Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems

Abstract: In many complex dynamical systems, artificial or natural, one can observe self-organization of patterns emerging from local rules. Cellular automata, like the Game of Life (GOL), have been widely used as abstract models enabling the study of various aspects of self-organization and morphogenesis, such as the emergence of spatially localized patterns. However, findings of self-organized patterns in such models have so far relied on manual tuning of parameters and initial states, and on the human eye to identify interesting patterns. In this paper, we formulate the problem of automated discovery of diverse self-organized patterns in such high-dimensional complex dynamical systems, as well as a framework for experimentation and evaluation. Using a continuous GOL as a testbed, we show that recent intrinsically-motivated machine learning algorithms (POP-IMGEPs), initially developed for learning of inverse models in robotics, can be transposed and used in this novel application area. These algorithms combine intrinsically-motivated goal exploration and unsupervised learning of goal space representations. Goal space representations describe the interesting features of patterns for which diverse variations should be discovered. In particular, we compare various approaches to define and learn goal space representations from the perspective of discovering diverse spatially localized patterns. Moreover, we introduce an extension of a state-of-the-art POP-IMGEP algorithm which incrementally learns a goal representation using a deep auto-encoder, and the use of CPPN primitives for generating initialization parameters. We show that it is more efficient than several baselines and equally efficient as a system pre-trained on a hand-made database of patterns identified by human experts.

Reviews and author responses:

We would like to thank Reviewer 2 for his time to provide us feedback, and for providing encouraging comments. 

First, as also stated in our response to R1, we agree (and apologize) that the Appendix in our initial submission was too long, not sufficiently well structured, and mixed materials that usefully complemented the main paper with materials that were much less useful. We have made a substantial rewrite and reduction of the Appendix, as detailed in response to R1.

Second, we agree that our results can be evaluated from two perspectives: 1) using quantitative metrics with statistical comparisons (as done in the main part of the paper); 2) using the human eye (which can be the eye of the scientist end-user of such a system, or a more “artistic” human eye). In order to enable readers to explore with their eyes, even more, the discoveries of the algorithms we study, we have built an interactive website to navigate all the patterns discovered by all goal exploration algorithms over many runs: https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/intrinsically-motivated-discovery/intrinsically-motivated-discovery.github.io/master/assets/media/tensorboard/projector_config.json

Our perspective is that the quantitative metrics are in practice very useful and relevant for several end-user applications we are now working on as next steps of this project. We are indeed right now working on using the methodology presented in this paper to enable 
1) bio-chemists to map the space of behaviours of certain complex biochemistry systems for which they do not have a good model (and even poor intuitive understanding), and then to optimize for target properties by leveraging the diversity of patterns found in the unsupervised discovery phase (previous papers on IMGEPs cited in our paper have shown that, in more traditional contexts of robotic control with hand-defined goals, finding a high diversity of behaviors enables to bootstrap very sample efficient optimization of a target behavior afterward)
2) neuroscientists to map the space of behaviours of complex neuro-muscular models and leverage the discovered diversity for later optimization of target behaviors.


> The paper refers to hand-designed goal spaces and talks, on p28, about “the statistical measures used to define the goal space”. At the same time, the analytic behavior space is [...] it is *not* referred to as hand-designed. At this point, the profusion of spaces and measures means that I am no longer sure what counts as hand-crafted or not. Please clarify. 

As we compare algorithms using different forms of goal spaces, and in addition we use an evaluation space (analytic behaviour space) for the evaluation and comparisons, there are indeed many spaces and our terminology in the initial text may indeed have complicated precise understanding. The analytic behavior space, as explained in 4.2 and detailed in section B.7.2 of the new Appendix, is the concatenation of a set of learned features (VAE over an “oracle” database of 42500 Lenia patterns) and of hand-designed features. The different kinds of spaces used by algorithms are described in section 4.3.


> The hypothesis on p34, sec E.4.2 that the VAE’s 8-dim bottleneck helps focus on animals rather than non-animals (which are differentiated more in terms of textures and details) is important and should be checked. 

As stated in the paper, this is indeed a speculative hypothesis we formulated as a result of analyzing the discoveries of the algorithms using learned VAEs. We formulated it in the Appendix because from our perspective it addresses a question that goes beyond the scope of the main 4 scientific questions we formulate in the main paper (section 5). We have begun thinking about how to make it more precise and test it, however, this is challenging as the bottleneck interacts with other factors (e.g. RGS with the same bottleneck does not incentivize animal discoveries, showing a bottleneck is not sufficient in itself + the VAEs learn representations that tend to produce blurred decoding). For this reason, we removed this specific hypothesis and only kept the discussion of the potential role of VAE’s difficulty in encoding sharp details. 


> Some of the decisions about what to check and vary are unclear. For example, section E.1 considers the effect of different initializations (“pytorch”, “xavier” and “kaiming”) [...]

The reason we initially included a comparison for different initializations was to ensure the RGS algorithm we used (randomly initialized VAE with no learning) was fairly compared to PGL and OGL (a poor initialization may project most data on the same embedding, e.g. through saturation). However, we agree this material can be omitted, which we have done in the new version of the Appendix, together with removing several other parts (e.g. VAE variants and HGS variants). 

We thank R4 for the detailed suggestions. We updated several parts of our paper accordingly. In detail:

>Section 3.1: It is not clear how the initial system state is established. In Section 3.1. the text states that 'parameters are randomly sampled and explored' before the process starts, but it is not clear why a random sampling is used and what this means for the subsequent sampling. Later in the text (3.3) it becomes more clear, but here this appears too unclear. 

We made some slight changes which hopefully improve readability.


>Section 3.1: "distribution over a hypercube in \mathcal{T} chosen to be large enough to bias exploration towards the frontiers of known goals to incentivize diversity." This sentence is not clear and needs more details. How is the distribution chosen exactly? 

Adapted the paragraph to be more exact:
“Different goal and parameter sampling mechanisms can be used within this architecture (Baranes & Oudeyer, 2013; Forestier & Oudeyer, 2016). In the experiments below, goals are sampled uniformly over a hyperrectangle defined in T. The hyperrectangle is chosen large enough to allow a sampling of a large goal diversity. The parameters are sampled by 1) given a goal, selecting the parameter from the history whose corresponding outcome is most similar in the goal space; 2) then mutating it by a random process.”


>Section 3.2 appears a bit repetitive and could be more concise. I don't think it is necessary >here to contrast manual vs learned features of the goal space. 

As unsupervised learning of goal spaces is one of the key points of the paper, we would like to emphasize it in this part by contrasting it to manually defined goal spaces.


>Section 3.2 (P3): the last sentence of this paragraph reads as if there exists no approaches for >VEAs in online settings. This should be toned down or backed up by a reference. 

We are unsure how the last sentence is implying this. Maybe this comment refers to a sentence in another section? We state in the beginning of Paragraph 3 of Section 3.2, that previous IMGEP approaches are not using online trained VAEs. We are not aware of any IMGEP approaches that use online trained VAEs. Yet, we mention some other methods that use online learned VAEs under the  “Intrinsically motivated learning” paragraph of the related work section (Sec. 2).


>Section 3.2: (last sentence): it is not clear how the history is used exactly to train the network. Which strategy is used to sample from the history of observations? 

We added extra information: “Importance sampling is used to give more weight to recently discovered patterns by using a weighted random sampler. It samples for 50% of the training batch samples patterns from the last K iterations and for the other 50% patterns from all other previous iterations”


> Section 3.3: What is meant by "The CPPNs are used of the parameters \{theta}"? The details provided after this sentence are not clear and need more details. 

Adapted the paragraph to give more context.


> Section 4.2: Please provide more details what "very large" dataset means. 

Added the number of patterns in the dataset: “... over a large dataset of 42500 Lenia patterns ...”


> Section 4.2: 'HGS algorithm' is not defined.

Adapted to not using the HGS abbreviation here.


> Section 5: It seems unnecessary to explain what t-SNE does as a method.

Removed.

We thank reviewer 1 for his time and effort, as well as for the encouraging comments. We especially appreciate the positive view on our introduction of a new problem framework, that may stimulate new and further research in machine learning, as it is for us a main objective (together with the study and comparison of particular algorithms). 

We would also like to apologize if the reading has been made difficult due to the length and/or structure of our Appendix. This was not intended, and in particular, it was not at all our aim to circumvent page limits. While with R1's review we realize we could have better organized and selected the material presented, we would like to explain our initial aim in structuring and building the paper, which was:
- Write a main paper where essential explanations (including presentation of a problem and context new to the readers) and main contributions were in the main paper, such that readers could understand their core aspects without reading the Appendix.
- Provide an Appendix that:
   1) includes full-page figures that provide complements to the main quantitative results (Figs. 3 and 4, main paper) through qualitative visual illustrations of examples of runs of the algorithm (Figs. 5-9, new version). 
   2) give all details enabling to reproduce all experiments (complementing the code)
   3) give all details enabling to understand all the techniques we use without needing to read the papers in the literature from which we reused them (e.g. Lenia’s complex system dynamics in section A, IMGEP implementation details, explanation of CPPNs, structure, and training of VAEs).
   4) show additional experimental results to show the robustness of our findings (e.g. showing that our results are robust to changes in the parameters of our diversity measure; or showing that the choice of hand-defined features used in HGS is fair by showing how it compares to other possible choices).
   5) show negative results for other algorithm variants we tried (e.g. different initialization methods for the randomized VAE (IMGEP-RGS) ), so that readers who would try to build on this work can benefit from this information.

As R1 and R2 remark, in the end this made a very long Appendix. As some papers accepted in previous editions of ICLR included similarly long Appendices, we did not try to reduce the Appendix at submission time. However, we agree that this should be improved. As a result, we updated significantly the Appendix by:

1) Removing large parts of the Appendix (we are thinking of providing this information rather on the Github of the code): 
- parts which were rather tutorials and summaries of other papers (e.g. non-essential explanations of the Lenia system, CPPNs or VAEs)
- parts presenting algorithm variants and hyperparameters we tried but which did not show good performances (HGS variants, VAE variants)
- some parts presenting an analysis redundant with the main paper
2) Summarizing many other parts to keep only the essential information
3) Structuring the Appendix in a clearer way:
Section A: Additional figures and results
Section B: Implementation details and hyperparameters (with a table of contents)

As a result, the new Appendix is now 19 pages shorter.

We did not make significant modifications to the main paper as we think it already provides the main results (we updated links to the Appendix trying to enable a more fluid reading and made several changes according to the suggestions of R4). We are of course open to suggestions from the reviewers if they think a particular additional figure or result is missing in the new Appendix.

> With regard to animal forms, it appears to me that Online goal learning harms the diversity of animal forms considerably compared to PGL and perhaps HGS. High-frequency spatial structure seems to be lost there. 
>  why non-animal types differ in PGL vs HGS, and why high-frequency spatial structure is lost in OGL

We agree that a difference between PGL and OGL is suggested by visual inspection of the example patterns in Figs. 27-31 (now Figs. 5-9). However, quantitative measures on Fig. 3 show that the diversity measure of OGL is as good as the one of PGL for animals (OGL and PGL use learned features), and way better than HGS (hand-engineered features).  A qualitative analysis of more patterns (these can be accessed through the database we provide now on the webpage) also shows this and that high frequency spatial structure in OGL is not lost. 


> why RGS produces the same kind of red linear patterns

It is true that the RGS shows a high abundance of “red linear patterns”. This is a result of the goal space of the RGS. It is random and discovered patterns are uniformly distributed in it (Fig. 10, new version). Thus, a goal exploration will result in a random selection of previous patterns and a mutation of them. Moreover, the initial random exploration of 1000 patterns results mainly in “red linear patterns” (this is visible from the patterns of the random exploration in Fig. 13 (new version)). Thus, during the goal exploration phase the RGS will mainly choose “red linear patterns” to mutate them. This produces in most cases again “red linear patterns”. As a result, “red linear patterns” are so abundant for RGS. This confirms that using a random latent representation for goals does not enable to organize exploration, as analyzed in Section 5. 


> Initial inspection reveals that hand-designed goal states produce the most interesting non-animal patterns. 
> Why HGS produces the distribution of pattern types in Figure 29, and why non-animal types differ in PGL vs HGS

We agree that subjectively non-animals found by HGS may have some dimensions of diversity not covered as well by the other algorithms. This is a result of their different goal spaces as pointed out under “How do goal space representations differ?” in Section 5 (p.9). The goal spaces of PGL and OGL represent well the form of small activity patterns (which are often animals). They do not represent well larger structures with interesting textures as seen in the HGS results. Thus they do not explore types of these patterns often resulting in a lower diversity of them.


> The results should NOT be shown just for the first repetition of the experiment but for all independent runs of the experiments, e.g averaged over 30 independent CPPN evolutions, for PGL, OGL, Random, and HGS! 

The core results (Fig. 3) include averages and standard deviations over 10 repetitions of each algorithm. We added now statistical results to the Figure showing that the algorithms produce statistically significant different diversities (Welch’s t-test, p < 0.01).

We thank Reviewer 4 for his time and efforts to review our paper.  We appreciate R4s comments and interest in our exploration results. There are some aspects of R4's comments we are not sure we fully understand, so we will be pleased to develop further our answers in case R4 would like us to address other points.


> The fitness is the closeness of a generated set of latents to a set of latents produced through one of several possible processes; hand-design, pretraining, or online training on previously generated CA settings. 

We would like to concisely provide two precisions:

1) The goal exploration algorithms we study generate a target uniform distribution of goals in a space of latent pattern features. These features are either learned or hand-engineered. From this generated distribution of goals, they try to find a distribution of parameters of the complex system (starting state+rules) that produces patterns covering well the target distribution of goal patterns. This is achieved through the dynamics of the POP-IMGEP algorithms, by iteratively sampling a goal (= a latent vector in case of learned goal features) and searching for the system parameters that approach that goal closest, leveraging all discoveries made so far. 

2) The quantitative measure used to evaluate our algorithms is a measure of diversity defined as the number of bins discovered in an evaluation space only known by the experimenter
. The dimensions of this evaluation space are a concatenation of hand-defined features and features of a learned embedding. The embedding is learned using a database with a large number of patterns found by all algorithms during all experiments. This measure of evaluation is only known and used by us, but it is not known by the individual exploration algorithms (as it uses a form of oracle knowledge to assess the discoveries the algorithms could make in principle).


> The core results are in Figures 27 to 31 in an appendix. 

Figs. 27-31 (now Figs. 5-9) are visualizations of particular examples of patterns discovered by the algorithms. For us the core results of the paper are the systematic quantitative measures presented in Fig. 3 and Fig. 4 (p. 8-9). Fig. 3, in particular, shows the average and standard deviation of the evolution of the diversity measure for several classes of patterns (all, animals and non-animals). These averages and standard deviations show the high-robustness of IMGEP-OGL and IMGEP-PGL to achieve the highest diversity in all classes (the low value of the standard deviation shows the high stability of these algorithms). 
Therefore, we believe that Figs. 27-31 (now Figs. 5-9), like the video on the accompanying web site, are complements to help readers visualize the kind of patterns that are discovered.


> ... in Figures 27 to 31 in an appendix. Initial inspection reveals that hand-designed goal states produce ... high frequency spatial structure is lost in OGL.

This qualitative analysis from R4 is made from looking at the examples of Figs. 27-31 (now Figs. 5-9). As explained above, the aim of these figures is to enable readers to have a visual sense of what "animals", "non-animals" and "dead" patterns look like, but they do not aim to be a way to quantitatively rank and compare the algorithms (Figs. 3 and 4 do this instead with objective statistical measures). As we aimed to introduce a novel scientific problem with this paper, we decided to focus on robust quantitative macroscopic measures of diversity within these different classes, which could be used as a basis for further investigations and comparisons with other algorithms (Figs 3 and 4). 

However, we agree that visual intuitions like the ones formulated by R4 through observing the discovered patterns with "human eyes" could help guiding the design of novel quantitative measures in future work. In order to enable readers to forge their own intuitions and possibly design new measures from them, we have now released a dataset of all discovered patterns: 
https://drive.google.com/file/d/1ZhVG2_uTLaT4SMqj0wKTKn568Y2XaypU/view?usp=sharing
Moreover, we released an interactive website enabling to view all discovered patterns projected into their goal spaces for all goal exploration algorithms and experimental repetitions: https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/intrinsically-motivated-discovery/intrinsically-motivated-discovery.github.io/master/assets/media/tensorboard/projector_config.jsonLINK 

The paper uses the continuous Game of Life as a testing ground for algorithms that discover diverse behaviors. The problem is interesting, under-explored, and rich. The  combines a variety of interesting ideas including compositional pattern producing networks (CPPNs) to learn structured primitives. Although the authors do propose formal measures of behavioral diversity and so show performance improvements, at the end of the day this work, like much empirical work on generative adversarial networks, is drifting towards art -- where performance is ultimately judged by human eyes rather than quantiative metrics. 

Comments:
The paper refers to hand-designed goal spaces and talks, on p28, about “the statistical measures used to define the goal space”. At the same time, the analytic behavior space is also defined in terms of statistical measures, but it is *not* referred to as hand-designed. At this point, the profusion of spaces and measures means that I am no longer sure what counts as hand-crafted or not. Please clarify.
The hypothesis on p34, sec E.4.2 that the VAE’s 8-dim bottleneck helps focus on animals rather than non-animals (which are differentiated more in terms of textures and details) is important and should be checked. 
Some of the decisions about what to check and vary are unclear. For example, section E.1 considers the effect of different initializations (“pytorch”, “xavier” and “kaiming”). The choice of initialization is important mostly to do with improving gradients to improve the rate of convergence (or convergence at all) in deep nets. It’s not clear why initializations are an parameter to vary when considering diversity of solutions. Or, rather, why initializations are more interesting to consider various other architectural considerations. More broadly, looking at Fig 17, the x-axis doesn’t make much sense. The experiments along the x-axis vary according to initialization, but also according to the nature of the goal space and other features. It seems a bit incoherent. 

Overall I think this is a good paper. The results are novel and even better, they are fun. However, the paper is extremely long, and it feels as though the authors have to some extent lost control of the material. I could add more comments but TL;DR it needs a lot of editing and pruning. 


The focus of the presented paper is on formulating the automated discovery of self-organized patterns in high-dimensional dynamic systems. The introduced framework uses cellular automata (game of life) as a testbed for experimentation and evaluation and existing machine learning algorithms (POP-IMGEPs). The goal of the paper is to show that these algorithms can be used to discover and represent features of patterns. Moreover, an extension of SOTA algorithms is introduced and several approaches to define goal space representations are compared. 

Overall, I have the impression this is an interesting paper that could be accepted to ICLR. The idea of applying IMGEPs to explore parameters of a dynamic system is novel and interesting, which could also simulate further research in this field. Furthermore, the paper well-written, technically sound, and the results are interesting. The overall contribution of the paper is in applying IMGEP algorithms to exploring parameters of dynamic systems and in comparing different algorithms along with an extensive set of experiments. As a point of criticism, a lot of (interesting) material was pushed to the Appendix. Resolving the references makes reading the paper harder. Moreover, given that this paper has more than 35 pages appendix material, it seems this work would better be suited for a journal as for a conference. There is a reason for papers to have a page limit and this work circumvents this limit by presenting a lot of additional material. Therefore, I am not willing to strongly support this work. 

Specific Comments: 

- Section 3.1: It is not clear how the initial system state is established. In Section 3.1. the text states that 'parameters are randomly sampled and explored' before the process starts, but it is not clear why a random sampling is used and what this means for the subsequent sampling. Later in the text (3.3) it becomes more clear, but here this appears too unclear.
-  Section 3.1: "distribution over a hypercube in \mathcal{T} chosen to be large enough to bias exploration towards the frontiers of known goals to incentivize diversity." This sentence is not clear and needs more details. How is the distribution chosen exactly?
- Section 3.2 appears a bit repetitive and could be more concise. I don't think it is necessary here to contrast manual vs learned features of the goal space.
- Section 3.2 (P3): the last sentence of this paragraph reads as if there exists no approaches for VEAs in online settings. This should be toned down or backed up by a reference.
- Section 3.2: (last sentence): it is not clear how the history is used exactly to train the network. Which strategy is used to sample from the history of observations?
- Section 3.3: What is meant by "The CPPNs are used of the parameters \{theta}"? The details provided after this sentence are not clear and need more details. 
- Section 4.2: Please provide more details what "very large" dataset means.
- Section 4.2: 'HGS algorithm' is not defined.
- Section 5: It seems unnecessary to explain what t-SNE does as a method. 

The paper describes an algorithm to find diverse patterns in Lenia (a continuous CA system) by using a CPPN to generate initial states, and a stochastic exploration algorithm to mutate parameters of the CPPN + CA parameters. The fitness is the closeness of a generated set of latents to a set of latents produced through one of several possible processes; hand-design, pretraining, or online training on previously generated CA settings. 

The core results are in Figures 27 to 31 in an appendix. Initial inspection reveals that handdesigned goal states produce the most interesting non-animal patterns. With regard to animal forms, it appears to me that Online goal learning harms the diversity of animal forms considerably compared to PGL and perhaps HGS. High frequency spatial structure seems to be lost there. 

I would like to see a further analysis of maybe 10000s of such images generated, and an understanding of exactly why RGS produces the same kind of red linear patterns, and why HGS produces the distribution of pattern types in Figure 29, and why non-animal types differ in PGL vs HGS, and why high frequency spatial structure is lost in OGL. How robust are these over many runs? The results should NOT be shown just for the first repetition of the experiment but for all independent runs of the experiments, e.g averaged over 30 independent CPPN evolutions, for PGL, OGL, Random, and HGS! 



Meta-review: The authors introduce a framework for automatically detecting diverse, self-organized patterns in a continuous Game of Life environment, using compositional pattern producing networks (CPPNs) and population-based Intrinsically Motivated Goal Exploration Processes (POP-IMGEPs) to find the distribution of system parameters that produce diverse, interesting goal patterns.

This work is really well-presented, both in the paper and on the associated website, which is interactive and features source code and demos. Reviewers agree that it’s well-written and seems technically sound. I also agree with R2 that this is an under-explored area and thus would add to the diversity of the program.

In terms of weaknesses, reviewers noted that it’s quite long, with a lengthy appendix, and could be a bit confusing in areas. Authors were responsive to this in the rebuttal and have trimmed it, although it’s still 29 pages. My assessment is well-aligned with those of R2 and thus I’m recommending accept. In the rebuttal, the authors mentioned several interesting possible applications for this work; it’d be great if these could be included in the discussion. 

Given the impressive presentation and amazing visuals, I think it could make for a fun talk.


Decision: accept-talk