Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems

1 INTRODUCTION . Self-organization of patterns that emerge from local rules is a pervasive phenomena in natural and artificial dynamical systems ( Ball , 1999 ) . It ranges from the formation of snow flakes , spots and rays on animal ’ s skin , to spiral galaxies . Understanding these processes has boosted progress in many fields , ranging from physics to biology ( Camazine et al. , 2003 ) . This progress relied importantly on the use of powerful and rich abstract computational models of self-organization ( Kauffman , 1993 ) . For example , cellular automata like Conway ’ s Game of Life ( GOL ) have been used to study the emergence of spatially localized patterns ( SLPs ) ( Gardener , 1970 ) , informing theories of the origins of life ( Gardener , 1970 ; Beer , 2004 ) . SLPs , such as the famous glider in GOL ( Gardner et al. , 1983 ) , are self-organizing patterns that have a local extension and can exist independently of other patterns . However , finding such novel self-organized patterns , and mapping the space of possible emergent patterns , has so far relied heavily on manual tuning of parameters and initial states . Moreover , the dependence of this exploration process on the human eye to identify “ interesting ” patterns is strongly limiting further advances . We formulate here the problem of automated discovery of a diverse set of self-organized patterns in such high-dimensional , complex dynamical systems . This involves several challenges . A first ∗Equal contribution . Source code and supplementary material at https : //automated-discovery.github.io/ challenge consists in determining a representation of patterns , possibly through learning , enabling to incentivize the discovery of diverse “ interesting ” patterns . Such a representation guides exploration by providing a measure of ( di- ) similarity between patterns . This problem is particularly challenging in domains where patterns are high-dimensional as in GOL . In such cases , scientists have a limited intuition about what useful features are and how to represent them . Moreover , low-dimensional representations of patterns are needed for human browsing and the visualization of the discoveries . Representation learning shall both guide exploration , and be fed by self-collected data . A second challenge consists in how to automate exploration of high-dimensional , continuous initialization parameters to discover efficiently “ interesting ” patterns , such as SLPs , with a limited budget of experiments . Sample efficiency is important to enable the later use of such discovery algorithms for physical systems ( Grizou et al. , 2020 ) , where experimental time and costs are strongly bounded . For example , in the continuous GOL used in this paper as a testbed , initialization consists in determining the values of a real-valued , high-dimensional 256×256 matrix besides 7 additional dynamics parameters . The possible variations of this matrix are too large for a simple random sampling to be efficient . More structured methods are needed . To address these challenges , we propose to leverage and transpose recent intrinsically motivated learning algorithms , within the family of population-based Intrinsically Motivated Goal Exploration Processes ( POP-IMGEPs - denoted simply as IMGEPs below , Baranes & Oudeyer ( 2013 ) ; Péré et al . ( 2018 ) ) . They were initially designed to enable autonomous robots to explore and learn what effects can be produced by their actions , and how to control these effects . IMGEPs self-define goals in a goal space that represents important features of the outcomes of actions , such as the position reached by an arm . This allows the discovery of diverse novel effects within their goal representations . It was recently shown how deep neuronal auto-encoders enabled unsupervised learning of goal representations in IMGEPs from raw pixel perception of a robot ’ s visual scene ( Laversanne-Finot et al. , 2018 ) . We propose to use a similar mechanism for automated discovery of patterns by unsupervised learning of a low-dimensional representation of features of self-organized patterns . This removes the need for human expert knowledge to define such representations . Moreover , a key ingredient for sample efficient exploration of IMGEPs for robotics has been the use of structured motion primitives to encode the space of body motions ( Pastor et al. , 2013 ) . We propose to use a similar mechanism to handle the generation of structured initial states in GOL-like complex systems , based on specialized recurrent neural networks ( CPPNs ) ( Stanley , 2006 ) . In summary , we provide in this paper the following contributions : • We formulate the problem of automated discovery of diverse self-organized patterns in high-dimensional and complex game-of-life types of dynamical systems . • We show how to transpose POP-IMGEPs algorithms to address the associated joint challenge of learning to represent interesting patterns and discovering them in a sample efficient manner . • We compare various approaches to define or learn goal space representations for the sample efficient discovery of diverse SLPs in a continuous GOL testbed . • We show that an extension of a state-of-the-art POP-IMGEP algorithm , with incremental learning of a goal space using a deep auto-encoder , is equally efficient than a system pretrained on a hand-made database of patterns . 2 RELATED WORK . Automated Discovery in Complex Systems Automated processes have been widely used to explore complex dynamical systems . For example , evolutionary algorithms have been applied to search specific patterns or rules of cellular automata ( Mitchell et al. , 1996 ; Sapin et al. , 2003 ) . However , their objective is to optimize a specific goal instead of discovering a diversity of patterns . Another line of experiments represent active inquiry-based learning strategies which query which set of experiments to perform to improve a system model , i.e . a mapping from parameters to the system outcome . Such strategies have been used in biology ( King et al. , 2004 ; 2009 ) , chemistry ( Raccuglia et al. , 2016 ; Reizman et al. , 2016 ; Duros et al. , 2017 ) and astrophysics ( Richards et al. , 2011 ) . However , these approaches have relied on expert knowledge , and focused on automated opti- mization of a pre-defined target property . Here , we are interested to automatically discover and map a diversity of unseen patterns without prior knowledge of the system . An exception is the concurrent work of Grizou et al . ( 2020 ) , which showed how a simple POP-IMGEP algorithm could be used to automate discovery of diverse patterns in oil-droplet systems . However , it used a low-dimensional input space , and a hand-defined low-dimensional representation of goal spaces , identified as a major limit of the system . Intrinsically motivated learning Intrinsically-motivated learning algorithms ( Baldassarre & Mirolli , 2013 ; Baranes & Oudeyer , 2013 ) autonomously organize an agent ’ s exploration curriculum in order to discover efficiently a maximally diverse set of outcomes the agent can produce in an unknown environment . They are inspired from the way children self-develop open repertoires of skills and learn world models . Intrinsically Motivated Goal Exploration Processes ( IMGEPs ) ( Baranes & Oudeyer , 2013 ; Forestier et al. , 2017 ) are a family of curiosity-driven algorithms developed to allow efficient exploration of high-dimensional complex real world systems . Population-based versions of these algorithms , which leverage episodic memory , hindsight learning , and structured dynamic motion primitives to parameterize policies , enable sample efficient acquisition of high-dimensional skills in real world robots ( Forestier et al. , 2017 ; Rolf et al. , 2010 ) . The discovered repertoires of diverse behaviors can be reused to solve hard exploration downstream tasks ( Colas et al. , 2018 ; Conti et al. , 2018 ) . Recent work ( Laversanne-Finot et al. , 2018 ; Péré et al. , 2018 ) studied how to automatically learn the goal representations with the use of deep variational autoencoders . However , training was done passively and in an early stage on a precollected set of available observations . Recent approaches ( Nair et al. , 2018 ; Pong et al. , 2019 ) introduced the use of an online training of variational autoencoders VAEs to learn the important features of a goal space similar to the methods in this paper . However , these approaches focused on the problem of sequential decisions in MDPs , incurring a cost on sample efficiency . This problem is observed in various intrinsically motivated RL approaches ( Bellemare et al. , 2016 ; Burda et al. , 2019b ) . These approaches are orthogonal to the automated discovery framework considered here with independent experiments allowing the use of memory-based sample efficient methods . A related family of algorithms in evolutionary computation is novelty search ( Lehman & Stanley , 2008 ) and quality-diversity algorithms ( Pugh et al. , 2016 ) , which can be formalized as special kinds of population-based IMGEPs . Representation learning We are using representation learning methods to learn autonomously goal spaces for IMGEPs . Representation learning aims at finding low-dimensional explanatory factors representing high-dimensional input data ( Bengio et al. , 2013 ) . It is a key problem in many areas in order to understand the underlying structure of complex observations . Many state-of-the-art methods ( Tschannen et al. , 2018 ) have built on top of Deep VAEs ( Kingma & Welling , 2013 ) , using varying objectives and network architectures . However , studies of the interplay between representation learning and autonomous data collection through exploration of an environment have been limited so far . 3 ALGORITHMIC METHODS FOR AUTOMATED DISCOVERY . 3.1 POPULATION-BASED INTRINSICALLY MOTIVATED GOAL EXPLORATION PROCESSES . An IMGEP is an algorithmic process generating a sequence of experiments to explore the parameters of a system by targeting self-generated goals ( Fig . 1 ) . It aims to maximize the diversity of observations from that system within a budget of N experiments . In population-based IMGEPs , an explicit memory of the historyH of experiments and observations is used to guide the process . The systems are defined by three components . A parameter space Θ corresponding to the controllable system parameters θ . An observation space O where an observation o is a vector representing all the signals captured from the system . For this paper , the observations are a time series of images which depict the morphogenesis of activity patterns . Finally , an unknown environment dynamic D : Θ→ O which maps parameters to observations . To explore a system , an IMGEP uses a goal space T that represents relevant features of its observations , computed by an encoding function ĝ = R ( o ) . For an exploration of patterns , such features may describe their form or extension . The exploration process iterates N times through : 1 ) sample a goal from a goal sampling distribution g ∼ G ( H ) ; 2 ) infer corresponding parameter θ using a parameter sampling policy Π = Pr ( θ ; g , H ) ; 3 ) roll-out an experiment with θ , observe outcome o , compute encodingR ( o ) ; 4 ) store ( θ , o , R ( o ) ) in historyH . The parameter sampling policy Π and in some cases the goal sampling distributionG take into account previous explorations stored in history H. Therefore , the history is first populated through exploring Ninit randomly sampled parameters . After this initial phase the described intrinsically motivated goal exploration process starts . Different goal and parameter sampling mechanisms can be used within this architecture ( Baranes & Oudeyer , 2013 ; Forestier & Oudeyer , 2016 ) . In the experiments below , goals are sampled uniformly over a hyperrectangle defined in T . The hyperrectangle is chosen large enough to allow a sampling of a large goal diversity . The parameters are sampled by 1 ) given a goal , selecting the parameter from the history whose corresponding outcome is most similar in the goal space ; 2 ) then mutating it by a random process .

Summary: The paper uses the continuous Game of Life as a testing ground for algorithms that discover diverse behaviors. The problem is interesting, under-explored, and rich. The  combines a variety of interesting ideas including compositional pattern producing networks (CPPNs) to learn structured primitives. Although the authors do propose formal measures of behavioral diversity and so show performance improvements, at the end of the day this work, like much empirical work on generative adversarial networks, is drifting towards art -- where performance is ultimately judged by human eyes rather than quantiative metrics.