Deep Innovation Protection
Authors: Sebastian Risi
Kenneth O. Stanley
Venue:  (2020)
Keywords: Neuroevolution
innovation protection
world models
genetic algorithm

Abstract: Evolutionary-based optimization approaches have recently shown promising results in domains such as Atari and robot locomotion but less so in solving 3D tasks directly from pixels. This paper presents a method called Deep Innovation Protection (DIP) that allows training  complex world models end-to-end for such 3D environments. The main idea behind the approach is to employ multiobjective optimization to temporally reduce the selection pressure on specific components in a world model, allowing other components to adapt. We investigate the emergent representations of these evolved networks, which learn a model of the world without the need for a specific forward-prediction loss. 

Under review as a conference paper at ICLR 2020
DEEP INNOVATION PROTECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Bvolutionary-based optimization approaches have recently shown promising re- sults in domains such as Atari and robot locomotion but less so in solving 3D tasks directly from pixels. This paper presents method called Deep Innovation Protection (DIP) that allows training complex world models end-to-end for such 3D environments. The main idea behind the approach is to employ multiobjective optimization to temporally reduce the selection pressure on specific components in a world model, allowing other components to adapt. We investigate the emer- gent representations of these evolved networks, which learn model of the world without the need for a specific forward preeadctpreiiiionnno loss.
INTRODUCTION
The ability of the brain to model the world arose from the process of evolution. It evolved because helped organisms to survive and strive in their particular enviromments and not because such forward prediction was explicitly optimized for. In contrast to the emergent neural representations in nature, current world model approaches are often directly rewarded for their ability to predict future states of the environment (Schmidhuher) 1990; Ha & Schmidhuber) 2018 Hafner et al.) 2018)] Wayne et al. 2018). While it is undoubtedly useful be able explicitly encourage a model to predict what will happen next, in this paper we are interested in what type of representations can emerge from the less directed process of artificial evolution and what ingredients might be necessary to encourage the emergence of such predictive abilities.
In particular, we are building on the recently introduced world model architecture introduced by Ha & Schmidhuber (2018). This agent model contains three different components: (1) a visual module, mapping high-dimensional inputs to a lower-dimensional representative code, (2) an LSTM- memory component, and (3) controller component that takes input from the visual and memory module to determine the agent's 's next action. In the original approach, each component of the world model was trained separately and to perform different and specialised function, such as predict- ing the future. While Risi & Stanley (2019) demonstrated that these models can also be trained end-to-end through a population-base performance, the approach was only applied to the simpler 2D car racing domain and it is an open question how such approach will scale the more complex 3D VizDoom task that first validated the effectiveness of the world model approach.
genetic algorithm (GA) that exclusively optimizes for final
Here we show that simple genetic algorithm fails to find a solution to solving the VizDoom task and ask the question what are the missing ingredients necessary to encourage the evolution of more powerful world models? The main insight this paper is that can view the optimization of heterogeneous neural network (such as world models) as a co-evolving system of multiple different sub-systems. The other important insight is that representational innovations discovered in one sub- system (e.g. the visual system learns to track moving objects) require the other sub-systems to adapt. In fact, the other systems are not given time to adapt, such innovation will likely initially have an adversarial effect on overall performance!
In order to optimize such co-evolving heterogeneous neural systems, we propose to reduce the se- lection pressure on individuals whose visual or memory system was recently changed, given the controller component time to readapt. This Deep Innovation Protection (DIP) approach is inspired by the recently introduced morphological innovation protection method of Cheney et al. (2018), which allows for the scalable co-optimization of controllers and robot body plans. Our approach is able to find solution to the VizDoom: Take Cover task, which first solved by the original

---

Under review as a conference paper at ICLR 2020
world model approach (Ha & Schmidhuber 2018). More interestingly, the emergent world models learned to predict events important for the survival of the agent. even though they were not explicitly trained predict the future. Additionally, allows evolution to carefully orchestrate the training of the components these heterogeneous ar- chitectures. We hope this work inspires more research that focuses on investigating representations emerging from approaches that do not necessarily only rely on gradient-base optimization.
investigates into the training process show that DIP
DEEP INNOVATION PROTECTION
The hypothesis in this paper is that to optimize complex world models end-to-end for more complex tasks requires each of its components to be carefully tuned to work well together. For example, an innovation in the visual or memory component could adversely impact the controller component, leading to reduced performance. However, in the long such innovation could allow an individual to outperform its predecessors.
The agent model is based the world model approach introduced by Ha & Schmidhuber (2018). The network includes a sensory component, implemented as a variational autoencoder (VAE) that compresses the high-dimensional sensory information into smaller 32-dimensional representative code (Fig. D.. This code is fed into memory component based on a recurrent LSTM (Hochreiter & Schmidhuber 1997). which should predict future representative codes based on previous informa- tion. Both the output from the sensory component and the memory component are then fed into a controller that decides on the action the agent should take each time step. Following Risi Stan- ley (2019). we train these world models end-to-end with a genetic algorithm, in which mutations add Gaussian noise to the parameter vectors of the networks: 0' + ae, where e ~ I0,
The approach introduced in this paper aims to train heterogeneous neural systems end-to-end by temporally reducing the selection pressure on individuals with recently changed modules, allowing other components adapt. For example, in case of the world model, in which mutation can either affect the VAE, MDN-RNN or controller, selection pressure should be reduced if a mutation affects the VAE MDN-RNN, giving the controller time to readapt to the changes in the learned representation. Inspired by the multi-objective morphological innovation protection introduced by Cheney al.] (2018), employ the well-known multiobjective optimization approach NSGA-II (Deb et al., 2202). in which a second "age" objective keeps track of when a mutation changes either the VAE or the MDN-RNN. Every generation an individual's s age is increased by 1, however, if mutation changes the VAE or MDN-RNN, this age objective is set to zero (lower is better). Therefore, if two neural networks reach the same performance (i.e. the same final reward), the one that had less time to adapt (i.e. whose age is lower) would have higher chance of heing selected for the next generation. The second objective is the accumulated reward received during an episode. Pseudocode of the approach applied to world models is shown in Algorithm]] [
is important to note that this approach different to the traditional usage of ""ge"" in multi- objective optimization, in which age is used to increase diversity and keeps track of how long indi- viduals have been in the population (Hornyy] 22066] (Schmidt & Lipson, (20IT). In the approach in this paper, age counts how many generations the controller component of individual had time to adapt to an unchanged visual and memory system.
In the original world model approach the visual and memory component were trained separately and through unsupervised learning based on data from random rollouts. In this paper they are optimized through genetic algorithm without evaluating each component individually. In other words, the VAE is not directly optimized to reconstruct the original input data and neither is the memory component optimized to predict the next time step; the whole network is trained in end- to-end fashion. Here we are interested in what type of neural representations emerge by themselves that allow the agent to solve the given task.
3 EXPERIMENTS
Following the original world model approach (Ha & Schmidhuber, 2018). in the experiments pre- sented here an agent is trained to solve the car racing tasks, and the more challenging VizDoom task (Kempka et al. 2016) from 64x64 RGB pixel inputs (Fig. 2 In the continuous control task
2

---

Under review as a conference paper at ICLR 2020
Algorithm Deep Innovation Protection for World Models
1:
Generate random population of size N with age objectives set to 0 for generation = 1 to i do
2:
3:
for Individual in Population do
Objective[1] = age
5:
Objective[2] = accumulated task reward
6:
Increase individual's s age by end for
7:
Assign ranks hased Pareto fronts
Generate set of non-dominated solutions
10:
Add solutions, starting from first front, until number solution N
11:
Generate child population through binary tournament selection and mutations
12: Reset age to for all individuals whose VAE or MDN-RN MON-RNN was mutated
13: end for
Environment
action
Reuu com 32x4
Rob corv
125
nome 253
DONTROLLES ER
EXRN 22
MEMORY
31x31x32
14x64
Observations 64 64
action
VISUAL COMPONENT (VAE Encoder)
Figure Agent Model. The agent model consists of three modules. visual component (the encoder of variational autoencoder) produces a latent code each time step f, which is concate- nated with the hidden state h of the LSTM-based memory component that takes 2; and previously performed action d input. The combined vector (zr,hy h2) is input into the controller component to determine the next action the agent. this paper, the agent model is trained end-to-end with multiobjective genetic algorithm.
CarRacing-vy v0 (Klimov 2016) the agent is presented with a new procedurally generated track every episode, receiving a reward of -0.1 every frame and reward of +100/N for each visited track tile, where N the total number of tiles in the track. The network controlling the agent (Fig. D has three outputs to control left/right steering, acceleration and braking. Further details on the network model. which is the same for both domains, can be found in the Appendix. In the VizDoom: Take Cover task the agent has to try to stay alive for 2,100 timesteps. while avoiding fireballs shot at it by strafing to the left or the right. The agent receives a reward for every frame is alive. The net- work controlling the agent has one output to control left (a -0.3) and right strafing (a 0.3), or otherwise standing still. In this domain, solution is defined as surviving for over 750 timesteps, averaged across 100 random rollouts (Kempka al. 2016)_
Following the NSGA-II approach, individuals for the next generation are determined stochastically through 2-way tournament selection from the 50% highest ranked individuals in the population (Algorithm T). No crossover operation was employed. The population size was 200. Because of the randomness in this domain, evaluate the top three individuals of each generation additional time to get a better estimate of the true elite. We compare a total of four different approaches:
1. Deep innovation protection (DIP): The age objective is reset to zero when either the VAE MDN-RNN is changed. The idea behind this approach is that the controller should get time to readapt if one of the components that precede it in the network change.
2. Controller innovation protection: Here the age objective is set to zero if the controller changes. This setting tests if protecting components upstream can effective in optimizing heterogeneous neural models.

---

Under review as a conference paper at ICLR 2020
PIP
(a) Original
Scaled (64 64
Original
(d) Scaled (64 x
Figure 2: In the CarRacing-vD task the agent has to learn to drive across many procedu- rally generated tracks as fast as possible from 64 x64 RGB color images. In the VizDoom: Take Cover domain the agent has to learn to avoid fireballs and to stay alive as long possible.
10
20.0
VAEE
- Champion Age
Average Pogulstiio Age
Contrsllr
Controller
175
Sansan DA renevation protectir;
Conturllr
15.0
2.0
12.5
1200
10.0
0900
2.0
7.5
750
5.0
10'
2.5
0.0
100
125
150
200
(a)
(b)
Figure VizDoom Evolutionary Training. Shown is (a) mean performance over generations gether with one standard error. For representative run of DIP (b), we plot the euclidean distances of the weights of the intermediate solutions (i.e. individuals with the highest task reward discovered so far) compared to the final solution in addition to their age and the average population age.
3. MDN-RNN & Controller innovation protection:
This setup is the same as the controller protection approach but we additionally reset age if the MDN-RNN changes. On average, this treatment will reset the age objective as often as DIP.
4. Random age objective: In this setup the age objective is assigned a random number be- tween [0, 20] at each evaluation. This treatment tests if better performance can be reached just through introducing more diversity the population.
5. Standard GA no innovation protection: In this non-multi-objecin setup, which is the same one as introduced in Risi & Stanley 2009 only the accumulated reward is taken into account when evaluating individuals.
For all treatments, a mutation has an equal probability to either mutate the visual, memory, or con- troller component of the network. Interestingly, while Risi & Stanley (2019) reported that this ap- proach performs similarly well to an approach that always mutates all components, we noticed that performs significantly worse in the more complicated VizDoom domain. This result suggests that the more complex the tasks, the more important it is to be able to fine-tune individual components in the overall world model architecture.
EXPERIMENTAL RESULTS
All results are averaged over ten independent evolutionary runs. In the car racing domain we find that there is no noticeable difference between an approach with and without innovation protection
We use the publicly avaible code from Risi & Stanley 2019 for our haseline (https//githuh.com/uee- researchggewoordd-mmeeee/
4

---

Under review as a conference paper at ICLR 2020
Time
Figure 4: Still frames of a learned policy: The evolved champion learned to primarily pay attention to the walls and fireballs, while ignoring the floor and ceiling. Interestingly the agent also seems to pay attention to the health and ammo indicator.
and both can solve the domain. However, in the more complex VizDoom task, the DIP approach that protects innovations in both VAE and MDN-RNN, outperforms all other approaches (Fig. 3p. The approach is able to find a solution to the task, effectively avoiding fireballs and reaching an average score of 824.33 (sd=491.55) calculate perturbation-bant environment the agent is paying most attention to (Fig. 4.. The idea behind perturbation-base saliency maps is to measure to what extent the output of the model changes if parts of the input image are altered (Greydanus et al. 2017). Not surprisingly, the agent learned to pay particular attention to the walls, fireballs, and the position of the monsters.
To better understand the decision-making process of the agent, we saliency maps (see Appendix for details) to determine the parts of the
The better performance of the random age objective compared to no innovation protection suggests that increasing diversity in the population improves performance but less effectively than selectivity resetting the age objective as in DIP. Interestingly, the controller and the MDN-RNN&Controllee protection approach perform less well, confirming our hypothesis that it is important to protect innovations upstream in the network for downstream components.
Learned Representations
We further investigate what type of world model can emerge from an evolutionary process that does not directly optimize for forward prediction reconstruction loss. To gain insights into the learned representations we employ the t-SNE dimensionality reduction technique (Maaten & Hinton 2008), which has proven valuable for visualizing the inner workings of deep neural networks (Such) et al. 2018 Mnih et al.) 2015). We are particularly interested in the information contained in the compressed 32-dimensional vector of the VAE and the information stored in the hidden states of the MDN-MNNN RNN (which are both fed into the controller that decides on the agent's 's action). Different combinations of sequences of these latent vectors collected during one rollout are visualized in two dimensions in 5 Interestingly, while the 32-dimensional vector from the VAE does not contain enough information to infer the correct action, either the hidden state alone or combination with results in grouping the states into two distinct classes (one for moving left and one for moving right). The temporal dimension captured by the recurrent network proves invaluable in deciding what action is best. For example, not getting stuck in a position that makes avoiding incoming fireballs impossible, seems to require a level of forward prediction by the agent. To gain a deeper understanding of this issue we look more closely into the learned temporal representation next.
Learned Forward Model Dynamics In order to analyze the learned temporal dynamics of the for- ward model, we are taking closer look at the average activation I; of all 256 hidden nodes at time step t and how much they differ from the overall average across all time steps X NV ΣN F5. The variance of Fe thus calculated as a; = (X )2, and normalized to the range [0. 1] before plot- ting. The hypothesis is that activation levels far from the mean might indicate a higher importance and should have a greater impact on the agent's controller component. In other words, they likely indicate critical situations in which the agent needs to pay particular attention to the predictions of the MDN-RNN Fig. 6 depicts frames from the learned policies in two different situations, which shows that the magnitude of LSTM activations model does not seem to react to fireballs by themselves but instead depends on the agent being in the line of impact of approaching fireball, which is critical information for the agent stay alive.
closely tied to specific situations. The forward
5

---

Under review as a conference paper at ICLR 2020
(a) L-SNE with 2+hidden
(b) I-SNE with Z alone
(c) I-SNE with hidden alone
Figure 5: I-SNE mapping of the latent+hidden vector (a), latent vector alone (b), and hidden vector alone (c). While the compressed latent vector is not enough to infer the correct action (b), the hidden LSTM vector alone contains enough information for the agent decide on the correct action (c). Color legend: red = strafe left, blue strafe right, black no movement.
(a)
(b)
Figure 6: Average activation levels of LSTM in two different situations. For visualization purposes only, game images colored more or less blue depending the LSTM activation values. The evolved forward model seems to have learned to predict a firehall would hit the agent at the current position. In (a) the agent can take advantage of that information avoid the fireball while the agent does not have enough time to get out of the way in situation (b) and gets hit. Shown on top the actions the agent takes in each frame.
Evolutionary Innovations addition to analyzing the learned representations of the final net- works, it is interesting study the different stepping stones evolution discovered to solve the Viz- Doom task. We show one particular evolutionary run in Fig. 7 with other ones following similar progressions. In the first 30 generations the agent starts learn to pay attention fireballs but only tries avoiding them by either standing still or moving to the right. A jump in performance happens around generation 34 when the agent starts discover moving either the left right; however, the learned representation between moving left or right is not well defined yet. This changes around generation 56, leading to another jump in fitness and some generations of quick fine-tuning later the agent is able to differentiate well between situations requiring different actions, managing to survive for the whole length of the episode. Motivated by the approach of Raghu et al. (2017) to analyse the gradient descent- based training of neural networks, we investigate the weight distances of the world model components of the best-perfoe peat-rrrffoingggg networks found during training to the final solution representation (Fig. 3D). The VAE is the component with the steepest decrease in distance with a noticeable jump around generation 60 due to another lineage taking over. The MDN-RR -NNN optimized slowest, which is likely due the fact that the correct forward model dynamics more complicated to discover than the visual component. These results suggest that DIP able to orchestrate the training of these heterogeneous world model architectures in automated way.

---

Under review as a conference paper at ICLR 2020
1750
1530
150
Generation
Gonadon
Generation
Generation
Generstion
Generation 56
Figure 7: Development of the evolved representation over generations. Shown are t-SNE mappings of the 288-dimensional vectors (32-dimensional latent vectors 256-dimensional hidden state vec- tor) together with saliency maps of specific game situations. Early on in evolution the agent starts paying attention to the fireballs (generation 24) but only moves to the right (blue) or stands still (black). Starting around generation 34 the agent starts to move to the left and right, with the saliency maps becoming more pronounced. From generation
on the compressed learned representation (latent vector+hidden state vector) allows the agent to infer the correct action almost all the time. The champion discovered in generation 145 discovered visual encoder and LSTM mapping that shows clear division for left and right strafing actions.
RELATED WORK
A variety of different RL algorithms have recently been shown to work well on a diverse set of problems when combined with the representative power of deep neural networks (Mnih al. 2015
Schulman et al. 2013 2017). While most approaches are based on variations of Q-learning Mmih al. (2015) or policy gradient methods (Schulman al., 2015 2017). recently evolutionary- methods have emerged as a promising alternative for some domains (Such et al.) [017] Salimans ctt al. 2017). Salimans et al. Salimans et al. (0017) showed that a type of evolution strategy (ES) can reach competitive performance in the Atari benchmark and at controlling robots in MuloCo. Additionally, Such et al. [Such et al. (2017) demonstrated that a simple genetic algorithm is in fact able to reach similar performance to deep RL methods such as DQN A3C. Earlier approaches that evolved neural networks for RL tasks worked well in complex RL tasks with nover-dimensional input spaces (Stanley & Mikkkulainen 2002
Floreano al. 2008 Risi & Togelius 2017).

---

Under review as a conference paper at ICLR 2020
However, when trained end-to-end these networks are often still orders of magnitude simpler than networks employed for supervised learning problems (Justesen et al.) 2019) or depend on additional losses that responsible for training certain parts of the network (Wayne et al. 2018).
For complex agent models, different network components can be trained separately (Wahlström al. 2015 Ha & Schmidhuber 2018). For example, in the world model approach (Ha & Schmidhu- ber] 2018). the authors first train a variational autoencoder (VAE) on 10,000 rollouts from a random policy compress the high-dimensional sensory data and then train recurrent network to predict the next latent code. Only after this process is a smaller controller network trained to perform the actual task, taking information from both the VAE and recurrent network as input determine the action the agent should perform.
Evolutionary approaches solving 3D tasks directly from pixels has so far proven difficult although a few notable approaches exist. Koutnik et al. (2013) evolved an indirectly encoded and recurrent controller for car driving TORCS, which learned to drive based on raw 64 64 pixel image. The approach was based on an indirect encoding of the network's 's weights analogous to the JPEG compression in images. To scale to 3D FPS tasks, [Alvernaz & Togelius (2017) first trained an autoencoder in unsupervised way and then evolved the controller giving the compressed rep- resentation as input. In another approach. Poulsen et al. (2017) trained an object recognizer in a supervised way and then in separate step evolved a controller module. More recently, Lehman et al. (2018) introduced an approach called safe mutations, in which the magnitude of mutations to weight connections is scaled based on the sensitivity of the network's output to that weight. The safe mutations approach allowed the evolution of large-scale deep networks for simple 3D maze task and is complementary approach that could be combined with DIP in the future.
The approach introduced in this paper can be viewed as a form of diversity maintenance, in which selection pressure on certain mutated neural networks is reduced. Many other methods for encour- aging diversity (Mouret & Doncicux 2012) were invented by the evolutionary computation com- munity, such as novelty search (Lehman & Stanley 2008). quality diversity (Pugh al. 2016). or speciation (Stanley & Mlikkulliinen 2002). Using the concept of age to maintain diversity has a long history in evolutionary algorithms. Kubota & Fukuda (1997) first introduced the idea of using age in genetic algorithm in which individuals are removed from the population if they reach specific age limit. the Age-Layered Population Structure approach introduced by [Horbyy (2066), the population is segregated into different age layers and newly generated individuals are introduced into the youngest age layer to increase diversity. (Schmidt & Lipson (201I) combine the idea of age with a multi-objective approach, in which individuals are rewarded for performance and low age. Inspired by the morphological innovation approach (Cheney et al. 2018) and in contrast to previous approaches (Hornby) 2006)] Schmidt& & Lipson] 2011). DIP does not introduce new random individ- uals into the generation but instead resets the age of individuals whose sensory or memory system have been mutated. That is, it is not a measure of how long the individual has been in the population, as in the traditional usage of age in multi-objective optimization.
Approaches to learning dynamical models have mainly focused on gradient descenttbassed methods, with early work on RNNs in the 1990s (Schmidhuber) 1990).. More recent work includes PILCO (Deisenroth & Rasmussen (011), which is probabilistic model-based policy search method and Black-DROPS (Chatzilygeroudis et al., (017). which employs CMA-ES for data-efficient optimiza- tion of complex control problems. Additionally, interest has increased in learning dynamical mod- els directly from high- nigh-diimesionall pixel images for robotic tasks (Watter et al. 2015 Hafner et al.) 2018) and also video games (Guzdial et al. (2177). Work on evolving forward models has mainly fo- cused on neural networks that contain orders of magnitude fewer connections and
feature vectors (Norouzzadeh & Clune 2016) than the models in this paper.
DISCUSSION AND FUTURE WORK
The paper demonstrated that a world model representation for a 3D task can emerge under the right circumstances without being explicitly rewarded for it. To encourage this emergence, we intro- duced deep innovation protection, an approach that can dynamically reduce the selection pressure for different components in a heterogeneous neural architecture. The main insight is that when com- ponents upstream in the neural network change, such as the visual or memory system in a world model, components downstream need time adapt to changes in those learned representations.
8

---

Under review as a conference paper at ICLR 2020
The neural model learned to represent situations that require similar actions with similar latent and hidden codes (Fig. 5 and 7) Additionally, without a specific newward-predictioe loss, the agent learned to predict "useful" events that necessary for its survival (e.g. predicting when the agent is in the line-of-fire of a firehall). In the future it will be interesting to compare the differences and similarities of emergent representations and learning dynamics resulting from evolutionary and gradient descent-based optimization approaches (Raghu et al., 2017).
Interestingly, without the need for a variety of specialized learning methods employed in the original world model paper, a simple genetic algorithm augmented with DIP can not only solve the simpler 2D car racing domain (Risi & Stanley, 2019). but also more complex 3D domains such as VizDoom. That the average score across 100 random rollouts is lower when compared to the one reported in the original world model paper (824 compared to 1092) maybe not surprising: if random rollouts are available, training each component separately can results in higher performance. However, in more complicated domains, which random rollouts might not be able to provide all relevant experiences (e.g. a random policy might never reach a certain level), the proposed DIP approach could become increasingly relevant. An exciting future direction is to combine the end-to-end training regimen of DIP with the ability of training inside the world model itself (Ha & Schmidhuber 2018). However, because the evolved representation is not directly optimized to predict the next time step and only learns to predict future events that are useful for the agent's survival, is an interesting open question how such a different version of a hallucinate environment could be used for training.
A natural extension to this work is to evolve the neural architectures in addition the weights of the network. Searching for neural architectures in RL has previously only been applied to smaller networks (Risi & Stanley 2012, Stanley Miikkulainen 2002, Stanley et al., 2019 Gaier & Hap 2019 Risi & Togelius] 2017 Floreano et al.) 2008) but could potentially now be scaled to more
complex tasks. While our innovation protection approach is based on evolution, ideas presented here could also be incorporated in gradient descent-basse approaches that optimize neural systems with multiple interacting components end-to-end.
A APPENDIX
A.1 OPTIMIZATION AND MODEL DETAILS
The size of each population was 200 and evolutionary runs had a termination criterion of 200 gen- erations. The genetic algorithm was determined empirically and set to 0.03 for the experiments in this paper. The code for the DIP approach can be found at: [Removed for anonymous review).
Table 1: Number of parameters and training procedures. The visual component of the agent (see Fig. D is effectively only utilizing and evolving the encoder part of the VAE, which has 755,744 parameters. The decoder network is composed of four deconvolutional layers and has 3,592,803 parameters.
Model
#Params
WM Training Ha & Schmidhuber (2018)
GA Training
VAE
4,348,547
SGD epoch
MDN-RNN
384,071
SGD - 20 epochs
Pop size 200 Rollouts 1
Controller
867
CMA-ES Pop 64
Rollouts 16
An overview of the agent model is shown in Fig. I which employs the same architecture as the original world model approach Ha & Schmidhuber (2018). The sensory model is implemented as a variational autoencoder that compresses the high-dimensional input to latent vector 2. The VAE takes as input an RGB image of size 64 64 x 3, which is passed through four convolutional layers, all with stride 2. Details on the encoder are depicted in the visual component shown in Fig. [ where layer details are shown as: activation type (e.g. ReLU), number of output channels x filter size. The decoder, which is in effect only used to analyze the evolved visual representation, takes as input tensor of size x x 104 and processes it through four deconvolutional layers each with stride and sizes of 128 5, 64 x 32 x 6, and 32 x 6. The network's 's weights set using the default

---

Under review as a conference paper at ICLR 2020
350
5200
300
4300
850
100
2300
350
20
23
20
Age
Figure 8: Average reward across ages and number of individuals per age.
PyTorch initilisation (He initialisation He et al. (2015), with the resulting tensor being sampled from u(ound,,, bound), where bound fan_in
The memory model (Ha & Schmidhuber 2018) combines recurrent LSTM network with a mixture density Gaussian model network outputs, known a MDN-RNN (Ha & Eck, 2017 Graves 2013b). The network has 256 hidden nodes and models P (e. 2e. he). where a, the action taken by the agent at time t and h, is the hidden state of the recurrent network. Similar models have previously been used for generating sequences sketches (Ha & Eck 2017) and handwriting (Graves) 20) The controller component is simple linear model that directly maps and h; to actions: a₂ = We[zgh]]bb where We and b. are weight matrix and bias vector. the parameter counts of the different world model components and how they trained here and in the world model paper. The code for all the experiments available at: [removed for anonymous review].
A.2 INTERACTION BETWEEN REWARD AND OBJECTIVE
We performed analysis of the (1) cumulative reward per age and (2) the number of individuals with a certain age averaged across all ten runs and all generations (Fig. 8). While the average reward increases with age, there are fewer and fewer individuals at higher age levels. This result suggest that the two objectives (minimising age and increasing cumulative reward) are in competition with each other, motivating the choice for multi-objective optimization approach. These results indi- cate that the multi-objectiv optimization is working as intended; staying alive for longer becomes increasingly difficult and a high age needs to compensated for by high task reward.
A.3 SALIENCY MAP CALCULATION
Similarly to the approach by Greydanus et al. (201)). we calculate perturbation-base saliency maps by applying Gaussian blur of 5 x 5 pixels the coordinates (i, .)) of an image / from the game. The Gaussian blur can be interpreted as adding uncertainty to a particular location of the screen. For example, if fireball is location (i, then adding noise to that location makes the agent less certain about the fireball's 's location. The saliency map intensity S(i.j) is calculated as the difference between the policy output given the original image I and modified image I' with added Gaussian blur at location (i,j): S(i.j) (T( (7')).
REFERENCES
Samuel Alvernaz and Julian Togelius. Autrencoder-augmeenee neuroevolution for visual doom playing. In Computational Intelligence and Games (CIG), 2017 IEEE Conference on, pp. 1-8. IEEE, 2017.
Konstantinos Chatzilygeroudis, Roberto Rama, Rituraj Kaushik, Dorian Goepp, Vassilis Vassiliades, and Jean- Baptiste Mouret. Black-box data-efficient policy search for robotics. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 51-58. IEFF, 2017.
10

---

Under review as a conference paper at ICLR 2020
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of morphology and control in embodied machines. Journal of The Royal Society Interface,
15(143), 2018. ISSN 1742-5689. doi: 10. 00.1088/sif.2211.0055.. URL http://rsif.
roya serllsociee stocietpubbl wisbiisshiie www.shiing..ooooooo
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions evolutionary computation, 6(2): 182-197, 2002.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-II), pp- 465-472, 2011.
Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence, 1(1):47-62, 2008
Adam Gaier and David Ha. Weight agnostic neural networks. arXiv preprint arXiv:1906.04338,
2019.
Alex Graves. Hallucination with recurrent neural networks. 2013a. URL /etpss://ww...
youtube, com/ xic.cchhceeeeyxxxxxeeeeexeeee
Alex Graves. Generating sequences with recurrent neural networks.
arXiv preprint
1rXiv:1308.080 2013b.
Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. arXiv preprint arXiv: 1.00138, 2017.
Matthew Guzdial, Boyang Li, and Mark O Riedl. Game engine learning from video. In IJCAI, pp- 3707-3713, 2017.
David Ha and Douglas Eck. arXiv: 1101114444.7777 2017.
neural representation sketch drawings. arXiv preprint
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 1467-2467, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamies for planning from pixels. arXiv preprint 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 2034,104,400 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Gregory S Horbby. Alps: the age-layered population structure for reducing the problem of pre- mature convergence. Proceedings of the 8th annual conference on Genetic and evolutionary computation, pp- 815-822. ACM, 2006.
Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep learning for video game playing. To appear in: IEEE Transactions on Games, 2019.
Michal Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojciech Jaskowski. Viz- doom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Oleg Klimov. Carracing-vll
2016.
URL https: https://gym.opena. www.a.....p.....m.mmmmmmom com/envs..eeeee/eeeeeee
ParRacing-vil
Jan Koutník, Giuseppe Cuccu, Jürgen Schmidhuber, and Faustino Gomez. Evolving large-scale neural networks for vision-based reinforcement learning. Proceedings of the 15th annual con- ference Genetic and evolutionary computation, pp. 1061-1068. ACM, 2013.
11

---

Under review as a conference paper at ICLR 2020
Naoyuki Kubota and Toshio Fukuda. Genetic algorithms with age structure. Soft Computing. 1(4): 155-161, 1997.
Joel Lehman and Kenneth O Stanley. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336, 2008.
Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O Stanley. Safe mutations for deep and recur- rent neural networks through output gradients. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 117-124, ACM, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov))2579-2605, 2008
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):22,, 2015.
J-B Mouret and Stéphane Doncieux. Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary computation, 20( 1):31-133, 2012.
Mohammad Sadegh Norouzzadeh and Jeff Clune. Neuromodulation improves the evolution of for- ward models. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 157-164. ACM, 2016.
Andreas Precht Poulsen, Mark Thorhauge, Mikkel Hvilshj Funch, and Sebastian Risi. Dine: A by- bridization of deep learning and neuroevolution for visual control. In Computational Intelligence and Games (CIG), 2017 IEEE Conference on, pp. 256-263. IEEE, 2017.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, 3:40, 2016.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085 2017.
Sebastian Risi and Kenneth O Stanley. An enhanced hypercube-basce encoding for evolving the placement, density, and connectivity of neurons. Artificial life, 18(4):331-36) 2012.
Sebastian Risi and Kenneth O. Stanley. Deep neuroevolution of recurrent and discrete world mod- cls. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '19, pp. 456-462, New York, NY, USA, 2019. ACM. ISBN 978-144503-6111-8 doi: 10.1145/3321707. 3321817. URL http::/ddae http:///tooaac...././/./
acm www.o/ggg........... 11t.//32111115331117700.. ottp://................///////11
Sebastian Risi and Julian Togelius. Neuroevolution in games: State of the art and open challenges. IEEE Transactions on Computational Intelligence and Al in Games, 91):25544, 2017.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. arXiv preprint trXXiv:703.03866 2017.
Jürgen Schmidhuber. An on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In 1990 IJCNN international joint conference neural networks, pp- 253-258. IEEE, 1990.
Michael Schmidt and Hod Lipson. Age-fitness pareto optimization. In Genetic programming theory and practice VIII, pp. 129-146 Springer, 2011.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiy preprint arXiwv1707.06347. 2017.
Kenneth O Stanley and Risto Mikkkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
12

---

Under review as a conference paper at ICLR 2020
Kenneth O Stanley, Jeff Clune, Joel Lehman, and Risto Mikkulainen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1):24-35, 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. arXiv preprinl arXiv:17/2.06566 2017.
Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yu- lun Li, Ludwig Schubert, Mare Bellemare, Jeff Clune, and Joel Lehman. An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. arXiv preprint arXiv: 1812.0766, 2018.
Niklas Wahlström, Thomas B Schön, and Mare Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:/502.02255/1 2015.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack Rac, Piotr Mirowski, Joel Z Leibo, Adam Santoro, al. Unsupervised predictive memory in a goal-directed agent, arXiv preprint arXiv: 1076, 2018
13


Reviews:
[{"review_id": "SkxpY7ZkqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review of “Deep Innovation Protection”\n\nThis paper builds on top of the World Models line of work that explores in more detail the role of evolutionary computing (as opposed to latent-modeling and planning direction like in Hafner2018) for such model-based architectures. Risi2019 is the first work that is able to train the millions of weights of the architecture proposed in Ha2018 end-to-end using modular-form of genetic algorithm (GA) to maximize the terminal cumulative reward.\n\nThis work expands on Risi2019, and proposes the use of multi-objective optimization (NSGA-II) to allow a world model to learn several subsystems by the fact that the GA optimizer is not only targeting the reward function of the environment, but it is also optimizing for diversity, similar to MAP-Elites line of literature (i.e. Cully2015, Mouret2015, and other more recent cited in this paper).\n\nThey demonstrate that end-to-end multi-objective optimization is not only able to solve both CarRacing and VizDoom tasks, but the world models learned have interesting features, i.e. the emerged world models can predict important survival events even though they were not trained explicitly for forward prediction.\n\nWhile the paper is interesting, I feel in its current state, it is not the level of an ICLR conference paper. Below is some feedback I want to give the authors to help them improve the work, so hopefully it can be improved either for this conference or the next. For the record, the score I wanted to give is a 4, but since I can only choose 3, or 6, I will assign a score of 3 for this review (although should really be a 4). I will breakdown the feedback into minor and major:\n\nMinor feedback:\n\n- Figure 3 does not indicate which task it is solving. Is it CarRacing or VizDoom?\n\n- What's the motivation for optimizing for both \"age\" and reward? Why not other multi-objectives?\n\nMedium Feedback:\n\nAs the authors are learning world models that can predict some form of a future, can they train agents inside an open-loop environment generated by such a world model (perhaps even with some hacks), and transfer such a policy back to the real env, as done in Ha2018 for VizDoom?\n\nMajor feedback:\n\nThe contribution of this paper isn't there, compared to Risi2019 which I feel has a much larger contribution to the line of work. I'm not convinced that the approach (multi-obj + end-to-end) which solves VizDoom cannot be solved using Risi2019. Authors mentioned this in the intro, but perhaps show some convincing experiments to make sure Risi2019 will definitely fail VizDoom? I'm not really convinced that an end-to-end algorithm will fail to find a solution for a poilcy (with our without a world model component in the architecture) for VizDoom Take Cover. Honestly I don't think VizDoom Take Cover is that hard of a task, and my intuition tells me that the method outlined in Risi2019, if carefully tuned properly, will be able to solve VizDoom Take Cover (but happy to be proven wrong if there is a strong experimental argument suggesting otherwise).\n\nI think for this paper to be convincing, the authors actually need to go beyond the CarRacing and VizDoom tasks and show the real power of multi-objective optimization for solving difficult problems that require model-based architectures. For instance, in Cully2015, the learning of several diverse sub-systems on the Pareto front allowed a robot to still be able to accomplish the walking task when the legs are disabled. I would advise the authors to look for a third task where they can clearly show that their approach can solve problems that current approaches (Ha2018, Risi2019, and possibly even DeepRL) cannot solve. Being able to do this has the potential to make this paper into an important work for the field. I can also recommend trying out some environments in the Animal Olympics (see refs) that will be very much suited for multi-obj optimization for animal-like survival environments where input space are pixels.\n\n[Ha2018] Recurrent World Models Facilitate Policy Evolution (NeurIPS2018)\n\n[Risi2019] Deep neuroevolution of recurrent and discrete world models (GECCO2019)\n\n[Hafner2018] Learning latent dynamics for planning from pixels (PlaNet)\n\n[Cully2015] “Robots that can adapt like animals” https://www.nature.com/articles/nature14422\n\n[Mouret2015] Illuminating search spaces by mapping elites\n\n[Animal Olympics] http://animalaiolympics.com/\n\n*** REVISED SCORE ***\n\nUpon looking at the revision, and the author response, I decided to increase the score of the paper (for the record: originally from a 4, which is rounded to a 4, to a 5, which is rounded to a 6).\n\nThe authors explained why Risi2019's approach cannot work on VizDoom with sufficient effort, and I buy their explanation. At the same time I feel this work is can benefit from addition, convincing tasks that highlight the full power of multi objective optimization, for this work to warrant a strong accept. Regardless of the final decision, I'm looking forward to see development of this interesting approach.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeI7qdCKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well organized. The applied methods are introduced in detail. But it lacks some more detailed analysis.\n\nMy concerns are as follows.\n1. The idea of using the concept of age is not new. There are many studies that have been conducted using the concept of aging in the literature of evolutionary algorithms.\n2. Multi-objective optimizations involve finding a set of optimal solutions for different and often competing objectives. However, the authors did not provide the analysis that the age and the accumulated task reward are conflicting with each other.\n3. It would be better the average survival time of the individuals is presented in this paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlQufDPur", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work improves upon the current state of applying neuroevolution to solve RL tasks, by using a modular \"world models\" architecture. While the original work used backpropagation to train the three different components on different tasks, later work used neuroevolution to train the entire architecture for final performance; however, the latter work was unable to demonstrate success on the more challenging task used in the original work. Given that the architecture has a heterogenous pipeline, the authors propose, and empirically show (over a set of seeds), that allowing extra time for components to adapt to other, more recently-changed components, enables neuroevolution to solve the second task. I would give this paper an accept, as this insight, along with a novel algorithm, could aid further work in neuroevolution but also more generally the optimisation of modular NNs.\n\nThe ablations are an important part of this work, and it would be good to study these further. The random age objective performing well does indeed indicate that diversity preservation plays a role in the better performance in DIP, but the fact that protecting the upstream controller does improve performance slightly makes me question if this effect has been properly decoupled. This ablation will reset the age 1/2 the time DIP does, reducing diversity/increasing selection pressure, and so another appropriate ablation is to see the effects of protecting MDN-RNN and controller innovations (with the VAE being the most downstream component).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a very borderline case. Mixed reviews. R2 score originally 4, moved to 5 (rounded up to WA 6), but still borderline. R1 was 6 (WA) and R3 was 3 (WR).  R2 expert on this topic, R1 and R3 less so. AC has carefully read the reviews/rebuttal/comments and looked closely at the paper. AC feels that R2's review is spot on and that the contribution does not quite reach ICLR acceptance level, despite it being interesting work. So the AC feels the paper cannot be accepted at this time. But the work is definitely interesting -- the authors should improve their paper using R2's comments and resubmit.  
Decision: Reject