Title: SAMPLE EFFICIENT DEEP NEUROEVOLUTION IN LOW DIMENSIONAL LATENT SPACE

Abstract: Current deep neuroevolution models are usually trained in a large parameter search space for complex learning tasks, e.g. playing video games, which needs billions of samples and thousands of search steps to obtain significant performance. This raises a question of whether we can make use of sequential data generated during evolution, encode input samples, and evolve in low dimensional parameter space with latent state input in a fast and efficient manner. Here we give an affirmative answer: we train a VAE to encode input samples, then an RNN to model environment dynamics and handle temporal information, and last evolve our low dimensional policy network in latent space. We demonstrate that this approach is surprisingly efficient: our experiments on Atari games show that within 10M frames and 30 evolution steps of training, our algorithm could achieve competitive result compared with ES, A3C, and DQN which need billions of frames.

1. INTRODUCTION:

Training an agent that can handle difficult tasks in complex environment is always challenging for artificial intelligence. Most works for solving such problems are using reinforcement learning algorithms, e.g. learning a value function (Watkins & Dayan, 1992) , or a policy function (Kakade, 2002) , or both (Sutton et al., 2000) . Combined with deep neural networks, the so-called deep reinforcement learning achieved great success in playing Atari games (Mnih et al., 2015) , operating robots (Yahya et al., 2017) , and winning human experts in challenging competitions like Go and Dota (Silver et al., 2016; OpenAI, 2018) . Recently, a portion of works are trying to combine deep neural networks with evolution strategies to solve similar problems. This approach, called deep neuroevolution, achieved competitive results compared to deep reinforcement learning on Atari Games and Mujoco tasks as well in much shorter training time due to its outstanding scalability and feasibility for parallelization (Salimans et al., 2017; Such et al., 2017) .

However, large-scale deep neuroevolution is both data and computation inefficient: it consumes thousands of CPU cores and needs billions of frames and thousands of generations to obtain significant results. In this work, we introduce a novel way to improve its efficiency: we train a variational autoencoder (VAE) to encode samples, and a recurrent neural network to model environment dynamics and handle temporal information; afterward, we train a two layer policy network with latent state vectors using covariance matrix adaption evolution strategy (CMA-ES).

We evaluate our sample efficient deep neuroevolution algorithm, or in short SEDN, on 50 Atari Games from OpenAI Gym (Brockman et al., 2016) , experiment shows that our approach surpassed DQN (Mnih et al., 2013) , A3C (Mnih et al., 2016) , and ES (Salimans et al., 2017) methods in several Atari games within much fewer state frames, shown in Table 1 . Our key findings are listed below:

• Evolution goes faster with latent state input. SEDN takes less than 30 evolution steps with only 32 children to surpass ES (Salimans et al., 2017) in several games: the latter needs thousands of children and generations.

• Training is more data efficient. Compared to ES and asynchronous RL methods which needs billions of frames to train, SEDN takes less than 10 million frames to achieve targets.

• Training is more computation efficient. On a workstation of one Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz and one Titan X Pascal GPU, SEDN takes less than 4 hours to finish the evolution, compared to ES which needs a cluster to finish.

Many real world decision making problems could be modeled as Markov Decision Process (MDP). Since proposed by Bellman (1957) , it has been applied in many disciplines, e.g. robotics, economics and game playing. A Markov decision process is defined by: a set of states S, a set of actions A, transition probabilities defining probability over next state given current state and action P :

S × A × S → [0, 1], a initial state distribution d : S → [0, 1],

and a reward function that maps states and actions into a scalar R : S × A → R. A finite time horizon episode τ of T steps is a sequence of state-action-reward tuples: (s 0 , a 0 , r 0 ), (s 1 , a 1 , r 1 ), . . . , (s t , a t , r t ), . . . (s T , ∅, ∅) called trajectory, where s t ∈ S, a t ∈ A, r t = R(s t , a t ), s t+1 ∼ P (s t , a t ), s 0 ∼ d 0 (s) where s T is called terminal state.

For a finite time horizon MDP, reinforcement learning (RL) algorithms learn a parameterized policy that defining probability of actions to be taken given current states π θ : S × A → [0, 1], and maximize episodic reward:

EQUATION

where

EQUATION

Above objective function takes the expectation over episodic rewards, which could be approximated by sampling N trajectories from policy π and environment dynamics P and d. We misuse notations such that r(τ ) donates episodic reward and p(τ ) represents trajectory probability, then we have below reformulation:

EQUATION

where

EQUATION

Vanilla policy gradient (PG) methods optimize L(π θ ) by taking its gradient:

EQUATION

and updates policy π θ to maximize expected episodic reward. Variants of policy gradient methods achieved great success in recent years, e.g. TRPO (Schulman et al., 2015) and PPO Schulman et al. (2017) .

Evolution strategies (ES), instead, take another approach to solve above optimization problem without any differentiation: it approximates the gradient by sampling, shown in Equation 7.

EQUATION

Vanilla ES solves MDP problems iteratively shown in Algorithm 1: firstly, it samples Gaussian noise from a normal distribution; secondly, it injects noise into current policy, and evaluates it in environment, obtaining episodic rewards; thirdly, it updates current policy parameters by the product of episode reward and noise injected. ES and PG are interpreted as two faces of Gaussian smoothing on policy: ES is on parameter space smoothing and PG is on action space smoothing (Salimans et al., 2017) .

Algorithm 1 Evolution Strategies:

Input: Learning rate η, noise standard derivation σ, initial policy parameter θ 1: for t = 0, 1, . . . do 2:

Sample N parameter noise: θi ∼ N (0, σ 2 I) for i = 1, . . . , N

3::

Execute perturbated deterministic policies π θi : a t = arg max a π θi (s t ) to produce trajectory τ i , where θ i = θ + θi for i = 1, . . . , N

4::

Update policy parameter θ ← θ + α/(N σ 2 ) N i=1 r(τ i ) θi 5: end for ES enjoys several advantages over PG. Firstly, it is a black-box optimization method that does not need gradients. In problems where gradients are not available, or state encoding is not feasible, or gradient is blocked by some operations in neural networks, ES exhibits its wider range of applications over PG. Secondly, in low dimensional optimization problems like Mujoco tasks, ES has competitive performance with PG (Salimans et al., 2017) , but is less sensitive to hyperparameters (Heidrich-Meisner & Igel, 2008) . Thirdly, the exploration-exploitation trade-off needs to be balanced in RL here is integrated: policy parameter perturbation ensures different behaviors during evaluation, whereas RL has to use a stochastic policy to explore. Lastly, ES is highly scalable therefore can be trained faster by a group of workers in parallel.

However, in n-dimensional search space where n is large, the exploration directions O(2 n ) increases in orders with n, therefore random sampling for gradient approximation would be very inefficient. For video games like Atari, high dimensional pixel state space induces more parameters of policy to be optimized, hence less efficient for ES. Surprisingly, Salimans et al. (2017) shows that even natural evolution strategies (NES) can beat common RL algorithms in several Atari games. Among the class of evolution strategies, the covariance matrix adaption evolution strategy (Hansen & Ostermeier, 2001) is the most successful one in solving low dimensional optimization problems. Would covariance matrix adaption evolution strategy (CMA-ES) perform better or faster over NES for high dimensional tasks? To answer this question, one problem has to be solved, that the application of CMA-ES on high dimensional space is infeasible since it needs to compute a large covariance matrix of size O(n 2 ). Recently advances in deep learning show promises to encode high dimensional spatial-temporal information into low dimensional latent vectors (Oh et al., 2015; Pathak et al., 2018; Ha & Schmidhuber, 2018) by modeling environment dynamics. In this case, we can convert high dimensional RL tasks into low dimensional tasks, where CMA-ES can take its advantage.

2.1. COVARIANCE MATRIX ADAPTION EVOLUTION STRATEGY:

CMA-ES improves ES in two ways. Firstly, vanilla ES samples noise from a fixed Gaussian distribution N (0, σ 2 I) of only one degree of freedom σ, the step size; whereas CMA-ES, samples noise from N (0, C) where covariance matrix C has n(n + 1)/2 degree of freedom. Secondly, CMA-ES updates sampling parameters σ, C by rank based method, rather than updating policy parameter by approximating its gradients.

We now explain rank-based ES. Rank-based ES is a selection-recombination process. Let f donates the objective function, λ be population size, µ be parent population size,

{w i |w 1 > w 2 > • • • > w µ > 0, µ i=1 w µ = 1, i = 1, . . . , µ} be weight coefficient for recombination. Firstly, Rank- based ES first samples λ points from Gaussian distribution x i ∼ m + σN i (0, C) for i = 1, . . . , λ.

Next, it evaluates sampled points on the objective function, and sort according to the cost such that

f (x 1 ) < f (x 2 ) < • • • < f (x λ ).

In the following, it recombines to get new mean and covariance matrix m

← µ i=1 w i x i , C ← µ i=1 w i (x i -m)(x i -m) T . This procedure repeats for G generations.

Similar to gradient based optimization methods, CMA-ES further uses momentum, step size control and path accumulation for more stable optimization. Let y (g+1) i ∼ N (0, C (g) ), i = 1, . . . , λ be sampled noise to be injected,

x i = m + σy i , suppose {y i |i = 1, . . . , µ} are sorted such that f (x 1 ) < f (x 2 ) < • • • < f (x λ )

, updating of covariance matrix with momentum gives:

C (g+1) = (1 -c µ )C (g) + c µ µ i=1 w i y (g+1) i y (g+1)T i (8)

where g is the number of generation and c µ is momentum constant. In favor to generate best point from the current generation, it further adds the best point's covariance term into Equation 8:

C (g+1) = (1 -c µ -c 1 )C (g) + c 1 y (g+1) 1 y (g+1)T 1 + c µ µ i=1 w i y (g+1) i y (g+1)T i (9)

In practice, exponential smoothing over parents' mean path, or evolution path, is used:

EQUATION

C (g+1) = (1 -c µ -c 1 )C (g) + c 1 p (g+1) c p (g+1)T c + c µ µ i=1 w i y (g+1) i y (g+1)T i (11)

Next, we explain how to control the step size σ. The step size is related to the evolution path: it should be decreased if the path is short and increased if the path is long. In contrast to evolution path p c , here we need to correct its directions by taking conjugate evolution path p σ :

EQUATION

To update step size, we compare conjugate evolution path with the expectation of length of vectors sampled from normal distribution N (0, I):

EQUATION

Hansen (2016) suggests default hyperparameters for CMA-ES, we summarize complete CMA-ES in Algorithm 2.

Algorithm 2 CMA-ES

Input: m ∈ R n , σ ∈ R + , λ, µ ∈ N, µ < λ, C = I, p c = 0, p σ = 0 w i such that µ w = 1/ µ i=1 w 2 i ≈ 0.3λ, w 1 ≥ • • • ≥ w µ > 0, µ i=1 w i = 1 c c , ≈ 4/n, c σ ≈ 4/n, c 1 ≈ 2/n 2 , c µ ≈ µ w /n 2 , c 1 + c µ ≤ 1, d σ ≈ 1 + µ w /n 1: for each generation step do 2: Sampling: y i ∼ N i (0, C), x i = m + σy i for i = 1, . . . , λ 3: Evaluation: obtain f (x i ), sort {x i |i = 1, . . . , λ} points such that f (x 1 ) ≤ • • • ≤ f (x λ ) 4: Mean update: m ← µ i=1 w i x i , y w = µ i=1 w i y i 5: Evolution path update: p c ← (1 -c c )p c + y w 1 -(1 -c c ) 2 √ µ w 6:

Conjugate evolution path update:

p σ ← (1 -c σ )p σ + C -1 y w 1 -(1 -c σ ) 2 √ µ w 7: Covariance matrix update: C ← (1 -c 1 -c µ )C + c 1 p c p c T + c µ µ i=1 w i y i y T i 8:

Step size update: σ ← σ exp( cσ dσ ( pσ E N (0,1) -1)) 9: end for CMA-ES addresses typical problems in non-linear optimization problems like ill-conditioning and ruggedness. The covariance matrix approximates the inverse Hessian matrix, and the updates approximate natural gradient by adapting the search metric into a sphere, hence increasing performance by orders of magnitude. Rank based selection ensures its invariance against translation and rotation in search space. The step size control facilitates fast convergence. However, it is not applicable to deep neural networks used in DRL which has millions of parameters and induces O(n 2 ) order of the size of the covariance matrix. This leads to an intuitive approach by encoding frames into small latent vectors to model the dynamics of the game environment, and evolve with a lower dimensional policy parameter space. 2018) also learned forward model dynamics using random policy demonstrations, and train a controller totally inside the dream.

In this work, we take similar approach to encode episodic data. Specifically, we first train a Variational Autoencoder (VAE) that encode spatial information; then train an recurrent neural network (RNN) to encode temporal information by learning model dynamics. Let φ e : S → R n , φ d : R n → S donates our encoder and decoder network where n is our latent space dimension, f : R n × A → R n be the our RNN, and c : R n × R m → A be our policy network where m is the dimension of our RNN's hidden state. The encoder φ e encodes state frame s t into a latent vector z t :

EQUATION

The decoder φ d decodes latent vector back to state frame ŝt . VAE is trained to minimize the reconstruction loss between s t , ŝt , and KL-divergence between latent vector and N (0, I):

EQUATION

LSTM f models the environment dynamics in latent state space, and encode temporal information:

EQUATION

LSTM is trained to minimize the KL-divergence between actual latent state from vae's encoder and predicted latent state:

EQUATION

The policy network takes the latent vector of the current state, and previous timestep's hidden state of RNN, we use deterministic policy in this work:

a t ∼ c(z t , h t-1 )

We illustrate our model in Figure 1 .

3. EXPERIMENTS:

Our complete training pipeline is summarized in Algorithm 3. We use OpenMPI's master-slave mode (Scarabello & Clipp, 2018) for parallel training of policy network during evolution steps, Hansen (2018) for CMA-ES optimizer, and Pytorch (Paszke et al., 2017) deep learning platform to train VAE and RNN. We train the three modules separately. Firstly, we collect episode data during training the policy network using CMA-ES. Secondly, we extract all video frames from collected data and train our variational autoencoder. Thirdly, we convert all state frames into latent state vectors using VAE trained. Lastly, we train our LSTM using the converted episode data. This training cycle iterates till the end.

During the training, we set our training data buffer to be a FIFO queue of size 1M : this will ensure the safety of workstation memory. The training process will stop when the maximum number of frames is reached or the maximum evolution step is reached, whichever is earlier. Our parallel training adopts a task-queue master-slave mode: the master node immediately distributes remaining tasks once there is any slave node available, and slave node requests new task right after it finishes a previous task. Unlike Salimans et al. (2017) , our master-slave communication frequency is so low that it only takes a small percentage of time compared to episode data sampling steps.

We conduct experiments on 50 Atari 2600 games in OpenAI Gym (Brockman et al., 2016) . We choose a larger encoder convolutional network compared to Vanilla DQN (Mnih et al., 2013) for better VAE performance: our experiment shows poor decoding performance if following Vanilla DQN's convolutional network. Our data pre-possessing is also different from Vanilla DQN: we resize image frames directly without frame stack, grayscale conversion or frame skip. We tried to stack latent state vector as input for policy network but found performance drop: the reason may be that increased dimension required for policy network slows down the evolution process for CMA-ES optimizer.

Our encoder network consists of 4 convolutional layers with 32, 64, 128, 256 channels followed by a flatten layer and a linear layer of 128 units. The convolutional layers use 4 × 4, 4 × 4, 3 × 3, 3 × 3 filter size and strides of 2, 2, 1, 1. Our decoder network reverses the operations except the last deconvolutional layer uses 6 × 6 filter size. Each convolutional and deconvolutional layer uses rectified activation, except that last convolutional layer uses tanh activation. Image frames are resized into (84, 84, 3) to feed into our network. We set our latent state dimension to be 128. We use one layer LSTM of 128 hidden size to model environment dynamics, therefore our input dimension of policy network is 256. Our policy network is a two-layer MLP of size 256 -32 -n a , where n a is the action space dimension.

The training of VAE and RNN is triggered every 8 evolution step. Each child policy is evaluated for 16 episodes, and the average episode reward is returned for CMA-ES optimizer to update. To visualize the training process of policy network, we make a boxplot of average episode reward distribution of Frostbite, shown in Figure 2 The maximum reward performance among policy networks increases significantly right after the training of VAE and RNN triggered at 8th, 16th generations. This agrees with our assumption that VAE and RNN encode spatial-temporal information from past experience and provides good feature representation for policy to evolve. It is worth to notice that for some game, e.g. Private Eye, policy performance increases significantly even without updating VAE and RNN; this observation agrees with Blundell et al. (2016) where features are extracted from random matrix projection. However, we also notice that for some games, policy networks performance drops significantly right after updating of VAE and RNN, e.g Atlantis shown in Figure 3 . We find that the cause may be the poor encoding performance of VAE. Shown in Figure 4 , we compare VAE's encoding-decoding on Frostbite and Atlantis. We find that image decoded from VAE will ignore tiny moving objects, leaving only a static background; this issue occurs especially in rare frames far from episode starts; similar phenomenon was found in games like Kangaroo, Gravitar and Breakout.

Experiment result is summarized in Table 1 . Our SEDN surpasses ES, DQN, and A3C in 7 of 50 Atari games, but with only 10M frames, whereas DQN using 200M frames, and ES and A3C using 1B frames. Due to the limitation of time and resources, we did not conduct hyperparameter tuning on SEDN. We believe better encoding techniques to address above issues will give a boost of performance on SEDN. We left this as an open direction in future works.

4. DISCUSSION:

Whereas traditional reinforcement learning algorithms need to balance between exploration and exploitation (Sutton et al., 1998) , and even add perturbation on parameters to ensure good exploration (Plappert et al., 2017) ; evolution strategy directly drives exploration by sampling from optimizer's distribution parameters (Salimans et al., 2017) during evolution: it integrates exploration and exploitation. This is what makes it so efficient especially in some low dimensional control tasks like CartPole. By encoding spatial-temporal information from high dimensional pixel space into low dimensional latent space, a challenging RL task could hence be converted into a simpler task where evolution strategies have advantages over deep reinforcement learning methods. 2017) are facing: ES does not perform well on some easy games like Enduro and Breakout. One possible reason is that ES's optimized policy is close to where it is randomly initialized due to evolution path control under ES, whereas optimal policy for dense reward games like Breakout are far away from where it is randomly initialized. Another reason is that, long episodic time horizon games like Enduro are very inefficient for ES to solve: given limited number of frames, a longer episode will result in less number of episodes, hence less and slower updates for ES. In addition, episode rewards ignore temporal reward information over time steps. This issue is critical in games like Enduro. Enduro is a car racing game, in which agent obtain rewards after passing other cars and penalty if it is passed by other cars, final ranking will be accounted as the episodic reward after a long episode. A random policy will always obtain a zero episodic reward under this case, hence ES learns nothing from such games. These two issues limit the application of vanilla ES and its variants, remaining to be resolved in future works. A possible direction is a mixture of ES and RL, e.g. using ES as a warm start of RL policy.

5. CONCLUSION:

Our work introduces a fast and efficient evolution algorithm that trains deep neural networks to play Atari games. Our experiment shows that encoding spatial-temporal information from high dimensional pixel space into low dimensional latent space makes ES fast, effective and efficient: SEDN outperforms DQN, A3C, and ES in 30 evolution steps with 10M frames experience in several Atari games. We conclude SEDN is a competitive approach to solving challenging RL tasks.

In future works, we plan to address issues discussed in section 4 by applying state of the art computer vision techniques like self-attention maps for better information encoding; we would also like to apply information encoding with RL to understand feature importance for RL. for child i from 1 to λ do 6: contrust policy p i from c i 7:

for each episode j from 1 to N do (φ enc (s 1 ), a 1 , . . . , φ enc (s T ), a T ) ← Y 

:



Figures and Tables:

Figure 2: Boxplot of the training process of Frostbite. Maximum reward performance among policy networks increases significantly after training of VAE and RNN triggered at 8th, 16th generations.

Figure 3: Boxplot of the training process of Atlantis. Maximum reward performance among policy networks drops significantly after training of VAE and RNN triggered at 8th generations.

Figure 4: Comparison of VAE's encoding-decoding on Atlantis and Frostbite. Upper: Original image; Lower: decoded image from the latent vector. Notice for Frostbite, decoded image agrees with original image at the 16th frame; but for Atlantis decoder does not recover the original small objects at 16th frame

{c 1 , . . . , c λ } from CMA-ES optimizer g 5:

data Y ij = (s 0 , a 0 , . . . , s T , a T ) and episode reward R ij with VAE, RNN and Policy:φ enc , f, p i 9:append episode data to training data X ← X ∪ {Y ij } ES optimizer g ← g(c 1 , R1 , c 2 , R2 , . . . , c λ , Rλ ) ,a 1 , . . . , s T , a T ) ← Y 18: I ← I ∪ {s 1 , s 2 , . . . , s T } VAE φ enc , φ dec with I 21: Extracted episode data S = {} 22:for each Y in X do 23:(s 1 , a 1 , . . . , s T , a T ) ← Y24:S ← S ∪ {φ enc (s 1 ), a 1 , . . . , φ enc (s T ), a T }

T , σ 1:T = φ enc (s 1:T ), z 1:T ∼ N (µ 1:T , σ 1:T ) 29: μ2:T , σ2:T = f (z 1:T -1 , a 1:T -1 ) 30: Calculate KL-divergence Loss L = KL[N (µ(ŝ t+1 ), σ(ŝ t+1 )) N (µ(s t+1 ), σ(s t+1))] final policy p f from CMA-ES optimizer's favorite solution 35: return φ dec , φ enc , f, p f

Result is obtained using SEDN on Atari games with 16 re-runs with up to 30 random initial no-ops compared withDQN Mnih et al. (2013),A3C Mnih et al. (2016), and ES Salimans et al. (2017); DQN, A3C, ES data are obtained from Salimans et al. (2017). Enduro's data is missing due to its long episode problem discussed in Section 4 Algorithm 3 Complete Training Pipeline Input: VAE encoder φ enc : x t → z t , VAE decoder φ dec : z t → x t RNN f : (z t , a t ) → z t+1 , Policy p : (z t , h t-1 ) → a t and CMA-ES optimizer: g 1: for each iteration do

Table content: <table><tr><td/><td>DQN</td><td>A3C, FF</td><td colspan="2">ES, FF SEDN, Ours</td></tr><tr><td>Frames</td><td>200 M</td><td>1 B</td><td>1 B</td><td>10 M</td></tr><tr><td>Time</td><td colspan="2">7-10 day 4 day</td><td>4 hour</td><td>4 hour</td></tr><tr><td>Amidar</td><td>133.4</td><td>283.9</td><td>112</td><td>33.4</td></tr><tr><td>Assault</td><td>3332.3</td><td>3746.1</td><td>1673.9</td><td>175.9</td></tr><tr><td>Asterix</td><td>124.5</td><td>6723</td><td>1440</td><td>387.5</td></tr><tr><td>Asteroids</td><td>697.1</td><td>3009.4</td><td>1562</td><td>519.4</td></tr><tr><td>Atlantis</td><td>76108</td><td>772392</td><td colspan="2">1267410 5237.5</td></tr><tr><td>Bank Heist</td><td>176.3</td><td>946</td><td>225</td><td>13.1</td></tr><tr><td>Battle Zone</td><td>17560</td><td>11340</td><td>16600</td><td>3625</td></tr><tr><td>Beam Rider</td><td>8672.4</td><td>13235.9</td><td>744</td><td>332.8</td></tr><tr><td>Berzerk</td><td/><td>1433.4</td><td>686</td><td>224.2</td></tr><tr><td>Bowling</td><td>41.2</td><td>36.2</td><td>30</td><td>97.6</td></tr><tr><td>Boxing</td><td>25.8</td><td>33.7</td><td>49.8</td><td>55.3</td></tr><tr><td>Breakout</td><td>303.9</td><td>551.6</td><td>9.5</td><td>8.8</td></tr><tr><td>Centipede</td><td>3773.1</td><td>3306.5</td><td>7783.9</td><td>3586.9</td></tr><tr><td>Chopper Command</td><td>3046</td><td>4669</td><td>3710</td><td>3718.8</td></tr><tr><td>Crazy Climber</td><td>50992</td><td>101624</td><td>26430</td><td>23662.5</td></tr><tr><td>Demon Attack</td><td colspan="2">12835.2 84997.5</td><td>1166.5</td><td>1812.4</td></tr><tr><td>Double Dunk</td><td>21.6</td><td>0.1</td><td>0.2</td><td>-8.8</td></tr><tr><td>Enduro</td><td>475.6</td><td>82.2</td><td>95</td><td/></tr><tr><td>Fishing Derby</td><td>2.3</td><td>13.6</td><td>49</td><td>-75.4</td></tr><tr><td>Freeway</td><td>25.8</td><td>0.1</td><td>31</td><td>23.8</td></tr><tr><td>Frostbite</td><td>157.4</td><td>180.1</td><td>370</td><td>387.5</td></tr><tr><td>Gopher</td><td>2731.8</td><td>8442.8</td><td>582</td><td>242.5</td></tr><tr><td>Gravitar</td><td>216.5</td><td>269.5</td><td>805</td><td>131.2</td></tr><tr><td>Ice Hockey</td><td>3.8</td><td>4.7</td><td>4.1</td><td>-2.9</td></tr><tr><td>Kangaroo</td><td>2696</td><td>106</td><td>11200</td><td>262.5</td></tr><tr><td>Krull</td><td>3864</td><td>8066.6</td><td>8647.2</td><td>1238.2</td></tr><tr><td colspan="2">Montezumas Revenge 50</td><td>53</td><td>0</td><td>0</td></tr><tr><td>Name This Game</td><td>5439.9</td><td>5614</td><td>4503</td><td>5403.1</td></tr><tr><td>Phoenix</td><td/><td>28181.8</td><td>4041</td><td>1258.1</td></tr><tr><td>Pit Fall</td><td/><td>123</td><td>0</td><td>0</td></tr><tr><td>Pong</td><td>16.2</td><td>11.4</td><td>21</td><td>21</td></tr><tr><td>Private Eye</td><td>298.2</td><td>194.4</td><td>100</td><td>6674.6</td></tr><tr><td>Q*Bert</td><td>4589.8</td><td>13752.3</td><td>147.5</td><td>160.9</td></tr><tr><td>River Raid</td><td>4065.3</td><td>10001.2</td><td>5009</td><td>531</td></tr><tr><td>Road Runner</td><td>9264</td><td>31769</td><td>16590</td><td>26430.8</td></tr><tr><td>Robotank</td><td>58.5</td><td>2.3</td><td>11.9</td><td>4</td></tr><tr><td>Seaquest</td><td>2793.9</td><td>2300.2</td><td>1390</td><td>128.8</td></tr><tr><td>Skiing</td><td/><td>13700</td><td colspan="2">15442.5 16948.4</td></tr><tr><td>Solaris</td><td/><td>1884.8</td><td>2090</td><td>2832.5</td></tr><tr><td>Space Invaders</td><td>1449.7</td><td>2214.7</td><td>678.5</td><td>148.8</td></tr><tr><td>Star Gunner</td><td>34081</td><td>64393</td><td>1470</td><td>337.5</td></tr><tr><td>Tennis</td><td>2.3</td><td>10.2</td><td>4.5</td><td>-2.1</td></tr><tr><td>Time Pilot</td><td>5640</td><td>5825</td><td>4970</td><td>1518.8</td></tr><tr><td>Tutankham</td><td>32.4</td><td>26.1</td><td>130.3</td><td>12.5</td></tr><tr><td>Up and Down</td><td>3311.3</td><td>54525.4</td><td>67974</td><td>8031.8</td></tr><tr><td>Venture</td><td>54</td><td>19</td><td>760</td><td>12.5</td></tr><tr><td>Video Pinball</td><td colspan="4">20228.1 185852.6 22834.8 23058.4</td></tr><tr><td>Wizard of Wor</td><td>246</td><td>5278</td><td>3480</td><td>606.2</td></tr><tr><td>Yars Revenge</td><td/><td>7270.8</td><td colspan="2">16401.7 2953.7</td></tr><tr><td>Zaxxon</td><td>831</td><td>2659</td><td>6380</td><td>262.5</td></tr></table>

Peer review (iclr2019):
 This paper proposes a combination of Evolutionary methods and variational representation learning to improve the sample efficiency of RL methods. They train a VAE on environment frames, as well as an action-conditioned Dynamics model to predict the next frames, and these form the representations fed into a policy network which is trained through ES. Overall, I find the problem setting interesting, and they try to tackle Atari games instead of simpler domains. The use of CMA-ES instead of NES is a good improvement, and the way they motivate using VAE representations to obtain manageable representation sizes is well put forward. [Edit: as mentioned by the other reviewers, this extension isn*t as novel, given Ha et al*s work, hence that reduces my confidence about accepting this paper further...] However, this paper suffers from several issues in its current state: 1. Its presentation is overly detailed about known literature. Section 2 goes in low-level details which are not necessary. It covers ES methods even though a citation to Salimans et al. 2017 would have been sufficient. Section 2.1 is a really complete coverage of CMA-ES, which should really just be a citation of the actual paper again or should be in the Appendix, this doesn’t warrant 1.5 pages of the main text. 2. The actual model presentation is too succinct and split into Section 2.2 and Section 3 (network architectures and parameters). It is never clear how many parameters are optimized by CMA-ES (I counted ~8200 parameters if the MLP of size 256 x 32 x n_a is used). Algorithm 3 however was extremely clear and helpful to fully understand the method. 3. There is no clear evaluation of the performance of the VAE representations and of the RNN dynamics model. Did they actually learn to represent anything at all? Figure 4 is not sufficient in providing evidence supporting this.  Compare this to Higgins et al. 2017, which used VAEs which represented enough information to perform at the same performance as non-variational representations. 4. This feeds into the biggest issue with the current results: The proposed method works rather badly, obtaining worst performance than ES on 35 out of 51 games (68%). On most of these games, the proposed method does not seem to be able to get off the ground at all.  Why is that the case? Obviously if the VAE+RNN do not represent the games well enough, the performance will be bad. Did the policy learning with CMA-ES converge well? (seeing learning curves might help) The fact that no gradients are passed back from the Policy to the VAE/RNN clearly emphasises that issue (The policy only affect the data on which the representations are periodically retrained on). In conclusion, even though I feel this paper tries to tackle an interesting problem, the results are not sufficient to support them as of now. Typos: - “Donates” instead of “denotes” in a few places. References: - Higgins et al., 2017: https://arxiv.org/abs/1707.08475