Sample Efficient Deep Neuroevolution in Low Dimensional Latent Space

1 INTRODUCTION . Training an agent that can handle difficult tasks in complex environment is always challenging for artificial intelligence . Most works for solving such problems are using reinforcement learning algorithms , e.g . learning a value function ( Watkins & Dayan , 1992 ) , or a policy function ( Kakade , 2002 ) , or both ( Sutton et al. , 2000 ) . Combined with deep neural networks , the so-called deep reinforcement learning achieved great success in playing Atari games ( Mnih et al. , 2015 ) , operating robots ( Yahya et al. , 2017 ) , and winning human experts in challenging competitions like Go and Dota ( Silver et al. , 2016 ; OpenAI , 2018 ) . Recently , a portion of works are trying to combine deep neural networks with evolution strategies to solve similar problems . This approach , called deep neuroevolution , achieved competitive results compared to deep reinforcement learning on Atari Games and Mujoco tasks as well in much shorter training time due to its outstanding scalability and feasibility for parallelization ( Salimans et al. , 2017 ; Such et al. , 2017 ) . However , large-scale deep neuroevolution is both data and computation inefficient : it consumes thousands of CPU cores and needs billions of frames and thousands of generations to obtain significant results . In this work , we introduce a novel way to improve its efficiency : we train a variational autoencoder ( VAE ) to encode samples , and a recurrent neural network to model environment dynamics and handle temporal information ; afterward , we train a two layer policy network with latent state vectors using covariance matrix adaption evolution strategy ( CMA-ES ) . We evaluate our sample efficient deep neuroevolution algorithm , or in short SEDN , on 50 Atari Games from OpenAI Gym ( Brockman et al. , 2016 ) , experiment shows that our approach surpassed DQN ( Mnih et al. , 2013 ) , A3C ( Mnih et al. , 2016 ) , and ES ( Salimans et al. , 2017 ) methods in several Atari games within much fewer state frames , shown in Table 1 . Our key findings are listed below : • Evolution goes faster with latent state input . SEDN takes less than 30 evolution steps with only 32 children to surpass ES ( Salimans et al. , 2017 ) in several games : the latter needs thousands of children and generations . • Training is more data efficient . Compared to ES and asynchronous RL methods which needs billions of frames to train , SEDN takes less than 10 million frames to achieve targets . • Training is more computation efficient . On a workstation of one Intel ( R ) Xeon ( R ) CPU E5-2640 v4 @ 2.40GHz and one Titan X Pascal GPU , SEDN takes less than 4 hours to finish the evolution , compared to ES which needs a cluster to finish . 2 METHOD . Many real world decision making problems could be modeled as Markov Decision Process ( MDP ) . Since proposed by Bellman ( 1957 ) , it has been applied in many disciplines , e.g . robotics , economics and game playing . A Markov decision process is defined by : a set of states S , a set of actions A , transition probabilities defining probability over next state given current state and action P : S × A × S → [ 0 , 1 ] , a initial state distribution d : S → [ 0 , 1 ] , and a reward function that maps states and actions into a scalar R : S × A → R. A finite time horizon episode τ of T steps is a sequence of state-action-reward tuples : ( s0 , a0 , r0 ) , ( s1 , a1 , r1 ) , . . . , ( st , at , rt ) , . . . ( sT , ∅ , ∅ ) called trajectory , where st ∈ S , at ∈ A , rt = R ( st , at ) , st+1 ∼ P ( st , at ) , s0 ∼ d0 ( s ) where sT is called terminal state . For a finite time horizon MDP , reinforcement learning ( RL ) algorithms learn a parameterized policy that defining probability of actions to be taken given current states πθ : S × A → [ 0 , 1 ] , and maximize episodic reward : ∑T−1 i=0 ri , or formally : maximize θ E T∑ i=0 R ( st , at ) ( 1 ) where s0 ∼ d ( s ) , at ∼ πθ ( st ) , st+1 ∼ P ( st , at ) ( 2 ) Above objective function takes the expectation over episodic rewards , which could be approximated by sampling N trajectories from policy π and environment dynamics P and d. We misuse notations such that r ( τ ) donates episodic reward and p ( τ ) represents trajectory probability , then we have below reformulation : maximize θ L ( θ ) = N∑ i=1 r ( τi ) p ( τi ) ( 3 ) where p ( τ ) = d ( s0 ) T−1∏ t=0 πθ ( st , at ) p ( st , at , st+1 ) , r ( τ ) = T−1∑ t=0 ri ( 4 ) Vanilla policy gradient ( PG ) methods optimize L ( πθ ) by taking its gradient : ∇θL ≈ 1 N N∑ i=1 r ( τi ) ∇θ log pθ ( τi ) ( 5 ) = 1 N N∑ i=1 r ( τi ) Ti−1∑ t=0 ∇θ log πθ ( st , at ) ( 6 ) and updates policy πθ to maximize expected episodic reward . Variants of policy gradient methods achieved great success in recent years , e.g . TRPO ( Schulman et al. , 2015 ) and PPO Schulman et al . ( 2017 ) . Evolution strategies ( ES ) , instead , take another approach to solve above optimization problem without any differentiation : it approximates the gradient by sampling , shown in Equation 7 . ∇θL ≈ 1 σ2 Eθ̂∼N ( 0 , σ2 ) ( L ( θ + θ̂ ) θ̂ ) ( 7 ) Vanilla ES solves MDP problems iteratively shown in Algorithm 1 : firstly , it samples Gaussian noise from a normal distribution ; secondly , it injects noise into current policy , and evaluates it in environment , obtaining episodic rewards ; thirdly , it updates current policy parameters by the product of episode reward and noise injected . ES and PG are interpreted as two faces of Gaussian smoothing on policy : ES is on parameter space smoothing and PG is on action space smoothing ( Salimans et al. , 2017 ) . Algorithm 1 Evolution Strategies Input : Learning rate η , noise standard derivation σ , initial policy parameter θ 1 : for t = 0 , 1 , . . . do 2 : Sample N parameter noise : θ̂i ∼ N ( 0 , σ2I ) for i = 1 , . . . , N 3 : Execute perturbated deterministic policies πθi : at = arg maxa πθi ( st ) to produce trajectory τi , where θi = θ + θ̂i for i = 1 , . . . , N 4 : Update policy parameter θ ← θ + α/ ( Nσ2 ) ∑N i=1 r ( τi ) θ̂i 5 : end for ES enjoys several advantages over PG . Firstly , it is a black-box optimization method that does not need gradients . In problems where gradients are not available , or state encoding is not feasible , or gradient is blocked by some operations in neural networks , ES exhibits its wider range of applications over PG . Secondly , in low dimensional optimization problems like Mujoco tasks , ES has competitive performance with PG ( Salimans et al. , 2017 ) , but is less sensitive to hyperparameters ( Heidrich-Meisner & Igel , 2008 ) . Thirdly , the exploration-exploitation trade-off needs to be balanced in RL here is integrated : policy parameter perturbation ensures different behaviors during evaluation , whereas RL has to use a stochastic policy to explore . Lastly , ES is highly scalable therefore can be trained faster by a group of workers in parallel . However , in n-dimensional search space where n is large , the exploration directionsO ( 2n ) increases in orders with n , therefore random sampling for gradient approximation would be very inefficient . For video games like Atari , high dimensional pixel state space induces more parameters of policy to be optimized , hence less efficient for ES . Surprisingly , Salimans et al . ( 2017 ) shows that even natural evolution strategies ( NES ) can beat common RL algorithms in several Atari games . Among the class of evolution strategies , the covariance matrix adaption evolution strategy ( Hansen & Ostermeier , 2001 ) is the most successful one in solving low dimensional optimization problems . Would covariance matrix adaption evolution strategy ( CMA-ES ) perform better or faster over NES for high dimensional tasks ? To answer this question , one problem has to be solved , that the application of CMA-ES on high dimensional space is infeasible since it needs to compute a large covariance matrix of size O ( n2 ) . Recently advances in deep learning show promises to encode high dimensional spatial-temporal information into low dimensional latent vectors ( Oh et al. , 2015 ; Pathak et al. , 2018 ; Ha & Schmidhuber , 2018 ) by modeling environment dynamics . In this case , we can convert high dimensional RL tasks into low dimensional tasks , where CMA-ES can take its advantage . 2.1 COVARIANCE MATRIX ADAPTION EVOLUTION STRATEGY . CMA-ES improves ES in two ways . Firstly , vanilla ES samples noise from a fixed Gaussian distribution N ( 0 , σ2I ) of only one degree of freedom σ , the step size ; whereas CMA-ES , samples noise from N ( 0 , C ) where covariance matrix C has n ( n+ 1 ) /2 degree of freedom . Secondly , CMA-ES updates sampling parameters σ , C by rank based method , rather than updating policy parameter by approximating its gradients . We now explain rank-based ES . Rank-based ES is a selection-recombination process . Let f donates the objective function , λ be population size , µ be parent population size , { wi|w1 > w2 > · · · > wµ > 0 , ∑µ i=1 wµ = 1 , i = 1 , . . . , µ } be weight coefficient for recombination . Firstly , Rankbased ES first samples λ points from Gaussian distribution xi ∼ m + σNi ( 0 , C ) for i = 1 , . . . , λ . Next , it evaluates sampled points on the objective function , and sort according to the cost such that f ( x1 ) < f ( x2 ) < · · · < f ( xλ ) . In the following , it recombines to get new mean and covariance matrix m ← ∑µ i=1 wixi , C ← ∑µ i=1 wi ( xi − m ) ( xi − m ) T . This procedure repeats for G generations . Similar to gradient based optimization methods , CMA-ES further uses momentum , step size control and path accumulation for more stable optimization . Let y ( g+1 ) i ∼ N ( 0 , C ( g ) ) , i = 1 , . . . , λ be sampled noise to be injected , xi = m + σyi , suppose { yi|i = 1 , . . . , µ } are sorted such that f ( x1 ) < f ( x2 ) < · · · < f ( xλ ) , updating of covariance matrix with momentum gives : C ( g+1 ) = ( 1− cµ ) C ( g ) + cµ µ∑ i=1 wiy ( g+1 ) i y ( g+1 ) T i ( 8 ) where g is the number of generation and cµ is momentum constant . In favor to generate best point from the current generation , it further adds the best point ’ s covariance term into Equation 8 : C ( g+1 ) = ( 1− cµ − c1 ) C ( g ) + c1y ( g+1 ) 1 y ( g+1 ) T 1 + cµ µ∑ i=1 wiy ( g+1 ) i y ( g+1 ) T i ( 9 ) In practice , exponential smoothing over parents ’ mean path , or evolution path , is used : p ( g+1 ) c = ( 1− cc ) p ( g ) c + cc ( m ( g+1 ) −m ( g ) ) /σ ( g ) ( 10 ) C ( g+1 ) = ( 1− cµ − c1 ) C ( g ) + c1p ( g+1 ) c p ( g+1 ) Tc + cµ µ∑ i=1 wiy ( g+1 ) i y ( g+1 ) T i ( 11 ) Next , we explain how to control the step size σ . The step size is related to the evolution path : it should be decreased if the path is short and increased if the path is long . In contrast to evolution path pc , here we need to correct its directions by taking conjugate evolution path pσ : p ( g+1 ) σ = ( 1− cσ ) p ( g ) σ + cσC ( g ) , − 1 2 ( m ( g+1 ) −m ( g ) ) /σ ( g ) ( 12 ) To update step size , we compare conjugate evolution path with the expectation of length of vectors sampled from normal distribution N ( 0 , I ) : lnσ ( g+1 ) = lnσ ( g ) + cσ dσ ( ‖pσ‖ E‖N ( 0,1 ) ‖ − 1 ) ( 13 ) Hansen ( 2016 ) suggests default hyperparameters for CMA-ES , we summarize complete CMA-ES in Algorithm 2 . Algorithm 2 CMA-ES Input : m ∈ Rn , σ ∈ R+ , λ , µ ∈ N , µ < λ , C = I , pc = 0 , pσ = 0 wi such that µw = 1/ ∑µ i=1 w 2 i ≈ 0.3λ , w1 ≥ · · · ≥ wµ > 0 , ∑µ i=1 wi = 1 cc , ≈ 4/n , cσ ≈ 4/n , c1 ≈ 2/n2 , cµ ≈ µw/n2 , c1 + cµ ≤ 1 , dσ ≈ 1 + √ µw/n 1 : for each generation step do 2 : Sampling : yi ∼ Ni ( 0 , C ) , xi = m + σyi for i = 1 , . . . , λ 3 : Evaluation : obtain f ( xi ) , sort { xi|i = 1 , . . . , λ } points such that f ( x1 ) ≤ · · · ≤ f ( xλ ) 4 : Mean update : m← ∑µ i=1 wixi , yw = ∑µ i=1 wiyi 5 : Evolution path update : pc ← ( 1− cc ) pc + yw √ 1− ( 1− cc ) 2 √ µw 6 : Conjugate evolution path update : pσ ← ( 1− cσ ) pσ + C−1yw √ 1− ( 1− cσ ) 2 √ µw 7 : Covariance matrix update : C← ( 1− c1 − cµ ) C + c1pcpcT + cµ ∑µ i=1 wiyiy T i 8 : Step size update : σ ← σ exp ( cσdσ ( ‖pσ‖ E‖N ( 0,1 ) ‖ − 1 ) ) 9 : end for CMA-ES addresses typical problems in non-linear optimization problems like ill-conditioning and ruggedness . The covariance matrix approximates the inverse Hessian matrix , and the updates approximate natural gradient by adapting the search metric into a sphere , hence increasing performance by orders of magnitude . Rank based selection ensures its invariance against translation and rotation in search space . The step size control facilitates fast convergence . However , it is not applicable to deep neural networks used in DRL which has millions of parameters and induces O ( n2 ) order of the size of the covariance matrix . This leads to an intuitive approach by encoding frames into small latent vectors to model the dynamics of the game environment , and evolve with a lower dimensional policy parameter space .

Summary: The main difficulty of neuroevolution---requiring a huge number of simulations for high dimensional problem---is addressed in this paper by introducing VAE to reduce the state space dimensionality and using a rather shallow controller network. This idea itself is very promising, however, it has been introduced in (Ha and Schmidhuber, 2018).  Still, there seems to be differences in how to gather histories and how to use them. Nevertheless, the differences are not well described in the text. The effect of the modification is not evaluated on experiments.