Sample Efficient Deep Neuroevolution in Low Dimensional Latent Space

1 INTRODUCTION . Training an agent that can handle difficult tasks in complex environment is always challenging for artificial intelligence . Most works for solving such problems are using reinforcement learning algorithms , e.g . learning a value function ( Watkins & Dayan , 1992 ) , or a policy function ( Kakade , 2002 ) , or both ( Sutton et al. , 2000 ) . Combined with deep neural networks , the so-called deep reinforcement learning achieved great success in playing Atari games ( Mnih et al. , 2015 ) , operating robots ( Yahya et al. , 2017 ) , and winning human experts in challenging competitions like Go and Dota ( Silver et al. , 2016 ; OpenAI , 2018 ) . Recently , a portion of works are trying to combine deep neural networks with evolution strategies to solve similar problems . This approach , called deep neuroevolution , achieved competitive results compared to deep reinforcement learning on Atari Games and Mujoco tasks as well in much shorter training time due to its outstanding scalability and feasibility for parallelization ( Salimans et al. , 2017 ; Such et al. , 2017 ) . However , large-scale deep neuroevolution is both data and computation inefficient : it consumes thousands of CPU cores and needs billions of frames and thousands of generations to obtain significant results . In this work , we introduce a novel way to improve its efficiency : we train a variational autoencoder ( VAE ) to encode samples , and a recurrent neural network to model environment dynamics and handle temporal information ; afterward , we train a two layer policy network with latent state vectors using covariance matrix adaption evolution strategy ( CMA-ES ) . We evaluate our sample efficient deep neuroevolution algorithm , or in short SEDN , on 50 Atari Games from OpenAI Gym ( Brockman et al. , 2016 ) , experiment shows that our approach surpassed DQN ( Mnih et al. , 2013 ) , A3C ( Mnih et al. , 2016 ) , and ES ( Salimans et al. , 2017 ) methods in several Atari games within much fewer state frames , shown in Table 1 . Our key findings are listed below : • Evolution goes faster with latent state input . SEDN takes less than 30 evolution steps with only 32 children to surpass ES ( Salimans et al. , 2017 ) in several games : the latter needs thousands of children and generations . • Training is more data efficient . Compared to ES and asynchronous RL methods which needs billions of frames to train , SEDN takes less than 10 million frames to achieve targets . • Training is more computation efficient . On a workstation of one Intel ( R ) Xeon ( R ) CPU E5-2640 v4 @ 2.40GHz and one Titan X Pascal GPU , SEDN takes less than 4 hours to finish the evolution , compared to ES which needs a cluster to finish . 2 METHOD . Many real world decision making problems could be modeled as Markov Decision Process ( MDP ) . Since proposed by Bellman ( 1957 ) , it has been applied in many disciplines , e.g . robotics , economics and game playing . A Markov decision process is defined by : a set of states S , a set of actions A , transition probabilities defining probability over next state given current state and action P : S × A × S → [ 0 , 1 ] , a initial state distribution d : S → [ 0 , 1 ] , and a reward function that maps states and actions into a scalar R : S × A → R. A finite time horizon episode τ of T steps is a sequence of state-action-reward tuples : ( s0 , a0 , r0 ) , ( s1 , a1 , r1 ) , . . . , ( st , at , rt ) , . . . ( sT , ∅ , ∅ ) called trajectory , where st ∈ S , at ∈ A , rt = R ( st , at ) , st+1 ∼ P ( st , at ) , s0 ∼ d0 ( s ) where sT is called terminal state . For a finite time horizon MDP , reinforcement learning ( RL ) algorithms learn a parameterized policy that defining probability of actions to be taken given current states πθ : S × A → [ 0 , 1 ] , and maximize episodic reward : ∑T−1 i=0 ri , or formally : maximize θ E T∑ i=0 R ( st , at ) ( 1 ) where s0 ∼ d ( s ) , at ∼ πθ ( st ) , st+1 ∼ P ( st , at ) ( 2 ) Above objective function takes the expectation over episodic rewards , which could be approximated by sampling N trajectories from policy π and environment dynamics P and d. We misuse notations such that r ( τ ) donates episodic reward and p ( τ ) represents trajectory probability , then we have below reformulation : maximize θ L ( θ ) = N∑ i=1 r ( τi ) p ( τi ) ( 3 ) where p ( τ ) = d ( s0 ) T−1∏ t=0 πθ ( st , at ) p ( st , at , st+1 ) , r ( τ ) = T−1∑ t=0 ri ( 4 ) Vanilla policy gradient ( PG ) methods optimize L ( πθ ) by taking its gradient : ∇θL ≈ 1 N N∑ i=1 r ( τi ) ∇θ log pθ ( τi ) ( 5 ) = 1 N N∑ i=1 r ( τi ) Ti−1∑ t=0 ∇θ log πθ ( st , at ) ( 6 ) and updates policy πθ to maximize expected episodic reward . Variants of policy gradient methods achieved great success in recent years , e.g . TRPO ( Schulman et al. , 2015 ) and PPO Schulman et al . ( 2017 ) . Evolution strategies ( ES ) , instead , take another approach to solve above optimization problem without any differentiation : it approximates the gradient by sampling , shown in Equation 7 . ∇θL ≈ 1 σ2 Eθ̂∼N ( 0 , σ2 ) ( L ( θ + θ̂ ) θ̂ ) ( 7 ) Vanilla ES solves MDP problems iteratively shown in Algorithm 1 : firstly , it samples Gaussian noise from a normal distribution ; secondly , it injects noise into current policy , and evaluates it in environment , obtaining episodic rewards ; thirdly , it updates current policy parameters by the product of episode reward and noise injected . ES and PG are interpreted as two faces of Gaussian smoothing on policy : ES is on parameter space smoothing and PG is on action space smoothing ( Salimans et al. , 2017 ) . Algorithm 1 Evolution Strategies Input : Learning rate η , noise standard derivation σ , initial policy parameter θ 1 : for t = 0 , 1 , . . . do 2 : Sample N parameter noise : θ̂i ∼ N ( 0 , σ2I ) for i = 1 , . . . , N 3 : Execute perturbated deterministic policies πθi : at = arg maxa πθi ( st ) to produce trajectory τi , where θi = θ + θ̂i for i = 1 , . . . , N 4 : Update policy parameter θ ← θ + α/ ( Nσ2 ) ∑N i=1 r ( τi ) θ̂i 5 : end for ES enjoys several advantages over PG . Firstly , it is a black-box optimization method that does not need gradients . In problems where gradients are not available , or state encoding is not feasible , or gradient is blocked by some operations in neural networks , ES exhibits its wider range of applications over PG . Secondly , in low dimensional optimization problems like Mujoco tasks , ES has competitive performance with PG ( Salimans et al. , 2017 ) , but is less sensitive to hyperparameters ( Heidrich-Meisner & Igel , 2008 ) . Thirdly , the exploration-exploitation trade-off needs to be balanced in RL here is integrated : policy parameter perturbation ensures different behaviors during evaluation , whereas RL has to use a stochastic policy to explore . Lastly , ES is highly scalable therefore can be trained faster by a group of workers in parallel . However , in n-dimensional search space where n is large , the exploration directionsO ( 2n ) increases in orders with n , therefore random sampling for gradient approximation would be very inefficient . For video games like Atari , high dimensional pixel state space induces more parameters of policy to be optimized , hence less efficient for ES . Surprisingly , Salimans et al . ( 2017 ) shows that even natural evolution strategies ( NES ) can beat common RL algorithms in several Atari games . Among the class of evolution strategies , the covariance matrix adaption evolution strategy ( Hansen & Ostermeier , 2001 ) is the most successful one in solving low dimensional optimization problems . Would covariance matrix adaption evolution strategy ( CMA-ES ) perform better or faster over NES for high dimensional tasks ? To answer this question , one problem has to be solved , that the application of CMA-ES on high dimensional space is infeasible since it needs to compute a large covariance matrix of size O ( n2 ) . Recently advances in deep learning show promises to encode high dimensional spatial-temporal information into low dimensional latent vectors ( Oh et al. , 2015 ; Pathak et al. , 2018 ; Ha & Schmidhuber , 2018 ) by modeling environment dynamics . In this case , we can convert high dimensional RL tasks into low dimensional tasks , where CMA-ES can take its advantage . 2.1 COVARIANCE MATRIX ADAPTION EVOLUTION STRATEGY . CMA-ES improves ES in two ways . Firstly , vanilla ES samples noise from a fixed Gaussian distribution N ( 0 , σ2I ) of only one degree of freedom σ , the step size ; whereas CMA-ES , samples noise from N ( 0 , C ) where covariance matrix C has n ( n+ 1 ) /2 degree of freedom . Secondly , CMA-ES updates sampling parameters σ , C by rank based method , rather than updating policy parameter by approximating its gradients . We now explain rank-based ES . Rank-based ES is a selection-recombination process . Let f donates the objective function , λ be population size , µ be parent population size , { wi|w1 > w2 > · · · > wµ > 0 , ∑µ i=1 wµ = 1 , i = 1 , . . . , µ } be weight coefficient for recombination . Firstly , Rankbased ES first samples λ points from Gaussian distribution xi ∼ m + σNi ( 0 , C ) for i = 1 , . . . , λ . Next , it evaluates sampled points on the objective function , and sort according to the cost such that f ( x1 ) < f ( x2 ) < · · · < f ( xλ ) . In the following , it recombines to get new mean and covariance matrix m ← ∑µ i=1 wixi , C ← ∑µ i=1 wi ( xi − m ) ( xi − m ) T . This procedure repeats for G generations . Similar to gradient based optimization methods , CMA-ES further uses momentum , step size control and path accumulation for more stable optimization . Let y ( g+1 ) i ∼ N ( 0 , C ( g ) ) , i = 1 , . . . , λ be sampled noise to be injected , xi = m + σyi , suppose { yi|i = 1 , . . . , µ } are sorted such that f ( x1 ) < f ( x2 ) < · · · < f ( xλ ) , updating of covariance matrix with momentum gives : C ( g+1 ) = ( 1− cµ ) C ( g ) + cµ µ∑ i=1 wiy ( g+1 ) i y ( g+1 ) T i ( 8 ) where g is the number of generation and cµ is momentum constant . In favor to generate best point from the current generation , it further adds the best point ’ s covariance term into Equation 8 : C ( g+1 ) = ( 1− cµ − c1 ) C ( g ) + c1y ( g+1 ) 1 y ( g+1 ) T 1 + cµ µ∑ i=1 wiy ( g+1 ) i y ( g+1 ) T i ( 9 ) In practice , exponential smoothing over parents ’ mean path , or evolution path , is used : p ( g+1 ) c = ( 1− cc ) p ( g ) c + cc ( m ( g+1 ) −m ( g ) ) /σ ( g ) ( 10 ) C ( g+1 ) = ( 1− cµ − c1 ) C ( g ) + c1p ( g+1 ) c p ( g+1 ) Tc + cµ µ∑ i=1 wiy ( g+1 ) i y ( g+1 ) T i ( 11 ) Next , we explain how to control the step size σ . The step size is related to the evolution path : it should be decreased if the path is short and increased if the path is long . In contrast to evolution path pc , here we need to correct its directions by taking conjugate evolution path pσ : p ( g+1 ) σ = ( 1− cσ ) p ( g ) σ + cσC ( g ) , − 1 2 ( m ( g+1 ) −m ( g ) ) /σ ( g ) ( 12 ) To update step size , we compare conjugate evolution path with the expectation of length of vectors sampled from normal distribution N ( 0 , I ) : lnσ ( g+1 ) = lnσ ( g ) + cσ dσ ( ‖pσ‖ E‖N ( 0,1 ) ‖ − 1 ) ( 13 ) Hansen ( 2016 ) suggests default hyperparameters for CMA-ES , we summarize complete CMA-ES in Algorithm 2 . Algorithm 2 CMA-ES Input : m ∈ Rn , σ ∈ R+ , λ , µ ∈ N , µ < λ , C = I , pc = 0 , pσ = 0 wi such that µw = 1/ ∑µ i=1 w 2 i ≈ 0.3λ , w1 ≥ · · · ≥ wµ > 0 , ∑µ i=1 wi = 1 cc , ≈ 4/n , cσ ≈ 4/n , c1 ≈ 2/n2 , cµ ≈ µw/n2 , c1 + cµ ≤ 1 , dσ ≈ 1 + √ µw/n 1 : for each generation step do 2 : Sampling : yi ∼ Ni ( 0 , C ) , xi = m + σyi for i = 1 , . . . , λ 3 : Evaluation : obtain f ( xi ) , sort { xi|i = 1 , . . . , λ } points such that f ( x1 ) ≤ · · · ≤ f ( xλ ) 4 : Mean update : m← ∑µ i=1 wixi , yw = ∑µ i=1 wiyi 5 : Evolution path update : pc ← ( 1− cc ) pc + yw √ 1− ( 1− cc ) 2 √ µw 6 : Conjugate evolution path update : pσ ← ( 1− cσ ) pσ + C−1yw √ 1− ( 1− cσ ) 2 √ µw 7 : Covariance matrix update : C← ( 1− c1 − cµ ) C + c1pcpcT + cµ ∑µ i=1 wiyiy T i 8 : Step size update : σ ← σ exp ( cσdσ ( ‖pσ‖ E‖N ( 0,1 ) ‖ − 1 ) ) 9 : end for CMA-ES addresses typical problems in non-linear optimization problems like ill-conditioning and ruggedness . The covariance matrix approximates the inverse Hessian matrix , and the updates approximate natural gradient by adapting the search metric into a sphere , hence increasing performance by orders of magnitude . Rank based selection ensures its invariance against translation and rotation in search space . The step size control facilitates fast convergence . However , it is not applicable to deep neural networks used in DRL which has millions of parameters and induces O ( n2 ) order of the size of the covariance matrix . This leads to an intuitive approach by encoding frames into small latent vectors to model the dynamics of the game environment , and evolve with a lower dimensional policy parameter space .

Summary: They propose "Sample Efficient Deep Neuroevolution" (SEDN) model and experiment on Atari games. In this model, they use a Variational Encoder (VAE) to encode state frames into a latent vector, and use an LSTM to encode the current latent vector and action to predict the next latent space. A policy network (trained using CMA-ES) takes the latent space, and hidden state of the RNN as an input, and outputs an action to execute.