Improving Evolutionary Strategies with Generative Neural Networks

1 INTRODUCTION . We are interested in the global minimization of a black-box objective function , only accessible through a zeroth-order oracle . In many instances of this problem the objective is expensive to evaluate , which excludes brute force methods as a reasonable mean of optimization . Also , as the objective is potentially non-convex and multi-modal , its global optimization can not be done greedily but requires a careful balance between exploitation and exploration of the optimization landscape ( the surface defined by the objective ) . The family of algorithms used to tackle such a problem is usually dictated by the cost of one evaluation of the objective function ( or equivalently , by the maximum number of function evaluations that are reasonable to make ) and by a precision requirement . For instance , Bayesian Optimization ( Jones et al. , 1998 ; Shahriari et al. , 2016 ) targets problems of very high evaluation cost , where the global minimum must be approximately discovered after a few hundreds of function evaluations . When aiming for a higher precision and hence having a larger budget ( e.g . thousands of function evaluations ) , a popular algorithm class is the one of Evolutionary Strategies ( ES ) ( Rechenberg , 1978 ; Schwefel , 1977 ) , a family of heuristic search procedures . ES algorithms rely on a search distribution , which role is to propose queries of potentially small value of the objective function . This search distribution is almost always chosen to be a multivariate Gaussian . It is namely the case of the Covariance Matrix Adaptation Evolution Strategies ( CMA-ES ) ( Hansen & Ostermeier , 2001 ) , a state-of-the-art ES algorithm made popular in the machine learning community by its good results on hyper-parameter tuning ( Friedrichs & Igel , 2005 ; Loshchilov & Hutter , 2016 ) . It is also the case for Natural Evolution Strategies ( NES ) ( Wierstra et al. , 2008 ) algorithms , which were recently used for direct policy search in Reinforcement Learning ( RL ) and shown to compete with state-of-the-art MDP-based RL techniques ( Salimans et al. , 2017 ) . Occasionally , other distributions have been used ; e.g . fat-tails distributions like the Cauchy were shown to outperform the Gaussian for highly multi-modal objectives ( Schaul et al. , 2011 ) . We argue in this paper that in ES algorithms , the choice of a standard parametric search distribution ( Gaussian , Cauchy , .. ) constitutes a potentially harmful implicit constraint for the stochastic search of a global minimum . To overcome the limitations of classical parametric search distributions , we propose using flexible distributions generated by bijective Generative Neural Networks ( GNNs ) , with computable and differentiable log-probabilities . We discuss why common existing optimization methods in ES algorithms can not be directly used to train such models and design a tailored algorithm that efficiently train GNNs for an ES objective . We show how this new algorithm can readily incorporate existing ES algorithms that operates on simple search distributions , Algorithm 1 : Generic ES procedure input : zeroth-order oracle on f , distribution π0 , population size λ repeat ( Sampling ) Sample x1 , . . . , xλ i.i.d∼ πt ( Evaluation ) Evaluate f ( x1 ) , . . . , f ( xn ) . ( Update ) Update πt to produce x of potentially smaller objective values . until convergence ; like the Gaussian . On a variety of objective functions , we show that this extension can significantly accelerate ES algorithms . We formally introduce the problem and provide background on Evolutionary Strategies in Section 2 . We discuss the role of GNNs in generating flexible search distributions in Section 3 . We explain why usual algorithms fail to train GNNs for an ES objective and introduce a new algorithm in Section 4 . Finally we report experimental results in Section 5 . 2 PRELIMINARIES . In what follows , the real-valued objective function f is defined over a compact X and π will generically denote a probability density function overX . We consider the global optimization of f : x∗ ∈ argmin x∈X f ( x ) ( 1 ) 2.1 EVOLUTIONARY STRATEGIES . The generic procedure followed by ES algorithms is presented in Algorithm 1 . To make the update step tractable , the search distribution is tied to a family of distributions and parametrized by a realvalued parameter vector θ ( e.g . the mean and covariance matrix of a Gaussian ) , and is referred to as πθ . This update step constitutes the main difference between ES algorithms . Natural Evolution Strategies One principled way to perform that update is to minimize the expected objective value over samples x drawn from πθ . Indeed , when the search distribution is parametric and tied to a parameter θ , this objective can be differentiated with respect to θ thanks to the log-trick : J ( θ ) , Eπθ [ f ( x ) ] and ∂J ( θ ) ∂θ = Eπθ [ f ( x ) ∂ log πθ ( x ) ∂θ ] ( 2 ) This quantity can be approximated from samples - it is known as the score-function or REINFORCE ( Williams , 1992 ) estimator , and provides a direction of update for θ . Unfortunately , naively following a stochastic version of the gradient ( 2 ) – a procedure called Plain Gradient Evolutionary Strategies ( PGES ) – is known to be highly ineffective . PGES main limitation resides in its instability when the search distribution is concentrating , making it unable to precisely locate any local minimum . To improve over the PGES algorithm the authors of Wierstra et al . ( 2008 ) proposed to descend J ( θ ) along its natural gradient ( Amari , 1998 ) . More precisely , they introduce a trust-region optimization scheme to limit the instability of PGES , and minimize a linear approximation of J ( θ ) under a Kullback-Leibler ( KL ) divergence constraint : argmin δθ J ( θ + δθ ) ' J ( θ ) + δθT∇θJ ( θ ) s.t KL ( πθ+δθ||πθ ) ≤ ( 3 ) To avoid solving analytically the trust region problem ( 3 ) , Wierstra et al . ( 2008 ) shows that its solution can be approximated by : δθ∗ ∝ −F−1θ ∇θJ ( θ ) where Fθ = Eπθ [ ∇θ log πθ ( x ) ∇θ log πθ ( x ) T ] ( 4 ) is the Fischer Information Matrix ( FIM ) of πθ . The parameter θ is therefore not updated along the negative gradient of J but rather along F−1θ ∇θJ ( θ ) , a quantity known as the natural gradient . The FIM Fθ is known analytically when πθ is a multivariate Gaussian and the resulting algorithm , Exponential Natural Evolutionary Strategies ( xNES ) ( Glasmachers et al. , 2010 ) has been shown to reach state-of-the-art performances on a large ES benchmark . CMA-ES Naturally , there exist other strategies to update the search distribution πθ . For instance , CMA-ES relies on a variety of heuristic mechanisms like covariance matrix adaptation and evolution paths , but is only defined when πθ is a multivariate Gaussian . Explaining such mechanisms would be out of the scope of this paper , but the interested reader is referred to the work of Hansen ( 2016 ) for a detailed tutorial on CMA-ES . 2.2 LIMITATIONS OF CLASSICAL SEARCH DISTRIBUTIONS . ES implicitly balance the need for exploration and exploitation of the optimization landscape . The exploitation phase consists in updating the search distribution , and exploration happens when samples are drawn from the search distribution ’ s tails . The key role of the search distribution is therefore to produce a support adapted to the landscape ’ s structure , so that new points are likely to improve over previous samples . We argue here that the choice of a given parametric distribution ( the multivariate Gaussian distribution being overwhelmingly represented in state-of-the-art ES algorithms ) constitutes a potentially harmful implicit constraint for the stochastic search of a global minimum . For instance , a Gaussian distribution is not adapted to navigate a curved valley because of its inability to continuously curve its density . This lack of flexibility will lead it to drastically reduce its entropy , until the curved valley looks locally straight . At this point , the ES algorithm resembles a hill-climber and barely takes advantage of the exploration abilities of the search distribution . An illustration of this phenomenon is presented in Figure 2 on the Rosenbrock function . Another limitation of classical search distribution is their inability to follow multiple hypothesis , that is to explore at the same time different local minima . Even if mixture models can show such flexibility , hyper-parameters like the number of mixtures have optimal values that are impossible to guess a priori . We want to introduce flexible search distributions to overcome these limitations . Such distributions should , despite their expressiveness , be easily trainable . We should also be concerned when designing them with their role in the exploration/exploitation trade off : a search distribution with too much capacity could over-fit some seemingly good samples , leading to premature convergence . To sum-up , we want to design search-distributions that are : • more flexible than classical distributions • yet easily trainable • while keeping control over the exploration / exploitation trade-off In the following section , we carefully investigate the class of Generative Neural Networks ( GNNs ) to find a parametric class of distributions satisfying such properties . 3 FLEXIBLE SEARCH DISTRIBUTIONS WITH GNNS . Generative Neural Networks ( MacKay , 1995 ) have been studied in the context of density estimation and shown to be able to model complex and highly multimodal distributions ( Srivastava et al. , 2017 ) . We propose here to leverage their expressiveness for ES , and train them in a principled way thanks to the ES objective : J ( π ) = Eπ [ f ( x ) ] As discussed in Section 2 , optimizing J ( π ) with gradient-based methods is possible through the score-function estimator , which requires to be able to compute and efficiently differentiate the logprobabilities of π . 3.1 GNN BACKGROUND . The core idea behind a GNN is to map a latent variable z ∈ Z drawn from a known distribution νω to an output variable x = gη ( z ) where gη is the forward-pass of a neural network . The parameter η represents the weights of this neural network while ω describe the degrees of freedom of the latent space distribution νω . We denote θ= ( ω , η ) and πθ ( x ) the density of the output variable x . For general neural network architectures , it is impossible to compute πθ ( x ) for samples x drawn from the GNN . This is namely why their are often trained with adversarial methods ( Goodfellow et al. , 2014 ) for sample generation purposes , bypassing the need of computing densities , but at the expense of a good density estimation ( mode-dropping ) . An alternative to adversarial methods was proposed with variational auto-encoders ( Kingma & Welling , 2013 ) however at the cost of learning two neural networks ( an encoder and a decoder ) . A less computationally expensive method consists in restricting the possible architectures to build bijective GNNs , also known as Normalizing Flows ( NF ) ( Rezende & Mohamed , 2015 ; Papamakarios et al. , 2017 ) , which allows the exact computation of the distribution ’ s density . Indeed , if gη is a bijection from Z to X with inverse hη , g−1η , the change of variable formula provides a way to compute πθ ( x ) : πθ ( x ) = νω ( hη ( x ) ) · ∣∣∣∣∂hη ( x ) ∂x ∣∣∣∣ ( 5 ) To have a tractable density one therefore needs to ensure that the determinant of the Jacobian |∂hη ( x ) /∂x| is easily computable . Several models satisfying these two properties ( i.e bijectivity and computable Jacobian ) have been proposed for density estimation ( Rippel & Adams , 2013 ; Dinh et al. , 2014 ; 2016 ) , and proved their expressiveness despite their relatively simple structure . NFs therefore answer two of our needs when building our new search distribution : flexibility and easiness to train . In this work , we will focus on one NF model : the Non-Linear Independent Component Estimation ( Dinh et al. , 2014 ) ( NICE ) model , for its numerical stability and volume preserving properties .

Summary: In ES the goal is to find a distribution pi_theta(x) such that the expected value of f(x) under this distribution is high. This can be optimized with REINFORCE or with more sophisticated methods based on the natural gradient. The functional form of pi_theta is almost always a Gaussian, but this isn't sufficiently flexible (e.g. multi-modal) to provide a good optimization algorithm. In response, the authors advocate for using a flexible family of generative neural networks for pi_theta. Using NICE as a generative model is desirable because it maintains volumes. This means that we can adjust volumes in latent space and this directly corresponds to volumes in x space. Doing so is useful to be able to tune how concentrated the search distribution is and to explicitly reason about the mode of the search distribution.