user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving Evolutionary Strategies with Generative Neural Networks

Paper content:
Evolutionary Strategies (ES) are a popular family of black-box zeroth-order optimization algorithms which rely on search distributions to efficiently optimize a large variety of objective functions.
This paper investigates the potential benefits of using highly flexible search distributions in ES algorithms, in contrast to standard ones (typically Gaussians).
We model such distributions with Generative Neural Networks (GNNs) and introduce a new ES algorithm that leverages their expressiveness to accelerate the stochastic search.
Because it acts as a plug-in, our approach allows to augment virtually any standard ES algorithm with flexible search distributions.
We demonstrate the empirical advantages of this method on a diversity of objective functions.
We are interested in the global minimization of a black-box objective function, only accessible through a zeroth-order oracle.
In many instances of this problem the objective is expensive to evaluate, which excludes brute force methods as a reasonable mean of optimization.
Also, as the objective is potentially non-convex and multi-modal, its global optimization cannot be done greedily but requires a careful balance between exploitation and exploration of the optimization landscape (the surface defined by the objective).
The family of algorithms used to tackle such a problem is usually dictated by the cost of one evaluation of the objective function (or equivalently, by the maximum number of function evaluations that are reasonable to make) and by a precision requirement.
For instance, Bayesian Optimization (Jones et al., 1998; Shahriari et al., 2016) targets problems of very high evaluation cost, where the global minimum must be approximately discovered after a few hundreds of function evaluations.
When aiming for a higher precision and hence having a larger budget (e.g. thousands of function evaluations), a popular algorithm class is the one of Evolutionary Strategies (ES) (Rechenberg, 1978; Schwefel, 1977) , a family of heuristic search procedures.
ES algorithms rely on a search distribution, which role is to propose queries of potentially small value of the objective function.
This search distribution is almost always chosen to be a multivariate Gaussian.
It is namely the case of the Covariance Matrix Adaptation Evolution Strategies (CMA-ES) (Hansen & Ostermeier, 2001 ), a state-of-the-art ES algorithm made popular in the machine learning community by its good results on hyper-parameter tuning (Friedrichs & Igel, 2005; Loshchilov & Hutter, 2016) .
It is also the case for Natural Evolution Strategies (NES) (Wierstra et al., 2008) algorithms, which were recently used for direct policy search in Reinforcement Learning (RL) and shown to compete with state-of-the-art MDP-based RL techniques (Salimans et al., 2017) .
Occasionally, other distributions have been used; e.g. fat-tails distributions like the Cauchy were shown to outperform the Gaussian for highly multi-modal objectives (Schaul et al., 2011) .
We argue in this paper that in ES algorithms, the choice of a standard parametric search distribution (Gaussian, Cauchy, ..) constitutes a potentially harmful implicit constraint for the stochastic search of a global minimum.
To overcome the limitations of classical parametric search distributions, we propose using flexible distributions generated by bijective Generative Neural Networks (GNNs), with computable and differentiable log-probabilities.
We discuss why common existing optimization methods in ES algorithms cannot be directly used to train such models and design a tailored algorithm that efficiently train GNNs for an ES objective.
We show how this new algorithm can readily incorporate existing ES algorithms that operates on simple search distributions, Algorithm 1: Generic ES procedure input: zeroth-order oracle on f , distribution π 0 , population size λ repeat (Sampling) Sample x 1 , . . . , x λ i.i.d
∼ π t (Evaluation) Evaluate f (x 1 ), . . . , f (x n ).
(Update)
Update π t to produce x of potentially smaller objective values.
until convergence; like the Gaussian.
On a variety of objective functions, we show that this extension can significantly accelerate ES algorithms.
We formally introduce the problem and provide background on Evolutionary Strategies in Section
2. We discuss the role of GNNs in generating flexible search distributions in Section
3. We explain why usual algorithms fail to train GNNs for an ES objective and introduce a new algorithm in Section
4. Finally we report experimental results in Section 5.
In this work, we motivate the use of GNNs for improving Evolutionary Strategies by pinpointing the limitations of classical search distributions, commonly used by standard ES algorithms.
We propose a new algorithm that leverages the high flexibility of distributions generated by bijective GNNs with an ES objective.
We highlight that this algorithm can be seen as a plug-in extension to existing ES algorithms, and therefore can virtually incorporate any of them.
Finally, we show its empirical advantages across a diversity of synthetic objective functions, as well as from objectives coming from Reinforcement Learning.
Beyond the proposal of this algorithm, we believe that our work highlights the role of expressiveness in exploration for optimization tasks.
This idea could be leverage in other settings where exploration is crucial, such a MDP-based policy search methods.
An interesting line of future work could focus on optimizing GNN-based conditional distribution for RL tasks -an idea already developed in Ward et al. (2019); Mazoure et al. (2019) .
Other possible extensions to our work could focus on investigating first-order and mixed oracles, such as in Grathwohl et al. (2017) ; Faury et al. (2018

assistant: text: We propose a new algorithm leveraging the expressiveness of Generative Neural Networks to improve Evolutionary Strategies algorithms.