DiBB: Distributing Black-Box Optimization

1 INTRODUCTION . Black Box Optimization ( BBO ) can be applied , by definition , to any problem independent of the specific application ( Audet & Hare , 2017 ) . In principle , this provides a method that is applicable to problems yet unsolved by the current state of the art . The most obvious catch lies in their computational requirements , which usually takes one of two forms : ( i ) simple black-box solvers suffers from high sample complexity ( i.e. , they are data hungry ) , in exchange for smaller computational requirements ( both in memory and CPU time ) . On the other hand , ( ii ) sophisticated solvers such as modern Evolution Strategies ( ES ; Hansen & Ostermeier 2001 ; Wierstra et al . 2014 ) have a high internal computational cost per sample , in exchange for a wide set of properties ( see Section 2 ) that make for significantly improved sample efficiency . In the first case ( simpler ES ) , the requirement for more samples can be mitigated—to some extent— through the embarrassingly parallel evaluation of a large population of candidate solutions . This approach however has its own limitations , as traversing a more complex fitness landscape often requires a disproportionate growth in the number of required samples as the dimensionality rises . In the second case however ( complex ES ) , algorithms relying on covariance matrix adaptation ( CMA ; e.g . CMA-ES by Hansen 1996 ) have a quadratic complexity in the number of variables for processing a sample . This strictly limits their application ( within a sensible time frame ) to problems within the tens of thousands variables . This limit has restricted the applicability of these methods to rather simple scenarios and applications so far . As a result , most successful applications of ES to high-dimensional problems ( e.g . training larger neural networks ) rely on simpler ES , such as the type ( i ) described above . Most of the per-sample complexity of type ( ii ) algorithms can be traced to the algorithm maintaining and updating the covariance information among the set of variables , which helps the sample efficiency of these algorithms ( as in convergence speed ) at the cost of longer run times ( as in wall-clock speed ) , eventually becoming intractable as the number of variables grows . Several algorithms ( Ros & Hansen , 2008 ; Schaul et al. , 2011 ) thereby implement an assumption of separability between the variables , relinquishing covariance information altogether for a greatly enhanced wall-clock speed per sample . Unsurprisingly , this comes at a cost in terms of convergence speed : these algorithms can require several orders of magnitude more samples to find comparable solutions , depending on the problem structure ( Hansen & Ostermeier , 2001 ) . The approach taken in this paper takes instead into account the fact that the correlation among variables is not uniform for most complex problems . Rather , certain groups of variables will display higher correlation between each other , with variables having lower correlation belonging instead into different groups . This induces two different assumptions on the variable set : groups having high intra-correlation support an assumption of partial correlation , while the low inter-group correlation displays the parallel argument of partial separability . We can represent this using a block-diagonal covariance matrix , which at the same time maintains full-covariance information within the variables belonging to one such group , and discarding the less useful ( and often negligible ) inter-group correlation altogether ( Cuccu & Gomez , 2012 ) . Such an approach allows leveraging covariance information where it is most efficient , while at the same time significantly reducing the memory footprint of the algorithm ( based on the number and size of the groups ) , and even offering additional opportunities for parallelization and distributed computing . We will use the term blocks to refer to the groups of variable with high intra-correlation in the rest of the paper . BBO algorithms are designed to ignore blocks , since they need to work in any unknowable ( blackbox ) environment . Even in a gray-box setting , black-box algorithm engineering can be useful to ensure that no illegitimate assumptions are made in their design . On the other hand , we aim to profit from the added information . Real applications often have a gray-box character , exposing a certain level of expert knowledge about the correlation between variables . Let us take for example the common application of Neuroevolution ( Stanley et al. , 2019 ) , where evolutionary algorithms ( such as ES ) are applied to learn the weights of a neural network . We know from the network ’ s equation that the weights of the connections entering the same neuron ( as opposed to entering different neurons ) are by necessity highly correlated , as they are composed in a linear fashion inside the neuron prior to activation . The same reasoning can be made for the weights of connections entering neurons belonging to the same layer , versus neurons in different layers , because outputs of a layer are linearly combined in the next layer . This principle remains true as the network expands : in complex applications such as those requiring deep networks , the correlation between weights of connections in the early versus late layers will be by design much more limited than in the neuron or layer examples above1 . Based on the above insights , this paper proposes a new framework for Distributed Black Box optimization , named DiBB , see Figure 1 . We leverage the assumption of partial correlation by partitioning the variables set into highly intra-correlated blocks ; then , switching the assumption to partial separability , we search each block with a different instance of the reference BBO algorithm , independently . This makes DiBB particularly suited for neuroevolution applications requiring larger/deep networks . In Section 2 , we provide rigorous theoretical arguments for the sample efficiency of this approach . From a practical perspective , this allows to search blocks asynchronously and even to distribute each BBO instance on different nodes in a cluster . Varying the partition of the variables enables the user to arbitrarily trade-off between wall-clock speed ( smaller blocks ) and sample efficiency ( larger blocks ) , while distributing the BBO instances decouples the complexity from the number of variables , as long 1The fundamental assumption of partial separability across network layers however does not seem to be very well studied in the literature , despite a considerable body of work on neural network loss landscapes ( Fort & Jastrzebski , 2019 ; Soltanolkotabi et al. , 2018 ) . This can be partially tracked to the limited availability of algorithms making use of partial correlation information . as enough machines are available to run the BBO instances . A major challenge comes with solution evaluation , as the variables constituting a sample are practically scattered across a network . This is addressed in Section 3 , by establishing a rather sparse communication scheme . It is important to notice at this point that no assumption or claim has yet been made on the underlying BBO algorithm of choice—nor will be made . While this paper explores the applicability of DiBB in the context of modern , sophisticated ESs , DiBB can be applied to any BBO algorithm with minimal interfaces . We analyze the performance of our framework on the classic COCO benchmark , both on the BBOB and BBOB large-scale suites , using the industry-standard CMA-ES as a reference , and as a blocklevel optimizer within DiBB . The sample complexity of our approach typically ( and as expected ) sits in between the full-covariance and the diagonal-covariance versions of CMA-ES , with few notable exceptions where maintaining extra covariance information is actually deceptive/misleading due to separability . Notably , on parallel hardward , DiBB is considerably faster in terms of wall clock time . To demonstrate the scalability of our algorithms , we present results on training a 20-layer neural network using 20 blocks on 20 machines , the largest neural network trained with neuroevolution to date , to the best of our knowledge . 1.1 CONTRIBUTIONS . Our key contributions are as follows : • We provide a novel way of parallelizing an ES , going beyond the parallel evaluation of candidate solutions forming a population , by optimizing blocks of variables with multiple ES instances . • We exploit the block structure to add covariance matrix information at low cost where it matters most , namely inside blocks of potentially highly correlated variables . • The resulting DiBB framework is particularly well-suited for neuroevolution . However , it can be applied to any BBO method . • We demonstrate that also large and deep networks can be trained efficiently with neuroevolution on highly parallel hardware . 2 A PRIMER ON EVOLUTION STRATEGIES . ES are direct search methods , which optimize a black-box objective function f : Rd → R by sampling candidate points from an adaptive Gaussian distribution . We briefly review the types of ES most relevant for our discussion . The classic variant is the ( 1+1 ) -ES ( Rechenberg , 1973 ) . Its central algorithmic mechanism is step size adaptation , i.e. , its ability to actively adapt the standard deviation σ > 0 of its Gaussian sampling distribution N ( m , σ2I ) to the current needs . For about 20 years , CMA-ES ( Hansen & Ostermeier , 2001 ) is the gold standard in ES research . Many variants exist , such as Natural Evolution Strategies ( NES ; Wierstra et al . 2014 ) . Its most important mechanism going beyond “ simple ” step-size adaptive ES is covariance matrix adaptation ( CMA ) , which means that not only the global step size σ , but also the full covariance matrix C of the Gaussian N ( m , σ2C ) is adapted to the problem at hand . CMA-ES is a powerful optimizer ; however , it was not designed for high-dimensional applications with hundreds of thousands of variables . Its internal parameters are not tuned with such a regime in mind , and learning a full covariance matrix with d ( d+1 ) 2 parameters is inherently slow . Such problems are best addressed by restricting C to a diagonal matrix ( Ros & Hansen , 2008 ) , to a diagonal plus a low-rank matrix ( Loshchilov , 2014 ; Akimoto et al. , 2014 ; Loshchilov et al. , 2018 ) , or to a blockdiagonal matrix ( Cuccu & Gomez , 2012 ) . The number of parameters of the covariance matrix can hence be chosen rather flexibly in the range d to d ( d+1 ) 2 . 2.1 ES FOR NEUROEVOLUTION . The application of ES to machine learning problems and to RL in particular has a long history ( Igel , 2003 ; Heidrich-Meisner & Igel , 2009 ) . In 2017 , the work of Salimans et al . ( 2017 ) sparked a renewed interest in ES by showcasing how to successfully exploit the embarrassing parallel nature of objective function evaluations in populations-based algorithms , considerably speeding-up the learning process . This triggered a large body of work on neuroevolution based on ES , see e.g . Plappert et al . ( 2017 ) ; Ha & Schmidhuber ( 2018 ) ; Chrabaszcz et al . ( 2018 ) ; Stanley et al . ( 2019 ) and references therein . 2.2 CONVERGENCE RATES AND COMPUTATIONAL COMPLEXITY . Due to Taylor ’ s theorem , local optima of C2 functions in d-dimensional space are well approximated ( up to O ( ‖x− x∗‖3 ) ) by convex quadratic functions f ( x ) = 12 ( x− x ∗ ) TH ( x− x∗ ) . The computational complexity of solving this problem with an ES to a fixed target precision ε > 0 is of the form O ( d · κ ( H ) · log ( 1/ε ) ) , where κ ( H ) denotes the condition number of the Hessian H ( Jägersküpper , 2006 ; Hansen et al. , 2015 ) . Hence , a step-size adaptive ES achieves linear convergence with rate O ( 1/ ( d · κ ( H ) ) ) . The linear dependency on d is optimal for comparison-based optimization ( Fournier & Teytaud , 2011 ) , but the dependency on H is sub-optimal . The advantage of maintaining covariance information is that the factor κ ( H ) is improved to κ ( HC∗ ) , where C∗ denotes the optimal covariance matrix available to the ES . When approaching C∗ = H−1 , methods maintaining full covariance achieve the optimal value κ ( HC∗ ) = 1 . In effect , as expected from a pseudo second order method , the convergence rate is independent of H . A diagonal covariance matrix C∗ acts as a diagonal pre-conditioner , with varying effectiveness depending on the problem . Obviously , the block-diagonal and the low-rank cases are in-between . The OpenAI-ES ( Salimans et al. , 2017 ) features neither step size adaptation nor covariance adaptation . Based on the NES framework of Wierstra et al . ( 2014 ) , it leverages the ability of ES to estimate the natural gradient of f from samples , and then applies the ADAM optimizer on top . In effect , this is roughly comparable to using diagonal C. CMA has a price in terms of algorithm internal complexity , and in addition the adaptation process is slow in terms of sample complexity . The above convergence rates measure time in terms of objective function evaluations ( sample complexity ) . When scaling up CMA to high dimensions , we need to take the following concepts into consideration . Algorithm internal complexity refers to the required ( amortized ) number of operations needed for creating a sample and for updating the internal state—the covariance matrix in particular . Regarding sample complexity , we distinguish between the number of samples needed to learn the covariance matrix , and the number of samples needed to solve the problem . Learning C with up to Θ ( d2 ) parameters is sample-inefficient . For example , a ( small ) network with d = 104 weights results in d ( d+1 ) 2 ≈ 5 · 10 7 parameters of the covariance matrix . Learning these takes hundreds of millions of samples , each of which can be a full RL episode . This is too slow for being useful . For larger d , even the storage of the full covariance matrix C becomes prohibitive . Furthermore , performing computations with C scales at least linear with the number of its parameters , which amounts to an internal complexity of Ω ( d2 ) for full CMA . Since network evaluation scales linearly with the number of weights d , CMA quickly becomes the computational bottleneck . Therefore , a different trade-off between fast convergence and internal complexity is needed , which can be realized for example with block-diagonal and low-rank structures , and combinations thereof .

Summary: This paper proposes Distributed Black Box optimization (DiBB), which involves using disjoint distributed pipelines to perform CMA/Hessian-based updates, over functions with assumed separability in terms of parameters. The method is explained in detail, and experiments are performed on BBOB functions (with varying dimensions), along with Mujoco Walker2d with a relatively large policy architecture (10K+ parameters). Other ablations such as wall-clock time and varying block sizes are also experimentally presented.