Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking

1 INTRODUCTION : STATE OF THE ART . Many real-world optimization challenges are black-box problems ; i.e. , instead of having an explicit problem formulation , they can only be accessed through the evaluation of solution candidates . These evaluations often require simulations or even physical experiments . Black-box optimization methods are particularly widespread in machine learning ( Salimans et al. , 2016 ; Wang et al. , 2020 ) , to the point that it is considered a key research area of artificial intelligence . Black-box optimization algorithms are typically easy to implement and easy to adjust to different problem types . To achieve peak performance , however , proper algorithm selection and configuration are key , since black-box optimization algorithms have complementary strengths and weaknesses ( Rice , 1976 ; Smith-Miles , 2009 ; Kotthoff , 2014 ; Bischl et al. , 2016 ; Kerschke & Trautmann , 2018 ; Kerschke et al. , 2018 ) . But whereas automated algorithm selection has become standard in SAT solving ( Xu et al. , 2008 ) and AI planning ( Vallati et al. , 2015 ) , a manual selection and configuration of the algorithms is still predominant in the broader black-box optimization context . To reduce the bias inherent to such manual choices , and to support the automation of algorithm selection and configuration , sound comparisons of the different black-box optimization approaches are needed . Existing benchmarking suites , however , are rather selective in the problems they cover . This leads to specialized algorithm frameworks whose performance suffer from poor generalizability . Addressing this flaw in black-box optimization , we present a unified benchmark collection which covers a previously unseen breadth of problem instances . We use this collection to develop a high-performing algorithm selection wizard , ABBO . ABBO uses high-level problem characteristics to select one or several algorithms , which are run for the allocated budget of function evaluations . Originally derived from a subset of the available benchmark collection , in particular YABBOB , the excellent performance of ABBO generalizes across almost all settings of our broad benchmark suite . Implemented as a fork of Nevergrad ( Rapin & Teytaud , 2018 ) , the benchmark collection , the ABBO wizard , the base solvers , and all performance data are open source . The algorithms are automatically rerun at certain time intervals and all Algorithm 1 High-level overview of ABBO . Selection rules are followed in this order , first match applied . d = dimension , budget b = number of evaluations . Details in ( Anonymous , 2020 ) . Case Choice Discrete decision variables only Noisy optimization with categorical variables Genetic algorithm mixed with bandits ( Heidrich-Meisner & Igel , 2009 ; Liu et al. , 2020 ) . alphabets of size < 5 , sequential evaluations ( 1 + 1 ) -Evolutionary Alg . with linearly decreasing stepsize alphabets of size < 5 , parallel case Adaptive ( 1 + 1 ) -Evolutionary Alg . ( Doerr et al. , 2019 ) . Other discrete cases with finite alphabets Convert to the continuous case using SoftMax as in ( Liu et al. , 2020 ) and apply CMandAS2 ( Rapin et al. , 2019 ) Presence of infinite discrete domains FastGA ( Doerr et al. , 2017 ) Numerical decision variables only , evaluations are subject to noise d > 100 progressive optimization as in ( Berthier , 2016 ) . d ≤ 30 TBPSA ( Hellwig & Beyer , 2016 ) b > 100 sequential quadratic programming Other cases TBPSA ( Hellwig & Beyer , 2016 ) Numerical decision variables only , high degree of parallelism Parallelism > b/2 or b < d MetaTuneRecentering ( Meunier et al. , 2020 ) Parallelism > b/5 , d < 5 , and b < 100 DiagonalCMA-ES ( Ros & Hansen , 2008 ) Parallelism > b/5 , d < 5 , and b < 500 Chaining of DiagonalCMA-ES ( 100 asks ) , then CMA-ES+metamodel ( Auger et al. , 2005 ) Parallelism > b/5 , other cases NaiveTBPSA as in ( Cauwet & Teytaud , 2020 ) Numerical decision variables only , sequential evaluations b > 6000 and d > 7 Chaining of CMA-ES and Powell , half budget each . b < 30d and d > 30 ( 1 + 1 ) -Evol . Strategy w/ 1/5-th rule ( Rechenberg , 1973 ) d < 5 and b < 30d CMA-ES + meta-model ( Auger et al. , 2005 ) b < 30d Cobyla ( Powell , 1994 ) For all other cases and all details , please refer to the source code data is exported to the public dashboard ( Rapin & Teytaud , 2020 ) . For ICLR reviewers , all code is available , thanks to github-anonymizer , at ( Anonymous , 2020 ) . In summary , our contributions are as follows . ( 1 ) OptimSuite Benchmark Collection : OptimSuite combines several contributions that recently led to improved reliability and generalizability of black-box optimization benchmarking , among them LSGO ( Li et al. , 2013 ) , YABBOB ( Hansen et al. , 2009 ; Liu et al. , 2020 ; Anonymous , 2020 ) , Pyomo ( Hart et al. , 2017 ; Anonymous , 2020 ) , MLDA ( Gallagher & Saleem , 2018 ) , and MuJoCo ( Todorov et al. , 2012 ; Mania et al. , 2018 ) , and others ( novelty discussed in Section 2 ) . ( 2 ) Algorithm Selection Wizard ABBO : Our algorithm selection technique , ABBO , can be seen as an extension of the Shiwa wizard presented in ( Liu et al. , 2020 ) . It uses three types of selection techniques : passive algorithm selection ( choosing an algorithm as a function of a priori available features ( Baskiotis & Sebag , 2004 ; Liu et al. , 2020 ) ) , active algorithm selection ( a bet-and-run strategy which runs several algorithms for some time and stops all but the strongest ( Mersmann et al. , 2011 ; Pitzer & Affenzeller , 2012 ; Fischetti & Monaci , 2014 ; Malan & Engelbrecht , 2013 ; Muñoz Acosta et al. , 2015 ; Cauwet et al. , 2016 ; Kerschke et al. , 2018 ) ) , and chaining ( running several algorithms in turn , in an a priori defined order ( Molina et al. , 2009 ) ) . Our wizard combines , among others , algorithms suggested in ( Virtanen et al. , 2019 ; Hansen & Ostermeier , 2003 ; Storn & Price , 1997 ; Powell , 1964 ; 1994 ; Liu et al. , 2020 ; Hellwig & Beyer , 2016 ; Artelys , 2015 ; Doerr et al. , 2017 ; 2019 ; Dang & Lehre , 2016 ) . Another core contribution of our work is a sound comparison of our wizard to Shiwa , and to the long list of algorithms available in Nevergrad . 2 SOUND BLACK-BOX OPTIMIZATION BENCHMARKING . We summarize desirable features and common shortcomings of black-box optimization benchmarks and discuss how OptimSuite addresses these . Generalization . The most obvious issue in terms of generalization is the statistical one : we need sufficiently many experiments for conducting valid statistical tests and for evaluating the robustness of algorithms ’ performance . This , however , is probably not the main issue . A biased benchmark , excluding large parts of the industrial needs , leads to biased conclusions , no matter how many experiments we perform . Inspired by ( Recht et al. , 2018 ) in the case of image classification , and similar to the spirit of cross-validation for supervised learning , we use a much broader collection of benchmark problems for evaluating algorithms in an unbiased manner . Another subtle issue in terms of generalization is the case of instance-based choices of ( hyper- ) parameters : an experimenter modifying the algorithm or its parameters specifically for each instance can easily improve results by a vast margin . In this paper , we consider that only the following problem properties are known in advance ( and can hence be used for algorithm selection and configuration ) : the dimension of the domain , the type and range of each variable , their order , the presence of noise ( but not its intensity ) , the budget , the degree of parallelism ( i.e. , number of solution candidates that can be evaluated simultaneously ) . To mitigate the common risk of over-tuning , we evaluate algorithms on a broad range of problems , from academic benchmark problems to real-world applications . Each algorithm runs on all benchmarks without any change or task-specific tuning . Use the ask , tell , and recommend pattern . Formalizing the concept of numerical optimization is typically made through the formalism of oracles or parallel oracles ( Rogers , 1987 ) . A recent trend is the adoption of the ask-and-tell format ( Collette et al. , 2010 ) . The bandit literature pointed out that we should distinguish ask , tell , and recommend : the way we choose a point for gathering more information is not necessarily close to the way we choose an approximation of the optimum ( Bubeck et al. , 2011 ; Coulom , 2012b ; Decock & Teytaud , 2013 ) . We adopt the following framework : given an objective function f and an optimizer , for i ∈ { 1 , . . . , T } , do x ← optimizer.ask and optimizer.tell ( x , f ( x ) ) . Then , evaluate the performance with f ( optimizer.recommend ) . The requirement of a recommend method distinct from the ask is critical in noisy optimization . A debate pointed out some shortcomings in the the noisy counterpart of BBOB ( Auger & Hansen , 2009 ) which was assuming that ask = recommend : ( Beyer , 2012a ; b ; Coulom , 2012a ) have shown that in the noisy case , this difference was particularly critical , and a framework should allow algorithms to “ recommend ” differently than they “ ask ” . A related issue is that a run with budget T is not necessarily close to the truncation of a run in budget 10T . Translation-invariance . Zero frequently plays a special role in optimization . For example , complexity penalizations often “ push ” towards zero . In control , numbers far from zero are often more likely to lead to bang-bang solutions ( and therefore extract zero signal , leading to a needle-inthe-haystack optimization situation ) , in particular with neural networks . In one-shot optimization , ( Cauwet et al. , 2019 ; Meunier et al. , 2020 ) have shown how much focusing to the center is a good idea in particular in high-dimension . Our experiments in control confirm that the scale of the optimization search is critical , and explains the misleading results observed in some optimization papers ( Section 4.2 ) . In artificial experiments , several classical test functions have their optimum in zero . To avoid misleading conclusions , it is now a standard procedure , advocated in particular in ( Hansen et al. , 2009 ) , to randomly translate the objective functions . This is unfortunately not always applied . Rotation and symmetrization . Some optimization methods may perform well on separable objective functions but degrade significantly in optimizing non-separable functions . If the dimension of a separable objective function is d , these methods can reduce the objective function into d onedimensional optimization processes ( Salomon , 1996 ) . Therefore , Hansen et al . ( 2009 ; 2011 ) have insisted that objective functions should be rotated to generate more difficult non-separable objective functions . However , Bousquet et al . ( 2017 ) pointed out the importance of dummy variables , which are not invariant per rotation ; and ( Holland , 1975 ) and more generally the genetic algorithms literature insist that rotation does not always makes sense – we lose some properties of a real-world objective function , and in some real-world scenarios rotating would , e.g. , mix temperature , distance and electric intensity . Permutating the order of variables is also risky , as their order is sometimes critical - k-point crossovers a la Holland ( Holland , 1975 ) typically assume some order of variables , which would be broken . Also , users sometimes rank variables with the most important first – and some optimization methods do take care of this ( Cauwet et al. , 2019 ) . In OptimSuite , we do include rotations , but include both cases , rotated or not . For composite functions which use various objective functions on various subsets of variables , we consider the case with rotations – without excluding the non-rotated case . An extension of symmetrization that we will integrate later in ABBO , which makes sense for replicating an objective function without exact identity , consists in symmetrizing some variables : for example , if the ith variable has range [ a , b ] , we can replace xi by b + a − xi . Applying this on various subsets of variables leads to 2d symmetries of an objective function , if the dimension is d. This variation can reduce the bias toward symmetric search operations ( Li et al. , 2013 ) . Benchmarking in OptimSuite . We summarize in Table 1 some existing benchmark collections and their ( desirable ) properties . We inherit various advantages from Nevergrad , namely the automatic rerun of experiments and reproducibility in one-line . Our fork includes PBT ( a small scale version of Population-Based Training ( Jaderberg et al. , 2017 ) ) , Pyomo ( Hart et al. , 2017 ) , Photonics ( problems related to optical properties and nanometric materials ) , YABBOB and variants , LSGO ( Li et al. , 2013 ) , MLDA ( Gallagher & Saleem , 2018 ) , PowerSystems , FastGames , 007 , Rocket , SimpleTSP , Realworld ( Liu et al. , 2020 ) , MuJoCo ( Todorov et al. , 2012 ) and others including a ( currently small ) benchmark of hyperparameters of Scikit-Learn ( Pedregosa et al. , 2011 ) and Keras-tuning , all of those being visible for review at the above-mentioned anonymized URL ( underlined means : the benchmark is either new , or , in the case of PowerSystems or SimpleTSP , significantly modified compared to previous works , or , in the case of LSGO or MuJoCo , included for the first time inside Nevergrad . For MuJoCo , we believe that interfacing with Nevergrad is particularly useful , to ensure fair comparisons , which rely very much on the precise setup of MuJoCo. . We note that , at present , we do not reproduce the extreme black-box nature of Loshchilov & Glasmachers ( 2017 ) . Still , by integrating such a wide range of benchmarks in a single open source framework , which , in addition , is periodically re-run , we believe that Nevergrad/OptimSuite provides a significant contribution to benchmarking , and this both for the optimization and the machine learning community , where most of the benchmark suites originate from .

Summary: The paper proposes a benchmarking suite to overcome the problem low generalizability with black box optimization algorithm. The benchmarking suite consists of standard academic benchmarks to real world optimization problems. It also covers several scenarios such as dynamic-static, small to large-scale, discrete to mixed-integer etc. This is a relevant contribution to the machine learning however there are several drawbacks which pushes back it's acceptance into ICLR.