Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking
Authors: Laurent Meunier
Herilalaina Rakotoarison
Jeremy Rapin
Paco Wong
Baptiste Roziere
Olivier Teytaud
Antoine Moreau
Carola Doerr
Venue:  (2021)
Keywords: black-box optimization
mujoco
wizard
benchmarking
BBOB
LSGO

Abstract: Existing studies in black-box optimization for machine learning suffer from low
generalizability, caused by a typically selective choice of problem instances used
for training and testing different optimization algorithms. Among other issues,
this practice promotes overfitting and poor-performing user guidelines. To address
this shortcoming, we propose in this work a benchmark suite, OptimSuite,
which covers a broad range of black-box optimization problems, ranging from
academic benchmarks to real-world applications, from discrete over numerical
to mixed-integer problems, from small to very large-scale problems, from noisy
over dynamic to static problems, etc. We demonstrate the advantages of such a
broad collection by deriving from it Automated Black Box Optimizer (ABBO), a
general-purpose algorithm selection wizard. Using three different types of algorithm
selection techniques, ABBO achieves competitive performance on all
benchmark suites. It significantly outperforms previous state of the art on some of
them, including YABBOB and LSGO. ABBO relies on many high-quality base
components. Its excellent performance is obtained without any task-specific
parametrization. The benchmark collection, the ABBO wizard, its base solvers,
as well as all experimental data are reproducible and open source in OptimSuite.

Under review as a conference paper at ICLR 2021
BLACK-BOX OPTIMIZATION REVISITED: IMPROVING ALGORITHM SELECTION WIZARDS THROUGH MAS- SIVE BENCHMARKING
Anonymous authors Paper under double-bling blind review
ABSTRACT
Existing studies in black-box optimization for machine learning suffer from low generalizability, caused by typically selective choice of problem instances used for training and testing different optimization algorithms. Among other issues, this practice promotes overfitting and poor poor-perfrring user guidelines. To ad- dress this shortcoming, we propose in this work a benchmark suite, OptimSuite, which covers broad range of black-box optimization problems, ranging from academic benchmarks to real-world applications, from discrete over numerical mixed-integer problems, from small to very large-scale problems, from noisy over dynamic to static problems, etc. We demonstrate the advantages of such a broad collection by deriving from Automated Black Box Optimizer (ABBO), a general-purpose algorithm selection wizard. Using three different types of al- gorithm selection techniques, ABBO achieves competitive performance on all benchmark suites. It significantly outperforms previous state of the art on some of them, including YABBOB and LSGO. ABBO relies on many high-quality base components. Its excellent performance is obtained without any task-specific parametrization. The benchmark collection, the ABBO wizard, its base solvers, well as all experimental data are reproducible and open source in OptimSuite.
INTRODUCTION: STATE OF THE ART
Many real-world optimization challenges are black-box problems; i.c., instead of having an explicit problem formulation, they can only be accessed through the evaluation of solution candidates. These evaluations often require simulations or even physical experiments. Black- box optimization meth- ods are particularly widespread in machine learning (Salimans al.] [016]] Wang al.) 2020). to the point that it is considered a key research area of artificial intelligence. Black-box optimization algorithms are typically easy to implement and easy to adjust to different problem types. To achieve peak performance, however, proper algorithm selection and configuration are key, since black-box optimization algorithms have complementary strengths and weaknesses (Rice. 1976. Smith-Miles! 2009 Kotthotf, 2014 BBschl et al. 2016 Kerschke & Trautmann 2018 Kerschke et al.
2018). But
whereas automated algorithm selection has become standard in SAT solving (Xu et al. 2008 and
Al planning (Vallati et al. 2015), manual selection and configuration of the algorithms still pre- dominant in the broader black-box optimization context. To reduce the bias inherent to such manual choices, and support the automation of algorithm selection and configuration, sound comparisons of the different black-box optimization approaches are needed. Existing benchmarking suites, however, are rather selective in the problems they cover. This leads to specialized algorithm frame- works whose performance suffer from poor generalizability. Addressing this flaw in black-box op- timization, we present a unified benchmark collection which covers previously unseen breadth of problem instances. We use this collection to develop high-performing algorithm selection wizard, ABBO. ABBO uses high-level problem characteristics to select one or several algorithms, which are run for the allocated budget of function evaluations. Originally derived from a subset of the avail- able benchmark collection, in particular YABBOB, the excellent performance of ABBO generalizes across almost all settings of our broad benchmark suite. Implemented & Teytaud, 2018). the benchmark collection, the ABBO wizard, the base solvers, and all perfor- mance data open source. The algorithms are automatically rerun at certain time intervals and all
fork of Nevergrad (Rapin)

---

Under review as a conference paper at ICLR 2021
Algorithm High-level overview of ABBO. Selection rules are followed in this order, first match applied. dimension, budget b number of evaluations. Details in (Anonymous) 2020).
Case
Choice
Discrete decision variables only
Noisy optimization with categorical variables
Genetic algorithm mixed with bandits Meidriih-MMissen & Igel
2009
al. 2020)
alphabets of size 5, sequential evaluations
(1
Evvoooooorrrryy Alg. with lineurly decreasing stepsize
alphabets of 5, parallel case
Adaptive (1 + 1). Exolutionay Alg. (Doerr et al. 2019
Other discrete cases with finite alphahets
Convert to the continuous case using Soorma II m [CB ot
2020 and apply CMandAS2 (Rapin el al. 2019
Presence of infinite discrete domains
FasiGA (Doem al.]] (2017)
Numerical decision variables only, evaluations are subject to moise
> 100
prograasive optimization as in (Bertherr 2016]]
d S 30
TBPSA (Tellwig & Beyer 2016)
>> 100 cases
sequential quatiratc programming
TBPSA (Hellwig Beyer 2016)
Numerical decision vartables only, high degree of parallelism
Parallelism >> 6/2 or b <
MetaTunneReentering (Meunier et (2220
Parallelism 6/5. d G, and 6 100
DiagonalCMA-ES (Ros Hanssn 2008
Parallelism > 6/5, d < 5, and b < 500
Chaining of Diagonal (1O0 aSk)), then CMA-ES+meta-
model (Auger el 2005)
Parallelism 6/5, other cases
NaiveIBPSA 3s CSuwet & Teyland 2020
Numerical decision variables only, sequential evalnations
> 6000 and d >
Chaining CMA. ES Powell, budget each.
b < 30d and d > 30
(1 + [)-Fvol. Strategy w/ 1/5-th nile (Rechenherg) 1973
< 5 and 6 30d
CMA-ES + meta-model (Auger al, 2005)
< 30d
Cohyla (Powell 1994).
For all other cases and all details, please refer to the source cude
data is exported to the public dashboard (Rapin & Teytaud 2020). For ICLR reviewers, all code is available, thanks to github-anonymizee at (Anonymous) 2020).
In summary, our contributions are as follows. (1) OptimSuite Benchmark Collection: Optim-
Suite combines several contributions that recently led to improved reliability and generalizability
of black-box optimization benchmarking, among them LSGO (Li et al. 2013). YABBOB (Hansen
eet al. 2009 Liu et al., 2020; Anonymous 2020), Pyomo (Hart et al, 2017 Anonymous 2020).
MLDA (Gallagher & Saleem 2018), and MuJoCo (Todorov et al. 2012 Mania et al. (2018), and
others (novelty discussed in Section 2 (2) Algorithm Selection Wizard ABBO: Our algorithm
selection technique, ABBO, can be seen as an extension of the Shiwa wizard presented in (Liu
et al. 2020). It uses three types of selection techniques: passive algorithm selection (choosing an algorithm as a function of a priori available features (Baskiotis & Sebag, 2004] Liu et al.) ( active algorithm selection (a bet-and-run strategy which runs several algorithms for some time and stops all but the strongest (Mersmann et al.] 201 Pitzer & Affenzeller, 2012 2012 Fischetti & Monaci 2014 Malan & Engelbrecht [013] Muñoz Acosta et al., 2015| Cauwet al.) 2016 Kerschke al.
2018 and chaining (running several algorithms in turn, in an priori defined order (Molina et al.,
2009
). Our wizard combines, among others, algorithms suggested in (Virtanen et al. 2019 Hansen
& Ostermeier 2003
Storn & Price 1997. PPweell 1964. 1994 Liu et al. 2020 Hellwig & Beyer 2015 Doerr et al.] 2017] 2019; Dang & Lehre| 2016). Another core contribution of
2016 Artelys
our work is sound comparison of wizard to Shiwa, and to the long list of algorithms available in Nevergrad.
2 SOUND BLACK-BOX OPTIMIZATION BENCHMARKING
We summarize desirable features and common shortcomings of black-box optimization benchmarks and discuss how OptimSuite addresses these.
Generalization. The most obvious issue in terms of generalization the statistical one: we need sufficiently many experiments for conducting valid statistical tests and for evaluating the robustness of algorithms' performance. This, however, is probably not the main issue. A biased benchmark, excluding large parts of the industrial needs, leads to biased conclusions, no matter how many ex- periments we perform. Inspired by (Recht et al. 2018) in the case of image classification, and similar to the spirit of cross-valine cross-validation for supervised learning, we use much broader collection of benchmark problems for evaluating algorithms in an unbiased manner. Another subtle issue in experimenter
terms of generalization is the case of Setance-based choices of
2

---

Under review as a conference paper at ICLR 2021
Testbed
BBOB
MuloCo
ESGO
Nevergrad
BRClomp
OptimSuite
Large
Translations
Symmetrizations rotations
One line reproducibilit
Periodic automated dashhoard
Complex real-world
Multimodal
Open sourced / no license
Askheellrecommend correct
Far-optimum
Human excluded / client
Properties of selected benchmark collections (details in appendix A). "+" means that the feature is present, "_" that the feature is missing, and empty case that it is not applicable. Far-optimum refers to problems with optimum far from the center or on the side of the domain; such benchmarks test the ability of optimization algorithms to answer promptly to bad initializa- tion (Chotard et al. 2012). "Translations" applies only to artificial benchmarks. A "+" in rows "Multimodal", "symmetrization", and "real-world" does not imply that all test functions have this property. "Open sourced" refers to open access to most algorithms involved in the published com- parison: here, *_" refers to license issues for the benchmark itself.
Table 1:
modifying the algorithm or its parameters specifically for each instance can easily improve results by a vast margin. In this paper, we consider that only the following problem properties are known in advance (and can hence be used for algorithm selection and configuration): the dimension of the domain, the type and range of each variable, their order, the presence of noise (but not its inten- sity), the budget, the degree of parallelism (i.e., number of solution candidates that can be evaluated simultaneously). To mitigate the common risk of over-tuning, we evaluate algorithms on a broad range of problems, from academic benchmark problems to real-world applications. Each algorithm runs on all benchmarks without any change or task-specific tuning.
Use the ask, tell, and recommend pattern. Formalizing the concept of numerical optimization is typically made through the formalism of oracles or parallel oracles (Rogers) 1987). A recent trend the adoption of the ask-and-tell format (Collette et al., 2010). The bandit literature pointed out that we should distinguish ask, tell, and recommend: the way we choose a point for gathering more information is not necessarily close the way we choose an approximation of the optimum (Bubeck) et al. 2011 Coulom, 2012b, Decoek & Teytaud 2013). We adopt the following framework: given an objective function / and an oplimizer, for i (1...., T}. do & optimizer.telle tell(x, f (x)). Then, evaluate the performance with requirement of recommend method distinct from the ask is critical in noisy optimization. A de- bate pointed out some shortcomings in the the noisy counterpart of BBOB (Auger & Hansen 2009) which was assuming that ask = recommend: (Beyer) 2012a b) Coulom 2012a) have shown that in the noisy case, this difference was particularly critical, and a framework should allow algorithms to "recommend" differently than they "ask". A related issue is that a run with budget is not necessarily close the truncation of a run in budget 107.
optimizer.co and The
Translation-invaiianee. Zero frequently plays a special role in optimization. For example, com- plexity penalizations often "push" towards zero. In control, numbers far from zero are often more likely to lead to bang-bang solutions (and therefore extract zero signal, leading to a needle-in- the-haystack optimization situation), in particular with neural networks. In one-shot optimiza- tion. (Cauwet et al. 2019 Meunier et al. 2020) have shown how much focusing to the center is a good idea in particular in high-dimension Our experiments in control confirm that the scale of the optimization search is critical, and explains the misleading results observed in some optimization papers (Section 4.2) In artificial experiments, several classical test functions have their optimum in zero. To avoid misleading conclusions, it is now standard procedure, advocated in particular in (Hansen et al. 2009), to randomly translate the objective functions. This unfortunately not always applied.
Rotation and symmetrization. Some optimization methods may perform well on separable ob- jective functions but degrade significantly in optimizing non-separable functions. If the dimension of a separable objective function is d, these methods can reduce the objective function into d one- dimensional optimization processes (Salomon 1996). Therefore, Hansen al. 2009 2011) have
3

---

Under review as a conference paper at ICLR 2021
insisted that objective functions should be rotated to generate more difficult non-separable objective functions. However, Bousquet et al. (2017) pointed out the importance of dummy variables, which not invariant per rotation; and (Holland 1975) and more generally the genetic algorithms lit- erature insist that rotation does not always makes sense - we lose some properties of a real-world objective function, and in some real-world scenarios rotating would, e.g., mix temperature, distance and electric intensity. Permutating the order of variables is also risky, as their order is sometimes critical- k-point crossovers la Holland (Holland 1975) typically assume some order of variables, which would be broken. Also, users sometimes rank variables with the most important first and some optimization methods do take care of this (Cauwet et al. (019). In OptimSuite, we do include rotations, but include both cases, rotated or not. For composite functions which use various objective functions on various subsets of variables, we consider the case with rotations without excluding the non-rotated case. An extension of symmetrization that we will integrate later in ABBO, which makes sense for replicating an objective function without exact identity, consists in symmetrizing some variables: for example, if the 7th variable has range 6], we can replace 24 by 6 + Z4- Applying this on various subsets of variables leads to symmetries of an objective function, if the dimension is d. This variation can reduce the bias toward symmetric search operations (Li et al., 2013).
Benchmarking in OptimSuite. We summarize in Table D some existing benchmark collections and their (desirable) properties. We inherit various advantages from Nevergrad, namely the automatic rerun of experiments and reproducibility in one-line. Our fork includes PBT (a small scale version of Population- -aaiattioo-BBeddd Training (Jaderberg et al. (0177), Pyomo (Hart et al. (017), Photonics (problems related optical properties and nanometric materials), YABBOB and variants, LSGO (Li et al. 2013) MLDA Gaallagher& & Saleem 2018), PowerSystems, FastGames, 007, Rocket, SimpleTSP, Realworld (Liu er |0,00 MujoCo Todorov et al. 2012)) and others including (currently small) benchmark of hyperparameters of Scikit-Learn (Pedregosa et al., (011) and Keras-tuning, all of those being visible for review at the above-mentioned anonymized URL (underlined means: the benchmark is either new, or, in the case of PowerSystems or SSP, significantly modified compared to previous works, or, in the case of LSGO or MuJoCo, included for the first time inside Nevergrad. For MuJoCo, we believe that interfacing with Nevergrad is particularly useful, to ensure fair comparisons, which rely very much on the precise setup of MuloCo. We note that, at present, do not reproduce the extreme black-box nature of Loshchilov & Glasmachers (2017). Still, by integrating such a wide range of benchmarks in single open source framework, which, in addition, provides a significant contribution to
periodically re-run, we believe that Nevergrad/OptimSuun benchmarking, and this both for the optimization and the machine learning community, where most of the benchmark suites originate from.
A NEW ALGORITHM SELECTION WIZARD: ABBO
Black-box optimization is sometimes dominated by evolutionary computation. Evolution strate- gies (Beyer & Schwefel 2002 Beyer, 200T Rechenberg 1973) have been particularly dominant in the continuous case, in experimental comparisons based on the Black-Box Optimization Benchmark
BBOB (Hansen et al. 2009) or variants thereof. Parallelization advantages (Salimans et al.] (016)) particularly appreciated in the machine learning context. However, Differential Evolution (Storn
& Price. 1997) is a key component of most winning algorithms in competitions based on variants of Large Scale Global Optimization (LSGO (Li et al.) 2013), suggesting a significant difference between these benchmarks. In particular, LSGO is more based on correctly identifying a partial decomposition and scaling to > 1000 variables, whereas BBOB focuses (mostly, except (Vare- las et al. 2018) on < 40 variables. Mathematical programming techniques (Powell 1964 1994]
Nelder & Mead 1965) Artelys| 2015) are rarely used in the evolutionary computation world, but they have won competitions (Artelys 2015) and significantly improved evolution strategies through memetic methods (Radcliffe & Surry 1994). Algorithm selection was applied to continuous black- box optimization and pushed in Nevergrad Liu et al.) (2020): their optimization algorithm com- bines many optimization methods and outperforms each of them when averaged over diverse test functions. Closer to machine learning, efficient global optimization (Jones et al. 1998) is widely used, although it suffers from the curse of dimensionality more than other methods (Snoek et al., 2012); (Wang et al. 2020) presented
of-te-of-the-art algorithm in black-box optimization JoCo, i.c., for the control of various realistic robots (Todorov et al.] 2012). We propose ABBO, which extends (Liu et al. 2020 by the following features: (1) Better use of chaining (Molina et al.
Mu-
4

---

Under review as a conference paper at ICLR 2021
Benchmark
Use for ABBO of contgs
ranking Shiwa/ CMA ES
ABBO best competitor
HDBO
Designing
2/21
1'
2
Shiwa
PARAMULTIMODAL
Designing
112
1/27
3
6
DiagounICMA-ES
Ros Hansen
2008
Realworld
Designing
486
1/6
2
3
Shiwa Liu
2020
Mlconii
Designing
1/24
3
8
Cobyla
Ilconnipara
Designing
5/28
7
3
MingonalCMA-ES
YABBOB
Designing
630
1/8
2
5
Shiwa
YAPARABBOB
Designing
630
1/6
5
MetaModel
YAHDBBOB
Designing
378
2/19
18
T)-ES
YANOISYBBOB
Designing
630
2/11
10
SQP
YAHDNOISYBROB
Designing"
630
4/24
13
SQP
YASMALLBBOB
Designing
Shiwa
HdMultimodal
Validation
42
1/14
2
4
Shiwa
Noisy
Validation
96
16/28
191
NA
Nccombbing@yiinsttoooooyyysssssd
RankNoisy
Validation
72
4/25
NA
19
PmpD13
AlIDEs
Validation
60
1/28
2
3
Shiwa Liu et al
2020
Pyomo
Evaluating
104
1/19
10
Shiwa Liu al.
2020
Rocket
Evaluating
5/18
DiagonaCCMA-Ls [ox Hansen
2008
SimpleeTSP
Evalunting
7
PontfolioDiscreee111
Scq. Fastgames
Evaluating
3/28
23
OptimisticDiserst((11
LSGO
Evaluating
1/6
MiniHHDDE + 1)-ES
Powersystems
Evaluating
48
10/6
NA
25
Table 2: Nevergrad maintains dashboard (Rapin Teytaud 2020). For each experiment, there are many configurations (budget, objective function, possibly dimension, and noise level). We sep- arate benchmarks used for designing ABBO, benchmarks used for validation, and those only used for testing. denotes benchmarks used for designing Shiwa (which is used inside ABBO). We present the rank based on the winning rate of ABBO in the dashboard. Since the submission of this paper, several variants of bandit-based algorithms have been added for nigh-ddmensiooal noisy opti- mization. They outperform ABBO, hence its poor rank for these cases. Detailed plots are available in Appendix B As expected, DE variants are strong on LSGO and CMA-ES variants are strong for YABBOB. ABBO also performs well on YABBOB, which was used for designing its ancestor Shiwa (see Fig. D. For the MuJoCo testbed, details are available in Table Figure Our modifications in the codebase implies an improvement of Shiwa compared to the version published in (Liu et al. 2020 for example, our chaining implies that the (k- + 1)th code starts from the best point obtained by the htth algorithm, which significantly improves in particular the chaining CMA-ES+ or CMA-ES+SQP. Experiments with "t" in the ranking of Shiwa correspond to this improved version of Shiwa.
2009) and more intensive use of mathematical programming techniques for the last part of the op- timization run, i.e., the local convergence, thanks Meta-Models (in the parallel case) and more time spent in Powell's method (in the sequential case). This explains the improvement visible in Section 4.T (2) Better performance in discrete optimization, using additional codes recently intro- duced in Nevergrad, in particular adaptive step-sizes. (3) Better segmentation of the different cases of continuous optimization. We still entirely rely on the base algorithms as available in Nevergrad; that is, we did not modify the tuning of any method. We acknowledge that our method only work thanks to the solid base components available in Nevergrad, which are based on contributions from various research teams. The obtained algorithm selection wizard, ABBO, is presented in Algo- rithm I The performances of ABBO is summarized in Table and a detailed dashboard is available at athttps: al. wwwippblicfiiiic.cc..
com/ www..///eeeeaaaaaaaa....ee allppp mwlxxxpssssst.i html
EXPERIMENTAL RESULTS
When presenting results on single benchmark function, we present the usual average objective function value for different budget values. When collection comprises multiple benchmark prob- lems, we present our aggregated experimental results with two distinct types of plots: (1) Average normalized objective value for each budget, averaged over all problems. The normalized objective value is the objective value linearly rescaled to [O, 1]. (2) Heatmaps, showing for each pair (x, y) of optimization algorithms the frequency at which Algorithm x outperforms Algorithm y. Algorithms ranked by average winning frequency. We use red arrows to highlight ABBO.
5

---

Under review as a conference paper at ICLR 2021
bmpdxjzzin
100
2x
22
My amcts 20282)
10"1
Hopeteed 0867)
Stlas 18:10
107
107
Avg. norm. loss
Heatmap
Figure 1: Average normalized loss and heatmap for YABBOB. Additional plots for High- dimensional (HD), NoisyHD, and Large budgets are available in Appendix B Other variants in- clude parallel, differences of budgets, and combinations of those variants, with excellent results for ABBO (see Nevergrad's dashboard https:: //dl. fba fbaipubll wellice of iles. com/ nevergrad/ allxps/ 1ist. .tmll publicly visible with anonymized entries, for the part of our code which is already merged).
DO MOATTS
TEL
(8722)
SPSOENTT (5-931)
(0.393)
R12 1820
Pyomo
Seq. Fastgames
Figure 2: Additional problems: Pyomo (covering Knapsack, P-median and others), Sequential- Fastgames (presented as heatmaps due to the high noise: GuessWho, War, Batawaf, Flip). Rockets, SimpleTSP. PowerSystems, and LSGO plots available in Appendix Figures and Pyomo and SimpleTSPP include discrete variables. Pyomo includes constraints. Rocket, PowerSystems, Se- quentialFastGames are based on open source simulators and are already merged from OptimSuite Nevergrad.
4.1 BENCHMARKS IN OPTIMSUITE USED FOR DESIGNING AND VALIDATING ABBO
YABBOB (Yet Another Black-Box Optimization Benchmark (Rapin al. 2019)) ) an adapta- tion of BBOB (Hansen et al. 2009), with extensions such as parallelism and noise management. It contains many variants, including noise, parallelism, high-dimension (BBOB was limited to dimen- sion < 50). Several extensions, for the high- -igeeniiensional, the parallel or the big budget case, have been developed: we present results Figures I and The high-dimensional one inspired by (Li et al. 2013). the noisy one is related to the noisy counterpart of BBOB but correctly implements the difference between ask and recommend as discussed Section 2 The parallel one generalizes YABBOB to settings in which several evaluations can be executed in parallel. Results PARA- MULTIMODAL are presented in Figure 6 (left). In addition, ABBO was run on ILLCONDI & ILLCONDIPARA (ill conditionned functions), HDMULTIMODAL (a multimodal case focusing on high-dimensioon, NOISY & RANKNOISY (two noisy continuous testbeds). YAWIDEBBOB (a broad range of functions including discrete cases and cases with constraints).

---

Under review as a conference paper at ICLR 2021
Task
Target
LA-MCTS resuits
ABBO result
LA-MC IS reward ABBO avg reward
Swimmer-v2
325
132 you
around
358
365
Tupper-v2 22
3120
2897 ios
around 3000 iler
3292
1787
Hailt Cheetah-22
3430
3877 loa
around 2000 iter
3227
4730
Walker2d v2x
4390
BR: 3114 (not reached) BR: 4398, hudget <, 64000 (reached!)
2709
2949
Ant-22
3580
BR: 2791 (noi reached) BR: 532S, budget 32000 (reached!)
2511
3532
-2mmannid-v2*
6000
BR: 3384 (not reached)
BR (hudget 500000): 4870
2511
4620
Table 3: Results for a linear policy in the black-box setting from the latest black-box paper (Wang ct al. 2020) and references therein, compared to results from ABBO. Two last columns = average reward for the maximum budget tested in (Wang et al., 2020), namely Ik, 4k, 4k, 40k, 30k, 40k, respectively. "ioa" iterations on average for reaching the target. "iter" iterations for target reached for median run. "*" refers to problems for which the target was not reached by Waagg et al. (2020): then BR means "best result in 10 runs". ABBO reaches the target for Humanoid and Ant whereas previous (black-box) papers did not; we get nearly the same ioa for Hopper and HalfCheetah (Nevergrad computed the expected value instead of computing the ioa, we cannot compare exactly; see Figure 33 for curves). ABBO is slower than LA-MCTS on Swimmer. Note that we keep the same method for all benchmarks whereas LA-MCTS modified the algorithm for 3 rows. On HDMULTIMODAL, ABBO performs better than LA-MCTS, as detailed in the text, and confirmed in (Wang et al. 2020), which acknowledges the poor results of LA-MCTS for high-dimensional Ackley and Rosenbrock.
AIIDEs and Hdbo are benchmark collections specifically designed to compare DE variants (AIIDEs) and high-dimensional Bayesian Optimization (Hdbo), respectively (Rapin & Teytaud, 2018). These benchmark functions are similar to the ones used in YABBOB, Many variants of DE (resp. BO) are considered. Results presented in Figure S They show that the performance of ABBO, relatively to DE or BO, is consistent over a wide range of parametrizations of DE or BO, at least in their most classical variants. All these variants are publicly visible in Nevergrad and/or in our anonymized branch.
Realworld: A test of ABBO is performed on the Realworld optimization benchmark suite proposed in (Rapin & Teytaud, 2018). This suite includes testbeds from MIDAA (Gallagher & Saleem 2018) and from (Liu et al.) 2020j. Results for this suite, presented in Figure 6 confirm that ABBO per- forms well also on benchmarks that were not explicitly used for its design however, this benchmark was used for designing Shiwa, which was the basis of our ABBO, A rigorous cross-validation, on benchmarks totally independent from the design of Shiwa, is provided in the next sections.
4.2 NEW BENCHMARKS IN OPTIMSUITE USED ONLY FOR EVALUATING ABBO
Pyomo is modeling language in Python for optimization problems (Hart et al. 2017). It is popular and has been adopted in formulating large models for complex and real-world systems, including energy systems and network resource systems. We implemented an interface to Pyomo for Never- grad and enriched our benchmark problems (Anonymous) 2020), which include discrete variables and constraints. Experimental results shown in Figure 2 They show that ABBO also performs decently in discrete settings and in constrained cases.
Additional new artificial and real-world functions: LSGO (large scale global optimization) com- bines various functions into an aggregated difficult testbed including composite highly multimodal functions. Correctly decomposing the problem is essential. Various implementations of LSGO exist; in particular we believe that some of them do not match exactly. Our implementation fol- lows (Li et al., 2013). which introduces functions with subcomponents (i.e., groups of decision variables) having non-uniform sizes and non-uniform, even conflicting, contributions to the objec- tive function. Furthermore, we present here experimental results on SequentialFastgames from the Nevergrad benchmarks, and three newly introduced benchmarks, namely Rocket, SimpleTST (a set of traveling salesman problems), power systems (unit commitment problems (Padhy) 2004)). Ex- perimental results are presented in Figures 2 and 8 They show that ABBO performs well on new benchmarks, never used for its design nor for that of the low-level heuristics used inside ABBO.
MuJoCo. Many articles (Sener & Koltun 2020 Wang et al. 2020 studied the MuJoCo testbeds Todorov et (012) in the black-box setting. MuJoCo tasks correspond to control prob- lems. Defined in (Wang et al. 2020: Mania et al. 2018), the objective is to learn linear mapping
7

---

Under review as a conference paper at ICLR 2021
dimasieni.. Nandttn das Seinmer arrenttration s
(113)
-15
TX
-20
Met 1228
Kunder 22501
ADSO Cl
Dusprnetii 11, Mi
Cldb 2.: 1311
Sliva 40400
3>
Swimmer (dim 16)
Hopper (dim 33)
Sincriien
20 wwwercodtoooooooo da Penee
-500
22
sudpet
Eudpet
Haaf-Chhetaah (dim 102)
Walker-2d 2d (dim 102)
parametr Lori ond ooy
4250
5
-2.0
30
30'
30
Ant (dim 888)
Humanoid (dim 6392)
Figure 3: Results on the MuJoCo testbeds. Dashed lines show the standard deviation. Compared to
the state of the art in (Wang al. 2020). with an algorithm adapted manually for the different tasks, we get overall better results for Humanoid, Ant, Walker. We get worse results for Swimmer (could match if we had modified our code for the casier tasks as done in (Wang et al,, (0200). similar for Hopper and Cheetah: we reach the target for 5 of the 6 problems (see text). Runs of Shiwa correspond to the improvement of Shiwa due chaining, as explained in Table?) 2
from states to actions. It turned out that the scaling is critical (Mania et al. 2018): for reasons men- tioned in Section solutions are close to O. We chose to scale all the variables of the problem at the power of 0.1 the closest to 1.2/d, for all methods run in Figure 3 We remark that ABBO and Shiwa perform well, including comparatively to gradient-based methods in some cases, while having the ability work when the gradient is not available. When the gradient is available, black-box methods do not require computation of the gradient, which saves time.
We use the same experimental setup as Wang et al. (2020) (linear policy, offline whitening of states). We get results better than LA-MCTS, in setting i.e., does not use any expensive surrogate model (Table 3. Our runs with CMA-ES and Shiwa are better than those in (Wang et al.) 2020). We acknowledge that LMRS (Sener & Koltun 2020) outperforms our method on all MuloCo tasks, using a deep network as a surrogate model: however, we point out that a part of their code is not open sourced, making the experiments not reproducible. In addition, when rerunning their repository without the non open sourced part, it solved Half- Cheetah within budget 56k, which is larger than ours. For Humanoid, the target was reached at 768k, which is again larger than budget. Results from ABBO are comparable to, and usually better than (for the hardest problems), results from LA-MCTS, while ABBO is entirely reproducible. In addition, it runs the same method for all
8

---

Under review as a conference paper at ICLR 2021
benchmarks and it is not optimized for each task specifically as in (Sener & Koltun 2020 Wang et al., 2020). In contrast to ABBO, (Wang et al. (2220) uses different underlying regression methods and sampling methods depending on the MuJoCo task, and it is not for some of the HDMULTIMODAL. ones. On the latter, ABBO performances are significantly better for Ackley and Rosenbrock in dimension 100 (expected results around 100 and 10-* after 10k iterations for Rosenbrock and Ackley respectively for ABBO, vs 0.5 and 500 in (Wang et al., 2020 From the curves in (Wang et al. 2020) and in the present work, we expect LA-MCTS to perform well with an adapted choice of parametrization and with low budget, for tasks related to MuloCo, whereas ABBO is adapted for wide ranges of tasks and budgets.
other benchmarks except
CONCLUSIONS
This paper proposes OptimSuite, very broad benchmark suite composed of real-world and of arti- ficial benchmark problems. OptimSuite is implemented as fork of Nevergrad (Rapin & Teytaud, 2018), from which it inherits strong reproducibility: our (Python) code open source (Anony- mous 2020), tests are rerun periodically, and up-to-date results are available in the public dash- board Rapin & Teytaud 2020). whole experiment can be done as one-liner. OptimSuite fixes several issues of existing benchmarking environments. The suite subsumes MuJoCo, Pyomo, LSGO, YABBOB, MLDA, and several new real-world problems. We also propose ABBO, an improved al- gorithm selection wizard. Despite its simplicity, ABBO shows very promising performance across the whole benchmark suite, often outperforming the previous state-of-the-arr, problem-speciffi solvers: (a) by solving of the 6 cases without any task-specific hyperparameter tuning, ABBO outperforms LA-MCTS, which was specialized for each single task, (b) ABBO outperforms Shiwa on YABBOB and its variants, which the benchmark suite used design Shiwa in the first place, (c) ABBO is also among the best methods on LSGO and almost all other benchmarks.
Further work. OptimSuite subsumes most of the desirable features outlined in Section 2 with one notable exception, the true black-box setting, which other benchmark environments have im- plemented through client-server interaction (Loshchilov & Glasmachers) 2017). A possible com- bination between our platform and such challenge, using the dashboard to publish the results, could be useful, to offer meaningful way for cross-validation. Further improving ABBO is on the roadmap. In particular, lection rules. Note, though, that important to us to maintain a high level of interpretability. which consider key for wide acceptance of the wizard. Another avenue for future work is proper configuration of the low-level heuristics subsumed by ABBO. At present, some of them are merely textbook implementations, and significant room for improvement can therefore be expected. Newer variants (Loshchilov) 2014 Akimmto & Hansen 2016 Loshchilov et al., 2018) of CMA- ES, of LMRS (Sener & Koltun 2020), recent Bayesian optimization libraries (e.g. Briksson al. (019 ), as well as per-instance a algorithm configuration such as Belkhir et al. (2017) are not unlikely to result in important improvements for various benchmarks. We also plan on extending OptimSuite further, both through interfacing existing benchmark collections/problems, and by designing new benchmark problems ourselves.
experimenting with the automation of the still hand-crafted
ACKNOWLEDGMENTS
We thank the anonymous reviewers for valuable suggestions that helped us improve the clarity and the presentation of our work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Youhei Akimoto and Nikolaus Hansen. Projection-basee baseed restricted covariance matrix adaptation for high dimension. In Proc. of Genetic and Evolutionary Computatiion (GECCO'16) pp. 197-204, ACM, 2016.
Anonymous.
Anonymized github repository with all source code of our work. https://
copen. htt.://.............e.ooe./eee20 2020.
SME Artelys. Artelys sqp wins the bbcomp competition, 2013. URL https:/t//// www. ini ini.rub
de/PEOPLE/qLaae s https//sshhooopp/oeeee///e inneee/ppiidx...eee html
Anne Auger and Nikolaus Hansen. Benchmarking the (1+1)-CMA-ES on the BBOB-2009 noisy testbed. In Proc. of Genetic and Evolutionary Computation Conference (GECCO'08 00,, Companion Material, pp. 1467-2472. ACM, 2009.
Anne Auger, Marc Schoenauer, and Olivier Teytaud. Local and global order 3/2 convergence of surrogate evolutionary algorithm. In Proc. of Genetic and Evolutionary Computation Conference (GECCO 005), pp. 857-864 864. ACM, 2005.
Nicolas Baskiotis and Michèle Scbag. C4.5 competence map: a phase transition-inspiin approach. In Proc. of International Conference on Machine Learning (ICMI. 004), 2004.
Nacim Belkhir, Johann Dréo, Pierre Savéant, and Mare Schoenauer. Per instance algorithm con- figuration of ema-es with limited budget. In Proc. of Genetic and Evolutionary Computation Conference (GECCO') 17), pp. 681-688. ACM, 2017.
Vincent Berthier. Progressive differential evolution on clustering real world problems. In Revised Selected Papers of the 12th International Conference on Artificial Evolution Volume 9554, pp- 71-82. Springer, 2016.
Hans-Georg Beyer. The Theory of Evolution Strategies. Natural Computing Series. Springer, Hei- deberg, 2001.
Hans-Georg Beyer.
2012 Apri1/000270 html
2012a.
Hans-Georg
Beyer.
http://lee...i..rr//jjrraabbbbbb
2012-Aprill April/000258 html
2012b.
Hans-Georg Beyer and Hans-Paul Schwefel. Evolution Strategies A Comprehensive Introduction. Natural Computing, 1(1):3-52, May 2002.
Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Thomas Marius Lindauer, Yuri Malitsky, Alexandre Fréchette, Holger H. Hoos, Frank Hutter, Kevin Meyton-Brown Kevin Tiemeyy and Joaquin Van- schoren. ASlib: A Benchmark Library for Algorithm Selection. Artificial Intelligence (AlJ), 237: 58, 2016.
Olivier Bousquet, Sylvain Gelly, Kurach Karol, Olivier Teytaud, and Damien Vincent. Critical hyper-parameters: No random, cry. CoRR, abs/1706.00200) 2017.
Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in finitely-armed and continuous-and armed bandits. Theor. Comput. Sci., 412((1):1832-1111 1852, 2011.
Marie-Liesse Cauwet and Olivier Teytaud. Population control meets Doob's martingale theorems: The noise-free multimodal case. In Proc. of Genetic and Evolutionary Computation (GECCO Companion Material), GECCO '20, pp. 321-322. ACM, 2020.
Marie-Liesse Cauwet, Jialin Liu, Baptiste Rozière, and Olivier Teytaud. Algorithm portfolios for noisy optimization. Annals of Mathematics and Artificial Intelligence, 76( 76(1-2):143-172, 2016.
Liesse Cauwet, Camille Couprie, Julien Dehos, Pauline Luc, Jérémy Rapin, Morgane Riviere, Fabien Teytaud, and Olivier Teytaud. Fully parallel hyperparameter search: Reshaped space- filling. arXiv preprint arXiv: 19/0.88406. To appear in Proc. of ICML. 2020, 2019.
10

---

Under review as a conference paper at ICLR 2021
Alexandre Adrien Chotard, Anne Auger, and Nikolaus Hansen. Cumulative Step-size Adaptation on Linear Functions: Technical Report. Research report, Inria Saclay, June 2012.
Yann Collette, Nikolaus Hansen, Gilles Pujol, Daniel Salazar, and Rodolphe Le Riche. On object-
oriented programming of optimizers examples in scilab. 01 2010.
Remi Coulom. http://list: 1ri. http:///......./aaa/aaaaaaaaaaaa 000252. html, 2012a.
Rémi Coulom. Clop:
Confident local optimization for noisyblack-box parameter tuning. In Ad- vances in Computer Games, pp. 146-157. Springer Berlin Heidelberg, 2012b.
Duc-Cuong Dang and Per Kristian Lehre. Self-adaptation of mutation rates in non-elitist popula- tions. In Proc. of Purallel Problem Solving from Nature (PPSN'16) pp. 803-813, 2016.
Jérémie Decock and Olivier Teytaud. Noisy optimization complexity under locality assumption. In Proceedings of the Twelfth Workshop on Foundations of Genetic Algorithms XII, FOGA XII '13, pp. 183-190. ACM, 2013.
Benjamin Doerr, Huu Phuoe Le, Régis Makhmara, and Duy Nguyen. Fast genetic algorithms. In Proc. of Genetic and Evolutionary Computation (GECCO 17), pp. 777-784. ACM, 2017.
Benjamin Doerr, Carola Doerr, and Johannes Lengler. Self-adjusting mutation rates with provably optimal success rules. Proc. of Genetic and Evolutionary Computation (GECCO' 19), pp- 1479-1487. ACM, 2019.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. imer, F. d Alché-Bus E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 5496-5507. Curran Associates, 2019.
Wallach, H. Larochelle, A. Beygelz-
Matteo Fischetti and Michele Monaci. Exploiting erraticism in search. Operations Research, 62(1): 114-122, 2014.
Marcus Gallagher and Sobia Saleem. Exploratory landscape analysis of the mlda problem set. In PPSN'18 workshop. 2018.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 11(1), 2003.
Nikolaus Hansen, Anne Auger, Steffen Finck, and Raymond Ros. Real-parameter black-box opti- mization benchmarking 2009: Experimental setup. Technical Report RR-6828, INRIA, France, 2009.
Nikolaus Hansen, Raymond Ros, Nikolas Mauny, Marc Schoenauer. and Anne Auger. Impacts of Invariance in Search: When CMA-ES and PSO Face Ill-onnitioned and Non-Separable Prob- lems. Applied Soft Computing, 11:5755-5769, 2011.
William E Hart, Carl D Laird, Jean-Pan Watson, David L Woodruff, Gabriel A Hackebeil, Bethany L Nicholson, and John D Siirola. Pyomo-optimization modeling in python, volume 67. Springer, 2017.
Verena Heidrich-Meisner and Christian Igel. Hoeffding and bernstein races for selecting policies in evolutionary direct policy search. In Proc. of International Conference on Machine Learning (ICML/09), pp. 401-408. ACM, 2009.
Michael Hellwig and Hans-Georg Beyer. Evolution under strong noise: A self-adaptive evolution strategy can reach the lower performance bound the PeCMSA-ES. In Proc. of Parallel Problem Solving from Nature (PPSN' (P)SN'16, pp. 26-36. Springer, 2016.
John H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, 1975.
Max Jaderberg, Valentin Dalibard, Simon Osindero. Wojciech M. Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks, 2017.

---

Under review as a conference paper at ICLR 2021
Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expen- sive black-box functions. Journal of Global Optinization, 13(4):455 492, Dec 1998.
Pascal Kerschke and Heike Trautmann. Automated Algorithm Selection Continuous Black-Box. Problems By Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation, pp. 28, 2018.
Pascal Kerschke, Holger H. Hoos, Frank Neumann, and Heike Trautmann. Automated Algorithm Selection: Survey and Perspectives. Evolutionary Computation, pp. 1-7,, 2018.
Lars Kotthoff. Algorithm Selection for Combinatorial Search Problems: A Survey, Al Magazine, 35(3):48 60, 2014.
Xiaodong Li, Ke Tang, Mohammmad Nabi Omidvar, Zhenyu Yang, and Kai Qin. Benchmark func- tions for the CEC'2013 special session and competition on large-scale global optimization. 01 2013.
Jialin Liu, Antoine Moreau, Mike Preuss, Jeremy Rapin, Baptiste Roziere, Fabien Teytaud, and Olivier Teytaud. Versatile black-box optimization. In Proc. of Genetic and Evolutionary Compu- tation (GECCO'20), pp. 620-628, 2020.
lyya Loshchilov. A computationally efficient limited memory cma-es for large scale optimization. In Proc. of Genetic and Evolutionary Computation (GECCO' 14), pp. 397-404, 2014.
Uya Loshchilov and T. Glasmachers. Black box optimization competition, 2017. URL https: html
7/ www. ini. rub. http///////aaattt///////ee//aaop
Ilya Loshchilov, Tobias Glasmachers, and Hans-Georg Beyer. Large scale black-box optimization by mimited-memory matrix adaptation. IEEE Transactions on Evolutionary Computation, 23(2):
353-358, 2018.
Katherine Mary Malan and Andries Petrus Engelbrecht. Survey of Techniques for Characterising Fitness Landscapes and Some Possible Ways Forward. Information Sciences (JIS), 241:148 163, 2013.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides competitive approach to reinforcement learning, 2018.
Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. Exploratory Landscape Analysis. In Proc. of Genetic and Evolutionary Computation Conferennce (GECCO'iI), pp. 829 836. ACM, 2011.
Laurent Meunier, Carola Doerr, Jérémy Rapin, and Olivier Teytaud. Variance reduction for better sampling continuous domains. Proc. of Parallel Problem Solving from Nature (PPSN'20), volume 12269 of LNCS, pp. 154-168. Springer, 2020.
D. Molina, M. Lozano, and F. Herrera. Memetic algorithm with local search chaining for continuous optimization problems: A scalability test. In 2009 Ninth International Conference on Intelligent Systems Design and Applicutions, pp. 1068-1073, 2009.
Mario Andrés Muñoz Acosta, Yuan Sun, Michael Kirley, and Saman K. Halgamuge. Algorithm Se- lection for Black-Box Continuous Optimization Problems: Survey on Methods and Challenges. Information Sciences (JIS), 317:224 245, October 2015.
John A. Nelder and Roger Mead. A simplex method for function minimization. Computer Journal, 7:308-30 -13, 1965.
Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014- URL. htt.:ttts:://gii htwp:h...ccoooeennaaaeeeeeeaaeee
zat in
Narayana Prasad Padhy. Unit commitment-a bibliographical survey. IEEE Transactions on Power Systems, 19(2): 196-1205, 2004.
12

---

Under review as a conference paper at ICLR 2021
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duch- esnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12: 2825-2830 2011.
Erik Pitzer and Michael Affenzeller. A Comprehensive Survey on Fitness Landscape Analysis. In János Fodor, Ryszard Klempous, and Carmen Paz Suárez Araujo (eds.). Recent Advances in Intelligent Engineering Systems, Studies in Computational Intelligence, pp. 161 191. Springer, 2012.
Michael J.D. Powell. An efficient method for finding the minimum of function of several variables without calculating derivatives. The Computer Journal, 7(2):155-162, 1964.
Michael J.D. Powell. A Direct Search Optimization Method That Models the Objective and Con- straint Functions by Linear Interpolation, pp. 51-67. Springer, 1994, ISBN 978-94-015--333-5.
Nicholas J. Radcliffe and Patrick D. Surry. Formal memetic algorithms. In T.C. Fogarty (ed.), Evolutionary Computing: AISB Workshop. pp. 1-16. Springer, LNCS 865, 1994.
Jeremy Rapin and Olivier Teytaud. Nevergrad A gradient-free optimization platform. https:
2018.
Jeremy Rapin and Olivier Teytaud. Dashboard of results for Nevergrad platform. https://dl.
Ebaipublicfiles wwt.eitwsss.oooo.o.oooooooooo.oe wwt.m//eeeerrrraalllppppp//////p
mtml 2020.
Jérémy Rapin, Pauline Dorval, Jules Pondard, Nicolas Vasilache, Marie-Liesse Cauwet, Camille Couprie, and Olivier Teytaud. Openly revisiting derivative optimization. In Proc. of Genetic and Evolutionary Computation (GECCO'19), pp. 267- ACM, 2019.
Ingo Rechenberg. Evolutionsstrategi:: Optimierung Technischer Systeme nach Prinzipien des Biol- ogischen Evolution. Fromman Frommon-Holzboog Verlag, 1973.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classi- fiers generalize CIFAR-10 ?, 2018.
John Rischard Rice. The Algorithm Selection Problem. Advances in Computers, 15:65 118, 1976.
Hartley Rogers. Theory of Recursive Functions and Effective Computability. MIT Press, 1987. ISBN 0262680521.
Raymond Ros and Nikolaus Hansen. simple modification in CMA-ES achieving linear time and space complexity. In Proc. of Parallel Problem Solving from Nature (PPSN 08), pp. 296-305. Springer, 2008
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016.
Ralf Salomon. Re-evaluating genetic algorithm performance under coordinate rotation of benchmark functions. a survey some theoretical and practical aspects of genetic algorithms. BioSystems, 39(3):263-277 1996.
Ozan Sener and Vladlen Koltun. Learning to guide random search. International Confer- Learning Representations, 2020. URL B1gHokBKWS
wttp:r//wwwww.eeeeeouuuuuuaa forumiid-d
net
Kate Amanda Smith-Miles. Cross-Disciplinary Perspectives on Meta-Leaming for Algorithm Se- lection. ACM Computing Surveys (CSUR), 41:1 - 25, January 2009.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NIPS), pp. 2951- 2959, 2012.
13

---

Under review as a conference paper at ICLR 2021
Rainer Storn and Kenneth Price. Differential evolution simple and efficient heuristic for global optimization over continuous spaces. J. of Global Optimization, 11(4):341-35, December 1997.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Mauro Vallati, Frank Hutter, Lukás Chrpa, and Thomas Leo McCluskey. On the effective con- figuration of planning domain models. Proc. of International Joint Conference on Artificial Intelligence (ICAA'15), 2015.
Konstantinos Varelas, Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Ouassim Ait ElHara, Yann Semet, Rami Kassab, and Frédéric Barbaresco. A comparative study of large-scale variants of CMA-ES. In Proc. of Parallel Problem Solving from Nature (PPSN' 18), volume 11101 of LNCS, pp. 3-15. Springer, 2018.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, Carey, ilhan Polat, Yu Feng, Eric Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. 0 Contributors. SciPy J0-Fundamental Algorithms for Scientific Computing in Python. arXiv e-prints, art. 1907. 10121, ,, Jul 2019.
Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimization using Monte Carlo Tree Search. arXiv:2007.00708. To appear in Proc. of NeurlPS 2020, 2020.
Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. SATzilla: Portfolio-based algo- rithm selection for SAT, J. Artif. Int. Res., 32(1):555-606, 2008
14

---

Under review as a conference paper at ICLR 2021
A
DETAILS ABOUT PROPERTIES OF BENCHMARKS
We specify the properties mentioned in Table
Large scale: includes dimension 2 1000.
Translations:
in unbounded continuous domains, standard deviation has be pro- vided, for example for sampling the first and second iterates of the optimization algorithm. Given standard deviation or we consider that there translation when optimas are ran- domly translated by a (O, o²) shift. Only interesting for artificial cases.
Far-optimum: optima are translated far from the optimum, with standard deviation at least (0, 25 x )²).
Symmetrizations/ / rotations (here assuming an optimum, up to translation, in O). Rotation: with a random rotation matrix M, the function x + f(x) is replaced by f(x) can be replaced by a diagonal matrix with each diagonal coefficient equal to 1 or -1 with probability 50%.
x + (x)( Symmetrization:
f(S(z)), with S
We do not request all benchmarks to be rotated: it might be preferable to have both cases considered.
One-line reproducibility: Where reproducibility requires significant coding, it is unlikely to be of great use outside of very small set of specialists. One-line reproducibility is given when the effort to reproduce an entire experiment does not require more than the execution of a single line. We consider this to be important feature.
Periodic automated dashboard: are algorithms re-run periodically on new problem in- stances? Some platforms do not collect the algorithms, and reproducibility hence not given. An automated dashboard is convenient also because new problems can be added "on the go" without causing problems, as all algorithms will be executed on all these new problem instances. This feature addresses what we consider to be one of the biggest bot- tlenecks the current benchmarking environments.
Complex or real-worid: Real-world is self-explanatory; complex means a benchmark involving a complex simulator, even if it is not real world. MuloCo is in the "complex" category.
Multimodal: whether the suite contains problems for which there are local optima which are not global optima.
Open sourced / no license: Are algorithms and benchmarks available under an open source agreement. BBOB does not collect algorithms, MuJoCo requires license, LSGO and BBOB are not realworld, Mujoco requires a license, BBComp is no longer maintained, Nevergrad before OptimSuite did not include complex ML problems without license issue before our work: some people have already applied Nevergrad to MuJoCo, but with our work MuJoCo becomes part of Nevergrad that people can upload their code in Nevergrad and it will be run on all benchmarks, including MuJoCo.
Ask/tell/recommend correctly implemented (Collette et al. 2010; Bubeck et al.) 201 The ask and tell idea (developped Collette et al. (2010) is that optimization al algorithm should not come under the format there many settings in which this is not possible: you might think of agents optimiz- ing concurrently their own part of an objective function, and problems of reentrance, or asynchronicity. All settings can be recovered from an ask/tell optimization method. This becomes widely used. However, as well known in the bandit literature (you can think of pure exploration bandits (Bubeck et al. 2011)). and recommend: the "recommend" method is the one which proposes an approximation of the optimum. Let us develop an example explaining why this matters: the domain is 2,, 3, 4}. and we have budget of 20 in noisy case. NoisyBBOB assumes that the optimum is found when "ask" returns the optimum arm: then, the status remains "found" even if the algorithm has no idea where is the optimum and never comes back nearby. So an algorithm which just iteratively "asks" 1,2, 2, reaches the optimum in at most 4 iterations. This does not mean anything in the noisy case, the challenge is to figure out which of the four numbers is the optimum. With a proper ask/tel/recommend, the optimizer chooses arm the end of the budget. A simple regret is then computed.
(objectiieeeeeeeeee - forioue-fnnnctiie function) because
necessary distinguish ask, tell
15

---

Under review as a conference paper at ICLR 2021
Actually this also matters in the noise-free case, but the issue is much critical in noisy optimization. The case continuous noisy optimization also has counter-exand cuumtes-xamples and all the best noisy optimization algorithms ask/tell/recommend We add the reference to the paper above.
Human excluded client-server: The problem instances are truly black-box. Algorithms only suggest points and observe function values, but neither the algorithm nor its de- signer have access to any other information about the problem apart from the number of variables, their type, ranges, and order. It is impossible hyperparameters without "paying" the budget of the HP tuning. This is something we could not do, as everything public and open sourced: however, we believe that we mitigate this issue by considering a large number of benchmarks.
repeat experiments for tuning
B ADDITIONAL FIGURES
ChePkssOne (72.3%)
ANSO (69 (0%)
80
Stika (916)
60
DiagonalcAa 16. 19%1
40
Maae (58 275)
20
MinDDE (56, EMS)
Onaifurcc (639)8330)
ABBO (530/330)
Shive (6300630)
DisgoralCMM (6300190 (6300930)
Wina huDF (e (630)630)
MINICE (530/630)
No Lolaantt (5306630)
CheinCM-PPeel 1630V930)
Spintpt Vni2ee (630VE50)
CMaandss: (5301530)
Spinopt (630)630)
Splecc t030/90001
Spi CONKS (ww.weeseeo......... (6:008330)
(sss (5309330)
conooo (630ve30)
DE (630V830))
Metehodee
High-dimensional (HD)
100
soo (78 1a8
Gevenn (ras sPA
75
50
Mestaanee (RR)
25
MonDE (77 8%)
390 ceseaae
swwo aeoooooo
mpsei (3008300
ABBO (030e300
Mintinsc (630/6 9o)
MiniDE (630/630)
(o3uu mn)
Prop (630/630)
boyDOpiiimeert (630/530)
Nglamess (630/630)
chainct/Pooel (630/830)
CMandASS (6201830)
CMA (629/830)
Metzifode (6301630)
SeltCMAatt (630/33))
Diagonatioal (G23/E30)
DE (625/033))
wowpesiinos (620/630)
spiluzieo (0201030)
splin serewneee (0271330)
sattccoo (0201200)
Prog)O0priminn
Noisy HD
Figure 4: YAHDBBOB (dimension > 50) and YANOISYHDBBOB (noisy + dimension 2 50) heatmaps.
16

---

Under review as a conference paper at ICLR 2021
ChainpEwithR (0.051)
CheinDEwit SeinDEiwiMMeaReeeeee Meteeeeeeeeeeteeegeeiimm (0.0503)
(o,0503)
1
ChainDEwithie EeinnEEiiiiieaatuue wtw.uneeeeeeeiinsstt......k
ChainDEEwttMMetaie
.
AlmossRattiinninnarrrt (p.0449)
10
..
ASCMADEMInG (0.035)
(0.034) MiniLhe (0.032)
10
OIDE 10.022 C307)
055
--
MinDD 10 0284)
apRotactoring woraaaornnnnanap (0.0277)
10
...
MiniOrDE (0 0.02351 0232)
ResMutDD (0.00 02733
ASCMA2PDER
(0.0155)
10
ESCMAGEORming (0.014)
(0.017)
www.oooiinnnnaameeeeeeeeeegee (0.0134)
- LMA to 54e 05 ClandASZ 37e-05)
10
ABOO (3.35e-05)
discreteDE (3.37e-07)
Sniwa (2.340-07)
10
10a
10a
10*
10a
budget
AlIDEs (d E (5, 20, 100).)
Chain8OwtthRoorTV Siiim (0_201)
Chain8Qwithh (0 73]
Chintttiieeeess serentering 225)
ChaiinBOwithhee Recenterinn (0.2177)
(0.215)
chaing thMtt Miaann Jumpeeeetmmeeeeeerrrrrn (0 208)
Chain8GW GiinBRiieeee
10
MidORBO wtw.eefauunReecmterinnnnnii
CMandAS2 149)
loss
10-2
ABBO (0.00551)
x
4
5 10
6 10
budget
HDBO (d E {20, 2000))
Figure 5: ABBO vs specific families of optimization algorithms (DE, and BO in the high- dimensional case) on Cigar, Hm, Ellipsoid, Sphere functions. Not all run algorithms are mentioned, for short. Bayesian optimization (Nevergrad uses Nogueira (2014))), often exploring boundaries first, is outperformed in high dimension (Wang et al. 2020).
17

---

Under review as a conference paper at ICLR 2021
MultSccleCMA (0.181)
MultiCMA (0.169)
CMangASZ (0.164)
CMandAS (.15
Partfolio 0.10))
ASCMA2PDEthie (0.108)
10-1
ASCMADEthird (0.099)
Notationinvarlantin (0.101)
LnsDE 0900
DE (0.0000) OnePointDE (0.0898)
OrDE 08.0877)
TwoPointsDE (0.0843)
CMA (0.0643)
Shiwa (0.0637)
MiniDE 0,0588)
10-2
MiniOrDE CM 00,0539
NaiveTBPSA (0.0435)
DiagonaICMA (0.002)
10-2
ABBO (0. (0.000335)
budget
PARAMULTIMODAL
6x
Meta|TuneRecentering (0.469)
NoLamcts (0.412)
40-1
PortoricDiscr leD.screeeeeecco. PecrrreoneellsOneee (0.392)
NaIVe (0.344)
l005
3x
DE [0,271]
OneelugQOs (6) (0.58) (MandASZ (0. 247)
TwoPointsDE (0.234)
DisgonalCMA (0.214)
CMA (0.212)
2x10-
(0.162)
Shiwa (0.169)
107
10a
10
budget
Realworld
Figure 6: Up: experiments for the parallel multimodal setting PARAMULTIMODAL. Budget up to Decpeeeeeeeeeitiiimaa----iii
100000, parallelism 1000, Bottom: Realworld benchmark from Nevergrad: games, Sammon mappings, clustering, small trav- eling salesman instance, small power systems.
18

---

Under review as a conference paper at ICLR 2021
Zero (0.914)
(0.829)
Meta TueeecuneReennn (0.697)
PSQ (0,64))
Scrlammerriie (0,557)
6 10
18P4A
Rap2om:5eRrch (0.531)
(0.531)
PortfolioDisc ContolliDDscrreeeOree
www.oiieeeeeeeeeeeuSneeeeee (0.463)
MiniDE 10 4151
NoIvE (0.41)
397 CMandASZ 357;
4x100
10 320)
ABBO
OYDE (0.356)
NgLamets (0. (4.34)
3x
Shiwa (0, (393
Diagonal (0. 3363))
OnePlusOne (0.286)
2 10
Powell (0.142)
10
10a
budget
Rocket
TBPSA (D.659)
6x 10
DE (0.565)
Onellusone (0.519)
SeftaiTnneecceeeenn (0.511)
TBPSA (0.509)
TwoPointsDE (0.433)
4x
P5O (0. 363)
(6.33)
CMandAS2 (C.355)
Shiwa (0.329)
10
ABBO (0 (0,212)
2 10
DiagonelÇMA
fottollooDscreeenneeeeeee (0.202)
10a
10a
10
budget
SimpleTSP
Figure 7: Additional problems (1): Rocket (26 continuous variables, budget up to 1600, sequential or parallelism 30) and SimpleTSPP (10 to 1000 decision variables).
19

---

Under review as a conference paper at ICLR 2021
10
SpitC (O (7.666))
PIoSONOPOS (0.074)
MInIDE (0.0737) Halto haltonsearch 0733).
ProSONOPOS (0.0678)
CMA 10 (0.0666)
096
LISDE 10 0.0667
SpIItCMA (0.0535)
0.0031
ProcODOPOS (0,0608)
SpiItCMAS (0,0503)
Prog Detpattiiiiieeeeeeeeleeee
Optimistic
(0.0589)
SDIRCMA13 (0.0568)
Optimistic Zero BetlccoiisOOnelssteeooo (0.054)
OnePluse usOne (.0401)
AB8O (0.0323)
ProgODOPOInf
ProgONOPO
10
10a
10a
10
10
budget
PowerSystems
MiniLhsDE (0.325)
DE (0.319)
loss
CMA (0.136)
chainMetaModdeSSPP (0.126)
10
ShiwaZ (0.0565)
10c
budget
Lsgo (15 functions)
Figure 8: Additional problems (2): PowerSystems (1806 to 9646 neural decision variables) and LSGO (mix of partially separable, overlapping, shifted cases as in Li et al. (2013)
20


Reviews:
[{"review_id": "mp8Dm6QIXJm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a benchmark suite for black-box optimization that covers more\ndifferent types of problems than existing benchmarks. They derive an algorithm\nselection system for black-box optimization from it and evaluate its performance\nempirically, comparing to other black-box optimization solvers.\n\nThe authors address an interesting problem and demonstrate some good empirical\nresults. It is certainly beneficial to have large benchmark suites to get a\ncomprehensive picture of the performance of different approaches.\n\nWhile the general motivation for a new benchmark suite is clear, the specific\nproperties the authors list are not. Some of the properties in Table 1 seem to\nbe complementary (far-optimum and translation), while it is unclear why others\nare important. It is certainly nice to have an automated dashboard (of what\nexactly?) and one-line reproducibility, but within the general context of the\npaper, which seems to focus more on generalizability of results, this seems\nunimportant. Whether a particular set of benchmarks is complex is to some extent\na question of the definition of \"complex\", which the authors do not make clear.\nI do not know what \"human-excluded/client-server\" means. While the majority of\nproperties listed in Table 1 are obviously important for black-box optimization,\nit seems that the authors started from the properties that their benchmark suite\nhas.\n\nThe benchmarks themselves seem to be mostly existing benchmarks that were\ncombined into a new collection -- it would help if the authors pointed out what\nbenchmarks were specifically created for OptimSuite.\n\nIt is unclear how exactly ABBO was created -- was the set of rules determined\nover all benchmark instances to maximize a performance metric? ABBO is listed as\none of the main contributions of the paper and it should be explained in more\ndetail how it was created and how it works.\n\nThe labels in the figures are too small.\n\nUpdate after rebuttal period: Thank you for your clarifications. I have revised my score accordingly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lDUQL2DGyEc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n** Summary **\n\nThis paper proposes a new benchmark suite for black box optimization algorithms, which contains a mixture of existing tasks such as nevergrad functions and RL tasks from MuJoCo. In addition, the authors propose a new automated approach to algorithm selection (ABBO), which uses a series of rules to select the best method.\n\n** Primary Reason for Score**\n\nThe benchmark suite is predominantly an amalgamation of existing tasks, and only appears to be an incremental improvement vs. Nevergrad. The proposed meta-algorithm is built on a set of “hand-crafted selection rules” (in the authors’ own words). This does not seem like a meaningful contribution to the ICLR community, but may be interesting in other, more applied venues. \n\n** Strengths **\n\n1) The proposed OptimSuite may be more convenient for users.\n2) The rules-based system works surprisingly well, which begs the question of whether a learned version could perform better, which may be of interest to the ICLR community.  \n\nPOST REBUTTAL:\n3) Including all benchmarks into one codebase could improve reproducibility.\n\n** Weaknesses **\n\n1) I don’t think ICLR is the right venue for this work, which seems more engineering focused and may be more suited to an applied venue.\n2) Most of the benchmarks in this suite are all included elsewhere already. Many recent papers proposing blackbox optimization algorithms include several of them. In particular, using Nevergrad + MuJoCo seems to achieve most of the desirable properties. \n3) The selection wizard is just a set of heuristics/hand-engineered rules. This might be more useful for industrial applications, rather than ICLR. \n4) Given that this paper is about benchmarking optimization algorithms, the presentation of the results is poor. It is almost impossible to read the plots, and there is no central table/comparison of the different methods. Reading this I learn very about the algorithms that are being benchmarked.\n5) I have several issues with the presentation of the RL results:\na) ARS is not even a blackbox optimization algorithm, it uses information about the MDP for state and reward normalization (v2-t). \nb) The “SOTA” results have been copy and pasted from another paper, which pasted them from the ARS paper, which referenced 2017 results. These are nowhere near “SOTA with grad” in RL. In fact, ARS didn’t say they were SOTA, it simply said that was what TRPO got as a baseline. SOTA now is probably MBPO or SAC/TD3, all of which solve these tasks in a fraction of the time.\nc) Missing relevant literature: there have been several approaches to blackbox optimization for RL (and other functions) since ARS. I included ones presented either at ICLR or similar venues, which should be discussed:\n  i) **Gradientless Descent: High-Dimensional Zeroth-Order Optimization**. Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, Qiuyi (Richard) Zhang. *ICLR 2020*.\n  ii)  **Learning to Guide Random Search**. Ozan Sener, Vladlen Koltun. *ICLR 2020*.\n  iii)  **From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization**. Krzysztof M. Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Vikas Sindhwani. *NeurIPS 2019*.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cNsoXs6ygv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a benchmarking suite to overcome the problem low generalizability with black box optimization algorithm. The benchmarking suite consists of standard academic benchmarks to real world optimization problems. It also covers several scenarios such as dynamic-static, small to large-scale, discrete to mixed-integer etc. This is a relevant contribution to the machine learning however there are several drawbacks which pushes back it's acceptance into ICLR.\n\n(a) The state of the art discussion was good but why benchmarking is crucial and how it is implemented in Optimsuite was very short in description. That was supposed to be the main highlight of the paper whereas discussion in that part was not clear at all. Specifically, more descriptions were needed in terms of what features to include/exclude in the benchmarking suite and how that helps in generalizability.\n\n(b) I know author(s) mentioned about interpretability as future work however I felt really challenging to understand the benchmarking suite especially when you have a combination of academic benchmarks and real world optimization problems together. I think that's a very significant challenge in implementing ABBO.\n\n(c) One thing was not clear to me how the tuning parameters that are often associated with several optimization problems are handled here? are you keeping the tuning parameter same across all the competing methods?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8g6h-yQIChy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n\nThe contribution of this work is two-fold: it collects an extremely wide range of benchmark problems for black-box optimization, and it proposes a new algorithm called ABBO. Both contributions are significant.\n\nThe proposed ABBO method is designed to be a swiss army knife solver, suitable for a wide range of different types of problems. To this end, naturally, it builds on existing components. Overall, the combination looks extremely convincing to me.\n\n\nCRITICISM\n\nAlready the first sentence of the abstract is problematic:\n\"Existing studies in black-box optimization suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing different optimization algorithms.\"\nThis statement is very general. However, black-box optimization is an extremely wide area, ranging (at least) from mathematical optimization over evolutionary computation all the way to machine learning. The statement applied to various degrees to most studies in various subfields. I completely agree with the statement only when restricted to machine learning papers, where experiments are typically limited to very few RL benchmarks. In other areas things are far from perfect, but generally much better (it is understood that avoiding a bias completely is near impossible), since benchmarks e.g. with (YA)BBOB aim at general insights, not (only) at demonstrating peak performance. Please qualify this statement accordingly.\n\nThere is one more problematic statement in the abstract:\n\"A single algorithm therefore performed best on these three important benchmarks, without any task-specific parametrization.\"\nWell, this \"single algorithm\" is really an algorithm selection machine. Technically it is \"a single algorithm\", but it is much more useful to think of ABBO as a selection and configuration method. In my understanding this makes a big difference. For optimal performance we need both: powerful components and powerful configurators. Please make absolutely clear that this \"single algorithm\" really is a configurator, which encompasses multiple components.\n\nPersonally I object some of the methods forming the basis of \"Algorithm 1\", although overall the choices look very solid. [Side note: I very much like the use of Powell's algorithms for fine tuning of approximate solutions found with more robust methods.] I have one suggestion: Diagonal CMA-ES is an outdated method. Please consider low-rank approaches as an alternative, like LM-CMA-ES, VD-CMA-ES and LM-MA-ES.\n@inproceedings{loshchilov2014computationally,\n  title={A computationally efficient limited memory CMA-ES for large scale optimization},\n  author={Loshchilov, Ilya},\n  booktitle={Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation},\n  pages={397--404},\n  year={2014}\n}\n@inproceedings{akimoto2016projection,\n  title={Projection-based restricted covariance matrix adaptation for high dimension},\n  author={Akimoto, Youhei and Hansen, Nikolaus},\n  booktitle={Proceedings of the Genetic and Evolutionary Computation Conference 2016},\n  pages={197--204},\n  year={2016}\n}\n@article{loshchilov2018large,\n  title={Large scale black-box optimization by limited-memory matrix adaptation},\n  author={Loshchilov, Ilya and Glasmachers, Tobias and Beyer, Hans-Georg},\n  journal={IEEE Transactions on Evolutionary Computation},\n  volume={23},\n  number={2},\n  pages={353--358},\n  year={2018},\n  publisher={IEEE}\n}\n\nI appreciate that the code is available. I understand that you start out by forking nevergrad. However, that's not a viable long-term strategy, at least when thinking in terms of utility for a wider community. Please put effort into merging your (im my opinion very significant) contributions back into nevergrad.\n\nAlgorithm configuration and selection for optimization is not entirely new. In the bbcomp results, the AS-AC-CMA-ES by Nacim Belkhir seems to be a very successful competitor method. I do not know whether the code is available or not -- I found Nacim's profile on github, but no code base corresponding to his competition entries. If possible, it would be very interesting to compare to his results.\n\nThis brings me to one of the few weak points of the paper. Experiments are performed for ABBO, but the authors rely (solely) on the nevergrad leader board for comparing with competitors. This has pros and cons. The huge advantage: results are not biased by running competitor methods with sub-optimal parameters. This is a huge plus; in effect, this is a rare case where I fully trust all experimental results. However, this means that some interesting baselines may be missing, in particular methods that predate nevergrad (which is still rather new), like AS-AC-CMA-ES.\n\n\nMINOR POINTS\n\nThe fonts in all plots in figures 2 to 5 are far too small, in particular when printed. On screen I need to zoom in quite a bit. I understand that there are space constraints, but in this form the presentation of the results is of limited value. \n\nI really do not understand why the machine learning community keeps talking about losses (and sometimes regrets) when it comes to optimization. The term \"fitness\" in evolutionary computation is no better. A long-established terminology exists already: the thing we minimize is an \"objective function\", and its value at a specific point is an \"objective value\". I vote for paying more attention to using the standard terminology (in general, not only in this paper), since it is compatible across multiple sub-communities of optimization.\n\nLast paragraph of section 2: \"Rocket\" is listed twice.\n\nIt seems that some of the URLs in the references do not work (any more). The bbcomp website has moved here: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/index.html\nI did not find a replacement for the Artelys link, but maybe referencing the bbcomp results does the job.\n\n\nRECOMMENDATION\n\nOverall this is a very nice and valuable paper with two significant contributions. I strongly recommend to accept the paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a benchmarking suite, primarily targeting the domain of evolutionary style optimization algorithms, and an effective heuristic algorithm selection procedure ABBO.  The reviewers seemed quite split in their reviews with significant variance, particularly with one outlier review (9) lifting up the average.  They all felt that there was significant value in the work presented and that the benchmark could be useful for designing and evaluating new methods.  However, there were concerns regarding details about the contributions (e.g. a detailed description of ABBO and which contributions to the suite were novel vs obtained from other benchmarks), the relevance of this work to the ICLR community, and choice of algorithms presented (i.e. not SOTA).  

In general, this seems like a useful contribution for the evolutionary algorithm community but this paper seems off-topic from the conference.  Certainly optimization is important and of interest to the community.  However, there is no machine learning component to the technical contribution of this paper, and it ignores many of the contributions to black-box optimization within this community (see e.g. the citations from AnonReviewer1, and the literature on surrogate-based black-box optimization - i.e. Bayesian optimization).  The RL optimization problems are somewhat relevant, but AnonReviewer1 raises concerns about the reporting of those results and the representation of the current literature.  There is an algorithm proposed in this work, but it's largely heuristic and no comparison is given to state-of-the-art portfolio optimization algorithms from the machine learning community (e.g. P3BO from Angermueller et al., ICML 2019).  A venue such as GECCO seems much more well suited to this work.
Decision: Reject