Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a benchmark suite for black-box optimization that covers more
different types of problems than existing benchmarks. They derive an algorithm
selection system for black-box optimization from it and evaluate its performance
empirically, comparing to other black-box optimization solvers.

The authors address an interesting problem and demonstrate some good empirical
results. It is certainly beneficial to have large benchmark suites to get a
comprehensive picture of the performance of different approaches.

While the general motivation for a new benchmark suite is clear, the specific
properties the authors list are not. Some of the properties in Table 1 seem to
be complementary (far-optimum and translation), while it is unclear why others
are important. It is certainly nice to have an automated dashboard (of what
exactly?) and one-line reproducibility, but within the general context of the
paper, which seems to focus more on generalizability of results, this seems
unimportant. Whether a particular set of benchmarks is complex is to some extent
a question of the definition of "complex", which the authors do not make clear.
I do not know what "human-excluded/client-server" means. While the majority of
properties listed in Table 1 are obviously important for black-box optimization,
it seems that the authors started from the properties that their benchmark suite
has.

The benchmarks themselves seem to be mostly existing benchmarks that were
combined into a new collection -- it would help if the authors pointed out what
benchmarks were specifically created for OptimSuite.

It is unclear how exactly ABBO was created -- was the set of rules determined
over all benchmark instances to maximize a performance metric? ABBO is listed as
one of the main contributions of the paper and it should be explained in more
detail how it was created and how it works.

The labels in the figures are too small.

Update after rebuttal period: Thank you for your clarifications. I have revised my score accordingly.
review_title: Promising approach with issues

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a benchmarking suite to overcome the problem low generalizability with black box optimization algorithm. The benchmarking suite consists of standard academic benchmarks to real world optimization problems. It also covers several scenarios such as dynamic-static, small to large-scale, discrete to mixed-integer etc. This is a relevant contribution to the machine learning however there are several drawbacks which pushes back it's acceptance into ICLR.

(a) The state of the art discussion was good but why benchmarking is crucial and how it is implemented in Optimsuite was very short in description. That was supposed to be the main highlight of the paper whereas discussion in that part was not clear at all. Specifically, more descriptions were needed in terms of what features to include/exclude in the benchmarking suite and how that helps in generalizability.

(b) I know author(s) mentioned about interpretability as future work however I felt really challenging to understand the benchmarking suite especially when you have a combination of academic benchmarks and real world optimization problems together. I think that's a very significant challenge in implementing ABBO.

(c) One thing was not clear to me how the tuning parameters that are often associated with several optimization problems are handled here? are you keeping the tuning parameter same across all the competing methods?

review_title: A good paper but slightly below the acceptance threshold

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
** Summary **

This paper proposes a new benchmark suite for black box optimization algorithms, which contains a mixture of existing tasks such as nevergrad functions and RL tasks from MuJoCo. In addition, the authors propose a new automated approach to algorithm selection (ABBO), which uses a series of rules to select the best method.

** Primary Reason for Score**

The benchmark suite is predominantly an amalgamation of existing tasks, and only appears to be an incremental improvement vs. Nevergrad. The proposed meta-algorithm is built on a set of “hand-crafted selection rules” (in the authors’ own words). This does not seem like a meaningful contribution to the ICLR community, but may be interesting in other, more applied venues. 

** Strengths **

1) The proposed OptimSuite may be more convenient for users.
2) The rules-based system works surprisingly well, which begs the question of whether a learned version could perform better, which may be of interest to the ICLR community.  

POST REBUTTAL:
3) Including all benchmarks into one codebase could improve reproducibility.

** Weaknesses **

1) I don’t think ICLR is the right venue for this work, which seems more engineering focused and may be more suited to an applied venue.
2) Most of the benchmarks in this suite are all included elsewhere already. Many recent papers proposing blackbox optimization algorithms include several of them. In particular, using Nevergrad + MuJoCo seems to achieve most of the desirable properties. 
3) The selection wizard is just a set of heuristics/hand-engineered rules. This might be more useful for industrial applications, rather than ICLR. 
4) Given that this paper is about benchmarking optimization algorithms, the presentation of the results is poor. It is almost impossible to read the plots, and there is no central table/comparison of the different methods. Reading this I learn very about the algorithms that are being benchmarked.
5) I have several issues with the presentation of the RL results:
a) ARS is not even a blackbox optimization algorithm, it uses information about the MDP for state and reward normalization (v2-t). 
b) The “SOTA” results have been copy and pasted from another paper, which pasted them from the ARS paper, which referenced 2017 results. These are nowhere near “SOTA with grad” in RL. In fact, ARS didn’t say they were SOTA, it simply said that was what TRPO got as a baseline. SOTA now is probably MBPO or SAC/TD3, all of which solve these tasks in a fraction of the time.
c) Missing relevant literature: there have been several approaches to blackbox optimization for RL (and other functions) since ARS. I included ones presented either at ICLR or similar venues, which should be discussed:
  i) **Gradientless Descent: High-Dimensional Zeroth-Order Optimization**. Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, Qiuyi (Richard) Zhang. *ICLR 2020*.
  ii)  **Learning to Guide Random Search**. Ozan Sener, Vladlen Koltun. *ICLR 2020*.
  iii)  **From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization**. Krzysztof M. Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Vikas Sindhwani. *NeurIPS 2019*.
review_title: Lacking novelty and relevance to the ICLR community

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: SUMMARY

The contribution of this work is two-fold: it collects an extremely wide range of benchmark problems for black-box optimization, and it proposes a new algorithm called ABBO. Both contributions are significant.

The proposed ABBO method is designed to be a swiss army knife solver, suitable for a wide range of different types of problems. To this end, naturally, it builds on existing components. Overall, the combination looks extremely convincing to me.


CRITICISM

Already the first sentence of the abstract is problematic:
"Existing studies in black-box optimization suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing different optimization algorithms."
This statement is very general. However, black-box optimization is an extremely wide area, ranging (at least) from mathematical optimization over evolutionary computation all the way to machine learning. The statement applied to various degrees to most studies in various subfields. I completely agree with the statement only when restricted to machine learning papers, where experiments are typically limited to very few RL benchmarks. In other areas things are far from perfect, but generally much better (it is understood that avoiding a bias completely is near impossible), since benchmarks e.g. with (YA)BBOB aim at general insights, not (only) at demonstrating peak performance. Please qualify this statement accordingly.

There is one more problematic statement in the abstract:
"A single algorithm therefore performed best on these three important benchmarks, without any task-specific parametrization."
Well, this "single algorithm" is really an algorithm selection machine. Technically it is "a single algorithm", but it is much more useful to think of ABBO as a selection and configuration method. In my understanding this makes a big difference. For optimal performance we need both: powerful components and powerful configurators. Please make absolutely clear that this "single algorithm" really is a configurator, which encompasses multiple components.

Personally I object some of the methods forming the basis of "Algorithm 1", although overall the choices look very solid. [Side note: I very much like the use of Powell's algorithms for fine tuning of approximate solutions found with more robust methods.] I have one suggestion: Diagonal CMA-ES is an outdated method. Please consider low-rank approaches as an alternative, like LM-CMA-ES, VD-CMA-ES and LM-MA-ES.
@inproceedings{loshchilov2014computationally,
  title={A computationally efficient limited memory CMA-ES for large scale optimization},
  author={Loshchilov, Ilya},
  booktitle={Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation},
  pages={397--404},
  year={2014}
}
@inproceedings{akimoto2016projection,
  title={Projection-based restricted covariance matrix adaptation for high dimension},
  author={Akimoto, Youhei and Hansen, Nikolaus},
  booktitle={Proceedings of the Genetic and Evolutionary Computation Conference 2016},
  pages={197--204},
  year={2016}
}
@article{loshchilov2018large,
  title={Large scale black-box optimization by limited-memory matrix adaptation},
  author={Loshchilov, Ilya and Glasmachers, Tobias and Beyer, Hans-Georg},
  journal={IEEE Transactions on Evolutionary Computation},
  volume={23},
  number={2},
  pages={353--358},
  year={2018},
  publisher={IEEE}
}

I appreciate that the code is available. I understand that you start out by forking nevergrad. However, that's not a viable long-term strategy, at least when thinking in terms of utility for a wider community. Please put effort into merging your (im my opinion very significant) contributions back into nevergrad.

Algorithm configuration and selection for optimization is not entirely new. In the bbcomp results, the AS-AC-CMA-ES by Nacim Belkhir seems to be a very successful competitor method. I do not know whether the code is available or not -- I found Nacim's profile on github, but no code base corresponding to his competition entries. If possible, it would be very interesting to compare to his results.

This brings me to one of the few weak points of the paper. Experiments are performed for ABBO, but the authors rely (solely) on the nevergrad leader board for comparing with competitors. This has pros and cons. The huge advantage: results are not biased by running competitor methods with sub-optimal parameters. This is a huge plus; in effect, this is a rare case where I fully trust all experimental results. However, this means that some interesting baselines may be missing, in particular methods that predate nevergrad (which is still rather new), like AS-AC-CMA-ES.


MINOR POINTS

The fonts in all plots in figures 2 to 5 are far too small, in particular when printed. On screen I need to zoom in quite a bit. I understand that there are space constraints, but in this form the presentation of the results is of limited value. 

I really do not understand why the machine learning community keeps talking about losses (and sometimes regrets) when it comes to optimization. The term "fitness" in evolutionary computation is no better. A long-established terminology exists already: the thing we minimize is an "objective function", and its value at a specific point is an "objective value". I vote for paying more attention to using the standard terminology (in general, not only in this paper), since it is compatible across multiple sub-communities of optimization.

Last paragraph of section 2: "Rocket" is listed twice.

It seems that some of the URLs in the references do not work (any more). The bbcomp website has moved here: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/index.html
I did not find a replacement for the Artelys link, but maybe referencing the bbcomp results does the job.


RECOMMENDATION

Overall this is a very nice and valuable paper with two significant contributions. I strongly recommend to accept the paper.

review_title: Solid paper with valuable contributions

Meta-review: This paper presents a benchmarking suite, primarily targeting the domain of evolutionary style optimization algorithms, and an effective heuristic algorithm selection procedure ABBO.  The reviewers seemed quite split in their reviews with significant variance, particularly with one outlier review (9) lifting up the average.  They all felt that there was significant value in the work presented and that the benchmark could be useful for designing and evaluating new methods.  However, there were concerns regarding details about the contributions (e.g. a detailed description of ABBO and which contributions to the suite were novel vs obtained from other benchmarks), the relevance of this work to the ICLR community, and choice of algorithms presented (i.e. not SOTA).  

In general, this seems like a useful contribution for the evolutionary algorithm community but this paper seems off-topic from the conference.  Certainly optimization is important and of interest to the community.  However, there is no machine learning component to the technical contribution of this paper, and it ignores many of the contributions to black-box optimization within this community (see e.g. the citations from AnonReviewer1, and the literature on surrogate-based black-box optimization - i.e. Bayesian optimization).  The RL optimization problems are somewhat relevant, but AnonReviewer1 raises concerns about the reporting of those results and the representation of the current literature.  There is an algorithm proposed in this work, but it's largely heuristic and no comparison is given to state-of-the-art portfolio optimization algorithms from the machine learning community (e.g. P3BO from Angermueller et al., ICML 2019).  A venue such as GECCO seems much more well suited to this work.
Decision: Reject