Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content:  ##########################################################################

Summary:
 
This paper proposes a benchmark suite of offline model-based optimization problems. This benchmark includes diverse and realistic tasks derived from real-world problems in biology, material science, and robotics contains a wide variety of domains, and it covers both continuous and discrete, low and high dimensional design spaces. The authors provide a comprehensive evaluation of existing methods under identical assumptions and get several interesting takeaways from
the results. They found there exists surprising efficacy of simple baselines such as naive gradient ascent, which suggests the need for careful tuning and standardization of methods in this area. 

##########################################################################

Pros: 
 
1. This paper tackles a valuable problem of benchmarking model-based optimization approaches. It will provide some insights in future algorithmic development
 
2. The paper is well written. They present the significance of model-based optimization, and clearly describe the problems, challenges and considerations of model-based optimization tasks. They conduct further analysis and discussions on the experimental results and find several interesting takeaways.
 
##########################################################################

Cons: 
 
1. Contribution is limited. There are no new ideas proposed and no significant findings revealed. Maybe the authors can look deeper into the simple takeaways and move forward to get more insights and draw some generalized conclusions.

2. In continuous control tasks such as HopperController, I think we cannot get a truly policy without trajectory data. I do not believe the datapairs of the weights of a neural network controller and the corresponding return values can be directly used to learn useful knowledge that contributes to calculate a reasonable policy. It is hard to get a generalizable function that maps from weights of a controller network to resulting values.

review_title: Official Blind Review #1

final_score: rating: 7: Good paper, accept
review_content: I liked the paper overall. The motivation of the paper is clear, and given that offline ML based optimization is beginning to take traction, this is a good time to set up benchmarks and evaluation metrics. The variety of domains considered, with careful consideration of complexity, are good characteristics of the benchmark.

The simple baseline considered makes sense and challenges the community to develop algorithms that can generalize to different problem characteristics.

A few suggestions for improving the paper:
- The hyper-parameters for each algorithm need not be the same for each task. It is possible that tuning the hyper-parameters for particular tasks improves the performance of the algorithms.
- As the paper points out, one cannot evaluate these algorithm at scale as they require real world experimentation. An important aspect that is not discussed in the paper is offline evaluation of the optimization algorithms. In a practical application, we need to know the efficacy of an algorithm for a particular task before they are deployed on the real task.
- None of the tasks considered have constraints on the problem. This is especially challenging for model based methods in continuous design space. 
- The algorithms implemented only consider model based methods. The more popular traditional methods such as genetic algorithms and mixed integer programming would be good to compare against. ML methods will only be adopted if they can beat existing established methods.
review_title: A meaningful benchmark for offline optimization

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
This paper focuses on model-based black box optimization problems in the offline setting. These are settings where access to ground truth is expensive, and instead the optimizer has access only to a trained model of the ground truth based on limited data. While optimizing on this surrogate space, a good optimizer often needs to account for model uncertainty and accuracy degradation. The main aim of the paper is to provide a test bed for algorithms that try to solve this challenge. 

Positives: 
This is a well-motivated line of work because there is a large interest in these problems across fields. There is indeed a need for better benchmarks and better libraries to make it easy to compare methods. I think the paper is executed cleanly and the it's well-written. I also think the work, when completed, has potential to be very useful. 

Areas for improvement: 

In my view, the paper has shortcomings in it's design, development and scope. A paper like this is helpful when it can:

 (i) Establish good practices across the board by streamlining workflow, and ensure the interface is used the same way when comparing methods.

(ii) Contribute code that make it easy and fast to use and develop on.

(iii)  Make it easy to collect and report relevant performance statistics the same way across algorithms, helping the field by making it easy to benchmark.

(iv) Includes challenges that are diverse but relevant to the use case of the algorithm. 

(v) Synthesize a suite of methods from the literature that are distinct from each other and of interest to the community as strong benchmarks. 

I don't think the paper is addressing these aspects sufficiently. Some detailed comments below. 

(1) The categorical choice of benchmark tasks is not clearly justified (e.g. Why should anyone care if a protein design algorithm is poor at designing a robot controller?) At the end of the day, these problems share little structure, and "no free lunch" arguments (Wolpert 95) would suggest that there is no one good algorithm for every challenge. The real world settings for these problems don't map well to each other. This paper as designed, could result in follow up work with meaningless comparisons between algorithms that have no business being compared (unless there is a meaningful connection between the challenges). If the authors feel like they can justify this particular set of challenges, I'm open to be convinced. E.g. if the argument is that there will be a "master algorithm" that is just good at designing everything, and this benchmarking set is designed to enable that, I could drop this point, but then other issues will be relevant. 

(2) The particular choices within each class of tasks is insufficiently justified. Why is GFP a good design challenge given that the ground truth is also necessarily a trained oracle with likely poor performance outside the training data? Just because some previous papers chose this as the design task doesn't make it a good benchmark. No statistics is provided for how good the GP model for GFP is.  As for other proteins, GB1 for instance is far more completely surveyed than GFP.  There are also better published models of GFP available (e.g. TAPE, UniRep). All of these would of course struggle with out-of-domain samples. Why not use a physical simulator like Rosetta that wouldn't have this issue? I believe when designing a benchmarking suite, these decisions should be considered more carefully, as it would quickly become the test bed for follow up work and a flawed design choice here amplifies in future work. 

(3) A body of literature for algorithms that can readily perform MBO has been neglected.  It is trivial to run a regular optimization algorithm with the model (instead of ground truth) and compare the proposed solutions to ground truth.  Quality-Diversity/EDA algorithms (e.g. genetic algorithms,  simulated annealing,  CMA-ES, or even pure CEM (rather than DbAS)...) consistently perform "well" in these high-dimensional optimization settings.  The success of the gradient-based method gives more reason to believe representatives of each of these classical approaches should be included and suggests that the claim that climbing proxy model will necessarily result in bad "ground truth" outcomes is a weak one. 

(4) For a benchmarking library like this, there needs to be mature code available for review (not submitted). I've checked the provided website multiple times, and while it is under active development, the code is not accessible. From what I gather the current code interface simply gives access to some data points and a ground truth. This is too little API. A good benchmarking tool would let the user abstract away the modeling part easily, and be able to readily port and run their algorithm against benchmarks, producing the results in the same way. It should also take care of running sanity checks/tests for the user and generating the same plots as those in the paper. 

(5) The fact that gradient-based methods outperform other methods presented here is only surprising in the sense that they were not included in the original papers (i.e. why weren't gradient-based methods benchmarked there? not this paper's fault of course).  The authors express  a general conviction where gradient based methods have done very poorly in other attempts for MBO, but provide no references, it would be great to cite relevant references for this claim.  

(6) As is, the paper/library only compares CbAS/DbAS with MINs and hill-climbing methods. I think it is not sufficient breadth of methods to make a "benchmarking" suite. As far as I can tell, CbAS/DbAS and MINs are not the best published algorithms in any of the domains suggested, so the authors should justify why they are the algorithms to benchmark against?

For instance, Angermueller et al 2020 ICLR, have an offline RL algorithm that can in principle solve all of these problems. In fact DynaPPO, PPO, and Ensemble-based Bayesian optimization all outperform CbAS in that study. Some sequence design challenges used there seem to be better benchmarks than GFP.  There is substantive work in molecular design on MBO,  but none of the SOTA algorithms are included (e.g the now-classic Gomez-Bombarelli 2016, or  perhaps adaptation of Zhou et al 2019 Scientific Reports). A good rule of thumb in my view is to include the SOTA or well-established algorithm for each task category. 

(6) I suggest the authors think carefully about what the evaluation criteria are. While optimization itself is a good metric, other factors, such as providing a way of evaluating diversity of solutions, or sensitivity to dataset size (e.g. by subsampling), are good to consider.

==========

I would like to encourage the authors to continue the pursuit of this work because it is relevant and well-motivated, and has great potential, in my view it is simply not ready. I think this work needs to be reviewed again when it is more mature,  with wider range of algorithms, better justified challenges, a larger set of metrics that can be easily collected, and available code such that the reviewer can vet the benchmarks and code properly. Right now, it's a comparative review of a small set of methods, not a good benchmarking suite.

If done well, it can be a very useful suite that can help researchers develop better algorithms. The danger of accepting it prematurely is that it will be a basis for future work that "game-ify" studies of algorithms against irrelevant/misleading set of benchmarks. That is only damaging to the development of good algorithms and could misguide research. As it stands,  I find the latter risk higher than it's contribution, and hence I believe it should be rejected and reviewed once more of these structural issues are addressed (and code is available to review).


~~~~~~~~~~~~~~

Post review and discussion remarks:
I think the authors have improved the paper significantly during the review period. However, three of my main concerns about the paper remain to a degree that I'm not confident about the paper's value (or risk of misleading followups). (1) That the set of challenges is somewhat arbitrary, some tasks are using "real" ground truths while others are simply running on known trained oracles.  (2) That implementation of strong offline RL benchmark algorithms are missing (because they don't exactly apply across domains) even though they can always be applied in this setting even if "exact" conditions are not applied in every case (just like Gradient Ascent or BO were) (3) That the API needs to offer more for this to be a good benchmarking suite. 

I've been most concerned about 2 and 3, and after reading the code, I find that it is still too "bare bones" to be a good package. I looked back at OpenAI gym, and there are several abstractions that they make, including actions, observations, environments, spaces that help the implementer unify how they deal with the complexity underneath. 

So far as I can tell, most of what design-bench does is load a csv matrix into an task.x, task.score(x) , and also lets the user access some approximate oracle task.y. This are critical to the process but their abstraction as related to the paper are not clear to me at this time. How is task.y computed, how is the ground truth actually representative of reality. How does optimization depend on the choice of oracle for task.y? 

Having read the code, useful elements are in there to make for a good package, I feel like it needs improvements and another review for scientific soundness.

I've updated my score to address the improvements made. The paper scores somewhere between a 4 and a 5 for me.
review_title: Valuable if done well, but needs substantial improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the evaluation of offline black-box optimization algorithms. The community currently lacks a standardized benchmark to compare the performance of methods. This paper presents a new suite of offline model-based optimization tasks and standardized evaluation procedures for the community. The evaluation criterion for the quality of a benchmark is the realism and diversity of the tasks, with special consideration for high-dimensional design space and the objective function's sensitivity. The paper then evaluates several algorithms on the benchmark. 

The creation of a useful benchmark is an important and challenging task. It seems that consideration was given to the choice of optimization problems. The result is a diverse set, with some representing real-world design optimization problems. However, there are a few areas in which the benchmark and evaluation should be improved. 

I do not recommend this paper for acceptance as there is insufficient support as to why these problems should be considered over others or what challenges these environments present for designing new algorithms. There are also some deficiencies in the evaluation protocol.

For the GFP, Molecule, and superconductor problems, an "expert model" is used as the oracle function. The expert model's use speeds up the evaluation and is undoubtedly the right choice in creating a benchmark, but it introduces bias when evaluating an algorithm. The goal of the paper is to present a benchmark for use in the development of novel algorithms. It is important that if others are to use this benchmark to design new algorithms, they should be aware of the benchmark's biases. A more detailed investigation of these specific optimization tasks and what type of algorithms they favored is warranted.  


In the morphology tasks, the data is generated using a policy trained with a given robot morphology and then later evaluated using a different morphology. This seems to bias the optimal choice of morphology to be the one the policy was used during training. Can the authors clarify what is intended to be learned from including these tasks in the benchmark?



The evaluation protocol is lacking in two areas: how hyperparameter tuning is considered and how results are compared. 

It is said that no hyperparameter tuning can use the oracle function. This is an obvious necessity, but it remains unclear to what extent algorithms are allowed to perform hyperparameter tuning. This makes it unclear if one algorithm performs better than another due to more hyperparameter tuning. Some recent works (Sivaprasad et al. 2020, Jordan et al. 2020, Dodge et al. 2019) address some of these issues when evaluating algorithms. 

In comparing the results, it is not clear how one determines which algorithm performs best. No aggregate measure is given to determine performance. Furthermore, it is unclear how the uncertainty of the results are quantified. These methods have some degree of stochasticity in the performance, be it from the algorithm or choice hyperparameters. It is unclear what sources of uncertainty are being considered in the results and how many trials have been executed. What do the +/- numbers represent in tables 2 and 3, and how are they computed?

Small notes for areas of improvement: 

In section 7 study, an ablation study is mentioned for gradient descent, but there is no discussion of how this study was carried out. Gradient descent is known to be sensitive to the scaling and parameterization of the function, which has led to the design of many optimizers that use preconditions on the gradients, e.g., Newton's method, natural gradient, Adagrad, RMSProp, etc. This problem has also been directly considered in finding optimal design points (Box and Draper 2007). 

The sensitivity (smoothness) of the objective function to small perturbations of its inputs is an important characteristic to consider. The provided example in Figure 2 claims that the objective function is highly sensitive, but the provided example does not appear to have high sensitivity but is rather discontinuous. Not to say that this is not a challenging problem, just that high sensitivity is perhaps not the correct interpretation for this example. 

In the MuJoCo environment tasks, the oracle objective function is evaluated 16 times and averaged to reduce noise. How was this number chosen, and how uncertain are the evaluations? It seems unlikely that it can be assumed that with 16 samples, the sample mean is normally distributed. 

Additionally, can the authors clarify why 100 and 1000 timesteps are sufficient for the MuJoCo tasks of interest? As presented, these seem like arbitrary choices.

Box, George EP, and Norman R. Draper. Response surfaces, mixtures, and ridge analyses. Vol. 649. John Wiley & Sons, 2007.

Sivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., & Fleuret, F. (2020). Optimizer benchmarking needs to account for hyperparameter tuning. In Proceedings of the 37th International Conference on Machine Learning.

Jordan, S. M., Chandak, Y., Cohen, D., Zhang, M., & Thomas, P. S. (2020). Evaluating the Performance of Reinforcement Learning Algorithms. In Proceedings of the 37th International Conference on Machine Learning.

Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. arXiv preprint arXiv:1909.03004.


review_title: A good start

Meta-review: This paper proposes a benchmark suite of offline model-based optimization problems. This benchmark includes diverse and realistic tasks derived from real-world problems in biology, material science, and robotics contains a wide variety of domains, and it covers both continuous and discrete, low and high dimensional design spaces. The authors provide a comprehensive evaluation of existing methods under identical assumptions and get several interesting takeaways from the results. They found there exists surprising efficacy of simple baselines such as naive gradient ascent, which suggests the need for careful tuning and standardization of methods in this area, and provides a test bed for algorithms that try to solve this challenge. However, most reviewers agreed that a more in-depth analysis and insightful explorations for the RL experiment results will help readers understand why their method has superiority even without trajectory data, and  that the paper needs another revision before being accepted. Therefore, I recommend rejection although all reviewers agreed that the tasks is very interesting and a good start.
Decision: Reject