Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary and contributions: The paper proposes a set of offline model-based optimization(MBO) datasets and evaluation methods for offline MBO model benchmarking. The tasks are derived from real-world problems in biology, material science, and robotics. The paper also benchmarked 9 popular offline MBO methods and reported results.
strengths: The motivation of the benchmarks is well justified.

The initial example of how offline MBO can find designs better than the best in the dataset is appreciated.

Code on GitHub is made available and looks relatively clean.

weaknesses: The choice of tasks are questionable in several regards:

1. The Hopper task, together with Ant and D’Kitty Morphology tasks, are classic tasks in reinforcement learning (RL) whose objective functions are computationally cheap to query. This makes them bad candidates for offline MBO settings. In fact, the RL community routinely performs online MBO in this type of task.

2. Non-exact oracles can be problematic. Of the 5 remaining tasks (excluding the 3 mentioned in the previous point), 4 of them use non-exact oracles. Using a trained model as an oracle is generally problematic, since these oracle models will make wrong predictions and will suffer from common issues such as distribution shift. In addition, the ResNet oracle for ChEMBL (rho=0.3208) is simply bad to begin with.

3. The choice of K=128 is not well justified. The authors cite reference 7,10,22 to justify the use of 100 percentile values but not the specific choice of K. I believe that K is an important hyperparameter for the benchmark for two reasons. (1) changing K will likely change the results and/or relative ranking of benchmarked methods. (2) in real world settings, the query time scales linearly with K. If a single query is expensive, one might prefer a model that performs well with small Ks. In summary, Different Ks should be explored and K=128 should be justified.

4. As an additional piece of evidence for poor task/experimental design, results for a few tasks are non-discriminative. For example, 7 out of 9 models score 0.864-0.865 in the GFP task, and results of all 9 models cluster around 0.650 in the UTR task.

MINs results are missing. Although the authors added a note to explain, this suggests the work is not well planned, especially considering that MINs is a model proposed by (a subset of) the authors previously.

Misc.: The paper has some typos such as ones in line 365: “unofmralized” should be “unormalized”, and “50%ile” is unconventional.

correctness: As discussed before, the correctness of non-exact oracles is questionable, the choice of K is not well justified, and thus results can’t be trusted.
clarity: Overall the writing is clear with one major exception. 

It is unclear what metric is being reported in the main table (Table 2). The first paragraph of section 7 describes the numbers as “the 100th percentile performance among them averaged over 8 independent runs”. One need to go to appendix C1 to see that the “performance” is a normalized relative performance bounded between 0 and 1 with higher values indicating better performance. In this regard, the main body is not self-contained.
relation to prior work: The authors clearly discussed how this work relates to previous contributions. It is a novel type of benchmark.
additional feedback: While I believe that the field of offline MBO would benefit from a high quality benchmark suite, I think many aspects of this work are fundamentally flawed. I would like to suggest that the authors carefully rethink the choice of tasks, oracles, as well as metrics.
review_title: Well motivated work but riddled with many flaws.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper presents a benchmark for offline black-box model-based optimization. The goal is to find an input to maximize an unknown objective function based on a collected dataset. The motivation is that querying the objective function could be expensive in practice, and one prefers to use the data one already has. This paper collects some previous datasets and presents a suite of tasks as a benchmark, including problems in biology, materials science, and robotics. The authors further benchmark many existing algorithms. The code is also open-sourced on GitHub under the MIT license.
strengths: 1.	Black-box model-based optimization is an interesting and general problem. Offline black-box model-based optimization addresses the scenarios when querying the objective function is expensive and time-consuming, which potentially has many real-world applications.
2.	The paper is clearly structured with detailed descriptions. The authors clearly explain the motivations, challenges, the details of the datasets, etc.
3.	The codebase looks neat. The README looks easy to follow, and the code seems to have sufficient comments with the design of base classes.
weaknesses: 1.	The design of the oracle function is confusing to me. The authors note that “use models built by domain experts as our ground-truth oracle functions… the training data provided for offline MBO is still real data”. Sim2Real itself is a very challenging problem. It is unclear to me why we should optimize based on the real data and evaluate with the models built by domain experts. The evaluation protocol seems questionable due to the gap between simulation and real data. Putting it another way, if an algorithm can perform well on the “real oracle” by using real data, how can we make sure it also performs well on the designed oracle function.
2.	The toy dataset in Figure 1 seems not realistic. The authors note that “but discard the samples that have the combination of best x and y.” In real-world applications, it seems not reasonable to mask some good samples. Rather, we may sample data following a distribution, such as uniform distribution. I am curious whether the offline black-box model-based optimization can discover better design in this setting.
correctness: The design of the oracle function seems problematic. See weakness.
clarity: The paper is well-structured with clear descriptions.
relation to prior work: The authors discussed sufficient related work
additional feedback: A typo:

Line 47, To address -> To address this
review_title: An interesting benchmark, but some designs are questinable

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors of Design-Bench propose a scheme to benchmark offline model-based optimisation techniques which do not have access to the true objective function and only have some data from which they can optimise. This field is not as studied as other settings where the objective is available, but is much harder and potentially more impactful to a broader audience. The offline-MBO community generally lacks standardised methods for comparing performance of algorithms across different domains. This work takes a first step towards proposing a scheme for doing this. The authors choose relevant datasets and tasks comparing current methods against standard oracles.

strengths: This paper has three main strengths; the paper is well-motivated, well-written, and very thorough.

The authors do an excellent job at describing the field and motivating the problem. They take great care to describe the many disparate and complex peculiarities of each task/dataset, oracle, and algorithm. (I do posit below that there is slightly too much detail on these specifics for the benchmark track. However, I'm confident that the authors can strike a balance for this venue.) I also enjoyed the space in the paper given to the considerations of what important characteristics are necessary for a benchmark in this field.

The authors are excellent communicators and the manuscript reads very well.
weaknesses: I see three main weaknesses in this paper:

Main weaknesses:

(WM1) The structure of the presentation of this benchmark is not yet sufficient for the NeurIPS dataset and benchmark track. See my comments in clarity for more on this. Essentially, I find the balance of motivation and benchmark methodology to be incorrect for a benchmark specific track. I'd like to see clarity and space given to the ways in which the evaluation is conducted. The ultimate goal of this benchmark is to (a) unify the community around the metrics and evaluation protocol presented, and (b) allow for further extension with new tasks/datasets/oracles and (most importantly) algorithms. I suggest the authors provide more details on the specifics of the evaluation methods and abstraction for the inclusion of a new task/dataset, oracle, and algorithm. Additionally, since a benchmark is only as good as its evaluation procedures, a subsection in the main paper which strongly advocates for your chosen procedures and what differences (in results) one might expect from different choices. For instance, if a different oracle is chosen, how much does that change the result?

(WM2) The only specific benefit detailed in this paper is the result regarding naive gradient ascent. Assuming that this is the type of result that the authors think validates the importance of this benchmark, the manuscript would seriously benefit from a more detailed discussion of it. How do the authors think this result will impact the community? What about other types of results that are derived from extensions of this benchmark?

(WM3) The ultimate reason why this benchmark is important is because it enables new offline MBO algorithms to be developed with broader applicability across the burgeoning field. This is necessary because MBO has so many applications in diverse domains that siloing of advancements can be deleterious to the whole. However, this paper, as written, does not convince me that this ultimate goal is appropriate given the highly-domain specific nature of each task. Essentially, why would the pursuit of a general purpose algorithm that works well on many offline MBO tasks be a better thing for the community than a more domain-specific solution which others adapt towards their own goal?



Minor weakness:

(Wm1) With n=8, it is standard practice to use a t-value instead of a std deviation in the tables.
correctness: Major point:

(CM1) Most evaluation decisions appear correct, though I believe that more discussion should be included in the main body of the paper to motivate some of the evaluation and design choices.

Minor points:

(Cm1)Two claims seemed to be in contradiction: the asterisk in the data tables corresponding to MINs and ChEMBL (indicating it was out of your computational budget) and the claim of Appendix C.4 are incongruous.

(Cm2) Figure 2, second figure appears to not "skew the distribution of values to the left" as described in the caption (though the main point about the thin manifold holds regardless).

clarity: The paper is well written. One main concept is not defined: 100 and 50th percentile performance. The meaning is not self-contained in the manuscript.

Looking at the structure of the manuscript, the majority is devoted to setting up the problem and motivation, and descriptions of the various tasks. The description of the evaluation methods (standardisation, evaluation protocol, etc) and results are relegated to either the appendix or (what feels like) a small and fast discussion at the end of the main paper in Section 7. Of course benchmark papers are very complex and require much more detail than could reasonably fit in the length of a NeurIPS submission. The authors made a reasonable choice to handle this inherent challenge.

I would have preferred more space in the main paper spent discussing the specific choices of the benchmarking procedure, namely the normalisation protocol (both score and gradient ascent), hyperparameter workflow, etc. As currently presented, the benchmark results do not feel self contained and many of the important considerations when evaluating a benchmark are relegated to the appendix or entirely missing. While always a challenge in motivating and distilling large and complex work, I think the reader would have benefitted from this description more than they would from the discussion of the tasks and design consideration motivations in Sections 4, 5, and 6.

relation to prior work: Based on my reading of the paper, this work is substantially different from prior work, and all pertinent references are included therein. The work primarily synthesises and benchmarks other offline MBO (of which there are not many) and other important baselines for comparison. In this latter category, the authors chose sufficiently relevant algorithms from a much broader and diverse landscape.

additional feedback: If the authors choose to update this manuscript for this track, I would encourage them to keep the overall structure of the paper, as it very much addresses many of the important aspects of a benchmark. My main suggestion for light revision is to rebalance the content to be less heavy on the domain-specific information and heavier on the specifics of the benchmark procedures.


The following are some small typos/grammatical errors in the manuscript. They didn't influence my opinion at all.

L97: improving -> improve

L106: objective -> objectives

L365: unofmralized -> unnormalized

L971: is -> are
review_title: A good paper in need of structural revision

Meta-review: The paper proposes a benchmark for model-based optimization (MBO). Reviewers praised the submission as well-motivated and well-written, and agreed that such a benchmark could be useful; reviewers also found the codebase to be clean and well-documented. However all reviewers raised concerns about the choice and design of the particular tasks included in the benchmark, and most notably about the use of non-exact oracle functions.

Of the eight tasks included in the benchmark, only four (Ant Morphology, D’Kitty Morphology, Hopper Controller, and TF Bind 8) have exact oracle functions. Reviewer VHeZ claims that the first three are standard tasks in RL; this is not exactly correct, since the standard RL formulations of these problems involve learning a policy to control a fixed morphology. However it is not clear that the formulations of these tasks are realistic. The Ant and D’Kitty Morphology tasks are to find a robot morphology that maximizes the performance of a fixed and pretrained controller; however it is hard to imagine realistic situations where robot morphology can be changed but the controller is fixed, since the former would require changing hardware but the latter can be achieved in software. The Hopper Controller task seems similarly contrived; while it is distinct from the standard RL task (L241-L244), it is not clear why any real-world learning scenario would give learners access to network weights paired with overall returns and without any trajectory information.

The other four tasks rely on learned oracle functions. All reviewers expressed concern with the use of learned oracle functions, since learned oracles may not precisely match the physical reality of the task. The AC is sympathetic to the author’s reply that the goal of the benchmark is not to improve performance on the underlying tasks but instead as a generic benchmark for MBO algorithms; however even with this motivation the authors should demonstrate that the tasks in the benchmark are reasonable tasks where MBO might be deployed, and that the tasks can indeed be used to differentiate MBO algorithms. From this perspective, the ChEMBL task seems ill-suited since its learned oracle has a very low rank correlation with a held-out validation set, and as Reviewer VHeZ points out the GFP and UTR tasks seem ill-suited since they do not clearly discriminate between the MBO algorithms analyzed.

Overall despite the submission’s clear motivations, concerns with the design of many of the tasks comprising the benchmark make it unclear whether it will truly be useful for the future development of MBO algorithms. The authors are encouraged to take these concerns into account and submit a revised version of the benchmark to a future venue.
Decision: Reject