Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization
Authors: Brandon Trabucco
Aviral Kumar
Xinyang Geng
Sergey Levine
Venue:  (2021)
Keywords: Model-Based Optimization
Benchmark
Offline

Abstract: Black-box model-based optimization (MBO) problems, where the goal is to find a design input that maximizes an unknown objective function, are ubiquitous in a wide range of domains, such as the design of drugs, aircraft, and robot morphology. Typically, such problems are solved by actively querying the black-box objective on design proposals and using the resulting feedback to improve the proposed designs. However, when the true objective function is expensive or dangerous to evaluate in the real world, we might instead prefer a method that can optimize this function using only previously collected data, for example from a set of previously conducted experiments. This data-driven offline MBO set- ting presents a number of unique challenges, but a number of recent works have demonstrated that viable offline MBO methods can be developed even for high- dimensional problems, using high-capacity deep neural network function approximators. Unfortunately, the lack of standardized evaluation tasks in this emerg- ing new field has made tracking progress and comparing recent methods difficult. To address this problem, we present Design-Bench, a benchmark suite of offline MBO tasks with a unified evaluation protocol and reference implementations of recent methods. Our benchmark suite includes diverse and realistic tasks derived from real-world problems in biology, material science, and robotics that present distinct challenges for offline MBO methods. Our benchmarks, together with the reference implementations, are available at sites.google.com/view/design-bench. We hope that our benchmark can serve as a meaningful metric for the progress of offline MBO methods and guide future algorithmic development.

Under review as a conference paper at ICLR 2021
DESIGN-BENCH: BENCHMARKS FOR DATA-DRIVEN OFFLINE MODEL-BASED OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Black-box model-based optimization (MBO) problems, where the goal is find a design input that maximizes an unknown objective function, are ubiquitous in wide range of domains, such as the design of drugs, aircraft, and robot mor- phology. Typically, such problems are solved by actively querying the black-box objective on design proposals and using the resulting feedback to improve the proposed designs. However, when the true objective function is expensive dangerous evaluate in the real world, we might instead prefer method that can optimize this function using only previously collected data, for example from a set of previously conducted experiments. This data-driven offline MBO set- ting presents number of unique challenges, but number of recent works have demonstrated that viable offline MBO methods can be developed even for high- dimensional problems, using high-capacity deep neural network function approx- imators. Unfortunately, the lack of standardized evaluation tasks in this emerg- ing new field has made tracking progress and comparing recent methods difficult. To address this problem, we present Design-Bench, a benchmark suite of offline MBO tasks with unified evaluation protocol and reference implementations of recent methods. Our benchmark suite includes diverse and realistic tasks derived from real-world problems in biology, material science, and robotics that present distinct challenges for offline MBO methods. Our benchmarks, together with the reference implementations. are available at sites.google.oo www.ggocoooooo..o..oooo com//iew//esignnbeeneeee We hope that our benchmark can serve a meaningful metric for the progress of offline MBO methods and guide future algorithmic development.
INTRODUCTION
Automatically synthesizing designs that maximize a desired objective function one of the most important problems in many scientific and engineering domains. From protein design in molecu- lar biology Shen et al. (2014) to superconducting material discovery in physics Hamidieh (2018), researchers have made significant progress in applying machine learning methods to such optimiza- tion problems over structured design spaces. Commonly, the exact form of the objective function is unknown, and the objective values for novel design can only be evaluated by running either com- puter simulations or physical experiments in the real world. The process of optimizing an unknown function is known as black-box optimization, and is typically solved in an online iterative manner, where in each iteration the solver proposes new designs and query the objective function for feed- back in order propose better design the next iteration Williams & Rasmussen (2006). In many domains however, the evaluation of the objective function is prohibitively expensive, because it re- quires manually conducting experiments in the real world. this setting, one cannot simply query the true objective function to gradually improve the design. Instead, collection of past records of designs and their corresponding objective values might be available, and therefore the optimization method must leverage the available data to synthesize the most optimal design possible. This is the setting of data-driven offline model-based optimization.
Although online black-box optimization has been studied extensively, the offline MBO problem has received comparatively less attention, and only small number of recent works study offline MBO in the setting with high-dimensional design spaces, where they utilize deep learning tech- niques Brookes et al. (2019); Kumar& & Levine (2019); Fannjiang & Listgarten (2020). This is partly due to the fact that methods for online design optimization cannot be easily applied in the offline

---

Under review as a conference paper at ICLR 2021
MBO setting. However, even with only a few existing methods, it is still hard to compare and track the progress in this field, as these methods are proposed and evaluated on different tasks with distinct evaluation protocols. To the best of our knowledge, there is commonly adopted set of benchmarks for offline MBO. To address this problem, in this paper we introduce a suite of offline MBO benchmarks with standardized evaluation protocols. We include realistic and diverse set of tasks that spans a wide range of application domains, from synthetic biology to robotics. The realism and diversity of the tasks is essential for the evaluation of offline data-driven model-based optimization methods. as it measures the generality of the methods being evaluated across multiple domains and verifies that they are not overfitting to a single task. Our benchmark tasks incorporate variety of challenging factors, such as high dimensional input spaces and sensitive discontinuous objective functions, which help better identify the strengths and weaknesses of MBO methods.
Along with the benchmark suite, we also present reference implementations of a number of existing offline MBO methods and baselines. We systematically evaluate them on all of the proposed bench- mark tasks and report our findings. A surprising discovery from our findings is that with proper data normalization, the simple baseline method of learning an objective value predictor and performing gradient ascent its input outperforms several prior MBO methods in our benchmark. We hope that our work can provide insight into the current progress of offline MBO methods and can also serve meaningful metric galvanize research in this area.
2 OFFLINE MODEL-BASED OPTIMIZATION PROBLEM STATEMENT
The goal in offline model-based optimization is to optimize an unknown (possibly stochastic) objec- tive function fxx), provided access to a static dataset D = {(x, 31}} of designs x; and a correspond- ing measurement of the objective value yt- Similar to batch Bayesian optimization (BBO) González et al. (2016), each algorithm 2 is allowed to consume the dataset DD and is required to produce a set of K candidate designs (DD K) = (x] : E {1...K]} These K candidates are evaluated under the ground truth objective function f(x), and the best performing design is reported as the final performance value. Abstractly, the objective for offline MBO is:
arg max [P([[(x)) x" E (D K)), a
(1)
where denotes the percentile function. Intuitively, this formulation ranks an offline MBO al- gorithm using the Arth percentile objective value obtained by it given fixed evaluation budget of s-samples. Common choices of N are 100, which represent the max objective value, and 50 which represents the median objective value among the candidates.
What makes offfine MBO especially challenging? The offline nature of the problem requires that the algorithm 2 not be tuned by peeking into the ground truth objective f, and this makes the offline MBO problem much more difficult than the online design optimization problem. One simple idea to tackle this problem is to learn an objective proxy using the dataset, and then convert this offfine MBO problem into an online problem by treating the learned objective proxy as the true objective. However, this idea may not work well, due to the intrinsic out-of-distribution nature of optimal designs. First of all, in a number of practical MBO problems such as optimization over proteins or robot morphologies, the designs with highest objective values in the dataset already lie on the tail of the dataset distribution, since they are better than most other designs. In order to improve upon the best designs, an optimization method needs to produce designs that are even further away from the dataset distribution. For such out-of- -iit----dittiittionn designs, is would be impossible to guarantee that the learned objective proxy is accurate, and hence any powerful optimization method would easily "exploit" the learned objective proxy and produce falsely promising designs that are drastically overestimated by the learned objective proxy. This conflict between the out-of-distribution nature of optimization and the in-distribution requirement of any learned model is indeed the core challenge of offline MBO. This challenge is often exacerbated in real-world problems by the high dimensionality of the design space and the sparsity of the available data, offline MBO method needs to carefully balance the two sides of the conflict, producing optimized designs that not too far from the data distribution.
will show in our benchmark. A good
2

---

Under review as a conference paper at ICLR 2021
Task
Dataset Size
Design Dimensions
Design Space
Discrete Categories
GFP-v0
5000
238
Discrete
20
GFP-v1
6852
238
Discrete
30
MoleculeAcctivtty-v0
4216
1024
Discrete
2
Superconductor-v0
16953
81
Continuous
N/A
HopperController-vo)
3200
5126
Continuous
N/A
AntMorphology-v0
12300
60
Continuous
N/A
DKittyMorpholog--WV
9546
56
Continuous
N/A
Table I: Overview of the tasks in our benchmark suite. Design-Bench includes a variety of tasks from different domains with both discrete and continuous design spaces and 3 tigh-dimensional tasks with > 200 design dimensions, making it suitable for benchmarking offline MBO methods.
RELATED WORK
While the extensive prior work on MBO, particularly Bayesian optimization, has made tremendous progress both in terms of enabling intelligent active selection of query points Kolda al. (2003); Lizotte (2008); Rubinstein & Kroese (2013); Powell (1998); Whitley (1994) and scaling up the di- mensionality of Bayesian optimization methods Snoek et al. (2012; 2015); Shahriari et al. (2015), significant number of real-world problems may be approached more naturally as offline MBO problems, given the availability several datasets which contain designs annotated with their cor- responding score values. Offline MBO presents a set up challenges, we believe that these challenges require a new set of benchmarks that emphasize offline data and high-dimensional design spaces.
discuss below, and
Researchers working on either online active design optimization or offline model-basee optimization have collected various datasets of designs that can be used to build tasks for offline MBO. Sarkisyan et al. (2016) 6) analyze the fluorescence of GFP proteins under blue and ultraviolet light, and Brookes al. (2019) use this dataset for optimization to find the protein with the highest fluorescence value. ChEMBL Gaulton et al. (2012) provides dataset for drug discovery, where molecule activities are measured against target assay. Hamidich (2018) analyze the critical temperatures for super- conductors and provide dataset to search for noom-temperature superconductors with potential in the construction of quantum computers. Some of these datasets have already been employed in the study of offline MBO methods Kumar & Levine (2019); Brookes et al. (2019); Fannjiang & List- garten (2020). However, these studies use different set of tasks and their evaluation protocols are highly field-specifie, making it difficult to use for general algorithm development. In bench- mark, we incorporate modified variants of some of these datasets along with our own tasks and provide a standardized evaluation protocol. We hope that these tasks can represent realistic MBO problems across a wide range of domains and that the standardized evaluation protocol can facilitate development of new and more powerful offline MBO algorithms.
Recently, researchers have proposed several methods specifically for the completely offline MBO problem. To make up for the fact that the true objective function is not available, these methods often make use of some type of learned objective function as proxy for the real one and a genera- tive model Kingma& Welling (2013); Goodfellow al. (2014) capture the distribution of valid designs, especially for high dimensional design spaces. Kumar & Levine (2019) learns objec- tive value conditioned generative model synthesize the designs with the desired objective value. Brookes et al. (2019); Fannjiang & Listgarten (2020) employ variational autoencoder Kingma & Welling (2013) to capture the distribution of designs in the dataset and a learned neural network objective function model. In our benchmark, we provide open-source reference implementations for these methods and systematically evaluate them in all our tasks.
DESIGN-BENCH BENCHMARK TASKS
In this section, we describe the set of tasks included in our benchmark. We first provide overview of the tasks in Table 1. Each task in our benchmark suite comes with a dataset D = {(,, y/ } along with oracle objective function f which is to be utilized for for test time evaluation only. An offline MBO algorithm should not query the oracle during training time, even for hyperparam- eter tuning. While in some of the tasks in our benchmark, such tasks pertaining to robotics
3

---

Under review as a conference paper at ICLR 2021
HopperController-vO0 DKityMorphology--o and AntMorphology-vo the oracle functions eval- uated by running computer simulations to obtain the true objective values, in the other tasks, the true objective values can only be obtained by conducting expensive physical experiments. While the eventual aim of offiin MBO is to make it possible to optimize designs in precisely such set- tings where querying the groundtruth objective is challenging, requiring real physical experiments for evaluation makes the design and benchmarking of new algorithms difficult and time consuming. Therefore, to facilitate evaluation, following evaluation methodology in prior work Brookes et al. (2019); Fannjiang & Listgarten (2020), and use models built by experts as our oracle function. Such models are typically also learned models, but with representations that are hand-designed, built-in domain-specific inductive biases and typically trained on much more data than is made available for solving the offline MBO problem, which increases the chance that this proxy "true function" can answer queries outside of the MBO training distribution. While this approach to evaluation dimin- ishes the realism of our benchmark, since these proxy "true functions" may not always be accurate, we believe that this trade off' is worth it to make benchmarking practical. The main purpose of our benchmark is to facilitate the evaluation and development of offline MBO algorithms, and we believe that it is important to include tasks in domains where the true objective values can only be obtained via physical experiments, which make up a large portion of the real-world MBO problems. For each task we discuss in this section, we provide detailed description of the data collection strategy and the data pre- pre-proeessing strategy in Appendix A.
GFP-v0 and GFP-v1: protein fluorescence maximization. The goal of this task is to design a derivative protein from the Aequorea victoria green fluorescent protein (GFP) that has maximum fluorescence, using real-world dataset mapping proteins to fluorescence collected by Sarkisyan et al. (2016). While we cannot exactly evaluate any novel protein, we employ an expert Gaussian process regression model with a protein-speciffe kernel built by Shen et al. (2014) as the oracle fune- tion, following the convention of Brookes et al. (2019); Fannjiang & Listgarten (2020). This expert model is built on a larger dataset than the one we use in this benchmark, making relatively accurate for proteins not in our dataset. The design space for GFP is discrete, consisting of a sequence of 238 categorical variables that can take one of 20 or 30 options, which corresponds to the choice of 238 amino acids that make up the green fluorescent protein from 20 types of amino acid. In brief, GFP-v0 uses a Gaussian process as the expert model, while GFP-vl adapts the 12-layer Transformer provided by the TAPE framework as the expert model. More details are in Appendix A.
MoleculeActivvtty-v.: drug activity maximization. This task is taken from the domain of drug discovery, where the goal is design the substructure of molecule that exhibits high activity when tested against a target assay. The dataset we provide with this task was originally collected by ChEMBL Gaulton et al. (2012). Again like for the GFP task, the true molecule activity against a target assay can only be evaluated with physical experiments. Therefore, we leverage an oracle used recently for meta-learning Yao et al. (2020) and adopt the expert random forest regression model built by Martin et al. (2019) as the oracle function. From the ChEMBL data, we pick a single target assay to form the dataset of this task, resulting in 4216 molecules in total. The design space for this task is a sequence of 1024 binary categorical variables that correspond to the Morgan radius 2 substructure fingerprints, making this task a high-dimensional task.
Superconductor-v0:: critical temperature maximization for superconductor materials. The Superconductor-vo task is taken from the domain of materials science, where the goal is to design a superconducting material that has high critical temperature. Developing superconducting materials has immense practical value, especially for room-temperature superconductors, which could greatly facilitate quantum computing. We adapt a real-world dataset used by materials scientists Hamidieh (2018). The dataset contains 21263 superconductors annotated with critical temperatures. Prior work has employed this dataset for the study of offline MBO methods Brookes et al. (2019), and we follow their convention of using an expert random forest model, detailed in Fannjiang & Listgarten (2020), for our oracle function. The design space for Superconductor-vo is an 81-channel vector of continuous variables, which describe properties of the material such as its atomic radius.
HopperConttroller-v:: robot neural network controller optimization. HopperController-vo) is a task created by us in the domain of robotics, where the goal is to optimize weights of neural network policy that maximizes expected return on the H2 task in OpenAl Gym Brockman et al. (2016). While this goal might look similar to that of reinforcement learning. it is important to note that the formulation is entirely different. Unlike reinforcement learning, we don't have
4

---

Under review as a conference paper at ICLR 2021
access to any form of trajectory data, neither by actively sampling from the environment nor from a pre-collected dataset. Instead, we only have a dataset of weights a neural network controller and the corresponding return values. Therefore, reinforcement learning methods cannot be applied to solve this problem. Unlike tasks introduced before, we evaluate the true objective value of any design during dataset generation by running 1000 steps of simulation in the MuJoCo simulator. This horizon length is chosen so that the learning problem is long-horizon, and difficult. The design space of this task is high-din dineeeisnionnl with 5126 continuous variables corresponding to the flattened weights of a neural net- work controller. The dataset is collected by training PPO reinforcement learning agent Schulman al. (2017) and recording the agent's 's weights every 10,000 samples.
AntMorphology-vd and MorthyMorpoooggy-ee robot morphology op- timization. We created these two tasks to optimize the morphological structure of two simulated robots: Ant from OpenAI Gym Brockman et al. (2016) and D' D'Kitty from ROBEL Ahn et al. (2019). For AntMorphology, the goal is optimize the morphology of an ant shaped robot, to run fast as possible, with a pre-trained neural network controller. For Doit phology, the goal is to optimize the morphology of D' Kitty, quadrupedal robot, such that a pre-trained neural network controller can navigate the robot to a fixed location. In this fashion, the goal for both tasks is recover a morphology with which the pre-trained con- troller is compatible. The variable morphology parameters of both robots include size, orientation, and location of the limbs, giving us 60 continuous values in total for Ant and 56 for D'Kitty. To evaluate the ground truth value for given design, we run robotic simulation in MuloCo Todorov al. (2012) for 100 time steps, averaging 16 independent trials. These parameters are chosen to reduce stochasticity, and allow the simulator to run in a minimal amount of time.
5 TASK PROPERTIES, CHALLENGES, AND CONSIDERATIONS
The primary goal of our proposed benchmark is provide general test bench for developing, evaluating, and comparing algorithms for offline MBO. While in principle any online active design optimization problem can be formulated into offline MBO problem by collecting dataset of designs and corresponding objective measurements, it is important to pick a subset of tasks that represent the challenges of real-world MBO problems in order convincingly evaluate algorithms and obtain insights about algorithm behavior. Therefore, several factors must be considered when choosing the tasks, which we discuss next.
Number samples
GFP-v0
Number samples
MoleculeActivity..n
Number samples
Supereonductor-vd
Number of samplo plopss
400
1000
200
1003
335
1000
Protein Dnoresomcce
Drug activity
Critical temperature
Average return
Sampled uniformly
Original
Figure 1: Histogram (frequency distribution) of objective values in the dataset compared to uniform re-sampling of the dataset from the design space. In every case, re-sampling skews the distribution of scores the left, suggesting that there exists thin manifold of valid designs the high-dimensional design space, and most of the volume in this space is occupied by low-scoring designs. The distribution of objective values in the dataset are often heave-tailed, for instance, in the case of MoleculeActivity--u and Superconductor-vO.
Diversity and Realism. First of all, the tasks need be diverse and realistic in order to prevent offline MBO algorithms from overfitting to particular problem domain and to expect that offline MBO methods performing well on this benchmark suite would also perform well on other real-world offline MBO problems. Design-Bench consists of tasks diverse in many respects: it includes both tasks with discrete and with continuous design spaces, which have distinct implications for offline MBO algorithms. Continuous design spaces, equipped with metric space and ordering structures, could make the problem easier to solve than discrete design spaces. On the other hand, discrete design spaces finite and therefore the dataset coverage might be better than some continuous
5

---

Under review as a conference paper at ICLR 2021
DKittMMorhooooggw.
Average
Succoods
22
Fails:
Leg orientation
Highly sensitive landscape of the groundtruth objective function in DKittyorpholog--o. A small change in single dimension of the design-space, for instance, by changing the orientation 0 (x-axis) of the base of the agent's front right leg, the performance value (y-axis) of the agent is critically impacted. The agent's 's design the original Kitty design and is held constant while varying uniformly from to n.
Figure 2:
tasks. A strong offline MBO algorithm needs to be able to handle both cases. Moreover, our tasks have varying dimensionality, ranging from 56 to 5126 dimensions. And finally, our tasks span multiple realistic domains ranging from biology and physical science robotics.
High-dimensional design space. In many real-world offline MBO problems such as drug discov- ery Gaulton et al. (2012), the design space is high--dimensional and the valid designs lie thin manifold in this high-dimensional space. Due to the curse of dimensionality, the available design data is often sparsely distributed in the design space. This property poses unique challenge for many MBO algorithms: to be effective on such problem domains, MBO methods need to strongly restrict optimization
thin manifold of the design space be able produce valid designs. Prior work Kumar & Levine (2019) has noted that this can be extremely hard in several cases. In our benchmark, we include the GFP-vO, MoleculeActivitty-v) and HopperController-v00 tasks with high-dimensional design spaces to capture this challenge. To intuitively understand this challenge, we performed a study on the HopperControoler--v0 task Figure 1, where we sampled 3200 designs uniformly at random from the design space and plotted histogram of their objective values against those the dataset we provide, that only consists of valid designs. Observe the clear discrepancy in the objective values, where randomly sampled designs attain objective values much lower than the dataset. This indicates that valid designs only lie on a thin manifold in the design space and therefore
very unlikely to hit valid design by uniform random sampling.
Highly sensitive objective function. Another important challenge that should be taken into con- sideration is the high sensitivity of objective functions, where closeness of two designs in design space need not correspond to closeness in their objective values, which may differ drastically. This challenge is naturally present in practical problems like protein synthesis Shen al. (2014), where the change of a single amino acid could significantly alter the property of the protein. The DKityMorphology-Wu and AntMorphology-vv tasks in our benchmark suite are also particularly challenging in this direction. To visualize the high sensitivity of the objective function, we plot one dimensional slice of the objective function around single sample our dataset in Figure 2. We see that with other variables kept the same, slightly altering one variable can significantly reduce the objective value, making hard for offline MBO methods produce the optimal design.
Heavy-tailed data distributions. Another important property that can pose challenge for offline MBO methods is the shape of the data distribution. In particular, machine learning algorithms are likely to exhibit poor learning behavior when the distribution of objective values in the dataset is heavy-tailed. This challenge is often present in black box optimization (Chowdhury & Gopalan, 2019) and can hurt the performance of MBO algorithms that generative model as well those that use a learned model the objective function. As shown Figure I, the Superconductor-v0 and MoleculeActivityy-e tasks in our benchmark exhibit this heavy-tailed structure.
ALGORITHM IMPLEMENTATIONS IN DESIGN-BENCE
To provide a baseline for comparisons in future work, we ran a number of recently proposed MBO algorithms each of our tasks. Since the dimensionality of our tasks ranges from 56 to 5126, we chose methods that have been shown in prior work to handle both the case offline training data (i.e.. no active interaction) and high nigh-dimensiona inputs. To that end, we include MINs Kumar & Levine (2019). CbAS Brookes et al. (2019), and autofocusing CbAS Fannjiang & Listgarten (2020) in comparisons, along with baseline naïve "gradient ascent" method that approximates the

---

Under review as a conference paper at ICLR 2021
true function f(x) with a deep neural network and then performs gradient ascent on the output of this model. In this section, we briefly discuss these algorithms, before performing a comparative evaluation in the next section.
Gradient Ascent. We evaluate a simple baseline that learns a model of the objective function, f(x) and optimizes x against this learned model via gradient ascent. Formally, the optimal solution x generated by this method can be computed as fixed point of the following update: X <
X; + a x In practice we perform T gradient steps, and report Xy as the final solution. Such methods are notoriously susceptible to falling off-the-manifold of valid inputs since nothing constrains the resulting XT to be on the manifold of valid-designs. We employ three variants of this procedure, the first optimizing a single learned objective function f(x), the second (Grad, Min) optimizing over the minimum prediction of N learned objective functions in an ensemble, and the third (Grad. Mean) optimizing over the mean ensemble prediction. We set N = 5 in our experiments. We provide details of the gradient ascent procedure in the Appendix Section E.
Conditioning by adaptive sampling (CbAS). CbAS learns a density model in the space of design inputs, px( that approximates the data distribution and gradually adapts it towards the optimized solution x". In a particular iteration I, CbAS alternates between (1) training a variational auto- encoder (VAE) Kingma & Welling (2013) on set of samples generated from the previous model D, {x, i ~ Pt-1(-) using a weighted version of the standard ELBO objective that puts a higher likelihood on estimated high-scoring designs (i.e., designs that have a score beyond a certain pre-specified threshold), and (2) generating new design samples from the autoencoder to serve as D1+1= = {x, ~ Pe (.).). In order to estimate the objective values for design samples from the learned density model px(x), CbAS utilizes separately trained models of the objective function, fxx), that trained via supervised regression to map an input x; its objective value yı- This training process, at a given iteration i, can formally be described as:
I
Po
D++1(x) == arg min m (x.)
2 log log where {x:) ~ Pt(+)
(2)
Autofocusing CbAS. Since CbAS uses a learned model of the objective function f(x) as proxy for the ground-truth function to iteratively adapt the generative model p(x) towards the optimized design, the f will inevitably be required to make predictions under shifting design distri- butions (x). Since these learned predictions are used ground-truth values, any inaccuracy in the learned function can adversely affect the optimization procedure. Autofocused CbAS aims to cor- fxx) (now denoted fx()) under the design distribution given by the current model, pt (x) via importance sampling, which is then used by CbAS.
rect for this distribution shift by re- re-training
f = arg min
1 |D]
Pr(xi) (x(x,)
(f(x.) V13
Model inversion networks (MINs). MINS learn an inverse map from the objective value to a design, f-1 y + X by using optimization and finally query the learned inverse map to produce the corresponding optimal design.
inverse maps, search for optimal y values during
These methods minimize a divergence measure (D) = (y-pp(() D(pD g). f (xly)) to train such an inverse map. During optimization, MINs obtain the optimal y-value that is used to query the inverse map, and obtains the optimized design by sampling form the inverse map.
Covariance matrix adaptation (CMA-ES), CMA-ES Hansen (2006) is an evolutionary algorithm that maintains a distribution over the optimal design, and gradually refines this distribution by adapt- ing the covariance matrix. Formally, let x, ~ N(4, Σ,) be the samples obtained from the Gaussian distribution an iteration of the algorithm, then CMA-ES computes the value of a learned ob- jective function f(xx) on samples X,, and fits Σ,41 to the highest scoring fraction of samples x, and iterates the process for multiple iterations. Typically CMA-ES utilizes the groundtruth objective function, but as evident from above, we utilize a learned model of the objective the offline setting. This learned model, f(x is learned via supervised regression.
REINFORCE (Williams, 1992). (Adaptation of Angermueller et al. (2020)) We also evaluated method that aims to optimize learned objective function, f(x), using the REINFORCF-style
7

---

Under review as a conference paper at ICLR 2021
Algorithm Name
GFP-v0
GFP-vi
Supercondactor-v)
Autofocus
1,331 0.057 (0.031)
1,580 + 0.001 (0.000)
0.920 - 0.053 (0.029)
[041 = 0.150 (0.083)
CbAS
1.638 +: 0.072 (0.039)
1,580 + 0.001 (0.000)
0.903 0,049 (0.027)
0.117 (0.064)
MINs
1.406 1: 0.082 (0.046)
1,580 1: 0.001 (0.000)
0 1: 0.089 (0.04))
1.084 1: 0.144 (0.079)
Gradient Ascent
0.356 1: 0.002 (0.000)
-1.775 0.010 (0.005)
1,029 0.025 (0.034)
1.211 4
(0.070)
Grad. Min
0,441 -: 0.185 (0.
-1.670 + 0.231 (0.127)
0.993 + 0.028 (0.015)
1,128 - 0.121 (0.066)
Grad. Mean
0.431 +: 0.097 (0.053)
-1,286 4: 1.017 (0.560)
0.987 0.044 (0.024)
1.189 = 0.141 (0.078)
REINFORCE
1.249 4: 0.121 (0:000)
1,576 4: 0.003 (0.005)
0.965 0.028 (0.000)
1,170 0.058 (0.000)
BO-qEI
0.999 + 0.000 (0.000)
1,579 + 0.000 (0.000)
1.222 + 0.062 (0.000)
1.213 = 0.000 (0.000)
CMA-ES
0.356
(0.000)
-1.755 + 0.006 (0.018)
(0.000)
1.332
(0.000)
Table 2: 100th percentile evaluations on the first four tasks in our benchmark. Results are averaged over 16 trials, the - indicates the standard deviation of the reported performance, and the parenthesis at the end of each cell indicate the 95th confidence interval of the sample mean by fitting a T distribution to the performance sample points, calculated using the T distribution from s scipy py. statts S.. Values are normalized to zero if the solution found by optimization performs like the worst sample in the training datasct, and a one if the solution performs like the best sample in the training dataset. A value > 1 indicates that solution better that the best observed sample is found. Similarly, a value < indicates that the optimizer has found a solution worse that the worst observed sample. Unnormalized results can be found in Table 6 in Appendix D
Algorithm Name
HopperController-vo
AntMorphology-v0
DKittyMorphology--u
Autofocus
0.326 + 0.105 (0.058)
2.158 + 0.044 (0.074)
1.787 1 0.233 (0.428)
CbAS
0,402 1 0.311 (0.171)
2.183 1 0.016 (0.029)
1.752 + 0.298 (0.547)
MINS
0,548 1 0.468 (0.257)
2.164 0.038 (0.021)
1.672 + 0.190 (0.1
Gradient Ascent
0.772 = 0.209 (0.100)
2.212 = 0.021 (0.040)
1.858 = 0.242 (0.322)
Grad, Min
1.434 + 0.327 (0.180)
running
running running
Grad, Mean
1.379 = 0.517 (0.285)
rumning
REINFORCE
0.406 + 0.217 (0.000)
running running running
running
BO-QEI
0.558 + 0.105 (0.000)
running running
CMA-ES
0.497 + 0.253 (0.000)
Table 3: 100th percentile evaluations on the final three tasks in our benchmark. Results are averaged over 16 trials, and the = indicates the standard deviation of the reported performance. For description of the score normalization methodology, refer to the caption of Table 2. Unnormalized results corresponding to this table can be found in Table 7 in Appendix D. "running" indicates that these runs have not yet completed
volicy-gradient estimator. This estimator parameterizes distribution (x(x) over the input space and then updates the parameters 0 of this distribution towards the design that maximizes f(x), using
the gradient, Ex (x) [Vo log my (x) fx)).. This method can be regarded as an adaptation of the DynaPPO method from Angermueller et al. (2020) to our setting. Not all our tasks can be formulated as a sequential peeisionnmmakig problem, therefore we pose offline MBO as a "one-step" problem. To mimic the selection of models from Angermueller et al. (2020), we train an ensemble of f(x) models and pick the subset of models that satisfy validation loss threshold This threshold is task-specific; for example, T S 0.25 is sufficient for Superconductor.vo.
Bayesian optimization (BO-qEI. We perform offline Bayesian Optimization to maximize the value of a learned objective function, f(x), by fitting Gaussian Process, proposing candi- date solutions, and labelling these candidates using f(x). To improve efficiency, we choose the quasi-Exppeted-IIppen Improvemmntt acquisition function Wilson et al. (2017), and the implementation of Bayesian Optimization provided in the BoTorch framework Balandat et al. (2019).
7 BENCHMARKING PRIOR METHODS
In this section, we provide a comparison of prior algorithms discussed in Section 6 on our proposed tasks. For purposes of standardization, easy benchmarking, and future algorithm development, we present results for all Design-Bennt Bench tasks in Table 2 and 4. As discussed Section 2, we pro- vide each algorithm with a dataset, and allow the method to produce K = 128 optimized design candidates. These K 128 candidates are then evaluated with the oracle function, and we report the th percentile and the thth percentile performance among them averaged over 16 independent trials, following the convention of prior offline MBO works Fannjiang & Listgarten (2020).
Algorithm setup and hyperparameter tuning. Since our goal is to generate high-performing solutions without any knowledge of the groundtruth, oracle function, any form of hyperparameter tuning on the parameters of the learned model should respect the evaluation boundary. We provide recommended method for tuning each algorithm described in Section 6 that is fully offline. To briefly summarize, for ChAS, tuning amounts to finding stable configuration for a VAE, such that samples
8

---

Under review as a conference paper at ICLR 2021
from the prior distribution map to on-manifold designs after reconstruction. We empirically found that a 3-VAE was essential for stability of CbAS and high values of B > 1 are especially important for modelling high- sigh-dimensional spaces like that of HopperController-vO. As general task-agnostic principle for selecting 8. we choose the smallest /3 such that the VAE's latent space does not collapse during importance sampling. Collapsing latent-spaces seem to coincide with diverging importance sampling, and the VAE's reconstructions collapsing to a single mode, For MINs, tuning amounts to fitting good generative model. We observe that MINs particularly sensitive to the scale of Y: when conditioning, which we resolve by normalizing the objective values. We implement MINs using WGAN-GP, and find that similar hyperparameters work well-across domains. For Gradient Ascent, while prior work has generally obtained extremely poor performance for naïve gradient- ascent based optimization procedures on top of learned models of the objective function, we find that by normalizing the designs x and objective values y to have unit Gaussian statistics, and by multiplying the learning rate f- avd where d is the dimension of the design space (discussed in Appendix E), a naive gradient ascent based procedure can perform reasonably well on most tasks, without any task specifici tuning. For discrete tasks, only the objective values normalized, and optimization is performed over log- pog-prbbabilities of designs. We then uniformly evaluate samples obtained by running 200 steps of gradient ascent starting from the highest scoring 128 samples in each dataset. Tuning instructions for each baseline in this work is available in Appendix F.
Results and discussion. The results for all tasks are provided in Table 2 and 4. There are several interesting takeaways from these results. First, these results indicate that there is no clear winner between the three prior methods MINs, CbAS and Auto- focused CbAS, provided they are all trained offline with no access to groundtruth evaluation for any form of hyperparameter tuning.
HopperControlle-en
Average
Second, somewhat surprisingly, the naïve gradient ascent baseline is competitive with several highly sophisticated MBO methods in 5 out of 7 tasks (Table 2), especially on nigh-dimensional tasks (c.g.. HopperController-VO0
Grafent sweet (ommeskan))
This result suggests that it might be difficult for generative models to capture high-dimensional task distributions with enough precision to be used for optimization. We conducted an ablation study to determine the reasons behind the surprisingly good performance of the naïve baseline. We found that the identical gradient-ascent baseline performed factor 1.4x worse on HopperController-vO,, when optimizing in the space of unnormalized designs and objective values, indicating that normalization is key in obtaining good performance with a naive gradient ascent baseline. For additional information on how normalization is performed, refer to Appendix Section E.
Gradient (unnecmilod))
Gonlient stego
Finally, we remark that the performance numbers for certain methods reported in Table 2 differ from the performance reported by prior works on certain tasks This difference stems from the standard- ization procedure employed in dataset generation (which we discuss in Appendix A), and the usage of uniform hyperparameters to ensure task-agnostic hyperparameter selection.
CONCLUSION
While online model-based optimization methods have received a lot of attention, especially in the Bayesian optimization community, offline model-based optimization is increasingly gaining pop- ularity as it carries the promise to be able to convert existing databases of designs into powerful optimizers, without the need for any expensive online interaction. However, due to the lack of stan- dardized benchmarks and evaluation protocols, it has been difficult to accurately track the progress in this field. To address this problem, we introduce Design- -enchh, a benchmark suite of offline MBO tasks that covers a wide variety of domains, and both continuous and discrete, low and high dimensional design spaces. We provide a comprehensive evaluation of existing methods under iden- tical assumptions. The somewhat surprising efficacy of simple baselines such as naive gradient ascent suggests the need for careful tuning and standardization of methods in this area. Another interesting avenue for future work is to devise methods that can be used to perform model-sclection and hyperparameter selection. One approach to address this problem is to devise methods for offline evaluation of produced solutions, which is also interesting topic for future work. We hope that our benchmark will be adopted as the standard metric in evaluating offline MBO algorithms and provide meaningful insight in future algorithmic development.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. Conference on Robot Learning (CoRL), 2019.
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International Conference on Learning Representations, 2020. URL https: http//ppeeee.eeeeeeeeee net forum?
id=HklxbgBKVv
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. Botorch: Programmable bayesian optimization in py- torch. CoRR, abs/1910.064033, 2019. URL http:: org abs 01900000660 06403.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.011 01540, 2016.
David H Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. arXiv preprint arXiv: /00000000000000 2019.
Sayak Ray Chowdhury and Aditya Gopalan. Bayesian optimization under heavy-tailed payoffs.. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' AAlhé-BBu,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 13890, 2019. URL http: http:::/papppsss/ nips. paper/ wtteer/933--baesiann--e utt.://see-aa.ooiiiiaaaooaoeeeee www.-oo--uunde---ee---eeeeeeeeee
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18853/v1/n19-1144 https: //oii .org//rgg 10. 186538663/3 /wll////1/ 1423.
URI
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 2185-2194. As- sociation for Computational Linguistics, 2019. doi: 10.18653/11D11--2222 1224. -1/211112....
URL https:
//dpi.
Sara El-Gebali, Jaina Mistry, Alex Bateman, Sean R. Eddy, Aurélien Luciani, Simon C. Potter, Matloob Qureshi, Lorna J. Richardson, Gustavo A. Salazar, Alfredo Smart, Erik L. L. Sonnham- mer, Layla Hirsh, Lisanna Paladin, Damiano Piovesan, Silvio C. E. Tosatto, and Robert D. Finn. The pfam protein families database in 2019. Nucleic acids research, 474D1)):477-D432 Jan 2019. ISSN 1362-4962. doi: 10. 1093/nar/gky995. URL https: /ttp//ptbeeee/e////// ncbá. mmmm nih gov/ 30357350. 30357350[pmid].
Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. arXiv preprint arXiv:2006.00 08052, 2020.
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Over- ington. Chembl: large-scale bioactivity database for drug discovery. Nucleic acids research, 40(Database issue): 100---11000011,,,, Jan 2012. ISSN 1362-4962, doi: 10. 1//3//////// URL nebi. nlm. nih. g 21948594[pmid].
ht htttss://
Javier González, Zhenwen Dai, Philipp Hennig. and Neil Lawrence. Batch bayesian optimization via local penalization. In Artificial intelligence and statistics, pp. 648 668,657, 2016.
10

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkes Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
T. Haarnoja, Aurick Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In JCML, 2018.
Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of super- conductor. Computational Materials Science, 154:346 - 354, 2018. ISSN 0927-0256. doi: https:
052.. URL http:/ www. www.sciieeee.... www.neeiieeedireccoo com/
Nikolaus Hansen. The CMA evolution strategy: A comparing review. In José Antonio Lozano, Pedro Larrañaga, Iñaki Inza, and Endika Bengoetxea (eds.), Towards a New Evolutionary Com- putation Advances in the Estimation of Distribution Algorithms, volume 192 of Studies in Fuzzi- doi: 10.1007/3-
ness and Soft Computing, pp. 102. Springer, 2006.
|44. URL
http///i.oog//a//////////oo 10010070111100000000111011111111 540- 32494-1_4.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. attpss//githie com/ com/hill Baallaa/ttable-aaselinaa 2018.
Scott M. Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip S. Thomas. Evaluat- ing the performance of reinforcement learning algorithms. CoRR, abs/2006.16958 2020. URL https: wwtpxiiiwrgga.////ooooooooo abss 16958..
Diederik P Kingma and Max Welling. 1rXiv:1312.6114, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Tamara G Kolda, Robert Michael Lewis, and Virginia Torezon. Optimization by direct search: New perspectives some classical and modern methods. SIAM review, 45(3):385 2003.
Aviral Kumar and Sergey Levine Model inversion networks for model-based optimization. arXiv preprint drXiv:1912.13464 2019.
Daniel James Lizotte. Practical bayesian optimization. University of Alberta, 2008.
Eric J. Martin, Valery R. Polyakov, Xiang-War Wei Zhu, Li Tian, Prasenjit Mukherjee, and Xin Liu. All-assay-man max2 pqsar: Activity predictions as accurate as four-concentrattion ic50s for 8558 no- vartis assays. Journal of Chemical Information and Modeling, 59(10):4450 4459, 2019. doi: 10.102 //aca./ccmm..b......... 9b00355. URL https:/ www.oooooo.oooooooooooooooooeeoe 101111...111111011 ww..////cc......
joim. 9b00375.
PMID: 31518124.
Michael JD Powell. Direct search algorithms for optimization calculations. Acta numerica, pp- 287-336, 1998.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John F. Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with TAPE, In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Confer- Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 9686-9698, 2019. URL htw.: /ttttt:://pppaa..... nips. cc/paper/ 93-eav uuat muating-proted tonnt-rrasfeee fornsferriie wtw.otteennnttaaaeeeeeeaameeeeee www.anree-aannnniiiiiteea
Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: unified approach to combina- torial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media, 2013.
Karen S. Sarkisyan, Dmitry A. Bolotin, Margarita V. Meer. Dinara R. Usmanova, Alexander S. Mishin, George V. Sharonov, Dmitry N. Ivankov, Nina G. Bozhanova, Mikhail S. Baranov, Onu- ralp Soylemez, Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny S. Egorov, Maria D. Logacheva, Alexey S. Kondrashov, Dmitry M. Chudakov, Ekaterina V. Putintseva, Ilgar Z. Mamedov, Dan S.

---

Under review as a conference paper at ICLR 2021
Tawfik, Konstantin A. Lukyanov, and Fyodor A. Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397 May 2016. ISSN 1476-4687. doi:
10.1038/nature11799 URL hitps: http://i.oooggo// http:://bo...rg//00 wttp://..10e//auueeee919991
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. CoRR, abs/1707.00347, 2017. URL http: /ttp://aaxiv.. org/abs/
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):
148-175, 2015.
Wen-Jun Shen, Hau-San Wong, Quan- Xiao, Xin Guo, and Stephen Smale. Introduction to the peptide binding problem of computational immunology: New results. Foundations of Computational Mathematics, 14(5),951-984, Oct 2014. ISSN 1615-3383.
doi: 10.1007/
$10208-013-9173-9, URL https:////// /ttp:::////....e// oreiioorg/100 1007 //t////00000011111311
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and François Fleuret. On the tunability of optimizers in deep learning. CoRR, abs/1910. 11758 2019. ARLhttp://aaxit
11758.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959) 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. International conference on machine learning, pp. 2171-2180, 2015.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ul- rike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL http:://apeeeeeee nips. cc/ cc/paper/ 7181-attent iisiiiooonnnil--eeeeeeeeeliellee is- asll--lou-el---eeedd
Darrell Whitley, A genetic algorithm tutorial. Statistics and computing, 4(2):65-85, 1994.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Ronald J. Williams.
Simple statistical gradient-following algorithms for connectionisttreiffocceeme learning. Mach. Learn, 8:229- 256, 1992. doi: 10.1007/BF00992666. URL https://doi. org/10 111111000///00992220000
James T. Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameter- ization trick for acquisition functions. CoRR, abs/1712.00424, 2017, URL http://arxiv.
org/abs/ aaaaabaa11 /11... 00424.
Huaxiu Yao, Longkai Huang, Ying Wei, Li Tian, Junzhou Huang, and Zhenhui Li. Don't overlook the support set: Towards improving generalization meta-learning. CoRR, abs/2007.13044, 2020. URL hwtph::/taaaaeee wtgg/bss//aa0 13040.
12

---

Under review as a conference paper at ICLR 2021
Appendices
A DATA COLLECTION
In this section, we detail the data collection steps used for creating each of the tasks in design-bench. We answer (1) where is the data from, and (2) what pre-processing steps are used?
A.1
GFP-vO
The GFP task provided is a derivative of the GFP dataset Sarkisyan al. (2016). The dataset we use in practice is that provided by Brookes et al. (2019) at the url htt / /giit tuub. com/ dhbrookes/ CbAS/ wttee/eee///eeeeeeeeeeeeeeeeeeee /eaeeerrre/////e data. We process the dataset such that a single training example consists of a protein represented as a tensor Top E. {0, 1) |,238x20, This tensor is a sequence of 238 one-hot vectors corresponding to which amino acid is present in that location the protein. We use the dataset format of Brookes et al. (2019) with no additional processing. The data was orig- inally collected by performing laboratory experiments constructing proteins similar to the Aequorea victoria green fluorescent protein and measuring fluorescence.
A.2 GFP-vI
This variant of the GFP task provided is a derivative of the GFP dataset Sarkisyan et al. (2016). The dataset we use in practice is that provided by Brookes et al. (2019) the url https: //github. com/ dhbrookes / /CAS// / ree mwsttr////////am/aaaaa... data. We process the dataset such that single training example consists of a protein represented as tensor IGFP E {0, 1] 11238x30 This ten- sor is sequence of 238 one-hot vectors corresponding to which amino acid is present in that location in the protein. We use the dataset format of Brookes et al. (2019) with no additional processing. The data was originally collected by performing laboratory experiments constructing proteins similar to the Aequorea victoria green fluorescent protein and measuring fluorescence.
A.3 MOLECULEACTIVVTT--V
The MoleculeActiivtty task is derivative of much larger dataset that is derived from ChEMBL Gaulton et al. (2012), a large database of chemicals and their properties. The data, similar to GFP, was originally collected by performing physical experiments on large number of molecules, and measuring their activity with a target assay. We have processed the original dataset presented in Martin et al. (2019), which consists of more than one million molecules and 1 11,000 assays, into smaller scale task with 4216 molecules and a single assay. We select this assay by first calculating the number of unique scores present in the dataset per assay, and sorting the assays by the number of unique scores. We select assay 600885 from Martin et al. (2019). This particular assay has 4216 molecules after pre-processing. Our pre-processing steps include converting each molecule into a one-hot tensor #Molecule E {0, 1 11022x2 This is performed by calculating the Morgan radius 2 substructure fingerprints of 1024 bits, which is implemented in RDKit. This calculation requires the SMILES representation for each molecule, which is provided by Martin et al. (2019). The final step of pre-processing, is sub sample the dataset by defining a percentile used to select and discard high- high-performing molecules, such that difficulty of the task is artificially increased. We use a split percentile of 80 for Molecule in the experiments in this paper.
A.4 SUPERCONDUCTOR-VO0
Superconductor-v0 is inspired by recent work Fannjiang & Listgarten (2020) that applies offline MBO to optimize the properties superconducting materials for high critical temperature. The data we provide in our benchmark is real-world superconductivity data originally collected by Hami- dieh (2018), and subsequently made available to the public at the url https: //archh///chiiiee ve. ics. uci. eddemmlmmm http/mmmddaaaseee datasets /Superconductittytteeeo...o perconductors featurized into vectors Supercondueetrr E 8 One issue with the original dataset is that the largest value of single dimension in the dataset is 22590.0, which appears cause learning
4. The original dataset consists of su-
13

---

Under review as a conference paper at ICLR 2021
instability. We follow Fannjiang & Listgarten (2020) and normalize each dimension of the design- space to have zero mean and unit variance. However, we deviate from the remaining pre- preopessssnng steps in Fannjiang & Listgarten (2020). In order to promote task realism, we directly use the super- conductivity data, whereas Fannjiang & Listgarten (2020) re-samples by collecting iid unit gaussian samples and labelling them with the task oracle function. This causes the scores in the dataset cor- respond exactly to the scores provided by the oracle. No other domain in design-bench re-samples nor re-labels static data, so we omit it here for consistency.
A.5 HOPPERCONTROLLER-VO
The HopperController task one that we provide ourselves. The goal of this task to design a set of weights for as neural network policy, in order to achieve high expected return when evaluating that policy. The data collected for HopperController was taken by training a three layer neural network policy with 64 hidden units and 5126 total weights the Hopper-v2 MuJoCo task using Proximal Policy Optimization Schulman et al. (2017). Specifically, we use the default parameters for PPO provided in stable baselines Hill et al. (2018). The dataset we provide with this benchmark has 3200 unique weights. In order to collect this many, we run 32 experimental trials of PPO, where we train for one million steps, and save the weights of the policy every 10,000 environment steps. The policy weights are represented originally as a list of tensors. We first traverse this list and flatten each of the tensors, and we then concatenate each of these flattened tensors into a single training example Hopper E R5126, The result is an optimization problem over neural network weights. After collecting these weights, we perform no additional pre-procsssing steps. In order collect scores we perform single rollout for each x using the Hopper-v2 MuloCo environment. The horizon length for training and evaluation is limited to 1000 simulation time steps.
A.6 ANTMORPHOL.OGY-VO & DKITYMOOPPHO.OGY-VW
Both morphology tasks are collected by us, and share methodology. The goal of these tasks is to design the morphology of quadrupedal robot- ant D' K'itty-such that the agent is able to crawl quickly in a particular direction. In order to collect data for this environment, we create variants of the MuJoCo Ant and the ROBEL D' Kitty agents that have parametric morphologies. The goal is to determine a mapping from the morphology of the agent to the average return that an agent trained for a particular intended morphology achieves. We implement this by pre-training neural network policy using SAC Haarnoja et al. (2018). For both the Ant and the D' D'Kitty, we train agents for up three million environments steps, and a maximum episode length of 1000, with all other settings as default. These agents are pre-trained for fixed gold-standarg morphology the default morphology of the Ant and D' Kitty respectively. Each morphology task consists of samples obtained by adding Gaussian noise with standard deviation 0.02 for Ant and 0.01 for DKitty times the design-space range to the gold-standard morphology. We label each sampled morphology by averaging the return of 16 rollouts of length 100 of an agent with that morphology.
B ORACLE FUNCTIONS
We detail oracle functions for evaluating ground truth scores for each of the tasks in design bench. common thread among these that the oracle, if trained, fit to a larger static dataset containing higher performing designs than observed by downstream MBO algorithm.
B.1 GFP-v0
GFP-vO uses the oracle function from Brookes et al. (2019). This oracle is Gaussian Process re- gression model with a protein-specine kernel proposed by Shen et al. (2014). The Gaussian Process fit to a larger dataset than the static dataset packaged with GFP-vO, making it possible to sample a protein design that achieve a higher score than any other protein seen during training. The oracle score for GFP-v0 is implemented the mean prediction of this Gaussian Process.
14

---

Under review as a conference paper at ICLR 2021
B.2 GFP-v1
GFP-vl oracle from Rao al. (2019). This oracle is a Transformer Vaswani al. (2017) model with 12 layers based on BERT Devlin et al. (2019), and pretrained using masked language modelling and the Pfam database El-Gebali al. (2019), Instructions for obtained this oracle can be found in the TAPE framework at https: /ttp::///gtthbb com/ http///nglaabaaaaaaaaaaaa
B.3 MOLECULEACHIVIT--OO
Following the procedure set by Martin et al. (2019), the oracle function we use for MoleculcActivity vO is random forest regression model. In particular, we use the RandomForestRegress scikit-learn, using identical hyperparameters to the random forest regression model used Martin et al. (2019). The random forest is trained on the entire task dataset. In practice, samples that score most the 80th percentile are observed by an MBO algorithm, which allows for sampling unobserved points that score higher than the highest training point.
provided
B.4 StPERCONDDUTOR-VOO
The Superconductor-v0 oracle function is also random forest regression model. The model we use it the model described by Hamidieh (2018). We borrow the hyperparameters described by them, and we use the Random RandomForestRerresor provided in seikit-learn. Similar to the setup for the previous two tasks, this oracle is trained on the entire static dataset, and the task is instantiated with a split percentile. Samples scoring at most in the 80th percentile are observed by an MBO algorithm, which allows for sampling unobserved points that score the unobserved top 20 percent.
B.5 HOPPERCONTRROLLER-V0
Unlike the previously described tasks, HopperController-vd and the remaining tasks implement exact oracle function. For HopperControllee-vv0 the oracle takes the form of a single rollout using the Hopper-v2 MuJoCo environment. The designs for HopperController-v0 are neural network weights, and during evaluation, a policy with those weights is instantiated-inn this case that policy is three layer neural network with 11 input units, two layers with 64 hidden units, and final layer with output units. The intermediate activations between layers are hyperbolic tangents. After building policy, the Hopper-v2 environment reset and the reward for 1000 time-steps is summed. That summed reward constitutes the score returned by the HopperControoler--v)) oracle. The limit of performance the maximum return that an agent can achieve in Hopper-v2 over 1000 steps.
B.6 ANTMORPHOOL.GGY.Y. & DKITTY
The final two tasks in design-bench both morphology tasks, the simulator performs 16 rollouts and averages the sum of rewards attained them. Each task is accompanied by pre-trained neural network policy. To perform evaluation, a morphology is passed to the Ant or D' DiKitty MuJoCo environments respectively, and dynamic- morphology agent is initialized inside these environments. These environments are very sensitive small morphological changes, and exhibit a high degree of stochasticity as a result. To compensate for the increased stochasticity, we average returns over 16 rollouts.
exact oracle function, using the MuJoCo simulator. For
C ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present the 50th percentile performance of the trials presented in main body. Sim- ilar to the 100th percentile performance reported in the main text, scores are calculated by evaluating solutions to each task found by an optimization method, subtracting the minimum score present in the corresponding task dataset, and dividing by the range of scores present in the corresponding task dataset. The result is performance of greater than one optimization converges to a solution with higher score that the best observed point in the corresponding task dataset.
15

---

Under review as a conference paper at ICLR 2021
Algorithm Name
GFP-v0
GFP-v1
MoleculeActivtt--il
Superconductor-vi
Autofocus
0.072
1.573 0.003
0.699 = 0.059
0,427 = 0.101
CbAS
1,575 = 0.000
0.692 - 0.046
0,435 = 0.098
MINs
0.047
1.535 0.029
0.717 0.029
0.504 = 0.142
Gradient Ascent
0.356
-1.780 0.007
0.941 0.070
0.731 = 0.068
Grad. Min
0.131
-1.772 0.011
0,938 0.033
0.693 = 0.044
Grad. Mean
0.395 + 0.049
-1,584 = 0.716
0.945 + 0.046
0.658 = 0.036
REINFORCE
0,634 1 0.003
-1,476 + 0.072
0.694 1 0.014
0,694 - 0.014
BO-qEI
0.999 1 0.000
1.532 0.000
0.791 0.008
0.991 - 0.000
CMA-ES
0.356 0.000
-1.779 0.001
0.793 0.013
0.725 1 0.020
Table 4: 50th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the = indicates the standard deviation of the reported performance. For description of the score normalization methodology, refer to the caption of Table 2.
Algorithm Name
HopperController-v
AutMorphology-v0
DKittyMoopphoogy-
Autofocus
0,085 + 0.014
1.284 0,249
0.919 + 0.044
CbAS
0,097 1 0.018
1.660 1 0.069
0.938 1 0.018
MINS
0.382 + 0.221
1.317 1 0.123
0.979 1 0.067 1.193 0.031
Gradient Ascent
0.136 0.054
1.871 + 0.050 running
Grad. Min
0.294 + 0.042
running running running running running
Grad, Mean
0,266 + 0.059
running
REINFORCE
0.031 = 0.002
running running running
BO-qEI
0.296 + 0.013
CMA-ES
0.035 0.003
Table 50th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the = indicates the standard deviation of the reported performance. For description of the score normalization methodology. refer the caption of Table
D UNNORMALIZED PERFORMANCE
In this section, we present the unnormalized raw performance of every experiment previously re- ported in the paper. These tables are displayed over the next several pages.
GFP-v0
GFP-y1
MoleculeActivity-v0
Superconductor-vu
Autofocus
3.365 + 0.023
3.805
0.001
6.345 +:2 0.141
77.07 + 11.11
CbAS
3.408 + 0.029
3.805 + 0.0
6.301 + 0.131
72.17 + 8.652
MINs
3.315 + 0.033
6.508 0.236
80.23 + 10.67
Gradient Ascent
2,894 0.001
1.461 0.007
6.636 0.066
89.64 9.201
Grad. Min
2,928 + 0.074
1.534 0.162
6
83.5 1 8.934
Grad. Mean
2.924 0.039
1.802 + 0.711
: 6.125100116
88.01 = 10.432
REINFORCE
3.252 0.049
3.803 + 0.002
6,465 + 0.075
86.58 + 4.27
BO-qEI
3.152 0.0
3.805 0.0
7.147 + 0.164
89.73 0.0
CMA-ES
2.894 + 0.0
1.474 0.004
6.774 + 0.096
98.59 + 5,271
Table 6: 100th percentile evaluations for baselines every task. Results averaged over 16 trials, and the + indicates the standard deviation of the reported performance. This table corresponds to the unnormalized performance originally in Table
E NORMALIZATION OF GRADIENT ASCENT
The goal of this section to discuss the normalization method that is used our Gradient As- cent baseline. For continuous design-space tasks, we normalize both the designs, and the scores to have unit Gaussian statistics. For discrete design-space tasks, we normalize only the scores to have unit Gaussian statistics. This is necessary part of the optimization workflow because scores vary by several orders of magnitude in the dataset, as low as 2.9 for GFP-vO and as high as 1400 for HopperController-vO. As we have shown empirically in Figure 7 The specific normalization
16

---

Under review as a conference paper at ICLR 2021
HopperControler-w
AntMorphology-vll
DKittyMorphology-Wu
Autofocus
443.8 + 142,9
386.9 + 10.58
376.3 =: 47.47
CbAS
547.1 +: 423,9
393.0 3.75
369.1 = 60.65
MINs
746.1 + 636.8
388.5 = 9.085
352.9 38.65
Gradient Ascent
1050.8 284.5
399.9 + 4.941 running
390.7 49.24
Grad. Min
1952.2 445.773
running running running
Grad, Mean
1877.1 + 704.203
running
REINFORCE
553.0 + 295,995
running
BO-qEI
759.4 142.341
running running
running running
CMA-ES
676.5 1 344.52
Table 7: 100th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the + indicates the standard deviation of the reported performance. This table corresponds to the unnormalized performance originally in Table 3.
GFP-vO
GFP-v1
MoleculeActivity-vl
Superconductor-vi
Autofocus
3.216 0.029
3.8 0.002
5.759 1 0.158
31.57
CbAS
3,269 + 0.018
3.802 + 0.002
5,742 : 0.123
32.21 7.255
MINN
3.135 0.019
3.774 0.02
5,806 + 0.078
37.32 + 10.5
Gradient Ascent
2.894 + 0.0
1.457 0.005
6.401 + 0.186
54.06 + 5.06
Grad. Min
2.916 0.053
1.462 0.008
6.393 0.087
51.292 + 3.285
Grad. Mean
2.91 0.02
1.594 + 0.501
6.412 + 0.123
48.716 + 2.637
REINFORCE
3,006 1 0.001
1.67 1 0.05
5.747 1: 0.038
51.358 1 1.059
BO-qEI
3.152 11522100 0.0
3.772 1 0.0
6.004 T 0.02
73.361 0.0
CMA-ES
2.894 0.0
1.458 0.0
6.01 0.033
53.641 1.452
Table 50th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the + indicates the standard deviation of the reported performance. This table corresponds to the unnormalized performance originally in Table
equation for continuous-valued designs is given below.
Xij x(x,)) o (x,)
XERNND
(3)
We also normalize the objective values in similar fashion to have unit Gaussian statistics. The result in set of designs & and objective values ý that is optimized over.
p(Us) Di.j Vi.j (13)
y E
(4)
The gradient ascent procedure is performed in the space of these normalized designs. Suppose T steps of gradient ascent have been taken, and a final normalized solution 87 is found. This solution is de-normalized using the following transformation.
(x = (xT), axx)) m(x,)
(5)
This normalization strategy is heavily inspired by data whitening. which is known reduce the variance of machine learning algorithms that learn discriminative mappings on that data. The learned model of the objective function is one such discriminative model, and normalization likely improves the consistency of Gradient Ascent across independent experimental trials.
F HYPERPARAMETER SELECTION WORKFLOW
Hyperparameter tuning under restricted computational budget is emerging as import research domain optimization Sivaprasad et al. (2019); Dodge et al. (2019); Jordan et al. (2020). Care must be taken when tuning each of the prescribed algorithms the task is used for hyperparameter selection. Formally, this means that the hyperparameters, H, conditionally independent of the particular value of the performance metric M, given the offline task dataset D. Examples of hyperparameter selection strategics that violate this requirement might, for example, perform grid search over H and take the that maximizes the performance metric,
that only offline information about
17

---

Under review as a conference paper at ICLR 2021
HopperContralle-ww
Anthoopphology-v)
DKittyMorpholgy--u
Autofocus
116.4 18.66
176.7 = 59.94
199.3 8.909
CbAS
132.5 23.88 23.88
267.3 16,55
203.2 +: 3.58
MINs
211.6 = 13.67
Gradient Ascent
185.0 + 72.88
318.0 + 12.05
255.3 6.379
Grad. Min
400.091 + 57.042
running running
running running running running running
Grad, Mean
362.547 + 80.091
REINFORCE
42.671 1 3,067
running
BO-qEI
403.363 1 17.854
running ranning
CMA-ES
47.392 4.416
Table 9: 50th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the + indicates the standard deviation of the reported performance. This table corresponds to the unnormalized performance originally in Table 5.
but this is not offline. An example of a tuning strategy that is fully offline tuning the parameters of a learned model such that is is a good fit for the task dataset D. One can choose H that minimizes validation loss, such as negative log likelihood. A detailed record of hyperparameters can be found in the experiment scripts located at the website for besign-bench:: https://siie com/ com/view / des gereeihhgggnh/.....
google.
We now present specific guidelines for hyperparameter selection (i.e. workflow) for some of the methods evaluated in our benchmark. These principles are general principles that can be used to tune the hyperparameters of these methods on new task in completely offline fashion.
F.1 STRATEGY FOR AUTOFOCUSED ORACLES
The main tunable components of Autofocused methods (Fannjiang & Listgarten, 2020) are the learned objective function, and the generative model fit to the data distribution. When training the learned objective function, tracking a validation performance metric like rank correlation is helpful to ensure that the resulting learned model is able to generalize beyond its training dataset. This track- ing is especially important for Autofocused methods because re- re-fitting the learned objective model during importance can lead to divergence if the importance weights generated by Autofocusing very large or very small in magnitude. The algorithm is tuned well if, for example, the validation rank correlation stays above a positive threshold, such as a threshold of 0.9.
The second component of Autofocused methods is the fit of the generative model used for sampling designs. The algorithm has the best chance of success if the generative model can generalize beyond the dataset in which it was trained. This can be monitored by holding out a validation set and track- ing metric such negative log likelihood on this held-out set. In the case when the generative generative model-for example, a VAE-other validation
model is not an exact likelihood-bart metrics can be used that measure the fit of the generative model on validation set. The generative model is especially impacted by the importance sampling procedure used by Estimation of Distribu- tion Algorithms (EDAs), and tracking the effective sample size of the importance weights can help diagnose when the generative model is failing to generalize to validation set.
F.2 STRATEGY FOR CBAS
The main tunable components of CbAS methods (Brookes et al., 2019) are the learned objective function, and the generative model fit to the data distribution. While the learned objective function not affected by the importance sampling weights generated by CbAS, the same tuning strategy described in section F.1 that focuses on generalization to a validation set is effective. Generative model tuning can also follow an identical strategy to that described in section F.I, which focuses on the ability for the generative model to represent samples outside of its training set. In the case of 3- VAE, which is used with CbAS in this work, the main parameter for controlling this generalization ability is the 3 parameter. We found that (3 is task specific, and must be found in order for the CbAS optimizer using 3- VAE to generate samples that are in the same distribution as its validation set. This value can be tuned in practice using validation metric like that in section F.1.
18

---

Under review as a conference paper at ICLR 2021
F.3 STRATEGY FOR MINS
The main tunable components of MINs (Kumar & Levine, 2019) are the learned objective function, and the generative model fit to the data distribution. The learned objective function is typically trained using maximum likelihood objective, and the validation log- --iklliheoood (or regression error) can be directly tracked. The learned objective function should train until a minimum validation loss reached, which ensured that the model will generalize well as possible beyond its training set. Since only the static task dataset is used for this-it may be split into a training and validation set set-this tuning strategy is fully offline.
The generative model for MINs is an inverse mapping x = f- z), conditioned on the objective value y. Training conditional generative models considerable less stable than unconditional gen- erative models, so in addition to monitoring the fit of validation set recommended in section F. 1, is also necessary to track the extent of the dependence of the generative model's predictions on the objective value y. This can be evaluated in practice by comparing the distribution of from the conditional generative model p(xly) to an unconditional generative model p(x) with an identical initialization, or by comparing if p(x (y) is independent of y by querying the inverse model for dif- ferent values of y and visualizing the similarity in the predictions of x. One metric for more formally
studying the extent of the dependence of x is the mutual information I [x: z). The conditional generative model has an appropriate fit if for some positive threshold e we have that 7 (xx; 2) > c.
F.4 STRATEGY FOR GRADIENT ASCENT
The main tunable components of Gradient Ascent MBO methods are the learned objective function, and the parameters for gradient ascent. The learned objective function typically trained using maximum likelihood objective, and the methodology for obtaining high-performing learned objective function is identical to that section F.3. The second aspect of gradient ascent MBO algorithms are the parameters of the gradient-based based optimizer for the designs- as its learning rate, and the number of gradient steps it performs. The learning rate should be small enough that the gradient steps taken increase the prediction of the learned objective function-iil the learning rate is too large, gradient steps may not follow the path of steepest ascent of the objective function. The number of gradient steps is more difficult to tune. The strategy we used is a fixed number of steps, and an offline model-selection criterion to select this parameter is future work.
F.5 STRATEGY FOR REINFORCE
The main tunable components of REINFORCE- MBO methods are the learned objective func- tion, and the parameters for the policy gradient estimator. The learned objective function is typically trained using maximum likelihood objective, and the methodology for obtaining a high- tigh-performing learned objective function is identical to that in section F.3. The remaining parameters to tune are specific to REINFORCE. The distribution of the policy should be carefully selected to be able to model the distribution of designs. For continuous MBO tasks, a Gaussian distribution is appropriate, and for discrete MBO tasks, categorical distribution is appropriate. In addition, the learning rate, and optimizer should be selected so that policy updates improve the -reedccped score. If the stability of the gradient estimator is suffering, due to high-variance updates, density ratio threshold like in PPO (Schulman et al., 2017) can be applied, and a baseline can be subtracted.
F.6 STRATEGY FOR BAYESIAN OPTIMIZATION
The main tunable components of Bayesian Optimization MBO methods (Balandat et al., 2019) are the learned objective function, and the parameters for the bayesian optimization loop. The learned objective function is typically trained using maximum likelihood objective, and the methodology for obtaining a nigh-performing learned objective function is identical to that in section F.3. For detailed review of the strengths and weaknesses of various Bayesian Optimization strategies and their hyperparameters, we refer the reader to the BoTorch documentation, available at the BoTrrc website https: / ///bttoocchhcccccc org/ docs/ / overview. In this work we employ Gaussian Pro- cess as the model, and the quasi-Monte Carlo Expected Improvement acquisition function, which has the advantage of scaling up our high-dimensional optimization problems.
19

---

Under review as a conference paper at ICLR 2021
F.7 STRATEGY FOR COVARIANCE MATRIX ADAPTATION (CMA-ES)
The main tunable components of Covariance Matrix Adaptation MBO methods are the learned ob- jective function, and the parameters for the evolution strategy. The learned objective function is typically trained using a maximum likelihood objective, and the methodology for obtaining a high- performing learned objective function is identical to that in Subsection F.3. For detailed review of the strengths and weaknesses of various Bayesian Optimization strategies and their hyperparameters, we refer the reader to an open-source implementation of CMA-ES and its corresponding documen- tation https: //gittttituu com/ CMA-ES / py cma. In this work we employ the default settings for CMA-ES reported this open source implementation, with o = 0.5.
G PERFORMANCE REPORTING CONFIDENCE INTERVALS
This section contains additional results for each baseline algorithm evaluated on every task, reporting confidence intervals rather than the standard deviation of the performance. We calculate the 90th, 95th, and 99th confidence interval for each method, and report that intervals using = notation next to the sample mean performance. Results calculated using 16 trials per method.
GFP-v0
GFP-v1
MoleculeActtivtt--v
Superconductor-vd
Autofocus
1.531
1.58:0.0
0.91911.029
1.411:0.083
CbAS
1.6371000399
1.58:10.0
0.903:10.027
0.975:0.064
MINs
1.00710.046
1.5810.0
0.98110,049
1.08410.079
Gradient Ascent
0.356+0..0
-1.775+00.05
1.029++0.04
1.215++0.07
Grad. Min
0.44+:0.102
-1.6740.127
0.993+0.015
1.128+0.066
Grad. Mean
0.43+:0.053
-1.286=00566
0.987+00024
1.189-10.078
REINFORCE
1,248.10.0
1.57610.005
0.00
1.17.0.00
BO-QEI
1.0.1.00
1.579
1.221
1.213
CMA-ES
0.356.10.0
-1.755.10.018
1.081
1.332.1..0
Table 10: 100th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the + indicates the 95th confidence interval of the reported performance.
HopperConntroller-vv)
AntMorphology--00
Dortty-Mophooogy-vv
Autofocus
0.326=0.058
2.217+40.074
1.787+00.228
CbAS
0.402=0.171
2.183++0.029
1.752+00.547
MINs
0.548=0.257
2.164++0.021
1.672+0.100
Gradient Ascent
0.753110.1
2.21511.04
16681103222
Grad. Min
1.43410.18
running
running running rumning running running
Grad. Mean
1.329
running
REINFORCE
0.406=+0.0
running running running
BO-QEI
0.558+00.
CMA-ES
0.49740.00
Table 11: 100th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the + indicates the 95th confidence interval of the reported performance.
GFP-v0
GFP-v1
MoleculeActtiity-vi
Superconductor-vi
Autofocus CbAS
1.16:10041
1.57310.002
0.69911.033
0.42710.055
1.292.10.255
1.57511.001
0.693:10.025
0.43511.005
MINs
0.958+0.0066
1.535+00.016
0.7174:0016
0.504:10.078
Gradient Ascent
0.356++0.0
-1.78-t0.004
0.95++0.032
0.731+0.033
Grad. Min
0.41+00.072
-1.772-40.006
0.938+000118
0.693+4..02
Grad. Mean
0.005
-1.584.10.994
0.945.1..025
0.658
REINFORCE
0.63410.00
-1.476.10.132
0.694
0.694
BO-QEI
0.999
1.532.1..0
0.70
1..0
CMA-ES
0.356+0..0
-1.779+0.002
0.793=0.0
0.725=0.0
Table 12: 50th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the + indicates the 95th confidence interval of the reported performance.
20

---

Under review as a conference paper at ICLR 2021
HopperController-vo)
AntMorphology-x0
DKitt.Morphology--d
Autofocus
0.086=00008
1.284:0.458
0.919
CbAS
0.097=0.01
1.66+0,122
0.038
MINs
0.382=0.122
1.317:10.068
0.979
Gradient Ascent
0.136=0.029
1.871+00.045
1.156::..177
Grad. Min
0.294=0.023
running
running
Grad. Mean
0.266=0.032
running
running
REINFORCE
0.031
running running running
running
BO-QEI
0.296110.0
running
CMA-ES
0.035++0.0
running
Table 13: 50th percentile evaluations for baselines every task. Results averaged over 16 trials, and the + indicates the 95th confidence interval of the reported performance.
GFP-y0
GFP-v1
MolcculeActtvity-x0
Superconductor-vl
Autolocus
H.536
L.58:0.00
0.919:00.024
1.041:0:068
ChAS
1.637+0.032
1.58++0.0
0.903++.022
0.975+00053
MINs
1.407++0.038
1.58++0.0
0.981+00.4
1.084++0.065
Gradient Ascent
0.35611..0
-1.77510.004
1.029110.026
1.215:100088
Grad. Min
0.44:100084
11.6710.105
0.99311.0013
1.12811.055
Grad. Mean
0.43:10.044
-1.286110.46
0.987-10.02
1.189:10.064
REINFORCE
1.248+00121
0.576+00.00
0.965+00002
1.17#00008
BO-QEI
1.0+0.00
1.579+0.0
1.221
1.213+0.0
CMA-ES
0.356+00.0
-1.755+00012
1.081
1.332+00001
Table 14: 100th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the + indicates the 90th confidence interval of the reported performance.
HopperConttoollee-vo)
AntMorphology-v00
DKittyMorphhoogy-vu
Autofocus
0.326=0.047
2.217::0.05
1.787
CbAS
0.402=0.141
2.183+:0.021
0.752+00404
MINs
0.222
2.164110.11
1.67210.086
Gradient Ascent
0.753
2.2151100022
I 16668 10.2338
Grad. Min
1.434
running running running running running
running running running running running
Grad. Mean
1.379=0.234
REINFORCE
0,406=0.217
BO-QEI
0.558=0. 105
CMA-ES
0.497
Table 15: 100th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the + indicates the 90th confidence interval of the reported performance.
GFP-v0
GFP-y1
MolcculeActivity--0
Superconductor-v0
Autolocus
1.16.10.033
1.77310.001
0.609
0.42710.046
CbAS
1.292.00.21
1.57510.001
0.693
0.435.10.044
MINs
0.0081
1.53510.013
0.717110013
0.504110.064
Gradient Ascent
0.356+0.0
-1.78+00.033
0.95=00.225
0.731+:0031
Grad. Min
0.419
-1.77240.005
0.938+0.001
0.693+10.02
Grad. Mean
0.395:10.022
-1.58441.324
0.945+10.021
0.65810.016
REINFORCE
0.634-10.003
-1.476+10.098
0.694+10.014
0.69410.014
BO-QEI
0.999+-0.0
1.532++0.0
0.791+00.008 0.793+-0001
0.991+0.0
CMA-FS
0.356+0.00
-1.779+00001
0.725+0002
Table 16: 50th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the + indicates the 90th confidence interval of the reported performance.
21

---

Under review as a conference paper at ICLR 2021
HopperController-v0
AntMorphology-x0
DKittyMorphology--d
Autofocus
0.086=00006
1.284
0.919=000599
CbAS
0.097=0.008
1.66=:00094
0.938=0.024
MINs
0.382-10.1
1.317:10.056
0.979-10.03
Gradient Ascent
0.136=0.024
1.871+0.0336
1.156:10.087
Grad. Min
0.294
running
running
Grad. Mean
0.266=0.027
running
running
REINFORCE
0.001
running running running
running running
BO-QEI CMA-ES
0.296=0.013
0.035=0.003
running
Table 17: 50th percentile evaluations for baselines every task. Results averaged over 16 trials, and the + indicates the 90th confidence interval of the reported performance.
GFP-y0
GFP-y1
MolcculeActivity-vl
Superconductor-vd
Autolocus
L.0000.00T
0.919=0004
1.041:00T14
ChAS
1.637+0.054
1.58++00001
0.903++0038
0.975+00089
MINs
1.407+00003
1.58+0.00
0.981+00068
1.084++0.11
Gradient Ascent
0.356110.0
-1.77510.007
1.02910.057
1.215=100097
Grad. Min
0.44110.14
-1.6710.176
0.99310.001
1.12811.092
Grad. Mean
0.43110.074
-1.286.10.774
0.987110.033
1.18910.107
REINFORCE
1.248+0.00
1.576=00001
0.965=00.0
1.17=0.0
BO-QEI
1.0+0.00
1.579+0.0
1.2
1.213=0.0
CMA-ES
0.356+00.0
-1.755=0004
1.0
1.332=0.0
Table 18: 100th percentile evaluations for baselines every task. Results are averaged over 16 trials, and the + indicates the 99th confidence interval of the reported performance.
HopperControolle--vo)
AntMorphology-x00
DKittyMorphology-vd
Autofocus
0.326::0.08
2.217+:0135
1.787+0.786
CbAS
0.402=0.237
2.183++0.053
1.752+ 1.004
MINs
0.358
2.164-10.029
1.67210.144 144
Gradient Ascent
0.753
2.215110.059
I 1.668 10.599
Grad. Min
1.434
running running running running running
running running
Grad. Mean
1.379=0,393
REINFORCE
0,406+0.0
running running running
BO-QEI
0.558+10.0
CMA-ES
0.497+40.0
Table 19: 100th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the + indicates the 99th confidence interval of the reported performance.
GFP-v0
GFP-y1
MoleculeActivity--0
Superconductor-v0
Autolocus
1.16.10.056
1.57310.002
0.69910.045
0.42710.077
CbAS
1.292.10.035
1.57510.002
0.693
0.435.10.075
MINs
0.008
1.535
0.717-1.022
0.504110.10 108
Gradient Ascent
0.356+0.0
-1.78=40.005
0.95=00.53
0.731+0.052
Grad. Min
0.41
-1.772400008
0.938+0.002
0.69340.034
Grad. Mean
0.395100.037
-1.58440.555
0.945::0.035
0.658:0.027
REINFORCE
0.634=10.0
-1.476:10.222
0.694=0.00
0.694=0.0
BO-QEI
0.999+40.0
1.532++0.0
0.791+0.0
0.991
CMA-ES
0.356+0.00
-1.779+00.044
0.793++0.0
0.725
Table 20: 50th percentile evaluations for baselines every task. Results averaged over 16 trials, and the + indicates the 99th confidence interval of the reported performance.
22

---

Under review as a conference paper at ICLR 2021
HopperConntroller-vo)
AntMorphology-v0
DKittyMorphology-v
Autofocus
0.086:+0.01
1.284-F0.84
0.919
CbAS
0.097=0.013
1.66+:02232
0.938+000059
MINs
0.382-0.166
0.317+00.03
0.979+0.051
Gradient Ascent
0.036
1871110006
1.15610.216
Grad. Min
0.294
running
running running running running
Grad. Mean
0.266-0.045
running
REINFORCE
0.001++0.0
running running running
BO-QEI
0.296+00.0
CMA-ES
0.035+0.0
running
Table 21: 50th percentile evaluations for baselines on every task. Results are averaged over 16 trials, and the + indicates the 99th confidence interval of the reported performance.
23


Reviews:
[{"review_id": "todhQSW7DU1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I liked the paper overall. The motivation of the paper is clear, and given that offline ML based optimization is beginning to take traction, this is a good time to set up benchmarks and evaluation metrics. The variety of domains considered, with careful consideration of complexity, are good characteristics of the benchmark.\n\nThe simple baseline considered makes sense and challenges the community to develop algorithms that can generalize to different problem characteristics.\n\nA few suggestions for improving the paper:\n- The hyper-parameters for each algorithm need not be the same for each task. It is possible that tuning the hyper-parameters for particular tasks improves the performance of the algorithms.\n- As the paper points out, one cannot evaluate these algorithm at scale as they require real world experimentation. An important aspect that is not discussed in the paper is offline evaluation of the optimization algorithms. In a practical application, we need to know the efficacy of an algorithm for a particular task before they are deployed on the real task.\n- None of the tasks considered have constraints on the problem. This is especially challenging for model based methods in continuous design space. \n- The algorithms implemented only consider model based methods. The more popular traditional methods such as genetic algorithms and mixed integer programming would be good to compare against. ML methods will only be adopted if they can beat existing established methods.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_Cs3nvyGs5J", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the evaluation of offline black-box optimization algorithms. The community currently lacks a standardized benchmark to compare the performance of methods. This paper presents a new suite of offline model-based optimization tasks and standardized evaluation procedures for the community. The evaluation criterion for the quality of a benchmark is the realism and diversity of the tasks, with special consideration for high-dimensional design space and the objective function's sensitivity. The paper then evaluates several algorithms on the benchmark. \n\nThe creation of a useful benchmark is an important and challenging task. It seems that consideration was given to the choice of optimization problems. The result is a diverse set, with some representing real-world design optimization problems. However, there are a few areas in which the benchmark and evaluation should be improved. \n\nI do not recommend this paper for acceptance as there is insufficient support as to why these problems should be considered over others or what challenges these environments present for designing new algorithms. There are also some deficiencies in the evaluation protocol.\n\nFor the GFP, Molecule, and superconductor problems, an \"expert model\" is used as the oracle function. The expert model's use speeds up the evaluation and is undoubtedly the right choice in creating a benchmark, but it introduces bias when evaluating an algorithm. The goal of the paper is to present a benchmark for use in the development of novel algorithms. It is important that if others are to use this benchmark to design new algorithms, they should be aware of the benchmark's biases. A more detailed investigation of these specific optimization tasks and what type of algorithms they favored is warranted.  \n\n\nIn the morphology tasks, the data is generated using a policy trained with a given robot morphology and then later evaluated using a different morphology. This seems to bias the optimal choice of morphology to be the one the policy was used during training. Can the authors clarify what is intended to be learned from including these tasks in the benchmark?\n\n\n\nThe evaluation protocol is lacking in two areas: how hyperparameter tuning is considered and how results are compared. \n\nIt is said that no hyperparameter tuning can use the oracle function. This is an obvious necessity, but it remains unclear to what extent algorithms are allowed to perform hyperparameter tuning. This makes it unclear if one algorithm performs better than another due to more hyperparameter tuning. Some recent works (Sivaprasad et al. 2020, Jordan et al. 2020, Dodge et al. 2019) address some of these issues when evaluating algorithms. \n\nIn comparing the results, it is not clear how one determines which algorithm performs best. No aggregate measure is given to determine performance. Furthermore, it is unclear how the uncertainty of the results are quantified. These methods have some degree of stochasticity in the performance, be it from the algorithm or choice hyperparameters. It is unclear what sources of uncertainty are being considered in the results and how many trials have been executed. What do the +/- numbers represent in tables 2 and 3, and how are they computed?\n\nSmall notes for areas of improvement: \n\nIn section 7 study, an ablation study is mentioned for gradient descent, but there is no discussion of how this study was carried out. Gradient descent is known to be sensitive to the scaling and parameterization of the function, which has led to the design of many optimizers that use preconditions on the gradients, e.g., Newton's method, natural gradient, Adagrad, RMSProp, etc. This problem has also been directly considered in finding optimal design points (Box and Draper 2007). \n\nThe sensitivity (smoothness) of the objective function to small perturbations of its inputs is an important characteristic to consider. The provided example in Figure 2 claims that the objective function is highly sensitive, but the provided example does not appear to have high sensitivity but is rather discontinuous. Not to say that this is not a challenging problem, just that high sensitivity is perhaps not the correct interpretation for this example. \n\nIn the MuJoCo environment tasks, the oracle objective function is evaluated 16 times and averaged to reduce noise. How was this number chosen, and how uncertain are the evaluations? It seems unlikely that it can be assumed that with 16 samples, the sample mean is normally distributed. \n\nAdditionally, can the authors clarify why 100 and 1000 timesteps are sufficient for the MuJoCo tasks of interest? As presented, these seem like arbitrary choices.\n\nBox, George EP, and Norman R. Draper. Response surfaces, mixtures, and ridge analyses. Vol. 649. John Wiley & Sons, 2007.\n\nSivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., & Fleuret, F. (2020). Optimizer benchmarking needs to account for hyperparameter tuning. In Proceedings of the 37th International Conference on Machine Learning.\n\nJordan, S. M., Chandak, Y., Cohen, D., Zhang, M., & Thomas, P. S. (2020). Evaluating the Performance of Reinforcement Learning Algorithms. In Proceedings of the 37th International Conference on Machine Learning.\n\nDodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. arXiv preprint arXiv:1909.03004.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PWRnhaLgG4C", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " ##########################################################################\n\nSummary:\n \nThis paper proposes a benchmark suite of offline model-based optimization problems. This benchmark includes diverse and realistic tasks derived from real-world problems in biology, material science, and robotics contains a wide variety of domains, and it covers both continuous and discrete, low and high dimensional design spaces. The authors provide a comprehensive evaluation of existing methods under identical assumptions and get several interesting takeaways from\nthe results. They found there exists surprising efficacy of simple baselines such as naive gradient ascent, which suggests the need for careful tuning and standardization of methods in this area. \n\n##########################################################################\n\nPros: \n \n1. This paper tackles a valuable problem of benchmarking model-based optimization approaches. It will provide some insights in future algorithmic development\n \n2. The paper is well written. They present the significance of model-based optimization, and clearly describe the problems, challenges and considerations of model-based optimization tasks. They conduct further analysis and discussions on the experimental results and find several interesting takeaways.\n \n##########################################################################\n\nCons: \n \n1. Contribution is limited. There are no new ideas proposed and no significant findings revealed. Maybe the authors can look deeper into the simple takeaways and move forward to get more insights and draw some generalized conclusions.\n\n2. In continuous control tasks such as HopperController, I think we cannot get a truly policy without trajectory data. I do not believe the datapairs of the weights of a neural network controller and the corresponding return values can be directly used to learn useful knowledge that contributes to calculate a reasonable policy. It is hard to get a generalizable function that maps from weights of a controller network to resulting values.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7rrdJAu4Bdn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper focuses on model-based black box optimization problems in the offline setting. These are settings where access to ground truth is expensive, and instead the optimizer has access only to a trained model of the ground truth based on limited data. While optimizing on this surrogate space, a good optimizer often needs to account for model uncertainty and accuracy degradation. The main aim of the paper is to provide a test bed for algorithms that try to solve this challenge. \n\nPositives: \nThis is a well-motivated line of work because there is a large interest in these problems across fields. There is indeed a need for better benchmarks and better libraries to make it easy to compare methods. I think the paper is executed cleanly and the it's well-written. I also think the work, when completed, has potential to be very useful. \n\nAreas for improvement: \n\nIn my view, the paper has shortcomings in it's design, development and scope. A paper like this is helpful when it can:\n\n (i) Establish good practices across the board by streamlining workflow, and ensure the interface is used the same way when comparing methods.\n\n(ii) Contribute code that make it easy and fast to use and develop on.\n\n(iii)  Make it easy to collect and report relevant performance statistics the same way across algorithms, helping the field by making it easy to benchmark.\n\n(iv) Includes challenges that are diverse but relevant to the use case of the algorithm. \n\n(v) Synthesize a suite of methods from the literature that are distinct from each other and of interest to the community as strong benchmarks. \n\nI don't think the paper is addressing these aspects sufficiently. Some detailed comments below. \n\n(1) The categorical choice of benchmark tasks is not clearly justified (e.g. Why should anyone care if a protein design algorithm is poor at designing a robot controller?) At the end of the day, these problems share little structure, and \"no free lunch\" arguments (Wolpert 95) would suggest that there is no one good algorithm for every challenge. The real world settings for these problems don't map well to each other. This paper as designed, could result in follow up work with meaningless comparisons between algorithms that have no business being compared (unless there is a meaningful connection between the challenges). If the authors feel like they can justify this particular set of challenges, I'm open to be convinced. E.g. if the argument is that there will be a \"master algorithm\" that is just good at designing everything, and this benchmarking set is designed to enable that, I could drop this point, but then other issues will be relevant. \n\n(2) The particular choices within each class of tasks is insufficiently justified. Why is GFP a good design challenge given that the ground truth is also necessarily a trained oracle with likely poor performance outside the training data? Just because some previous papers chose this as the design task doesn't make it a good benchmark. No statistics is provided for how good the GP model for GFP is.  As for other proteins, GB1 for instance is far more completely surveyed than GFP.  There are also better published models of GFP available (e.g. TAPE, UniRep). All of these would of course struggle with out-of-domain samples. Why not use a physical simulator like Rosetta that wouldn't have this issue? I believe when designing a benchmarking suite, these decisions should be considered more carefully, as it would quickly become the test bed for follow up work and a flawed design choice here amplifies in future work. \n\n(3) A body of literature for algorithms that can readily perform MBO has been neglected.  It is trivial to run a regular optimization algorithm with the model (instead of ground truth) and compare the proposed solutions to ground truth.  Quality-Diversity/EDA algorithms (e.g. genetic algorithms,  simulated annealing,  CMA-ES, or even pure CEM (rather than DbAS)...) consistently perform \"well\" in these high-dimensional optimization settings.  The success of the gradient-based method gives more reason to believe representatives of each of these classical approaches should be included and suggests that the claim that climbing proxy model will necessarily result in bad \"ground truth\" outcomes is a weak one. \n\n(4) For a benchmarking library like this, there needs to be mature code available for review (not submitted). I've checked the provided website multiple times, and while it is under active development, the code is not accessible. From what I gather the current code interface simply gives access to some data points and a ground truth. This is too little API. A good benchmarking tool would let the user abstract away the modeling part easily, and be able to readily port and run their algorithm against benchmarks, producing the results in the same way. It should also take care of running sanity checks/tests for the user and generating the same plots as those in the paper. \n\n(5) The fact that gradient-based methods outperform other methods presented here is only surprising in the sense that they were not included in the original papers (i.e. why weren't gradient-based methods benchmarked there? not this paper's fault of course).  The authors express  a general conviction where gradient based methods have done very poorly in other attempts for MBO, but provide no references, it would be great to cite relevant references for this claim.  \n\n(6) As is, the paper/library only compares CbAS/DbAS with MINs and hill-climbing methods. I think it is not sufficient breadth of methods to make a \"benchmarking\" suite. As far as I can tell, CbAS/DbAS and MINs are not the best published algorithms in any of the domains suggested, so the authors should justify why they are the algorithms to benchmark against?\n\nFor instance, Angermueller et al 2020 ICLR, have an offline RL algorithm that can in principle solve all of these problems. In fact DynaPPO, PPO, and Ensemble-based Bayesian optimization all outperform CbAS in that study. Some sequence design challenges used there seem to be better benchmarks than GFP.  There is substantive work in molecular design on MBO,  but none of the SOTA algorithms are included (e.g the now-classic Gomez-Bombarelli 2016, or  perhaps adaptation of Zhou et al 2019 Scientific Reports). A good rule of thumb in my view is to include the SOTA or well-established algorithm for each task category. \n\n(6) I suggest the authors think carefully about what the evaluation criteria are. While optimization itself is a good metric, other factors, such as providing a way of evaluating diversity of solutions, or sensitivity to dataset size (e.g. by subsampling), are good to consider.\n\n==========\n\nI would like to encourage the authors to continue the pursuit of this work because it is relevant and well-motivated, and has great potential, in my view it is simply not ready. I think this work needs to be reviewed again when it is more mature,  with wider range of algorithms, better justified challenges, a larger set of metrics that can be easily collected, and available code such that the reviewer can vet the benchmarks and code properly. Right now, it's a comparative review of a small set of methods, not a good benchmarking suite.\n\nIf done well, it can be a very useful suite that can help researchers develop better algorithms. The danger of accepting it prematurely is that it will be a basis for future work that \"game-ify\" studies of algorithms against irrelevant/misleading set of benchmarks. That is only damaging to the development of good algorithms and could misguide research. As it stands,  I find the latter risk higher than it's contribution, and hence I believe it should be rejected and reviewed once more of these structural issues are addressed (and code is available to review).\n\n\n~~~~~~~~~~~~~~\n\nPost review and discussion remarks:\nI think the authors have improved the paper significantly during the review period. However, three of my main concerns about the paper remain to a degree that I'm not confident about the paper's value (or risk of misleading followups). (1) That the set of challenges is somewhat arbitrary, some tasks are using \"real\" ground truths while others are simply running on known trained oracles.  (2) That implementation of strong offline RL benchmark algorithms are missing (because they don't exactly apply across domains) even though they can always be applied in this setting even if \"exact\" conditions are not applied in every case (just like Gradient Ascent or BO were) (3) That the API needs to offer more for this to be a good benchmarking suite. \n\nI've been most concerned about 2 and 3, and after reading the code, I find that it is still too \"bare bones\" to be a good package. I looked back at OpenAI gym, and there are several abstractions that they make, including actions, observations, environments, spaces that help the implementer unify how they deal with the complexity underneath. \n\nSo far as I can tell, most of what design-bench does is load a csv matrix into an task.x, task.score(x) , and also lets the user access some approximate oracle task.y. This are critical to the process but their abstraction as related to the paper are not clear to me at this time. How is task.y computed, how is the ground truth actually representative of reality. How does optimization depend on the choice of oracle for task.y? \n\nHaving read the code, useful elements are in there to make for a good package, I feel like it needs improvements and another review for scientific soundness.\n\nI've updated my score to address the improvements made. The paper scores somewhere between a 4 and a 5 for me.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark suite of offline model-based optimization problems. This benchmark includes diverse and realistic tasks derived from real-world problems in biology, material science, and robotics contains a wide variety of domains, and it covers both continuous and discrete, low and high dimensional design spaces. The authors provide a comprehensive evaluation of existing methods under identical assumptions and get several interesting takeaways from the results. They found there exists surprising efficacy of simple baselines such as naive gradient ascent, which suggests the need for careful tuning and standardization of methods in this area, and provides a test bed for algorithms that try to solve this challenge. However, most reviewers agreed that a more in-depth analysis and insightful explorations for the RL experiment results will help readers understand why their method has superiority even without trajectory data, and  that the paper needs another revision before being accepted. Therefore, I recommend rejection although all reviewers agreed that the tasks is very interesting and a good start.
Decision: Reject