Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization

1 INTRODUCTION . Automatically synthesizing designs that maximize a desired objective function is one of the most important problems in many scientific and engineering domains . From protein design in molecular biology Shen et al . ( 2014 ) to superconducting material discovery in physics Hamidieh ( 2018 ) , researchers have made significant progress in applying machine learning methods to such optimization problems over structured design spaces . Commonly , the exact form of the objective function is unknown , and the objective values for a novel design can only be evaluated by running either computer simulations or physical experiments in the real world . The process of optimizing an unknown function is known as black-box optimization , and is typically solved in an online iterative manner , where in each iteration the solver proposes new designs and query the objective function for feedback in order to propose better design in the next iteration Williams & Rasmussen ( 2006 ) . In many domains however , the evaluation of the objective function is prohibitively expensive , because it requires manually conducting experiments in the real world . In this setting , one can not simply query the true objective function to gradually improve the design . Instead , a collection of past records of designs and their corresponding objective values might be available , and therefore the optimization method must leverage the available data to synthesize the most optimal design possible . This is the setting of data-driven offline model-based optimization . Although online black-box optimization has been studied extensively , the offline MBO problem has received comparatively less attention , and only a small number of recent works study offline MBO in the setting with high-dimensional design spaces , where they utilize deep learning techniques Brookes et al . ( 2019 ) ; Kumar & Levine ( 2019 ) ; Fannjiang & Listgarten ( 2020 ) . This is partly due to the fact that methods for online design optimization can not be easily applied in the offline MBO setting . However , even with only a few existing methods , it is still hard to compare and track the progress in this field , as these methods are proposed and evaluated on different tasks with distinct evaluation protocols . To the best of our knowledge , there is no commonly adopted set of benchmarks for offline MBO . To address this problem , in this paper we introduce a suite of offline MBO benchmarks with standardized evaluation protocols . We include a realistic and diverse set of tasks that spans a wide range of application domains , from synthetic biology to robotics . The realism and diversity of the tasks is essential for the evaluation of offline data-driven model-based optimization methods , as it measures the generality of the methods being evaluated across multiple domains and verifies that they are not overfitting to a single task . Our benchmark tasks incorporate a variety of challenging factors , such as high dimensional input spaces and sensitive discontinuous objective functions , which help better identify the strengths and weaknesses of MBO methods . Along with the benchmark suite , we also present reference implementations of a number of existing offline MBO methods and baselines . We systematically evaluate them on all of the proposed benchmark tasks and report our findings . A surprising discovery from our findings is that with proper data normalization , the simple baseline method of learning an objective value predictor and performing gradient ascent on its input outperforms several prior MBO methods in our benchmark . We hope that our work can provide insight into the current progress of offline MBO methods and can also serve as a meaningful metric to galvanize research in this area . 2 OFFLINE MODEL-BASED OPTIMIZATION PROBLEM STATEMENT . The goal in offline model-based optimization is to optimize an unknown ( possibly stochastic ) objective function f ( x ) , provided access to a static datasetD = { ( xi , yi ) } of designs xi and a corresponding measurement of the objective value yi . Similar to batch Bayesian optimization ( BBO ) González et al . ( 2016 ) , each algorithm A is allowed to consume the dataset D , and is required to produce a set of K candidate designs A ( D , K ) = { x∗i : i ∈ { 1 ... K } } . These K candidates are evaluated under the ground truth objective function f ( x ) , and the best performing design is reported as the final performance value . Abstractly , the objective for offline MBO is : arg max A [ P ( { f ( x∗ ) : x∗ ∈ A ( D , K ) } , N ) ] , ( 1 ) where P denotes the percentile function . Intuitively , this formulation ranks an offline MBO algorithm using the N th percentile objective value obtained by it given a fixed evaluation budget of K−samples . Common choices ofN are 100 , which represent the max objective value , and 50 which represents the median objective value among the candidates . What makes offline MBO especially challenging ? The offline nature of the problem requires that the algorithm A not be tuned by peeking into the ground truth objective f , and this makes the offline MBO problem much more difficult than the online design optimization problem . One simple idea to tackle this problem is to learn an objective proxy using the dataset , and then convert this offline MBO problem into an online problem by treating the learned objective proxy as the true objective . However , this idea may not work well , due to the intrinsic out-of-distribution nature of optimal designs . First of all , in a number of practical MBO problems such as optimization over proteins or robot morphologies , the designs with highest objective values in the dataset already lie on the tail of the dataset distribution , since they are better than most other designs . In order to improve upon the best designs , an optimization method needs to produce designs that are even further away from the dataset distribution . For such out-of-distribution designs , is would be impossible to guarantee that the learned objective proxy is accurate , and hence any powerful optimization method would easily “ exploit ” the learned objective proxy and produce falsely promising designs that are drastically overestimated by the learned objective proxy . This conflict between the out-of-distribution nature of optimization and the in-distribution requirement of any learned model is indeed the core challenge of offline MBO . This challenge is often exacerbated in real-world problems by the high dimensionality of the design space and the sparsity of the available data , as we will show in our benchmark . A good offline MBO method needs to carefully balance the two sides of the conflict , producing optimized designs that are not too far from the data distribution . 3 RELATED WORK . While the extensive prior work on MBO , particularly Bayesian optimization , has made tremendous progress both in terms of enabling intelligent active selection of query points Kolda et al . ( 2003 ) ; Lizotte ( 2008 ) ; Rubinstein & Kroese ( 2013 ) ; Powell ( 1998 ) ; Whitley ( 1994 ) and scaling up the dimensionality of Bayesian optimization methods Snoek et al . ( 2012 ; 2015 ) ; Shahriari et al . ( 2015 ) , a significant number of real-world problems may be approached more naturally as offline MBO problems , given the availability of several datasets which contain designs annotated with their corresponding score values . Offline MBO presents a new set up challenges , as we discuss below , and we believe that these challenges require a new set of benchmarks that emphasize offline data and high-dimensional design spaces . Researchers working on either online active design optimization or offline model-based optimization have collected various datasets of designs that can be used to build tasks for offline MBO . Sarkisyan et al . ( 2016 ) analyze the fluorescence of GFP proteins under blue and ultraviolet light , and Brookes et al . ( 2019 ) use this dataset for optimization to find the protein with the highest fluorescence value . ChEMBL Gaulton et al . ( 2012 ) provides a dataset for drug discovery , where molecule activities are measured against a target assay . Hamidieh ( 2018 ) analyze the critical temperatures for superconductors and provide a dataset to search for room-temperature superconductors with potential in the construction of quantum computers . Some of these datasets have already been employed in the study of offline MBO methods Kumar & Levine ( 2019 ) ; Brookes et al . ( 2019 ) ; Fannjiang & Listgarten ( 2020 ) . However , these studies use different set of tasks and their evaluation protocols are highly field-specific , making it difficult to use for general algorithm development . In our benchmark , we incorporate modified variants of some of these datasets along with our own tasks and provide a standardized evaluation protocol . We hope that these tasks can represent realistic MBO problems across a wide range of domains and that the standardized evaluation protocol can facilitate development of new and more powerful offline MBO algorithms . Recently , researchers have proposed several methods specifically for the completely offline MBO problem . To make up for the fact that the true objective function is not available , these methods often make use of some type of learned objective function as a proxy for the real one and a generative model Kingma & Welling ( 2013 ) ; Goodfellow et al . ( 2014 ) to capture the distribution of valid designs , especially for high dimensional design spaces . Kumar & Levine ( 2019 ) learns a objective value conditioned generative model to synthesize the designs with the desired objective value . Brookes et al . ( 2019 ) ; Fannjiang & Listgarten ( 2020 ) employ a variational autoencoder Kingma & Welling ( 2013 ) to capture the distribution of designs in the dataset and a learned neural network objective function model . In our benchmark , we provide open-source reference implementations for these methods and systematically evaluate them in all our tasks . 4 DESIGN-BENCH BENCHMARK TASKS . In this section , we describe the set of tasks included in our benchmark . We first provide an overview of the tasks in Table 1 . Each task in our benchmark suite comes with a dataset D = { ( xi , yi ) } , along with an oracle objective function f which is to be utilized for for test time evaluation only . An offline MBO algorithm should not query the oracle during training time , even for hyperparameter tuning . While in some of the tasks in our benchmark , such as tasks pertaining to robotics – HopperController-v0 , DKittyMorphology-v0 and AntMorphology-v0 , the oracle functions are evaluated by running computer simulations to obtain the true objective values , in the other tasks , the true objective values can only be obtained by conducting expensive physical experiments . While the eventual aim of offline MBO is to make it possible to optimize designs in precisely such settings where querying the groundtruth objective is challenging , requiring real physical experiments for evaluation makes the design and benchmarking of new algorithms difficult and time consuming . Therefore , to facilitate evaluation , following evaluation methodology in prior work Brookes et al . ( 2019 ) ; Fannjiang & Listgarten ( 2020 ) , and use models built by experts as our oracle function . Such models are typically also learned models , but with representations that are hand-designed , built-in domain-specific inductive biases and typically trained on much more data than is made available for solving the offline MBO problem , which increases the chance that this proxy “ true function ” can answer queries outside of the MBO training distribution . While this approach to evaluation diminishes the realism of our benchmark , since these proxy “ true functions ” may not always be accurate , we believe that this trade off is worth it to make benchmarking practical . The main purpose of our benchmark is to facilitate the evaluation and development of offline MBO algorithms , and we believe that it is important to include tasks in domains where the true objective values can only be obtained via physical experiments , which make up a large portion of the real-world MBO problems . For each task we discuss in this section , we provide a detailed description of the data collection strategy and the data pre-processing strategy in Appendix A. GFP-v0 and GFP-v1 : protein fluorescence maximization . The goal of this task is to design a derivative protein from the Aequorea victoria green fluorescent protein ( GFP ) that has maximum fluorescence , using a real-world dataset mapping proteins to fluorescence collected by Sarkisyan et al . ( 2016 ) . While we can not exactly evaluate any novel protein , we employ an expert Gaussian process regression model with a protein-specific kernel built by Shen et al . ( 2014 ) as the oracle function , following the convention of Brookes et al . ( 2019 ) ; Fannjiang & Listgarten ( 2020 ) . This expert model is built on a larger dataset than the one we use in this benchmark , making it relatively accurate for proteins not in our dataset . The design space for GFP is discrete , consisting of a sequence of 238 categorical variables that can take one of 20 or 30 options , which corresponds to the choice of 238 amino acids that make up the green fluorescent protein from 20 types of amino acid . In brief , GFP-v0 uses a Gaussian process as the expert model , while GFP-v1 adapts the 12-layer Transformer provided by the TAPE framework as the expert model . More details are in Appendix A. MoleculeActivity-v0 : drug activity maximization . This task is taken from the domain of drug discovery , where the goal is to design the substructure of a molecule that exhibits high activity when tested against a target assay . The dataset we provide with this task was originally collected by ChEMBL Gaulton et al . ( 2012 ) . Again like for the GFP task , the true molecule activity against a target assay can only be evaluated with physical experiments . Therefore , we leverage an oracle used recently for meta-learning Yao et al . ( 2020 ) and adopt the expert random forest regression model built by Martin et al . ( 2019 ) as the oracle function . From the ChEMBL data , we pick a single target assay to form the dataset of this task , resulting in 4216 molecules in total . The design space for this task is a sequence of 1024 binary categorical variables that correspond to the Morgan radius 2 substructure fingerprints , making this task a high-dimensional task . Superconductor-v0 : critical temperature maximization for superconductor materials . The Superconductor-v0 task is taken from the domain of materials science , where the goal is to design a superconducting material that has a high critical temperature . Developing superconducting materials has immense practical value , especially for room-temperature superconductors , which could greatly facilitate quantum computing . We adapt a real-world dataset used by materials scientists Hamidieh ( 2018 ) . The dataset contains 21263 superconductors annotated with critical temperatures . Prior work has employed this dataset for the study of offline MBO methods Brookes et al . ( 2019 ) , and we follow their convention of using an expert random forest model , detailed in Fannjiang & Listgarten ( 2020 ) , for our oracle function . The design space for Superconductor-v0 is an 81-channel vector of continuous variables , which describe properties of the material such as its atomic radius . HopperController-v0 : robot neural network controller optimization . HopperController-v0 is a task created by us in the domain of robotics , where the goal is to optimize weights of a neural network policy that maximizes expected return on the Hopper-v2 task in OpenAI Gym Brockman et al . ( 2016 ) . While this goal might look similar to that of reinforcement learning , it is important to note that the formulation is entirely different . Unlike reinforcement learning , we don ’ t have access to any form of trajectory data , neither by actively sampling from the environment nor from a pre-collected dataset . Instead , we only have a dataset of weights of a neural network controller and the corresponding return values . Therefore , reinforcement learning methods can not be applied to solve this problem . Unlike tasks introduced before , we evaluate the true objective value of any design during dataset generation by running 1000 steps of simulation in the MuJoCo simulator . This horizon length is chosen so that the learning problem is long-horizon , and difficult . The design space of this task is high-dimensional with 5126 continuous variables corresponding to the flattened weights of a neural network controller . The dataset is collected by training a PPO reinforcement learning agent Schulman et al . ( 2017 ) and recording the agent ’ s weights every 10,000 samples . AntMorphology-v0 and DKittyMorphology-v0 : robot morphology optimization . We created these two tasks to optimize the morphological structure of two simulated robots : Ant from OpenAI Gym Brockman et al . ( 2016 ) and D ’ Kitty from ROBEL Ahn et al . ( 2019 ) . For AntMorphology , the goal is to optimize the morphology of an ant shaped robot , to run as fast as possible , with a pre-trained neural network controller . For DKittyMorphology , the goal is to optimize the morphology of D ’ Kitty , a quadrupedal robot , such that a pre-trained neural network controller can navigate the robot to a fixed location . In this fashion , the goal for both tasks is to recover a morphology with which the pre-trained controller is compatible . The variable morphology parameters of both robots include size , orientation , and location of the limbs , giving us 60 continuous values in total for Ant and 56 for D ’ Kitty . To evaluate the ground truth value for a given design , we run robotic simulation in MuJoCo Todorov et al . ( 2012 ) for 100 time steps , averaging 16 independent trials . These parameters are chosen to reduce stochasticity , and allow the simulator to run in a minimal amount of time .

Summary: This paper focuses on model-based black box optimization problems in the offline setting. These are settings where access to ground truth is expensive, and instead the optimizer has access only to a trained model of the ground truth based on limited data. While optimizing on this surrogate space, a good optimizer often needs to account for model uncertainty and accuracy degradation. The main aim of the paper is to provide a test bed for algorithms that try to solve this challenge.