Package: design-bench
Design-BenchDesign-Bench is abenchmarking frameworkfor solving automatic design problems that involve choosing an input that maximizes a black-box function. This type of optimization is used across scientific and engineering disciplines in ways such as designing proteins and DNA sequences with particular functions, chemical formulas and molecule substructures, the morphology and controllers of robots, and many more applications.These applications have significant potential to accelerate research in biochemistry, chemical engineering, materials science, robotics and many other disciplines. We hope this framework serves as a robust platform to drive these applications and create widespread excitement for model-based optimization.Offline Model-Based OptimizationThe goal of model-based optimization is to find an inputxthat maximizes an unknown black-box functionf. This function is frequently difficulty or costly to evaluate---such as requiring wet-lab experiments in the case of protein design. In these cases,fis described by a set of function evaluations: D = {(x_0, y_0), (x_1, y_1), ... (x_n, y_n)}, and optimization is performed without queryingfon new data points.InstallationDesign-Bench can be installed with the complete set of benchmarks via our pip package.condainstallrdkit-cconda-forge
pipinstalldesign-bench[all]==2.0.20
pipinstallmorphing-agents==1.5.1Alternatively, if you do not have MuJoCo, you may opt for a minimal install.pipinstalldesign-bench==2.0.20Available TasksIn the below table, we list the supported datasets and objective functions for model-based optimization, where a ✅ indicates that a particular combination has been tested and is available for download from our server.Dataset \ OracleExactGaussian ProcessRandom ForestFully ConnectedLSTMResNetTransformerTF Bind 8✅✅✅✅✅✅✅GFP✅✅✅✅✅✅ChEMBL✅✅✅✅✅✅UTR✅✅✅✅✅✅Hopper Controller✅✅✅✅Superconductor✅✅✅Ant Morphology✅✅✅✅D'Kitty Morphology✅✅✅✅Combinations of datasets and oracles that are not available for download from our server are automatically trained on your machine on task creation. This currently only affects approximate oracles on user-defined MBO tasks. Below we provide the preferred oracle for each task, as well as meta data such as the number of data points measured.Task NameDatasetOracleDataset SizeSpearman's ρTFBind8-Exact-v0TF Bind 8Exact65792GFP-Transformer-v0GFPTransformer560860.8497ChEMBL-ResNet-v0ChEMBLResNet405160.3208UTR-ResNet-v0UTRResNet2800000.8617HopperController-Exact-v0Hopper ControllerExact3200Superconductor-RandomForest-v0SuperconductorRandom Forest212630.9155AntMorphology-Exact-v0Ant MorphologyExact25009DKittyMorphology-Exact-v0D'Kitty MorphologyExact25009Performance Of BaselinesWe benchmark a set of 9 methods for solving offline model-based optimization problems. Performance is reported in normalized form, where the 100th percentile score of 128 candidate designs is evaluated and normalized such that a 1.0 corresponds to performance equivalent to the best performing design in thefull unobserveddataset assoctated with each model-based optimization task. A 0.0 corresponds to performance equivalent to the worst performing design in thefull unobserveddataset. In circumstances where an exact oracle is not available, thisfull unobserveddataset is used for training the approximate oracle that is used for evaluation of candidate designs proposed by each method. The symbol ± indicates the empirical standard deviation of reported performance across 8 trials.Method \ TaskGFPTF Bind 8UTRChEMBLAuto. CbAS0.865 ± 0.0000.910 ± 0.0440.650 ± 0.0060.470 ± 0.000CbAS0.865 ± 0.0000.927 ± 0.0510.650 ± 0.0020.517 ± 0.055BO-qEI0.254 ± 0.3520.798 ± 0.0830.659 ± 0.0000.333 ± 0.035CMA-ES0.054 ± 0.0020.953 ± 0.0220.666 ± 0.0040.350 ± 0.017Grad.0.864 ± 0.0010.977 ± 0.0250.639 ± 0.0090.360 ± 0.029Grad. Min0.864 ± 0.0000.984 ± 0.0120.647 ± 0.0070.361 ± 0.004Grad. Mean0.864 ± 0.0000.986 ± 0.0120.647 ± 0.0050.373 ± 0.013MINs0.865 ± 0.0010.905 ± 0.0520.649 ± 0.0040.473 ± 0.057REINFORCE0.865 ± 0.0000.948 ± 0.0280.646 ± 0.0050.459 ± 0.036Performance On Discrete Tasks.Method \ TaskSuperconductorAnt MorphologyD'Kitty MorphologyHopper ControllerAuto. CbAS0.421 ± 0.0450.884 ± 0.0460.906 ± 0.0060.137 ± 0.005CbAS0.503 ± 0.0690.879 ± 0.0320.892 ± 0.0080.141 ± 0.012BO-qEI0.402 ± 0.0340.820 ± 0.0000.896 ± 0.0000.550 ± 0.118CMA-ES0.465 ± 0.0241.219 ± 0.7380.724 ± 0.0010.604 ± 0.215Grad.0.518 ± 0.0240.291 ± 0.0230.874 ± 0.0221.035 ± 0.482Grad. Min0.506 ± 0.0090.478 ± 0.0640.889 ± 0.0111.391 ± 0.589Grad. Mean0.499 ± 0.0170.444 ± 0.0810.892 ± 0.0111.586 ± 0.454MINs0.469 ± 0.0230.916 ± 0.0360.945 ± 0.0120.424 ± 0.166REINFORCE0.481 ± 0.0130.263 ± 0.0320.562 ± 0.196-0.020 ± 0.067Performance On Continuous Tasks.Reproducing Baseline PerformanceIn order to reproduce this table, you must first install the implementation of the baseline algorithms.gitclonehttps://github.com/brandontrabucco/design-baselines
condaenvcreate-fdesign-baselines/environment.yml
condaactivatedesign-baselinesYou may then run the following series of commands in a bash terminal using the command-line interface exposed in design-baselines. Also, please ensure that the conda environmentdesign-baselinesis activated in the bash session that you run these commands from in order to access thedesign-baselinescommand-line interface.# set up machine parametersNUM_CPUS=32NUM_GPUS=8forTASK_NAMEin\gfp\tf-bind-8\utr\chembl\superconductor\ant\dkitty\hopper;doforALGORITHM_NAMEin\autofocused-cbas\cbas\bo-qei\cma-es\gradient-ascent\gradient-ascent-min-ensemble\gradient-ascent-mean-ensemble\mins\reinforce;do# launch several model-based optimization algorithms using the command line interface# for example:# (design-baselines) name@computer:~/$ cbas gfp \#                                        --local-dir ~/db-results/cbas-gfp \#                                        --cpus 32 \#                                        --gpus 8 \#                                        --num-parallel 8 \#                                        --num-samples 8$ALGORITHM_NAME$TASK_NAME\--local-dir~/db-results/$ALGORITHM_NAME-$TASK_NAME\--cpus$NUM_CPUS\--gpus$NUM_GPUS\--num-parallel8\--num-samples8donedone# generate the main performance table of the paperdesign-baselinesmake-table--dir~/db-results/--percentile100th# generate the performance tables in the appendixdesign-baselinesmake-table--dir~/db-results/--percentile50th
design-baselinesmake-table--dir~/db-results/--percentile100th--no-normalizeThese commands will run several model-based optimization algorithms (such asCbAS) contained in design-baselines on all tasks released with the design-bench benchmark, and will then generate three performance tables from those results, and print a latex rendition of these performance tables to stdout.The Train-Test DiscrepencyFor tasks where an exact numerical ground truth is not available for evaluating the performance of previously unseen candidate designs, we provide several families of approximate oracle models that have been trained using a largerheld outdataset of designs x and corresponding scores y.Using a learned oracle for evaluation and training an MBO method using real data creates a train-test discrepency. This discrepency can be avoided byrelabellingthe y values in an offline MBO dataset with the predictions of the learned oracle, which is controlled by the following parameter when building a task.importdesign_bench# instantiate the task using y values generated from the learned oracletask=design_bench.make('GFP-Transformer-v0',relabel=True)# instantiate the task using y values generated from real experimentstask=design_bench.make('GFP-Transformer-v0',relabel=False)Task APIDesign-Bench tasks share a common interface specified indesign_bench/task.py, which exposes a set of input designstask.xand a set of output predictionstask.y. In addition, the performance of a new set of input designs (such as those output from a model-based optimization algorithm) can be found usingy = task.predict(x).importdesign_benchtask=design_bench.make('TFBind8-Exact-v0')defsolve_optimization_problem(x0,y0):returnx0# solve a model-based optimization problem# solve for the best input x_star and evaluate itx_star=solve_optimization_problem(task.x,task.y)y_star=task.predict(x_star)Many datasets of interest to practitioners are too large to load in memory all at once, and so the task interface defines an several iterables that load samples from the dataset incrementally.importdesign_benchtask=design_bench.make('TFBind8-Exact-v0')forx,yintask:pass# train a model hereforx,yintask.iterate_batches(32):pass# train a model hereforx,yintask.iterate_samples():pass# train a model hereCertain optimization algorithms require a particular input format, and so tasks support normalization of bothtask.xandtask.y, as well as conversion oftask.xfrom discrete tokens to the logits of a categorical probability distribution---needed when optimizingxwith a gradient-based model-based optimization algorithm.importdesign_benchtask=design_bench.make('TFBind8-Exact-v0')# convert x to logits of a categorical probability distributiontask.map_to_logits()discrete_x=task.to_integers(task.x)# normalize the inputs to have zero mean and unit variancetask.map_normalize_x()original_x=task.denormalize_x(task.x)# normalize the outputs to have zero mean and unit variancetask.map_normalize_y()original_y=task.denormalize_y(task.y)# remove the normalization applied to the outputstask.map_denormalize_y()normalized_y=task.normalize_y(task.y)# remove the normalization applied to the inputstask.map_denormalize_x()normalized_x=task.normalize_x(task.x)# convert x back to integerstask.map_to_integers()continuous_x=task.to_logits(task.x)Each task provides access to the model-based optimization dataset used to learn the oracle (where applicable) as well as the oracle itself, which includes metadata for how it was trained (where applicable). These provide fine-grain control over the data distribution for model-based optimization.importdesign_benchtask=design_bench.make('GFP-GP-v0')# an instance of the DatasetBuilder class from design_bench.datasets.dataset_builderdataset=task.dataset# modify the distribution of the task datasetdataset.subsample(max_samples=10000,distribution="uniform",min_percentile=10,max_percentile=90)# an instance of the OracleBuilder class from design_bench.oracles.oracle_builderoracle=task.oracle# check how the model was fitprint(oracle.params["rank_correlation"],oracle.params["model_kwargs"],oracle.params["split_kwargs"])Dataset APIDatasets provide a model-based optimization algorithm with information about the black-box function, and are used in design bench to fit approximate oracle models when an exact oracle is not available. All datasets inherit from the DatasetBuilder class defined indesign_bench.datasets.dataset_builder.All datasets implement methods for modifying the format and distribution of the dataset, including normalization, subsampling, relabelling the outputs, and (for discrete datasets) converting discrete inputs to real-valued. There are also special methods for splitting the dataset into a training and validation set.Display code snippetfromdesign_bench.datasets.discrete.gfp_datasetimportGFPDatasetdataset=GFPDataset()# convert x to logits of a categorical probability distributiondataset.map_to_logits()discrete_x=dataset.to_integers(dataset.x)# normalize the inputs to have zero mean and unit variancedataset.map_normalize_x()original_x=dataset.denormalize_x(dataset.x)# normalize the outputs to have zero mean and unit variancedataset.map_normalize_y()original_y=dataset.denormalize_y(dataset.y)# remove the normalization applied to the outputsdataset.map_denormalize_y()normalized_y=dataset.normalize_y(dataset.y)# remove the normalization applied to the inputsdataset.map_denormalize_x()normalized_x=dataset.normalize_x(dataset.x)# convert x back to integersdataset.map_to_integers()continuous_x=dataset.to_logits(dataset.x)# modify the distribution of the datasetdataset.subsample(max_samples=10000,distribution="uniform",min_percentile=10,max_percentile=90)# change the outputs as a function of their old valuesdataset.relabel(lambdax,y:y**2-2.0*y)# split the dataset into a validation settraining,validation=dataset.split(val_fraction=0.1)If you would like to define your own dataset for use with design-bench, you can directly instantiate a continuous dataset or a discrete dataset depending on the input format you are using. The DiscreteDataset class and ContinuousDataset are built with this in mind, and accept both two numpy arrays containing inputsxoutputsy.Display code snippetfromdesign_bench.datasets.discrete_datasetimportDiscreteDatasetfromdesign_bench.datasets.continuous_datasetimportContinuousDatasetimportnumpyasnp# create dummy inputs and outputs for model-based optimizationx=np.random.randint(500,size=(5000,43))y=np.random.uniform(size=(5000,1))# create a discrete dataset for those inputs and outputsdataset=DiscreteDataset(x,y)# create dummy inputs and outputs for model-based optimizationx=np.random.uniform(size=(5000,871))y=np.random.uniform(size=(5000,1))# create a continuous dataset for those inputs and outputsdataset=ContinuousDataset(x,y)In the event that you are using a dataset that is saved to a set of sharded numpy files (ending in .npy), you may also create dataset by providing a list of shard files representing using the DiskResource class. The DiscreteDataset class and ContinuousDataset accept two lists of sharded inputsxand outputsyrepresented by DiskResource objects.Display code snippetfromdesign_bench.disk_resourceimportDiskResourcefromdesign_bench.datasets.discrete_datasetimportDiscreteDatasetfromdesign_bench.datasets.continuous_datasetimportContinuousDatasetimportosimportnumpyasnp# list the disk resource for each shardos.makedirs("new_dataset/")x=[DiskResource("new_dataset/shard-x-0.npy"),DiskResource("new_dataset/shard-x-1.npy")]y=[DiskResource("new_dataset/shard-y-0.npy"),DiskResource("new_dataset/shard-y-1.npy")]# create dummy inputs and outputs for model-based optimizationxs=np.random.randint(500,size=(5000,43))ys=np.random.uniform(size=(5000,1))# save the dataset to a set of shard filesnp.save("new_dataset/shard-x-0.npy",xs[:3000])np.save("new_dataset/shard-x-1.npy",xs[3000:])np.save("new_dataset/shard-y-0.npy",ys[:3000])np.save("new_dataset/shard-y-1.npy",ys[3000:])# create a discrete dataset for those inputs and outputsdataset=DiscreteDataset(x,y)# create dummy inputs and outputs for model-based optimizationxs=np.random.uniform(size=(5000,871))ys=np.random.uniform(size=(5000,1))# save the dataset to a set of shard filesnp.save("new_dataset/shard-x-0.npy",xs[:3000])np.save("new_dataset/shard-x-1.npy",xs[3000:])np.save("new_dataset/shard-y-0.npy",ys[:3000])np.save("new_dataset/shard-y-1.npy",ys[3000:])# create a continuous dataset for those inputs and outputsdataset=ContinuousDataset(x,y)Oracle APIOracles provide a way of measuring the performance of candidate solutions to a model-based optimization problem, found by a model-based optimization algorithm, without having to perform additional real-world experiments. To this end, oracle implement a prediction functionoracle.predict(x)that takes a set of designs and makes a prediction about their performance. The goal of model-based optimization is to maximize the predictions of the oracle.Display code snippetfromdesign_bench.datasets.discrete.gfp_datasetimportGFPDatasetfromdesign_bench.oracles.tensorflowimportTransformerOracle# create a dataset and a noisy oracledataset=GFPDataset()oracle=TransformerOracle(dataset,noise_std=0.1)defsolve_optimization_problem(x0,y0):returnx0# solve a model-based optimization problem# evaluate the performance of the solution x_starx_star=solve_optimization_problem(dataset.x,dataset.y)y_star=oracle.predict(x_star)In order to handle when an exact ground truth is unknown or not tractable to evaluate, Design-Bench provides a set of approximate oracles including a Gaussian Process, Random Forest, and several deep neural network architectures specialized to particular data modalities. These approximate oracles may have the following parameters.Display code snippetfromdesign_bench.datasets.discrete.gfp_datasetimportGFPDatasetfromdesign_bench.oracles.tensorflowimportTransformerOracle# parameters for the transformer architecturemodel_kwargs=dict(hidden_size=64,feed_forward_size=256,activation='relu',num_heads=2,num_blocks=4,epochs=20,shuffle_buffer=60000,learning_rate=0.0001,dropout_rate=0.1)# parameters for building the validation setsplit_kwargs=dict(val_fraction=0.1,subset=None,shard_size=5000,to_disk=True,disk_target="gfp/split",is_absolute=False)# create a transformer oracle for the GFP datasetdataset=GFPDataset()oracle=TransformerOracle(dataset,noise_std=0.0,# parameters for ApproximateOracle subclassesdisk_target="new_model.zip",is_absolute=True,fit=True,max_samples=None,distribution=None,max_percentile=100,min_percentile=0,model_kwargs=model_kwargs,split_kwargs=split_kwargs)defsolve_optimization_problem(x0,y0):returnx0# solve a model-based optimization problem# evaluate the performance of the solution x_starx_star=solve_optimization_problem(dataset.x,dataset.y)y_star=oracle.predict(x_star)Defining New MBO TasksNew model-based optimization tasks are simple to create and register with design-bench. By subclassing either DiscreteDataset or ContinuousDataset, and providing either a pair of numpy arrays containing inputs and outputs, or a pair of lists of DiskResource shards containing inputs and outputs, you can define your own model-based optimization dataset class. Once a custom dataset class is created, you can register it as a model-based optimization task by choosing an appropriate oracle type, and making a call to the register function. After doing so, subsequent calls todesign_bench.makecan find your newly registered model-based optimization task.Display code snippetfromdesign_bench.datasets.continuous_datasetimportContinuousDatasetimportdesign_benchimportnumpyasnp# define a custom dataset subclass of ContinuousDatasetclassQuadraticDataset(ContinuousDataset):def__init__(self,**kwargs):# define a set of inputs and outputs of a quadratic functionx=np.random.normal(0.0,1.0,(5000,7))y=(x**2).sum(keepdims=True)# pass inputs and outputs to the base classsuper(QuadraticDataset,self).__init__(x,y,**kwargs)# parameters used for building the validation setsplit_kwargs=dict(val_fraction=0.1,subset=None,shard_size=5000,to_disk=True,disk_target="quadratic/split",is_absolute=True)# parameters used for building the modelmodel_kwargs=dict(hidden_size=512,activation='relu',num_layers=2,epochs=5,shuffle_buffer=5000,learning_rate=0.001)# keyword arguments for building the datasetdataset_kwargs=dict(max_samples=None,distribution=None,max_percentile=80,min_percentile=0)# keyword arguments for training FullyConnected oracleoracle_kwargs=dict(noise_std=0.0,max_samples=None,distribution=None,max_percentile=100,min_percentile=0,split_kwargs=split_kwargs,model_kwargs=model_kwargs)# register the new dataset with design_benchdesign_bench.register('Quadratic-FullyConnected-v0',QuadraticDataset,'design_bench.oracles.tensorflow:FullyConnectedOracle',dataset_kwargs=dataset_kwargs,oracle_kwargs=oracle_kwargs)# build the new task (and train a model)task=design_bench.make("Quadratic-FullyConnected-v0")defsolve_optimization_problem(x0,y0):returnx0# solve a model-based optimization problem# evaluate the performance of the solution x_starx_star=solve_optimization_problem(task.x,task.y)y_star=task.predict(x_star)CitationThanks for using our benchmark, and please cite our paper!@misc{
    trabucco2021designbench,
    title={Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization},
    author={Brandon Trabucco and Aviral Kumar and Xinyang Geng and Sergey Levine},
    year={2021},
    url={https://openreview.net/forum?id=cQzf26aA3vM}
}