Title: Generative Adversarial Model-Based Optimization via Source Critic Regularization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Offline Optimization

Bayesian Optimization

Surrogate Objectives
Abstract: Offline model-based optimization seeks to optimize against a learned surrogate model without querying the true oracle objective function during optimization. Such tasks are commonly encountered in protein design, robotics, and clinical medicine where evaluating the oracle function is prohibitively expensive. However, inaccurate surrogate model predictions are frequently encountered along offline optimization trajectories. To address this limitation, we propose *generative adversarial model-based optimization* using **adaptive source critic regularization (aSCR)**—a task- and optimizer- agnostic framework for constraining the optimization trajectory to regions of the design space where the surrogate function is reliable. We propose a computationally tractable algorithm to dynamically adjust the strength of this constraint, and show how leveraging aSCR with standard Bayesian optimization outperforms existing methods on a suite of offline generative design tasks. Our code is available at https://github.com/michael-s-yao/gabo.

Generative Adversarial Model-Based Optimization via Source Critic Regularization

Abstract

Offline model-based optimization seeks to optimize against a learned surrogate model without querying the true oracle objective function during optimization. Such tasks are commonly encountered in protein design, robotics, and clinical medicine where evaluating the oracle function is prohibitively expensive. However, inaccurate surrogate model predictions are frequently encountered along offline optimization trajectories. To address this limitation, we propose generative adversarial model-based optimization using adaptive source critic regularization (aSCR) -a task- and optimizer- agnostic framework for constraining the optimization trajectory to regions of the design space where the surrogate function is reliable. We propose a computationally tractable algorithm to dynamically adjust the strength of this constraint, and show how leveraging aSCR with standard Bayesian optimization outperforms existing methods on a suite of offline generative design tasks. Our code is available at https://github.com/michael-s-yao/gabo.

1 Introduction

In many real-world tasks, we often seek to optimize the value of an objective function over some search space of inputs. Such optimization problems span across a wide variety of domains, including molecule and protein design (Guimaraes et al., 2017; Brown et al., 2019; Maus et al., 2022), patient treatment effect estimation (Kim & Bastani, 2021; Berrevoets et al., 2022; Xu & Bastani, 2023), and resource allocation in public policy (Bastani et al., 2021; Ramchandani et al., 2021). A number of algorithms have been explored for online optimization in these domains, including first-order methods, quasi-Newton methods, and Bayesian optimization (Sun et al., 2020).

However, in many situations it may prove difficult or costly to estimate the objective function for any arbitrary input configuration. Evaluating newly proposed molecules requires expensive experimental laboratory setups, and testing multiple drug doses for a single patient can potentially be dangerous. In these scenarios, the allowable budget for objective function queries is prohibitive, thereby limiting the utility of out-of-the-box online policy optimization methods.

To overcome this limitation, recent work has investigated the utility of optimization methods in the offline setting, where we are unable to query the objective function during the optimization

Hamsa Bastani

Osbert Bastani ∗

process and instead only have access to a set of prior observations of inputs and associated objective values; this problem can often be referred to as offline model-based optimization ( MBO ) (Trabucco et al., 2021; Mashkaria et al., 2023). While one may naïvely attempt to learn a surrogate black-box model from the prior observations that approximates the true oracle objective function, such models can suffer from overestimation errors, yielding falsely promising objective estimates for inputs not contained in the offline dataset. As a result, offline optimization against the surrogate objective may yield low-scoring candidate designs according to the true oracle objective function-a key limitation of traditional policy optimization techniques in the offline setting ( Fig. 1 ).

In this work, we propose a novel offline MBO algorithm that leverages source critic models to optimize a surrogate objective while simultaneously remaining in-distribution when compared against a reference offline dataset. In this setting, an optimizer is rewarded for proposing optima that are 'similar' to reference data points, thereby minimizing overestimation error and allowing for more robust oracle function optimization in the offline setting. Inspired by recent work on generative adversarial networks (Goodfellow et al., 2014), we quantify design similarity by proposing a novel method that regularizes a surrogate objective model using a source critic actor, which we call adaptive source critic regularization ( aSCR ). We show how our algorithm can be readily leveraged with optimization methods such as Bayesian optimization (BO) and first-order methods.

Contributions: Our contributions are as follows: (1) We propose a novel approach for MBO that formulates the task as a constrained primal optimization problem, and we show how this framework can be used to solve for the optimal

D

n

=

{

(

x

k

, y

k

)

}

n

k

=1

Oracle Objective

Fitted Surrogate Model

tradeoff between naïvely optimizing against the surrogate model and staying in-distribution relative to the offline dataset. (2) We introduce a computationally tractable method-which we call adaptive source critic regularization (aSCR)-to implement this framework with two popular optimization methods: Bayesian optimization and gradient ascent. (3) We show that compared to prior methods, our proposed algorithm with Bayesian optimization empirically achieves the highest rank of 3.8 (second best is 5.5) on top-1 design evaluation, and highest rank of 3.0 (second best is 4.6) on top-128 design evaluation across a variety of tasks spanning multiple scientific domains.

2 Related Work

Leveraging source critic model feedback for adversarial training of neural networks was popularized by works such as Goodfellow et al. (2014), where a generator and adversarial discriminator 'play' a zero-sum minimax game to train a generative model. However, such discriminators often suffer from mode collapse and training instability in practice. To overcome these limitations, Arjovsky et al. (2017) introduced the Wasserstein generative adversarial net (WGAN), which instead utilizes a source critic that learns to approximate the 1-Wasserstein distance between the generated and training distributions. However, WGANs and similar networks primarily aim to generate samples that look in-distribution from a latent space prior, rather than optimize against an objective function. In our work, we adapt WGAN-inspired source critic models for Wasserstein distance estimation.

Separately in the field of optimization, Brookes et al. (2019) introduced a method for conditioning by adaptive sampling (CbAS) that learns a density model of the input space that is gradually adapted towards the optimal solution. However, such prior works have focused on solving low-dimensional, online optimization tasks (Hansen & Ostermeier, 1996; Brookes et al., 2019). More recently, Trabucco et al. (2021) introduced conservative objective models ( COM ) specifically for offline optimization tasks; however, their method requires directly modifying the parameters of the surrogate function

during optimization, which is not always feasible for any general task. Mashkaria et al. (2023) proposed Black-box Optimization Networks ( BONET ) to learn the dynamics of optimization trajectories using a causally masked transformer model, and Krishnamoorthy et al. (2023) introduced Denoising Diffusion Optimization Models ( DDOM ) to learn the generative process via a diffusion model. Furthermore, Yu et al. (2021) and Chen et al. (2022) describe Robust Model Adaptation ( RoMA ) and Bidirectional learning via infinite-width networks ( BDI ), respectively. RoMA regularizes the gradient of surrogate objective models by enforcing a local smoothness prior at the observed inputs, and BDI learns bidirectional mappings between low- and high- scoring candidates. Finally, Nguyen et al. (2023) introduce Experiment Pretrained Transformers ( ExPT ) to learn a general model for optimization using unsupervised methods. While these recent works and others propose promising algorithms for offline optimization tasks, they are often evaluated using expensive oracle query budgets that are often not achievable in practice-especially for potentially dangerous tasks such as patient care and other high-stakes applications.

3 Background

3.1 Offline Model-Based Optimization

In many real-world domains, we often seek to optimize an oracle objective function f ( x ) over a space of design candidates X to solve for x ∗ = argmax x ∈X f ( x ) . Examples of such problems include optimizing certain desirable properties of molecules in molecular design (Guimaraes et al., 2017; Brown et al., 2019; Maus et al., 2022), and estimating the optimal therapeutic intervention for patient care in clinical medicine (Kim & Bastani, 2021; Berrevoets et al., 2022; Xu & Bastani, 2023). In practice, however, the true objective function f may be costly to compute or even entirely unknown, making it difficult to query in optimizing f ( x ) . Instead, it is often more feasible to obtain access to a reference labeled dataset of observations from nature D n = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } where y i = f ( x i ) . Optimization methods may use a variety of different strategies to leverage D n in the offline setting (Mashkaria et al., 2023; Krishnamoorthy et al., 2023; Chen et al., 2022); one common approach used by Trabucco et al. (2021) and others is to learn a regressor model f θ parametrized by

as a surrogate model for the true oracle objective f ( x ) . Rather than querying the oracle f as in the online setting, we can instead solve the related optimization problem

with the hope that optimizing f θ will also lead to desirable oracle values of f as well. Solving (2) is one instantiation of offline model-based optimization (MBO) for which a number of techniques have been developed, such as gradient ascent and Bayesian optimization ( BO ) (Sun et al., 2020).

Of note, it is difficult to guarantee the reliability of the model's predictions for x / ∈ D n that are almost certainly encountered in the optimization trajectory. Thus, naïvely optimizing the surrogate objective f θ can result in 'optima' that are low-scoring according to the oracle objective f .

3.2 Optimization Over Latent Spaces

In certain cases, the search space X for an optimization task may be discretized over a finite set of structured inputs, such as amino acids for protein sequences or atomic building blocks for molecules. However, many historical optimization algorithms do not generalize well to these settings for a number of different reasons, such as the lack of gradients with respect to the input designs to guide the optimization trajectory. Instead of directly optimizing over X , recent work leverages deep variational autoencoders (VAEs) to first map the input space into a continuous, (often) lower dimensional latent space Z and then performing optimization over Z instead (Tripp et al., 2020; Deshwal & Doppa, 2021; Maus et al., 2022). A VAE is composed of a two components: (1) an encoder with parameters ϕ that learns an approximated posterior distribution q ϕ ( z | x ) for x ∈ X , z ∈ Z ; and (2) a decoder with parameters φ that learns the conditional likelihood distribution p φ ( x | z ) (Kingma & Welling, 2013). The encoder and decoder are co-trained to maximize the evidence lower bound (ELBO)

where D KL is the Kullback-Leibler (KL) divergence and p VAE ( z ) is the prior distribution. A common choice is to set p VAE = N (0 , I ) (i.e., the standard normal distribution). Optimization can then be

performed over the continuous latent space Z of the VAE to propose 'latent space designs' that can be readily decoded using the decoder φ back into the original input space.

One such optimization method over VAE latent spaces is Bayesian optimization (BO) , a sampleefficient framework for solving expensive black-box optimization problems (Mockus, 1982; Osborne et al., 2009; Snoek et al., 2012). While the utility of BO has primarily been explored for expensiveto-evaluate black-box functions in prior literature, recent work has shown that BO also outperforms baseline optimization methods in offline tasks involving models that are relatively inexpensive to evaluate, such as the neural network surrogates used in model-based optimization (MBO). Multiple prior works have shown that BO and related methods consistently outperform both first-order gradientbased and stochastic evolutionary methods (Eriksson et al., 2019b; Maus et al., 2022; Hvarfner et al., 2024; Eriksson & Jankowiak, 2021; Astudillo & Frazier, 2019).

3.3 Wasserstein Metric Between Probability Distributions

The Wasserstein distance is a distance metric between any two probability distributions, and is closely related to problems in optimal transport. We define the p = 1 Wasserstein distance between a reference distribution P and a generated distribution Q using distance metric d ( · , · ) as

where Γ is the set of all couplings between P and Q . For empirical distributions where p n ( q n ) is based on n observations { z ′ j } n j =1 ( { z i } n i =1 ), (4) can be simplified to

where the infimum is over all permutations σ of n elements. Leveraging the Kantorovich-Rubinstein duality theorem (Kantorovich & Rubinstein, 1958), (5) can be equivalently written as

where c ( z ) is a source critic and || c || L is the Lipschitz norm of c ( z ) . In the Wasserstein GAN (WGAN) model proposed by Arjovsky et al. (2017), a generative network and source critic are cotrained in a minimax game where the generator (critic) seeks to minimize (maximize) the Wasserstein distance W 1 between the training and generated distributions. Such an optimization schema enables the generator policy to learn the distribution of training samples from nature.

4 A Framework for Generative Adversarial Optimization

In this section we describe our proposed framework for generative adversarial model-based optimization using adaptive source critic regularization ( aSCR ). Our method uses a K -Lipschitz source critic model to dynamically regularize the optimization objective to avoid extrapolation against the proxy surrogate model f θ in offline MBO.

4.1 Constrained Optimization Formulation

In offline generative optimization, we aim to optimize against a surrogate objective function f θ . In order to ensure that we are achieving reliable estimates of the true, unknown oracle objective, we can add a regularization penalty to keep generated samples 'similar' to those from the training dataset of f θ according to an adversarial source critic trained to discriminate between generated and offline samples. That is, in contrast to (2), aSCR instead considers a closely related constrained problem

over some configuration space Z ⊆ R d , and where we define c ∗ as a source critic model that maximizes E z ′ ∈ P [ c ∗ ( z ′ )] -E z ∈ Q [ c ∗ ( z )] over all K -Lipschitz functions as in (6). We can think of E z ′ ∈ P [ c ∗ ( z ′ )] -c ∗ ( z ) as the contribution of a particular generated datum z to the overall p = 1

Wasserstein distance between the generated candidate ( Q ) and reference ( P ) distributions of designs as in (6). In practice, we model c ∗ as a fully connected neural net. Intuitively, the imposed constraint restricts the feasible search space to designs that score at least as in-distribution as the average sample in the offline dataset according to the source critic. Therefore, c ∗ acts as an adversarial model to regularize the optimization policy. Of note, this additional constraint in (7) may be highly non-convex for general c ∗ , and so it is often impractical to directly apply (7) to any arbitrary MBO policy.

4.2 Dual Formulation

To solve this implementation problem, we instead look to reformulate (7) in its dual space by first considering the Lagrangian L of our constrained problem:

where λ ≥ 0 is the Lagrange multiplier associated with the constraint in (7). We can equivalently think of λ as a hyperparameter that controls the relative strength of the source critic-penalty term: λ = 0 equates to naïvely optimizing the surrogate objective, while λ ≫ 1 asymptotically approaches a WGAN-like optimization policy. Minimizing L thus minimizes a relative sum of -f θ and the Wasserstein distance contribution from any particular generated datum z with relative weighting dictated by the hyperparameter λ . From duality, minimizing L over z and simultaneously maximizing over λ ∈ R + is equivalent to the original constrained problem in (7).

The challenge now is in determining this optimal value of λ : if λ is too small, then the objective estimates may be unreliable; if λ is too large, then the optimization trajectory may be unable to adequately explore the input space. Prior work by Trabucco et al. (2021) has previously explored the idea of formulating offline optimization problems as a similarly regularized Lagrangian (albeit with a separate regularization constraint), although their method tunes a static hyperparameter by hand. In contrast, aSCR treats λ as a dynamic parameter that adapts to the optimization trajectory in real time.

4.3 Computing the Lagrange Multiplier λ

Continuing with our dual formulation of (7), the Lagrange dual function g ( λ ) is defined as g ( λ ) = inf z ∈ R n L ( z ; λ ) . The z = ˆ z that minimizes the Lagrangian in the definition of g is evidently a function of λ . To show this, we use the first-order condition that ∇ z L = 0 at z = ˆ z . Per (8), we have

In general, solving (9) for ˆ z is computationally intractable-especially in high-dimensional problems. Instead, we can approximate ˆ z by relaxing the condition in (9) according to

Our key insight is that although minimizing the loss term in (10) is not practical when the feasible set is naïvely uniform over R n , we can instead choose to focus our attention on latent space coordinates with high associated probability according to the VAE prior distribution p VAE ( z ) . This is because in optimization problems acting over the latent space of any variational autoencoder, the majority of the encoded information content is embedded according to p VAE ( z ) due to the Kullback-Leibler (KL) divergence contribution to V AE training. Put simply, the encoder distribution q ϕ ( z | x ) is trained so that D KL [ q ϕ ( z | x ) || p VAE ( z ))] is optimized as a regularization term in (3). We argue that it is thus sufficient enough to approximate ˆ z ( λ ) using a Monte Carlo sampling schema with random samples Z N = ( z 1 , z 2 , . . . , z N ) ∼ p VAE ( z ) :

We can now concretely write an approximation of the Lagrange dual problem of (7):

where ˆ z is as in (11). Defining the surrogate variable α such that λ = α 1 -α , we can rewrite (12) as

In practice, we discretize the search space for α to 200 evenly spaced points between 0 and 1 inclusive. From weak duality, finding the optimal solution to (12) provides a lower bound on the optimal solution to the primal problem in (7). Algorithm 1 can now be used to choose the optimal α (and hence λ ) adaptively during offline optimization: we refer to our method as Adaptive SCR (aSCR) .

4.4 Overall Algorithm

Using Adaptive SCR, we now have a proposed method for dynamically computing α (and hence the Lagrange multiplier λ ) of the constrained optimization problem in (7). Importantly, aSCR can be integrated with any standard function optimization method by optimizing the Lagrangian objective in (8) over the candidate design space as opposed to the original unconstrained objective f θ . We refer to this algorithm as Generative Adversarial Model-Based Optimization (GAMBO). To evaluate aSCR empirically, we instantiate two flavors of GAMBO: (1) G enerative A dversarial B ayesian O ptimization ( GABO , Algorithm 2 ); and (2) G enerative A dversarial G radient A scent ( GAGA ). 2

We implement GABO using a quasi-expected improvement (qEI) acquisition function, iterative sampling budget of T = 32 , sampling batch size of b = 64 , and GAGA using a step size of η = 0 . 05 , T = 128 , and b = 16 . Of note, the optimization objective using aSCR is time-varying and causally linked to past observations made during the optimization process via intermittent training of the source critic c . Prior works from Nyikosa et al. (2018) and Aglietti et al. (2022) have examined optimization against dynamic objective functions, although have either entirely disregarded causal relationships between variables or only examined causality between inputs as opposed to inputs and the objective. We leave such methods for future work given that aSCR works well in practice.

5 Experimental Evaluation

5.1 Datasets and Tasks

To evaluate our proposed algorithm, we focus on a set of eight tasks spanning multiple domains with publicly available datasets in the field of offline model-based optimization. (1) The Branin function is a well-known synthetic benchmark function where the task is to maximize the two-dimensional Branin function f br : [ -5 , 10] × [0 , 15] → R . (2) The LogP task is a well-studied optimization problem (Zhou et al., 2019; Chen et al., 2021; Flam-Shepherd et al., 2022) where we search over candidate molecules to maximize the penalized water-octanol partition coefficient (logP) score, which is an approximate measure of a molecule's hydrophobicity (Ertl & Schuffenhauer, 2009) that also rewards structures that can be synthesized easily and feature minimal ring structures. We use the publicly available Guacamol benchmarking dataset from Brown et al. (2019) to implement this task.

Tasks (3) - (7) are derived from Design-Bench, a publicly available set of MBO benchmarking tasks (Trabucco et al., 2022): (3) TF-Bind-8 aims to maximize the transcription factor binding efficiency of an 8-base-pair DNA sequence (Barrera et al., 2016); (4) GFP the green fluorescence of a 237-amino-acid protein sequence (Brookes et al., 2019; Rao et al., 2019); (5) UTR the gene expression from a 50-base-pair 5'UTR DNA sequence (Sample et al., 2019; Angermueller et al., 2020); (6) ChEMBL the mean corpuscular hemoglobin concentration (MCHC) biological response of a molecule using an offline dataset collected from the ChEMBL assay CHEMBL3885882 (Gaulton et al., 2012); and (7) D'Kitty the morphological structure of the D'Kitty robot (Ahn et al., 2020).

Finally, (8) the Warfarin task uses the dataset of patients on warfarin medication from Consortium (2009) to estimate the optimal dose of warfarin given clinical and pharmacogenetic patient data. Of note, in contrast to tasks (1) - (7) and other traditional MBO tasks in prior literature (Trabucco et al., 2022), the Warfarin task is novel in that only a subset of the input design dimensions may be optimized over (i.e., warfarin dose) while the others remain fixed as conditioning variables (i.e., patient covariates). Such a task can therefore be thought of as conditional model-based optimization.

5.2 Policy Optimization and Evaluation

For all experiments, the surrogate objective model f θ is a fully connected net with two hidden layers of size 2048 and LeakyReLU activations. f θ takes as input a V AE-encoded latent space datum and

returns the predicted objective function value as output. The V AE encoder and decoder backbone architectures vary by MBO task and are detailed in Supplementary Table A1 . Following GómezBombarelli et al. (2018) and Maus et al. (2022), we co-train the V AE and surrogate objective models together using an Adam optimizer with a learning rate of 3 × 10 -4 for all tasks. For the optimization tasks over continuous design spaces (i.e., Branin, Warfarin, and D'Kitty), we fix the V AE encoder and decoders as the identity functions, such that the latent and input spaces are equivalent.

The source critic agent c in (7) is implemented as a fully connected net with two hidden layers with sizes equal to four (one) times the number of input dimensions for the first (second) layer. To constrain the Lipschitz norm of c as in (6), we clamp the weights of the model between [-0.01, 0.01] after each optimization step as done by Arjovsky et al. (2017). The model is trained using gradient descent with a learning rate of 0.001 to maximize the Wasserstein distance between the dataset and generated candidates in the V AE latent space.

During optimization, both GABO and GAGA alternate between sampling new designs and training the source critic actor c ( z ) until there is no improvement to the Wasserstein distance W 1 according to c after 100 consecutive weight updates. We find that training c every n generator = 4 sampling steps is a good choice across all tasks assessed, similar to prior work Arjovsky et al. (2017).

All MBO methods were evaluated using a fixed surrogate query budget of 2048. We focus on two evaluation metrics: 100th percentile (1) top k = 1 ; and (2) top k = 128 oracle score. The top k = 128 evaluation metric is commonly reported in prior offline MBOliterature (Mashkaria et al., 2023; Trabucco et al., 2021; Yu et al., 2021); the top k = 1 metric better accounts for the limited oracle query budget of the real-world tasks in which offline MBO would be of use. In both settings, an optimizer selects the top k design that minimize the Lagrangian function value in (8) from the 2048 assessed designs to evaluate using the true oracle function, and the maximum score of those k designs is reported across 10 random seeds.

We evaluate both GABO and GAGA against a number of pre-existing baseline algorithms on one internal cluster with 8 NVIDIA RTX A6000 GPUs. We include vanilla Bayesian Optimization ( BO -qEI) and gradient ascent ( Grad. ) in our evaluation to assess the utility of our proposed aSCR algorithm. Furthermore, we evaluate limited-memory BFGS ( L-BFGS ) Liu & Nocedal (1989), CMA-ES Hansen & Ostermeier (1996), and

Algorithm 1 Adaptive Source Critic Regularization (SCR)

,

Algorithm 2 Generative Adversarial BayesOpt (GABO)

Input: surrogate objective f θ : R d → R , offline dataset D n = { z ′ j } n j =1 , acquisition function a , iterative sampling budget T , sampling batch size b , number of generator steps per source critic training n generator, oracle query budget k AdaptiveSCR Input: α step size ∆ α , search budget B , norm threshold τ Define: Differentiable source critic c : R d → R Define: Lagrangian L ( z ; α ) : R d × R → R // Eq. (8) L ( z ; α ) = -f θ ( z ) + α 1 -α [ E z ′ ∼D n [ c ( z ′ )] -c ( z )] Sample candidates Z 1 ←{ z 1 i } b i =1 ∼ SobolSequence // Train the source critic per Eq. (6) to optimality: c ← argmax || c || L ≤ K W 1 ( D n , Z 1 ) = argmax || c || L ≤ K [ E z ′ ∼D n [ c ( z ′ )] -E z ∼Z 1 [ c ( z )]] α ← AdaptiveSCR ( f θ , c, D n , ∆ α, B , τ ) // Alg. (1) Evaluate candidates Y 1 ←{ y 1 i } b i =1 = {-L ( z 1 i ; α ) } b i =1 Place Gaussian Process (GP) prior on f θ for t in 2 , 3 , . . . , T do Update posterior on f θ with D t -1 = { ( Z m , Y m ) } t -1 m =1 Compute acquisition function a using fitted posterior Sample candidates Z t ←{ z t i } b i =1 according to a α ← AdaptiveSCR ( f θ , c, D n , ∆ α, B , τ ) Evaluate samples Y t ←{ y t i } b i =1 = {-L ( z t i ; α ) } b i =1 if t mod n generator equals 0 then // Train the source critic per Eq. (6) to optimality: c ← argmax || c || L ≤ K W 1 ( D n , Z t ) = argmax || c || L ≤ K [ E z ′ ∼D n [ c ( z ′ )] -E z ∼Z t [ c ( z )]] end if end for return the top k samples from the T × b observations D T = {{ ( z i m , y i m ) } b i =1 } T m =1 according to y i m simulated annealing ( Anneal ) Kirkpatrick et al. (1983). We also compare our method against the more recently introduced algorithms TuRBO -qEI (Eriksson et al., 2019a), COM (Trabucco et al.,

2021), RoMA (Yu et al., 2021), BDI (Chen et al., 2022), DDOM (Krishnamoorthy et al., 2023), BONET (Mashkaria et al., 2023), ExPT (Nguyen et al., 2023), ROMO (Chen et al., 2023), and BootGen (Kim et al., 2023). Because BootGen is proposed by Kim et al. (2023) as an optimization method specifically for biological sequence design, we only assess this baseline method on the five relevant tasks in our evaluation suite.

Conditional MBO Tasks. To our knowledge, prior work in conditional model-based optimization is limited, and so previously reported algorithms are not equipped to solve such tasks out-of-the-box. Chen et al. (2023) explore such tasks in their work, but primarily focus on conditional tasks that are built by arbitrarily fixing certain design dimensions from unconstrained problems, which are not representative of true conditional optimization problems in the real world. In our work, we introduce the Warfarin task to assess methods on their ability to design an optimal therapeutic drug regiment conditioned on a fixed patient state and lab values. To assess existing methods on this task, we implement conditional proxies of all baselines employing a first-order optimization schema via partial gradient ascent to only update the warfarin dose dimension while leaving the patient attribute conditional dimensions unchanged. Conditional BO-based methods are implemented by fitting separate Gaussian processes for each patient. In conditional DDOM, we exchange the algorithm's diffusion model-based backbone with a conditional score-based diffusion model (Gu et al., 2023).

Of note, the BONET algorithm (Mashkaria et al., 2023) requires multiple observations for any given patient to construct synthetic optimization trajectories. However, the key challenge in conditional MBO is that each condition (i.e., patient) has no past observations (i.e., warfarin doses), and instead relies on learning from offline datasets constructed from different permutations of condition values As a result, the BONET algorithm is unable to be evaluated on conditional MBO tasks.

5.3 Main Results

Scoring of one-shot optimization candidates is shown in Table 1 . Across all eight assessed tasks spanning a wide range of scientific domains, GABO with our aSCR algorithm achieved the best average rank of 3.8 when compared to other existing methods (next best is 5.5). Furthermore, GABO was able to propose top k = 1 candidate designs that outperform the best design in the pre-existing offline dataset for 6 of the 8 tasks-greater than any of the other methods assessed. If a larger oracle evaluation budget is available (i.e., k = 128 ), GABO with aSCR performs even better, achieving the best average rank of 3.0 (next best is 4.6). GABO is also the best algorithm on 3 of the 8 tasks and second best on 2 tasks according to this evaluation metric. Altogether, our results suggest that GABO is a promising method for proposing optimal design candidates in offline MBO.

Importantly, our aSCR algorithm improves upon both the naïve BO-qEI and Grad. Ascent parent optimizers assessed. GABO outperforms both baseline BO-based optimization methods in our evaluation suite: BO (TuRBO) only achieves a rank of 8.8 (9.0) on the top k = 1 evaluation metric and a rank of 6.6 (7.4) on the top k = 128 metric. Similarly, GAGA scores an average rank of 7.4 (7.6) on the top k = 1 ( k = 128 ) evaluation metric; by leveraging aSCR, GAGA outperforms its base parent optimizer (Grad. Ascent), which only achieves an average rank of 9.0 and 11.0 on the same two evaluation metrics, respectively. Our results show that using aSCR to adaptively penalize the objective of two popular optimization methods can improve their offline performance.

Qualitative Evaluation: Penalized LogP Task. We evaluate GABO against naïve BO-qEI for the LogP task by inspecting the three-dimensional chemical structures of the top-scoring candidate molecules. As a general principle, molecules that are associated with high Penalized LogP scores are hydrophobic with minimal ring structures and therefore often feature long hydrocarbon backbones (Ertl & Schuffenhauer, 2009). In Figure 2 , we see that BO-qEI using the unconstrained surrogate objective generates a candidate molecule of hydrogen and carbon atoms. However, the proposed candidate includes two rings in its structure, resulting in a suboptimal oracle Penalized LogP score.

We hypothesize that this may be due to a lack of ring-containing example molecules in the offline dataset, as only 6.7% (2.7%) of observed molecules contain at least one (two) carbon ring(s). As a result, the surrogate objective model estimator returns more inaccurate Penalized LogP estimates for input ring-containing structures (surrogate model root mean squared error (RMSE) = 25.5 for offline dataset molecules with at least 2 rings; RMSE = 16.5 for those with at least 1 ring; and RMSE = 4.6 for those with at least 0 rings), leading to sub-par BO-qEI optimization performance as the unconstrained algorithm extrapolates against the surrogate to find 'optimal' molecules that

are out-of-distribution. In contrast, GABO generates a candidate molecule with a long hydrocarbon backbone and no rings, resulting in a penalized logP score of 22.1-greater than the best observed value in the offline dataset for the task.

Ablation Experiments. Taking inspiration from (Trabucco et al., 2021), it is possible to utilize our SCR algorithm in GABO without dynamically computing α (and hence the Lagrange multiplier λ ). To better characterize the utility of aSCR, we ablate Algorithm 1 by treating λ instead as a hand-tunable constant hyperparameter, and test our method using different values of λ = α/ (1 -α ) ( Table 3 ). Setting α = 0 (i.e., λ = 0 ) corresponds to naïve BO against the unconstrained surrogate model, while α = 1 (i.e., λ → ∞ ) is equivalent to a WGAN-like policy. Evaluating constant values of α ranging from 0 to 1, we find that there is no consistently optimal constant value for all eight optimization tasks. In contrast, our method achieves an average rank of 1.9 ( 2.4 ) on the top-1 (top-128) evaluation metric, and is one of the top two methods when compared to the ablations for at least five of the eight tasks. These results suggest that the 'adaptive' nature of aSCR is an important component in solving the constrained optimization problem in (7).

C1(CC(C(C2)=CC1C3

C=4C3C5)C67C5=4)

C6(C)C72

CCCCCCCCCCCCSCCCCCCCCCCCCC

CCCCCCCCCCCCCCCCCC(=O)N(CC)

CCCCCCCCCCC(=O)NCC(=O)O

Structure

CCCCCCCCCCCCCCC(C)CCCCCCC(C)CC(C)CCCCCCCCCCCC1

CCC(C)C1C(C)C(C)CCCCC(C)CCCCCC(CCC)CCCCCC(CC)

(C)C(CC)CC(C(C)C(C)CC)C(C)C(CC)C(C)C(CC)C

(CC=CCB)CBC(CC)C(C)CCC(C)C(C)CCCC

Penalized

LogP Score

11.3

-11.4

22.1

Offline Dataset

BO-qEI

GABO

SMILES

Of note, the top designs found across different constant values of α can be very similar for certain tasks. This reflects the inherent challenge in developing task-agnostic methods for policy regularization-if the magnitudes of the unconstrained objective and regularization function vastly differ, then constant values of α may over- or under- constrain the objective. Adaptive SCR overcomes this problem by dynamically setting α as an implicit function of prior observations.

6 Conclusion

We propose adaptive source critic regularization (aSCR) to solve the problem of off-distribution objective evaluation in offline MBO. When leveraged with vanilla Bayesian optimization, aSCR outperforms baseline methods to achieve an average rank of 3.8 ( 3.0 ) in one-shot k = 1 (few-shot k = 128 ) oracle evaluation, and most consistently proposes designs better than the offline dataset.

Limitations. One limitation of aSCR is that our algorithm requires preexisting knowledge of the prior distribution over the input space in order to be computationally tractable. While we have focused our experimental evaluation on tasks amenable to imposed latent space priors, further work is needed to adapt aSCR to any arbitrary configuration space. Future work may also extend aSCR to improve parent optimization methods more sophisticated than BO-qEI and Gradient Ascent explored herein.

Impact Statement. Offline policy optimization methods, such as those discussed in this work, have the potential to benefit society. Such examples may include helping develop more effective drugs and individualizing patient therapies. However, as with any real-world algorithm, these methods can also be leveraged to generate potentially harmful design candidates. Careful oversight by domain experts and researchers is required to ensure that the contributions proposed herein are used for social good.

References

Aglietti, V., Dhir, N., González, J., and Damoulas, T. Dynamic causal Bayesian optimization. In Proc NeurIPS , 2022. doi: 10.48550/arXiv.2110.13891.

Ahn, M., Zhu, H., Hartikainen, K., Ponte, H., Gupta, A., Levine, S., and Kumar, V. ROBEL: Robotics benchmarks for learning with low-cost robots. In Proc Conf Robot Learn , volume 100 of CoRL'20 , pp. 1300-1313. PMLR, 2020.

Angermueller, C., Belanger, D., Gane, A., Mariet, Z., Dohan, D., Murphy, K., Colwell, L., and Sculley, D. Population-based black-box optimization for biological sequence design. In Proc Int Conf Mach Learn , volume 119 of ICML'20 , pp. 324-34. PMLR, 2020.

Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In Proc Int Conf Mach Learn , volume 70 of ICML'17 , pp. 214-23. PMLR, 2017.

Astudillo, R. and Frazier, P. I. Bayesian optimization of composite functions. In Proc Int Conf Mach Learn , volume 97 of ICML'17 , pp. 354-63. PMLR, 2019.

Barrera, L. A., Vedenko, A., Kurland, J. V., Rogers, J. M., Gisselbrecht, S. S., Rossin, E. J., Woodard, J., Mariani, L., Kock, K. H., Inukai, S., Siggers, T., Shokri, L., Gordân, R., Sahni, N., Cotsapas, C., Hao, T., Yi, S., Kellis, M., Daly, M. J., Vidal, M., Hill, D. E., and Bulyk, M. L. Survey of variation in human transcription factors reveals prevalent DNA binding changes. Science , 351 (6280):1450-4, 2016. doi: 10.1126/science.aad2257.

Bastani, H., Drakopoulos, K., Gupta, V., Vlachogiannis, I., Hadjichristodoulou, C., Lagiou, P., Magiorkinis, G., Paraskevis, D., and Tsiodras, S. Efficient and targeted COVID-19 border testing via reinforcement learning. Nature , 599:108-13, 2021. doi: 10.1038/s41586-021-04014-z.

Berrevoets, J., Verboven, S., and Verbeke, W. Treatment effect optimisation in dynamic environments. Journal of Causal Inference , 10(1):106-22, 2022. doi: 10.1515/jci-2020-0009.

Branin, F. H. Widely convergent method for finding multiple solutions of simultaneous nonlinear equations. IBM Journal of Research and Development , 16(5):504-22, 1972. doi: 10.1147/rd.165. 0504.

Brookes, D., Park, H., and Listgarten, J. Conditioning by adaptive sampling for robust design. In Proc Int Conf Mach Learn , volume 97 of ICML'19 , pp. 773-82. PMLR, 2019.

Brown, N., Fiscato, M., Segler, M. H. S., and Vaucher, A. C. GuacaMol: Benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling , 59:1096-108, 2019. doi: 10.1021/acs.jcim.8b00839.

Chen, C., Zhang, Y., Fu, J., Liu, X., and Coates, M. Bidirectional learning for offline infinite-width model-based optimization. In Proc NeurIPS , 2022. doi: 10.48550/arXiv.2209.07507.

Chen, M., Zhao, H., Zhao, Y., Fan, H., Gao, H., Yu, Y., and Tian, Z. Romo: Retrieval-enhanced offline model-based optimization. In Proc International Conf Dist Artif Intell , number 10 in DAI'23, pp. 1-9. ACM, 2023.

Chen, Z., Min, M. R., Parthasarathy, S., and Ning, X. A deep generative model for molecule optimization via one fragment modification. Nat Mach Intell , 3:1040-9, 2021. doi: 10.1038/ s42256-021-00410-2.

Consortium, T. I. W. P. Estimation of the warfarin dose with clinical and pharmacogenetic data. New England Journal of Medicine , 360(8):753-64, 2009. doi: 10.1056/NEJMoa0809329.

Deshwal, A. and Doppa, J. Combining latent space and structured kernels for Bayesian optimization over combinatorial spaces. In Proc NeurIPS , volume 34, pp. 8185-200, 2021.

Eriksson, D. and Jankowiak, M. High dimensional Bayesian optimization with sparse axis-aligned subspaces. In Proc Unc Arti Intel , volume 161 of UAI'21 , pp. 493-503. PMLR, 2021.

Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., and Poloczek, M. Scalable global optimization via local Bayesian optimization. In Proc NeurIPS , pp. 5496-507, 2019a. doi: 10.48550/arXiv. 1910.01739.

Eriksson, D., Pearce, M., Gardner, J. R., Turner, R., and Poloczek, M. Scalable global optimization via local Bayesian optimization. In Proc NeurIPS , 2019b. doi: 10.48550/arXiv.1910.01739.

Ertl, P. and Schuffenhauer, A. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. J Cheminform , 1(8), 2009. doi: 10.1186/1758-2946-1-8.

Flam-Shepherd, D., Zhu, K., and Aspuru-Guzik, A. Language models can learn complex molecule distributions. Nature Communications , 13(3293), 2022. doi: 10.1038/s41467-022-30839-x.

Gaulton, A., Bellis, L. J., Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y ., McGlinchey, S., Michalovich, D., Al-Lazikani, B., and Overington, J. P. ChEMBL: A large-scale bioactivity database for drug discovery. Nucleic Acids Res , 40:D1100-7, 2012. doi: 10.1093/nar/gkr777.

Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., and Aspuru-Guzik, A. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science , 4:268-76, 2018. doi: 10.1021/acscentsci.7b00572.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Proc NeurIPS , pp. 2672-80, 2014. doi: 10.48550/arXiv.1406.2661.

Gu, X., Yang, L., Sun, J., and Xu, Z. Optimal transport-guided conditional score-based diffusion models. In Proc NeurIPS , 2023. doi: 10.48550/arXiv.2311.01226.

Guimaraes, G. L., Sanchez-Lengeling, B., Outeiral, C., Farias, P. L. C., and Aspuru-Guzik, A. Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models. arXiv Preprint , 2017. doi: 10.48550/arXiv.1705.10843.

Hansen, N. and Ostermeier, A. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proc IEEE Int Conf on Evolutionary Comp , pp. 312-7, 1996. doi: 10.1109/ICEC.1996.542381.

Hvarfner, C., Hellsten, E. O., and Nardi, L. Vanilla Bayesian optimization performs great in high dimensions. arXiv Preprint , 2024. doi: 10.48550/arXiv.2402.02229.

Kantorovich, L. and Rubinstein, G. S. On a space of totally additive functions. Vestnik Leningrad. Univ , 13:52-9, 1958.

Kim, C. and Bastani, O. Learning interpretable models with causal guarantees. arXiv Preprint , 2021. doi: 10.48550/arXiv.1901.08576.

Kim, M., Berto, F., Ahn, S., and Park, J. Bootstrapped training of score-conditioned generator for offline design of biological sequences. In Proc NeurIPS , 2023. doi: 10.48550/arXiv.2306.03111.

Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv Preprint , 2013. doi: 10.48550/arXiv.1312.6114.

Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. Optimization by simulated annealing. Science , 220 (4598):671-80, 1983. doi: 10.1126/science.220.4598.671.

Krenn, M., Häse, F., Nigam, A., Friederick, P., and Aspuru-Guzik, A. Self-referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology , 1(4):045024, 2020. doi: 10.1088/2632-2153/aba947.

Krishnamoorthy, S., Mashkaria, S., and Grover, A. Diffusion models for black-box optimization. In Proc Int Conf Mach Learn , volume 202 of ICML'23 , pp. 17842-57. JMLR, 2023.

Liu, D. C. and Nocedal, J. On the limited memory BFGS method for large scale optimization. Mathematical Programming , 45:503-28, 1989. doi: 10.1007/BF01589116.

Mashkaria, S., Krishnamoorthy, S., and Grover, A. Generative pretraining for black-box optimization. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of ICML'23 , pp. 24173-97. JMLR, 2023.

Maus, N. T., Jones, H. T., Moore, J. S., Kusner, M. J., Bradshaw, J., and Gardner, J. R. Local latent space Bayesian optimization over structured inputs. In Proc NeurIPS , 2022. doi: 10.48550/arXiv. 2201.11872.

Mockus, J. The Bayesian approach to global optimization. In System Modeling and Optimization , pp. 473-481. Springer, 1982.

Nguyen, T., Agrawal, S., and Grover, A. Expt: Synthetic pretraining for few-shot experimental design. In Proc NeurIPS , 2023. doi: 10.48550/arXiv.2310.19961.

Nyikosa, F. M., Osborne, M. A., and Roberts, S. T. Bayesian optimization for dynamic problems. arXiv Preprint , 2018. doi: 10.48550/arXiv.1803.03432.

Osborne, M. A., Garnett, R., and Roberts, S. J. Gaussian processes for global optimization. In Int Conf Learn Intell Opt , pp. 1-15, 2009.

Ramchandani, P., Bastani, H., and Wyatt, E. Unmasking human trafficking risk in commercial sex supply chains with machine learning. Social Science Research Network , 2021. doi: 10.2139/ssrn. 3866259.

Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., and Song, Y. S. Evaluating protein transfer learning with TAPE. In Proc NeurIPS , 2019. doi: 10.48550/arXiv. 1906.08230.

Sample, P. J., Wang, B., Reid, D. W., Presnyak, V., McFadyen, I. J., Morris, D. R., and Seelig, G. Human 5'UTR design and variant effect prediction from a massively parallel translation study. Nature Biotechnology , 37:803-9, 2019. doi: 10.1038/s41587-019-0164-5.

Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. In Proc NeurIPS , volume 25, pp. 2951-9, 2012. doi: 10.48550/arXiv.1206.2944.

Sobol, I. M. On the distribution of points in a cube and the approximate evaluation of integrals. Mathematical Physics , 7:86-112, 1967. doi: 10.1016/0041-5553(67)90144-9.

Sun, S., Cao, Z., Zhu, H., and Zhao, J. A survey of optimization methods from a machine learning perspective. IEEE Transactions on Cybernetics , 50(8):3668-81, 2020. doi: 10.1109/TCYB.2019. 2950779.

Székely, G. J., Rizzo, M. L., and Bakirov, N. K. Measuring and testing dependence by correlation of distances. Annals of Statistics , 35(6):2769-94, 2007. doi: 10.1214/009053607000000505.

Todorov, E., Erez, T., and Tassa, Y . MuJoCo: A physics engine for model-based control. In Proc Int Conf Intell Rob Sys , pp. 5026-33, 2012. doi: 10.1109/IROS.2012.6386109.

Trabucco, B., Kumar, A., Geng, X., and Levine, S. Conservative objective models for effective offline model-based optimization. In Proc Int Conf Mach Learn , volume 139 of ICML'21 , pp. 10358-68. PMLR, 2021.

Trabucco, B., Geng, X., Kumar, A., and Levine, S. Design-Bench: Benchmarks for data-driven offline model-based optimization. In Proc Int Conf Mach Learn , volume 162 of ICML'22 , pp. 21658-76. PMLR, 2022.

Tripp, A., Daxberger, E., and Hernández-Lobato, J. M. Sample-efficient optimization in the latent space of deep generative models via weights retraining. In Proc NeurIPS , volume 33, pp. 11259-72. Curran Associates, Inc., 2020.

Truda, G. and Marais, P. Evaluating warfarin dosing models on multiple datasets with a novel software framework and evoluationary optimisation. J Biomedical Informatics , 113:103634, 2021. doi: 10.1016/j.jbi.2020.103634.

Weininger, D. Smiles, A chemical language and information system. J Chem Inf Comput Sci , 28(1): 31-6, 1988. doi: 10.1021/ci00057a005.

Wildman, S. A. and Crippen, G. M. Prediction of physicochemical parameters by atomic contributions. J Chem Inf Comput Sci , 39:868-73, 1999.

Xu, K. and Bastani, H. Multitask learning and bandits via robust statistics. arXiv Preprint , 2023. doi: 10.48550/arXiv.2112.14233.

Yu, S., Ahn, S., Song, L., and Shin, J. RoMA: Robust model adaptation for offline model-based optimization. In Proc NeurIPS , 2021. doi: 10.48550/arXiv.2110.14188.

Zhou, Z., Kearnes, S., Li, L., Zare, R. N., and Riley, P. Optimization of molecules via deep reinforcement learning. Scientific Reports , 9(10752), 2019. doi: 10.1038/s41598-019-47148-x.

A Additional Implementation Details

Oracle Functions. All oracle functions for the tasks assessed are either exact functions or approximate oracles developed by domain experts. Specifically, the Branin and TF-Bind-8 tasks utilize exact oracles described in detail by Branin (1972) and Barrera et al. (2016), respectively. The oracle for the Penalized LogP task is an approximate oracle from Wildman & Crippen (1999) that is the same oracle used by domain experts in the Guacamol benchmarking study (Brown et al., 2019). The GFP , UTR , and ChEMBL tasks feature approximate oracles from Snoek et al. (2012), Angermueller et al. (2020), and Trabucco et al. (2022), respectively, that were trained on a larger, hidden datasets inaccessible to us for the respective tasks. The D'Kitty morphology task uses a MuJoCo (Todorov et al., 2012) simulation environment and learned control policy from Trabucco et al. (2022) to evaluate proposed designs. Finally, the Warfarin task uses a linear model (Consortium, 2009) to estimate a patient's optimal warfarin dose given their pharmacogenetic attributes.

Data Preprocessing. (1) For the Branin task, we sample 1000 points from the square input domain [ -5 , 10] × [0 , 15] to construct the offline dataset, and remove the top 20%-ile according to the oracle function to make the task more challenging in line with prior work (Mashkaria et al., 2023). In this continuous task (along with the D'Kitty and Warfarin tasks), we treat input designs as their own latent space mappings, such that the VAE encoder and decoder for this task are both the identity function with zero trainable parameters. (2) The offline dataset of the Penalized LogP task is the validation partition of the Guacamol dataset from Brown et al. (2019), which consists of 79,564 unique molecules and their corresponding penalized LogP scores. The input molecules are represented as SMILES strings (Weininger, 1988), which is a molecule representation format shown to frequently yield invalid molecules in prior work (Krenn et al., 2020). Therefore, we encode the molecules instead as SELFIES strings, an alternative molecule representation from Krenn et al. (2020) with 100% robustness.

(3) - (5) The TF-Bind-8 , GFP , and UTR tasks are assessed as-released by Design-Bench from Trabucco et al. (2022)-please refer to their work for task-specific descriptions. (6) - (7) In the ChEMBL and D'Kitty tasks, we normalize all objective values y in the offline dataset to ˆ y = ( y -y min ) / ( y max -y min ) as done in prior work (Mashkaria et al., 2023), where ˆ y is the corresponding normalized objective value and y min ( y max) is the minimum (maximum) observed objective value in the full, unobserved dataset. Because only the bottom 60%-ile (40%-ile) from the full dataset is used in the available offline dataset for the ChEMBL (D'Kitty) task, the respective maximum ˆ y values are less than 1.0 ( Supplementary Table A1 ). We also translate the original SMILES string representations in the ChEMBL task into SELFIES strings (Krenn et al., 2020) as in the LogP task.

(8) Finally, the Warfarin task uses the dataset of pharmacogenetic patient covariates published by Consortium (2009). We split the original dataset of 3,936 unique patient observations into training (validation) partitions with 3,736 (200) datums. The patient attributes in the Warfarin dataset consist of a combination of discrete and continuous values. All discrete attributes are one-hot encoded into binarized dimensions, and continuous values are normalized to zero mean and unit variance using the training dataset. Missing patient values were imputed following prior work (Truda & Marais, 2021). We define the cost c ( z | x ) accrued by a patient with attributes x ∈ R 32 as a function of the input dose z ∈ R is c ( z | x ) = ( z -d oracle ( x )) 2 , where d oracle : R 32 → R is the domain-expert oracle warfarin dose estimator from Consortium (2009). The observed objective values y associated with each of the training datums is calculated as y = [ c (¯ z | x ) -c ( z | x )] /c (¯ z | x ) , where ¯ z is the mean warfarin dose over the training dataset and z is the true dose given to the patient. Using this constructed offline dataset, our task is then to assign optimal doses to the 200 validation patients to maximize y with no prior warfarin dosing observations.

B Additional Experimental Results

In this section, we provide additional experimental results that help better characterize both the strengths and limitations of GABO and GAGA.

B.1 How do sub-optimal design candidates proposed by GABO and GAGA perform?

To evaluate the robustness of optimization methods, we report one-shot 90th percentile oracle scores in Supplementary Tables B1 and B2 . For each method, all proposed designs are ranked according

to the surrogate forward model ((8) for Generative Adversarial Bayesian Optimization (GABO) and Generative Adversarial Gradient Ascent (GAGA)), and the single 90th percentile design according to this ranking is selected and evaluated using the oracle function. We report the oracle score of this suboptimal design averaged over 10 seeds.

Our results show that GABO and GAGA do not propose suboptimal designs that are better than those proposed by other methods, such as BONET (Mashkaria et al., 2023), Simulated Annealing (Kirkpatrick et al., 1983), L-BFGS (Liu & Nocedal, 1989), and ExPT (Nguyen et al., 2023). This is not surprising, as aSCR is not designed to target this metric (and it is not our primary metric of interest). Separately for GABO, we also hypothesize that the algorithm's performance according to this metric may partially be explained by the limitations of the underlying Bayesian optimization (BO) optimization algorithm. Because BO is not an iterative first-order algorithm, the designs proposed by any BO-based algorithm often have high variance in practice-this is indeed what we observe across all of our experiments, including in Table 1 and Supplementary Tables B1 and B2 .

Finally, we note that in most applications of offline optimization, the 90th percentile metric-or any metric that does not use the best proposed design(s)-is not as useful as the other metrics assessed where GABO does perform well. This is because in offline optimization tasks with a restricted budget to query the hidden, expensive-to-evaluate oracle function, we are not interested in 'wasting' this limited budget on subpar design candidates. While the 90th percentile and similar metrics can be helpful to understand the limitations of algorithms (as in this case), we believe that the alternative evaluation metrics reported in the main text-namely, the 100th percentile top-1 and top-128 oracle score metrics-are more useful and practical in assessing each of the optimization algorithms.

To further characterize the distribution of designs and their associated oracle scores proposed by GABO, Figure B1 plots a histogram of the oracle scores of (1) all 2,048 oracle scores, and (2) the oracle scores of the top 256 designs according to the penalized surrogate objective in (8) for the LogP task. Compared with the other optimization methods assessed, we notice that the range of oracle scores is larger for BO-based optimization methods compared with the baseline methods assessed. This helps motivate our design choice to leverage aSCR and Algorithm 1 with BO-qEI, as BO is able to explore a larger region of the design space and is an effective parent optimizer for complex design spaces. Secondly, we also find that the distribution of scores is similar between BO-qEI and GABO, even though the performance of these two methods is remarkably different in Tables 1 and 2 . This is likely due to the fact that while BO enables us to explore a larger effective region of the design space (compared with first-order iterative methods), aSCR more accurately ranks proposed designs using the penalized surrogate so that we can identify promising candidates even in the low-budget oracle evaluation regime .

B.2 Are offline objectives and oracle function values correlated?

A key component of GABO with Adapative SCR critical to the above discussion in Section B.1 is that generated designs score similarly according to the hidden oracle function and the regularized Lagrangian objective as in (8) in order to solve the problem of surrogate objective overestimation encountered in traditional offline optimization settings ( Fig. 1 ). To assess this quantitatively, we computed the distance covariance dCov n [ {L ( x k ; λ ∗ ) } n k =1 , { f ( x k ) } n k =1 ] between the oracle scores f ( x k ) and the constrained Lagrangian scores L ( x k ; λ ∗ ) with λ = λ ∗ ( t ) computed using our Adaptive SCR algorithm. The empirical distance covariance metric is computed over the n = 2048 design candidates generated using our GABO algorithm. Briefly, the distance covariance is a nonnegative measure of dependence between two vectors which may be related nonlinearly; a greater distance covariance implies a greater degree of association between observations (Székely et al., 2007). We focus our subsequent discussion on the Penalized LogP task.

Across five random seeds, GABO with Adaptive SCR achieves a distance covariance score of 0.535 ± 0.067 (mean ± standard deviation). In contrast, naïve BO-qEI (i.e., λ = 0 ) only achieves a distance covariance score of 0.392 ± 0.040. Using p < 0 . 05 as a cutoff for statistical significance, the distance covariance scores are significantly different between these two methods ( p ≈ 0 . 006 , unpaired two-tailed t -test). These results help support our conclusion that GABO with Adaptive SCR is able to provide better estimates of design candidate performance according to the hidden oracle function when compared to the corresponding unconstrained BO policy.

B.3 Is adaptively computing α in aSCR important for the performance of GAGA?

In our ablation experiments presented in Table 3 , we showed how that 'adaptive' nature of aSCR is an important component in solving the constrained optimization problem in (7) for GABO, and outperforms alternative approaches that manually hand-tune α (and hence λ ) as a constant hyperparameter. We explore whether this conclusion also applies for GAGA as well here.

For clarity, we first offer the explicit formulation of GAGA in Supplementary Algorithm 3 . We ablate Algorithm 1 in GAGA by instead evaluating our method using different values of λ = α/ (1 -α ) . As a reminder, setting α = 0 (i.e., λ = 0 ) corresponds to naïvely performing gradient

0.0

0.1

0.2

0.3

0.4

Grad.

0.0

0.1

0.2

Grad. Mean

0.0

0.1

0.2

Grad. Min

0.00

0.02

0.04

0.06

L-BFGS

0.00

0.05

0.10

0.15

BOBYQA

0.00

0.02

0.04

0.06

CMA-ES

0.0

0.1

0.2

0.3

0.4

Sim. Anneal

0.000

0.025

0.050

0.075

0.100

BO-qEI

0.00

0.05

0.10

0.15

TuRBO-qEI

0.0

0.2

0.4

BONET

0.00

0.05

0.10

0.15

0.20

DDOM

0.0

0.1

0.2

COM

50

25

0

25

50

75

100

Oracle Penalized LogP Score

0.0

0.1

0.2

RoMA

50

25

0

25

50

75

100

Oracle Penalized LogP Score

0.000

0.025

0.050

0.075

0.100

GABO

ascent against the unconstrained surrogate model; setting α = 1 (i.e., λ → ∞ ) is equivalent to a WGAN-like generative policy.

Algorithm 3 Generative Adversarial Gradient Ascent (GAGA)

Input: surrogate objective f θ : R d → R , offline dataset D n = { z ′ j } n j =1 , iterative sampling budget T , sampling batch size b , number of generator steps per source critic training n generator, oracle query budget k , step size η AdaptiveSCR Input: α step size ∆ α , search budget B , norm threshold τ Define: Differentiable source critic c : R d → R Define: Lagrangian L ( z ; α ) : R d × R → R = -f θ ( z ) + α 1 -α [ E z ′ ∼D n [ c ( z ′ )] -c ( z )] // Eq. (8) Sample Z 1 ←{ z 1 i } b i =1 as the top b designs in D n according to their previously observed oracle scores // Train the source critic per Eq. (6) to optimality: c ← argmax || c || L ≤ K W 1 ( D n , Z 1 ) = argmax || c || L ≤ K [ E z ′ ∼D n [ c ( z ′ )] -E z ∼Z 1 [ c ( z )]] α ← AdaptiveSCR ( f θ , c, D n , ∆ α, B , τ ) // Alg. (1) Evaluate candidates Y 1 ←{ y 1 i } b i =1 = {-L ( z 1 i ; α ) } b i =1 for t in 2 , 3 , . . . , T do Z t ←{ z t i } b i =1 = { z t -1 i -η ∇ z t -1 i L ( z t -1 i ; α ) } b i =1 α ← AdaptiveSCR ( f θ , c, D n , ∆ α, B , τ ) Evaluate samples Y t ←{ y t i } b i =1 = {-L ( z t i ; α ) } b i =1 if t mod n generator equals 0 then // Train the source critic per Eq. (6) to optimality: c ← argmax || c || L ≤ K W 1 ( D n , Z t ) = argmax || c || L ≤ K [ E z ′ ∼D n [ c ( z ′ )] -E z ∼Z t [ c ( z )]] end if end for return the top k samples from the T × b observations D T = {{ ( z i m , y i m ) } b i =1 } T m =1 according to y i m

Our results are shown in Supplementary Table B3 : similar to the analogous ablation results for GABO in Table 3 , dynamically adjusting the strength of source critic regularization using our aSCR algorithm outperforms manually setting the value of α to a constant in both the one-shot k = 1 and few-shot k = 128 evaluation settings.

B.4 What is the impact of dynamic updates to the source critic over the optimization trajectory?

In Algorithm 2 and Supplementary Algorithm 3 , we describe how generative adversarial optimization alternates between batched acquisition steps according to the optimizer and re-training the source critic on the newly sampled trajectory points. To better interrogate the significance of dynamically re-training the source critic during optimization, we compare the performance of the default GABO and GAGA algorithms (with n generator = 4 as the number of acquisition steps per critic retraining step) against the respective methods without source critic re-training (i.e., n generator = ∞ ) in Supplementary Table B4 . Across all three evaluation metrics and all eight tasks, dynamically retraining the source-critic improves upon the performance of the GABO when n generator = ∞ by 67.4% in the top-1 evaluation metric; 0.0% in the top-128 evaluation metric; and 33.5% in the 90%-ile evaluation metric. Intuitively, these results align with the value of the source critic in being able to implicitly set the value of the regularization strength α in (8) according to the sampled trajectory points-especially in the constrained budget oracle evaluation setting.

Interestingly, we do not observe similar performance improvements with dynamic re-training of the source critic in GAGA. Qualitatively, we find that this is because of the iterative first-order nature of the parent gradient ascent algorithm-because the sampled designs are clustered in the same regions of the design space over the course of optimization, the energy landscape of the penalized surrogate (i.e., the negative of the Lagrangian expression in (8)) does not change significantly during source critic re-training. This further reinforces the optimizer to stay roughly in the same regions of the design space. As a result, it is likely that no major updates are often made to the source critic when aSCR is used in conjunction with a first-order optimization method, and so the benefit of using a finite n generator hyperparameter value is largely reduced when compared to its utility in GABO.

B.5 How does initialization affect the performance of GABO?

Per Algorithm 2 , GABO is based on the BO-qEI baseline optimization policy, which involves initializing the gaussian process (GP) to approximate the offline surrogate model. Consistent with prior work (Eriksson et al., 2019a; Maus et al., 2022), we initialize the GP using the pseudo-random Sobol sequence (Sobol, 1967) at the beginning of the optimization procedure. However, an alternative approach is to instead initialize the GP using the top n init samples from the offline dataset. In particular, this strategy is already employed in both related work describing the baseline first-order optimization methods assessed herein, with the idea that better designs can be generated by initializing from better designs. We compare these two GP initialization strategies in Supplementary Table B5 .

Interestingly, our results show that initializing the GABO GP from the Sobol sequence consistently outperforms initialization from the top candidates in offline dataset. We hypothesize that this may be due to the fact that top-scoring candidates likely lie in similar regions of the input space, which significantly alters the ability of the optimizer to explore other regions of the design space over the course of the optimization process. Future work may help better interrogate the relationship between GP initialization and offline optimization, which is outside the scope of this work.

B.6 Can the Gaussian process (GP) in GABO be directly used as the surrogate forward model?

In Algorithm 2 , we leverage a surrogate forward model f θ in model-based optimization and a separate GP to acquire samples in the Bayesian optimization framework. However, it may be possible to use the GP directly as the surrogate forward model. Our results in Supplementary Table B6 suggest that this is not an effective strategy with which to use GABO-using even the simple neural-network as the surrogate function (as done in our approach in Algorithm 2 ) outperforms the alternative GP-based

approach in six of the eight tasks in the top-1 evaluation setting, and is non-inferior to the alternative GP-based approach in all eight tasks in the top-128 evaluation setting. These results suggest that using a more complex neural-network surrogate function for GABO leads to better optimization results than directly using the GP as the surrogate function.

B.7 What is the computational cost of running aSCR (i.e., Algorithm 1)?

At first glance, Adaptive SCR may appear to be a computationally expensive algorithm: it requires us to dynamically re-train a source critic neural network and compute the Lagrangian hyperparameter at each step through a grid search. However, in the implementation used for our experiments, the grid search to compute α is highly vectorized, and the source critic re-training patience and learning rate are such that the computational cost from re-training is not too significant. As a result, we are able to run Adaptive SCR with both Bayesian Optimization (BO) and Gradient Ascent (GA) using an experimental setup with one 24-core Intel Xeon CPU and one NVIDIA RTX A6000 GPU. To benchmark our implementation, we evaluate BO and GA both with and without our Generative Adversarial (GA) source critic regularization algorithm on the Branin and Penalized LogP optimization tasks. As a reminder, the Branin task is a standard benchmarking task for offline optimization, and the Penalized LogP task is subjectively the most challenging task assessed in our manuscript with the highest dimensional design space out of the eight assessed tasks.

Our results are shown in Supplementary Table B7 . On the Branin toy task, aSCR increases the compute time by 257% for BO and 680% for GA, which is a significant computational cost. However, on the more challenging LogP task more representative of the tasks encountered in the applications of offline optimization, aSCR only introduces a 6.9% increase in compute time for GA and 28.9%

increase for BO. Furthermore, while there are evidently additional compute costs associated with running our aSCR algorithm, we note that in most applications of offline optimization, obtaining labeled data is the main bottleneck in many practical applications. Thus, it is often worth spending this extra compute to ensure the best results for a given evaluation budget using aSCR.

B.8 How do the performance of GABO and other optimization methods vary with the allowed oracle query budget k ?

To investigate this question, we vary the number of allowed k -shot oracle calls in the Penalized LogP task ( Supplementary Fig. B2 ). While the majority of first-order optimization methods we evaluated

are able to reach local optima rapidly, the proposed designs from such approaches are suboptimal compared to those from GABO (and GAGA) with Adaptive SCR as the oracle query budget size increases. Separately, comparing the curves for GABO and vanilla BO-qEI, we see that GABO with Adaptive SCR is able to propose consistently superior design candidates in the small query budget regime often encountered in real-world settings. This is due to the fact that GABO regularizes the surrogate function estimates such that the proposed candidates are both high-scoring according to the surrogate objective and relatively in-distribution. Our results demonstrate that especially for real-world tasks like molecule design with complex objective function landscapes, methods such as GABO with Adaptive SCR are able to explore diverse, high-performing design candidates effectively even in the setting of small oracle query budgets.

0

500

1000

1500

2000

Oracle Query Budget

50

0

50

100

150

Best Observed Oracle Penalized LogP Score

Offline MBO Method

Grad.

L-BFGS

CMA-ES

BO-qEI

TuRBO-qEI

BONET

DDOM

COM

RoMA

BDI

BootGen

GAGA

GABO

B.9 Is the optimization budget sufficient for optimization convergence?

For all of our experimental results, we restrict the surrogate query budget to a total of 2048 allowed offline surrogate model queries in order to ensure a fair comparison between different optimization methods. To ensure that such a budget is sufficient for optimizer convergence across different optimization methods, we plot the best achieved oracle Penalized LogP value (i.e., assuming an unlimited oracle evaluation budget) as a function of the number of optimizer surrogate queries ( Supplementary Fig. B3 ) for the Penalized LogP task. These results show that our methods are indeed able to converge over the course of the optimization trajectory.

0

500

1000

1500

2000

Number of Surrogate Queries

0

20

40

60

80

100

120

140

Best Observed Oracle Penalized LogP Score

Offline MBO Method

Grad.

L-BFGS

CMA-ES

BO-qEI

TuRBO-qEI

BONET

DDOM

COM

RoMA

BootGen

ROMO

GAGA

GABO

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction are supported by the experimental results presented. The introduction includes the contributions made in the paper, and important assumptions and limitations are included where relevant.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Please see Section 6 in the main text for a focused 'Limitations' section and relevant discussion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the relevant information needed to reproduce all reported experimental results of the paper. We have made our code required to reproduce our experimental results available in the Supplementary Material ZIP file.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data and code associated with this paper is open access and includes sufficient instructions to faithfully reproduce the experimental results reported herein. We have made the data and code available in the Supplementary Material ZIP file. The public link to the GitHub repository containing the code will be included in the Abstract in the final version (currently not linked so as to comply with double-blind review). All datasets used herein are publicly available without any limitations in public accessibility. Our data and code adhere to the NeurIPS code and data submission guidelines.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting is clearly described in Section 5.2 and Appendix A , and is also reproducible via the code released alongside the paper in the Supplementary Material ZIP file.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All statistics and results included in the paper are accompanied by confidence intervals. We clearly describe the factors of variability in the confidence intervals in Section 5.2 and Appendix A . All additional details are included in relevant table captions.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information for the resources required to reproduce the experiments are included in Section 5.2 in the main paper. More specifically, all experiments were performed on a single internal cluster with 8 NVIDIA RTX A6000 GPUs. Any particular experimental configuration required no more than 24 hours to complete using our setup. The full research project did not require more compute than the experiments reported in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and assert that the research described herein conforms with the Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see the conclusion (i.e. Section 6 ) in the main text for a focused 'Impact Statement' subsection and relevant discussion on the potential societal impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not introduce any assets that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The only existing assets used in the paper are the seven MBO datasets used for experimental evaluation as described in Section 5.1 . All of the relevant datasets are publicly available, and the references for each of the datasets are cited in the aforementioned section. The licenses associated with each of the seven datasets made available in each of the relevant citations are properly respected. There are no restrictions with respect to accessing any of the datasets used in the paper. No scraped data was used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only asset introduced in and released alongside the paper is the experimental code to reproduce the reported results. The repository containing the code is included in the Supplementary Material ZIP file. All datasets used for the experiments discussed herein are publicly available and available online via the appropriate references in this paper and via online links included in the Supplementary Material ZIP file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The research described herein does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research described herein does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed GABO, a novel Bayesian optimization method for offline model-based optimization problems. GABO regulates the surrogate model with source critic actor so that the BO procedure remains in the in-distribution. Experiment results validate that GABO outperforms several baselines in terms of the mean rank.\n\nSTRENGTHS:\n- Bayesian optimization is a widely used algorithm for black-box optimization but unexplored in offline MBO settings. As far as I know, it is the first paper that improves BO for offline MBO settings\n\n- Mathematical formulation seems to be valid, and practical algorithm is given\n\nWEAKNESSES:\n- It seems that the tasks used for evaluation from the Design Bench are discrete, **biological sequence design tasks**. There are several continuous tasks, such as Superconductor, Ant, and Dkitty, which are also high-dimensional and challenging problems. I think the research scope is then limited to offline biological sequence design, not general offline optimization tasks. Then, authors should compare their method with papers specialized in biological sequence designs, such as BIB[1] and BootGen[2].\n\n- If my understanding is correct, the **evaluation procedure is a bit different from the offline MBO conventions**. The authors first train a surrogate model and choose the top-1 candidate among 2048 candidates according to the predicted score of the surrogate model. As the surrogate model gives inaccurate predictions on the data points outside of the offline dataset, it is not convincing to choose the best one with the surrogate model for evaluation. The authors should elaborate on the reason why they changed the evaluation setting.\n\n[1] Chen, Can, et al. \"Bidirectional learning for offline model-based biological sequence design.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Kim, Minsu, et al. \"Bootstrapped training of score-conditioned generator for offline design of biological sequences.\" Advances in Neural Information Processing Systems 36 (2024).\n\nQUESTIONS:\n- In the experiment part, there is a new task called Warfarin, which is a conditional offline MBO task. I think the problem setting is practical and important. However, it seems that the proposed method lags behind other baselines in terms of performance. Could authors elaborate on why this phenomenon happens?\n\n- It is also hard for me to accept the results of BONET and DDOM in the Branin task, as both papers deeply analyze the behavior of their methods in the Branin function. Is it due to the different evaluation settings?\n\nLIMITATIONS:\nAs written in the weakness section, authors should specify the scope of the research and compare their method with proper baselines. Furthermore, the authors should explain why the evaluation setting has been changed.\n\nThere are a few minor comments on the manuscript.\n\n- In the related work part, it might be beneficial that authors clearly state the limitations of prior methods rather than just explain the methods.\n\n- In the background part, it might mislead readers if we define offline MBO as solving the optimization problem in Eq (2). There are several methods that formulate the problem as conditional generative modeling. Even for forward approaches, they do not solve the problem in Eq (2) and propose various approaches such as regularization.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles offline Bayesian optimization using adaptive source critic regularization. The authors propose generative adversarial Bayesian optimization, which optimizes against a learned surrogate model without querying the true oracle function during optimization. It utilizes a Lipschitz-bounded source critic model to constrain the optimization trajectory. Finally the authors provide experimental results against various baseline methods.\n\nSTRENGTHS:\n- The setup of offline Bayesian optimization seems interesting.\n\nWEAKNESSES:\n- The rationale behind the proposed method is unclear.\n- Details of experiments are missing.\n\nQUESTIONS:\n- Could you explain how these examples \"Evaluating newly proposed molecules requires expensive experimental laboratory setups, and testing multiple drug doses for a single patient can potentially be dangerous\" agree with your problem formulation?\n- This sentence \"Leveraging source critic model feedback for adversarial training of neural networks was first introduced by Goodfellow et al. (2014)\" is not true. The authors should revise it.\n- I think that this work just proposes a better regression method in the offline setting.\n- Can you provide the details of $f_\\theta$? Why did you choose such parameterization?\n- How does a learned surrogate model provide additional information without querying the true oracle function? Source critic regularization doesn't add anything.\n- When do you evaluate a true objective function? Didn't you evaluate query points when comparing your method to baseline methods?\n- Equation (10): Is argmin correct? Should it be min?\n- Can you explain the rationale behind $\\lambda = \\alpha / (1 - \\alpha)$?\n- Tables 1 and 2: How can the baseline methods show the results over the best in $\\mathcal{D}$? I think that it is bounded in $\\mathcal{D}$.\n- Tables 1 and 2: What is the purpose of the row of the best? How can I interpret the results better than the best?\n- It is a minor thing, but \"bolded\" is not a correct word. It should be just \"bold.\"\n\nLIMITATIONS:\nThere is no particular societal limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considered an offline optimization problem where a surrogate objective function instead of the oracle objective can be queried. The surrogate objective function is trained using a reference offline dataset and thus may falsely predict the optimum due to overestimation errors. To resolve this issue, this paper proposed a Generative Adversarial Bayesian Optimization (GABO) algorithm which exploits the adaptive Source Critic Regularization (SCR) to achieve robust oracle function optimization. The proposed method is demonstrated to outperform other tested offline optimization baselines on several synthetic and real-world datasets.\n\nSTRENGTHS:\n1. The idea of adding a regularization penalty to the original offline optimization problem is interesting. The technical issues are clearly introduced.\n\n2. The empirical performance improvement of the proposed GABO is significant compared to the other BO and model-based optimization algorithms.\n\nWEAKNESSES:\nI have reviewed this paper before and had multiple rounds of discussions with the authors in the previous review process. My major concern is still about the motivation and positioning of this work. The key contribution of this work is the design of the surrogate objective in (7) and its dual formulation in (8) for a model-based optimization (MBO) problem. This new objective is independent of BO and can actually be optimized using the zero-order, one-order, or any other optimization algorithm. Even though this paper has highlighted and empirically shown that BO is an effective choice among multiple optimization methods, I still think the combination of SCR and BO is trivial and should not be the focus of this work.\n\nI cannot consider the proposed GABO algorithm as a novel **BO** framework since all the new techniques proposed are not specifically designed for BO. The challenges of applying BO in the generative adversarial optimization problem are unclear. I highly suggest the authors to reconsider the title and motivation of this work such that its contributions can be clearer.\n\nQUESTIONS:\nIn Lines 210-213, it is mentioned that the objective of GABO is time-varying, which is an interesting and non-trivial issue of applying BO to this problem. Do you have any idea about how to resolve this issue?\n\nLIMITATIONS:\nThis limitations are clearly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of offline black-box optimization where only limited (zero-shot or few-shot) online interactions with the objective function is available. Existing approaches commonly train a neural net parametrized surrogate model of the objective using the offline data. The paper proposes to use a source critic model, inspired by discriminator training in generative adversarial networks, to regularize this surrogate. This is accomplished by formulating a constrained optimization problem which constrains the optimization (over the surrogate model) trajectory samples to be similar to the training data using the source critic model. The Lagrangian version of this constrained optimization is used as the objective for a standard batch mode Bayesian optimization algorithm. Experiments are performed on tasks from Design-bench benchmark and a new Warfarin task.\n\nSTRENGTHS:\n- One thing I really like about the paper is its evaluation which truly mirrors the offline optimization setting. The paper compares the proposed approach and baselines on a single evaluation from the oracle which I believe is the right way to evaluate algorithms for offline model-based optimization.\n\n- The proposed approach shows good performance across all the tasks on both zero-shot and few-shot evaluation. The ablation study in 5.5 also shows that adaptively tuning the lagrangian coefficient is also important for good performance.\n\n- The constrained optimization formulation is well-formulated and principled approach to tackle this problem. The design choice of constraining the surrogate model using the source critic model makes sense and is relevant for the problem.\n\n- Some of the tasks in design-bench benchmark have multiple errors which makes them not so informative for evaluation. I like the paper didn't include superconductor task where the original offline dataset itself has multiple copies of the same inputs but with different outputs. Similarly, I think the ChemBL and UTR tasks are also not useful. In my practical experience, samples searched over ChemBL generates a lot of syntactic errors. I like the fact that the paper introduces new tasks like LogP and Warfarin which will be useful to the broader community.\n\nWEAKNESSES:\n- One question I have is about the broader picture of using Bayesian optimization (BO) for this problem space. Why should we first train the neural network surrogate model and then fit the Gaussian process (GP) on top of the neural surrogate model to do few steps of BO? Why can't we just fit a GP (with a latent space presumably to handle high/structured dimensionality) directly on the offline data and do one step of BO? I am assuming the TurBO-qEI baseline doesn't do that and also fits the Turbo's GP on the neural network surrogate.\n\n- It would be nice to see more justification for the BO hyperparameters (like sampling budget (T) and batch size (b)). Is the method sensitive to the choice of these parameters?\n\n- One thing I like about the paper is that it can work with any choice of the surrogate model. I would also like to point out one very recent related work that also constrains the optimization given any surrogate model by formulating the search as an offline RL problem.\n\n        - Yassine Chemingui, Aryan Deshwal, Nghia Hoang, Janardhan Rao Doppa. Offline Model-based Black-Box Optimization via Policy-guided Gradient Search. Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI), 2024.\n\nQUESTIONS:\nPlease see weaknesses section.\n\nLIMITATIONS:\nPlease see weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper proposes a novel approach for offline MBO that combines Source Critic Regularization and Bayesian Optimization. Offline MBO aims to train a surrogate model from offline data and subsequently extrapolates it to find an optimal design (as opposed to online MBO which collects more data as it trains the surrogate model). One key challenge of offline MBO is knowing when to trust the surrogate model, since it could be highly erroneous at ood inputs. This paper handles said challenge by incorporating a critic constraint that penalizes candidate that are not sufficiently similar to the training samples.

STRENGTHS:
- Clear description, easy to follow
- Practical and novel method for MBO. As a regularization method it can be incorporated into many other MBO frameworks.
- Hyperparameter \lambda is chosen methodically
- Perform well on 100th percentile metric

WEAKNESSES:
- Doesn't seem to be very good on 90th percentile metric (i.e., mean rank ~ 8). Does this mean that the method is very good at picking out high-reward outlier, but the overall quality of the surrogate is about the same as everything else? If that's the case is there any justification for this behavior? I would prefer if the authors plotted out the entire scatter plot of 100 candidates for each method, as that would allow us to compare performance on a distributional level.

QUESTIONS:
- Why run BO on the learned surrogate when it is already differentiable and can be numerically optimized using GA? (I saw the ablation study on GAGA but just want to hear an argument for the benefit of GABO).

LIMITATIONS:
No potential negative societal impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use an adversarial regulariser in the BO setting. In particular, the authors propose a systematic way to compute the regularization parameter through a lagrange duality. Overall the regularized method performs on average better than existing methods across a suite of benchmark datasets.\n\nSTRENGTHS:\n- The paper is the first to propose an adversarial regularization objective for BO.\n- The method is well motivated and a practical way to obtain the hyperparameter is given, albeit through a grid.\n- The proposed methods performs on par with baselines and on average is better across several benchmark datasets.\n\nWEAKNESSES:\n- to clarify, the way you choose alpha is through a grid. Wouldnt that mean your computational cost is significantly larger than baseline methods, please add a section on computational time as well as cost to better understand the tradeoffs of the proposed method over existing methods. Please comment on the above and clarify any misunderstandings please. Especially this part: \"To ensure a fair evaluation schema, all MBO methods were evaluated using a fixed surrogate query budget of 2048\". Can you confirm that no extra data was used to obtain the alpha in your algorithm? There might be a misunderstanding here, in particular what how exactly to pick the best alpha based on the grid of 200 and the associated costs?\n- VAEs as well as GAN are often time hard to train and need very specific architectures to work well. How, were these chosen and is there any ablation study? how does the quality of \"c\" affect the problem setting? My concern is that there are so many moving parts that is becomes hard to understand what the contribution of each part is\n- I dont understand D(best) how come some values in the table are able to be better than the oracle? Am i reading the table correctly?\n\n\nI am more than happy to change my score if the above have been clarified\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}