Local Latent Space Bayesian Optimization over Structured Inputs
Authors: Natalie Maus
Haydn Thomas Jones
Juston Moore
Matt Kusner
John Bradshaw
Jacob R. Gardner
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian optimization
ML for molecules

Abstract: Bayesian optimization over the latent spaces of deep autoencoder models (DAEs) has recently emerged as a promising new approach for optimizing challenging black-box functions over structured, discrete, hard-to-enumerate search spaces (e.g., molecules). Here the DAE dramatically simplifies the search space by mapping inputs into a continuous latent space where familiar Bayesian optimization tools can be more readily applied. Despite this simplification, the latent space typically remains high-dimensional. Thus, even with a well-suited latent space, these approaches do not necessarily provide a complete solution, but may rather shift the structured optimization problem to a high-dimensional one. In this paper, we propose LOL-BO, which adapts the notion of trust regions explored in recent work on high-dimensional Bayesian optimization to the structured setting. By reformulating the encoder to function as both an encoder for the DAE globally and as a deep kernel for the surrogate model within a trust region, we better align the notion of local optimization in the latent space with local optimization in the input space. LOL-BO achieves as much as 20 times improvement over state-of-the-art latent space Bayesian optimization methods across six real-world benchmarks, demonstrating that improvement in optimization strategies is as important as developing better DAE models.

Local Latent Space Bayesian Optimization over Structured Inputs
Natalie Maus
Haydn T. Jones
Department of Computer and Information Science University of Pennsylvania
Los Alamos National Laboratory Los Alamos, NM hj hjones@lan! gov
Philadelphia, PA nmaus@seas upenn. edu
Juston Moore
Matt J. Kusner Centre for Artificial Intelligence, University College London London, UK m. kusueeruucl..... ac. uk
Los Alamos National Laboratory
Los Alamos, NM juston@lanl.gon
John Bradshaw
Jacob R. Gardner
Department of Chemical Engineering, Massachusetts Institute of Technology
Department of Computer and Information Science
University of Pennsylvania Philadelphia, PA jacobrgdseas uponn. edu
Cambridge, MA jbrad@mit. edu
Abstract
Bayesian optimization the latent spaces of deep autoencoder models (DAEs) has recently emerged as a promising new approach for optimizing challenging black-box functions structured, discrete, hard-to-enumerate search spaces (e.g., molecules). Here the DAE dramatically simplifies the search space by map- ping inputs into continuous latent space where familiar Bayesian optimization tools can be more readily applied. Despite this simplification, the latent space typically remains these approaches do not necessarily provide a complete solution, but may rather shift the structured optimization problem hign-dimensional one. In this paper, we propose LOL-BO, which adapts the notion of trust regions explored in recent work on high- dineesninni Bayesian optimization to the structured setting. By refor- mulating the encoder to function as both an encoder for the DAE globally and as deep kernel for the surrogate model within a trust region, we better align the notion of local optimization in the latent space with local optimization in the input space. LOL-BO achieves as much as 20 times improvement over state-of state-of-the-ar latent space Bayesian optimization methods across six real-world benchmarks, demon- strating that improvement in optimization strategies as important as developing better DAE models.
dibddmmmnnional.. Thus. even with well-suited latent space,
Introduction
Many challenges across the physical sciences and engineering require the optimization of structured objects. For example, in the drug design setting, it common seek a molecule that maximizes some property, e.g., binding affinity for a target protein [[7] or similarity to a known drug molecule
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

These structured optimization problems are particularly challenging, as the objective function now typically over a large combinatorial space of discrete objects.
under some metric B.
These challenging optimization problems have motivated a large amount of very recent interest in methods that perform Bayesian optimization (BO) over latent spaces. Using drug design running example, a deep auto-encoder (DAE) is first trained on a large set of unsupervised molecules. Bayesian optimization is then run as normal in the continuous latent space of the autoencoder to produce candidate latent codes. These candidate latent codes are then decoded back to molecules, and the objective function evaluated. Bayesian optimization then proceeds as normal, with the surrogate model trained on a dataset of latent codes and their resulting objective values after decoding.
Rapid progress has been made in this emerging topic. In large part, this has been to improve the DAE model. The early work of Kusner et al. (28) and Jin et al. [21) investigated domain specific autoencoder architectures that much more reliably produced valid molecules. The latest work of Tripp et al. [45] and Grosnit et al. [4] investigate methods to inject supervised signal from objective function evaluations into the latent space, enabling better surrogate modeling. However, the optimization component of latent space Bayesian optimization remains under-explored: indeed, it common practice directly apply standard Bayesian optimization with expected improvement once a latent space has been constructed (all of the above cited methods do this). We conjecture that, despite the ability of deep encoder to dramatically simplify structured input space, optimization remains challenging due to the high- sigh-dmmennionalityy of the extracted latent space.
In this paper, we propose jointly co-design the latent space model and a local Bayesian optimization strategy. Concretely, we make the following contributions:
1. We develop local latent Bayesian optimization, LOL-BO, for the structured optimization setting, addressing the inherent mismatch between the notion of a trust region in the latent space and trust region in the original structured input space.
2. We derive a semisupervised DAE model that better satisfies the implicit assumptions of trust region methods by performing joint variational inference over the DAE and sparse GP surrogate model (this model is similar in spirit to Kingma et al. 25
Eissman et al. 66
3. We propose a deep autoencoder architecture using transformer layers over the recently proposed SELFIES representation for molecules [2. We find that the SELFIES string representation provides significant improvements to optimization performance.
4. We show the potential of co-adapting latent space modeling with local Bayesian optimization in an empirical evaluation of LOL-BO against top-performing baselines, achieving up to 22x improvements in objective value on commonly used benchmark tasks over prior latent space BO approaches.
2 Background
In black-box optimization, we seek to find the minimizer of a function, arg mingex (x). Commonly, fxx) is assumed to be expensive to evaluate and unknown (i.e., a "black box"). It is therefore desirable to use fewer function evaluations to achieve given objective value.
Most classically, this problem has been considered in the setting where the search space X' is continuous (e.g., X' C RR. However, many applications across the natural sciences and engineering require optimizing over discrete and structured input spaces. De novo molecule design using machine learning has received particular attention, where A' is a search space of small organic molecules and f(x some desirable property to optimize
is framework for solving these black-box optimization problems, most commonly when X' is a continuous, real-valued search space. A surrogate model is trained on a set of inputs and corresponding objective values D; [x, Val This surrogate model used to suggest candidate inputs to query next. The candidates are then labeled, added to the dataset, and the process repeats. Gaussian process models f' f((x) ~ GP(m(x), k(x, x')) are commonly used as surrogate models, as access to calibrated predictive (co-)variances enables search strategies that trade off exploration and exploitation.
Bayesian optimization
+
30
41

---

Deep autoencoder models. A deep autoencoder model consists of an encoder + : X + 2 that maps from structured and discrete space (c.g., the space of molecules) X' to continuous, real-valued latent space 2, and decoder that does the reverse r(.): 2 - X. A variational autoencoder [24] uses a probabilistic form of these two functions, a variational posterior (( X) and likelihood T(X Z). The parameters 0g. OT of T and I are trained by maximizing the evidence lower bound (ELBO):
VAVR = EO,O 0r) Ra(z)) [og [(x Z)] ( X) (lp()))
Commonly, the (amortized) variational posterior (z X) and the prior p() are chosen to be Gaussian, which leads to a convenient analytic expression for the KL divergence. Note that, with respect to specific optimization tasks we might want to perform over the structured space A, the ELBO as defined above is unsupervised.
Latent space Bayesian optimization. Latent space Bayesian optimization seeks to leverage the representation learning capabilities of deep autoencoders models (DAEs) to reduce a discrete, struc- tured search space X' to continuous one Z unsupervised with respect to an optimization task, a large set of unlabeled inputs Du S X is first used to train the DAE. A Gaussian process prior is again placed over the objective function, but here function of the latent codes, f(z) ~ GP(m(z) k(2, z z²)). To train the GP initially, we collect a small set of n inputs Dj X < [D.,] with known labels yı- The inputs are passed through the encoder to obtain latent representations = [2: 1-11 and the surrogate model is fit to the dataset [[z,, y: y:)] Standard BO strategies may then be applied using this surrogate model to select candidate latent vector 2, which is then decoded back to an input & and evaluated.
43 5
Because the ELBO above is
3 Methods
Recent latent space Bayesian optimization (LS-BO) approaches have focused on improving the architecture of the VAE used to compress the search space 23 21], and on reorganizing the subsequent latent space so that it is more amenable to BO standard BO with expected improvement once the latent space has been constructed, despite the fact that this latent space often remains high-dimensional (21. on the order of tens or hundreds of dimensions. In this paper, we explore whether lessons learned in the high-dimensional BO setting can be adapted to improve optimization in high-dimensional latent spaces.
4
4 However, it is commonplace to apply
3.1 Local Bayesian optimization - Fixed latent space
One approach to adapting high-dimensional BO strategies is to apply them in the extracted latent space Z. Because Z is synthetic latent space, it's unlikely that the objective function decomposes additively over this space without explicitly encouraging this 2 9 47 Likewise, Z is already a compression of the much larger X,, so methods that exploit low-dimensional substructure may not be
effective 48 32
Local Bayesian optimization is strategy that avoids over-exploration in these search spaces without simplifying assumptions about the objective. Commonly this done by restricting the search each iteration to small region. For example, Eriksson et al. [ proposes TuRBO-M, an optimization algorithm which maintains M local optimization runs, each of which is limited to search within its own hyper-rectangular trust region. Each trust region is centered at the current best input found by given run. Since Eriksson et al. [ found that TuRBO-1 is competitive or even better than TuRBO-m where > 1 on most tasks, in this paper we focus only TuRBO-1. Following Eriksson et al. we will henceforth use TuRBO as short-hand for TuRBO-1.
Adapting trust regions to LS- BOOBORR is straightforward. Suppose we have a VAE with encoder ((-) and decoder r--) trained on an unsupervised dataset D,, of structured inputs, and supervised set Z of latent codes corresponding to labeled inputs D; and their objective values y. Denote by 2 the incumbent latent code corresponding to the input x with best observed objective value ý so far.
The trust region is an axis aligned hyper-rectangular subset of the latent space Tg C Z, centered at the incumbent latent vector à. It has base side length L. with the actual side length for each dimension X,k
rescaled according to its lengthscale X; in the GP's ARD covariance function, L, TT X,)
3

---

Candidate latent vectors to evaluate next are restricted to be selected from Tg. Following Nelder and Mead [33. Eriksson al. [8]. the base trust region length is halved after That iterations where no progress is made and doubled after Tince iterations in row where progress is made. When new incumbent ż' is found with y² < tj. the trust region is recentered at 2'.
3.2 Local Bayesian optimization Adaptive latent space
By defining "locality" in terms of small rectangular region centered at the incumbent, trust region methods implicitly assume smoothness, that points close to the center have highly correlated objective values with it. In the standard BO setting with GP surrogates, this assumption matches that made by the GP, as many covariance functions measure covariance as function of distance.
However, the latent space Z of DAE is only trained so that latent vector z (xx) reconstructs the original input when passed through the decoder, [(z) R x. Therefore, it is possible that small changes to the current incumbent latent vector 2 may result in large changes the corresponding inpul &, violating the assumptions of both the GP surrogate model and the trust region method.
Adapting the latent space the GP prior. We propose solve this mismatch by performing inference jointly over both the Gaussian process surrogate defined over the latent space p(f Z) and the Bayesian VAE model p(X, Z) p(X Z)p(Z), while assuming in the prior conditional
independence between f and X given Z:
(f,, X Z) = p(f Z)T(X Z)p(Z)
By performing inference over f and Z jointly in the above model end-to-end, encourage the latent space to organize in a way that matches the assumptions of the GP prior p(f Z).
Relation to prior work. Learning the VAE and supervised model jointly in semisupervised fashion is related to prior work on semisupervised VAE modelling as first proposed Kingma et al. (25). This adapted for BO in Eissman al. 6 who use auxiliary supervised neural network to inject supervision before fitting the GP. Here we consider joint variational inference over the VAE and sparse GP, directly encouraging the latent space to satisfy the assumptions of the GP prior. In the global BO setting, Siivola et al. [40) finds semisupervised adaptation of VAEs to not be critical; however, in the local setting, we experimentally find this results in significant improvement due to the assumptions underlying trust region methods.
Sparse GP augmentation. Following Hensman et al. 15) the above prior p(f Z) can be augmented with global set of learned inducing variables u representing latent function values at of inducing locations V, p(f Z) p(f u, Z, V)u( V). This results in the following full joint density over the latent function f, structured inputs X, latent representation Z, objective values y, and inducing variables u (ommitting dependencies on inducing locations V for compactness):
ply, X,f, u, Z) = p(y (f)(( u, ())(u)T(X Z))(Z)
(1)
ELBO derivation. Our goal is to derive an evidence lower bound (ELBO) for the above model to perform variational inference over both the inducing variables u and VAE latent variables Z. The log evidence p(y, X) for this model is log jog((y, X) = log p(y, X, f, u, Z)dfdudZ. To aid in lower bounding the log evidence, we introduce a variational distribution q(u, Z) (Z X). This is a factored joint variational distribution over the inducing values u as in 15 and over the latent space of the encoder Z.
doy,(y. X) log / ply, X, f. u, Z) Z)df
(Z X)
du
dZ
q(u)
( X
log (z)x
E (q(
p(y. X, f. Z df
(2)
(Z
qq())
Because the joint distribution p(y, X, f, u, Z) factors as in Equation
we can further rewrite the
integral of the joint distribution over as:
f ply, X. u, df Ep(riu,z) (p(y (f)(())((X)))))) Z)p(Z)

---

Plugging this expected value form of the joint distribution integral into Equation 2 and repeatedly applying Jensen's inequality, we derive an evidence lower bound
[lo f)] + (z)x) [log ['x Z)] logp(y, X) 2 ( Z X p(Z) - (L) Ilp)
(3)
The ELBO in practice. Rephrasing Equation 3 in terms of the standard SVGP (sparse variational GP: 15 and VAE ELBOs, we see that:
Cjoint (x(Z(x) x [Csvar(Opp, O Z)] + Lvvn Or; X)
The ELBO can estimated as follows. Select a minibatch of inputs (e.g., molecules) X, some of which have known labels y, and some of which are unlabeled. Pass the minibatch X through the encoder and generate latent codes Z. For latent codes Zy corresponding to labeled inputs, compute LSVGP on the supervised dataset Yz). For all latent codes Z, pass through the decoder and add the standard VAE loss. Because the variational posteriors (Z X) and q(u) support reparameterization, computing derivatives of the ELBO with respect to 0z. 0r, Ocp is straightforward. Because the encoder parameters 0g are updated through the SVGP loss, tb can be viewed as simultaneously acting encoder for the VAE and as (stochastic) deep kernel 52 ST
for the GP model.
Recentering Under this joint model, local BO can still be applied using the same setup and hyperparameters discussed in subsection 3.1 However, when using the above loss, the VAE-and therefore the latent space-is now modified after each iteration of BO. Therefore, the center of the trust region å and latent codes of inputs within we pass observed data X back through the updated encoder to determine the new location of the corresponding latent points Z before selecting candidates, and update the trust region accordingly.
the trust region may change. To remedy this,
Computational considerations. Updating the full encoder ((-) can be significantly more expensive than training the surrogate model only. A practical extension to the above idcalized process is to only train in joint fashion periodically. Thus, the variational GP parameters 0gp are updated in each iteration of optimization, but not necessarily 04, 0p.
We propose to tie updates to fg. OT the sequential successes and failures already tracked by the trust region method. If the optimization procedure is making rapid progress, updating 0s, Or may be an unnecessary expense. Therefore, we update the full set of parameters 0g. 0r, 0gp after Tretrin successive failures. Experimentally we use Tretrain < updates for each time a trust region shrinks.
Tail as this allows for multiple joint model
Related Work
Bayesian optimization for molecule discovery can be performed without DAEs over fixed lists of molecules [50] 16 [3. However, the need to pre-define the molecules that can be queried upfront puts limit to the chemical space such methods can consider: the largest virtual libraries order of s 10t0 in size 445), which is tiny compared to the space of all possible compounds [26].
ties the optimization algorithm to DAE so that, in theory, can generate any molecule. We broadly group many recent advancements in this approach into two categories: (a) those that investigate new training losses to improve surrogate 6. 5 and (b) those that develop new decoder architectures, for instance improving the validity of the generated molecules by building in grammars or working explicitly in the graph domain [2X] all 38 22 Decoder architectures have also been designed to Gaussian
Latent-space Bayesian optimization 12
learning of the objective 14 43
apply these ideas to different discrete domains such as neural network architectures 55
process kernels
Components of our method including the joint training procedure described in subsection 3.2 are similar to Eissman et 66 which considers injecting semisupervision into the VAE via an auxiliary neural network and Grosnit et al. [4] which does so using triplet loss. In particular, the triplet loss of Grosnit et al. [14] likely encourages similar latent space organization to the method described in subsection 3.2 due to the use of distances in both the triplet loss and the GP covariance function. However, these works use off-the-shelf global Bayesian optimization algorithms, such as expected improvement, the latent optimizer.
5

---

Alternative optimization algorithms have also been proposed for molecule discovery, such as those Markov chain Monte Carlo [53], ideas from reinforcement 39 use the cross entropy method to guide the generation of complete SMILES strings from an autoregressive recurrent neural network. You et al. [54] train a policy to build molecules up from atoms and bonds using an RL algorithm. While such methods are often able to find molecules with better property scores than early LS-BO approaches, these methods frequently neglect sample efficiency, impeding their use practical problems.
based on genetic algorithms 20 34
learning (RI.) RG 54 56 Segler et al.
5 Experiments
Penaliced LogP
T-LBO (Graat) et 2021)
Expressions
LOL-BO (SELFIES)
LXLL-10
Best Score Found (Hightt Battrr
500
LOL-B0 (TTVA))
Stundard LS
W-LBO (Tripo 2020)
W-LDO (Trpp etal, 2020)
Hett Dastett (4.53)
Regrett Rognett (Lovee is Bettsc)
T-1 (Cremit al, 2213
7.0
2.8
0.0
Number of Steps
Number of Steps
Figure 1: Optimization results for the log P and arithmetic expressions tasks. We compare recent competitive latent BO methods 14 43 using the same DAE (LOL-BO (JTVAE)) and the new SELFIES VAE (LOL-BO (SELFIES)).
We apply LOL-BO to six high-dimensional, structured BO tasks over molecules metic expressions. We additionally investigate the impact of various components of our method in subsection 5.4 and subsection 5.4 log P and Expressions (subsection 5.1) tasks that typically evaluated in low budget settings. The Guacamol benchmarks dsubsection 5.2) afford larger function evaluation budgets, and the protein docking task (subsection 5.3) both low budget and the oracle is expensive. Error bars in all plots show standard error over runs. We implement and GPyTorch (TO Code and model weights are available
IR 19] and arith-
LOL-BO leveraging BoTorch 2
https://gh //g thub. com htto/nataliienn ceattaalemaasslaa /lolbo
Datasets All methods have access to the same amount of supervised and unsupervised data for each task. To initialize all molecular optimization tasks, we generate labels for random subset of 10,000 000 molecules from the standardized unlabeled dataset of 1.27M molecules from the Guacamol benchmark software 3 For the expressions task, expressions from |E! to initialize all methods. Furthermore, wherever different VAEs are used for given task, unsupervised data also remains consistent. In particular, all molecular VAFs are pre-trained on the same aforementioned dataset of 1, 1.27M molecules from Brown et al.
the same labeled dataset of 40, 000
the same unsupervised dataset train the VAEs so that the amount of
A variety of VAE models have been used for the benchmarks tasks we consider below. For Expressi ons, we follow recent work 228
Deep autoencoder models.
35) in using GrammarVAE
model 2
For the remaining de novo molecule design tasks, we consider two models.
Junction Tree VAE (JTVAE).. Molecules are commonly represented initially using the Simplified Molecular-Input Line-Entry System (SMILES) 49 string representation. SMILES representations are well-known to be brittle. in that single character mutations can easily result in strings that no longer encode valid molecule. The JTVAE model Jin et 21 addresses the problem of generating valid strings by using molecular graph structure to build each molecule out of valid sub-components so that the generated molecules are almost always valid. As a result, the JTVAE has been widely
BoTooch and GorTorch released under the MIT license

---

used as VAE model for latent space optimization due to its much higher rate of decoding to valid molecules
35
We follow prior work, using latent dimensionality of 56.
SELFIES VAE The SELFIES string representation for molecules is recently proposed alternative to the SMILES string representation 27 Notably, nearly every permutation of SELFIES tokens encodes a valid molecule. As result, purely sequence based models using the SELFIES representa- tion are capable of generating valid molecules rate competitive with the JT-VAE while avoiding expensive computation involving graph structures.
In our molecule design experiments below, therefore evaluate VAE architecture based on the SELFIES string representation in addition to the commonly used JTVAE. We train a VAE model with 6 transformer encoder and transformer decoder layers and latent space dimensionality of 256 [46 We use our SELFIES VAE for all molecular optimization tasks below. We also use the JT'VEE model to enable direct comparison to prior work.
Hyperparameters. For the trust region dynamics, all hyperparameters including the initial base and minimum trust region lengths Linit, Lmins and success and failure thresholds Tanes, Thail are set to the TuRBO defaults as used in Eriksson et al. [X. Our method introduces only one new hyperparameter, Tretrann which we set to 10 in all experiments.
5.1 log P and Expressions
In this section, consider two tasks commonly used in the latent space BO literature -oteetiziiotiiiiiiggggg the penalised water-octanol partition coefficient (log P) over molecules, and generating single variable
arithmetic expressions (e.g., x sin x)) 4 43 5
35 23 1. Because of the small evaluation budget typically considered for these benchmarks, we use a batch size of 1.
Baselines. For these tasks, we compare to standard latent space BO, as well as variety of recent prior work in this area. We compare to LS-BO with weighted retraining (w- (W-LBO) as described in Tripp et al. 4 and LS-BO with triplet loss retraining (T-LBO) from Grosnit et al. [4]. A number of techniques outside of the BO literature consider this task as well. These papers report final objective values 30, but do not consider sample efficiency. For example, GEGL [I which do compare to in subsecctiiin 5.2
uses more than 500 evaluations per iteration. See Ahn et al. [ for an evaluation
on log
Results. In Figure we plot the cumulative best objective value for the penalized log P and expressions tasks, averaged over three runs. For log P. LOL-BO using the SELFIES transformer VAE achieves an average log P score of 545. 17 after 500 iterations. Using a JTVAE, which is more directly comparable to prior work, we still achieve average score of 123.54-higher than previous state-of-the-arr scores using latent space BO of 26. 11 14.. For the expressions task, LOL-BO achieves final regret of 0.079 average, versus 0.194 achieved by T-LBO.
Molecule quality. While molecules found using LOL-BO for the log P task are "valid" according to software commonly used to compute these scores, the molecules produced by this search clearly abandon any notion of reality. A number of methods have been proposed to evaluate molecule quality
3 and indeed even some recent work in LS-BO has explored encouraging higher quality solutions However, our primary goal in this paper is to develop stronger optimization routines in this space, and indeed the baselines we consider also treat log P as unconstrained optimization problem. Therefore, this result should be taken primarily as evidence that the log objective can be exploited by sufficiently strong optimizer, rather than as evidence of novel interesting molecules.
35
5.2 Guacamol Benchmark Tasks
The Guacamo benchmark suite 3 contains scoring oracles for variety of molecule design tasks, with scores ranging between 0 and 1. We select three tasks among the most challenging in terms of the best objective value achieved: Perindopril MPO, Ranolazine MPO, and Zaleplon MPO. The goal of each of these tasks is to design a molecule with high fingerprint similarity to a known target drug but that differs in a specified target way. The task definitions are discussed in Brown al. B.
Guacamol is released under the MIT license
7

---

Zaloplon MPO
Perindogeil MPO
Kanolazine MPO
6.75
0.50
0.75
Blet Scoce Ioand
00.0
6.50
Tunction Evaluations
Function Evaluations
Total Function Evaluations
10L (SILITES)
LOC-30 (ITAAD
Sradard
GIGL 2011
3021
Coog CAtlesrr
Figure Optimization three Guacamol benchmark tasks. We compare T-LBO, the most compet- itive latent space BO method from above. using the same DAE and SMILES representation (LOL-BO (JTVAE)) well as our proposed SELFIES transformer (LOL-BO (SELFIES)), We additionally compare to two recent orthogonal methods for these tasks that do not use deep autoencoders or BO (GraphGA, GEGL). GEGL. uses a combination of SMILES representations and directly modifying atomic graph structure, while GraphGA operates entirely on graph structures and only uses string representations for saving and loading molecules.
Zalephon MPO
Perindopiil MPO
Banolazine MPO
0.50
8270
Best
8.55
3.5
30000
40000
0000
$3030
2000
42003
00000
53000
30000
4900
(000
19939
196393 12000
Total Finction Boaliations
Total Function Evalustions
Total Finction Exalustions
LOL-BO (SELFIES)
TukBo (SELF1E5)
StnseedLS-B0 (SELFIES)
LOL-BO (TVAR)
TaRBO (TVAB)
Stanchni LS-50
We evaluate (1) the gain in optimization performance achieved in moving from global BÔ to local (Standard LS-BO vs TuRBO), and (2) the gain in optimization performance achieved adapting the latent space to be suitable for the trust region method (TuRBO vs LOL-BO), Improving the optimization algorithm results in significant improvements using either VAE.
Figure 3:
Baselines. We compare T-LBO, the most competitive Bayesian optimization method overall from subsection 5.1 Additionally, variety of techniques from beyond the BO literature using reinforcement learning and genetic programming have been evaluated on these tasks. We compare to GEGL and GraphGA, with GraphGA (20] among the top methods on the Guacamol leaderboard. We also report the best score achieved over the x1.27 million molecules in the full Guacamol dataset, which serves as a performance cap for virtual screening (see section 4p.
The results of optimization on these three benchmarks are shown Figure We again report results for LOL-BO with both JTVAE and SELFIES VAE deep autoencoder. LOL-BO with SELFIES VAE optimizes significantly faster than other methods in the evaluation budget used here.
Results.
5.3 DRD3 Receptor Docking Affinity
The Therapeutics Data Commons (TDC) benchmark suite 17 contains oracles that evaluate the binding propensity of ligand (small molecule) to a target protein. These docking oracles ineur nontrivial computation cost, requiring the order of minutes to evaluate, making them ideal

---

Table 1: (Left.) Comparing SELFIES VAE and JTVAE on metrics computed over molecules in the GuacaMol test dataset 3. Decode stddev is the standard deviation Perindopril MPD objective value when decoding fixed latent 20 times, averaged over a subset of 3500 latent vectors from the dataset. (Right.) Docking scores achieved by various methods after the four evaluation budgets considered on the TDC leaderboard (lower is better). We compare to three BO baselines and variety of orthogonal approaches from the TDC leaderboard.
Method
Best Score Found in Evaluation Budget evals
evals
1000 evals
evals
LOL,-80 FIEF)SS)
-13.1 0.3
-13.3 0.1
-1388 + 0.5
-1544 1.3
SELFIES VAE
JTVAE
TuRBO (SELFIES)
-12.6.00
-12.7 0.3
-12.9 1 0.4
-13.0
15.0100 0.4
Reconstruction Acc.
0.913
0.767
LS-BO (SELFIES)
-..6600.0
-126:000
-126:0.0
-126:0.0
Avg. Decode Time (s)
0.02
3,14
LOL-80 (JTVAE)
-126=0.0
-12.
-128#02
-13.1
Decode Stddey
0.04
0.1
TuRBO (TIVE))
-226:000
-126100
-12.7
-
Validity
1.0
1.0
LS-BO (TVAE)
-126000
-126#00
-126
-126=0.0
GP Test NLL (Perindopril)
-2.156
-1.481
T-LBO
-226000
-126#00
-126+000
-128
GP Test NLL(Zaleplon)
-2.705
-1.781
Graph-GA
-1181111
-125
-33
-6.5
GP Test NLL (Ramolazine)
-2.127
-1.649
SMILSTM
-11.100.6
-11.4.0.6
-120102
1.5
MARS
-912007
-98203
-11111011
-114
MolDQN
-70=002
-76+02
-78=001
-2
GCPN
-116422
-.20
-.2.0
-12.3 1.0
benchmark tasks for Bayesian optimization. In this paper, we focus on designing ligands that bind to dopamine receptor D3 (DRD3), the TDC website maintains leaderboard of results for DRD3 We compare LOL-BO on the DRD3 docking task to the TDC leaderboard and three BO methods-T-LBO, standard LS-BO, and fixed latent space TuRBO on this task.
Results of optimization are in Table | (right). LOL-BO achieves the best performance among the BO baselines, with only TuRBO making significant progress on the task by 5000 evaluations. Compared to the baselines from the TDC leaderboard, LOL-BO significantly more sample efficient, achicving results comparable to the best method with factor of 10x fewer oracle calls.
5.4 Ablation studies
In this section, evaluate the components of LOL-BO described in subsection 3.2 beyond the direct application of TuRBO in the latent space descrihed in subsection 3.1] We run TuRBO on the three Guacamol tasks considered in subsection 5.2 using fixed latent space extracted by the pretrained SELFIES VAE and JTV.E. To run TuRBO, we use the same experimental setup as for LOL-BO, but (-)) and I'(-) are fixed the start of optimization. We compare LS-BO, TuRBO, and LOL-BO on all three tasks in Figure 3 The use of SVGP is controlled across methods.
LOL-BO outperforms TuRBO across all DAE models and tasks. While these results demonstrate the impact of joint training, the excellent performance of TuRBO in a static, pre-trained latent space highlights the value of high dimensional Bayesopt in this setting.
To further analyze whether the jointly trained latent space better-matches the distance based assump- tions of the GP model, we include an additional analysis directly comparing scores of molecules sampled from a local region around the top-scoring molecule in the latent space of the pretrained SELFIES-VAL before optimization has begun and at the end of optimization (see subsection A.3).
Since LOL-BO achieves better performance with the SELFIES VAE than with the JTVAE, evaluate a series of metrics for both DAE models. In Table I left, we compare reconstruction error and validity, as well as the standard deviation in Perindopril MPO score from decoding fixed latent vector 20 times, averaged over 3500 distinct z. We report supervised learning performance for a GP trained on 10, 000 molecules taken from the initial Guacamol dataset, using the pretrained latent spaces. Across standard metrics, the SELFIES VAE achieves excellent performance. Beyond these, the improvement in objective variance for single z notable: because Guacamol objective values are in the range [0, 1). standard deviation of 0.1 represents significant additional noise. Thus, the DAE can be significant source of noise even for an otherwise deterministic objective function.

---

Conclusion
Designing latent spaces for Bayesian optimization is challenging because the VAE and the optimizer often compete: too small of latent space renders the VAE incapable of capturing the structure present in the input space, while too large latent space makes optimization challenging.
The empirical results we present here demonstrate the importance of viewing these latent spaces as high-dimenssooal and tackling optimization with nigh-dimensional BO strategies, even though these latent spaces are often significantly smaller than the input space. Furthermore, we show that high-dimensional BO methods can be significantly improved when translating them to the latent space setting, demonstrating the potential of further research at the intersection of these areas.
7 Acknowledgements
JRG and NM were supported by NSF award IIS-2145644. JM and HJ were supported by the Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20210043DR.
References
[1] Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration, 2020.
[2] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: Framework for Efficient Corlo-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL URL http:///axx http://aaxiv..ogaaaaaa abg/bbb
06403
[3] Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. Guacamol: Benchmarking models for de novo molecular design. Journal Chemical Information and Modeling, 1963):006-11108 Mar 2019. ISSN 1549-960X. doi: 10. Jeb00003,,
8b00839, URL http::/dx. dol. org/1100.......1.00
acs j cim. 8b00839
[4] Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. for structured data. In International Conference Learning Representations 2018, 2018.
variational autoencoder
[5] Aryan Deshwal and Janardhan Rao Doppa. Combining latent space and structured kernels for Bayesian optimization over combinatorial spaces. CoRR, abs/2111.01 1,01 2021. URL htttss//////
2111.01 01186
[6] Stephan Eissman, Daniel Levy. Rui Shu, Stefan Bartzsch, and Stefano Ermon. Bayesian optimization and attribute adjustment. In Proc. 34th Conference on Uncertainty in Artificial Intelligence. 2018
[7] David Eriksson and Martin Jankowiak. High-dimensional Bayesian optimization with sparse axis-aligned subspaces. arXiv preprint arXiv:2103.003499 2021.
[8] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloezek. Scalable global optimization via local Bayesian optimization. In H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-Buc, E. Fox, and Garnett, editors, Advances Neural Information Processing Systems. volume 32. Curran Associates, Inc., 2019. URL htt.:/tttt//poceeeeeee
ceeurip..cc/ppper//
pdf
[9] Jacob Gardner, Chuan Guo, Kilian Weinherger, Roman Garnett, and Roger Grosse. Discovering and exploiting additive structure for Bayesian optimization. Artificial Intelligence and Statistics, pages 1311-1319. PMLR, 2017.
[10] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpy- torch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. arXiv preprint ar09v:1809.1 166,, 2018.
[11] Roman Gareet,. Michael A Osbome,, and Philipp Hennig. Active learning of linear embeddings for Gaussian processes. Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence. 2014.
10

---

[12] Rafael Baacchez-Lengelnnge Dennis Sheberla, Jorge -palteraaparrggurreee Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamií
[13] David E Graff, Fugene Shakhnovich, and Connor W Coley. Accelcrating high-throughput virtual screening through molecular pool-based active learning. Chemical science, 12(22):78667788 April 2021.
[14] Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander Imani Cowen- Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang. Jan Peters, and Haitham Bou-Ammar. High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning. CoRR, abs/2106.03600 2021. URL htttttts////rsstaa org/ hbs///b///// abs/21/6..1../ 03609
[15] James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, 2013.
[16] José Miguel Hermández-Lobato, James Requeima, Edward O Pyzer-Knaph and Alán Aspuru-Guzik. Parallel and distributed Thompson sampling for large-scale accelcrated exploration of chemical space. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70, pages
PMLR, 2017.
[17] Kexin Huang. Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks drug discovery and development. Proceedings of Neural Information Processing Systems, NeurIPS Datasels and Benchmarks, 2021.
[18] John Irwin and Brian Shoichet. Zinc- free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177-18) 182, 2005.
[19] John Irwin, Khanh Tang, Jennifer Young, Chinzorig Dandarchuluun, Benjamin R Wong, Munkhzul Khurelbaatar, Yurii S Moroz, John Mayfield, and Roger A Sayle. Zinc20 free ultralarge-scale chemical database for ligand discovery. Journal of chemical information and modeling, 60(12) 60(12):6065-6073, 2020.
[20] Jensen. A based genetic algorithm and generative model/Monte Carlo tree search for the exploration of chemical space. chem sci 10 (12): 3567-3572, 2019.
[21] Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning. PMLR, 2018.
[22] Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In International Conference on Machine Learning, pages 3183-3191 3191. PMLR, 2019.
[23] Kirthevasan Kandasamy. Jeff Schneider. Barnabás P6czos. High dimensional Bayesian optimisation and bandits via additive models. In International conference on machine learning, pages 295-304, PMLR, 2015.
[24] Diederik Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 132 2013.
[25] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 18899, 2014.
[26] Peter Kirkpatrick and Clare Ellis. Chemical space. Nature, 432(7019):823-3833 823, December 2004.
[27] Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru-Guzik, SELFIES: a robust representation of semantically constrained graphs with an example application in chemistry. CoRR, abs/1905. 13741, 2019. URL httl:/ttt://arriiii http::////rrii..///abaa/a/aa///a 19t.. 13741
[28] Matt Kusner, Brooks Paige, and José Miguel Hernández-Lobann Grammar variational autocncoder. In International Conference on Machine Learning, pages 1945-1954. PMLR, 2017.
[29] Xiaoyu Lu, Javier Gonzalez, Zhenwen Dai, and Neil Lawrence. Structured variationally auto-encoded optimization. In Jennifer Dy and Andreas Krause. editors, Proceedings the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3267-3275 Stockholmsmässan, Stockholm Sweden, 2018. PMLR.
[30] Jonas Mockus The Bayesian approach to global optimization. In System Modeling and Optimization. pages 473 481. Springer, 1982.

---

[31] Varnavas D Mouchlis. Antreas Afantitis, Angela Serra, Michele Fratello, Anastasios G Papadiamantis. Vassilis Aidinis. Iseult Lynch, Dario Greco, and Georgia Melagraki. Advances in de novo drug design: from conventional to machine learning methods. International journal of molecular sciences, 22(4):1676, 2021.
[32] Amin Nayebi, Alexander Munteanu, and Matthias Poloczek. A framework for Bayesian optimization in embedded subspaces. In International Conference on Machine Learning, pages
PMLR, 2019.
[33] John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal, 7 (4):308-313, 1965.
[34] Akshatkumar Nigam, Pascal Friederich, Mario Krenn, and Alan Aspuru Aspuru-Guzik. Guzik. Augmenting genetic algorithms with deep neural networks for exploring the chemical space. In International Conference on Learning Representations, 2020.
[35] Pascal Notin, José Miguel Hermández-Lobato, and Yarin Gal. Improving black-box optimization in VAE latent space using decoder uncertainty. Advances in Neural Information Processing Systems, 34, 2021.
[36] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):48, September 2017.
[37] Michael A Osborne, Roman Gamett, and Stephen J Roberts. Gaussian processes for global optimization. In 3rd international conference on learning and intelligent optimization (LION3), pages 1-15, 2009.
[38] Bidisha Samanta, Abir De, Gourhari Jana, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Gomez- Rodriguez. NeVAE: A deep generative model for molecular graphs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33 of 01, pages 1100111,, 2019.
[39] Marwin H. S. Segler, Thierry Kogej, Christian Tyrchan, and Mark P. Waller. Generating facused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4(1):120-131, 2018. doi: 10.102 /accsccentsci.70055511 12. URL htt.:hhtt////eet org/ 10 1tt.:0.1211//accc.e.e 29392184.
7b00512
PMID:
[40] Eero Säivola, Andrei Paleyes, Javier González, and Aki Vehtari. Good practices for Bayesian optimization of high dimensional structured spaces. Applied Al Letters, 2(2):224, 2021.
[41] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
[42] Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Artificial intelligence and statistics, pages 567-574. PMLR, 2009.
[43] Austin Tripp, Erik A. Daxberger, and José Miguel Hernández-Lobato Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems 33, 2020.
[44] Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artificial-intelliennee powered drug discovery. Nature machine intelligence, 4:189-191, 2020.
[45] Niek van Hilten, Florent Chevillard, and Peter Kolb. Virtual compound libraries in Computer-Assisted drug discovery. Journal of chemical information and modeling, 59(2)6644-651, Fehruary 2019.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pages 5998-60000 6008, 2017.
[47] Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large scale Bayesian optimiza- tion in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics, pages 745-754. PMI.R, 2018
[48] Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. Bayesian optimization in billion dimensions via random embeddings. Journal of Artificial Intelligence Research, 55:361-387, 2016.
[49] David Weininger. SMILES, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28 28(1):31-36 1988. doi: 10. 00.1021/c0005570000 URL https:://ubbe acs org/ httg//oo/aab//see 1021000000000000000000
12

---

[50] Kevin Williams, Elizabeth Bilsland, Andrew Sparkes, Wayne Aubrey, Michael Young, Larisa N Soldatova, Kurt De Grave, Jan Ramon, Michaela de Clare, Worachart Sirawarapom, Stephen G Oliver, and Ross D King. Cheaper faster drug development validated by the repositioning of drugs against neglected tropical discases, Journal of the Royal Society, Interface / the Royal Sociery, 12(104):20141288 March 2015.
[51] Andrew Wilson, Zhiting Hu, Russ Salakhutdinov, and Eric Xing. Stochastic variational deep kemel learning. Advances in Neural Information Processing Systems, 29:2586-2555 2016.
[52] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-3 378. PMLR, 2016.
[53] Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang. Weinan Zhang. Yong Yu, Li, MARS: Markov molecular sampling for multt--bjeective drug discovery. International Conference on Learning Represen- tations, 2021.
[54] Jiaxuan You, Bowen Liu, Zhitao Ying. Vijay Pande, and Jure Leskovec. Graph convolutional policy network for Goal- Directed molecular graph generation. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi, and R Garnett, editors, Advances in Neural Information Processing Systems 31, pages 6410-6421. Curran Associates, Inc., 2018.
[55] Muhan Zhang, Shali Jiang. Zhicheng Cui, Roman Gamett, and Yixin Chen. D-VAE: A variational autoencoder for directed acyclic graphs. In Advances Neural Information Processing Systems 32, pages 1588-1600. Curran Associates, Inc., 2019.
[56] Zhenpeng Zhou, Steven Kearnes, Li Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 911::10752, July 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] (see subsection A.7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] (see subsection A.7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] (See (SeeSeccion 5) where will provide link github repository our code)
(h) Did you specify all the training details (e.g., data splits, hyperparamcters, how they were chosen)?
[Yes] (See Section 5)
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [Yes] (Seefsection
(d) Did you include total amount of compute the type resources used (e.g.. type GPUs, internal cluster, or cloud provider)? [Yes] (See subsecctonn A.4)
4. If you using existing assets (e.g.. code, data, models) curating/releasing
assets...
(a) your work uses existing assets, did you cite the creators? [Yes] (See section S
(b) Did you mention license of the assets? [Yes] (See Subsectinn A.5
(c) Did you include any new assets either the supplemental material or as URL? [Yes] (See Section 5) where we will provide a link to a github repository with our code)
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "cPyzw-nJSNm", "reviewer": "Reviewer_RDPg", "summary": "The paper proposes a fused autoencoder/VGP approach which unlocks substantial improvements in Bayesian optimization on high dimensional spaces such as molecule graphs.", "questions": "Equation 2 has identical numerator/denominator terms with $q(u)$ and $\\Phi(Z|X)$, which seems incorrect for the variational approximation? (At a glance, should it be p/q and $\\frac{\\Gamma}{\\Phi}$)?\n\nWhy not compare SELFIES vs JTVAE in terms of score-achieved-by-iterations, as done w/ TuRBO vs LOL-BO in Table 1 rhs? Adding a row for either TuRBO-JTVAE or LOL-BO-JTVAE to Table 1 right would be helpful to set a clearer baseline.\n\n", "limitations": "Bayesopt on biologics might be a space where a societal impact statement might be in order. De novo biologicals design would constitute a mixed-use technology.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nThe paper brings together a few lines of work: Most importantly, it fuses trust region bayesopt (TuRBO)+sparse/variational GP with learned index point representations (via VAE), reconciling the trainable representation learning with the trust region approach by jointly training with a semisupervised objective. Secondly, the paper incorporates representation improvements (JTVAE=>SELFIES) which provide substantial posterior/eval speedups and higher chances of emitting valid representations in the discrete molecule space. That said, because \"all it does\" is bring these lines of work together, this work is more about unlocking performance as opposed to proposing a radically original idea -- it's somewhat incremental to TuRBO, to VGP, to SELFIES.\n\nQuality\nWell written, problem well framed. Experiments against multiple Guacamol targets + DRD3 as well as Expressions dataset go beyond toy problem to address interesting industry benchmarks. Ablation study is helpful for disentangling the contributions of joint training (LOL-BO) vs representation change (SELFIES). It is evident that both are contributing substantially. It would be nice to have \"best score by iterations\" type metrics for JTVAE vs SELFIES. Is this an issue of computation budget (100x more expensive JTVAE prohibitive for bayesopt)?.\nL310 spelling SELFIS\n\nClarity\nProblem space, experiments are well explained. Joint variational training approach is well explained. I wonder if there is an error in Eq 2 (see questions below).\n\nSignificance\nThe topic area (bayesopt on high dim discrete spaces) is timely and important, and one which the field has not fully solved to-date.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XQuF0cFBn8W", "reviewer": "Reviewer_B8Dz", "summary": "The paper deals with solving the high dimensional Bayesian optimization problem with latent space deep autoencoder models (DAE) primarily focusing on structured input. The idea revolves around the high dimensionality issues that persist even after projecting the structured input into the latent with DAE models. The primary novelty lies in efficiently incorporating a trust region in the Bayesian optimization step to deal with this high-dimensionality issue, not explicitly addressed in the earlier research. The empirical performance demonstrates significant improvements over the latest SOTA latent space Bayesian optimization algorithms across various complex real-world benchmarks. The paper tries to emphasize improving the optimization methodologies as well along with the improvement of Deep generative models and shows its effectiveness for latent space BO with a combination of both.\n", "questions": "1. What was the initial sample size for both the unsupervised and supervised cases? Can a comparison be provided with the other methods as well? I am specifically interested to see when the initial samples especially supervised information less how the performance compares? \n\n2. Joint-inferencing and using Transformer architectures will definitely require more initial sets of samples right? Unsupervised information can be available but there should be a proper justification for that. Hence, I am curious to understand the initial information comparison.\n\n3. TuRBO maintains m trust regions simultaneously. It uses a TS type approach to choose the best sample across the trust regions which can be paralleled and results in better efficiency. However, this work doesn't consider multiple trust regions, but still, there is an improvement over TurBo. Is it due to the Sparse GP Augmentation aspect with a global set of learned-inducing variables?", "limitations": "The authors have addressed the computation challenges to some extent and have carefully addressed the societal challenges which are appreciable. They clearly mention that experts should be heavily involved and that FDA guidelines for drug development and approval should be strictly followed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength revolves around addressing the mismatch b/w the trust-region in the latent space and  trust-region in the original structured input space. Although an extremely efficient formulation of trust region-based local optimization has been proposed by Eriksson et. al [2] that doesn't deal with the high-dimensionality with latent space BO. One of the most important contributions of this paper lies in formulating the trust-region for a high dimensional latent space Bayesian optimization problem. This paper points out a very interesting and critical point that even a small change in the latent space generated by BO i.e $||z - z'|| \\leq \\epsilon$ can be drastically different in the original space since the closeness in DAE is defined w.r.t to the decoder and BO doesn't explicitly incorporates that, hence it can still be that $||x - x'|| \\geq \\Delta$ which contradicts the purpose of trust-region based BO objective as pointed in Eriksson et. al. The main strength of this paper, according to me lies in the joint inference formulation which allows simultaneous optimization of GP with VAE parameters. The paper proposes an interesting methodology combining several popular LSBO/ Trust-region BO methods with State of the art DAE models in an intelligent framework. The empirical evaluations are quite explicit and detailed. In Figure 1, they show that LOL-BO (SELFIES) outperforms drastically all other benchmarks which are interesting. But what is even more critical is that the LOL-BO (JT-VAE) outperforms other baselines as well.\n\nThe primary weakness of this research lies in its uniqueness and novelty since it seemed like combining the best aspects of the latest advancements in the field of LSBO, Trust-region BO, and High dimensional BO in general. For ex: The semi-supervised variational inference was introduced by Kingma et. al and has been used to replace the weighted re-training mechanism popular in the LSBO literature by Miguel et al. Similarly, the paper by Eriksson et. al had already introduced the notion of trust-region in Bayesian optimization but it didn't have the latent space aspect to it. Also, the authors incorporate transformer layers on the recently published SELFIES representation which shows significant improvements. So, it's not extremely clear the exact novelty of this research. Another challenge lies in jointly learning multiple parameters and also using deep generative models should require a larger initial dataset both supervised and unsupervised and a comparison of that is not explicitly given as the major purpose of BO and GP, in general, is to be effective in a low-data regime where the initial supervised information is less. Hence, a comparison is missing with the initial supervised (and unsupervised) data on which the GP is trained? Also, related to this lies the concern in the simultaneous updating of the parameters, which the authors have dealt with various heuristics of periodically training, etc. However, there are no guarantees of convergence provided for the same.\n\n### Post Rebuttal\nI am happy with the author's response and increased my score.\n\n\nReferences :\n\n[1]. Austin Tripp, Erik A. Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems 33, 2020 \n\n[2]. David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In H. Wallach, H. Larochelle, A. Beygelzimer,  F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/6c990b7aca7bc7058f5e98ea909e924b-Paper.pdf \n\n[3]. Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander Imani Cowen370 Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, Jan Peters, and Haitham Bou-Ammar. High-dimensional Bayesian optimization with variational autoencoders and deep metric learning. CoRR, abs/2106.03609, 2021. URL https://arxiv.org/abs/2106.03609\n\n[4]. Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581–3589, 2014", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RUHEojCK9XB", "reviewer": "Reviewer_rFLV", "summary": "The paper considers the problem of optimizing black-box functions defined over structured inputs. The solution is situated within the general framework of Bayesian optimization (BO) over latent space where a deep-autoencoder model is learned from unsupervised data and subsequently used for employing standard Bayesian optimization algorithms in the continuous (latent) embedding. One challenge with this general approach is the high-dimensionality of the latent space which converts a black-box optimization problem over combinatorial space to that over high-dimensional continuous space. The paper proposes a trust region based BO algorithm in the latent space as a way to address this challenge. Based on the hypothesis that there is a mismatch between the trust region on the latent space and the corresponding trust region in the original structured space, a joint inference procedure is proposed over the Gaussian Process (GP) surrogate model and the deep autoencoder model i.e., treating the deep autoencoder as sort of a deep kernel for the GP model. Experiments are performed on multiple molecular optimization and arithmetic expression tasks.", "questions": "Please see above.", "limitations": "There is a good description of limitations in the supplementary section. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The considered problem of black-box functions defined over structured inputs is very significant with many real world applications including molecular design/drug discovery.\n\n- The idea of trust region Bayesian optimization for high-dimensional spaces is well known now especially with the work of Eriksson et al (TurBO)[1]. Therefore, the novelty of the proposed method is relatively limited but it does improve over the standard application of TurBO in latent space.\n\n- The paper presents empirical analysis for the proposed approach. I found the empirical analysis sound and effective in showing the efficacy of the approach on the overall BO performance. The proposed approach outperforms other baselines on multiple benchmarks and it is especially refreshing to see the discussion around usefulness of objectives like log-p metric in molecular optimization (line 308 to 321).\n\n- I found the literature review to be comprehensive and representative of the field.\n\nIn summary, the paper considers an important real-world application with an effective solution that outperforms multiple baselines. Barring some time complexity issues with the proposed inference procedure, it should be a useful method for practical applications.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept