Advancing Bayesian Optimization via Learning Correlated Latent Space
Authors: Seunghun Lee
Jaewon Chu
Sihyeon Kim
Juyeon Ko
Hyunwoo J. Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Bayesian optimization
smoothness regularization
variational autoencoder
TL;DR: We propose CoBO to narrow the inherent gap in latent Bayesian optimization with proposed two regularizations and a loss weighting scheme.

Abstract: Bayesian optimization is a powerful method for optimizing black-box functions with limited function evaluations. Recent works have shown that optimization in a latent space through deep generative models such as variational autoencoders leads to effective and efficient Bayesian optimization for structured or discrete data. However, as the optimization does not take place in the input space, it leads to an inherent gap that results in potentially suboptimal solutions. To alleviate the discrepancy, we propose Correlated latent space Bayesian Optimization (CoBO), which focuses on learning correlated latent spaces characterized by a strong correlation between the distances in the latent space and the distances within the objective function. Specifically, our method introduces Lipschitz regularization, loss weighting, and trust region recoordination to minimize the inherent gap around the promising areas. We demonstrate the effectiveness of our approach on several optimization tasks in discrete data, such as molecule design and arithmetic expression fitting, and achieve high performance within a small budget.

Advancing Bayesian Optimization via Learning Correlated Latent Space
Seunghun Lee, Jaewon Chu, Sihyeon Kim, Juyeon Ko, Hyunwoo J. Kim
Computer Science & Engineering
Korea University
(1lashh319, allonsy07, sh_bs15, juyon98, byumwoojkin/kkorea.ccm kr.
Abstract
Bayesian optimization is powerful method for optimizing black-box functions with limited function evaluations. Recent works have shown that optimization latent space through deep generative models such variational autoencoders leads to effective and efficient Bayesian optimization for structured or discrete data. However, the optimization does not take place in the input space, leads to an inherent gap that results in potentially suboptimal solutions. To alleviate the discrepancy, we propose Correlated latent space Bayesian Optimization (CoBO). which focuses on learning correlated latent spaces characterized by a strong cor- relation between the distances in the latent space and the distances within the ob- jective function. Specifically, our method introduces Lipschitz regularization, loss weighting, and trust region recoordination to minimize the inherent gap around the promising areas. We demonstrate the effectiveness of our approach on sev- eral optimization tasks in discrete data, such as molecule design and arithmetic expression fitting, and achieve high performance within small budget.
Introduction
Bayesian optimization (BO) is a standard method for wide range of science and engincering prob- lems such as chemical design 114. reinforcement learning [5], and hyperparameter tuning [6]. Re- lying surrogate model typically modeled with Gaussian process (GP), BO estimates the com- putationally expensive black-bol objective function to solve the problem with minimum number of function evaluations 77]. While it is known powerful method continuous domains [8] 9, applying BO is often obstructed by structured or discrete data, as the objective values are in a com- plex combinatorial space (1O). This challenge has motivated recent interest in latent space Bayesian optimization (LBO) methods [TH-14) which aim to find solutions in low-dimensional, continuous embeddings of the input data. By adopting deep generative models such as variational autoen- coders (VAEs) [15] to map the input space to the latent space, LBO has successfully addressed the difficulties of such optimization problems.
However, the fact that optimization is not directly conducted in the input space gives rise to inherent gaps, which may lead to failures in the optimization process. First, tween the input space and the latent space. Seminal works have been developed to address this gap, with generative models such B 3-VAE (16) emphasizing the importance of controlling loss weights, while WGAN [17] introduces improved regularization. Both aim to learn the improved latent space that better aligns with the input data distribution. Second, considering the BO problems, addi- tional gap emerges between the proximity of solutions in the latent space and the similarity of their
think of the gap be-
"cqual contributions "Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

black-box objective function values. This is observed in many prior works [18 19 that learn la- tent space by minimizing only reconstruction enrors without considering the surrogate model. This often leads to suboptimal optimization results. A recent study [00] has highlighted the significance of joint training between VAE and the surrogate model, yet it only implicitly encourages the latent space to align with the surrogate model. This limitation is observed in Figure Sa where the latent space's landscape, with respect to objective values, remains highly non-smooth with joint training.
To this end, we propose our method Correlated latent space Bayesian Optimization (CoBO) to ad- dress the inherent gaps in LBO. First, we aim to minimize the gap between the latent space and the objective function by increasing the correlation between the distance of latent vectors and the differ- ences in their objective values. By calculating the lower bound of the correlation, we introduce two regularizations and demonstrate their effectiveness in enhancing the correlation. Especially, one of these regularizations, called the Lipschitz regularization, encourages smoother latent space, allow- ing for a more efficient optimization process (see Figure 5b). Moreover, we suggest loss weighting with respect to objective values of each input data point to particularly minimize the gap between the input space and the latent space around promising areas of high objective values. Finally, we propose the concept of trust region recoordination to adjust the search space line with the updated latent space. We experimentally validate our method with qualitative and quantitative analyses on nine tasks using three benchmark datasets on molecule design and arithmetic expression fitting.
To summarize, our contributions are as follows: We propose Correlated latent space Bayesian Optimization (CoBO) to bridge the inherent gap in latent Bayesian optimization.
We introduce two regularizations align the latent space with the black-box objective function based on increasing the lower bound of the correlation between the Euclidean distance of latent vectors and the distance of their corresponding objective values.
We present a loss weighting scheme based on the objective values of input points, aiming to close the gap between the input space and the latent space focused on promising areas.
We demonstrate extensive experimental results and analyses nine tasks using three benchmark datasets on molecule design and arithmetic expression fitting and achieve state- of-the-art in all nine tasks.
2 Methods
In this section, we describe the main contributions of our method. Section 2.1 introduces several preliminaries on Bayesian optimization. In Section 2.2 we propose two regularizations to align the latent space with the black-box objective function. In Section 2.3 we describe our loss weighting scheme with the objective values. Lastly, in Section 2.4 we explain the overall architecture of our method.
2.1 Preliminaries
Bayesian optimization. Bayesian optimization (BO) 6 g mization method that aims to solve the problem
is a classical, sample-efficient opti-
x arg max (x),,
(1)
xEX
where X is a feasible set and black-box objective function. Since the function evaluation is assumed to be expensive, BO constructs probabilistic model of the black-box objective function. There are two main components of BO, first surrogate model g that provides posterior probability distribution over ff fx) conditioned on observed dataset D {(x; y:) based prior over objective function. Second is an acquisition function a for deciding the most promising next query point x;+1 based on the posterior distribution over fx). BO is a well-established method, however, applying BO to high-dimensional data can challenging due to the exponential growth the search space. To alleviate the challenge, recent approaches [10] 20 restrict the search space to a trust region centered on the current optimal input data point. In this paper, we adopt this regiin- based BO for handling high-dimensional search space.

---

Latent space Bayesian optimization. BO over structured or discrete input space X' is partieu- larly challenging. as search space over the objective function becomes large combinatorial one. In an effort to reduce a large combinatorial search space to continuous space, latent space Bayesian optimization (LBO)) [10
21 18 suggests BO over continuous latent space Z. A pre- 15 is commonly used as the mapping function, where the latent space is learned to follow the prior distribution (Gaussian distribution). Given a pretrained VAE, an en- coder 9s X + Z maps the input x; to the latent vector z; and the surrogate model g takes z; as the input. After the acquisition function a suggests the next latent query point Zi+1, decoder Pe 2 X' reconstructs z to X;+1, so that it can be evaluated by the black-box objective function, i.e., f(x(+1).
13
14 19
trained VAE = {go- P8j
LBO is promising optimization approach for discrete or structured inputs, yet, there are two main gaps to be considered. Firstly, there is gap between the input space and the latent space. Our focus is on narrowing the gap, especially within the promising areas, i.e. samples with high objective function values. Secondly, a gap exists between the proximity of solutions within the latent space and the similarity of their corresponding objective function values. This arises because the objective value originates from a black-box function in the discrete input space X. distinct from the latent space Z where our surrogate model g is learned. In the previous work, [0]] has suggested closing the gap by jointly optimizing VAE and the surrogate GP model to align the latent space with current top-k samples.
Here, we propose CoBO that explicitly addresses those two gaps by training a latent space with Lipschitz regularization, which increases the correlation between the distance of latent samples and the distance of objective values, and loss weighting with objective values to focus on relatively important search space.
2.2 Aligning the latent space with the objective function
Our primary goal is to align the latent space Z with the black-box objective function f. which can be achieved by increasing the correlation between the Euclidean distance of latent vectors and the differences in their corresponding objective values, i.e., Corr(lIz) the objective function f is an L-Lipschitz continuous function, we can establish a lower bound of Corr[[ 22|12, My y2l). In general, the function f is L--ipschitz continuous, it defined as
f2112, 81 y2l).. Assuming
vzi, 22 E R", (y())))), f(2))) S Ldz(zi, 22),
where dz and dy are distance metrics in their respective spaces for z and y. Then, the lower bound of Corr( [| 22|12, l22 ly1 //2) can be obtained as Theorem [
Theorem 1. Let Dz dz (,, 72) and Dy = dy (J(Z)) Z1, Za are i.i.d. random variables, I is an T--Lipschitz continuous function, and dz dy are distance functions. Then, the correlation between Dz and Dy is lower bounded
be random variables where
Dy S LDDz + CorrDz.Dv 2 10 + + Lpig
Vo
where JD2, ob PDv, and By are the mean and variance of Dz and Dy respectively.
Theorem iimplies that under the assumption of the L-Lipschitz continuity of f. we can increase the lower bound of Corr[] 22|12, 1y y/2) by reducing Lipschitz constant L while rbb.- ob2 b and of, remain constants. Based Theorem we propose two regularizations. The first one is Lipschitz regularization, which encourages a smooth latent space 2 w.r.t. the objective function f.
Σ max (0,
[i yı - L.) [z,
Lip
(3)
2,
where N is the number of training data points. Here, set the Lipschitz constant L as the median of all possible gradients of slopes. By penalizing slopes with larger gradients than an adaptively adjusted L, we encourage a decrease in L itself, leading learning correlated latent space.
Next, is beneficial to keep rbb.. ob ub, and ob, as constants. Given that 1.,,, and ob are function-specific constants, where the blx function is unchanged throughout the optimization,
3

---

we can treat them as fixed values. Then, we want to constrain nD as constant with the second regularization C. by penalizing the average distance between the latent vectors z be a constant c:
2
2 - 22
N2
JJSN
We set e as the expected Euclidean norm between two standard normal distributions, which is the prior of the variational autoencoder. That is the mean of the noncentral chi distribution [22] which is sum of squared independent normal random variables:
- vi)a = =E = 21( A
c= E V
(U,
U. ~ N
(5)
r(
Eq v3) WTW? = C, W. ~ N 0. (0.V2).
(6)
where [()) denotes the gamma function, C denotes the random variable with noncentral chi distri- bution NCxx and denotes the degrees freedom which is the same value as dimension n of the latent vector. Then is dependent only on the dimension of the latent vector, z R For By. preliminary observations indicate that it stays in reasonable range as long LLp is not overly penalized. Thus, we safely conclude that there is no need to explicitly constrain o We refer to the supplement for further analysis and the proof of Theorem
2.3 Loss weighting with objective values
Our focus now shifts to addressing the gap between the input space X and the latent space Z for LBO. Especially, we aim to minimize the gap in promising areas that offer better optimization oppor- tunities, i.e., significant points with high objective values. To achieve this, we prioritize input data points based on their respective objective values by weighting the reconstruction loss term. Follow- ing [23], we utilize the cumulative density function of the Gaussian distribution for the weighting. Specifically, the weighting function w.r.t. objective value y is:
X(y) P(Y yg),
with Y Ny, o²). where Y4 represents specific quantile of the distribution of Y. and hyperpa- rameter a denotes the standard deviation of Y. The weighted reconstruction loss is as follows:
Lreco__ = A(y) Creeon = - Exy)) (x)x; (og
(8)
Moreover, we also apply the weighting scheme to the Lipsehitz regularization term to promote smoother latent space when the objective value is higher. The weighted Lipschitz regularization is defined with the geometric mean of the weights of two input data points:
(AlUe)A(s)) max (0. 1y ツyl
Lip
Σ
T.)
2 2,112
JJEN
2.4 Overall architecture of CoBO
In this section, we explain the overall architecture of CoBO. We first introduce the training schema of latent space in CoBO to encourage high correlation between the distance in the latent space and the distance within the objective function. Next, we describe updating strategy the surrogate model for modeling the black-box function and further present generating procedure of the next candidate inputs for the black-box objective function through the acquisition function in the trust region. The overall procedure of our CoBO is in Algorithm
Learning the latent space, Our method learns the latent space by optimizing the encoder 4s and decoder pe of the pretrained VAE and updating the surrogate model in the latent space with our final loss:
LCOBO
LLip + L, Lrecun w + LXL. + L
(10)
LxL (z)g((
(11)

---

where Lipp_w and Lg is the regularization term in the latent space at Section 2.2 and 2.3 Crecon_ w is the weighted reconstruction loss term, Lxi, the KL divergence between the latent space distribution and the prior, and Laur is the loss for optimizing the surrogate model. We adopt the joint training scheme, training the surrogate model and the encoder do of the VAE model jointly [TO]. Under computational considerations, we retrain a latent space after Nan accumulated failure of updating the optimal objective value.
Updating the surrogate model. After After jointly optimizing the latent space and the surrogate model, we freeze the parameter of VAE and train our surrogate model. Note that this update is executed only after consecutive failures of updating optimal value, we also train the surrogate model in every iteration. As exact Gaussian process (GP), i.e., f(x) ~ GP(mmx), x')), where m is a mean function and is a covariance function, infeasible to handle large datasets due to cubic computa- tional complexity O(N3) for N data points, we employ sparse GP [24] as surrogate model which computationally efficient via inducing point method. To alleviate cubic complexity, sparse GP approximates black-box function with M < N pseudo-training samples called 'inducing points' that reduce complexity to O(M N2). We select the most widely used RBF kernel as sparse GP kernel function. Finally, we adopted deep kernel learning (DKL.) [5 in conjunction with sparse GP for our final surrogate model.
Generating candidates through acquisition function. Candidate samples for the acquisi- tion function are determined by random points in a trust region centered on the current op- timal value. We take a simple and powerful method, Thompson sampling as acquisition function within the context of Bayesian opti- mization. The surrogate model acts as the prior belief about our objective function. Thompson sampling uses this model to draw samples, leverag- ing its uncertainty to decide the next point to evaluate. In detail, we first select candidate samples in the trust region and score each candidate based on the posterior of the surrogate model to get the most promising values. Also, we recoordinate the center of the trust region to 2", which is obtained by passing the current optimal latent vector z into updated VAE, i.e., qo (pe (z")), as shown in Fig- ure [] We empirically showed that trust region recoordination helps to find better solution within limited budget (see Table Following [20), the base side length of the trust region is halved after consecutive failures of updating optimal objective value and doubled after consecutive successes.
Trust region
Z
2'
Latent space 2
Figure 1: Trust region recoordination.
Experiments
In this section, we demonstrate the effectiveness and efficiency of CoBO through various optimiza- tion benchmark tasks. We first introduce tasks and baselines. Then, in Section 3.1 we present evaluations of our method and the baselines for each task. In Section [3.2] we conduct an ablation study on the components of our approach. Finally, in Section 3.3 we provide qualitative analyses on the effects of our suggested regularizations and the necessity
regularization.
Tasks. We evaluate CoBO to nine tasks on a discrete space in three different Bayesian optimiza- tion benchmarks, which consist of arithmetic expression fitting tasks, molecule design tasks named dopamine receptor D3 (DRD3) in Therapeutics Data Commons (TDC) 26 and Guacamol bench- marks (27) The arithmetic expression task is generating polynomial expressions that are close to specific target expressions (e.g., 1/3+ x sin(r x 2)) [10 13 14 28], we set the number of ini- tialization points Wol to 40k, and max oracle calls to 500. In Guacamol benchmark, we select seven challenging tasks to achieve high objective value, Median molecules 2, Zaleplon MPO, Perindopril MPO, Osimertinib MPO, Ranolazine MPO, Aripiprazole similarity, and Valsartan SMART. The re- sults for the last three tasks are in the supplement. The goal of each task is to find molecules that have the most required properties. For every task of Guacamol benchmark, we set the number of initialization points to 10k, and max oracle calls to 70k. DRD3 task in the TDC benchmark aims to find molecules with the largest docking scores to target protein. In DRD3, the number of initial- ization points is set to 100, and the number of oracle calls set to 3k. We use SELFIES VAE [TO in Chemical design, and Grammar VAE 11 in arithmetic expression.
5

---

Algorithm Correlated Bayesian Optimization (CoBO)
Input: Pretrained VAE encoder Oá decoder po. black-box function f. surrogate model g, acquisition
function a, previously evaluated dataset Dg = {X,, Vi, zi) Ji= oracle budget T, latent update interval Nta batch size No, loss for surrogate model Lurrs proposed loss for joint training CCoBo
1: D + +D
2: Ppai +0
3: for t= = 1,
do D' +- D[-Ni :] (D)
4:
5:
if Meail 2 Nua then
6:
Tifall +0 Train qo. Pa. with LCOBO, D'
7:
Eq. 10
8:
Z < (x:( (xi, Mi, 2.) E D'}
9:
+ t D (
((pp(z))))))))))))))))()))))(( z:) i E }}
10:
end if
11:
Train g with Lsu. D' if * clse Do
12:
(x' "x",""Z"") arg (xX
13:
2' (p(((zz)))
trust region recoordination
14:
Get candidate set Zeand with random points trust region around 2*
15:
Zuext + argmaxz=Z.... ((z))
16:
if [Pol(Zneu)) I then Refall +- Thrail +
D DUDU
1 [poo (Zmexa) )), Zmex))
18:
end for
19:
return x
Baselines. We compare our methods with four BO baselines: LOL--O 10 W-LBO 13.
TuRBO (20) and LS-BO. LOL-BO proposed the joint loss close the gap between the latent space and the discrete input space. Additionally, W-LBO weighted data points to emphasize important samples with regard to their objective values. To handle discrete and structured data, we leverage TuRBO in latent space, TuRBO-L through encoder of pretrained frozen VAE and reconstruct it by decoder Py be evaluated by the objective function. For more details about TuRBO-L, see [10]. In our LS-BO approach, we adopt the standard LBO methodology. Specifically, we use a VAE with pre-trained parameters that are frozen. For sampling, the candidates are sampled from standard normal Gaussian distribution, NN(,
3.1 Results on diverse benchmark tasks
Figure represent the graphs that depict the number of oracle calls, i.e., the number of the black- box function evaluations, and the corresponding mean and standard deviation of objective value. The maximum number of oracles is set to 70k with Gucamol benchmarks, 3k with the DRD3 benchmark, and 500 with the arithmetic expression task. As shown in Figure 2 our method outperformed all the baselines in all the tasks. Note that the goal of the arithmetic task in Figure 3a and the DRD3 task Figure 3bb is minimizing the objective function, while others aim to maximize it. In the case of Figure 2a and Figure 2d LS-BO and TuRBO-L markedly failed to search for the one that has the best score. Especially in Figure 2a LS-BO and TuRBO-L, which only search with the fixed latent space, failed to update the initial best point despite searching 70k molecules. It implies that the pretrained VAE cannot generate better molecules unless the latent space is updated.
3.2 Ablation study
Table An ablation of CoBO's components.
Table An ablation on trust region recoordi- nation.
C,
((y)
Lip
Score
Recoordination
Score
0.7921
V
V
0.7835
0.7921
X
X
V
0.7733
X
0.6844
X
x
0.7504

---

0.38
0.00
0.34
0.55
0.32
0.50
10000 2000
40000
60000
80000
10000
20000
40000
80000
80000
Num Oracle
Num Oracle
(a) Median molecules 2 (med2)
(h) Zaleplon MPO (zalc)
0.75
0.70
0.65
0.55
0.50
1000
20000
40000
60000
80000
10000 20000
40000
60000
00000
Num Oracle
Num Oracle
(c) Perindopril MPO (ndop)
(d) Osimertinib MPO (osmb)
CoBO (Ours)
LOL-BO
W-LBO
TuRBO-L
LS-BO
Figure 2: Optimization results with four different tasks on the Guacamol benchmark. The lines and range are the mean and standard deviation three repetitions with the same parameters.
1.75
1:25
-300
Score
1.00
14.0
-145
0.25
0.00
-15.5
40000
40100
40200
40300
40400
40500
Num Oracle
Num Orecle
(a) Arithmetic expression
(b) TDC DRD3
CoBO (Ours)
LOL-BO
W-LBO
TuRBO-L
LS-BO
Figure 3: Optimization results with the arithmetic expression and TDC DRD3 benchmark. The lines and range are the mean and standard deviation of three repetitions with the same parameters.
In this section, we evaluate the main components of our model to analyze their contribution. We employ Perindopril MPO (pdop) task for our experiment and note that all scores of ablation studies are recorded when the oracle number 20k to compare each case in limited oracle numbers as the original BO intended.
We ablate the three main components of our CoBO and report the results Table As latent space regularization E, comes from Lipschitz regularization and loss weighting schema aims prioritize penalized input points, we conducted cascading experiments. Notably, we observed that perfor- mance decreased as the components were omitted, and the largest performance dropped (0.0229)

---

0.8
w/o Casor
yu Ma Manne
0.6
w/o Cangn
Correlation
Enaign
Pearson Correlation
0.5
Religo
0.4
0.4
/ Momp yary nay
0.3
0.2
0.2
0.1
0.0
0.0
-0.1
30
42
Epoch
Epoch
(a) Perindopril MPO (pdop)
(b) Arithmetic expression
Figure 4: Effects of Calggn The plot depicts the Pearson correlation between the distance of the latent vectors and the distance the objective values over the Bayesian optimization process. Each line represents training with a different loss for the latent space during the optimization process: one with Galign (orange) and the other without Lalign (bluc), where Lalign = LLip + Lzz We measure the correlation after every VAE update.
when Lipschitz regularization was not employed. Table reports the effectiveness of trust region re- coordination. We observe that applying trust region recoordination makes a considerable difference within a small oracle budget, as the score increases 0.6844 to 0.7921 with recoordination.
3.3 Analysis on proposed regularizations
All the analyses were conducted on the Perindopril MPO (pdop) task from the Guacamol benchmark. We include an additional analysis with the arithmetic data. For convenience, we define the Lalign as follows:
Lalign = Lrip L.,,
(12)
which is the loss that aligns the latent space with the black-box objective function.
Effects of Calign on correlation. In Section 2.2 we theoretically prove that Lalign increases the correlation between the distance in the latent space and the distance within the objective function. Here, we demonstrate our theorem with further quantitative analysis to show correlation changes during the Bayesian optimization process. Figure shows Pearson correlation value between the latent space distance ||z, Zz ||2 and the objective value distance wy: Vÿl. The blue and orange lines indicate the models with and without our align loss Lalign in Eq. 12 respectively. We measure the correlation with 10 data point and every 10 pair. The data is selected as the top 10 points with the highest value among 10 data, which are from training data for the VAE model and surrogate model. Over the training process, the Pearson correlation values with our align loss Latign were overall higher compared to the baseline. Moreover, Latign increases the Pearson correlation value high over 0.7 Figure 44a which normally regarded as the high correlation value. This means align loss increases the correlation between the distance of the latent vectors and the distance of the objective values effectively, leading to narrowing the gap between the latent space and the objective function.
Effects of Latign on smoothness. To validate that our model encourages the smooth latent space, we qualitatively analyze our model with Lalign and without Lalign by visualizing the landscape of the latent space after the training finishes. In Figure we visualize the top-k objective value and the corresponding latent vector in 2D space with the 2D scatter plot and the corresponding 3D plot for better understanding. We reduce the dimension of the latent vector to two dimensions using principal component analysis (PCA). The color means the normalized relative objective score value. The landscape of objective value according the latent space with Latign (right) is smoother than the case without Lalign (left). It demonstrates our Calign loss encourages smoothness of the latent space with respect to the objective function. Note that due to the inherent discreteness of the black-box objective function, it's expected to observe some spaces between the clusters in the 2D plot. Still, this does not detract from our purpose of effectively aligning latent vectors with respect to their discrete objective values.
8

---

1.00
Objective Value
Objective Value
0.67
0.67
0.33
0.33
0.00
(a) Without Calign
(b) With Lalign
Figure 5: Visualizations on the landscape of latent space for Lalign ablation. The scatter plot and corresponding 3D plot of the latent vectors with objective values. The landscape becomes much smoother with applying Lalign- A colorbar indicates the normalized objective value, where yellow means higher value and purple means lower value.
4 Related Works
4.1 Latent space Bayesian optimization
Latent space Bayesian optimization [10 12-14, 21 29 30 aims to resolve the issues in optimiza- tion over high-dimensional, or structured input space by introducing Bayesian optimization over la- tent space. As the objective for structured inputs is usually defined over large, complex space, the challenges can be alleviated by the lower-dimensional and continuous latent space. Variational au- toencoders (VAEs) [15] are commonly leveraged to learn the continuous embeddings for the latent space Bayesian optimizers. Some prior works propose novel architectures for decoders 12 BT- 33). while others introduce loss functions improve the surrogate for learning the objective func- tion B 14 18 [29]. Note that while the surrogate model (typically GP) is modeled based on the latent space, it is the input space which the objective value is obtained from. Although this results in an inherent gap in latent space Bayesian optimization, many prior works [12 14
21 29 do not update the generative model for the latent space. LOL-BO [10] seeks address this gap by adapting the latent space to the GP prior, while 13 suggests periodic weighted retraining to update the latent space.
4.2 Latent space regularization
Several previous studies in latent space have focused on learning the appropriate latent space for their tasks by incorporating additional regularizations constraints alongside the reconstruction loss. These approaches have been applied a wide range of areas, including molecule design (34]
36). domain adaptation [37 38 semantic segmentation [39 40 representation learning 441.33,
and reinforcement learning 44). Notably, the Lipschitz constraint is commonly employed to pro-
introduces Lipschitz regular- ization in learning implicit neural functions to encourage smooth latent space for 3D shapes while 45 penalizes the data pairs that violate the Lipschitz constraint in adversarial training. Addition- ally, CoFLO [46] leverages Lipschitz-like like regularization in latent space Bayesian optimization. Our method proposes regularization to close the gap inherent the latent space Bayesian optimization. Concretely, we introduce Lipschitz regularization to increase the correlation between the distance of latent space and the distance of objective value and give more weight to the loss the promising areas.
mote smoothness in diverse optimization problems. For example, 42

---

Conclusion and Discussion
In this paper, we addressed the problem of the inherent gap in the latent space Bayesian optimization and proposed Correlated latent space Bayesian Optimization. We introduce Lipschitz regularization which maximizes the correlation between the distance of latent space and the distance objective value to close the gap between latent space and objective value. Also, we reduced the gap between latent space and input space with loss weighting scheme, especially the promising areas. Ad- ditionally, by trust region recoordination, we adjust the trust regions according to the updated latent space. Our experiments on various benchmarks with molecule generation and arithmetic fitting tasks demonstrate that our CoBO significantly improves state-of-the of-the-at methods in LBO.
Limitations and broader impacts.
Given the contribution of this work to molecular design opti- mization, careful consideration should be given to its potential impacts on the generation of toxic or harmful substances in the design of new chemicals. We believe that our work primarily has the positive potential of accelerating the chemical and drug development process, setting a new standard latent space Bayesian optimization.
Acknowledgments and Disclosure of Funding
This work was partly supported by ICT Creative Consilience program (ITP--2033-200--001811)) supervised by the IIТР, the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2023RI A2222005332) and Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-ITI701-51.
References
[1] Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. Chembo: Bayesian optimization of small organic molecules with synthesizable recommendations. In ICAIS, 2020.
[2] José Miguel Hernández-Lobate James Requeima, Edward O Pyzer-Knapp. and Alán Aspuru- Guzik. Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In JCML, 2017.
[3] Ryan-Rhye Griffiths and José Miguel Hernandez-Lobato Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science, 2020.
[4] Ke Wang and Alexander W Dowling. Bayesian optimization for chemical products and func- tional materials. Current Opinion in Chemical Engineering, 2022.
[5] Roberto Calandra, André Seyfarth, Jan Peters, and Mare Peter Deisenroth. Bayesian optimiza- tion for learning gaits under uncertainty; An experimental comparison on a dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence, 2016.
[6] Jasper Snoek, Hugo Larochelle, and Ryan Adams. Practical bayesian optimization of ma- chine learning algorithms. In NeurIPS, 2012.
[7] Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of pensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Arxiv, 2010.
[8] Peter Frazier. A tutorial on bayesian optimization. Azxiv, 2018
[9] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 2015.
[10] Natalie Maus, Haydn Jones, Juston Moore, Matt Kusner, John Bradshaw, and Jacob Gardner. Local latent space bayesian optimization structured inputs. In NeurlPS, 2022.
[11] Matt Kusner, Brooks Paige, and José Miguel Hernández-Loohtto Grammar variational au- toencoder. In ICML., 2017.
10

---

[12] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML, 2018.
[13] Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimiza- tion in the latent space of deep generative models via weighted retrainin. In NeurIPS, 2020.
[14] Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I Jowen-Rivery Lin Yang. Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, et al. High- dimensional bayesian optimisation with variational autoencoders and deep metric learning. Arxiv, 2021.
[15] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[16] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual con- cepts with a constrained variational framework. In ICLR, 2017.
[17] Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan. Arxiv, 2017.
[18] Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for bayesian optimization over combinatorial spaces. In NeurlPS, 2021.
[19] Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sänchez-Lennglling Dennis Sheberla, Jorge Aguilera-Iparragee Timothy D Hirzel, Ryan Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data- driven continuous representation of molecules. ACS central science, 2018.
[20] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scal- able global optimization via local bayesian optimization. In NeurlPS, 2019.
[21] Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In ICLR, 2022.
[22] John Lawrence. Moments of the noncentral chi distribution. Sankhya A, 2021.
[23] David H Brookes and Jennifer Listgarten. Design by adaptive sampling. Arxiv, 2018.
[24] Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In NeurIPS, 2005.
[25] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In AISTATS, 2016.
[26] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In NeurlPS, 2021.
[27] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain Vaucher. Guacamol: bench- marking models for de novo molecular design. Journal of chemical information and modeling, 2019.
[28] Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimiza- tion with genetic exploration. In NeurIPS, 2020.
[29] Stephan Eissman, Daniel Levy, Rui Shu, Stefan Bartzsch, and Stefano Ermon. Bayesian opti- mization and attribute adjustment. In UAI, 2018.
[30] Eero Siivola, Andrei Paleyes, Javier González, and Aki Vehtari. Good practices for bayesian optimization of high dimensional structured spaces. Applied Al Letters, 2021.
[31] Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Kumar Chattaraj. Niloy Ganguly, and Manuel GodezzRodriguez. Nevae: A deep generative model for molecular graphs. In AAAI, 2019.

---

[32] Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In ICML, 2019.
[33] Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. Syntax-directed variational autoencoder for structured data. In ICLR, 2018.
[34] Egbert Castro, Abhinav Godavarthi, Julian Rubinfien, Kevin Givechian, Dhananjay Bhaskar, and Smita Krishnaswamy. Transformer-based protein generation with regularized latent space optimization. Nature Machine Intelligence, 2022.
[35] Pascal Notin, José Miguel Hernández- and Yarin Gal. Improving black-box optimiza- tion in vae latent space using decoder uncertainty. In NeurlPS, 2021.
[36] ANM Abeer. Nathan Urban, M Ryan Weil, Francis J Alexander, and Byung-Jun Yoon. Multi- objective latent space optimization of generative molecular design models. Arxiv, 2022.
[37] Guoliang Kang, Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019.
[38] Lei Tian, Yongqiang Tang, Liangchen Hu, Zhida Ren, and Wensheng Zhang. Domain adapta- tion by class centroid matching and local manifold self-learning. IEEE Transactions on Image Processing, 2020.
[39] Francesco Barbato, Marco Toldo, Umberto Michieli, and Pietro Zanuttigh. Latent space regu- larization for unsupervised domain adaptation in semantic segmentation. In CVPRW, 2021.
[40] Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion- attraction of sparse and disentangled latent representations. In CVPR, 2021.
[41] Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto- encoders. In NeurlPS, 2021.
[42] Hsueh-Ti Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. In SIGGRAPH, 2022.
[43] Ramana Subramanyam Sundararaman, Riccardo Marin, Emanuele Rodola, and Maks Ovs- janikov. Reduced representation of deformation fields for effective non-rigid shape matching. In NeurlPS, 2022.
[44] Mete Kemertas and Tristan Aumentado-Armstrony Towards robust bisimulation metric learn- ing. In NeurlPS, 2021.
[45] Dávid Terjék. Adversarial lipschitz regularization. In ICLR, 2020.
[46] Fengxue Zhang, Yair Altas, Louise Fan, Kaustubh Vinchure, Brian Nord, and Yuxin Chen. Design of physical experiments via collision-free latent space optimization. In NeurIPSW. 2020.
12


Reviews:
[{"review_id": "zKJS63D4kg", "reviewer": "Reviewer_Bvzj", "summary": "This paper proposes a latent space Bayesian Optimization approach based on the intuition that distances in the latent space should be correlated with differences in objective value. CoBO iteratively updates a variational autoencoder (VAE) to align distances in latent space with differences in objective function. The method leverages a lower bound on correlation to introduce new regularizers on the VAE objective. Combined with loss weighting and latent space “recoordination”, the method achieves state-of-the-art performance on diverse benchmarks.", "questions": "1. The paper ignores the fact that the VAE decoder in many applications is stochastic (e.g. the SMILES decoder from Maus et al.). This would motivate a probabilistic treatment of f(Z) in Theorem 1.\n2. In Theorem 1, won’t it always be the case that $D_Y \\le D_Z$ by the Lipschitz assumption? Why is this stated as a sufficient condition?\n3. On line 120, is this Lipschitz constant computed as the median __per mini-batch__?\n4. Equations 4-6 do not appear to be consistent. You derive the distribution of inter-point distances, so why regularize all distances to be the same? Do you think of this as an additional discrepancy measure between your VAE prior and posterior? Can this be connected to the MMD regularization in InfoVAEs?\n5. The paper describes the method in terms of standard Gaussian Process kernels. Do you not use a deep kernel, even for the drug discovery tasks?\n6. How are hyperparameters determined, particularly in the ablation study?\n7. How do you choose the initialization points?\n8. Does Figure 4 include the recoordination step?\n9. In Figures 4-5, would it be possible to compare to the prior state-of-the-art (LOL-BO) to substantiate the claim that that method does not effectively reorganize the latent space, rather than simply comparing to a fixed latent space?", "limitations": "I believe that CoBO can only be expected to be effective on objectives that are Lipschitz continuous; this limitation is not mentioned. The author’s consideration of possible malicious uses for drug discovery models is much appreciated.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper mostly applies existing techniques to achieve incremental but statistically significant improvement over prior state-of-the-art. Theorem 1 provides a useful result that is a significant contribution of its own, if it is in fact a new bound.\n\nThe paper is written very well and will likely serve as a benchmark for future BO algorithms. Algorithm 1 provides a nice, approachable, and thorough description of CoBO. Figures 4-5 show strong evidence that the CoBO objective has a strong positive effect on organizing the latent space.\n\nThe experimental results are convincing. An ablation study justifies the inclusion of all components of the proposed VAE objective, albeit on only one optimization benchmark (hopefully these results hold for all benchmarks studied).\n\nI would suggest trying to fit Table 3 from the Supplemental into the body of the paper, as it would strengthen the experimental evaluation. The results show strong performance especially in the low-budget setting.", "weaknesses": "It does not seem natural to assume that the mapping $f$ is Lipschitz for a VAE trained in an unsupervised manner. Won’t it be the case that the objective is not Lipschits in the input space of many useful models, especially over discrete spaces? Specifically, do you have any experiments demonstrating Lipschitz continuity of the objective by sampling points in the latent space?\n\n_Note that the “recoordination” technique claimed by this paper previously existed in Maus et al. under the name “recentering”._\n\nEvaluation is a little limited. Do the results hold for all Guacamole tasks? For example, Maus et al evaluates on additional GuacaMol tasks such as ranolazine.\n\nEvalutions on DRD3 only go up to 1,000 evaluations of the objective function. This is understandable given the expense of running the docking objective, but on the other hand, this is the most “real-world” objective studied. Would it be possible to run for longer to determine whether CoBO can eventually beat LOL-BO or GraphGA?\n\nThe axes for all plots start at 0; it would be more helpful to include initialization points in the evaluation budget for comparison with prior work.\n\nSpecific issues:\n* Articles are missing in several locations, including in the abstract (line 12). Please proofread for small errors.\n* Equation 4 shows a regularizer that encourages __every__ pair of distances to be exactly $c$; it does not regularize the mean as claimed in the text. Is the text or the equation incorrect?\n* Do you recoordinate all points during fine tuning? If so, add this step to Algorithm 1 (I believe it currently only shows recentering for the trust region, not the surrogate).\n* Is there a reason that you report the objectives in Arithmetic expressions as a minization problem, but transform the DRD3 task into a maximization task? Retaining the convention of the TDC leaderboard would improve the accessibility of the paper to a wide audience.\n\nSmall comments:\n* Define “an inherent gap” in the abstract. This notion is referred to multiple times in the paper, but never precisely. Is the claim that there do not exist points in the latent space $z$ that correspond to an optimal point in input space $x$? Or that these regions are small and hard to find?\n* It is not immediately obvious in the introduction that this method will leverage a pre-trained VAE that is trained in a purely unsupervised manner.\n* In Equation 7, notation seems to be overloaded. Do you mean a quintile of the empirical distribution of objective values seen during training, $y_i$? At line 140, is $sigma$ a hyperparameter or equal to $\\sigma_Y$?\n* Algorithm 1 line 14: typo “is the best score than”\n* When discussing “initialization points” in Section 3, it would be useful to the the notation $D_0$ from Algorithm 1.\n* In Figure 2, should “TDL” be “TDC”?\n* It would be helpful to label the y-axes in Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough and detailed feedback. We will address the issues below.\n\n**Q1. More evaluations on DRD3.**\n\nWe conducted the experiment with DRD3 for a duration of up to 5 days, reaching a maximum of 2500 oracle calls within the available resources. We also provide the optimization curve in the PDF file. CoBO shows the superior performance over baselines.\n\n| DRD3 | CoBO (Ours) | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **15.6** | 14.0 | 14.2 | 12.4 | 12.2 |\n| Oracle calls | **2500** | **2500** | **2500** | **2500** | **2500** |\n|  |  |  |  |  |  |\n\n**Q2. Additional tasks for Guacamol.**\n\nWe report three additional tasks (rano, valt, adip) for Guacamol dataset. With limited resources, we set the Oracle budget to 30K. To ensure fairness, we also include the Found Best Score for the same wall-clock time. We also provide the optimization curve in the PDF file. It demonstrated superior performance in additional tasks, aligning with the results of the main paper.\n\n| rano | CoBO (Ours) | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.9548** | 0.9244 | 0.9155 | 0.8184 | 0.7775 |\n| Oracle calls | **50K** | **50K** | **50K** | **50K** | **50K** |\n| Wall clock time (hr) | 14.5 | 11.8 | 9.1 | 5.8 | **4.8** |\n|  |  |  |  |  |  |\n| Found Best Score | **0.9530** | 0.8329 | 0.9014 | 0.8184 | 0.7775 |\n| Oracle calls | 32K | 15K | 17K | 40K | **50K** |\n| Wall clock time (hr) | **4.8** | **4.8** | **4.8** | **4.8** | **4.8** |\n|  |  |  |  |  |  |\n\n| valt | CoBO (Ours)  | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.9981** | 0.9961 | 0.9727 | 5e-15 | 2e-31 |\n| Oracle calls | **50K** | **50K** | **50K** | **50K** | **50K** |\n| Wall clock time (hr) | **3.2** | 10.9 | 21 | 11.5 | 8.0 |\n|  |  |  |  |  |  |\n| Found Best Score | **0.9981** | 0.9759 | 2e-26 | 2e-26 | 2e-31 |\n| Oracle calls | **50K** | 25K | 10K | 15K | 20K |\n| Wall clock time (hr) | **3.2** | **3.2** | **3.2** | **3.2** | **3.2** |\n|  |  |  |  |  |  |\n\n| adip | CoBO (Ours)  | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.8321** | 0.7746 | 0.7817 | 0.6983 | 0.6983 |\n| Oracle calls | **50K** | **50K** | **50K** | **50K** | **50K** |\n| Wall clock time (hr) | 27 | 6.7 | 32.6 | **3.1** | **3.1** |\n|  |  |  |  |  |  |\n| Found Best Score | **0.8321** | 0.7518 | 0.7534 | 0.6983 | 0.6983 |\n| Oracle calls | 27K | 33K | 14K | **50K** | **50K** |\n| Wall clock time (hr) | **3.1** | **3.1** | **3.1** | **3.1** | **3.1** |\n|  |  |  |  |  |  |\n\n**Q3. Is it natural to assume that the mapping f is Lipschitz for a VAE pretrained unsupervisedly, especially over discrete spaces?**\n\nGood question! We do NOT assume that the pretrained VAE is Lipschitz continuous. That’s why we proposed the regularizers in Eq. 3 and 4 in the main paper. Also, we observe that although the input/output of VAE are discrete values, except for the first and last layers, the VAE internally treats the data like continuous variables. All intermediate features including latent variable $z$ are continuous values. Therefore, imposing a Lipschitz constraint on VAE is a reasonable remedy to improve the latent space and reduce the gap between the latent space and the unknown objective function.\n\n**Q4. Is the claim that there do not exist points in the latent space that correspond to an optimal point in input space? Or that these regions are small and hard to find?**\n\nGreat question! In general, the latent space has a point that corresponds to an optimal solution in the input space. First, the VAE returns a probability mass function with p in (0,1). Since it cannot be strictly 0, or 1, the generated PMF covers all possible discrete solutions. In addition, the Gaussian distribution in the continuous latent space has a infinite support so the latent vector that corresponds to an optimal solution has non-zero density. Indeed, multiple latent vectors correspond to the same optimal solution. In sum, the optimal point must exist in the latent space but the regions can be small or the likelihood to get the optimal latent vector could be small.\n\n**Q5. The stochastic nature of the VAE decoder.**\n\nWe treat the decoder as deterministic by considering the expectation $\\mathbb{E}_\\theta(p\\_\\theta(\\mathbf{x}|\\mathbf{z}))$. We acknowledge that considering the stochastic nature of the decoder can provide a more precise understanding, thus we will reflect to our future works.\n\n**Q6. About \"recoordination\".**\n\nThe concept of both [10] and CoBO is similar, but there are some differences in implementation, especially in deciding the center. In [10], there's a possibility of retrieving the latent space before updating VAE, due to implementation issues. [10] determines the center based on latent space $\\mathbf{z}$ with the decoded input w.r.t. the best objective value $y$, whereas ours uses the latent space $\\mathbf{z}$ with the input w.r.t. the best objective value $y$. We empirically showed better performance over [10]. Also, we only recoordinate the center of the trust region, rather than all points during fine-tuning.\n\n**Q7. Do you have experiments showing Lipschitz continuity of the objective?**\n\nWe agree that understanding the Lipschitz continuity of the objective value is an essential aspect of our study. To demonstrate the Lipschitz continuity, we measure the change of Lipschitz loss over multiple methods (i.e., we do not utilize Lipschitz loss for baselines). The curve is in the PDF file. The Lipschitz loss increased at the beginning of the training process, due to the differences in the observed objective values.\n\n**Q8. Additional questions/suggestions.**\n\nDue to limited space, we were unable to address all raised issues. We commit to correcting all typos and clarifying notations in line with your valuable suggestions, for the final version."}, {"review_id": "QwbBnkACzb", "reviewer": "Reviewer_4zSv", "summary": "This paper proposes several heuristic regularization constraints for learning a Bayesian optimization latent space. It argues that the learned latent space needs to be aligned to the black-box function values, and this is achieved via keeping the Lipschitz constant small and the mean latent distance (of training samples) constant. It also argues that promising points need to be prioritized during the latent space optimization, and this is achieved via weighting the reconstruction loss and the Lipschitz regularization loss above at each training point by its function values. The paper evaluates the proposed method on several benchmark functions, yielding positive results.", "questions": "1. I don't understand why $\\mu_{D_Z}$ and $\\sigma_{D_Z}$ are not constants the same way $\\mu_{D_Y}$ and $\\sigma_{D_Y}$ are? And what are they constants with respect to?\n\n2. Eq. (4) seems like a much stricter objective than what is needed -- it means that the points are optimized towards being equidistant. Also, when every $||z_i - z_j||_2 \\simeq c$, what will remain for L_lip to optimize?\n\n3. Why does blurring y with Gaussian noise in Eq. (7) make sense? Couldn't we just directly let $\\lambda(y) = y_q - y$  (and maybe normalize this by y_max so that the weight is between 0/1) ?\n", "limitations": "The authors have discussed broader impacts & limitations. I don't see any potential negative social impact.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed approach makes some practical sense, and yields good performance on several benchmarks. The approach is also novel, so I believe the paper has some intellectual and practical merits. ", "weaknesses": "1. I think the problem is not very clearly described and motivated. What exactly are the \"gaps\" between the latent space and the input space? Throughout the paper, I have not seen any technical description of this issue. To quote the manuscript, the two main gaps are:\n- \"First, even though the surrogate model g learned in the latent space the objective value is still obtained by a black-box function defined in the discrete input space X so the gap between objective values and latent spaces leads to poor optimization performance\".\n- \"Second, since the distribution of samples expected to have high objective values is different from that of samples observed in pretraining, there is a gap between the input space and the latent space that makes the optimization inaccurate\".\n\nNeither statements give a precise description of what these gaps are, so I cannot be convinced that the proposed solutions are meaningful.\n\n2. A lot of the technical details also lack motivation. Please refer to the specific questions below.\n\n3. Different BO approaches have different per-iteration cost, and it would be slightly unfair to only compare the performance vs. number of oracle calls. I would strongly suggest providing a plot to show performance vs. wall-clock time, or a table documenting the runtime per iteration of each baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. Below, we present comprehensive responses addressing the questions raised by the reviewer.\n\n**Q1. Performance comparison w.r.t. wall-clock time.**\n\nWe provide the results on Guacamol dataset with respective tasks with respect to wall-clock time as requested. We report the Found Best Score corresponding to the same wall-clock time,  determined by the fastest baseline model to reach the 70K oracle calls.\n| zale | CoBO (Ours) | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.773** | 0.745 | 0.606 | 0.566 | 0.511 |\n| Oracle calls | 53K | 33K | 13K | 46K | **70K** |\n| Wall clock time (hr) | **2.6** | **2.6** | **2.6** | **2.6** | **2.6** |\n|  |  |  |  |  |  |\n\n| med2 | CoBO (Ours) | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.379** | 0.352 | 0.330 | 0.312 | 0.312 |\n| Oracle calls | 58K | 35K | 20K | **70K** | 58K |\n| Wall clock time (hr) | **2.9** | **2.9** | **2.9** | **2.9** | **2.9** |\n|  |  |  |  |  |  |\n\n| osmb | CoBO (Ours) | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.928** | 0.910 | 0.898 | 0.834 | 0.844 |\n| Oracle calls | 67K | 43K | 21K | **70K** | 64K |\n| Wall clock time (hr) | **3.5** | **3.5** | **3.5** | **3.5** | **3.5** |\n|  |  |  |  |  |  |\n\n| pdop | CoBO (Ours) | LOL-BO | W-LBO | TuRBO | LS-BO |\n| --- | --- | --- | --- | --- | --- |\n| Found Best Score | **0.834** | 0.796 | 0.635 | 0.587 | 0.565 |\n| Oracle calls | 41K | 35K | 17K | **70K** | 52K |\n| Wall clock time (hr) | **2.7** | **2.7** | **2.7** | **2.7** | **2.7** |\n|  |  |  |  |  |  |\n\n**Q2. Why use Gaussian noise in Eq. (7) for blurring y instead of directly using $\\lambda(y)=y_q-y$?**\n\nWe opted for the weight function from [21], as it demonstrated superior performance in contrast to other weight functions. We performed preliminary experiments for weight function selection on Guacamol (w/ pdop). We provide the experimental results and formulations of each weight function in the PDF file. Among these weight functions, Rwr closely aligns with your suggestion (with the addition of a temperature parameter). It is clear that [21] (indicated by the blue line) showcases the most effective performance.\n\n**Q3. Why are $\\mu_{D_Z}$ and $\\sigma_{D_Z}$ are not constants the same way $\\mu_{D_Y}$ and $\\sigma_{D_Y}$ are? And what are they constants with respect to?**\n\nSince the input data points are associated with fixed objective values of the black box function, we can consider the $\\mu_{D_Y}$ and $\\sigma_{D_Y}$ as constants. However, the distance in the latent space (i.e., ${D_Z}$) depends on the mapping function (encoder). Because the mapping function can be changed during the optimization process, $\\mu_{D_Z}$  and $\\sigma_{D_Z}$  are not treated as constants.\n\n\n\n"}, {"review_id": "Hzp4V5o0kG", "reviewer": "Reviewer_aw6W", "summary": "The paper addresses the problem of Bayesian optimization with the help of a low-dimensional latent space (here learned using a VAE augmented with ad-hoc loss terms).\n\nThe paper contributes:\n- An analysis/recap of the state-of-the-art highlighting issues raised in other works, particularly the need for smoothness (in the latent space) around the optima.\n- A concrete loss function augmenting previous work [10] with two additional regularisation terms to ensure smoothness in the latent space w.r.t. the objective function.\n- Experimental study demonstrating the advantages over a suitable set of benchmarks on six problems.\n- An ablation study examining the effect of some of the loss terms on the final BO task.", "questions": "The questions are included in the above.", "limitations": "See above for technical limitations/issues. There is no need to consider the broader societal impact of the specific paper.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-\tImportant and relevant research area.\n-\tThe background and motivation for the derived algorithm are well-described.\n-\tThe included experiments indicate improvements compared to a set of reasonable baselines.\n-\tThe resulting algorithm seems sufficiently novel and builds on the intuition about smoothens w.r.t. to the objective function, but I have some concerns/questions…\n", "weaknesses": "Overall weaknesses/comments/suggestions:\n\n-\tThe setup does not come across as principled or consistent from a pure modeling perspective…\n  -\tI appreciate the algorithmic-centered view (vs. a probabilistic modeling perspective) adopted in the paper. Still, I am always a bit skeptical when rather ad-hoc loss terms are added to the VAE loss or the joint setup from [10] (which both originate from a proper probabilistic analysis). The combination of and interaction among the many loss terms (in particular the $L_z$, $L_lip$, vs. the VAE prior) would need quite a lot of analysis to work out the exact effect of the combination (and weighting). Currently, I feel the paper lacks this insight.\n - $z$ is a random variable, yet the computation of the 2-norm in Eq. 9 is performed without recognizing this. I’d expect an expectation w.r.t.. q(z|x) to be involved in $L_z$ and $L_{lip_W}$ (or a non-central chi distribution if treated analytically)?\n -\tI am unsure why $L_z$ is needed – can this not be controlled directly via the variance on the prior, $p(z)$? Can the smoothness constraints be incorporated/formulated as a more principled prior for z, thus providing a complete probabilistic view and derivation of the algorithm?\n-\tMissing assumptions. \n -\tI think it needs to be argued that it makes sense to globally measure distance in the latent space using a Euclidian distance function.\n - It is generally unclear if you assume y is noiseless or not. This seems important in Eq. 9 (and Theorem 1), i.e., do you use $p(y|x, \\cdot)$ or simply the observed y when computing $|y_i-y_j$? \n-\tThe model/loss is not fully specified in the paper, as far as I can tell. Specifically,\n -\tThe weights on the loss terms are left out in Eq 11, giving the impression that there are no hyperparameters related to the loss itself, yet the supplementary clearly indicates that some manual fine-tuning is needed. The main paper should be transparent about this. How sensitive is the performance to the weights on the loss terms, and how should they be set in practice? \n -\tl 155: $L_{joint}$ should be explained in more detail to provide a self-contained definition of the model/algorithm.\n -\tThere are no details in the paper or supplementary about the likelihood ($p(x|z)$); I’d suggest adding more information about this beyond a reference to the particular VAEs.\n -\tThere are no details about the structure and complexity of the problems or the VAE (e.g. dimensionality of z and its influence)$); I’d suggest providing this at least in the supplementary.\n-\tExperimental suggestions: \n -\tIf possible, I would suggest providing a very simple 2D (maybe 3D ) synthetic example (trained with the proposed model) to provide the reader with a better intuition about what’s going on with the many loss terms (c.f. previous comment).\n -\tThe ablation study (on one dataset) is interesting yet should probably include removing the $L_{joint}$ term as well.\n\n\nMinor questions/comments:\n\n-\tI’d suggest providing (possibly in the appendix) an optimization trace for one example with the total loss and all the individual loss terms included individually.\n-\tl120: I think adaptively changing setting L using the median would require more justification and explanation to clarify the properties of this scheme.\n-\tl212: I am unsure what is meant with “….Thompson sampling in $N(0,I)$...”?\n-\tFigure 5: Is this a result based on a VAE with a latent space with dimension 2 or a projection onto 2D?\n - l 263: Fig 5 (b) seems less smooth than Fig 5 (a) as is; consider making the statement clearer (i.e. wrt to the objective)\n-\tMost figures lack indication of axis labels (on the figure itself or in the caption)\n-\tIt would be helpful with a few more words/details attached to Eq 7 (the proof) in the supplementary material.\n-\tSome sections contain grammatical issues interrupting the flow, e.g. sec 2.4, l. 165  (missing “the”, “a” etc.)\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. We present comprehensive responses below.\n\n**Q1. Unsure why $\\mathcal{L}\\_z$ is needed**\n\nThe $\\mathcal{L}\\_{\\mathbf{z}}$ serves a specific purpose in our approach. If we were to rely solely on the Lipschitz loss and exclude $\\mathcal{L}\\_{\\mathbf{z}}$, it could lead to a trivial solution where the scale of $\\mathbf{z}$ is merely growing. This would be shown in the expression ${|y\\_i-y\\_j| \\over ||\\mathbf{z}\\_i-\\mathbf{z}\\_j||\\_2}$, which is the Lipschitz loss we minimize, resulting in a collapse of the distance metric in the denominator. To prevent this and ensure a meaningful scaling of $\\mathbf{z}$, we introduced the $\\mathcal{L}\\_{\\mathbf{z}}$ loss term. Furthermore, as described in lines 124-125, this term is closely related to Theorem 1, providing additional theoretical justification for its inclusion. We agree that the VAE prior can control these aspects, however, we expect the current formulation of $\\mathcal{L}\\_{\\mathbf{z}}$ provide a targeted and effective solution to a specific challenge we identified in our approach. We also empirically verified its effectiveness.\n\n**Q2. I think it needs to be argued that it makes sense to globally measure distance in the latent space using a Euclidian distance function.**\n\nWe appreciate your insight regarding the measurement of distance function. In the context of the Variational Autoencoder, the Euclidean space assumption is widely adopted [a], and this assumption guided our choice to measure distances in the latent space using the Euclidean distance function. It would be good for future work to define and use the geodesic distance on the manifold based on the Riemannian manifold [b]. \n\n[a] Kalatzis, D., Eklund, D., Arvanitidis, G., & Hauberg, S. Variational Autoencoders with Riemannian Brownian Motion Priors. International Conference on Machine Learning, 2020.\n[b] Pennec, X. Intrinsic statistics on Riemannian manifolds: Basic tools for geometric measurements. Journal of Mathematical Imaging and Vision, 2006.\n\n**Q3. l 263: Fig 5 (b) seems less smooth than Fig 5 (a) as is; consider making the statement clearer.**\n\nWe understand your point regarding Figure 5 (b) appearing less smooth than Figure 5 (a). It is possible to interpret the discontinuous portion in the middle of Figure 5 (b) as lacking smoothness. However, our intention was to illustrate the smoothness of the objective value at each location, rather than the visual continuity of the plot itself. Each color in the figure represents a different objective value, and we aimed to show that nearby points have similar colors, reflecting similar objective values. We will clarify the explanation of visual representation in Figure 5 (b).\n\n**Q4. l120: I think adaptively changing setting L using the median would require more justification and explanation to clarify the properties of this scheme.**\n\nWe choose $L$ as the median to create a more robust and adaptive mechanism that can respond to varying properties within the data. This adaptive process allows the model to dynamically align with the characteristics of the data at different stages of training or across different datasets. This alleviates sensitiveness of the data and enhance stability in the learning process. We will provide additional analysis in the final version.\n\n**Q5. Unsure on “….Thompson sampling in $N(0,I)$...”?**\n\nWe aim to convey is that in our approach using Gaussian Processes (GP), we sample the function and then draw candidate samples from the normal distribution $N(0,I)$ to find the optimal input value. \n\n**Q6. Do you use or simply the observed y when computing $|y\\_i-y\\_j|$?**\n\nIn all the tasks we run, the objective function was noiseless, and we also assumed that y is noiseless. We will clarify this in the final version.\n\n**Q7. How does the combination and interaction of the various loss terms (such as $\\mathcal{L}\\_z$, $\\mathcal{L}\\_{lip}$, and the VAE prior) work, and what analysis has been done to understand their exact effect, considering the current paper seems to lack this insight?**\n\nThe design of these loss terms each has a crucial role to achieve a specific goal including Lipschitz regularization and latent space regularization. Also, we showcased and analyzed that these loss terms lead to effective optimization in the latent space. We agree that further theoretical analysis would make the paper stronger and will add in the final version.\n\n**Q8. $\\mathbf{z}$ is a random variable, yet the computation of the 2-norm in Eq. 9 is performed without recognizing this. I’d expect an expectation w.r.t.. q(z|x) to be involved in $\\mathcal{L}_z$ and $\\mathcal{L}\\_{lip\\\\_W}$ (or a non-central chi distribution if treated analytically)?**\n\nYour expectation is correct. As you say, we were calculating the loss using the sampled z from Eq.9 and some other equations. We were using the bold to denote the vector. We will remove the bold for distinguishing sampled z and clarify it.\n\n\n**Q9. Additional questions/suggestions.**\n\nDue to limited space, we were unable to address all raised issues. We commit to correcting all typos and clarifying notations, adding analyses based on your valuable suggestions for the final version.\n\n\n\n"}, {"review_id": "GTZnmkJUGb", "reviewer": "Reviewer_ynAb", "summary": "Recent advances in Bayesian optimization have shown that it is possible to exploit latent spaces of variational auto-encoders or generative models to perform the optimization of any function defined over a structured space. However, since the optimization takes place in the latent space. There is an inherent gap between the original problem formulation and the optimization of the latent space. This paper proposed a series of new losses based on the smoothness map of the latent space to enforce a correlation between the function values in the latent space and the original space. The authors propose several losses based on Lipschitz constants and local searches to improve the optimization properties. Finally, the author proposes a series of experiments to show the benefit of their method.", "questions": "The questions raised have been asked above.", "limitations": "The limitations are quickly summarized in the conclusion. It could be nice to have a larger section in the Appendix", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I liked Theorem 1 which provides a nice theoretical justification of the innovations of the paper. \n\nMoreover, it provides a nice storyline to explain the main elements of the paper and the loss introduced.\n\nThe paper is very clear and easy to follow.\n\nThere are a lot of numerical experiments showing the benefits of the method.", "weaknesses": "Overall, I really like the problem tackled in the paper. However, I have some questions with regards to the contributions, and more particularly with regards to lots of heuristics used in the algorihm:\n\n- although there is a nice theoretical justification of the new losses introduced (Eq 3, 4 and 9), the method presented in Algorithm 1 relies on a large number of parameters such as k, the kernel used in the Gaussian process, the batch size, the latent space update interval Nfail which might make the method hard to use in practice. For instance, it seems like some parameters presented in the Appendix such as the batch size and k are different depending on the use case. Is there a way to know how to choose those hyperparameters that work in most cases?\n\n- Similarly, for the halving of the trust region. It seems to be a full heuristic. Is there a justification for the value of the frequency to half the search space?\n\n- In Eq 7, how to you choose the value of $y_q$? What is the precise meaning of this term in the loss?\n\n- Bayesian optimization is mainly used when the black-box function is expensive (or costly) to evaluate. Moroever, there exists a large number of other methods for black-box optimization when the function is cheap to evaluate. However, when looking at the different experiments, it seems like the experiments can go up to 60K+ iterations. In this case, how would those methods compare to existing techniques (such as fine-tuned simulated annealing, genetic algorithms, and partitioning techniques such as the DIRECT algorithm) or even multi-start local search methods? Does using Bayesian optimization make a real difference?\n\n- Overall and as a last comment, since the method relies on a large number of hyper-parameters, it seems like it is not realistic to choose them in practice in the case of expensive-to-evaluate black-box functions (which is the core justification of Bayesian optimization) in which functions evaluations are scarce. How could we improve that in practice?\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly value the feedback provided by the reviewer. We offer responses to address the issues below.\n\n**Q1. Is there a justification for the value of the frequency to halve the search space?**\n\nWe adopt every setting for the Trust region from TURBO [16], which is an established technique. For example, we use 32 for the number of consecutive failures as in [16]. We also followed [16], without specifically tuning the value of the frequency to halve the search space.\n\n**Q2. In Eq 7, how do you choose the value of $y_q$? What is the precise meaning of this term in the loss?**\n\nThe value of $y_q$ corresponds to a specific quantile of the distribution of $y$. We use the value of $y_q$ as the same as in [21]. We assign weights based on Gaussian Cumulative Distribution Function (CDF), which is centered on $y_q$.\n\n**Q3. Compared to existing techniques like simulated annealing, genetic algorithms, or partitioning techniques, does using Bayesian optimization make a significant difference?**\n\nHere, our work aims to address the challenges of current (latent) Bayesian Optimization approaches. As a result, our primary focus for comparison is directed towards BO methods. While we acknowledge the potential for valuable insights through comparisons with conventional methods, we will add comparisons in the final version.\n\n\n**Q4. More detailed limitations**\n\nWe will improve the limitations and reflect on Appendix in the final version.\n"}]

Author rebuttal:
We thank all reviewers for their thorough and thoughtful feedback. We will address all issues raised by the reviewers below. Especially, we will provide answers to common questions that we have received from multiple reviewers in this General section.

**Q1. Regarding hyperparameter search and selection. (Bvzj, aw6W, ynAb)**

We grid search coefficients of our proposed regularizations $\mathcal{L}\_{\text{lip\\_w}}$ and $\mathcal{L}\_{\mathbf{z}}$, in the range of [10,100,1000]/[0.1,1]. The grid search was each conducted for 20 percent of the total oracle budget. The final coefficients are mentioned in Appendix E.1.

For other hyperparameters such as coefficients for other losses, batch size, learning rate, … etc, we fix the coefficient as in Table 5 from Appendix E.1, as we did not do further searching. Rather, we utilized hyperparameters from previous methods, e.g., the hyperparameters related to trust region in TURBO [16]. For the batch size, we chose values within the range allowed by GPU memory, and for initialization points, we mostly followed [15] and if exist, we followed the official evaluation settings from the benchmark dataset (e.g., DRD3).

We thank all reviewers for suggestions and we will reflect the above details in the Appendix for the final version.

**Q2. Confusion regarding Eq. 4 in the main paper. (Bvzj, 4zSv)**

We apologize for the confusion. Our intention is to control the average of the $\mathbf{z}$ distances, not to optimize for all distances between points to be equidistant. Thus, Eq. 4 should be revised as below:


$\displaystyle\mathcal{L}_{\text{z}} = {1\over N^2} \sum\_{i,j\le N} \bigg|||\mathbf{z}_i - \mathbf{z}_j||_2-c\bigg| \rightarrow\mathcal{L}\_{\text{z}} = \bigg|{1\over N^2} \sum\_{i,j\le N} ||\mathbf{z}_i - \mathbf{z}_j||_2-c\bigg|$

**We provide additional experiments on the attached PDF file.**
These are details on references in the PDF file:
[A] Rubinstein, R. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1999.
[B] Tripp, A., Daxberger, E., & Hernández-Lobato, J. M. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 2020.
[C] Peters, J., & Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. International conference on Machine learning, 2007.

Meta-review: I think this is a pretty good paper, and I'm unconvinced that several concerns raised by reviewers are actually concerns specific to this paper. For example, Reviewer ynAb points out the modeling design choices that must be made, but the specific choices listed are going to be true for any TuRBO-like BayesOpt algorithm (e.g., choosing a trust region length) or some for even any BO algorithm in general (for example, the kernel). I don't think we should suddenly start penalizing this paper for the existence of these considerations, some of which have existed in the BayesOpt literature at least since it has used Gaussian processes. Reviewer aw6W argues that the paper might not be "principled from a modelling perspective," but I'm just not sure whether this should matter specifically on optimization tasks like those over molecules where guarantees seem essentially impossible to achieve. LOL-BO can be viewed as a joint variational inference problem, yes, but it's unclear if that actually helps us guarantee anything about optimization performance.

I do think there are a few concerns raised that the authors *should* address, beyond the extensive results they've included in response to Reviewer Bzvj:

- The authors are pretty fast and loose with terminology like the "inherent gap" between the objective function and the latent space. I know what the authors mean, but they should be more precise about this. In Maus et al., 2022 for example, it is described as correlations in the objective function not being well measured by the GP prior used to model the objective. 

- I'm additionally unsure about the results on expressions and DRD3 -- Maus et al., 2022 reports substantially better performance on both tasks than seen in this paper (~0.1 on Expressions by 500 evaluations, and ~13.8+/-0.5 on DRD3 after 1000 evaluations). I'm curious about the source of this gap, as the results on the Guacamol tasks match fairly closely those in the original paper. Perhaps some hyperparameter (e.g., number of failures before shrinking the trust region) was tuned differently on those tasks due to the much smaller evaluation budgets involved?

Despite the above (and while I'd urge the authors to address them), I'm recommending the paper be accepted. Getting optimization working on these tasks is not easy. Even if the DRD3 and expressions results are fixed and much closer to matching at those evaluation budgets, I think the results are pretty impressive on the Guacamol tasks, and it seems as though the higher budget DRD3 results reported in the author feedback period will be better anyways.
Decision: Accept (poster)