Title: Advancing Bayesian Optimization via Learning Correlated Latent Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian optimization

smoothness regularization

variational autoencoder
Abstract: Bayesian optimization is a powerful method for optimizing black-box functions with limited function evaluations. Recent works have shown that optimization in a latent space through deep generative models such as variational autoencoders leads to effective and efficient Bayesian optimization for structured or discrete data. However, as the optimization does not take place in the input space, it leads to an inherent gap that results in potentially suboptimal solutions. To alleviate the discrepancy, we propose Correlated latent space Bayesian Optimization (CoBO), which focuses on learning correlated latent spaces characterized by a strong correlation between the distances in the latent space and the distances within the objective function. Specifically, our method introduces Lipschitz regularization, loss weighting, and trust region recoordination to minimize the inherent gap around the promising areas. We demonstrate the effectiveness of our approach on several optimization tasks in discrete data, such as molecule design and arithmetic expression fitting, and achieve high performance within a small budget.

Advancing Bayesian Optimization via Learning Correlated Latent Space

Abstract

Bayesian optimization is a powerful method for optimizing black-box functions with limited function evaluations. Recent works have shown that optimization in a latent space through deep generative models such as variational autoencoders leads to effective and efficient Bayesian optimization for structured or discrete data. However, as the optimization does not take place in the input space, it leads to an inherent gap that results in potentially suboptimal solutions. To alleviate the discrepancy, we propose Co rrelated latent space B ayesian O ptimization ( CoBO ), which focuses on learning correlated latent spaces characterized by a strong correlation between the distances in the latent space and the distances within the objective function. Specifically, our method introduces Lipschitz regularization, loss weighting, and trust region recoordination to minimize the inherent gap around the promising areas. We demonstrate the effectiveness of our approach on several optimization tasks in discrete data, such as molecule design and arithmetic expression fitting, and achieve high performance within a small budget.

1 Introduction

Bayesian optimization (BO) is a standard method for a wide range of science and engineering problems such as chemical design [1-4], reinforcement learning [5], and hyperparameter tuning [6]. Relying on a surrogate model typically modeled with a Gaussian process (GP), BO estimates the computationally expensive black-box objective function to solve the problem with a minimum number of function evaluations [7]. While it is known as a powerful method on continuous domains [8, 9], applying BO is often obstructed by structured or discrete data, as the objective values are in a complex combinatorial space [10]. This challenge has motivated recent interest in latent space Bayesian optimization (LBO) methods [11-14], which aim to find solutions in low-dimensional, continuous embeddings of the input data. By adopting deep generative models such as variational autoencoders (VAEs) [15] to map the input space to the latent space, LBO has successfully addressed the difficulties of such optimization problems.

However, the fact that optimization is not directly conducted in the input space gives rise to inherent gaps, which may lead to failures in the optimization process. First, we can think of the gap between the input space and the latent space. Seminal works have been developed to address this gap, with generative models such as β -VAE [16] emphasizing the importance of controlling loss weights, while WGAN [17] introduces improved regularization. Both aim to learn the improved latent space that better aligns with the input data distribution. Second, considering the BO problems, an additional gap emerges between the proximity of solutions in the latent space and the similarity of their

black-box objective function values. This is observed in many prior works [18, 19] that learn a latent space by minimizing only reconstruction errors without considering the surrogate model. This often leads to suboptimal optimization results. A recent study [10] has highlighted the significance of joint training between V AE and the surrogate model, yet it only implicitly encourages the latent space to align with the surrogate model. This limitation is observed in Figure 5a, where the latent space's landscape, with respect to objective values, remains highly non-smooth with joint training.

To this end, we propose our method Co rrelated latent space B ayesian O ptimization ( CoBO ) to address the inherent gaps in LBO. First, we aim to minimize the gap between the latent space and the objective function by increasing the correlation between the distance of latent vectors and the differences in their objective values. By calculating the lower bound of the correlation, we introduce two regularizations and demonstrate their effectiveness in enhancing the correlation. Especially, one of these regularizations, called the Lipschitz regularization, encourages a smoother latent space, allowing for a more efficient optimization process (see Figure 5b). Moreover, we suggest loss weighting with respect to objective values of each input data point to particularly minimize the gap between the input space and the latent space around promising areas of high objective values. Finally, we propose the concept of trust region recoordination to adjust the search space in line with the updated latent space. We experimentally validate our method with qualitative and quantitative analyses on nine tasks using three benchmark datasets on molecule design and arithmetic expression fitting.

To summarize, our contributions are as follows:

We propose Correlated latent space Bayesian Optimization (CoBO) to bridge the inherent gap in latent Bayesian optimization.

We introduce two regularizations to align the latent space with the black-box objective function based on increasing the lower bound of the correlation between the Euclidean distance of latent vectors and the distance of their corresponding objective values.

We present a loss weighting scheme based on the objective values of input points, aiming to close the gap between the input space and the latent space focused on promising areas.

We demonstrate extensive experimental results and analyses on nine tasks using three benchmark datasets on molecule design and arithmetic expression fitting and achieve stateof-the-art in all nine tasks.

2 Methods

In this section, we describe the main contributions of our method. Section 2.1 introduces several preliminaries on Bayesian optimization. In Section 2.2, we propose two regularizations to align the latent space with the black-box objective function. In Section 2.3, we describe our loss weighting scheme with the objective values. Lastly, in Section 2.4, we explain the overall architecture of our method.

2.1 Preliminaries

Bayesian optimization. Bayesian optimization (BO) [6, 8, 9] is a classical, sample-efficient optimization method that aims to solve the problem

where X is a feasible set and f is a black-box objective function. Since the function evaluation is assumed to be expensive, BO constructs a probabilistic model of the black-box objective function. There are two main components of BO, first is a surrogate model g that provides posterior probability distribution over f ( x ) conditioned on observed dataset D = { ( x i , y i ) } n i =1 based prior over objective function. Second is an acquisition function α for deciding the most promising next query point x i +1 based on the posterior distribution over f ( x ) . BO is a well-established method, however, applying BO to high-dimensional data can be challenging due to the exponential growth of the search space. To alleviate the challenge, recent approaches [10, 20] restrict the search space to a hyper-rectangular trust region centered on the current optimal input data point. In this paper, we adopt this trust-regionbased BO for handling high-dimensional search space.

Latent space Bayesian optimization. BO over structured or discrete input space X is particularly challenging, as a search space over the objective function becomes a large combinatorial one. In an effort to reduce a large combinatorial search space to continuous space, latent space Bayesian optimization (LBO) [10, 13, 14, 19, 21, 18] suggests BO over continuous latent space Z . A pretrained VAE = { q ϕ , p θ } [15] is commonly used as the mapping function, where the latent space is learned to follow the prior distribution (Gaussian distribution). Given a pretrained VAE, an encoder q ϕ : X ↦→ Z maps the input x i to the latent vector z i and the surrogate model g takes z i as the input. After the acquisition function α suggests the next latent query point z i +1 , a decoder p θ : Z ↦→ X reconstructs z i +1 to x i +1 , so that it can be evaluated by the black-box objective function, i.e ., f ( x i +1 ) .

LBO is a promising optimization approach for discrete or structured inputs, yet, there are two main gaps to be considered. Firstly, there is a gap between the input space and the latent space. Our focus is on narrowing the gap, especially within the promising areas, i.e ., samples with high objective function values. Secondly, a gap exists between the proximity of solutions within the latent space and the similarity of their corresponding objective function values. This arises because the objective value originates from a black-box function in the discrete input space X , distinct from the latent space Z where our surrogate model g is learned. In the previous work, [10] has suggested closing the gap by jointly optimizing V AE and the surrogate GP model to align the latent space with current topk samples.

Here, we propose CoBO that explicitly addresses those two gaps by training a latent space with Lipschitz regularization, which increases the correlation between the distance of latent samples and the distance of objective values, and loss weighting with objective values to focus on relatively important search space.

2.2 Aligning the latent space with the objective function

Our primary goal is to align the latent space Z with the black-box objective function f , which can be achieved by increasing the correlation between the Euclidean distance of latent vectors and the differences in their corresponding objective values, i.e. , Corr ( || z 1 -z 2 || 2 , | y 1 -y 2 | ) . Assuming the objective function f is an L -Lipschitz continuous function, we can establish a lower bound of Corr ( || z 1 -z 2 || 2 , | y 1 -y 2 | ) . In general, if the function f is L -Lipschitz continuous, it is defined as

where d Z and d Y are distance metrics in their respective spaces for z and y . Then, the lower bound of Corr ( || z 1 -z 2 || 2 , | y 1 -y 2 | ) can be obtained as Theorem 1.

Theorem 1. Let D Z = d Z ( Z 1 , Z 2 ) and D Y = d Y ( f ( Z 1 ) , f ( Z 2 )) be random variables where Z 1 , Z 2 are i.i.d. random variables, f is an L -Lipschitz continuous function, and d Z , d Y are distance functions. Then, the correlation between D Z and D Y is lower bounded as

where µ D Z , σ 2 D Z , µ D Y , and σ 2 D Y are the mean and variance of D Z and D Y respectively.

Theorem 1 implies that under the assumption of the L -Lipschitz continuity of f , we can increase the lower bound of Corr ( || z 1 -z 2 || 2 , | y 1 -y 2 | ) by reducing Lipschitz constant L while µ 2 D Z , σ 2 D Z , µ 2 D Y , and σ 2 D Y remain as constants. Based on Theorem 1, we propose two regularizations. The first one is Lipschitz regularization, which encourages a smooth latent space Z w.r.t. the objective function f .

where N is the number of training data points. Here, we set the Lipschitz constant L as the median of all possible gradients of slopes. By penalizing slopes with larger gradients than an adaptively adjusted L , we encourage a decrease in L itself, leading to learning a correlated latent space.

Next, it is beneficial to keep µ 2 D Z , σ 2 D Z , µ 2 D Y and σ 2 D Y as constants. Given that µ 2 D Y and σ 2 D Y are function-specific constants, where the black-box function is unchanged throughout the optimization,

we can treat them as fixed values. Then, we want to constrain µ 2 D Z as a constant with the second regularization L z, by penalizing the average distance between the latent vectors z to be a constant c :

We set c as the expected Euclidean norm between two standard normal distributions, which is the prior of the variational autoencoder. That is the mean of the noncentral chi distribution [22] which is sum of squared independent normal random variables:

where Γ( · ) denotes the gamma function, C denotes the random variable with noncentral chi distribution NC χ k and k denotes the degrees of freedom which is the same value as dimension n of the latent vector. Then c is dependent only on the dimension of the latent vector, z ∈ R n . For σ 2 D Z , preliminary observations indicate that it stays in a reasonable range as long as L Lip is not overly penalized. Thus, we safely conclude that there is no need to explicitly constrain σ 2 D Z . We refer to the supplement for further analysis and the proof of Theorem 1.

2.3 Loss weighting with objective values

Our focus now shifts to addressing the gap between the input space X and the latent space Z for LBO. Especially, we aim to minimize the gap in promising areas that offer better optimization opportunities, i.e. , significant points with high objective values. To achieve this, we prioritize input data points based on their respective objective values by weighting the reconstruction loss term. Following [23], we utilize the cumulative density function of the Gaussian distribution for the weighting. Specifically, the weighting function w.r.t. objective value y is:

with Y ∼ N ( y, σ 2 ) , where y q represents a specific quantile of the distribution of Y , and hyperparameter σ denotes the standard deviation of Y . The weighted reconstruction loss is as follows:

Moreover, we also apply the weighting scheme to the Lipschitz regularization term to promote a smoother latent space when the objective value is higher. The weighted Lipschitz regularization is defined with the geometric mean of the weights of two input data points:

2.4 Overall architecture of CoBO

In this section, we explain the overall architecture of CoBO. We first introduce the training schema of latent space in CoBO to encourage a high correlation between the distance in the latent space and the distance within the objective function. Next, we describe updating strategy of the surrogate model for modeling the black-box function and further present a generating procedure of the next candidate inputs for the black-box objective function through the acquisition function in the trust region. The overall procedure of our CoBO is in Algorithm 1.

Learning the latent space. Our method learns the latent space by optimizing the encoder q ϕ and decoder p θ of the pretrained V AE and updating the surrogate model in the latent space with our final loss:

where L Lip_W and L z is the regularization term in the latent space at Section 2.2 and 2.3, L recon_W is the weighted reconstruction loss term, L KL is the KL divergence between the latent space distribution and the prior, and L surr is the loss for optimizing the surrogate model. We adopt the joint training scheme, training the surrogate model and the encoder q ϕ of the VAE model jointly [10]. Under computational considerations, we retrain a latent space after N fail accumulated failure of updating the optimal objective value.

Updating the surrogate model. After jointly optimizing the latent space and the surrogate model, we freeze the parameter of VAE and train our surrogate model. Note that this update is executed only after consecutive failures of updating optimal value, we also train the surrogate model in every iteration. As exact Gaussian process (GP), i.e. , f ( x ) ∼ GP ( m ( x ) , k ( x , x ′ )) , where m is a mean function and k is a covariance function, is infeasible to handle large datasets due to cubic computational complexity O ( N 3 ) for N data points, we employ sparse GP [24] as a surrogate model which is computationally efficient via inducing point method. To alleviate cubic complexity, sparse GP approximates black-box function with M ≪ N pseudo-training samples called 'inducing points' that reduce complexity to O ( MN 2 ) . We select the most widely used RBF kernel as a sparse GP kernel function. Finally, we adopted deep kernel learning (DKL) [25] in conjunction with sparse GP for our final surrogate model.

Generating candidates through acquisition function. Candidate samples for the acquisition function are determined by random points in a trust region centered on the current optimal value. We take a simple and powerful method, Thompson sampling as an acquisition function within the context of Bayesian optimization. The surrogate model acts as the prior

belief about our objective function. Thompson sampling uses this model to draw samples, leveraging its uncertainty to decide the next point to evaluate. In detail, we first select candidate samples in the trust region and score each candidate based on the posterior of the surrogate model to get the most promising values. Also, we recoordinate the center of the trust region to ˆ z ∗ , which is obtained by passing the current optimal latent vector z ∗ into updated VAE, i.e ., q ϕ ( p θ ( z ∗ )) , as shown in Figure 1. We empirically showed that trust region recoordination helps to find a better solution within a limited budget (see Table 2). Following [20], the base side length of the trust region is halved after consecutive failures of updating optimal objective value and doubled after consecutive successes.

3 Experiments

In this section, we demonstrate the effectiveness and efficiency of CoBO through various optimization benchmark tasks. We first introduce tasks and baselines. Then, in Section 3.1, we present evaluations of our method and the baselines for each task. In Section 3.2, we conduct an ablation study on the components of our approach. Finally, in Section 3.3, we provide qualitative analyses on the effects of our suggested regularizations and the necessity of z regularization.

Tasks. We evaluate CoBO to nine tasks on a discrete space in three different Bayesian optimization benchmarks, which consist of arithmetic expression fitting tasks, molecule design tasks named dopamine receptor D3 (DRD3) in Therapeutics Data Commons (TDC) [26] and Guacamol benchmarks [27]. The arithmetic expression task is generating polynomial expressions that are close to specific target expressions ( e.g. , 1 / 3+ x +sin( x × x ) ) [10, 11, 13, 14, 28], we set the number of initialization points | D 0 | to 40k, and max oracle calls to 500. In Guacamol benchmark, we select seven challenging tasks to achieve high objective value, Median molecules 2, Zaleplon MPO, Perindopril MPO, Osimertinib MPO, Ranolazine MPO, Aripiprazole similarity, and Valsartan SMART. The results for the last three tasks are in the supplement. The goal of each task is to find molecules that have the most required properties. For every task of Guacamol benchmark, we set the number of initialization points to 10k, and max oracle calls to 70k. DRD3 task in the TDC benchmark aims to find molecules with the largest docking scores to a target protein. In DRD3, the number of initialization points is set to 100, and the number of oracle calls is set to 3k. We use SELFIES V AE [10] in Chemical design, and Grammar VAE [11] in arithmetic expression.

Algorithm 1 Correlated Bayesian Optimization (CoBO)

Input: Pretrained VAE encoder q ϕ , decoder p θ , black-box function f , surrogate model g , acquisition function α , previously evaluated dataset D 0 = { ( x i , y i , z i ) } n i =1 , oracle budget T , latent update L L

̸

Baselines. We compare our methods with four BO baselines: LOL-BO [10], W-LBO [13], TuRBO [20] and LS-BO. LOL-BO proposed the joint loss to close the gap between the latent space and the discrete input space. Additionally, W-LBO weighted data points to emphasize important samples with regard to their objective values. To handle discrete and structured data, we leverage TuRBO in latent space, TuRBOL through encoder q ϕ of pretrained frozen VAE and reconstruct it by decoder p θ to be evaluated by the objective function. For more details about TuRBOL , see [10]. In our LS-BO approach, we adopt the standard LBO methodology. Specifically, we use a VAE with pre-trained parameters that are frozen. For sampling, the candidates are sampled from a standard normal Gaussian distribution, N (0 , I ) .

3.1 Results on diverse benchmark tasks

Figure 2, 3 represent the graphs that depict the number of oracle calls, i.e. , the number of the blackbox function evaluations, and the corresponding mean and standard deviation of objective value. The maximum number of oracles is set to 70k with Gucamol benchmarks, 3k with the DRD3 benchmark, and 500 with the arithmetic expression task. As shown in Figure 2, our method outperformed all the baselines in all the tasks. Note that the goal of the arithmetic task in Figure 3a and the DRD3 task Figure 3b is minimizing the objective function, while others aim to maximize it. In the case of Figure 2a and Figure 2d, LS-BO and TuRBOL markedly failed to search for the one that has the best score. Especially in Figure 2a, LS-BO and TuRBOL , which only search with the fixed latent space, failed to update the initial best point despite searching 70k molecules. It implies that the pretrained V AE cannot generate better molecules unless the latent space is updated.

3.2 Ablation study

nation.

Recoordination

✓

×

Score

0.7921

0.6844

(a) Median molecules 2 (med2)

(c) Perindopril MPO (pdop)

0.75

0.38

0.70

0.36

Score

Score

0.65

0.34

0.60

0.55

0.32

0.50

1000020000

40000

60000

80000

10000 20000

40000

60000

80000

NumOracle

0.85

0.94

0.80

0.92

0.75

Score

0.70

Score

0.90

S

0.65

S

0.88

0.60

0.86

0.55

0.84

0.50

1000020000

40000

60000

80000

10000 20000

40000

60000

80000

CoBO (Ours)

LOL-BO

W-LBO

TuRBO-L

LS-BO

(a) Arithmetic expression

(b) TDC DRD3

1.75

1.50

12.0

12.5

1.25

13.0

1.00

Score

13.5

S

0.75

S

-14.0

0.50

14.5

0.25

-15.0

0.00

15.5

40000

40100

40200

40300

40400

40500

100

1000

2000

3000

CoBO (Ours)

LOL-BO

W-LBO

TuRBO-L

LS-BO

In this section, we evaluate the main components of our model to analyze their contribution. We employ Perindopril MPO (pdop) task for our experiment and note that all scores of ablation studies are recorded when the oracle number is 20k to compare each case in limited oracle numbers as the original BO intended.

We ablate the three main components of our CoBO and report the results in Table 1. As latent space regularization L z comes from Lipschitz regularization and loss weighting schema aims to prioritize penalized input points, we conducted cascading experiments. Notably, we observed that performance decreased as the components were omitted, and the largest performance dropped (0.0229)

(a) Perindopril MPO (pdop)

(b) Arithmetic expression

0.8

0.6

M

lation

w/ Calign

Correlation

0.5

0.6

0.4

Correl

0.4

0.3

Pearson

Pearson

0.24

0.2

0.1

0.0

0.0

-0.1

0

25

50

75

100

125

150

175

200

0

10

20

30

40

50

60

70

Epoch

Epoch

when Lipschitz regularization was not employed. Table 2 reports the effectiveness of trust region recoordination. We observe that applying trust region recoordination makes a considerable difference within a small oracle budget, as the score increases 0.6844 to 0.7921 with recoordination.

3.3 Analysis on proposed regularizations

All the analyses were conducted on the Perindopril MPO (pdop) task from the Guacamol benchmark. We include an additional analysis with the arithmetic data. For convenience, we define the L align as follows:

which is the loss that aligns the latent space with the black-box objective function.

Effects of L align on correlation. In Section 2.2, we theoretically prove that L align increases the correlation between the distance in the latent space and the distance within the objective function. Here, we demonstrate our theorem with further quantitative analysis to show correlation changes during the Bayesian optimization process. Figure 4 shows Pearson correlation value between the latent space distance ∥ z i -z j ∥ 2 and the objective value distance | y i -y j | . The blue and orange lines indicate the models with and without our align loss L align in Eq. 12, respectively. We measure the correlation with 10 3 data point and every 10 6 pair. The data is selected as the top 10 3 points with the highest value among 10 3 data, which are from training data for the V AE model and surrogate model. Over the training process, the Pearson correlation values with our align loss L align were overall higher compared to the baseline. Moreover, L align increases the Pearson correlation value high over 0 . 7 in Figure 4a which is normally regarded as the high correlation value. This means align loss increases the correlation between the distance of the latent vectors and the distance of the objective values effectively, leading to narrowing the gap between the latent space and the objective function.

Effects of L align on smoothness. To validate that our model encourages the smooth latent space, we qualitatively analyze our model with L align and without L align by visualizing the landscape of the latent space after the training finishes. In Figure 5, we visualize the topk objective value and the corresponding latent vector in 2D space with the 2D scatter plot and the corresponding 3D plot for better understanding. We reduce the dimension of the latent vector to two dimensions using principal component analysis (PCA). The color means the normalized relative objective score value. The landscape of objective value according to the latent space with L align (right) is smoother than the case without L align (left). It demonstrates our L align loss encourages smoothness of the latent space with respect to the objective function. Note that due to the inherent discreteness of the black-box objective function, it's expected to observe some spaces between the clusters in the 2D plot. Still, this does not detract from our purpose of effectively aligning latent vectors with respect to their discrete objective values.

(a) Without

L

align

(b) With

L

align

1.00

Objective Value

0.67

0.33

0.00

1.00

Objective Value

0.67

0.33

0.00

4 Related Works

4.1 Latent space Bayesian optimization

Latent space Bayesian optimization [10, 12-14, 19, 21, 29, 30] aims to resolve the issues in optimization over high-dimensional, or structured input space by introducing Bayesian optimization over latent space. As the objective for structured inputs is usually defined over a large, complex space, the challenges can be alleviated by the lower-dimensional and continuous latent space. Variational autoencoders (VAEs) [15] are commonly leveraged to learn the continuous embeddings for the latent space Bayesian optimizers. Some prior works propose novel architectures for decoders [11, 12, 3133], while others introduce loss functions to improve the surrogate for learning the objective function [13, 14, 18, 29]. Note that while the surrogate model (typically GP) is modeled based on the latent space, it is the input space which the objective value is obtained from. Although this results in an inherent gap in latent space Bayesian optimization, many prior works [12, 14, 19, 21, 29] do not update the generative model for the latent space. LOL-BO [10] seeks to address this gap by adapting the latent space to the GP prior, while [13] suggests periodic weighted retraining to update the latent space.

4.2 Latent space regularization

Several previous studies in latent space have focused on learning the appropriate latent space for their tasks by incorporating additional regularizations or constraints alongside the reconstruction loss. These approaches have been applied in a wide range of areas, including molecule design [3436], domain adaptation [37, 38], semantic segmentation [39, 40], representation learning [41-43], and reinforcement learning [44]. Notably, the Lipschitz constraint is commonly employed to promote smoothness in diverse optimization problems. For example, [42] introduces Lipschitz regularization in learning implicit neural functions to encourage smooth latent space for 3D shapes while [45] penalizes the data pairs that violate the Lipschitz constraint in adversarial training. Additionally, CoFLO [46] leverages Lipschitz-like regularization in latent space Bayesian optimization. Our method proposes regularization to close the gap inherent in the latent space Bayesian optimization. Concretely, we introduce Lipschitz regularization to increase the correlation between the distance of latent space and the distance of objective value and give more weight to the loss in the promising areas.

5 Conclusion and Discussion

In this paper, we addressed the problem of the inherent gap in the latent space Bayesian optimization and proposed Correlated latent space Bayesian Optimization. We introduce Lipschitz regularization which maximizes the correlation between the distance of latent space and the distance of objective value to close the gap between latent space and objective value. Also, we reduced the gap between latent space and input space with a loss weighting scheme, especially in the promising areas. Additionally, by trust region recoordination, we adjust the trust regions according to the updated latent space. Our experiments on various benchmarks with molecule generation and arithmetic fitting tasks demonstrate that our CoBO significantly improves state-of-the-art methods in LBO.

Limitations and broader impacts. Given the contribution of this work to molecular design optimization, careful consideration should be given to its potential impacts on the generation of toxic or harmful substances in the design of new chemicals. We believe that our work primarily has the positive potential of accelerating the chemical and drug development process, setting a new standard in latent space Bayesian optimization.

Acknowledgments and Disclosure of Funding

This work was partly supported by ICT Creative Consilience program (IITP-2023-2020-0-01819) supervised by the IITP, the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2023R1A2C2005373), and Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT1701-51.

References

Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. Chembo: Bayesian optimization of small organic molecules with synthesizable recommendations. In ICAIS , 2020.

José Miguel Hernández-Lobato, James Requeima, Edward O Pyzer-Knapp, and Alán AspuruGuzik. Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In ICML , 2017.

Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science , 2020.

Ke Wang and Alexander W Dowling. Bayesian optimization for chemical products and functional materials. Current Opinion in Chemical Engineering , 2022.

Roberto Calandra, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty: An experimental comparison on a dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence , 2016.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NeurIPS , 2012.

Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Arxiv , 2010.

Peter I Frazier. A tutorial on bayesian optimization. Arxiv , 2018.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 2015.

Natalie Maus, Haydn Jones, Juston Moore, Matt J Kusner, John Bradshaw, and Jacob Gardner. Local latent space bayesian optimization over structured inputs. In NeurIPS , 2022.

Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. In ICML , 2017.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML , 2018.

Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retrainin. In NeurIPS , 2020.

Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I Cowen-Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, et al. Highdimensional bayesian optimisation with variational autoencoders and deep metric learning. Arxiv , 2021.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , 2014.

Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR , 2017.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. Arxiv , 2017.

Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for bayesian optimization over combinatorial spaces. In NeurIPS , 2021.

Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a datadriven continuous representation of molecules. ACS central science , 2018.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. In NeurIPS , 2019.

Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In ICLR , 2022.

John Lawrence. Moments of the noncentral chi distribution. Sankhya A , 2021.

David H Brookes and Jennifer Listgarten. Design by adaptive sampling. Arxiv , 2018.

Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In NeurIPS , 2005.

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In AISTATS , 2016.

Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In NeurIPS , 2021.

Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling , 2019.

Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. In NeurIPS , 2020.

Stephan Eissman, Daniel Levy, Rui Shu, Stefan Bartzsch, and Stefano Ermon. Bayesian optimization and attribute adjustment. In UAI , 2018.

Eero Siivola, Andrei Paleyes, Javier González, and Aki Vehtari. Good practices for bayesian optimization of high dimensional structured spaces. Applied AI Letters , 2021.

Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Gomez-Rodriguez. Nevae: A deep generative model for molecular graphs. In AAAI , 2019.

Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In ICML , 2019.

Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. Syntax-directed variational autoencoder for structured data. In ICLR , 2018.

Egbert Castro, Abhinav Godavarthi, Julian Rubinfien, Kevin Givechian, Dhananjay Bhaskar, and Smita Krishnaswamy. Transformer-based protein generation with regularized latent space optimization. Nature Machine Intelligence , 2022.

Pascal Notin, José Miguel Hernández-Lobato, and Yarin Gal. Improving black-box optimization in vae latent space using decoder uncertainty. In NeurIPS , 2021.

ANMAbeer, Nathan Urban, M Ryan Weil, Francis J Alexander, and Byung-Jun Yoon. Multiobjective latent space optimization of generative molecular design models. Arxiv , 2022.

Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR , 2019.

Lei Tian, Yongqiang Tang, Liangchen Hu, Zhida Ren, and Wensheng Zhang. Domain adaptation by class centroid matching and local manifold self-learning. IEEE Transactions on Image Processing , 2020.

Francesco Barbato, Marco Toldo, Umberto Michieli, and Pietro Zanuttigh. Latent space regularization for unsupervised domain adaptation in semantic segmentation. In CVPRW , 2021.

Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsionattraction of sparse and disentangled latent representations. In CVPR , 2021.

Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational autoencoders. In NeurIPS , 2021.

Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. In SIGGRAPH , 2022.

Ramana Subramanyam Sundararaman, Riccardo Marin, Emanuele Rodola, and Maks Ovsjanikov. Reduced representation of deformation fields for effective non-rigid shape matching. In NeurIPS , 2022.

Mete Kemertas and Tristan Aumentado-Armstrong. Towards robust bisimulation metric learning. In NeurIPS , 2021.

Dávid Terjék. Adversarial lipschitz regularization. In ICLR , 2020.

Fengxue Zhang, Yair Altas, Louise Fan, Kaustubh Vinchure, Brian Nord, and Yuxin Chen. Design of physical experiments via collision-free latent space optimization. In NeurIPSW , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecent advances in Bayesian optimization have shown that it is possible to exploit latent spaces of variational auto-encoders or generative models to perform the optimization of any function defined over a structured space. However, since the optimization takes place in the latent space. There is an inherent gap between the original problem formulation and the optimization of the latent space. This paper proposed a series of new losses based on the smoothness map of the latent space to enforce a correlation between the function values in the latent space and the original space. The authors propose several losses based on Lipschitz constants and local searches to improve the optimization properties. Finally, the author proposes a series of experiments to show the benefit of their method.\n\nSTRENGTHS:\nI liked Theorem 1 which provides a nice theoretical justification of the innovations of the paper. \n\nMoreover, it provides a nice storyline to explain the main elements of the paper and the loss introduced.\n\nThe paper is very clear and easy to follow.\n\nThere are a lot of numerical experiments showing the benefits of the method.\n\nWEAKNESSES:\nOverall, I really like the problem tackled in the paper. However, I have some questions with regards to the contributions, and more particularly with regards to lots of heuristics used in the algorihm:\n\n- although there is a nice theoretical justification of the new losses introduced (Eq 3, 4 and 9), the method presented in Algorithm 1 relies on a large number of parameters such as k, the kernel used in the Gaussian process, the batch size, the latent space update interval Nfail which might make the method hard to use in practice. For instance, it seems like some parameters presented in the Appendix such as the batch size and k are different depending on the use case. Is there a way to know how to choose those hyperparameters that work in most cases?\n\n- Similarly, for the halving of the trust region. It seems to be a full heuristic. Is there a justification for the value of the frequency to half the search space?\n\n- In Eq 7, how to you choose the value of $y_q$? What is the precise meaning of this term in the loss?\n\n- Bayesian optimization is mainly used when the black-box function is expensive (or costly) to evaluate. Moroever, there exists a large number of other methods for black-box optimization when the function is cheap to evaluate. However, when looking at the different experiments, it seems like the experiments can go up to 60K+ iterations. In this case, how would those methods compare to existing techniques (such as fine-tuned simulated annealing, genetic algorithms, and partitioning techniques such as the DIRECT algorithm) or even multi-start local search methods? Does using Bayesian optimization make a real difference?\n\n- Overall and as a last comment, since the method relies on a large number of hyper-parameters, it seems like it is not realistic to choose them in practice in the case of expensive-to-evaluate black-box functions (which is the core justification of Bayesian optimization) in which functions evaluations are scarce. How could we improve that in practice?\n\nQUESTIONS:\nThe questions raised have been asked above.\n\nLIMITATIONS:\nThe limitations are quickly summarized in the conclusion. It could be nice to have a larger section in the Appendix", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of Bayesian optimization with the help of a low-dimensional latent space (here learned using a VAE augmented with ad-hoc loss terms).\n\nThe paper contributes:\n- An analysis/recap of the state-of-the-art highlighting issues raised in other works, particularly the need for smoothness (in the latent space) around the optima.\n- A concrete loss function augmenting previous work [10] with two additional regularisation terms to ensure smoothness in the latent space w.r.t. the objective function.\n- Experimental study demonstrating the advantages over a suitable set of benchmarks on six problems.\n- An ablation study examining the effect of some of the loss terms on the final BO task.\n\nSTRENGTHS:\n-\tImportant and relevant research area.\n-\tThe background and motivation for the derived algorithm are well-described.\n-\tThe included experiments indicate improvements compared to a set of reasonable baselines.\n-\tThe resulting algorithm seems sufficiently novel and builds on the intuition about smoothens w.r.t. to the objective function, but I have some concerns/questions…\n\nWEAKNESSES:\nOverall weaknesses/comments/suggestions:\n\n-\tThe setup does not come across as principled or consistent from a pure modeling perspective…\n  -\tI appreciate the algorithmic-centered view (vs. a probabilistic modeling perspective) adopted in the paper. Still, I am always a bit skeptical when rather ad-hoc loss terms are added to the VAE loss or the joint setup from [10] (which both originate from a proper probabilistic analysis). The combination of and interaction among the many loss terms (in particular the $L_z$, $L_lip$, vs. the VAE prior) would need quite a lot of analysis to work out the exact effect of the combination (and weighting). Currently, I feel the paper lacks this insight.\n - $z$ is a random variable, yet the computation of the 2-norm in Eq. 9 is performed without recognizing this. I’d expect an expectation w.r.t.. q(z|x) to be involved in $L_z$ and $L_{lip_W}$ (or a non-central chi distribution if treated analytically)?\n -\tI am unsure why $L_z$ is needed – can this not be controlled directly via the variance on the prior, $p(z)$? Can the smoothness constraints be incorporated/formulated as a more principled prior for z, thus providing a complete probabilistic view and derivation of the algorithm?\n-\tMissing assumptions. \n -\tI think it needs to be argued that it makes sense to globally measure distance in the latent space using a Euclidian distance function.\n - It is generally unclear if you assume y is noiseless or not. This seems important in Eq. 9 (and Theorem 1), i.e., do you use $p(y|x, \\cdot)$ or simply the observed y when computing $|y_i-y_j$? \n-\tThe model/loss is not fully specified in the paper, as far as I can tell. Specifically,\n -\tThe weights on the loss terms are left out in Eq 11, giving the impression that there are no hyperparameters related to the loss itself, yet the supplementary clearly indicates that some manual fine-tuning is needed. The main paper should be transparent about this. How sensitive is the performance to the weights on the loss terms, and how should they be set in practice? \n -\tl 155: $L_{joint}$ should be explained in more detail to provide a self-contained definition of the model/algorithm.\n -\tThere are no details in the paper or supplementary about the likelihood ($p(x|z)$); I’d suggest adding more information about this beyond a reference to the particular VAEs.\n -\tThere are no details about the structure and complexity of the problems or the VAE (e.g. dimensionality of z and its influence)$); I’d suggest providing this at least in the supplementary.\n-\tExperimental suggestions: \n -\tIf possible, I would suggest providing a very simple 2D (maybe 3D ) synthetic example (trained with the proposed model) to provide the reader with a better intuition about what’s going on with the many loss terms (c.f. previous comment).\n -\tThe ablation study (on one dataset) is interesting yet should probably include removing the $L_{joint}$ term as well.\n\n\nMinor questions/comments:\n\n-\tI’d suggest providing (possibly in the appendix) an optimization trace for one example with the total loss and all the individual loss terms included individually.\n-\tl120: I think adaptively changing setting L using the median would require more justification and explanation to clarify the properties of this scheme.\n-\tl212: I am unsure what is meant with “….Thompson sampling in $N(0,I)$...”?\n-\tFigure 5: Is this a result based on a VAE with a latent space with dimension 2 or a projection onto 2D?\n - l 263: Fig 5 (b) seems less smooth than Fig 5 (a) as is; consider making the statement clearer (i.e. wrt to the objective)\n-\tMost figures lack indication of axis labels (on the figure itself or in the caption)\n-\tIt would be helpful with a few more words/details attached to Eq 7 (the proof) in the supplementary material.\n-\tSome sections contain grammatical issues interrupting the flow, e.g. sec 2.4, l. 165  (missing “the”, “a” etc.)\n\nQUESTIONS:\nThe questions are included in the above.\n\nLIMITATIONS:\nSee above for technical limitations/issues. There is no need to consider the broader societal impact of the specific paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes several heuristic regularization constraints for learning a Bayesian optimization latent space. It argues that the learned latent space needs to be aligned to the black-box function values, and this is achieved via keeping the Lipschitz constant small and the mean latent distance (of training samples) constant. It also argues that promising points need to be prioritized during the latent space optimization, and this is achieved via weighting the reconstruction loss and the Lipschitz regularization loss above at each training point by its function values. The paper evaluates the proposed method on several benchmark functions, yielding positive results.\n\nSTRENGTHS:\nThe proposed approach makes some practical sense, and yields good performance on several benchmarks. The approach is also novel, so I believe the paper has some intellectual and practical merits.\n\nWEAKNESSES:\n1. I think the problem is not very clearly described and motivated. What exactly are the \"gaps\" between the latent space and the input space? Throughout the paper, I have not seen any technical description of this issue. To quote the manuscript, the two main gaps are:\n- \"First, even though the surrogate model g learned in the latent space the objective value is still obtained by a black-box function defined in the discrete input space X so the gap between objective values and latent spaces leads to poor optimization performance\".\n- \"Second, since the distribution of samples expected to have high objective values is different from that of samples observed in pretraining, there is a gap between the input space and the latent space that makes the optimization inaccurate\".\n\nNeither statements give a precise description of what these gaps are, so I cannot be convinced that the proposed solutions are meaningful.\n\n2. A lot of the technical details also lack motivation. Please refer to the specific questions below.\n\n3. Different BO approaches have different per-iteration cost, and it would be slightly unfair to only compare the performance vs. number of oracle calls. I would strongly suggest providing a plot to show performance vs. wall-clock time, or a table documenting the runtime per iteration of each baseline.\n\nQUESTIONS:\n1. I don't understand why $\\mu_{D_Z}$ and $\\sigma_{D_Z}$ are not constants the same way $\\mu_{D_Y}$ and $\\sigma_{D_Y}$ are? And what are they constants with respect to?\n\n2. Eq. (4) seems like a much stricter objective than what is needed -- it means that the points are optimized towards being equidistant. Also, when every $||z_i - z_j||_2 \\simeq c$, what will remain for L_lip to optimize?\n\n3. Why does blurring y with Gaussian noise in Eq. (7) make sense? Couldn't we just directly let $\\lambda(y) = y_q - y$  (and maybe normalize this by y_max so that the weight is between 0/1) ?\n\nLIMITATIONS:\nThe authors have discussed broader impacts & limitations. I don't see any potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a latent space Bayesian Optimization approach based on the intuition that distances in the latent space should be correlated with differences in objective value. CoBO iteratively updates a variational autoencoder (VAE) to align distances in latent space with differences in objective function. The method leverages a lower bound on correlation to introduce new regularizers on the VAE objective. Combined with loss weighting and latent space “recoordination”, the method achieves state-of-the-art performance on diverse benchmarks.\n\nSTRENGTHS:\nThe paper mostly applies existing techniques to achieve incremental but statistically significant improvement over prior state-of-the-art. Theorem 1 provides a useful result that is a significant contribution of its own, if it is in fact a new bound.\n\nThe paper is written very well and will likely serve as a benchmark for future BO algorithms. Algorithm 1 provides a nice, approachable, and thorough description of CoBO. Figures 4-5 show strong evidence that the CoBO objective has a strong positive effect on organizing the latent space.\n\nThe experimental results are convincing. An ablation study justifies the inclusion of all components of the proposed VAE objective, albeit on only one optimization benchmark (hopefully these results hold for all benchmarks studied).\n\nI would suggest trying to fit Table 3 from the Supplemental into the body of the paper, as it would strengthen the experimental evaluation. The results show strong performance especially in the low-budget setting.\n\nWEAKNESSES:\nIt does not seem natural to assume that the mapping $f$ is Lipschitz for a VAE trained in an unsupervised manner. Won’t it be the case that the objective is not Lipschits in the input space of many useful models, especially over discrete spaces? Specifically, do you have any experiments demonstrating Lipschitz continuity of the objective by sampling points in the latent space?\n\n_Note that the “recoordination” technique claimed by this paper previously existed in Maus et al. under the name “recentering”._\n\nEvaluation is a little limited. Do the results hold for all Guacamole tasks? For example, Maus et al evaluates on additional GuacaMol tasks such as ranolazine.\n\nEvalutions on DRD3 only go up to 1,000 evaluations of the objective function. This is understandable given the expense of running the docking objective, but on the other hand, this is the most “real-world” objective studied. Would it be possible to run for longer to determine whether CoBO can eventually beat LOL-BO or GraphGA?\n\nThe axes for all plots start at 0; it would be more helpful to include initialization points in the evaluation budget for comparison with prior work.\n\nSpecific issues:\n* Articles are missing in several locations, including in the abstract (line 12). Please proofread for small errors.\n* Equation 4 shows a regularizer that encourages __every__ pair of distances to be exactly $c$; it does not regularize the mean as claimed in the text. Is the text or the equation incorrect?\n* Do you recoordinate all points during fine tuning? If so, add this step to Algorithm 1 (I believe it currently only shows recentering for the trust region, not the surrogate).\n* Is there a reason that you report the objectives in Arithmetic expressions as a minization problem, but transform the DRD3 task into a maximization task? Retaining the convention of the TDC leaderboard would improve the accessibility of the paper to a wide audience.\n\nSmall comments:\n* Define “an inherent gap” in the abstract. This notion is referred to multiple times in the paper, but never precisely. Is the claim that there do not exist points in the latent space $z$ that correspond to an optimal point in input space $x$? Or that these regions are small and hard to find?\n* It is not immediately obvious in the introduction that this method will leverage a pre-trained VAE that is trained in a purely unsupervised manner.\n* In Equation 7, notation seems to be overloaded. Do you mean a quintile of the empirical distribution of objective values seen during training, $y_i$? At line 140, is $sigma$ a hyperparameter or equal to $\\sigma_Y$?\n* Algorithm 1 line 14: typo “is the best score than”\n* When discussing “initialization points” in Section 3, it would be useful to the the notation $D_0$ from Algorithm 1.\n* In Figure 2, should “TDL” be “TDC”?\n* It would be helpful to label the y-axes in Figure 4.\n\nQUESTIONS:\n1. The paper ignores the fact that the VAE decoder in many applications is stochastic (e.g. the SMILES decoder from Maus et al.). This would motivate a probabilistic treatment of f(Z) in Theorem 1.\n2. In Theorem 1, won’t it always be the case that $D_Y \\le D_Z$ by the Lipschitz assumption? Why is this stated as a sufficient condition?\n3. On line 120, is this Lipschitz constant computed as the median __per mini-batch__?\n4. Equations 4-6 do not appear to be consistent. You derive the distribution of inter-point distances, so why regularize all distances to be the same? Do you think of this as an additional discrepancy measure between your VAE prior and posterior? Can this be connected to the MMD regularization in InfoVAEs?\n5. The paper describes the method in terms of standard Gaussian Process kernels. Do you not use a deep kernel, even for the drug discovery tasks?\n6. How are hyperparameters determined, particularly in the ablation study?\n7. How do you choose the initialization points?\n8. Does Figure 4 include the recoordination step?\n9. In Figures 4-5, would it be possible to compare to the prior state-of-the-art (LOL-BO) to substantiate the claim that that method does not effectively reorganize the latent space, rather than simply comparing to a fixed latent space?\n\nLIMITATIONS:\nI believe that CoBO can only be expected to be effective on objectives that are Lipschitz continuous; this limitation is not mentioned. The author’s consideration of possible malicious uses for drug discovery models is much appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}