Latent Bayesian Optimization via Autoregressive Normalizing Flows
Authors: Seunghun Lee
Jinyoung Park
Jaewon Chu
Minseo Yoon
Hyunwoo J. Kim
Venue: ICLR 2025 Oral (2025)
Keywords: Bayesian optimization
normalizing flow
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Bayesian Optimization (BO) has been recognized for its effectiveness in optimizing expensive and complex objective functions.
Recent advancements in Latent Bayesian Optimization (LBO) have shown promise by integrating generative models such as variational autoencoders (VAEs) to manage the complexity of high-dimensional and structured data spaces.
However, existing LBO approaches often suffer from the value discrepancy problem, which arises from the reconstruction gap between input and latent spaces.
This value discrepancy problem propagates errors throughout the optimization process, leading to suboptimal outcomes.
To address this issue, we propose a Normalizing Flow-based Bayesian Optimization (NF-BO), which utilizes normalizing flow as a generative model to establish one-to-one encoding function from the input space to the latent space, along with its left-inverse decoding function, eliminating the reconstruction gap. Specifically, we introduce SeqFlow, an autoregressive normalizing flow for sequence data.
In addition, we develop a new candidate sampling strategy that dynamically adjusts the exploration probability for each token based on its importance.
Through extensive experiments, our NF-BO method demonstrates superior performance in molecule generation tasks, significantly outperforming both traditional and recent LBO approaches.

Published as a conference paper at ICLR 2025
LATENT BAYESIAN OPTIMIZATION VIA AUTOREGRESSIVE NORMALIZING FLOWS
Seunghun Lee', Jinyoungg Park's Jaewon Chu', Minseo Yoon', Hyunwoo J. 2im
"Department of Computer Science and Engineering. Korea University Schooo of Computing, KAIST
{11sshh319 1pmn 678, al lonsy07, cookk.6113jkcorra..c.k hyunwoo jkinkkaisi........m...m. ac.kr
ABSTRACT
Bayesian Optimization (BO) has been recognized for its effectiveness in optimiz- ing expensive and complex objective functions. Recent advancements in Latent Bayesian Optimization (LBO) have shown promise by integrating generative mod- els such as variational autoencoders (VAEs) to manage the complexity of high- dimensional and structured data spaces. However, existing LBO approaches often suffer from the value discrepancy problem, which arises from the reconstruction gap between input and latent spaces. This value discrepancy problem propagates errors throughout the optimization process, leading to suboptimal outcomes. To address this issue, we propose Normalizing Flow-hased Bayesian Optimization (NF-BO), which utilizes normalizing flow as generative model to establish one- to-one encoding function from the input space to the latent space, along with its left-inverse decoding function, eliminating the reconstruction gap. Specifically, we introduce SeqFlow, autoregressive normalizing flow for sequence data. In addition, we develop a new candidate sampling strategy that dynamically adjusts the exploration probability for each token based its importance. Through ex- tensive experiments, our NF-BO method demonstrates superior performance molecule generation tasks, significantly outperforming both traditional and recent LBO approaches.
INTRODUCTION
1964) has been broadly applied across various areas Bayesian optimization (BO) (Kushner 1962.
such as chemical design (Wang & Dowling, 2022), material science (Ament et al.) (02T), and hy- perparameter optimization (Wu
2019). BÖ aims to probabilistically optimize an expensive and black-box objective function using a surrogate model to find an optimal solution with mini- mal cost. Although BO is effective in continuous spaces, its application to a discrete input space still remains challenging (Oh et al.] 2019 Deshwal & Doppa 2021). Latent Bayesian Optimiza- tion (LBO) (Gömez-Bombane
al., 2018 Tripp et al. (020)) addresses this challenge by per- forming BO in lower-dimensional latent space learned by a generative model such as Variational AutoEncoders (VAEs) (Kingma & Welling 2014). LBO performs optimization in continuous space by mapping the discrete input into continuous latent space with the VAEs (Kssner et al. 2017 Jin et al. 2018 Samanta et al.) 2019).
Objective Values of Adip Task
However, the reconstruction of VAE is not always per- fect, leading to value discrepancy problem, which indi- cates that given sample encoded the latent space, its decoding may not result in the same sample in the input space. F the value dis- crepancy problem by presenting the distributions of ob- jective values before and after the reconstruction using a pretrained SELFIES VAE (Maus et al. 2022), focus- ing on data with the top 10% of objective values.
Original top-c
After reconstruction
embedding in
Density
10.
0.0
0.2
0.4
0.6
Value
Figure Visualization of value discrep- ancy problem.
Corresponding author

---

Published as a conference paper at ICLR 2025
y
y
Encoder
Decoder
Flow
Inverse g 11
X
Z
X
Z
q Pe
g
(a)
(b)
(a) Most existing LBO approaches suffer from the value discrepancy problem y * induced by the reconstruction gap, Po (x)) * x. This results in that the latent representation corresponds to different evaluation values y and ! due the reconstruction error, where x * X. (b) Our NF-BO effectively addresses the value discrepancy problem by employing a normalizing flow model that ensures one-to-one mapping between x and z via the invertible flow and inverse processes, g and g i.e., (g(x)) = So, the latent representation is consistently associated with the same evaluation value y.
Figure 2:
During the optimization process, these models often refine the latent space by training on newly searched data and their corresponding objective values (Maus et al.) 2022] Lee et al. (023). It requires re-encoding data points to find their latent representations, which also makes the value discrepancy problem. The previous method (Chu et al. (024) addressed this by inverse the data with an iterative approach.
To address the problems efficiently without re-evaluations/iterative procedures, we propose Nor- malizing Flows-based Bayesian Optimization, referred to as NF-BO, that leverages an invertible function for discrete sequence data. This approach establishes one-to-one encoding function from the input space to the latent space, along with its left-inverse decoding function, effectively resolving the value discrepancy problem.
Figure 2ereeeexplln explains the value discrepancy problem in (a) and how our NF-BO model addresses it using flow and inversion (b). Apart from the value discrepancy problem, additionally introduce token- level adaptive candidate sampling for more effective local search. The sampling scheme dynamically adjusts the sampling distribution based on the importance of each token to more focus on promising areas.
The contributions of our research are as follows.
We propose NF-BO to address the value discrepancy problem, which commonly occurs in Latent Bayesian Optimization (LBO). NF-BO leverages normalizing flows to establish one-to-one encoding function from the input space the latent space, with its left-inverse decoding function ensuring accurate reconstruction. To the best of our knowledge, NF-BO is the first work to integrate normalizing flows into LBO.
We propose a Token-level Adaptive Candidate Sampling (TACS), enabling effective local search by adjusting the sampling distribution based the token-level importance.
Our extensive experiments multiple benchmarks demonstrate the superiority of the pro- posed method in optimizing high--ameesiionl and structured data, consistently outperform- ing existing latent Bayesian optimization and traditional optimization methods.
2 RELATED WORKS
Latent Bayesian Optimization. Latent Bayesian Optimization (LBO)
et al.,
2018
Eissman et al. 2018 Tripp et al.) 2020 Griffiths & Hernández-Lobato, 2020) Grosnit et al,
Siivola et al. 2021 has emerged an effective approach overcome the limitations of tra- ditional Bayesian Optimization (BO), particularly in nigh-dimensional or discrete input spaces. By embedding discrete sequences into continuous latent space, typically using Variational Autoen- coders (VAEs) (Kingma Welling, 2014; Higgins et al.) 2017), LBO enables efficient optimization of complex problems, discussed in (González-Duque al." 2024 with comprehensive review.
2021
2

---

Published as a conference paper at ICLR 2025
To improve this mapping, prior works have proposed novel architectures improve reconstruction
quality (Kusner et al. 2017; Jin et al. 2018: Lu et al. 2018: Samanta et al. 2019) or utilize uncer-
tainty for increased robustness (Notin et al. 2021] Verma et al. [022]. In particular, LaMBO (Stan-
ton et al. )022) introduced a masked language model-based architecture, and LaMBO-2 (Gruver
al.
2024) developed inffuusionbassed approach extend prior methods.
Recent LBO works, such as LOL-BO (Maus et al.] 2022) have introduced the concept of trust re- gions (Eriksson et al., 2019) in the latent space. ROBOT (Maus et al.) (023) have emphasized the importance of incorporating diversity measures to further support diverse solutions. CoBO (Lee) al. (023) implements a novel loss function to improve the alignment between the latent space and the objective function. However, these methods still encounter the value discrepancy problem, where the output value from the decoded input inconsistent with the original value.
Normalizing Flows. Normalizing Flows (NFs) (Rezende & Mohamed 2015) are a class of gen- erative models that transform simple, known probability distribution into more complex one and vice versa. Each layer in these models is designed to be invertible, with a tractable Jacobian deter- minant, which facilitates efficient computation and flexible modeling of complex data distributions. Early NF models (Dinh et al. 2015 2017 Kingma & Dhariwal [20 8: Ho et al. 2019 Durkan et al.) 2019) have demonstrated their effectiveness generating righ-quuaity images using coupling-basee techniques, ensuring tractability and scalability.
More recently, NFs have also been developed not only for generating images but also for expanding their applicability to a wider range of data types. For instance, methods like (Ziegler & Rush 2019) specifically addressed the challenges in modeling discrete data by integrating NFs within a VAE framework, jointly learning latent distributions and improving the expressivity of the latent space.
To the best of our knowledge, our work is the first work that applies NFs in the context of LBO to deal with the value discrepancy problem by introducing new model SeqFlow in Section 4.2
3 PRELIMINARIES
Bayesian optimization (BO) has widely been applied to optimize black-box (unknown) objective functions where their evaluations are expensive. Let X' and be the input space and solution, respectively. The goal of BO is to find the optimal solution x that maximizes a black-box objective function f. which can be formulated as:
argmax f(x)- XEX
Since f is unknown, BO typically constructs surrogate model ff to approximate the true function f. With the surrogate model, BO searches for the optimal points with an acquisition function a as follows:
X = argmax oxx; J.D), XEXXand
(2)
where D - ((x(0 y 31 represents the accumulated data, X is data point selected based on the acquisition function, and Xeand C. X' is a candidate set. In trust region- local Bayesian optimization such as TuRBO (Eriksson et al. 2019), Xeand is selected within trust region that is often centered at a current optimal point (e.g., anchor point). The trust region limits the search space to promising small regions, thereby easing the difficulty of optimization.
Normalizing Flows (NFs) (Rezende & Mohamed 2013) are a class of generative models for mod- eling the data distributions p(x) through a sequence of invertible transformations, offering exact density evaluation and sample generation. NFs are formulated as follows:
(3) where g and g denote the forward and inverse transformation, parameterized by 6, ensuring that
==g(x;0) 0), x= 9=;-1(2;0))
each mapping is bijective and differentiable. The determinant of Jacobian det Jy (x) |1 computes
the change in volume induced by g. which is important for density calculations. The training of these flows involves minimizing the following negative log-likelihood:
2g dx
L -= - [logp(x)] Exxxx logp(z) + log det
(4)
This ensures that the model accurately captures the underlying data distribution, allowing efficient generation.
3

---

Published as a conference paper at ICLR 2025
4 METHODS
We propose Normalizing Flow-based Bayesian Optimization (NF-BO), which leverages Normal- izing Flows (NFs) generative model combined with adaptive candidate sampling for effective optimization. To begin with, we introduce Latent Bayesian Optimization (LBO) and the value dis- crepancy problem induced by incomplete reconstruction of the generative model used in LBO (Sec- tion 4.J. Next, we present an autoregressive NF model, SeqFlow, specifically tailored for sequence generation, which addresses the value discrepancy problem by accurate reconstruction (Section (42). Additionally, we propose Token-level Adaptive Candidate Sampling (TACS), which constructs di- verse candidate set within trust regions (Section 4.3) Finally, we delineate the overall process of our NE-BO (Section 4.4)
4.1 PROBLEM STATEMENT
Although BO has shown its effectiveness in various optimization tasks, it has difficulty performing over the discrete domain, such chemical design (Griffiths & Hernández- Lobato 22200; Wang & Dowling, 2022). To address this issue, recent works (Gómez-Bombarelli et al. 2018 Tripp et al., 2020) have studied Latent Bayesian Optimization (LBO) that performs BO in continuous latent space after embedding the discrete input data into the latent space. LBO can be formulated as:
z" argmax (pp((z)),
where Z is latent space and pa 2 - X' is the decoder parameterized by 0. LBO uses an encoder-decoder structure to map complex inputs into an effective representation in the latent space and then performs search in this latent space. Note that the formulation assumes the decoder pe is deterministie. I,OO searches for the optimal points using acquisition function a as follows:
X (e where argmax olz; f,D).
(6)
Ji=1 represents the accumulated data, X and ă are the next evaluation point and its corresponding latent vector in the candidate set Zeand S Z. s Z is a surrogate model for the composite function f o Pe 2 - y
D ((x(0,2)) ())
Value Discrepancy Problem. LBOs generally learn a surrogate model in the latent space and construct the data ((x(1) (0) y (0) D where y(() [x)()). (() (x(x(()) with the encoder
9s. assuming complete reconstruction (D) pe (e(gg(( (x))) and identical function values, i.e., y (6)
f(x()) = (z))(x))) (Tipp et al. 2020 Maus et al. 2022 Lee et al. 2023 Chen et al. 2024
However, in practice, there exists a reconstruction gap in VAE and it results in the discrepancy
between the function values evaluated at input data x and its reconstruction & as follows:
+ & and (x) * f(x), where & po (x)))
This value discrepancy problem propagates errors throughout the optimization process, leading to suboptimal optimization results. To mitigate this issue, an ideal generative model in LBO should exhibit perfect reconstruction, ensuring that any point in the input space can be accurately mapped to the latent space and vice versa. This property resolves the value discrepancy problem by ensur- ing that the generated data accurately reflects the characteristics of the original data. As result, error propagation during optimization is minimized, leading to improved optimization performance. Motivated by this, we introduce a new LBO built on normalizing flows.
4.2 SEQFLOW
To address the value discrepancy problem in existing LBOs, propose Normalizing Flow-based Bayesian Optimization (NF-BO), leveraging NF's ability in modeling the data distribution via a one-to-one mapping between the input space and the latent space. To efficiently perform NF-BO on a long sequence of discrete data, we propose a novel discrete normalizing Flow model Flow)low).
autoregressive
SeqFlow learns the distribution p(x) of the sequence of discrete data x = [ xr.). where No is sequence of token indices, using two components: (i) mapping function between
4

---

Published as a conference paper at ICLR 2025
Autoregressive Normalizing Flow
Token Embedding
V = K
zk-1
g
Z
M
M
Embedding To
= z
z K-1
Z 6
Z
Token
Figure 3: Overall pipeline of SeqFlow. Given the input space of sequence discrete values x, SeqFlow first maps the discrete values continuous representation v and efficiently transforms them via autoregressive transformations {9 Ji-o latent representation z" in the encoding phase (top pathway). In the decoding phase (bottom pathway). SeqFlow reconstructs x from z through the inverse of transformations, SeqFlow ensures the perfect reconstruction of the discrete input.
the continuous representation RLFF and a discrete input x and (ii) a density model p(v) (i.e, normalizing flow). Here, L represents the number of tokens in a sequence and F is the embedding dimension. The mapping function (i) is defined as:
X; = arg max sim (Vi:
where sim(:;) is the cosine similarity, e; E IR is an embedding vector of j-th token. All em- beddings are initialized by random vectors drawn from a normal distribution after L2 normaliztion, i.., ||ey12 = for all j. As a result, x; is the index of the token whose embedding vector e, is most similar to the continuous representation vector V,. Based on the density model p(v) and the mapping function, we define the likelihood of input discrete sequence p(x) as follows:
p(x) = - p(v)) [ dv,
p(x:/v;) 8x,,x, where X; arg max sim(v;, ey),
where ò is the Kronecker delta function and &; is the index of the most similar embedding vector v.. However, directly calculating Eq. (9) is intractable. So, we introduce the variational distribu- tion ((4) (Ho et al. (2019) and optimize the likelihood p(x) by maximizing the evidence lower bound (ELBO), which derived as:
logp(x)
logp(v) Σ (logp(x:lvv)) - log (ogg(v;(x)))
(10)
We define the distribution (vv, as an isotropic Gaussian distribution centered at the embedding of x;, iee,, (ex,,, T²T). Additionally, we sample only v, from g(v,/x;) that satisfies p(x:/v;) 1. The constrained version q(v,(x;) defined as:
g(vi)bx.)
if ((x: otherwise
g'(v.(x;)
(11)
0,
where Z is normalization constant. We accept a sample v; with probability g/v Through the constrained sampling within the domain where the condition holds, we effectively make the practical sampling distribution <(v.(x.) closer to p(vs[x;). The example of the distribution o' depicted in the Appendix G
d(vx)))))
We employ a negative log likelihood to maximize logp(v), which serves as a normalizing flow loss that enhances the model's ability to generate valid continuous representations The Negative Lik-Liielihood LNLL is defined as follows:
K
Ōg* 2zk+I
LNL = - logg(vv = log p(z)
log det
(12)
5

---

Published as a conference paper at ICLR 2025
where gf represents k-th transformation in the flow sequence g and 2 is the output of the k-th transformation.
Also, implement simple variant of the contrastive loss to maximize the cosine similarity be- tween v; and ex, for x; and distance from other embeddings:
Σ sim(vi, vi, Cx,)
1 Σ sim, oy), oj (nix(E((ex)))
Ssiml (. 0) - -
(13)
where ey an embedding uniformly sampled from embedding set except for ex which corre- sponds to the token x, The contrastive loss encourages diverse token embeddings in a given con- text. To train our SeqFlow model, we combine the similarity loss with the Negative Log-Like Ligeliied (NLL) loss of normalizing flows. The final loss of our model is given by:
LNEBO CNLL + smm (v,am((ve)) (v,e),
(14)
where Å is the hyperparameter that balances the NLL loss and the similarity loss.
To effectively represent a long sequence of discrete values, we adopt an autoregressive normalizing flows (Ziegler & Rush 2019). Our model defines the flow for encoding:
Autoregressive Normalizing Flows.
V g (z: 0), z = (v;
(15)
where 9. 9 entire flow and its inverse transformation. respectively. To be specific, autore- gressive NF is composed of K series of autoregressive transformation blocks and each block for EE (O..... K 1} operates follows:
z:
(zf:z (
0t), and 2 = (z, :
1 1+ .0)), 2i
(16)
where z denotes i-th token output vector of the k--th block. The initial input to the first block is Our autoregressive block (g*) consists of
z" and the output of the final block is z several coupling layers, which aggregate information from the previous tokens. This helps the flow model to capture the long-range dependencies within the sequence for effective sequence modeling. More details on the architecture of the autoregressive normalizing flow model is in the Appendix H
Injectivity of our SeqFlow. The SeqFlow ensures injectivity through the invertibility of the trans- formation function g- This function maps the embedding ex to a latent representation z, and the decoding process serves as the left inverse of this encoding. As stated in Proposition [ and Propo- sition 2. this guarantees that for every input x, the operation g(ex) and its inverse will precisely reconstruct x.
Proposition 1. Let g be Normalizing Flows and h is an injective function with nonempty domain X. Then, f := gol is left invertible, i.e., f-1 o idx, where h- -1 is the left inverse of h and f- = h 1
Remarks. Proposition Timplie that our construction provides perfect reconstruction. To be specific, SeqFlow consists of two functions: (i) function h to map discrete sequence data to sequence of embeddings in the continuous space and (ii) Normalizing Flows g defined in the continuous space. the function h is injective, with its left inverse and the inverse of NFs, SeqFlow achieves the perfect reconstruction.
Proposition 2. Assume the elements of embedding set E (er, eg.ee......... are distinct and 12-normalized, i.e., O₁ Øjz for all + j and [ee 1. Given list of L natural num- bers x [2,,X2,.... .2.,,,..,, x.] NL, a mapping function h. is defined as h(x) == Cx where Cx [ex, "xp, .xp...., 1x, Then, h is injective and the function ---(v)= : [arg max, sim(v, ei) where sim(e,, e;) = e. e,, is a left inverse of h, i.e., h (h(x)) =
The proofs of Proposition] T and Proposition 2 provided in Appendix This approach ensures all information is preserved during encoding and decoding through a one-to-one function and its left-inverse. This is crucial for applications that demand exact input reconstruction.
Moreover, the reliability of the decoding function h(()) ensures that any generated latent variable ac- curately reverts to its corresponding input sequence. This capacity is essential for resolving the value

---

Published as a conference paper at ICLR 2025
discrepancy problem often observed in other latent-based optimization models, where reconstructed outputs might not match the original inputs. This enhancement increases the overall efficacy of the optimization process, making SeqFlow robust framework for handling discrete sequence optimiza- tion tasks.
4.3 LEVEN-LEVEL ADAPTIVE CANDIDATE SAMPLING
In this section, we present Token-level Adaptive Candidate Sampling (TACS) improve the can- didate sampling process of trust region-based local search BO methods (Eriksson et al., 2019; Maus al. 2022 Lee al. 2023). These local search BO methods search next query points constrained in promising areas centered around an anchor points, derived from the best input found in the data history.
Most previous rust-region-basee approaches utilize Thompson sampling finite set of candidate points Zeand by perturbing subset of dimensions of an anchor point (Eriksson et al. 2019). We observe that the existing approaches select subset of dimensions to be perturbed uniformly, which can lead to less effective exploration especially when it is applied to our SeqFlow. To address this, propose Token-level Adaptive Candidate Sampling (TACS), which samples candidates regarding the importance of each latent token. Specifically, we sample subset latent tokens an anchor point for perturbation from token-level probability distribution, defined by the relative importance of each token. This allows TACS to perform a dense search over important tokens while sparsely exploring less important ones with limited resources.
To identify important tokens at the anchor point (x, z), we utilize the Pointwise Mutual Information (PMI) between each token z, and the sequence x.
(z)z)) PMI(x, z ()) log p(x/z) log p(x(z) p(x(z_.)
(17)
p(x|z)= - p(xlv) = I p(xx/vi).
Zi+l...... } A Monte Carlo approximation is employed to es- timate p(x[z_;), and to stabilize computations, small constant is added to p(x,/v;). This PMI score (z) measures the impact of latent token the sequence enabling efficient exploration along the most important dimensions. Using the PMI score, we define the token-level sampling probability (z) as:
where Z-; [21,
exp (wi(z)/T) m zz) min (x), s,(z) Σ, exp (w) (2)/T)
(18)
where k is constant scaling factor, and indicates the temperature. The softmax with temperature allows for flexible adjustment in focusing on the importance of different tokens. For example, has higher value, the candidate set is uniformly sampled, disregarding the token-level impor- tance. Conversely, a lower т concentrates sampling more densely on the tokens with the highest importance.
4.4 OVERALL BAYESIAN OPTIMIZATION PROCESS
In this section, we present our overall NF-BO framework, which illustrated in Figure 4 For each iteration, the BF-BO framework begins by training the SeqFlow model with the loss function L as defined in Eq. (14). using the dataset D {xx(1) yl)).. For training the ScgFlow model, we sample variational vector following the distribution g', as described in Eq. (11). After training the SeqFlow, we construct the latent vector (*) corresponding to the input x(0 and then use it to train the surrogate model f. Then, we select anchor points Zame based on their corresponding objective values and generate trust regions centered on them. To perform local search, the candidate set Zeand is drawn within the trust region, using the Token-level Adaptive Candidate Sampling (TACS) method. Finally, the acquisition function a determines next query point followed by decoding and evaluating to update the best score. This procedure is repeated until the allocated oracle budget T expended, continuously improving the SegFlow model throughout the optimization process. For better understanding, the pseudocode for NF-BO is provided in the Appendix F
7

---

Published as a conference paper at ICLR 2025
Dictienary
[0]
X
v RAKP R
o, E 2
[N]
Fg
[C]
[(x, y)}
Flow
(z, y)
Evaluate & Add
Embeaking
Embrskdngg
over Latent Space
[
(8)
V=
Inverse
X
V.
S
Embeiiing
{Z, y)3
Train
Sumogatcc Model
Acquisition F'unction
(Z)
Token-level Adipiive Candidate Sampling (TACS)
Candidates
Sample and Perturbate Tokens
X
Eq. (17)-(18 (183
Figure 4: Overview of NF-BO. We employ normalizing flows, SeqFlow, mapping func- tion between a discrete input space and continuous latent space. Each discrete input token x; mapped to its corresponding embedding vector from the dictionary. A surrogate model is then trained using the latent representation encoded by the flow model g and the associated function value to emulate the objective function. To enhance the efficiency of trust region-based local search, we propose Token-level Adaptive Candidate Sampling (TACS). In TACS, candidates token-level
for the acquisition function generated by perturbing tokens, sampled according sampling probability specified in Eq. (18). Given these candidates and the surrogate model, we select the next query points ž by the acquisition function. Next, the inverse model generates the embedding v and searches the most similar embedding and return the corresponding index as a X.
5 EXPERIMENTS
5.1 TASKS
We validate NF-BO across various benchmarks focusing de novo molecular design tasks. Initially, we conduct experiments on the Guacamol benchmarks (Brown et al.) 2019). specifically targeting seven challenging tasks where optimal solutions are not readily found. For these bench- marks, we evaluate NF-BO and the baselines under three different settings, each varying the number of initial data points and the additional oracle budget: (100, 500), (10,000, 10,000). and (10,000, 70,000). Subsequently, we evaluate our method on the PMO benchmarks (Gao et al. 2022) which consists of tasks, including albuterol similarity, and amlodipine MPO.
5.2 BASELINES
In the Guacamol benchmark, use LSBO, TuRBO-L (Eriksonn et al. 2019). W-LBO Tripp al. 2020), LOLBO (Maus et al., (0022), CoBO (Lee et al.. 2023). and PG-LBO (Chen et al. 2024) as the baselines. In the PMO benchmarks,
compare method with 25 molecular design algorithms.
These include generative models (e.g., GANs and VAEs), machine learning models (e.g., s, Reinforce- ment Learning), and optimization algorithms (e.g.. MCTS and GA). More detailed explanations of the baselines are in Appendix
5.3 IMPLEMENTATION DETAILS
We employ Thompson sampling (Eriksson et al. 2019) as the acquisition function, and our surrogate model is sparse variational Gaussian process (Snelson & Ghahramani 2005 (Hnnman al. 2015 Matthews 2017) enhanced with a deep kernel (Wilson et al. (016). For the Guacamol and PMO benchmarks, pretrain using 1.27M unlabeled Guacamol and 250K ZINC datasets, respectively,
8

---

Published as a conference paper at ICLR 2025
Amiodipine MPO (edip)
Modian melecules 2 (med2)
645
Amiodiiine NPO (adip)
240
Median molecules 2 (medZ)
2.33
0.14
Score
8.2
Beet Scooee
5.00
Scores
0.33
030
2.22
230
200
660
602
500
126
54X
3BC
10C
0:33
220
301
26
Number of Dradee
Number
TuRBO-L
CoBO
Figure 5: Optimization results of NF-BO on Guacamol benchmarks comparing performance with baselines under two oracle budget settings: (100, 500) (left) and (10,000, 10,000) (right). The shaded regions indicate the standard error over 5 trials.
Table I: PMO results across various methods and assembly. The table presents scores and rankings for 6 evaluation metrics illustrating the comparative performance each method. Score is the sum of all 23 tasks constituting the PMO benchmark computed summarize the overall performance.
Top-1 Score (Rank)
Top-10 Score (Rank)
Top-100 Score (Rank)
AUC TuCTl
AUC LLC'Top-10 Score (Rank)
AUC Toop-100 Score (Rank)
Methods
Assembly
Score (Runk)
Bayesian
Optimization
NF-BO
SELFIES
18.095 (1)
17,692 (1)
17,037 (1)
15,539 (1)
14.737 (1)
13.423
GP BO
Pragments
15.345 (7)
14,940 (6)
14.365 (6)
13.798 (5)
13.156 (5)
12.122
VAEE BO
SELFIES
11.423 (17)
9,788 (19)
7.622 (22)
10.589 (17)
8.887 (19)
6.899 (22)
VAF: BO
SMILES
10.926 (21)
9.435 (21)
7.623 (21)
10.197 (19)
8.587 (21)
6.909 (21)
JT-VAE BO
Fragments
10.296 (23)
8.671 (24)
7,037 (24)
9,973
8.358 (24)
6.740 (23)
Reinforcement Learning
REINVENT SMILES
16.772 (2)
16.654 (2)
16.297 (2)
14.711 (2)
14.196 (2)
13.445
REINVENT MoIDQN
SELFIES
16.059 (5)
15.889 (4)
15.377 (3)
14.077 (4)
13.471 (4)
12,455
Atoms
7.143 (26)
6,495 (26)
5,435 (26)
6.332 (26)
5.620 (26)
4.528 (26)
Genetic Algorithm
Graph GA
Fragments
16.244 (4)
15.946 (3)
15.342 (4)
14.356 (3)
13.751 (3)
12.696 (3)
STONED
SELFIES
14.257 (8)
14.201
(8)
14.017 (7)
13.256 (7)
13.024
(6)
12.518 (4)
SMILES GA
SMILES
13.123
12,997 (9)
12,824 (9)
12.357 (10)
12,054
(8)
11.598 (7)
SynNet
Synthesis
13.105 (12)
12.279
10.768
12,425 (9)
(9)
9,914
GA+D
SELFIES
11.942 (16)
11,696 (15)
(13)
9.387 (24)
8.964 (18)
8.280 (15)
Hill Climbing
LSTM HC
SMILES
16.754 (3)
15.880 (5)
14.621 (5)
13.611 (8)
12.223 (7)
10.365 (8)
LSTM HC
SELFIES
13.770 (9)
12.894 (10)
11.657 (12)
11.441 (14)
10.246 (15)
8.595 (13)
DoG-Gen MIMOSA
Synthesis
15.633 (6)
14.772 (7)
13.653 (8)
12.721 (8)
11.456 (10)
9.635 (12)
Fragments
12.524 (15)
12.223 (13)
11.717
11.378
10.651 (13)
9.708 (11)
following the previous settings. We employ 1,000 initial data points and an additional 9,000 oracle calls following the PMO benchmarks.
5.4 RESULTS ON GUACAMOL BENCHMARKS
We compare the optimization results of our NF-BO with six LBO baselines in two experimental settings: 500 and 10K additional oracle budgets on two Guacamol tasks. Figure 5 presents the main experimental results, while the other results five tasks provided in Appendix B The experimental results demonstrate that our proposed NF-BO consistently outperforms other VAE- based LBO methods in all tasks and settings.
5.5 RESULTS ON PMO BENCHMARKS
We also conduct experiments demonstrate the effectiveness of our NF-BO against 25 baseline models, including various generative models and optimization algorithms, across 23 PMO bench- mark tasks. Our evaluation metrics included Top-1, 10, and 100-100 scores, well the Area

---

Published as a conference paper at ICLR 2025
Distinct Sample
Amild pinc MPO (acip)
Median molecules (med21
0.85
0.40
TACS
0.00
Best Scone
Dutto
0.20
Dest
642
0.75
0.0
HF 10:80 (er TACS) Jann TACO
0.22
NR 80 TACS)
0.70
(ecn
1dpp
me3t
Task
Number
Figure 6: Comparison of distinct sample ratios with and without TACS in two Guacamol tasks.
Figure 7: Comparison of performance with and without TACS in Guacamol benchmarks. The shaded regions indicate the stan- dard error over trials.
Under the Curve (AUC) for these metries, all based on Oracle calls. The experimental results are in Table
The scores for each individual task detailed in Appendix A The table shows that NF-BO achieves the best performance with 1st rank on five out of six metrics, In particular, NF-BO signifi- cantly enhances the performance of VAE BO, which also uses SELFIES, improving its average rank from 19th to 1st.
ANALYSIS
In this section, we provide the analysis of our NF-BO on the Guacamol. Experiments were imple- mented with 10,000 initial data points and an additional oracle budget of 10,000.
6.1 CANDIDATE DIVERSITY WITH TACS IMPLEMENTATION
We evaluate the proportion distinct samples within a set of 1,000 candidates generated in two different Guacamol tasks with and without TACS. Each experimental setup subjected to Monte Carlo approximation times to estimate expectation in Eq. (17), and we conducted five independent experiments averaging the results. We pre-trained Seq model and 10 different anchor points to generate trust regions.
In Figure 6 the result with TACS has higher ratio of distinct samples compared to those without TACS, underscoring its effectiveness in enhancing the diversity the candidate pool. This implies TACS improves the exploration capacity of the BO, which is crucial for optimization performance. We provide optimization performances with different temperatures in TACS in Appendix C
6.2 ABLATION STUDY
Figure our ablation studies that illustrates the effectiveness of our Token-level Adaptive Candidate Sampling (TACS) strategy, shows its impact on performance across these tasks in the Guacamol benchmark. From the analysis, it is evident that the incorporation of TACS significantly enhances performance, confirming its benefit in optimizing the search process.
CONCLUSION
In conclusion, the proposed NF-BO method, which leverages normalizing flows, makes significant improvements in the domain of Bayesian optimization, especially for handling molecular data. This approach not only addresses the value discrepancy problem through one-to-one function from the input space to the latent space and its left-inverse function but also enhances the effectiveness of the search process with a novel token-level adaptive candidate sampling strategy. Our comprehensive evaluations across diverse benchmarks have demonstrated the superiority of NF-BO over traditional methods and other LBO techniques, confirming its potential to reshape the landscape of optimization strategies in various scientific and engineering applications.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For reproducibility, we elaborate on the overall pipeline of our work in Section 4 In our main paper and appendix, we also illustrate our overall pipeline and pseudocode for NF-BO, respectively. Code available at bttpp::/gihhbb..om////aaaaa/
ETIICS STATEMENT
Our main contribution, NF-BO, aims to design molecules with desired properties, e.g. Amlodipine MPO in the Guacamol task. However, this could lead unintended consequences, such as the creation of harmful substances like illicit drugs, requiring the exercise of extreme caution.
ACKNOWLEDGEMENT
This research was supported by the ASTRA Project through the National Research Foundation (NRF) funded by the Ministry of Science and ICT (No. RS-2024-00439619,
REFERENCES
Sebastian Ament, Maximilian Amsler, Duncan Sutherland, Ming-Chian Chang, Dan Gue- varra, Aine B Connolly, John M Gregoire, Michael O Thompson, Carla P Gomes, and R Bruce Van Dover. Autonomous materials synthesis hierarchical active learning of nonequilibrium phase diagrams. Sci. Adv., xeabb44,,, 2021.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain Vaucher. GuacaMol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3): 1096-1108, 2019.
Taicai Chen, Yue Duan, Dong Li, Lei Qi, Yinghuan Shi, and Yang Gao. PG-LBO: Enhancing high- dimensional Bayesian optimization with pscudo-label and gaussian process guidance. In AAAI, 2024.
Jaewon Chu, Jinyoung Park, Seunghun Lee, and Hyunwoo Kim. Inversion- based latent bayesian optimization. In NeurlPS, 2024.
Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for Bayesian opti- mization over combinatorial spaces. In NeurlPS, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. ICLR Workshop, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In NewrIPS, 2019.
Stephan Eissman, Daniel Levy, Rui Shu, Stefan Bartzsch, and Stefano Ermon. Bayesian optimiza- tion and attribute adjustment. In UAI, 2018.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. NeurIPS, 2019.
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: benchmark for practical molecular optimization. In NeurIPS, 2022.
Rafael Gómez Bombarll, Jennifer Wei, David Duvenaud, José Miguel Hernández-Lobate Benjamín Sánchez-Lengelinng Dennis Sheberla, Jorge Aguilera-Iaarrguing Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Gude Guzik. Automatic chemical design using a data-driven contin- representation of molecules. ACS Cent. Sci., 4(2):268- 2766 2018.

---

Published as a conference paper at ICLR 2025
Miguel González- -uqqe, Richard Michael, Simon Bartels, Yevgen Zainchkovskyy. Søren Hauberg, and Wouter Boomsma. A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences. 27Xx::4406.04739, 2024.
Ryan-Rhys Griffiths and José Miguel Hernández-Loban Constrained Bayesian optimization for automatic chemical design using variational autoencoders. Chem. Sci., 11(2):577-586, 2020.
Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander Cowen- Rivers, Lin Yang. Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, et al. High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning. urXiv:2106.03609 2021.
Nate Gruver, Samuel Stanton, Nathan Frey, Tim Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. NeurlPS, 2024.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational gaussian pro- cess classification. In Artificial Intelligence and Statistics, pp. 351-360. PMIR, 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerehner. VAE: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In ICML, 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In ICML, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In NeurlPS, 2018.
H. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng., 86(1):97-106, 1964.
Harold J Kushner. A versatile stochastic model of function of unknown and time varying form. J. Math. Anal. Appl., 5(1):150-167, 1962.
Matt Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoen- coder. In ICML, 2017.
Seunghun Lee, Jaewon Chu, Sihyeon Kim, Juyeon Ko, and Hyunwoo J Kim. Advancing Bayesian optimization via learning correlated latent space. In NeurlPS, 2023.
Xiaoyu Lu, Javier Gonzalez, Zhenwen Dai, and Neil D Lawrence. Structured variationally auto- encoded optimization. In ICML, 2018.
Alexander Graeme de Garis Matthews. Scalable Gaussian process inference using variational meth- ods. PhD thesis, 2017.
Natalie Maus, Haydn Jones, Juston Moore, Matt Kusner, John Bradshaw, and Jacob Gardner. Local latent space Bayesian optimization over structured inputs. NeurlPS, 2022.
Natalie Maus, Kaiwen Wu, David Eriksson, and Jacob Gardner. Discovering many diverse solutions with Bayesian optimization. In AISTATS, 2023.
Pascal Notin, José Miguel Hernández-Lobbto, and Yarin Gal. Improving black-box optimization in VAE latent space using decoder uncertainty. In NeurIPS, 2021.
Changyong Oh, Jakub Tomezak, Efstratios Gavves, and Max Welling. Combinatorial Bayesian optimization using the graph cartesian product. In NewrlPS, 2019.
12

---

Published as a conference paper at ICLR 2025
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Chattaraj, Niloy Ganguly, and Manuel Gomez-Rodriguez. NeVAE: A deep generative model for molecular graphs. In AAAI, 2019.
Eero Siivola, Andrei Paleyes, Javier González, and Aki Vehtari. Good practices for Bayesian opti- mization of high dimensional structured spaces. Applied Al Letters, 2(2):22,, 2021.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pscudo-inputs Advances in neural information processing systems, 18, 2005.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Green- side, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In ICML, 2022.
Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato, Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In NeurlPS, 2020.
Ekansh Verma, Souradip Chakraborty, and Ryan-Rhy Rhys Griffiths. High dimensional Bayesian opti- mization with invariance. In ICML Workshop, 2022.
Ke Wang and Alexander W Dowling. Bayesian optimization for chemical products and functional materials. Curr. Opin. Chem. Eng., 36: 36:100728, 2022.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning- In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Jia Wu, Xiu- Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng. Hyperparame- ter optimization for machine learning models based on Bayesian optimization. J. Electron. Sci. Technol., 17(1):26-40, 2019.
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In ICML, 2019.
13

---

Published as a conference paper at ICLR 2025
Table 2:
Detailed results on PMO benchmarks. The table presents scores and standard deviations across 6 evaluation metrics, with each score representing the mean of independent runs. Addition- ally, the sum for each column is computed to summarize the overall performance.
Top-1
Top-10
Top-100
AUC Top-1
AUC Top-10
AUC Top-100
albuterol_site simtirrliimm
1.000 0.000
0,967 +: 0.011
0.847
0.862 + 0.014
0.010
0.708
amlodipine.mpo
0.802 1 0,028
0.798 1: 0.024
0.788
0.013
0.688 1 0.023
0.672 : 0.021
0.642 0.020
0.799 + 0.164
0.699 1 0.076
0.634 0.063
0.605 1 0.069
0.546
0.031
0.481 0.024
deco_hop
0.725 0.006
0.724 1: 0.007
0.724 0.007
0.685 0.004
0.675 0.003
0.662 0.003
drd2
1.000 1 0.000
1,000 1 0,000
0.999
0.001
0.932 1 0.004
0,875 = 0.005
0.788 1: 0.004
fexofenadine mpo
0.854 i 0.012
0.854 : 0.012
0.852 0.012
0.797 1 0.008
0.784 = 0.008
0.756 0.007
gsk3b
0.990 1 0,015
0,952 1 0.041
0.903 0.069
0.820 1 0.032
0.754
0.010
0.664 0.028
isomers c7h8n202
1.000 + 0.000
0.841 + 0.076
0.619 0.160
0.916 + 0.00S
0.748
0.062
0.525 + 0.126
isomers 10no2222l
0.946 + 0.028
0.935 + 0.008
0.933 0.007
0,881 + 0.010
0,842
0.009
0.757 0.009
jak3 median
0.894 + 0.052
0,884 0.061
0.866 0.076
0.709 + 0.036
0.649
0.037
0.574 + 0.040
0.422 + 0.022
0.419 + 0.023
0.409 = 0.021
0.352 + 0.007
0.340 = 0.006
0.307 0.004
median2
0.313 0.022
0,311 + 0.021
0.305 0.019
0.269 0.013
0,260
0.011
0.244 0.010
mestranol mestranol_similartyy
0.758 + 0.058
0,758 + 0.058
0.758
0.058
0.629 + 0.028
0.607
0.024
0.570 0.018
osimertinib.no mpo
0.880 1 0.010
0.878 1 0.010
0.872
0.012
0.838 1 0.004
0.828
0.005
0.788 0.005
perindopril_mpo
0.678 + 0.034
0.678 + 0.034
0.677 0.034
0.598 + 0.028
0.027 0.586
0.560 0.026
ged ranolazine_me _mpo
0.948 1 0.000
0,948 1 0,000
0.948
0.000
0.943 1 0.000
0,941
0.000
0.931 1 0.000
0.844 + 0.012
+ 0.011
0.838
0.012
0.698
0.010
0.647
scaffold hop sitagliptio.mp
0.769 1 0.172
0.767 1 0.170
0.733 0.141
0.646 1 0.087
0.629
0.087
0.608 0.085
0.764 + 0.075
0.757 + 0.079
0.722 = 0.090
0.578 + 0.032
0.516
0.029
0.427 + 0.025
thiothixene rediscovery
0.639 0.121
0.623 1 0.100
0.602
0.524 0.061
0.496 0.048
0.459 0.037
toglitazone_redinn rediscovery
0.476 + 0,040
0.475
+ 0.039
0.473 0.039
0,386 + 0.020
0.375
0.019
0.352 0.018
valsartan smarts
0.998 + 0.001
0.996 + 0.002
0,974 0.012
0.633 + 0.041
0.594 = 0.037
0,514 0.033
zaleplon_mpo
0.593 + 0.016
0.584 + 0.016
0.561 0.016
0.524 + 0.011
0.504
0.011
0.460 0.010
Sum
18,095
17.692
17.037
15,539
14.737
13.423
A DETAILED RESULTS ON PMO BENCHMARKS
We conducted experiments to demonstrate the effectiveness our NF-BO across 23 PMO bench- mark tasks. The full experimental results, including detailed scores and standard deviations for each task, are provided in Table 2 The evaluation metrics we used include Top-1, Top-10, and Top- 100 scores, well the Area Under the Curve (AUC) for these metrics, all based oracle calls. Our main findings show that NF-BO consistently achieves competitive performance across various tasks. Additionally, the AUC scores show comparable results in terms of further highlighting NF-BO's 's ro- bustness. These results suggest that NF-BO not only excels at identifying the best solutions but also maintains consistent performance across different tasks.
B ADDITIONAL RESULTS ON GUACAMOL BENCHMARKS
As referenced in Section 5.4 we compare our NF-BO with six LBO baselines across seven tasks in the Guacamol benchmarks. In this section, we present the results of the remaining tasks for the (100, 500) and (10,000, 10. 10.000) oracle settings, which were not covered in the main section, along with the results for the (10,000, 70,000) oracle settings. Figures 8 9 and TO display the results for the (100, 500), (10,000, 10,000), and (10,000, 70,000) oracle settings, respectively. In the case of PG-LBO (Chen 2024), we were unable to include results for the (10,000, 10,000) and (10,000, 70,000) settings due to infeasibility caused by excessive experimental time.
14

---

Published as a conference paper at ICLR 2025
Osimertinib (osmb)
Perindopril MPC (pdop)
Ranolazine MPO (rano)
0.80
0.51
Best Score
0.79
Best Score
6 o
Dest
0.48
0.68
0.76
400 Number Oracle
Number Oracle
Number of Oracle
Val Saisartan smarts (valt)
Zaleplon (zale)
10
0.48
Best 5core
10
LS80
0urs
10-17
Best
0.40
W-L00
LUUIO
10-2
PS-LB
0.36
CuRD
10-2
400 Number Oracle
100
200
3010
400
500
600
Number of Oracle
Figure 8: Optimization results Guacamol benchmarks under 500 additional oracle settings. Note that the valt task, the y-axis represented on log scale, and for this task, we also added two nonzero data points in the initial dataset of 100 for all methods. The shaded regions indicate the standard error over trials.
Osimertinib MPO (osmb)
Perindopril MPO (pdop)
Ranolazine MPO (rano)
0.92
0.85
0.90
0.90
Best Score
Best Score
0.70
Best Score
0.88
0.65
0.65
0.80
0.86
0.55
0.50
0.70
10K
12K
14K
-0K
13K
20K
13K
12K
14K
-6C
13K
20K
13k
12K
14K
20k
13K
20K
Number of Oracle
Number of Oracle
Number of Oracle
Valsartan smarts (valt)
Zaleplon MPO (zale)
1.0
0.80
0.75
0.5
0.70
0.6
0.65
1S50
TURBO
Best
0.4
CoOO
0.2
0.0
0.45
Number of
Number of Oracle
Figure 9: Optimization results on Guacamol benchmarks under 10K additional oracle settings. The shaded regions indicate the standard error over 5 trials.
15

---

Published as a conference paper at ICLR 2025
Amlodipine MPO (adip)
Median molecules 2 (mme22)
OsimerUinib MPO losmbb)
0.96
0.94
Best Score
Best Score
Best Score
0.92
0.39
0.90
0.36
0.88
0.75
0.06
0.70
0.84
30K
20K
40K
GUC B0K
13K
20K
4UK
60K
B0X
13K
2UK
4UK
60C
B0K
Number of Oracle
Number of Oracle
Number of Oracle
Perindopril MPO (pdop)
Ranolazine MPO (rano)
0.85
0.85
0.80
Best Score
6.75
Score
0.70
0.65
0.00
0.55
10K
10K
BOX
10K
20K
<OK
60K
Number of Oracle
Number of Oracle
smarts
Zaleplon MPO (zale)
0.80
1.0
0.75
Best Score
Best Score
Ours
0.65
TuROC-
0.4
0.60
WAUD
LOLIO
0.55
Ca60
0.50
0.45
10C
20K
40K
60K
80K
10K
20K
40K
60K
80K
Number of Oracle
Number Oracle
Figure 10: Optimization results on Guacamol benchmarks under 70K additional oracle settings. The shaded regions indicate the standard error over 5 trials.
16

---

Published as a conference paper at ICLR 2025
C DISTINCT SAMPLE RATIO WITH VARIOUS TACS TEMPERATURE
The distinct sample ratio quantifies the diversity of generated candidates by measuring the pro- portion of distinct samples within the total candidates. In Figure [T explore how varying the temperature parameter TACS affects this ratio on the Guacamol benchmark. Lower temperatures generally promote exploration by sampling impactful tokens within the input sequences in the latent space, increasing the diversity of candidates.
Distinct Sample Ratio by TACS Temperature
0.96
Unique Sample Ratio
0.94
med2
osmb
0.90
pdop
rano
valt
0.86
0.84
200
100
TACS Temperature
Figure 11: Distinct sample ratio with various TACS temperatures Guacamol benchmarks.
The experimental setup follows the same configuration as detailed in the analysis section of the paper. As result, we observe that for six of the seven tasks (excluding Rano), the distinet sample ratio increases as the temperature decreases, indicating that lower temperatures encourage a broader exploration of distinct candidates.
D ANALYSIS OF POINTWISE MUTUAL INFORMATION IN SEQFLOW
Pointwise Mutual Information (PMI) Value on Pretrained SeqFlow
60
80
100
Token Index i
Figure 12: Pointwise Mutual Information (PMI) value wi between each latent token z; and sequence x.
We analyzed the Pointwise Mutual Information (PMI) values of each latent token across different points in the sequence. The PMI values, denoted w; PMI(x, zi(z-),, were calculated 10 different points, and Monte Carlo methods were employed 10 times to ensure accuracy. The x-axis Figure 12 represents the token index in the latent z, while the y-axis measures the PMI value between each z; and Different colors stacked in the figure represent the cumulative PMI values measured from various points.
As observed in Figure there is a trend where the PMI values decrease as the token index increases. This tendency reflects the autoregressive nature of our model used, where earlier tokens tend to influence larger part of the sequence, exerting significant impacts subsequent tokens. This shows that early tokens in our model are important to the sequence generation and optimization processes.
17

---

Published as a conference paper at ICLR 2025
E PROOF OF LEFT INVERTIBILITY OF SEQFLOW
Proposition 1. Let g be Normalizing Flows and h is an injective function with nonempty domain X. Then f g is left invertible, i.e., f-1 o idx, where h- is the left inverse of and f- = h
Proof. NF g has inverse function g by definition and h has left inverse since every injective function with a nonempty domain has a left inverse. Let h- denote the left inverse of h. Then, f == h-1 g ogoo idx.
Proposition 2. Assume the elements of embedding set E = (0l, 02. C....,,. cjai are distinct and 12-normalieed, i.e, Bi * ej for all + j and |e,12 = Given a list of natural num- NL, a mapping function h is defined as h(x) := Cx where ex [ex] exp...., ex,]² Then, h is injective and the function h (v) == [arg max, sim(v;, eş) e5)l where sim 0j) = c/ cj is left inverse of h, ie., h- (h(x)) x.
bers x [11 x....,
Proof. Since function with nonempty domain is injective if and only if the function has left inverse, we show that h is the left inverse of
By definition, we have
h (M(x)) 1. (ex)
arg max sim e,)
(19)
Since the embeddings are distinct and 1.2-normalized, sim(e; 0ÿ) of ej satisfies
c/; 1 1, otherwise.
i
(20)
Thus, for each i, the maximum value of sim e;) occurs at j arg max, sim,ex; e₁) x1, Vi. Therefore, (hx()) = (-- [xi] x.
meaning
F PSEUDOCODE OF NF-BO
This section provides the pseudocode of NF-BO frameworks on Algorithm topk in the algorithm refers to selecting the top A data points with the highest objective values from the dataset D. The number of data specified in Table
G VISUALIZATION OF SAMPLING DISTRIBUTION: FEASIBLE REGIONS IN LATENT SPACE
Figure 13 illustrates the simplified example of constrained sampling distribution g' ('(vv (x. based on Eq. (11). In the figure, the Voronoi cells represent the spatial partitioning the input space. For simple and clear description, this space is based random points. Each cell is shaded based an isotropic Gaussian distribution centered at the cell's origin. The shading intensity reflects the density of accepting a sample based on the condition p(x:/v.) Gaussian values, and hence higher likelihoods of sample acceptance. Light sky blue areas indicate regions with lower density compared the darker regions. This visualization demonstrates the selective nature of our sampling method, focusing only on feasible solutions during optimization.
Darker regions indicate higher
H ARCHITECTURE DETAILS
Each autoregressive block 9tt includes several coupling layers g 1.,, The transformation of each layer operates as follows:
kel (z, 1,1+1 A(Z k.L ok.))
2
(21)
18

---

Published as a conference paper at ICLR 2025
Algorithm NF-BO
Input: black-box objective function f. SeqFlow model g, embedding set & (c1, . C
surrogate model f. acquisition function token-level importance w, oracle budget T, number of query points Ng. initial data D = {(x(*) _
1: for .... while the oracle budget remains do
2:
Dtr + CONCAT -N,,N, :]. (D))
3: Train g. & with LO Dtr
Eq. 14
4:
Train f on Der if else D
5:
(Xanc: Mane) - sample based on y values from D
6:
Zane g (x,x))
7:
Zeand - Draw Ng candidate points with TACS in trust region centered on Zanc
>> Eq. 18
8:
Select subset Z based alz; f), where Zand
9:
X
x/xx larg max, sim(v,, e (z), Z
10: D (x, [x
11: D CONCAT (D, Dnew)
end for
13: (x*, _,² .y*) + arg maX(xxx.))D
14: return x*
Figure 13: Sampling Distribution Visualization. Voronoi cells represent different regions, and color intensity indicates the likelihood of accepting a sample.
For each block g², the input represented by =0: zz, and the output of the final layer in each block sets the initial condition for the next block, - The final output after the last layer of the last block is K.L v. Each coupling layer further refines the data representation, informed by previous tokens. The function A, which we implemented as LSTM, aggregates information from prior tokens, enhancing the model's ability to capture long-range dependencies of sequence data.
I IMPLEMENTATION DETAILS
In our experiments, parameters were adjusted based on the specific requirements each benchmark setting. For (the batch size of trust regions, the number of query points N, per trust region), set these parameters to (5, 10) for the Guacamol benchmark with an additional oracle call setting of 500. For other settings, these parameters adjusted (10, 100).
19

---

Published as a conference paper at ICLR 2025
We explored the temperature T for the Token-level Adaptive Candidate Sampling (TACS) across the values { {400, 200, 100) to find the optimal setting. The sequence length L was determined based on the longest sequence in the initial dataset. For details on the other fixed parameters, please refer to Table 3
Table 3: Fixed parameters for all tasks and settings.
Parameter
Value
Scaling factor in TACS
0.1 Sequence length 0.1
Standard deviation of variational distribution !
# of topk data for training
1000
Coefficient of similarity loss Lsim
Typically, the anchor point within trust region is selected based on the current best observed value from the accumulated data. However, in our approach, we enhance exploration by sampling anchor points based on their objective values. We apply a softmax function to the objective values of the data points to determine their probabilities of being selected as anchor points. This probability defined as: p(x(1) = exp(x (r)) where y(() is the objective value point and r' is the Σ, exp(
temperature parameter set 0.1, facilitating a more explorative selection by emphasizing higher objective values. This method ensures that points with higher objective values are more likely to be selected, promoting diverse exploration of the solution space.
J BASELINES
In the Guacamol benchmark, we use the following LBO methods the baselines:
LSBO: searches the entire latent space without any modifications.
TuRBO-L (Eriksson et al. 2019): employs a trust region strategy, focusing the search on promising areas around the current best score.
W-LBO (Trpp et al. 2020 utilizes weighted retraining to better adapt the model based promising new data.
LOLBO (Maus et al., 2022): integrates joint training between the surrogate and generative models to optimize performance.
CoBO (Lee et al. 2023): uses Lipschitz regularization enhance the correlation between
the latent space and the objective function, aiming to improve the model's 's predictive align- ment with desired outcomes.
PG-LBO (Chen et al., 2024): applies pseudo-labeling techniques to predict labels of unla- beled data points, potentially uncovering valuable areas of the search space.
K ADDITIONAL EXPERIMENTAL RESULTS
Analysis of Flo for value discrepancy problem. We presented ablation study of our gen- erative model (SeqFlow) to demonstrate the impact of the value discrepancy problem. We compare NF models by applying different mapping functions: Eq. 8 (ours) and BiLSTM (TextFlow (Ziegler & Rush 2019). Both models utilize a same Normalizing Flow (NF) framework. However, TextFlow does not ensure the accurate reconstruction of the inputs since applies BiI.TTM to the mapping function. The optimization results are in Table Please note that we do not apply TACS solely to compare generative models. From the table, our SeqFlow model achieves better perfor- mance with fewer parameters compared to the baseline model. SeqFFlow and TextFlow use the same NF model, but TextFlow includes more components and therefore has more parameters. Although TextFlow has more parameters, our SeqFlow model resolves the value discrepancy problem, result- ing in higher optimization performance. This shows that addressing the value discrepancy problem important in effective Bayesian optimization.
20

---

Published as a conference paper at ICLR 2025
Table 4:
Optimization results according to different generative models. Each score represents the mean and standard deviation of independent runs.
Methods BO
SeqFlow (Ours) NE-BO w/o TACS Autoregressive NF Eq. (8, 9) O 31M
TextFlow ZZiglerr & Rush
2019
NF-BO w/o TACS
Base Model
Autoregressive NF BiLSTM X 54M
Mapping Functions
Complete Reconstruction Params
adip
0.778 + 0.016 0.372 + 0.012
0.716 + 0.017 0.347 + 0.010
med2
Measurements of the value diserepancy between actual and latents. To demonstrate that our FeqFlow effectively addresses the value discrepancy problem, we measure the ratio of instances where y # g. comparing the score of the input data y and the reconstructed data ÿ. We use top 1,000 data points from 10,000 initial data points across all Guacamol tasks. The experimental results are Table 3 From the table, our SeqHlow model accurately reconstructs every data point, unlike the TextFlow model, which indicates that our SeqFlow model is appropriate NF model to address the value discrepancy problem.
Table 5: Quantitive measurement of value discrepancy. We measure the ratio of y 9.
Model
SeqFlow
TextFlow (Ziegler & Rush 2019)
adip 0.000
0.548
med2
0.000
0.609
osmb
0.000
0,630
pdop
0.000
0.502
rano
0.000
0.814
zale 0.000
0.750
valt 0.000
0.001
Exploration abilities of TACS according to the number of initial points. To verify the explo- ration abilities and effectiveness of our TACS, conduct ablation study of TACS using initial data point and 10,000 initial data points in Table 6 Each experiment is repeated 5 times and we report the average and standard deviation of the results. From the table. NF-BO with TACS con- sistently shows better optimization results compared to NF-BO without TACS when using initial data point. Moreover, NF-BO with TACS is shown to robust to the number of initial points by comparing the optimization results between NF-BO w/ TACS (init 1) and NF-BO w/ TACS (init 10K). This suggests that TACS has strong exploration capabilities. Interestingly, some tasks like Adip, NF-BO w/ TACS (init 1) performs better than NF-BO w/ TACS (init 10K), which highlights the effectiveness of TACS under low-data scenarios. We maintained the TACS temperature at 400, consistent with our main experiments.
Choice TACS temperature. To choose the TACS temperature our main experiments, we initially conduct a simple search for the TACS temperature on one of the Guacamol tasks within the range [400, 200, 100). Based on this search, we fix the temperature 400 for all benchmarks and tasks.
To further demonstrate the robustness of the TACS temperature, we provide a sensitivity analysis in Table 7 This analysis is performed across seven Guacamol tasks, with results averaged over five runs per task and summed. Both the oracle budget and the number of initial data were set to 10,000. From the table, TACS temperatures above 200 consistently show better optimization results compared to not using TACS, highlighting the robustness of our approach to the choice of TACS temperature across all tasks.
21

---

Published as a conference paper at ICLR 2025
Table 6: Performance on low-data scenarios with initial data. Bold values indicate the highest performance among methods with only initial data point. Each score represents the mean and standard deviation of 5 independent runs.
Method
NE-BO w/o TACS (init 1) NF-BO w/ TACS (init 1)
NE-BO w/ TACS (init 10K)
adip
0.765 + 0.038
0.818 + 0.051
0.809 0.059
med2
0,306 + 0.014
0.307 + 0.027
0.380 - 0.014
osmb
0.848 + 0.037
0.855 + 0.007
0.897 0.016
pdop
0.564 + 0.045
0.623 + 0.043
0.759 0.023
0.846 + 0.019
0.848 + 0.021
0.941 0.006
valt
0.198 + 0.443
0.786 + 0.439
0.995 + 0,004
zale
0.586 + 0.013
0.589 + 0.033
0.760 0.012
Table Sensitivity analysis of TACS temperature on Guacamol tasks. Each task's performance is averaged over five trials.
TACS Temperature
Score Guacamol tasks
(w/o TACS)
5.495
2000
5,495 (+0.000)
1000
5.523 (+0.028)
400
5.544 (+0.049)
200
5.565 (+0.070)
100
5,481 (-0.014)
50
5,453 (-0.042)
20
5,418 (-0.077)
Sensitivity analysis for coefficient À to similarity loss. We conduct sensitivity analysis evalu- ate the performance across different À values. These experiments are carried out on seven Guacamol tasks, with the results in Table [ The table demonstrates that our NF-BO model maintains compet- itive performance across various À settings. Also, tasks such osmb and rano show robustness to variations in
Table 8: Sensitivity À on model performance. Each score represents the mean and standard deviation of independent runs.
Coefficient À
0.1
adip
0.783 t 0.024
0.809 + 0.059
0.771 + 0.023
med2
0.366 0.018
0.380 + 0.014
0.367 0.012
osmb
0.895 + 0.004
0.897 + 0.016
0.900 + 0.009
pdop
0.768 0.033
0.759 + 0.023
0.785 0.028
0.938 + 0.004
0.941 + 0.006
0.940 0.005
valt
0.989 + 0.006
0.995 + 0.004
0.975 0.031
zale
0.737 + 0.014
0.760 + 0.012
0.752 0.013
Illustration of the value discrepancy problem Guacamol tasks. We additionally include
illustrations similar to those presented in the main paper (see Figure] tasks, which are shown in Figure 14
1 for five additional Guacamol
L DISCUSSION WITH LAMBO AND LAMBO-2
In contrast to our model, where the decoder is formulated as problem statement's decoder x = (z) (Eq. 6), the decoders in methods like LaMBO (Stanton et al., 2022 and LaMBO-2 (Gru-) ver al. 2024 (e.g., MAE) operate differently since they utilize the original in the decoder to
22

---

Published as a conference paper at ICLR 2025
Objective Values of Med2 Task
Objective Values of Csmb Tosk
Objective Values of Pdop Task
Org nal k coda
Original top dita Aftor rcconstractin
20
Drigg nal top-4 date After reconntratioon
After reconstruct
40
15
Density
Dersity
50
20
0.25
Objective Values Task
Objective Values of Zale Task
elte reconstruction
reconistuction
Donaty
Density
1.5
0.5
0.2
0.4
0.6
0.0
0.1
0.2
0.5
0.4
0.5
Value
Value
Figure 14: Visualizations of the value discrepancy problem across 5 Guacamol tasks.
restore inputs for unmasked input tokens, thus cannot define the value discrepancy problem, which defined as Equation 7 Instead, these models elegantly align the latent space and input space by repeatedly decoding or encoding new candidates. One key difference between LaMBO, LaMBO-2 and ours that our method does not need additional encoding and decoding all candidates. Our method decodes only the latent vector chosen by the acquisition function thanks to the one-to-one function from the input space to the latent space and its left-inverse function.
M GLOSSARY OF NOTATION
Table 9: Glossary of notation.
Sequence of discrete data, each x, is token index from V, where N Continuous representation corresponding to discrete data where E L Number of tokens Dimension of the embedding space Embedding vector of the j-th token, where e, E R
X
V
L
sequence
F
ej
sim(.. (...)
Cosine similarity function used to measure the similarity between two vectors Function that returns the index of the most similar embedding vector to v; Conditional probability of the token index x; given the continuous vector v; Equals if x; is the index returned by a(v:) and 0 otherwise Optimal value the optimization
a(v.)
plx VV)
x
f
Objective function Objective value of input x
y
z
Latent vector of input
& 5, z
Next query data selected by the acquisition function
g
Flow transformation
g
Sequence of flow transformations Pointwise mutual information of x and z; Token-level sampling probability on the latent z
W.(C
(2)
K6
Constant scaling factor varied by sequence length
Temperature parameter for Token-level Adaptive Candidate Sampling (TACS)
23


Reviews:
[{"review_id": "cL27uj4yJ8", "reviewer": "Reviewer_2nEz", "summary": "This paper addresses the value discrepancy problem in latent Bayesian optimization, where improvements to the latent variable objective value can fail to translate to improvements to objective value of the actual decision variables after decoding. The authors propose to address this issue by parameterizing the generative model as an invertible mapping, specifically a normalizing flow.", "questions": "- The tradeoff of the restrictive parameterization of a normalizing flow is typically a loss of expressivity. I believe this is why normalizing flows have largely been abandoned by most of the generative modeling community. Are the authors at all concerned about this tradeoff?\n\n- Have used techniques like TACS for some time, I've noticed it has a weakness in the exploration context, namely that it will tend not to choose positions that have not been varied in the past, and is thus heavily reliant on a relatively complete characterization of the search space in the initial data package. Have you considered how you might \"cold start\" your solver when only a small local region of the input space is represented in the initial dataset?\n\n- Similarly the choice of temperature in the TACS procedure is a critical hyperparameter that was presumably tuned by the authors by looking at performance curves on the evaluation tasks. While this is common practice in the literature, it tends to cause the actual performance of these algorithms to be overestimated when turned to real consequential tasks where little or no tuning budget is available. Have the authors considered any heuristics for choosing this hyperparameter, perhaps in an adaptive way?\n\n- If you are interested in more potential test functions and baselines you may want to check out the [poli](https://github.com/MachineLearningLifeScience/poli) and [poli-baselines](https://github.com/MachineLearningLifeScience/poli-baselines) project. In particular you may be interested in Ehrlich test functions [5], which are available [standalone](https://github.com/prescient-design/holo-bench) or as part of the poli package. Ehrlich functions are inexpensive yet relatively difficult closed-form test functions for rapid prototype development and more extensive ablation experiments.\n\n\nReferences\n\n- [5] Stanton, S., Alberstein, R., Frey, N., Watkins, A., & Cho, K. (2024). Closed-Form Test Functions for Biophysical Sequence Optimization Algorithms. arXiv preprint arXiv:2407.00236.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors tackle an important problem in the latent BayesOpt literature, a problem which has been previously been observed since at least [1].\n\n- The TACS procedure for selecting token positions to vary is very sensible and closely resembles a similar technique employed by MaskGIT [2] and LaMBO-2 [3], albeit with some minor differences in what signal is normalized into a sampling distribution\n\nReferences\n\n- [1] Stanton, S., Maddox, W., Gruver, N., Maffettone, P., Delaney, E., Greenside, P., & Wilson, A. G. (2022, June). Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning (pp. 20459-20478). PMLR.\n\n- [2] Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022). Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11315-11325).\n\n- [3] Gruver, N., Stanton, S., Frey, N., Rudner, T. G., Hotzel, I., Lafrance-Vanasse, J., ... & Wilson, A. G. (2024). Protein design with guided discrete diffusion. Advances in neural information processing systems, 36.", "weaknesses": "This is a reasonable paper, if somewhat incremental and with an incomplete view of previous literature on the topic. For an example of a good literature review on this area, see [4].\n\nWhile more test functions and baseline experiments could always be added, I want to focus my feedback on the lack of a key experiment that verifies the authors are correctly attributing their improved performance to the elimination of value discrepancy. I find it to be a reasonable hypothesis, but the current comparisons to baselines do not prove the hypothesis to be correct because the experiment is confounded by all the other differing implementation choices in those baselines. This is not about proving that your program _as a whole_ outperforms other programs on the benchmarks, this is about critically testing your hypothesis that value discrepancy is the causal factor in this experiment. I would encourage you to think carefully about how you might design an ablation to evaluate this hypothesis. It could be as simple as making the smallest possible change to your current code to intentionally reintroduce the value discrepancy problem, while minimizing any other potential confounders in the experiment.\n\nIt is also a bit concerning that the value gap between actual and latents does not seem to have been systematically measured and reported, although that could simply be a choice of presentation. Unless I am mistaken the only place the value gap is quantified at all is in an illustrative figure in the intro.\n\nYou should also compare to the simplest possible solution to the value discrepancy problem which was employed by [1] and [3], which is simply decoding frequently and checking the actual objective value [1], and reinitializing the latent variables from actual improved solutions [3].\n\n\nReferences\n\n- [4] González-Duque, M., Michael, R., Bartels, S., Zainchkovskyy, Y., Hauberg, S., & Boomsma, W. (2024). A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences. arXiv preprint arXiv:2406.04739.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FET4gxAjRU", "reviewer": "Reviewer_ajp3", "summary": "The authors focus on latent-space Bayesian Optimization (LBO), which in recent years has emerged as an important area of research. The authors' main contribution is two-fold. (i) To remove the discrepancy between the encoder and decoder of the VAE models commonly used in LBO approaches, they learn a normalizing-flow model that preserves a one-to-one mapping and thus a loss-less mapping from observed to encoded space. (ii) an improved approach to sample new candidates from the latent space based on their relative importance. \nThe method is evaluated on a range of benchmarks with strong performance improvements upon the baselines.", "questions": "- Q1: The paper contains the ablation of running the method with and without TACS, showing a clear improvement when including it. Can the authors speculate (or provide if time permits) on the expected results of a second ablation that combines TACS with a VAE model instead of a flow? Would its relative performance improve to a similar extent as in the proposed approach, or more/less?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written and the proposal is well-motivated. \n- The evaluation is extensive and demonstrates a strong improvement\n- Each of the parts, the autoregressive flow as well as the sampler seem straightforward to implement and use for further applications", "weaknesses": "The weaknesses listed below are only minor.\n\n- Unless I am mistaken, the final acquisition function used is never specified in the text or appendix.\n- In l255 the relation to Ho et al. (2019) remains unclear in the text\n- The embedding e is never fully specified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXwum6bKK9", "reviewer": "Reviewer_xNso", "summary": "&nbsp;\n\nThe authors present the first architecture featuring a normalizing flow as the encoder-decoder model for latent space Bayesian optimization. The invertibility of the normalizing flow is important in addressing the value discrepancy problem, namely that encoded points are not guaranteed to be reconstructed perfectly by the decoder. This pathology in latent space Bayesian optimization has been identified in recent work [11] under the name of the \"misalignment problem\". It should be noted that [11], to the best of my knowledge is not yet publicly available, and hence the authors should not be required to consider this work. Furthermore, the authors introduce a token-level adaptive sampling scheme (TACS) and demonstrate that it improves performance of the NF-BO method in an ablation study. I think the method is innovative and the empirical results are impressive. However, I have major concerns about the reproducibility of the results, given that the codebase is not provided. I believe crucial details to reproduce the experiments are missing as detailed below. If these issues can be addressed during the rebuttal phase I will be more than happy to increase my score and recommend acceptance.\n\n&nbsp;", "questions": "&nbsp;\n\n1. What is the relationship between the similarity loss in Equation 13 and deep metric learning [9]? Do the authors think that incorporating label information y into the similarity loss could yield a better surrogate fit as in [9]? Did the authors consider incorporating label information into the loss formulation?\n\n2. For Equation 14, did the authors consider coordinate ascent on the two terms in the objective as opposed to scalarization with the lambda parameter? Coordinate ascent may be more efficient and would avoid setting an arbitrary scaling parameter.\n\n3. On line 331, the authors state, \"In practice, sampling the full posterior function from the GP posterior distribution is infeasible\". What do the authors mean by this and why is it relevant. Do the authors assume an acquisition function like Thompson Sampling is being used?\n\n4. In Appendix C, do the authors have any intuition for why lower temperature yields reduced diversity for the rano task?\n\n5. In Appendix F, how is the embedding dictionary specified?\n\n6. In Algorithm 1, what is topK on line 2?\n\n&nbsp;", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "&nbsp;\n\n1. The work is the first to introduce normalizing flows as an encoder-decoder model in latent space Bayesian optimization. This represents a novel departure to existing work.\n\n2. The authors methodology addresses an important pathology in latent space Bayesian optimization by enforcing invertibility on the encoder-decoder architecture.\n\n3. The authors demonstrate strong empirical results and obtain state-of-the-art performance on a widely-used benchmark (PMO) where they demonstrate that NF-BO outperforms a suite of other optimization approaches including RL, genetic algorithms, and other Bayesian optimization methods. As such, the current work is of very broad interest to the optimization community.\n\n&nbsp;", "weaknesses": "&nbsp;\n\n**MAJOR POINTS**\n\n&nbsp;\n\n1. My principal concerns relate to the reproducibility of the empirical results since the authors do not provide their code. An Anonymous GitHub link or similar could be provided during the rebuttal phase. In its current form, I don't believe the results are reproducible from the manuscript alone since crucial details such as the GP surrogate model (vanilla GP/sparse GP) and the choice of acquisition function (EI, PI, Thompson sampling) appear to be missing.\n\n2. I experienced difficulty navigating Section 4.2 as the notation is very confusing. On line 239, what is L, the number of tokens in a sequence? If so why is each token a vector? In the case of SMILES or SELFIES strings of molecules each token will be an alphabetic character which should not be a vector quantity? What is F? The dimension of the embedding space? What is Equation 8 implying? In other words what does choosing the j that maximizes sim(v_i, e_j) entail? The argmax variable is not clear as only the index is given. A similar point is applicable for the definition of a_i(v). How are the embeddings e computed? Perhaps a glossary of notation would be beneficial for clarity?\n\n&nbsp;\n\n**MINOR POINTS**\n\n&nbsp;\n\n1. In the abstract, with reference to the sentence, \"accurate and one-to-one mappings between latent and input spaces\". Is the term \"accurate\" redundant since a one-to-one mapping implies perfect reconstruction?\n\n2. When introducing Bayesian optimization, it may be worth citing the originating papers for the method [1, 2] as discussed in [3].\n\n3. In the first paragraph, some citations are not parenthetical \\citet in LateX vs. \\cite and should be parenthetical.\n\n4. The sentence on line 50, \"This makes the value discrepancy problem more and makes\nthe searching challenging.\" Should be revised.\n\n5. In the related work section on latent space Bayesian optimization, it would also be worth discussing the following papers [4-8] and their relationship to the current work e.g. [8] explicitly considers the diversity in the candidate solutions in a similar fashion to the Token-level Adaptive Candidate Sampling (TACS) employed by the authors.\n\n6. There are some missing capitalizations in the references e.g. \"Bayes\" instead of \"bayes\" in the citation for Kingma and Welling, 2014.\n\n7. On line 147, it would be worth changing the notation for the label y to be a scalar since f is defined as a scalar quantity.\n\n8. Missing full stop at the end of Equation 4.\n\n9. The decision to use the variable v in Equation 3 is slightly confusing. Why not just use x?\n\n10. The problem formulation of latent space Bayesian optimization in Equation 5, differs from that in e.g. [9] where the goal is to identify z^* that maximizes the expected f(p(z)) under a probabilistic decoder. I'm assuming that the formulation presented assumes a deterministic decoder since the normalizing flow is capable of producing a bijective mapping. It may hence be worth emphasizing the distinction between the problem formulation with normalizing flows (deterministic decoder) and the problem formulation with VAEs (probabilistic decoder).\n\n11. Typo line 200, y^(i) should be a scalar, similarly line 201.\n\n12. For the contrastive loss in Equation 13, it would be good to discuss the intent in relation to deep metric learning in [9]. The goal of the contrastive loss would appear to be to structure the latent space. As found in [9] it might be an idea to leverage label information y to structure the normalizing flow latent space. This could yield improvements to the GP fit.\n\n13. In Table 1, what is the score? The sum of the scores of each individual task constituting the PMO benchmark? It would be worth specifying this directly in the main paper in addition to the appendix (I only found this information later when I read the appendix.). Additionally, it would be worth mentioning that the full breakdown of results from the PMO benchmark [10] is provided in the appendix.\n\n14. Typo line 484, the number of the referenced table is missing.\n\n15. Typo line 485, \"metrics\".\n\n16. In Figures 5 and 7, the method for computing the errorbars should be reported e.g. is the standard error or the standard deviation over 5 trials being depicted? It would also be worth stating the number of trials directly in the figure captions.\n\n17. Typo line 893, \"number of query points\".\n\n&nbsp;\n\n**REFERENCES**\n\n&nbsp;\n\n[1] Kushner, HJ., [A Versatile Stochastic Model of a Function of Unknown and Time\nVarying Form](https://www.sciencedirect.com/science/article/pii/0022247X62900112). Journal of Mathematical Analysis and Applications 5(1):150–167. 1962.\n\n[2] Kushner HJ., [A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise](https://asmedigitalcollection.asme.org/fluidsengineering/article-abstract/86/1/97/392213/A-New-Method-of-Locating-the-Maximum-Point-of-an?redirectedFrom=fulltext). Journal of Basic Engineering 86(1):97–106. 1964.\n\n[3] Garnett, R., [Bayesian optimization](https://bayesoptbook.com/). Cambridge University Press. 2023.\n\n[4] Verma, Ekansh, Souradip Chakraborty, and Ryan-Rhys Griffiths. [High dimensional Bayesian optimization with invariance.](https://realworldml.github.io/files/cr/paper53.pdf) In ICML Workshop on Adaptive Experimental Design and Active Learning. 2022.\n\n[5] Stanton, S., Maddox, W., Gruver, N., Maffettone, P., Delaney, E., Greenside, P. and Wilson, A.G., 2022, June. [Accelerating Bayesian optimization for biological sequence design with denoising autoencoders.](https://proceedings.mlr.press/v162/stanton22a/stanton22a.pdf) In International Conference on Machine Learning (pp. 20459-20478). PMLR.\n\n[6] Notin, P., Hernández-Lobato, J.M. and Gal, Y., 2021. [Improving black-box optimization in VAE latent space using decoder uncertainty.](https://openreview.net/pdf?id=F7LYy9FnK2x) Advances in Neural Information Processing Systems, 34, pp.802-814.\n\n[7] Lu, X., Gonzalez, J., Dai, Z. and Lawrence, N.D., 2018, [Structured variationally auto-encoded optimization.](https://proceedings.mlr.press/v80/lu18c.html) In International Conference on Machine Learning (pp. 3267-3275). PMLR.\n\n[8] Maus, N., Wu, K., Eriksson, D. and Gardner, J., 2023, [Discovering Many Diverse Solutions with Bayesian Optimization.](https://proceedings.mlr.press/v206/maus23a/maus23a.pdf) In International Conference on Artificial Intelligence and Statistics (pp. 1779-1798). PMLR.\n\n[9] Grosnit, Antoine, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I. Cowen-Rivers, Lin Yang, Lin Zhu et al. [High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning.](https://arxiv.org/abs/2106.03609) arXiv preprint arXiv:2106.03609. 2021.\n\n[10] Gao, Wenhao, Tianfan Fu, Jimeng Sun, and Connor Coley. [Sample efficiency matters: a benchmark for practical molecular optimization.](https://proceedings.neurips.cc/paper_files/paper/2022/hash/8644353f7d307baaf29bc1e56fe8e0ec-Abstract-Datasets_and_Benchmarks.html) Advances in Neural Information Processing Systems 35, 21342-21357. 2022.\n\n[11] Chu et al. [Inversion-Based Latent Bayesian Optimization](https://nips.cc/virtual/2024/poster/95013), NeurIPS 2024.\n\n&nbsp;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ggDexBAup", "reviewer": "Reviewer_TFzn", "summary": "This paper introduces a Latent Bayesian Optimization (L-BO) method, called NF-BO, that aims to improve BO performances by avoiding the value discrepancy problem typical of existing L-BO methods.\nThis method uses a sequential normalizing flow-based model to ensure a 1 to 1 injective mapping between latents and input data points, therefore avoiding the value-discrepancy problem. The authors also introduce TACS, an adaptive sampling methods for candidate points to increase the efficiency of the search by better focusing on areas with higher potential impact.\nThe effectiveness of NF-BO is demonstrated on a wide range of molecule generation tasks.", "questions": "See weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Efficient BO methods are key to finding optimal solution to a wide range of complicated problems with expensive and black-box objective functions. Many of these problems have inherently discrete components, and as such it's key to develop methods like NF-BO that can effectively deal with discrete spaces in a principled manner.\n2. The authors give a good overview of the limitations of current methods, and the reason why normalizing flows are a better choice for this task.\n3. To the best of my knowledge, this method is novel in the usage of NF, the SeqFlow architecture and the TACS sampling. \n4. In the experimental benchmarks NF-BO  together with TACS sampling greatly outperforms competing methods, leading to much better solutions in a fraction of the oracle calls", "weaknesses": "1. Overall, I found some of the SeqFlow explanations in section 4.2 a bit too abstract to be easily understood by the reader. I think the section could benefit from using a simple running example to show in practice (a) how an embedding dictionary could look like (b) a sample surrogate model and acquisition function\n2. In the paper you claim that current L-BO methods fail because of the value discrepancy problem. Are you sure that the issues are due to the value discrepancy problem itself, and not for example due to a \"poor\" learned latent space? In other words, is \"perfect reconstruction\" a must, or \"good enough reconstruction\" would also work?\n3. How sensitive is the method to different values of $\\lambda$  in the similarity loss in (14)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)