Learning Representation for Bayesian Optimization with Collision-free Regularization
Authors: Fengxue Zhang
Brian Nord
Yuxin Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: Latent space
Bayesian Optimization
Collision

Abstract: Bayesian Optimization has been challenged by the large-scale and high-dimensional datasets, which are common in real-world scenarios. Recent works attempt to handle such input by applying neural networks ahead of the classical Gaussian process to learn a (low-dimensional) latent representation. We show that even with proper network design, such learned representation often leads to collision in the latent space: two points with significantly different observations collide in the learned latent space, leading to degraded optimization performance. To address this issue, we propose LOCo, an efficient deep Bayesian optimization framework which employs a novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to the objective value to be Lipschitz continuous. LOCo takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance. We provide a rigorous theoretical justification for LOCo by inspecting the regret of this dynamic-embedding-based Bayesian optimization algorithm, where the neural network is iteratively retrained with the regularizer. Our empirical results further demonstrate the effectiveness of LOCo on several synthetic and real-world benchmark Bayesian optimization tasks.

Under review as a conference paper at ICLR 2022
LEARNING REPRESENTATION FOR BAYESIAN OPTI- MIZATION WITH COLLISION-FREE REGULARIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Bayesian Optimization has been challenged by the large-scale and high- dimensional datasets, which common in real-world scenarios. Recent works attempt to handle such input by applying neural networks ahead of the classical Gaussian process to learn a (lowddimensiona) latent representation. We show that even with proper network design, such learned representation often leads to collision in the latent space: two points with significantly different observations collide in the learned latent space, leading to degraded optimization performance. To address this issue, we propose LOOCO, an efficient deep Bayesian optimiza- tion framework which employs novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to the ob- jective value to be Lipschitz continuous. LOCo takes in pairs of data points and penalizes those too close in the latent space compared to their target space dis- tance. We provide a rigorous theoretical justification for LOCo by inspecting the regret of this dynamic-embedding- bbaadd Bayesian optimization algorithm, where the neural network iteratively retrained with the regularizer. Our empirical re- sults further demonstrate the effectiveness of LOCO on several synthetic and real- world benchmark Bayesian optimization tasks.
INTRODUCTION
Bayesian optimization is classical sequential optimization method and is widely used in various fields in science and engineering, including recommender systems (Galuzzi et al., 2019), medical trials (Sui et al., 2018), 8), robotic controller optimization (Berkenkamp et al., 2016), scientific exper- imental design (Yang et al., 2019), and hyper-parameter tuning (Snock et al., 2012), among many others. Many of these applications involve evaluating an expensive blackbox function; therefore, the number of queries should be minimized. A common way to model the unknown function is via Gaussian processes (GPs) (Rasmussen and Williams, 2006), which have been extensively stud- ied under the bandit setting, as an effective surrogate model of the unknown objective function in a broad class of blackbox function optimization problems (Srinivas al., 2010; Djolonga al., 2013).
A key computational challenge for learning with GPs lies in optimizing specific kernels used for modeling the covariance structures. Such an optimization task depends on the dimension of the input space. For high-dimensional data, it is often prohibitive to train a GP model. Meanwhile, local kernel machines are known to suffer from the curse of dimensionality (Bengio et al., 2005), while the required number of training samples could grow exponentially with the dimensionality of the data. Therefore, dimensionality reduction and representation learning algorithms are needed to optimize the learning process.
Recently, Gaussian process optimization has been investigated in the context of latent space models. For example, deep kernel learning (Wilson et al., 2016) learns (ow-dimensiooaa) data represen- tation and a scalable kernel simultaneously via an end-to-end trainable deep neural network. In general, the neural network is trained to learn a simpler latent representation with reduced dimen- sion and has the structure information already embedded for the GP. Combining the representation leamned via neural network with GP could improve the scalability and extensibility of classical Bayesian optimization, but also poses new challenges for the optimization task, such as dealing with the tradeoff between representation learning and function optimization (Tripp al., 2020).

---

Under review as a conference paper at ICLR 2022
As we later demonstrate, critical challenge brought by introducing representation learning into Bayesian optimization is that the latent representation is prone to collisions: two points with signifi- cantly different observations can get too close, and therefore collide in the latent space. The collision effect in latent space models for Bayesian optimization is especially evident when information is lost during dimensionality reduction and/or when the training data is limited in size. As illustrated in Figure 1, when passed through the neural network, data points with drastically different observations are mapped to close positions in the latent space, Such collisions could be regarded as additional noise introduced by the neural network. Although Bayesian optimization is known to be robust to mild noisy observations (Bogunovic al., 2018), the collision in latent space could be harmful to the optimization performance, it is non-trivial to model the collision into the acquisition function explicitly. Also, the additional noise induced by the collision effect will further loosen the regret bound for classical Bayesian optimization algorithms (Srinivas et al., 2010).
Overview of main results To mitigate the collision effect, we propose novel regularization scheme that can be applied as a simple plugin amendment for the latent space based Bayesian optimization models. The proposed algorithm, namely Latent Space Optimization via Collision-frree regularization (LOOCo) leverages a regularized regression loss function to optimize the latent space for Bayesian optimization periodically.
reqularized Latent Space
150
Next
Current Sample
125
Current Moan
Obrsevations
Diserved oprinum
Ophmum
50
25
Concretely, our collision-free regularizer is encoded by a novel pairwise collision penalty function defined jointly on the latent space and the output domain. order to mitigate the risk collision in the latent space (and consequently boost the opti- mization performance). LOCO applies the regularizer to mini- mize the collisions uniformly in the latent space.
0.2 Latert
The collision effect in latent space-based BO tasks when having 100 data points and observa- tions. Because the data points around the optimum severely collided, BO is misguided to the sub-optimum. Dif- ferent from noise which normally as- sumed to be Gaussian, the collision could be much more divergent. See extended discussion in Appendix B.
Figure 1:
We further note that for Bayesian optimization tasks, collisions in regions close to the optimum more likely mislead the optimization algorithm. Based on this insight, we propose an optimization-awarn regularization scheme that assigns higher weight to the collision penalty on those pairs of points closer to the optimum region in the latent space. This algorithm. which we refer to as Dynamically- Weighted LOCo, is designed to dy- namically assess the importance of a collision during optimiza- tion. Compared with the uniform collision penalty in the latent space, the dynamic weighting mech- anism has demonstrated drastic improvement over the state-of-the-art latent space based Bayesian optimization models.
We summarize our key contributions as follows:
I. We investigate latent space based Bayesian optimization, and expose the limitations of existing latent space optimization approaches due to the collision effect on the latent space (Section 3). II. We propose a novel regularization scheme as simple plugin amendment for latent space based Bayesian optimization models. Our regularizer penalizes collisions in the latent space and effectively reduces the collision effect. Furthermore, dynamic weighting mechanism for adjusting the collision penalty to improve the effectiveness of regularization for Bayesian optimization (Section 4).
propose an optimization-awan
III. We provide theoretical analysis for the performance of Bayesian optimization on regularized latent space (Section 5).
IV. We conducted extensive empirical study several synthetic and real-world datasets, in- cluding real-world case study for cosmic experimental design, and demonstrate the promis- ing empirical performance for our algorithm (Section 6).
2 RELATED WORK
This section provides short survey recent work in Bayesian learning, which was designed to overcome the high-dimensionallty challenge for Gaussian process regression tasks and Bayesian optimization.
2

---

Under review as a conference paper at ICLR 2022
Different surrogate models with internal latent space Some alternative surrogate models have been proposed to replace classical GP in Bayesian optimization to overcome the challenge of high- dimensional and highly-structured input in BO. Deep Nenwork for Global Optimization (DNGO) Snoek et al. (2015) uses a pre-trained deep neural network with a Bayesian linear regressor at the last hidden layer of the network the surrogate model. More generally, Deep Kernel Learning (DKL) combines the power of the Gaussian process and neural network by introducing a deep neural network g to learn a mapping 9 X' + Z from the input domain X' latent space Z (Wilson et al., 2016). It uses the latent representation & E Z as the input of the base Gaussian process. The neural network 9 and a spectral mixture-based kernel form a scalable expressive closed-form deep covariance kernel, denoted by kn (2i, + (({()))..({;)) The deep kernel allows end-to-end learning and Bayesian optimization on the original input space.
Representation learning and latent space optimization Instead of reducing the dimensionality and performing optimization in an end-to-end process, other methods aim to optimize in a related low-dimensional space first and then map the solution back to the original input space. Djolonga et al. (2013) assume that only a subset of input dimensions varies, and the kernel is smooth (i.e. with bounded RKHS norm). Under these assumptions, the underlying subspace is learned via low-rank matrix recovery. Random Rentom.eatuuee feature is another solution under this setting (Rahimi et al., 2007; Letham al., 2020; Binois et al., 2015; Nayebi et al., 2019; Wang et al., 2016). is known that a random representation space of sufficiently large dimension is guaranteed to contain the optima with high probabilityy Mutný and Krause (2019) consider Quadrature Fourier Features (QFF)- as opposed to Random Fourier Feature (RFF) in Rahimi et al. (2007)-1o overcome the variance starvation prob- lem, and proved that Thompson sampling and GP-UCB achieve no-regret with squared exponential kernel in optimization tasks. However, both RFF and QFF methods rely on key assumption that the function to be optimized has a low effective dimension. In contrast, as discussed in Section 6 and the supplemental materials, we show that LOCo performs well for challenging high-dimensional BO problems where algorithms relying on the low effective dimension assumption may fail.
Another line of work on latent space optimization uses autonncoeers to learn din-dimesiional rep- resentations of the inputs to improve the scalability and capability to leverage the structural infor- mation (Mathieu al., 2019), (Ding et al., 2020), (Gómez-Bommarelli et al., 2018; Huang et al., 2015; Tripp et al., 2020; Lu et al., 2018). Mathieu et al. (2019), Ding et al. (2020) focus on disentan- gled representation learning that breaks down, or disentangles, each feature into narrowly defined variables and encodes them as separate dimensions. Tripp et al. (2020) iteratively train the autoen- coder with a dynamic weighting scheme when performing optimization to improve the embedding. Griffiths and Hernández-Lobato (2020) and Letham et al. (2020) enforce certain properties on the representation space to improve the optimization performance. To the best of the authors' knowl- edge, collision of the embeddings has not been explicitly studied. Binois et al. (2015) propose warped kernel to guarantee the injectivity the random linear embedding, which is not applicable in neural network- based methods.
A common challenge in applying these techniques to generic optimization tasks lies in the assump- tion on the accessibility of training data: Bayesian optimization often assumes limited access to labeled data, while surrogate models built on deep neural networks often rely on abundant access to data for pretraining. Another problem lies in the training objective: During the training phase, these surrogate models typically focus on improving the regression performance, and do not explicitly address the artifact caused by collisions of the learned embeddings, which-as as we later demonstrate in Section 3.3 could be harmful to sequential decision-making tasks.
3 PROBLEM STATEMENT
In this section, we introduce necessary notations and formally state the problem. We focus on the problem of sequentially optimizing a function f X + TR, where C Rd is the input domain. At iteration i, we pick a point Ja E X, and observe the function value perturbed by additive noise: yt = f(x)) + Et with & ~ (O0 o²) being i.i.d. Gaussian noise. Our goal is to maximize the sum of rewards ET, (1t) over T iterations, or equivalently, to minimize the cumulative regret Rr: ΣT rt, where T max maxf(x) f(x) - f(x) denotes the instantaneoos regret of Zt.
ZEX
3

---

Under review as a conference paper at ICLR 2022
3.1 BAYESIAN OPTIMIZATION
Formally, we assume that the underlying function f is drawn from a Gaussian process (P(m(zz),(aa ((xx)²)), where m(x) the mean function and k(xxz²) is the covariance function. and the corresponding noisy evalua- tions Yt = W....., yel yel", the posterior over f also takes the form of GP, with mean (y(x) (x)) (K, + + - Yr, covariance k (z, 2') = k(z, xz) ks (x) (Ke + o² "2')"" and vari- ance of (z) = k,,x, 2), where k,(z) ( ()........... ,xx, and K, is the positive definite kernel matrix [k<ar, x. E,,, (Rasmussen and Williams, 2005). After obtaining the posterior, one can compute the acquisition function a : X' - IH, which used to select the next point to be evaluated. Various acquisition functions have been proposed in the literature, including popular choices such as Upper Confidence Bound (UCB) (Srinivas et al., 2010) and Thompson sampling (TS) (Thompson, 1933).
At iteration 6, given the selected points At = {æ T
Remark. Regret is commonty used as performance metric for BO methods. In this work we focus on the simple regret ri max (x) max (xx) and cumulative regret R(T) =Etm Tt.
-T
=EX'
tST
3.2 LATENT SPACE OPTIMIZATION
Recently, Latent Space Optimization (LSO) has been proposed solve Bayesian optimization prob- lems on complex input domains (Gómez-Bombarelli et al., 2018; Huang et al., 2015; Tripp et al., 2020; Lu et al., 2018). LSO learns latent space mapping 9 X + 2 to convert the input space X' to the latent space Z. Then, it constructs an objective mapping h z + R such that (z) h(g(z)), V2 E Z. In this paper, we model the latent space mapping 9 as neural net- work; the neural network and the base kernel & together are regarded as deep kernel. denote by kmn(r, x²) (g(()), gaa")) (Wilson et al., 2016). In this context, the actual input space for BO is the latent space 2 and the objective function is h. With the acquisition function Ona (1) == a(g(z)), we do not compute an inverse mapping g as opposed to the aforementioned autoencoder-based
LSO algorithms (e.g. Tripp al. (2020)), since BO directly select 2t = arg max a'n (mn()) (x) Vi: < T. In
zEX'
our analysis, use squared exponential kernel, i.e. ksr(1,, x²) o exp (m-n'
3.3 THE COLLISION EFFECT OF LSO
When the mapping g : X + Z is represented by a neural network, it may cause undesirable collisions between different input points in the latent space Z. Under the noise-free setting, we say there exists a collision in 2, if Ex; 23 E X, such that when gxx;) = ((z,). lf;;; - (x;)) > O. Such collision could be regarded the neural network g. Given a representation function y. noisy observations
additional (unknown) noise on the observations introduced by f(x)++ +e, we say that there exists collision, if for X > O, there exist T;, 3y E X, such that (g(a.) 9(5;) < Alyz øزl-
When the distance between a pair of points (x., Zz)) in the latent space is too close compared to their difference in the output space, the different output values yt, 3/ for the collided points in the latent space could be interpreted as the effect of additional observation noise for g(z;) (or g(zz)). In general, collisions could degrade the performance of LSO. Since the collision effect is a priori unknown, it is often challenging to be dealt with in LSO, even if we regard it as additional obser- vation noise and increase the (default) noise variance in the Gaussian process. Thus, it is necessary to mitigate the collision effect by directly restraining it in the representation learning phase. One potential method to avoid collision could be tuning the design of neural networks. However, we empirically show that increasing the network complexity often does not help reduce the collision. The study is posed in Appendix B.2.
We consider low-noise setting where the collision can play a more significant role in degrading the optimization performance. As is shown in Figure that the collision could result in larger difficulty in the optimization task. And when collision exists, it is hard to distinguish it from the observation. Therefore we focused on treating the collision when defining the penalty instead of dealing with the stochasticity.
4

---

Under review as a conference paper at ICLR 2022
4
LATENT SPACE OPTIMIZATION VIA COLLISION-FREE REGULARIZATION
We now introduce LOCo as novel algorithmic framework to mitigate the collision effect.
4.1 OVERVIEW OF THE LOCo ALGORITHM
The major challenge in restraining collisions in the latent space is thak-unlike the formulation of the classical regression loss-we cannot quantify it based on single training example. We can, however, quantify collisions by grouping pairs of data points and inspecting their corresponding observations.
We define the collision penalty based pairs of inputs and further introduce pair loss function to characterize the collision effect. Based this pair loss, we propose a novel regularized latent space optimization algorithm', as summarized in Algorithm 1. The proposed algorithm concurrently feeds the pair-wise input into the same net- work and calculates the pair loss function. We demon- strate this process in Figure 2.
space
coliation penalty
pair
squaro Ioos
Given a of labeled data points, we can train the neu- ral network to create an initial latent space representation similar to DKL (Wilson et al., 2016)². Once provided with the initial representation, we can then refine the la- tent space by running LOCo and periodically update latent space (i.e. updating the learned representation after collecting batch of data points) to mitigate the collision effect as we gather more labels.
Figure 2: Schematic LOCO
Algorithm Latent Space Optimization via Collision-free Regularization (LOCo)
1: Input: Penalty parameter À (cf. Equation 1), regularization weight (cf. Equation 3), impor- tance weight parameter (cf. Equation 2), neural network g. parameters (0n,3, gg,)) total time steps T;
2: for 1 to T do
3:
<- arg max (gg( 0g,z))
> maximize acquisition funniittionnfunction
4: Yt <- evaluation on It
D update observation
5: 0 retrain 0, with the pair loss function (pp.A.c.9 Dz) in Equation 3
6: Output: max
4.2 COLLISION PENALTY
This subsection aims to quantify the collision effect based on the definition proposed in Section 3.3. As illustrated in Figure 2, we feed pairs of data points into the neural network and obtain their latent space representations. Apart from maximizing the GP's likelihood, we concurrentyy calculate the amount of collision on each pair and incur a penalty when the value positive. For Is; 2y fx;) + yj f(r, + are the corresponding observations, and 2; g(aa),, 2j 9(T,) are the corresponding latent space representations. We define the collision penalty as
Pij max (Aly, シy - 12, 2)1.0)
where À is penalty parameter that controls the smoothness of the target function h: - As rule of thumb, can estimate À by sampling from the original data distribution P(X,Y) Y) X x R. and choose the to be the maximum value such that Σ max (Alg) (i.c. to provide an upper bound for by keeping the total collision in the input domain to be zero).
2 =
Note that we have introduced several hyper-paramette in the algorithm design; we will defer discus- sion on choice these parameters Section
2To obtain an initial emhedding in the latent space, the process does not require the labels to he exact and allows the labels be collected from related task of cheaper cost.
5

---

Under review as a conference paper at ICLR 2022
4.3 IMPORTANCE- WEIGHTED COLLISION-FRED REGULARIZER
Note that it is challenging to universally reduce the collisions by minimizing the collision penalty and the GP's regression loss-this is particularly the case with limited amount of training data. Fortunately, for optimization tasks, it is often unnecessary to learn fine-grained representation for suboptimal regions. Therefore, we can dedicate more training resources improve the learned latent space pertaining to the potentially near-optimal regions, Following this insight, we propose to use a weighted collision penalty function, which uses the objective values for each pair as an importance weight in each iteration. Formally, for any pair ((x) 2j, Mý), (Ii, Zi, V:)) in a batch of observation pairs D, = [((mm Zm, ym), (Ins 2ns Bn)) ( m. where Iny Jm E A, and yn, ym E Yr. we define the impeghence-weiighted penalty function
e((++()))
Pig = Piy
with
Wij
(2)
Σ
(m.n)DD
Here the importance weight ( is used to control the aggressiveness of the weighting strategy.
Combining the kernel learning -egettvve log likelihood and the collision penalty for GP, we define the puir loss function Lpp.1..9
(p,, D.) = log(P()((A,,))) ||
P
Σ Pij
(3)
iED..EDD.
Here, |o,, 0.)) = -b (K, - + o2 log(2T) is the learning objective for the GP (Rasmussen and Williams, 2005). P denotes the regularization weight; demonstrate in Section 6, we initialize the regularization weight p to keep the penalty at the same order of magnitude as the negative log likelihood. An alternative training process is minimize the regression loss and the collision penalty alternatively. We observe in our empirical study that both training processes could lead to reasonable convergence behavior of the LOCO training loss.
DISCUSSION AND THEORETICAL INSIGHT
This subsection discusses the theoretical insight underlying the collision-free regularizer, by inspect- ing the effect of regularization on the regret bound of LOCo where the constantly trained neural network feeds a dynamic embedding to the Gaussian process.
While the key idea for bounding the regret of UCB-based GP bandit optimization algorithms follows the analysis of Srinivas al. (2010), two unique challenges are posed in the analysis of LOCo. Firstly, unlike previous work in Srinivas et al. (2010), the neural network is constantly retrained along with the new observations. Thus, the input space for the downstream Gaussian process could be highly variant.
For the discussion below, consider stationary and monotonic kernel, and assume that retraining the neural network g does not decrease the distance between data points in the latent space. It is worth noting that, although not strictly enforced, such monotonicity behavior is naturally encour- aged by our proposed regularization, which only penalizes the pair of too-close data in the latent space. Under the above assumption, the internal complexity of neural network training still makes it challenging to bound the regret w.r.t the dynamics of the neural network. Thus, we investigate the dynamics of the mutual information term in the regret bound, and justify the proposed collision- free regularizer by showing that penalizing the collisions tends to reduces the upper bound on the regret.
We first consider discrete decision set and then leverage the desired Lipschitz continuity on the regularized space to extend our results to the continuous setting (cf full proofs in Appendix A).
Proposition 1. Let Z be a finite discrete set. Let E (0, 1). and define 3r = 2 log (2)(2/66) Suppose that the objective function h: Z x 0 + R defined on Z and parameterized by 0 is sam- ple from GP. Furthermore, consider a stationary and monotonic kernel, and assume that retraining the neural network g does decrease the distance between data points in the latent space. Run- ning GP-UCB with Se for a sample h of a GP with mean function zero and stationary covariance function k((z, x²), we obtain a regret bound of (<log( ( I(h(2r, On,o); ΦT)) with high probability.

---

Under review as a conference paper at ICLR 2022
More specifically, with C1 8/ 1/1og(11 0²2), we have
IP Rr S C (1(22r On.)i br)))
1 - 8.
Here "YT is the maximun information gain after T iterations, and ΦT as the identification of the collided data points Z. YT is defined "r max (UA,, O(),0r)) AC2,1A/=T
The collision regularization reduced the maximum mutual information by specific term dependent on the distribution of the noise caused by the collision of data points. The distribution is dynamic and determined by the complex learning process of the neural network. In the following, we show that the mutual information bounded within a given interval:
Assume d, is random variable that identify Zt E 2, V/ y from its collided points, and the variance the collision is o then we have
S I (h(22, On.u): oT) S 1/2 [
This means that if oe is a random variable sampled from a Gaussian distribution defined on h, then II (h(2T, 0r,00; Φr) maximized.
The regularization also constrains the function / to be Lipschitz-contimuous with a Lipschitz con- stant, enabling a slightly narrower regret bound.
Proposition 2. Let Z C r0, be compact and convex, d N,r 0 and X 2 0. Sup- pose that the objective function h Z x + R defined on Z and parameterized by 0 sample from GP and is Lipschitz continuous with Lipschitz constant x. Let 5 E 10, and define Bt 2 logg( (/66) /60) + 2dgogdrdd()) (Ardi²). Furthermore, consider stationary and mono- tonic kernel, and assume that retraining the neural network does not decrease the distance between data points in the latent space. Running GP-UCB with 3, for sample h of GP with mean function zero and stationary covariance function k(x, x'), we obtain a regret bound of O" (VdT(rr - (h(zr, 0n.0); Φr)) with high probability.
More specifically, with C1 = 8/ log(1 + o-²), we have
Rr / CTT - (h(2r, 0n,0); (r)) + + 2 -
Here "T is the maximum information gain after T iterations, and âT as the identification of the collided data points on Z. .r is defined as "r == max (y ΦT; ((,,0r)).
AC2.
EXPERIMENTS
In this section, we empirically evaluate algorithm several synthetic and real-world bench- mark blackbox function optimization tasks. All experiments are conducted on Google Cloud GPU instance (4 vCPUs, 15 GB memory, Tesla T4 GPU) and Google CoLab high-RAM GPU instance.
6.1 EXPERIMENTAL SETUP
We consider five baselines in our experiments. Three popular optimization algorithss-partice swarm optimization (PSO) (Miranda, 2018), Tree-structured Parzen Estimator Approach (TPE) (Bergstra et al., 2011 1), a BoToorch (Balandat et al., 2020) implementation of Trust Region Bayesian Optimization (TuRBO) (Friksson et al., 2019), and standard Bayesian optimization (BO) (Nogueira, 2014) which uses Gaussian processes the statistical model-are tuned in each task. Another base- line we consider is the sample-efficient LSO (SE LSO) algorithm, which is implemented based on the algorithm proposed by Tripp al. (2020). We also compare the non-regularized latent space optimization (LSO), LOCo with uniform weights (i.e. synamically-weighee LOCo (i.e. with Ç > 0, referred to as DW LOCo) proposed in this paper.
O, referred to as LOCo), and the
One crucial problem in practice is tuning the tuned for periodically retraining in the optimization process by minimizing the loss function
The hyper- papeerrrrrmeterss for GP are
7

---

Under review as a conference paper at ICLR 2022
Max Area
HDBO
Rastrigin 2D
PSO TUARO
Z50
PSO TuARO
80
Simple regret
LSO
Simple regret
LS0
Simple regret
TFC
LOKo
WUCo
LS0
LOCo
DW LOCO
iterations
Iterations
Iterations
(a) Max Area
(b) Sum 200D
(c) Rastrigin 2D
Water Converter
Supernova
SPOKES
27000
80
10-
80
80
250000
PSD
Simple regret
230000
TPE
Simple regret
TPP
TuABO
Simple regret
TPE
TUABO
X2000
LSO
LS0
LSO
LSO
2500
LOCo
32
LOCo
LOCo
DW LOCo
DW LOCo
LOCo
2000
0030
13-
Iterations
Iterations
tterations
(d) Water Converter
(c) Supernova
(f) SPOKES
Figure 3: Experiment results on six pre-collected datasets. Each experiment repeated at least eight times. The colored area around the mean curve denotes the vn Here ā denotes the empirical standard deviation. denotes the number of cases repeated in experiments. Note that the BO and TPE implementation we tested can not finish in reasonable time for figure 3a and figure 3b.
validation set. For all tasks, we choose simple neural network architecture due the reasoning in section 3.3, as well as due to limited and expensive access to labeled data under the BO setting. The coefficient p is, in general, selected to guarantee similar order for the collision penalty to GP loss. The A should be tolerant of the additive noise in the evaluation. In practice, we choose the simple setting
1 and find it perform well. We also include study of the parameter choice in the appendix. controls the aggressiveness of the importance weight. While Ç should not be too close to zero (equivalent uniform weight), an extremely high value could make the regularization overly biased. Such severe bias could allow heavily collided representation in most of the latent space and degrade regularization effectiveness. The value choice is similar to the inverse of the temperature parameter of softmax in deep learning (Hinton al., 2015). Here use Ç for simplicity and find it robust to different tasks. All experiments are conducted on the pre-collected datasets. We defer the detailed experimental setup to Appendix C.
6.2 DATASETS AND RESULTS
We now evaluate LOCo on three synthetic datasets and three real-world datasets. We demon- strated the improvement in LOCo that is enabled by the explicit collision mitigation in the lower- dimensional latent space in terms of average simple regret.
2D Shape Area Maximization The dSprites dataset (Matthey et al., 2017) consists of images of size 64 x containing 2d Shapes with different scales, rotations, and positions. Each pixel value of the images are binary, hence E {0, 1] 166161 The goal is to generate a shape x with maximum area, which is equivalent to finding arg max Σ. 6ix61 x; where i corresponds to the pixel index. The
neural network is pretrained on 50 data points. To meet the limitation of memory on our computing instance, uniformly sample 10000 points from the original dataset and approximately maintain the original distribution of the objective value. The DW LOCo and LOCo outperform or matches the baseline methods this dataset.
8

---

Under review as a conference paper at ICLR 2022
Sum 200D We create synthetic dataset Sum 200D of 200 dimensions. Each dimension is inde- pendently sampled from a standard normal distribution to maximize the uncertainty on that dimen- sions and examine the algorithm's 's capability to solve the geedium-dimensione maximize the label f(x) _i=1 ext which bears an additive structure and of non-linearity. The neural network is pretrained on 100 data points. As illustrated by figure 3b DW LOCO and LOCo could significantly outperform baselines that do not specifically leverage the additive structures of the problem.
problem. We want
Rastrigin-2D The Rastrigin function is a non-convex function used as a performance test problem for optimization algorithms. It was first proposed by Rastrigin (1974) and used popular bench- mark dataset for evaluating Gaussian process regression algorithms (Cully et al., 2018). Formally, the 2D Rastrigin function is [(x) = 10d + Ei= x 10 cos((2t:;)) d = 2. For convenience of comparison, we take the -x()) as the objective value make the optimization tasks maximization task.
Supernova-3D Our first real-world task is to perform maximum likelihood inference on three cosmological parameters, the Hubble constant Hn E (60, 80), the dark matter fraction Ωt (0, 1), and the dark energy fraction 22 E (0, 1). The likelihood is given by the Kobertson-Walker metric, which requires one-dimensional numerical integration for each point in the dataset from Davis et al. (2007). The neural network is pretrained on 100 data points. As illustrated by figure 3e, both LOCO and DW LOCo demonstrate its consistent robustness. Among them, DW LOCo slightly outperform LOCo the early stage.
Water Converter Configuration-161 This UCI dataset consists of positions and absorbed power outputs of wave energy converters (WECs) from the southern coast of Sydney. The applied converter model is fully submerged threc-tether converter called CETO. 16 WECs locations are placed and optimized in a size-constrained environment.
Redshift Distribution-14) Careful accounting of all the requirements and features of these ex- periments becomes increasingly necessary to achieve the goals of a given cosmic survey. SPOKES (SPectrOscopic KEn Simulation) is an end-to-end framework that can simulate all the operations and critical decisions of cosmic survey (Nord et al., 2016). In this work, galaxies within a specified window distances from Earth. We then minimize the Hausdorff dis- tance between the desired redshift distribution and the simulation of specific cosmological surveys generated by SPOKES. In our experiments, the neural network is pretrained with 400 data points. As illustrated by figure 3f, the simple regret of DW LOCO drops slower yet eventually outperforms or matches other baselines' performances.
SPOKES to generate
In general, our experimental results consistently demonstrate the robustness of methods against collisions in the learned latent space. Our method outperforms or matches the performance of the best baselines all scenarios. When compared the sample-efficient LSO, DW LOCO performs better in most cases and shows steady capability to reach the optimum by explicitly mitigating the collision the latent space. Due the dynamics of representation learning process, is difficult to claim that the performance improvement brought by dynamic weighting is universal. This aligned with the observation in the experiments that DW LOCO brings observable improvement the regret curve at certain stage for an optimization task and achieve an ultimate performance that at least matches LOCo, In contrast, the sample-efficient LSO might fail due to the collision problem.
7 CONCLUSION
We have proposed novel regularization scheme for latent-space-basee Bayesian optimization. Our namely LOCo-addressa addresses the collision problem induced by dimensionality reduction and improves the performance for latent space-based optimization algorithms. We show that the regularization effectively mitigates the collision problem in the learned latent spaces and, therefore, can boost the performance of the Bayesian optimization in the latent space. We demonstrate solid empirical results for LOCo on several synthetic and real-world datasets. Furthermore, demon- strate that LOCo can deal with high-dimensional input that could be highly valuable for real-world experiment design tasks such cosmological survey scheduling.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL http:// fetp://arxivv http://axxxvv...aas..s.ss /1900/9900..0 066033.
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of dimensionality for local kernel machines. Techn. Rep, 1258:12, 2005.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parament optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, edi- tors, Advances in Neural Information Processing Systems, volume 24, pages 2546-2554, Curran Associates, Inc., 2011.
Felix Berkenkamp, Angela Schoellig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. 2016.
Mickaël Binois, David Ginsbourger, and Olivier Roustant. A warped kernel improving robustness in bayesian optimization via random embeddings. In International Conference on Learning and Intelligent Optimization, pages 281-286. Springer, 2015.
Ilija Bogunovic, Jonathan Scarlett, Stefanie Jegelka, and Volkan Cevher. Adversarially robust opti- mization with gaussian processes. In NeurIPS, 2018.
A. Cully, K. Chatzilygeroudis, F. Allocati, and J.-B. Mouret. Limbo: A Flexible High perthheerfrmmeeee Library for Gaussian Processes modeling and Data- Efficient Optimization. The Journal of Open Source Software, 3(26):545, 2018. doi: 10.21 25/joss.00545.
T. M. Davis. E. Mortsell, J. Sollerman, A. C. Becker, Blondin. Challis, A. Clocchiatti, A. V. Fil- ippenko, R. J. Foley, P. M. Garnavich, S. Jha, K. Krisciunas, R. P. Kirshner, B. Leibundgut, W. Li, T. Matheson, G. Miknaitis, G. Pignata, A. Rest, A. G. Riess, P. Schmidt, R. C. Smith, Spy- romilio, C. W. Stubbs, N. B. Suntzeff, J. I.. Tonry, W. M. Wood- Vasey, and A. Zenteno. Scruti- nizing exotic cosmological models using ESSENCE supernova data combined with other cosmo- logical probes. The Astrophysical Journal, 666(2):716-72) sep 2007, doi: 00.106/511988.
Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang. Max Welling, and Zhuowen Tu. Guided variational autoencoder for disentanglement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7920-7929, 2020.
Josip Djolonga, Andreas Krause, and V. Cevher. High- nigh-dimmessioal gaussian process bandits. In Neural Information Processing Systems, 2013.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in Neural Information Processing Systems, 09144566777,,, 2019.
Bruno Giovanni Galuzzi, Ilaria Giordani, Antonio Candelieri, Riccardo Perego, and Francesco Archetti. Bayesian optimization for recommender system. In World Congress Global Op- fimization, pages 751-760. Springer, 2019.
Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson Gpytoreh: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
Rafael Gómez-Bombarelll, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato. Benjamín Sánchez-Lengelingg Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru- Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268-276) 2018 88.
Timothy D Hirzel,
Kyan-Rhys Griffiths and José Miguel Hernández-Lobato Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical science, 11(2):577-586, 2020.
10

---

Under review as a conference paper at ICLR 2022
Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015.
Wenbing Huang, Deli Zhao, Fuchun Sun, Huaping Liu, and Edward Chang. Scalable gaussian process regression using deep neural networks. In Proceedings of the 24th International Con- ference on Artificial Intelligence, IJCAP15, pages 3576 3582. AAAI Press, 2015. ISBN
9781577357384.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14Xi 6980, 2014.
Ben Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-examining linear embeddings for high-dimennional bayesian optimization. Advances in Neural Information Processing Systems, 33, 2020.
Xiaoyu Lu, Javier Gonzalez, Zhenwen Dai, and Neil Lawrence. Structured variationally auto- encoded optimization. volume 80 Proceedings of Machine Learning Research, pages 3267- 3275, Stockholm International Fairs, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In International Conference on Machine Learning, pages 4402-440444 4412. PMLR, 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http:///tttbb.oommdeeeeemnn///ts
2017.
Lester James V. Miranda. PySwarms, research-toolks for Particle Swarm Optimization in Python. Journal of Open Source Software, 3, 2018. doi: 10.21 105/joss.00433..
Mojmír Mutný and Andreas Krause. Efficient high dimensional bayesian optimization with addi- tivity and quadrature fourier features. Advances in Neural Information Processing Systems 31, pages 9005-9016 2019.
Amin Nayebi, Alexander Munteanu, and Matthias Poloczek. A framework for bayesian optimization in embedded subspaces. International Conference on Machine Learning, pages 4752-4761. PMLR, 2019,
Fernando Nogucira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014.
B. Nord, A. Amara, A. Réfrégier, La. Gamper, Lu. Gamper, B. Hambrecht, C. Chang, J.E. Forero- Romero, S. Serrano. C. Cunha, O. Coles, Nicola, M. Busha, A. Bauer, W. Saunders, S. Jouvel, D. Kirk, and R. Wechsler. Spokes: An end-to-end simulation facility for spectroscopic cosmolog- ical surveys. Astronomy and Computing, 15:1 15, 2016. ISSN 2213-1337.
Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In Neural Information Processing Systems, volume 3, page 5. Citeseer, 2007.
C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Leonard Andreevič Rastrigin. Systems of extremal control. Nauka, 1974.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In 26th Annual Conference on Neural Information Processing Systems 2012, pages 2951-2959, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pages 2171-2180. PMLR, 2015.
11

---

Under review as a conference paper at ICLR 2022
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference International Conference Machine Learning, ICML'10, pages 1015-1022, Madison, WI, USA, 2010. Omnipress. ISBN 9781 605589077.
Yanan Sui, Joel Burdick, Yisong Yue, et al. Stagewise safe bayesian optimization with gaussian processes. In International Conference on Machine Learning, pages 4781-4789. PMLR, 2018.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294 1933.
Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Infor- mation Processing Systems, 33, 2020.
Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. Bayesian op- timization in billion dimensions via random embeddings. Journal of Artificial Intelligence Research, 55:361-387, 2016.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning- volume 51 of Proceedings of Machine Learning Research, pages 770-378, Cadiz, Spain, 09-11 May 2016. PMLR.
Kevin K Yang, Zachary Wu, and Frances H Amoll. Machine-learming-guidee directed evolution for protein engineering. Nature methods, 16(8):687-694, 2019.
12

---

Under review as a conference paper at ICLR 2022
A
PROOFS
In this section, we provide proofs for our main theoretical results (Proposition and Proposition 2).
A.1 PROOF PROPOSITION REGRET BOUND ON DISCRETE DECISION SET
We follow the proof structure in Srinivas et al. (2010) and introduce new notations to characterize the learning process of the neural network and the collision in the proof.
Before proving Proposition 1, we first introduce a few useful lemmas.
Lemma 1. Pick E (0, 1) and set Bg = 2 log( /8), where E1z1 Tx Te. >> 0. Then with probability 2 Va E D,
(ar((r (,t-1),,0-11 -e S f2 oz
- (g(x, 0gH-11)
Here 0g is the parameter for g at time step t - 1. 0p is the parameter for h at time step t 1.
Proof. Vx E D. Z 1, Conditioned on Y = (v) Vr-1): [-i,,.., F2-1} 1 are deterministic, and h(g(a, (0,(2,,,,,))),0,.-11 -n) sianity of h, we have
(g(z, 0g.-)))) Then using the subgaus- N ( (
Pr([h(o(2,0 0g,t-1), -4,, 14,1 01-1( (00r,0g,--0 (g.--1))3 e B./2
Applying the union bound, with probability 2 21 D 9:12 Vx E
0g((2,00,-1);00 - Mel S 8. 0 9g.-11)
Let Die 9.12 2012 8/T,, applying the union bound for Vt e: N the statement holds.
Lemma 2. Consider stationary and monotonic kernel, and assume that retraining the neural network g does not decrease the distance between data points in the latent space. Then Vt 2
re(0:-1) 5 2 -,, $ 2/3/2 11-1(g(1; 0g.r))
(1-1
Here 0g. is the parameter for g time step T.
Proof.
re(0c-1) = -/g(=", 0gg0-11), 0),00 hani)-h((ggla ())) 0n,
$ 122 0--1(g(0' 0g.t-11) 1)) + 14-1 0g,011) - h[g((e, 0y,t-1), 0n,t-1)
S 1/12 01-1 -11-
0g,0-1) Mg(ar, 1g. 01,e-1)
2/3/ 00//00000(())) 0 (,,))))
< 22, 0g.r))
The last line is because the non- con-decreasing distance between g(x 0g.r) and ( Vx, x' D leads to larger variance a, when using a stationary and monotonic kernel.
Ll
Lemma 3. The information gain for the points selected can be expressed in terms of the predictive variance. by = (h(g(z,, fy 0),r)) E RT:
T Σ
I (yr; h7) 2
log + - (7.
t=l
13

---

Under review as a conference paper at ICLR 2022
Proof. First, get (yr: hy) H(yr) log I Then,
(yr) (yr--1) (1 (2T-1)
H(yr-1) log ( (g.T))))
Since 21, ... vT are deterministic conditioned on yr-1- The result follows by induction.
Lemma 4. The gap of the mutual information between collision-free g(ø,, gg.)) and unregularized g( 0g.0) is
I [yr: h(r, 0nrr) = II [yr; h (arr,,,,,r1 (n,r)) )lr)
(yr: h(2r, 0n,o))oT)
= (yr, Φr; h(zr, 0n,u)) - I (h(2r, On,u): Φr))
Here 24 g( 9gT)). 6 is the identification of collided data points.
The result is simple application of the chain rule of mutual information. I1 (yT; hr) I (yr; h(rr, (),r)) corresponds the information gain under fully regularized and assumed collision-Fie setting. [yr, 01; h(21, 0n,o)) corresponds to information gain under unregularized setting.
Lemma 5. Pick 8 E (0, and let Be be defined in Lemma 1. Then, the following holds with probability 2 VT 2 1,
T
Σ
r s (yCillyT;;br) S C13r(1T
Here C1
1og(1+o-1) 8
Proof.
We first observe that
(g.x,, 0g.r)) S 48:00 20- fg.r)))
< 43.00
log( + 30 ((,((1)00gT)) 0g.T))
log(1 + t
Combining the above inequality with Lemma 2, Lemma 3 and Lemma 4 completes the proof.
Now we are ready to prove Proposition 1.
Proof of Proposition I. Proposition is simple consequence of Lemma 4 and Lemma and Mauchy-Schward inequality.
A.2 PROOF OF PROPOSITION 2:
REGRET BOUND WITH LIPSCHITZ-CONTINOOON ORJECTIVE FUNCTION
We first modify Lemma 5.7 and Lemma 5.8 in Srinivas et al. (2010) since we are assuming the deter- ministic Lipschitz-continuity for h. Use the same analysis tool Ze defined set of discretization Z, C Z where Z, will be used at time f in the analysis.
We choose a discretization Z, of size dr.) so that Vz E Z.
||2 [z]elli S rd/T.
where [z], denotes the closest point in Z, z.
Lemma 6. Pick & (0, 1) and set 3 log( + 2d log (TrilP), where Eiz1 75 1, म. > 0.
Let = Lrdi2. Hence then
11(z",n,tz11 00,0.11) Mr- ---1 8, 0 0 1/8 W21
holds with probability 2 1 5.
14

---

Under review as a conference paper at ICLR 2022
Proof. Using the Lipschitz-continiity and equation we have that Vz E Z, (1(2,, fn - h([z]es: 0m.-11) - Lrd/T
By choosing Lrdt?, we have [z|| (Lrdtt) and
V2 Z, (2, 0h,r-1) h([zles 0n,r-1)1 (n.t--)) Then using Lemma we reach the expected result.
Based on Lemma and Lemma 6, we could have the following result directly.
Lemma 7. Pick E (0, 1) and set = 2 6
(Lrdt"), where Σ121 Te = 1, ne > 0.
Then with probability 2 1 6, for all E N, the regret is bounded as follows:
23, /2 ( 1/t
Proof. Using the union bound of 5/2 in both Lemma 2 and Lemma 6, we have that with probability 8:
rt h(=)(((--)>))) h((z)
s 1/ 1 + 1/12² + (1-1(2)) - h(z))
s 2 0--)(2)) + +1/12
which complete the proof.
Now are ready to prove Proposition 2.
Proof of Proposition 2. Using Lemma 7, we have that with probability 2 21 1 6:
Σ
48,07-1(z)) (187((n-I(((77.0AA (n.))rr)) VT > 1
By Cauchy-Schwarz
T Σ 23 (w)) S VGBrltr - ((((z),0no):(r)) v [=1
Finally, substitute मe with 7222 (since 11/2= x2/6). Proposition 2 follows.
15

---

Under review as a conference paper at ICLR 2022
B DEMONSTRATION OF THE COLLISION EFFECT
B.1 VISUALIZATION THE COLLISION EFFECT LATENT SPACE
We demonstrate the collision effect in the latent space. We train the neural network Feyn- man dataset with 101 data points which demonstrate the latent space after two retrains with the retrain interval larization parameter p le penalty parameter le-2, retrain interval T,, weighting parameter 1e- and the base kernel set to be square exponential kernel. The non-regularized one employs LSO.
be 50 data points. The regularized one employs DW LOCo, with the regu-
Zom
Canoe
Gakive
510
036
CD loc Fato
Colsuen
(a) 1-D regularized latent space
(b) I-D non-regularized latent space
Figure 5: Illustrate the 1-D latent space Feynman III.9.52 dataset. The second row shows the ratio that the penalty define equation is non-zero. The third row shows point-wise estimation of A. 5a shows regularized latent space with a few observable collisions. 5b shows non-regularized latent space with bumps of collisions especially around the maxima among the observed data points. Besides, having fewer collisions the latent space contribute to the optimization through improving the learned Gaussian process. We observe in this comparison that the next point selected by the acquisition function of the regularized version is approaching the global optima, while the next point in the non-regularized version is trying solve the uncertainty brought by the severe collision near the currently observed maxima.
B.2 THE COLLISION EFFECT ON PROPER NEURAL NETWORKS
In this section, we provide empirical results supporting the claim in section 3.3 that increasing the network complexity often does not help to reduce the collision in the latent space.
For Feynman task, we test the single-layer neural network, which consists of 10, 1001, 5000, or 6000 neurons with Leaky Relu activation functions.
For Max Area task, we test the three-layer dense neural network. The first layer consists of 50, 100, 1000 1500 neurons with Tanh activation functions. The second layer consists of 50 neurons with Tanh activation functions. The third layer consists neurons with Leaky Relu activation functions.
For Rastrigin-2D task, we test the single-layer neural network, which consists of 10, 100, 1000, or 5000 neurons with Leaky Relu activation functions.
For SPOKES task, we test the single-layer neural network, which consists of 10, 100, 1000, or 2000 neurons with Leaky Relu activation functions.
16

---

Under review as a conference paper at ICLR 2022
Calldden
10
Feynman
ran
Het moc
Cellzenn Laterr g000s
3.30
1000
700
100
3000
6030
2006
130
0006
(b) Max Area
Tran 1580
Text mor
Calluze Lanart space
27000
20000
125040
Rastrigin-21
Tren mow
Ieil mia
Colleen Latert-pee
00025
130050
0005
50000
SPOKES
1507
lest
Sert of Latert space
3.3
20
3.5
(e) Water Converter
Figure 6: These curves show the network design test results. The collision value shown here the penalty term proposed in equation 1. The x-axis denotes the neural network's general complexity. The collisions for model with lowest test MSE are still significant.
For the Water Converter task, test the three-layer dense neural network. The first layer consists of 512, 1024, 2048, or 4096 neurons with Tanh activation functions. The second layer consists of half of the first layer's neurons with Tanh activation functions. The third layer consists of half of the second layer's ss neurons with Leaky Relu activation functions.
We demonstrate the collision effect regression task Rastrigin-2D..
17

---

Under review as a conference paper at ICLR 2022
N.Lor Rastiggn
USE on Hastiggh 10
Beg evrbed
200.30
170
32033
20000
250000
26
1550
2500
2008
9.5
1200
1330
2540
2006
-150
-50
100
(a) non-regularized latent space
(b) Negative Log Likelihood
(c) Mean Squared Error
Figure 7: Illustrate regression task on Rastrigin-2D dataset. 7a shows non-regularized latent space after sufficiently trained as is demonstrated in 7b. 7b shows the NLL of the training process. 7a shows the corresponding MSE.
As illustrated in figure 7a even after being sufficiently trained after 3000 iterations, there is still collisions in the latent space especially around the optima. figure 7a shows that by regularizing the latent space, the ultimate MSE could also be improved.
18

---

Under review as a conference paper at ICLR 2022
C SUPPLEMENTAL MATERIALS ON ALGORITHMIC DETAILS
Our implementation of LOCo and DW LOCo built upon the open source package GPytorch (Gardner et al., 2018). The deep kernel is trained with back propogation. We use the Adam (Kingma and Ba, 2014) optimizer with learning rate set to be le le- Below we discuss the detailed configura- tion of the underlying neural network and the choice of the key parameters used by the algorithm.
C.I ALGORITHMIC DETAILS ON NEURAL NETWORK ARCHITECTURE
As the primary goal of our paper was to showcase the performance of novel collision-free reg- ularizer, we pick our network architectures to be basic multi-layer dense neural network. We use 4-layer dense neural network. Its hidden layers consist of 1000, 500, 50 neurons respectively, each with Leaky Relu activation functions. The output layer also uses Leaky Relu as its activation function and generates I-dimensional output.
st hidden layer
th hidden layer
input layer
output layer
Mo
20
(L)
(T+1)
271
yi
(L)
90
D
V22
Figure 8: Network graph of a (L + 1) llyee dense network with D input units and 1 output units. In our experiments, L is be 3.
Pre-training the Neural Network We randomly select the points from the bottom of the dataset to avoid the pre-training simplifying the optimization task while helping with initializing the param- eters of the model. The points are used initialize the neural network instead of being served as initial selections of the optimization task. Without the pre-training stage, the latent embedding fed by the neural network the gaussian process would random. The practical problem with such randomness could be much larger variance for the results since it influences the following neural network training process and the optimization process. In practice, it's always possible collect pre-training datasets from relative domains when aiming optimizing unknown objective functions. This pre-training method was also reported in the literature (Snoek et al., 2015).
19

---

Under review as a conference paper at ICLR 2022
C.2 PARAMETER CHOICES
We investigate the robustness of parameter choices of the regularization parameter on the Rastrigin 2D dataset. We show the results in the figure below.
Study
LS0
L=C.0=0001
A0Co1
LOCo
LOCo A-1000
A-100.0
15
DW LOCO
DW LOCo
DW LOCo
DW LOCo
DW LLCo000100.
10
Simple Regret
20
80
100
Iteration
Figure 9: Simple regret under different parameter settings on the Rastrigin 2D dataset. The colored area represents the standard error of the tests certain iteration. Each experiments repeated eight times. The figure shows that a moderately large À suffices to achieve decent performance in terms of simple regret. We believe that the wide range of objective values of the test dataset, which otherwise would hurt the optimization performance, can be regularized by the collision penalty. The curves demonstrate the decent performance of DW LOCo as long as the parameters are set to be too small.
20

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL RESULTS
Random EMbedding Bayesian Optimization (REMBO) (Wang et al., 2016) leverages simple random linear transformations improve the efficiency now-efieetieeddimesion high-dimensional tasks. We compare LOCO and DW LOCo with the performance of this wawdlo--mmbeddingbba-eeese method and empirically exposed the failure case of REMBO when its modeling assumption does not hold (i.e. when dealing with dataset that has large effective dimensions).
Supermova
SPOKES
Rastrigin
100
Simple
Simple
Simple
071
Iterationn Water Converter
Iterations Max Area
Iterations
HDBO
50000
$0.00
Simpe regg
Simple regres
Simple
338
Figure 10: Experiment results on six synthetic real datasets. Each experiment repeated least eight times. The shaded area around the mean curve denotes the & standard deviation. denotes the number of cases repeated in experiments. As illustrated in the figure, the uundom-embedding. DW LOCo. We place the discussion over REMBO here for two reasons. Firstly, there has been several problems about REMBO as discussed in section 2. Secondly, the experiments are conducted on tasks where the effective dimensions are similar scale the dimensionality of the original inputs and doesn't 't align with the assumption of REMBO.
vn Here â denotes the empirical
methods have been significantly outperformed by LOCo and
21


Reviews:
[{"review_id": "ZALNptbQuS", "reviewer": "Reviewer_Knwx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Merits\n\n- Adapting Bayesian optimization for high dimensional search spaces is arguable one of the key challenges at the moment, and learning embeddings is one of the most promising directions. The paper addresses an important practical problem of these embeddings\n\n- Overall I found the paper to be well written and easy to follow.\n\n- The paper convincingly motivates the occurrence of collisions and the proposed penalty terms seems sound.\n\n- While the empirical evaluation exhibit some flaws (see below), it seems that the proposed approach indeed improves upon baselines.\n\n\n## Concerns\n\n- Figure 1 is confusing to me. What is the original function? What means current sample? It would be great to visualize the true objective to see how far off the GP prediction is?\n\n- Last sentence in Section 4.2: How exactly do you select \\lambda in the input space? Shouldn't lambda be proportional to epsilon (i.e the observation noise)?\n\n- Do you adapt rho during optimization, since the order of magintude of the likelihood might change over time?\n\n- How do you select the pretrained datapoints to train the neural network?\n\n- I am not sure what you mean with \"... to meet the limitation of memory on our computing instance, we uniformly sample 10000 points from the original dataset \". This doesn't seem to be a lot of data? How many datapoints does the original benchmark have?\n\n- Why is TPE and BO missing for the Max Area and the HDBO problem?\n\n- It seems that the weighted version of LOCo does not lead to any gain in performance. Only for the Supernova DW LOCo outperforms LOCo in the early stage of the optimization, which is surprising to me, since the collisions of well-performing configuration should happen in later stages of the optimization process. This is somewhat contradicting the intuition provided by the authors.\n\n- How many independent trials do you conduct for each method in Figure 3?\n\n- Overall, Figure 3 might be somewhat missleading. If I am not mistaken you have to first sample several hundred data points to train the neural networks, which is not necessary for  TPE, PSO and BO. This initial phase is not reflecting in Figure 3 and puts these method into a disadvantage.\n\n- Looking at Figure 9 it seems that LOCo is quite sensitive to the choice of lambda. For non-weighted version of LOCo lambda = 0.1 seems to work best, however for DW LOCo this choice of lambda performs worst and larger lambda (i.e 10) seem to work better.\n\n- I am missing additional ablation studies for the other hyperparameters: \\zeta and \\rho\n\n\n\n## Minor Comments\n\n- Typo in Section 4.2: yi is defined twice instead of yj\n\n- Increase font size of the legend in Figure 3\n\n- Increase the line width in Figure 3\n\n", "summary_of_the_paper": "Despite its success, Gaussian process based Bayesian optimization still struggles in high dimensional search spaces. Current approaches aim to learn an embedding to optimize the objective in a low dimensional continuous latent space. This paper provides evidence that with current approaches, different data points in the input space can be mapped to same point in the latent space. To avoid these collisions, the paper proposes a new regularization technique based on pairs of observed datapoints.", "summary_of_the_review": "Overall I think the paper tackles an important problem and provides a sensible solution to it. However, I am underwhelmed by the empirical results of the paper and not yet convinced that the proposed approach actually works reliably in practice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WBF1XqdJjdD", "reviewer": "Reviewer_aX8w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "MAJOR COMMENTS:\n\nContribution I:\n\nThe paper claims to \"expose the limitations of existing latent space optimization approaches due to the collision effect on the latent space.\" I do not think the paper provides sufficient evidence to support this contribution. Sections 3 and 4 state that collisions are an issue, but nowhere does the paper _demonstrate_ that it is an issue. Fig. 4 in the appendix shows that there are collisions happening, but the paper does not provide any evidence that collisions are detrimental for Bayesian optimization. Simply raising awareness that collisions can happen is not a contribution because that has already been done (see Moroconi et al., 2020). The contribution would be to show that this actually limiting performance of current methods and that was not shown. In Figure 3, as best as I can tell, LSO is the same as LOCo but without the regularization. The performance of LSO on the benchmarks does not appear to be significantly worse than that of LOCo, so it appears that collisions are not a primary limitation of existing latent space optimization approaches.\n\nAnother issue related to motivation and the necessity of the methods introduced in the paper is that it isn't clear to me why end-to-end learning (described in the Introduction) isn't sufficient to avoid the most serious collisions that would be harm GP modeling. Consider the two latent spaces in Fig. 7 in the Appendix. Could the authors report the GP marginal log-likelihood for each of them? It looks clear to me that the latent space for the regularized latent space will have much higher log-likelihood than the non-regularized latent space, because having a big high-variance area is going to be bad for likelihood of a model that assumes homoskedastic noise. If the marginal log-likelihood is higher for the regularized latent space as I suspect, then why was the regularization necessary at all? If I fit the latent space end-to-end (i.e., to maximize marginal log likelihood of the GP) then why doesn't that  smooth out the collisions to better match the homoskedastic noise assumption? This is my main concern with the paper: I acknowledge that collisions could be an issue, but I think that end-to-end training is sufficient to resolve the issue, and so I'm not convinced (and did not see evidence in the paper) that the regularization is actually necessary.\n\nContribution IV:\n\nThere are three vital issues with the empirical evaluation, any one of which would be sufficient for me to recommend revision of the paper.\n\n(1) Only one HDBO baseline is included in the comparison (SE LSO). In a field with so much recent development as HDBO I do not think this is appropriate. At the very least I would insist on adding TuRBO (Eriksson et al., 2019) to the comparison since it has a widely available implementation (https://github.com/uber-research/TuRBO) and shows state-of-the-art performance for HDBO problems of the type used in this paper.\n\n(2) Figures show 1 standard error. They should show 2, so that they provide a 95% confidence interval. As is, the error bars are showing a 68% CI for the means, which is both highly non-standard, and also makes it hard to assess which of these differences are significant. It looks like there are some significant differences, but most differences are not significant at a 95% level.\n\n(3) Section 6.2 describes that the neural networks were pre-trained with 100 or 400 data points. How were these chosen? More importantly, were all methods given 100 or 400 extra data points for initialization? In particular, was BO given 100 or 400 points for its initialization so that all points get the same amount of data? I worry that it was not because the text specifies neural networks as getting the additional data. If not, it would be inappropriate to compare the methods by total number of iterations when some methods were allowed more data than others. So please verify that all methods were given the same total amount of data. For instance, when TuRBO is added as an additional baseline, it will be important to set n_init as the same 100 or 400 used as pre-training for the neural network-based methods.\n\n\nOTHER COMMENTS:\n\n* Missing prior work: Moroconi et al. (2020) studies the issue of latent space collisions for linear embeddings and should be referenced.\n\n* For (1), this is not scale invariant on z. If I multiply z_i and z_j each by C, then I will require a larger value of \\lambda to achieve the same collision penalty. The paper proposes setting \\lambda using the original X. I don't think that can reasonably be done when the penalty is not scale invariant in Z. I can keep X the same as it is (so lambda stays fixed) and then by adjusting the weights in the neural net could linearly expand Z in a way that doesn't affect the GP, but would drastically change p_ij by changing the scale of Z relative to \\lambda. I think this is problematic for the definition of the penalty (it should be scale invariant to the embedding), and also based on this fact I disagree with the claim that \\lambda could be set other than through hyperparameter tuning.\n\n* The chosen setting of \\lambda = 1 is entirely unjustified and would depend on the scale of f at the very least.\n\n* Unclear notation: Section 3.3 defines the collision property in terms of |g(x_i) - g(x_j)|. g(x_i) is a point in the presumably multi-dimensional latent space, so it is unclear what norm is being used here. Was the latent space always 1-d in this paper? It isn't in others. Same issue in (1).\n\n* Fig. 1: I don't know what \"Current Sample\" and \"Current Mean\" are. There is no y axis label.\n\n* theta_h and theta_g are not defined in the paper. I think that theta_h are the GP parameters and theta_g are the NN parameters, but please define.\n\n* Algo 1 Line 3: Shouldn't alpha also be a function of theta_h if those are the GP parameters? If not then what is theta_h?\n\n* Algo 1 Line 5 says that the theta_t, which I understand to be both the entire neural network and the GP hyperparameters, are trained every iteration. But Section 6.1 says \"The hyper-parameters for GP are tuned for periodically retraining in the optimization process.\" Does periodically mean every iteration? If not what is being retrained in Algo 1 Line 5?\n\n* The paper needs editing for grammar.\n\n\nReferences:\n\nMoroconi R, Kumar KSS, Deisenroth MP, High-dimensional Bayesian optimization with projections using quantile Gaussian processes, Optimization Letters, 2020\n\nEriksson D, Pearce M, Gardner J, Turner R, Poloczek M, Scalable global optimization via local Bayesian optimization, NeurIPS, 2019", "summary_of_the_paper": "The paper studies the use of embeddings for high-dimensional Bayesian optimization and introduces a regularization term into the model training to ensure smoothness in the latent space, so that it will be suitable for GP modeling and Bayesian optimization.", "summary_of_the_review": "The idea of regularization in the embedding is interesting but I feel is underdeveloped and I do not find that the paper provides sufficient evidence for its claims that the method is necessary or useful. As such I do not think it is ready for publication at this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PqohPLtIBu_", "reviewer": "Reviewer_DSCs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have several questions that I would like to kindly ask the authors, hoping this can improve my assessment of the paper: \n\n1. What is the relation of this work with the recently proposed method of combining metric learning with latent space Bayesian optimisation (Grosnit et. al 2021). At first sight, the problems seem equivalent yet no comparison is made. Can the authors please clarify the main differences in that work? \n2. Concerning the theoretical results, I am a bit confused about the novelty of the proof. Does it seem to be a direct application of Srinivas 2010? I do understand the changing neural network idea, however, resolving this by an assumption of monotonic increase seems to answer the difficulty of the proof with an assumption. I would be really grateful if the authors help me understand the challenges and novelty in the proof. Additionally, when presenting the proof, it would be great if there is a section listing all assumptions made. I found it extremely disturbing to have to read the proof not knowing the assumptions made up-front. Please try to improve the proof's exposition. \n3. I am a bit confused about the stochastic treatment of the approach. The authors seem to add noise everywhere but do not study its effects in, for example, equation 1. Like equation 1 is defined over observations of y and z and not in expectation? Why would that be a good idea and how does this affect the variance of p_ij? \n4. Can the authors please help me understand the relationship between lambda and the Lipschitz constant of the function? If they are the same, how is it possible to estimate that? Estimating the Lipschitz constant is an extremely hard task to do. Additionally, if we are enforcing the net to be Lipshitz, there is a large body of work that is ignored in this paper that should be referred to. In fact, that work leads us to understand that difficulty beyond MLPs to more sophisticated networks. Was this the reason the authors chose simple networks to handle? \n5. Arriving at Equation 3 is a bit unprincipled from my point of view. I would appreciate a rigorous derivation from probabilistic assumptions of GPs. What kernels would those amount to and how can Equation 3 be derived from GP log-marginals? \n6. Reading the experiments section, I found it very hard to understand. I also tried to find the additional details and descriptions in the appendix but could only find ablation studies. It would be really useful if the authors improve the exposition of this section clearly stating the dimensions of each of the considered tasks. The 2D shape area maximisation is a Toy domain and so is the sum 200D. In the sum 200 D the black-box clearly exhibits an affinity structure to which specific GPs with affine kernels have been developed. I urge the authors to compare to one of those. In the supernova and redshift what are the dimensions of the problem? Are those high-dimensional tasks? I think the authors should run on molecule domains which were one of the motivations of the work in Tripp et. al. 2020. They should also reference and compare to a wide literature of work that handles high-dimensional discrete structures (e.g., Junction Tree VAEs and many others). \n6. Figure 1 is unclear: what are the original space points and what are the latent points here? If the star y-value is the true optimum and its x-value is the optimiser, it is true that it is hard to model for the GP as the region where the optimiser is located is surrounded by low y-value points. But I don’t see the point of that Figure, it’s just a panel about how it can be difficult for a GP to model such data. It would have been better to have a real example maybe.\n7. What does it mean to minimise uniformly the collisions? Surely the regulariser will be proportional to the difference in function values.\n8. Is the GP training jointly with the neural network? Can you please clarify this? If so, what is the relation to GPLVMs?\n9. It is said that it is more efficient to learn how to eliminate collisions in regions around the optimiser in the latent space due to limited resources (i.e. labels) in BO. But the dynamic method does not solve the data requirement problem. It merely weighs the penalties based on function values, which is a good first step. I agree that this weighting enables the regulariser to penalise more aggressively collided pairs with high function values. But to compute the weights you nevertheless need all points to have labels. Can you please clarify this point? \n10. In the appendix are these architectures really adapted? If I understand your latent space is always 1-dimensional. Isn’t that limiting? Isn’t that actually creating collisions? \n\nMinor: \n\nI think the K_SE in section 3.2 is missing a square inside the exponential. \nThe paper is really hard to read and understand. I would urge the authors to: 1) write a clear set of assumptions during the proof, 2) clear listings of dimensionalities and structures of networks used in the experiments, and 3) Improve the quality of the Figures using vectorised graphics. \nThe authors could also cite these two papers: https://arxiv.org/pdf/1812.02833.pdf and https://openaccess.thecvf.com/content_CVPR_2020/papers/Ding_Guided_Variational_Autoencoder_for_Disentanglement_Learning_CVPR_2020_paper.pdf", "summary_of_the_paper": "This paper is concerned with latent space Bayesian optimisation that typically involves a step of learning a lower-dimensional latent representation. The authors focus on non-linear embeddings generated through neural networks. They observe a collision problem in the latent space and attempt to resolve it by introducing a regulariser based on Lipschitz continuity. In a set of experiments, they demonstrate that such a method is effective in various benchmarks. Although interesting, I still find this paper lacking as presented in the next section. ", "summary_of_the_review": "See above. This paper seems interesting but is not convincing enough for an acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-n5WGjQqB-l", "reviewer": "Reviewer_G6dp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths\n1. The paper addresses the problematic behavior (collision) that it deserves and that has not been explicitly discussed well and proposes a solution to the collision\n2. The paper adds a proof technique which may be useful for future GP bandit methods using evolving data points.\n3. The paper is well written so that it is easy to follow.\n- Weaknesses\n1. By introducing the collision penalty, the loss function has more hyper-parameters which unavoidably calls for some heuristic. Many hyper-parameters including neural network architectures are different between BO experiments and collision ablation study in the appendix. For the same hyperparameters used in BO exp, is it consistently observed that the collision is well suppressed?\n2. With the added collision penalty, GP kernel hyper-parameters cannot be optimized contrast to ordinary GP training. Would it be possible to frame this approach in more Bayesian way (e.g interpreting collision penalty as some sort of prior on network parameter structure) so that more principled approach to handle newly introduced hyper-parameters? \n3. “Dynamical weighting” does not seem to make noticeable differences empirically, any formal argument to support this?\n- Minor points\n1. “GP are not intrinsically designed to deal with structured input” sounds a bit controversial, any reference for this?", "summary_of_the_paper": "Focusing on Bayesian optimization using latent spaces to overcome the difficulties coming from the high dimensionality of the search space, the paper proposes identifies a problematic behavior called the collision, and proposes a regularizer to mitigate the collision and in turn to enhance the BO performance.", "summary_of_the_review": "The paper focuses on an issue in latent space based Bayesian optimization and proposed a solution for it. It also appears to provide a useful theoretical tool in regret analysis. However, the added complexity in the pair-wise loss and how new hyperparameters are handled deserves more through theoretical or more extensive empirical support. Therefore, I think an improvement is required for the paper’s acceptance. I will be expecting that some of my concerns are answered in the discussion period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the problem of identifying a low-dimensional latent space for high-dimensional Bayesian optimization (BO) is considered. In particular, the authors focus on the problem of collision, where different points in the original space become identical in the latent space, and propose a regularization method to avoid this problem. Latent space identification for high-dimensional Bayesian optimization is an interesting and the authors' approach sounds reasonable. However, many reviewers pointed out that the discussion and results in the paper do not provide sufficient evidence for the authors' claims. Therefore, we have to conclude that the paper cannot be accepted at this time.
Decision: Reject