Learning Collision-free Latent Space for Bayesian Optimization
Authors: Fengxue Zhang
Yair Altas
Louise Fan
Kaustubh Vinchure
Brian Nord
Yuxin Chen
Venue:  (2021)
Keywords: Latent space
Bayesian Optimization
Collision

Abstract: Learning and optimizing a blackbox function is a common task in Bayesian optimization and experimental design. In real-world scenarios (e.g., tuning hyper-parameters for deep learning models, synthesizing a protein sequence, etc.), these functions tend to be expensive to evaluate and often rely on high-dimensional inputs. While classical Bayesian optimization algorithms struggle in handling the scale and complexity of modern experimental design tasks, recent works attempt to get around this issue by applying neural networks ahead of the Gaussian process to learn a (low-dimensional) latent representation. We show that such learned representation often leads to collisions in the latent space: two points with significantly different observations collide in the learned latent space. Collisions could be regarded as additional noise introduced by the traditional neural network, leading to degraded optimization performance. To address this issue, we propose Collision-Free Latent Space Optimization (CoFLO), which employs a novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to objective value to be Lipschitz continuous. CoFLO takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance. We provide a rigorous theoretical justification for the regularizer by inspecting the regret of the proposed algorithm. Our empirical results further demonstrate the effectiveness of CoFLO on several synthetic and real-world Bayesian optimization tasks, including a case study for computational cosmic experimental design.

Under review as a conference paper at ICLR 2021
LEARNING COLLISION-FRES LATENT SPACE FOR BAYESIAN OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning and optimizing blackbox function is common task in Bayesian optimization and experimental design. In real-world scenarios (e.g.. tuning hyper- parameters for deep learning models, synthesizing a protein sequence, etc.), these functions tend to be expensive to evaluate and often rely on high-dimensional inputs. While classical Bayesian optimization algorithms struggle in handling the scale and complexity of modern experimental design tasks, recent works attempt to get around this issue by applying neural networks ahead of the Gaussian process to learn a )inddimensional) latent representation. We show that such learned representation often leads to collision in the latent space: two points with significantly different observations collide in the learned latent space. Collisions could be regarded as additional noise introduced by the traditional neural network, leading degraded optimization performance. To address this issue, we propose Collision- Free Latent Space Optimization (CoFLO), which employs novel regularizer to reduce the collision the learned latent space and encourage the mapping from the latent space to objective value to be Lipschitz continuous. CoFLO takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance. We provide a rigorous theoretical justification for the regularizer by inspecting the regret of the proposed algorithm. Our empirical results further demonstrate the effectiveness of CoFLO on several synthetic and real-world Bayesian optimization tasks, including a case study for computational cosmic experimental design.
INTRODUCTION
Bayesian optimization is a classical sequential optimization method and is widely used in various fields, including recommender systems, scientific experimental design, hyper-parameter optimization, etc. Many of theses applications involve evaluating an expensive blackbox function; therefore the number of queries should be minimized. common way to model the unknown function is via Gaussian processes (GPs) Rasmussen and Williams (2006). GPs have been extensively studied under the bandit setting, and has proven to be an effective approach for addressing broad class of black-box function optimization problems. One of the key computational challenges for learning with GPs concerns with optimizing specific kernels used to model the covariance structures of GPs. As such optimization task depends on the dimension of feature space, for high dimensional input, it is often prohibitively expensive to train a Gaussian process model. Meanwhile, Gaussian processes are not intrinsically designed to deal with structured input that has strong correlations among different dimensions, e.g., the graphs and time sequences. Therefore, dimensionality reduction algorithms are needed to speed up the learning process.
Recently, it has become popular to investigate GPs in the context of latent space models. As an example, deep kernel learning (Wilson et al., 2016) simultaneously learns a (low-dimensional) data representation and scalable kernel via an end-to-end trainable deep neural network. In general, the neural network is trained to learn a simpler latent representation with reduced dimension and has the structure information already embedded for the Gaussian process. Such combination of neural network and Gaussian process could improve the scalability and extensibility of classical Bayesian optimization, but it also poses new challenges for the optimization task (Tripp et al.. 2020). As we later demonstrate, one critical challenge brought by introducing the neural network is that the latent representation is prone collisions: two points with significant different observations get

---

Under review as a conference paper at ICLR 2021
Non-regularized Latent
150
Current Sample
Current Mean
100
Observations
Observed Optmum
75
Optimum
25
0-
-0.4
-0.2
0.0
0.2
0.4
0.6
Latent Value
Figure I: Illustration of the collision effect in latent space-based Bayesian optimization tasks. Since the data points around the optimum severely collided, BO misguided to the sub-optimum.
too close in the latent space. The collision effect is especially evident when information lost by dimension reduction, and/or when the training data is limited in size in Bayesian optimization.
As illustrated Figure 1, when passed through the neural network, data points with drastically different observations are mapped close positions in the latent space. Such collisions could be regarded as additional noise introduced by the neural network. Although Bayesian optimization is known be robust mild noisy observations. the collision in latent space could be harmful the optimization performance, as it is non-trivial to explicitly model the collision into the acquisition function. In addition, the additional noise induced by the collision effect will further loosen the regret bound for classical Bayesian optimization algorithms (Srinivas et al., 2010).
Overview of main results To mitigate the collision effect, we propose novel regularization scheme which can be applied as a simple plugin amendment for the latent space-based Bayesian optimization models. The proposed algorithm, namely Collision-Fii Latent Space Optimization (CoFLO), leverages regularized regression loss function, to periodically optimize the latent space for Bayesian optimization.
Concretely, our regularizer is encoded by novel pairwise collision penalty function defined jointly on the latent space and the output domain. In order to mitigate the risk of collision in the latent space (and consequently boost the optimization performance). one can apply the regularizer uniformly to the latent space to minimize the collisions. However, in Bayesian global optimization tasks, we seek to prioritize the regions close to the possible optimum, collisions in these regions are more likely to mislead the optimization algorithm. Based on this insight, we propose a optimization- aware regularization scheme, where assign higher weight for the collision penalty those pairs of points closer to the optimum region in the latent space. This algorithm-which we refer to dynamically-weighee
CoFLO-is designed dynamically assess the importance of collision during optimization. Comparing to the uniform collision penalty over the latent space, the dynamic weighting mechanism has demonstrated drastic improvement over the of-te----rrrrtt latent space- based Bayesian optimization models.
We summarize our the key contributions below:
We propose a novel regularization scheme, a simple plugin amendment for latent space- based Bayesian optimization models. Our regularizer penalizes collisions the latent space and effectively reduces the collision effect.
We propose an optimization- awarre dynamic weighting mechanism for adjusting the collision penalty to further improve the effectiveness of regularization for Bayesian optimization.
We provide theoretical analysis for the performance of Bayesian optimization regularized latent space.
We conducted an extensive empirical study four synthetic and real-world datasets, including a real-world case study for cosmic experimental design, and demonstrate strong empirical performance for our algorithm.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Bayesian optimization has demonstrated promisming performance in various cost-sensitive global optimization tasks (Shahriari et al., 2016). However, due to its intrinsic computational limitation in the high-dimensional regime, its applicabiliity has been restricted to relatively simple tasks. In this section, we provide a short survey on recent work in Bayesian learning, which were designed to overcome the high-dimmeniionaity challenge for both Bayesian optimization and regression tasks.
Deep kernel learning Deep kernel learning (DKL) (Wilson et al., 2016) combines the power of the Gaussian process and that of neural network by introducing a deep neural network to learn a mapping 9 X + 2 from the input domain X' latent space Z, and use the latent representation E 2 as the input of the Gaussian process. The neural network g and spectral mixture base kernel & forms a scalable expressive closed-form covariance kernel, denoted by kD (b),()))))))) - k(g(z)) g(æ;)), for Gaussian processes. Despite of encouraging results in numerous regression tasks, it remains unclear whether DKL readily applicable to Bayesian optimization. One key difference in Bayesian regression and optimization tasks is the assumption on the accessibility of training data: Bayesian optimization often assumes limited access to labeled data, while DKL for regression relies on abundant access to data in order to train deep kernel function. Another problem lies in the difference between the objective functionn. While DKL focuses on improving the general regression performance, it does not specifically address the problem caused by collisions, which-as we later demonstrate in section 3.3 could be harmful for sequential decision making tasks.
Representation learning and latent space optimization Aiming at improving the scalability and extensibility of the Gaussian process, various methods are proposed to reduce the dimensionality of the original input. Djolonga et al. (2013) assume that only a subset of input dimensions varies, and the kernel is smooth (i.e. with bounded RKHS norm). Under these assumptions, they underlying subspace via low-rank matrix completion. Huang et al. (2015) use Autoencoder to learn a low- dimensional representation of the inputs to increase GP's scalability in regression tasks. Snoek al. (2015) further propose to learn a pre-trained encoder neural network before BO, Lu et al. (2018) learn variational auto-encoder iteratively during sequential optimization embed the structure of the input. The challenge for combining latent space learning with Bayesian optimization lies in that a pre-trained neural network may not extract adequate information around the more promising region of the input space. Furthermore, the latent space could be outdated without continuous updates with the latest acquired observation. Tripp et al. (2020) propose to periodically retrain the neural network to learn a better latent space, in order to minimize the number of iterations needed for LSO. They claim that by prioritizing the loss of more promising data points in the original input space (i.c. by assigning higher weight to these data points), the model could focus more learning high- value regions and allow a substantial extrapolation in the latent space to accelerate the optimization. However, such framework does not explicitly deal with collisions in the latent space, which found to be a key factor in the poor performance of modern latent space optimization algorithms.
3 PROBLEM STATEMENT
In this section, introduce necessary notations and formally state the problem. We focus the problem of sequentially optimizing the function f X' + R, where S Rd is the input domain. In each round f,, pick point 2; E X, and observe the function value perturbed by additive noise: Ye = f(x)) + 4 with Q ~ NO, o2) being i.i.d. Gaussian noise. Our goal is to maximize the sum of rewards Et= over T iterations, or equivalently, to minimize the cumulative regret Ri Σ rs, where r. == max f fx) (2x) denote the instantaneous regret of 24.
3.1 BAYESIAN OPTIMIZATION
Bayesian optimization typically employs Gaussian processes as the statistic tool for modeling the unknown objective function. The major advantage of using GP is that it presents a computationally tractable solution to depict sophisticated and consistent view across the space of all possible function (Rasmussen and Williams, 2005), which allows closed-form posterior estimation in the function space. BO methods starts with prior on the black-box function. Upon observing new
3

---

Under review as a conference paper at ICLR 2021
labels, BO then iteratively updates the posterior distribution in the function space, and maximizes an acquisition function measuring each point's contribution to finding the optimum, in order to select the next point for evaluation.
Formally, in Bayesian optimization we assume that f follows a (P(m(2)) k(xx,")) where m(x) is the mean function, (xx, x²) is the kernel covariance function. Throughout this paper, use (--=x') where the length scale I determines
squared exponential kernel, x3)) = dăg exp
the length of the "wiggles" and the output variance d'gg determines the average distance of the function away from its mean. At iteration T, given the historically selected points Ar = {Ti, ... Zt) and the corresponding noisy evaluations yr [./,... the posterior over f also takes the form of a GP, with mean Pr (z), covariance (2, x²), and variance (z):
ur(x)= = kT(=)" (Kr + o221
(r(x,1) = x(x,x')
o2T)
xx) (T(
where kx(x) [8(11, k(rr, =)}² and Kr is the positive definite kernel matrix [k: x' ='../A/ After obtaining the posterior, one can compute the acquisition function a : X R, which is used select the next point be evaluated. Various acquisition functions have been proposed in the literature, including popular choices such as Upper Confidence Bound (UCB) (Srinivas et al., 2010) and Thompson sampling (TS) (THOMPSON, 1933). UCB uses the upper confidence bound (x) [=(x))+ -g1/// with (ar) being the confidence coeffecient, and enjoys rigorous sublinear regret bound. TS usually outperform UCB in practice and has been shown to enjoy a similar regret bound Agrawal and Goyal (2012). It samples a function f₃ from the GP posterior Je ~ (P(r(()), x') and then uses the sample as an acquisition function: (r)(()) = (z).
Remark, Regret is commonly used as performance metric for BO methods. In this work we focus on the simple regret rip max
(xx max f((x) and cumulative regret R(T)- = =! Tb ZEX
KT
3.2 LATENT SPACE OPTIMIZATION
Recently, Latent Space Optimization (LSO) has been proposed to solve Bayesian optimization problems on complex input domains (Tripp et al., 2020). LSO first learns a latent space mapping objective 9 X Z to convert the input space X' to the latent space 2. Then, it constructs
mapping h Z - R such that f(x) = h(g(x)), Vz E Z. The latent space mapping g and base deep kernel, denote by kx, x²) k(g(()), g(=)). Thus, the actual input space for BO is the latent space and the objective function is h. With acquisition function Gna (2)) = (g(()) ). it is unnecessary compute an inverse mapping g -1 as discussed in Tripp al. (2020), as BO could directly select _, = arg max am)(x) Vt S T and evaluate f. In the meantime, n,EX'
kernel & could be regarded
BO can leverage the latent space mapping g. usually represented by a neural network, to effectively learn and optimize the target function
lower-dimension input space.
3.3 THE COLLISION EFFECT OF LSO
When the mapping 9 : X' - Z is represented by neural network, it may cause undesirable collisions between different input points in the latent space Z. Under the noise-free setting, we say there exists a collision in 2, if Ex; 2y X, such that when g(x;) = g(r,), [s(x;) f(x;))] ) > 0. Such collision could regarded as additional (unknown) noise on the observations introduced by the neural network g. For noisy observation y f(x) + e, we define collision follows: for p > 0, Ho,, Iy E D, [g(x;) - g(2,), < ply- - yy|-
When the distance between pair of points in the latent space is too close comparing to their difference the output space, the different output values for the collided points in the latent space could be interpreted as the effect of additional observation noise. In general, collisions could degrade the performance of LSO. Since the collision effect is priori unknown, is often challenging to deal with collisions in LSO, even if we regard it additional observation noise and increase the (default)
4

---

Under review as a conference paper at ICLR 2021
noise variance in the Gaussian process. Thus, it is necessary mitigate the collision effect, by directly restraining the representation learning phase.
4 COLLISION-FREE LATENT SPACE OPTIMIZATION
In this section, we introduce Collision-Free Latent Space Optimization (CoFLO), an algorithmic framework designed to mitigate the collision effect.
4.1 OVERVIEW OF THE COFLO ALGORITHM
The major challenge in restraining collisions in the latent space that, unlike the traditional regression problem, we cannot quantify it on single point's observation. We can, however, quantify collisions by grouping pairs of data points and inspecting their corresponding observations.
We define the collision penalty based on pairs of inputs, and further introduce a pair loss function to characterize the collision effect. Based on this pair loss, we propose a novel regularized latent space optimization algorithm', as summarized in Algorithm 1. The proposed algorithm first uses the pair-wise input and concurrently feeds them into the same network and then calculates the pair loss function. We demonstrate this process Figure 2.
Given a set labeled data points, we can train the neural network to create an initial latent space representation", similar DKL (Wilson et al., 2016). Once provided with the initial representation, we can then refine the latent space by running CoFLO and periodically update the latent space (i.e. updating the latent representation after collection batch of data points) to mitigate the collision effect as we collect more labels
Algorithm Collision-Reeguarizze Latent Space Optimization (CoFLO)
1: Input: Regularization weight p (cf. Equation 3), penalty parameter (cf. Equation 1), retrain interval T, importance weight parameter y (cf. Equation 2), neural network Mo, base kernel Ko, prior mean μo. total time steps Tü
2: for t = 1 to T do
3: Tt. < arg max or (M))
b maximize acquisition function
4:
Ve evaluation Tę
update observation
5: if 0 (mod T') then
6:
M,+1, <- retrain M, and K, with the pair loss function L p, (M,, Kr, D,) as periodical retrain
defined in equation
7: end
8: end for
9: Output: max
4.2 COLLISION PENALTY
Istent space
coliition penalty
In this subsection, we aim to quantify the collision effect based on the definition proposed Section 3.3. As illustrated in Figure 2, we feed pairs of data points into the neural network and obtain their latent space representations. Apart from maximizing the GP's likelihood,
par lose
concurrently calculate the amount of collision on each pair, and penalize only if the value is positive. For vi, Ty E X, yı = ((x; + 6, 9: = f(x;) +e the corresponding observations. and g(x,), 23 g(x,) are the corresponding latent space representations.
Figure 2: CoFLO schematic
Note that have introduced several hyper-parameters in the algorithm design; we will defer discussion on choice these parameters to Section 5.
2To obtain an initial latent space representation, the labels do not have be exact and could be collected from related task of cheaper cost cost.

---

penalty function Under review as a conference paper at ICLR 2021
We define the collision penalty
Pij max (Alys Uýن - -211
(1)
where À is penalty parameter that controls the smoothness of the target function h: 2 R.
4.3 DYNAMIC WEIGHT
Note that it is challenging universally eliminate the collision effect by minimizing the collision penalty and the GP's regression loss-this is particularly true with limited amount training data. Fortunately, in the optimization tasks it is often unnecessary to learn equally good representation for suboptimal regions. Therefore, we can dedicate more training resources to improve the learned latent space by focusing on the (potentially) near-optimal regions. Following this insight, we propose to use a weighted collision penalty function, which uses the objective values for each pair as importance weight in each iteration. Formally, for any pair ((z). 2j Vj), ) (I, 25 y:)) in a batch of observation pairs Dr {((mm: 3m, Vmm) (In; Rm; Vn m. we define the aapporaace-weigeeen as
ed(u)ta))
Pi
WiyWay
with
Wij
Σ
e3 +9a)
(2)
(m,n)eD,
Here the importance weight 7 is used to control the aggressiveness of the weighting strategy.
Combining the collision penalty and regression loss of GP, we define the pair loss function as L.p. Kr, Dc)
1
(GPk, y1)² + (GPk, (Mi(zz)) v,)² + PPtj SED,JED,
Σ
||D&|P
(3)
Here, (Pr denotes the Gaussian process's posterior mean on x; with kernel Ke and neural network M, at timestep I. denotes the regularization weight; as we demonstrate in Section 5, in practice we often choose to keep the penalty at a order close to the regression loss.
4.4 THEBRETICCL ANALYSIS
In this subsection, provide theoretical justification for the collision-free regularizer, by inspecting the effect of regularization on the regret bound of CoFLO. We first connect the proposed collision penalty in Equation to Lipschitz-continuity and then integrate it into the regret analysis to provide an improved regret bound.
Lipschitz continuity of the target function h The collision penalty encourages the Lipschitz- continuity for h. Formally, the proposed regularization promotes to learn latent space where Vr E D, z = g(xx), 2j = - E Z. 1g(x;) 9(x,) S -
The above inquality reduces to the Lipschitz-continiity for h. Unlike typical smoothness assumptions in GPs, function can be non-smooth and still Lipschitz continuous. Recently, Ahmed et al. (2019) leverage the Lipschitz-continuity of the objective function to propose improved acquisition functions based on the common acquísition functions, and provide an improved regret bound both theoretically and empirically. In the following, we show that running GP-UCB on the collision-fres latent space amounts to an improvement terms of its regret bound:
Theorem 1. Let Z C r] be compact and convex, E N,r > 0, 2 O. Suppose that the sample from GP and Lipschitz continuous with Lipschitz constant X. Let 8 E (0, 1), and define 3t = 2 log (2 /68) + 2d log (xrd().. Running the GP-UCB with 3e for sample h of GP with mean function zero and covariance function k(x,x'), we obtain a regret bound of O (Vdl (TTr) with high probability.
objective function defined on
Precisely, with C₁ = 8/ log(1 + -2-), we have
IP R S VCTBTTT + 21 5.
Here "yr is the maximum information gain after iterations, defined as 7 := max
T(yA;ha).
AC2.[A|=T

---

Under review as a conference paper at ICLR 2021
50
BO
P50
140
Simple regre!
SE
Simple regret
120
CORLD
LaFtD
OW CAFLD
100
DW CoftO
11-
80
60
300
125
200
300
200
30
Iterations
Iterations
(a) Rastrigin 2D
(b) Feynman 111.9.52
50
80
107
P50
PS0
RS
TPE
Simple regret
Simple regret
10
DW CaTLD
DW CoTLO
10a
01
20
60
150
40
100
Iterations
Iterations
(c) Supernova
(d) SPOKES
Figure 3: Experiment results on four pre-collected datasets. Rach experiment is repeated at least times.The colored area around the mean curve denotes the 3. Here & denotes the empirical standard deviation. n denotes the number of cases repeated in experiments. The hyper-parameten are set as the following. The retrain interval T are set to be 100 iterations for 8c 8a and 8d, 200 for 8b. The Regularization parameters p are set to be le for 8c 8a and 8d, 1e for 8b. The penalty parameter À are set to le-2, The weighting parameter y are set be 12- The prior mean Mo all set be O. The squared exponential kernel is used the GP covariance for all the four experiments. We also demonstrate the median curves in the Appendix.
Comparing the above result to Theorem 2 of Srinivas et al. (2010) which offers the regret bound with the sub-gaussianii assumption on the objective functions derivative, the second part of our regret bound does not rely on The coefficients are also smaller as the deterministic bound on derivative of ʃ avoids union bound.
The collision penalty encourages h to be Hipschitz-continunn on the latent space. Ideally, when the collision penalty Pt.j (X) term converges to zero for all data points in the latent space, we can claim that h is Lipschitz-contimun
Remark.
with Lipschitz constant at most X. Applying Theorem with 3. 2 log ( 2x2 /65) + 2d log (rrtt), we can reduce the regret bound by choosing smaller X. However, in practice, since the observation can be noisy, we need to choose a X big enough to tolerant the noise. small could make difficult to learn a meaningful representation.
5 EXPERIMENTS
In this section, we empirically evaluate our algorithm on two synthetic blackbox function optimization tasks and two real-world optimization problems.

---

Under review as a conference paper at ICLR 2021
5.1 EXPERIMENTAL SETUP
We consider four baselines in our experiments. The rudimentary random selection algorithm (RS) shows the task complexity. Three popular optimization algorithms, namely particle swarm optimization (PSO) (Miranda, 2018), Tree-structured Parzen Estimator Approach (TPE) (Bergstra al., 2011), and standard Bayesian optimization (BO) (Nogueira, 2014) which uses Gaussian process as the statistical model and the upper confidence bound (UCB) as its acquisition function, are tuned in each task. Another baseline we consider the sample-efficient LSO (SE LSO) algorithm, which is implemented based on the algorithm proposed by Tripp et al. (2020). We also compare the non-regularized latent space optimization (LSO), Collision-Free Latent Space Optimization (CoFLO) and the dynamically-weighen CoFLO (DW CoFLO) proposed in this paper. The performance for each task measured on 10,000 pre-collected data points.
One crucial problem in practice is tuning the syper-parameters. The hyper-parameterr parameters for GP tuned for periodically retraining in the optimization process, by minimizing the loss function on validation set. For all our tasks, choose simplistic neural network architecture M, due to limited and expensive access to labeled data under the BO setting. The coefficient p is, in general, selected to guarantee a similar order for the collision penalty GP loss. The À should be estimated according to the first several sampled data and tolerant the additive noise in the evaluation. controls the aggressiveness of the importance weight. While y should not be too close to zero (which is equivalent to uniform weight), an extremely high value could make the regularization overly biased. Such severe bias could possibly allow a heavily collided representation in most of the latent space and degrade regularization effectiveness. The value choice is similar to the inverse of the temperature parameter of softmax in deep learning Hinton al. (2015). Here we use the first batch of observed samples to estimate the order of all observations and choose the appropriate 7
5.2 DATASETS AND RESULTS
We now evaluate CoFLO on two synthetic datasets and two real-world datasets. In the experiments, all the input data points are mapped to one-dimensional latent space by via the neural network. We demonstrated the improvement of CoFL. brought by the explicit collision mitigation in the lower- dimensional latent space in terms of average simple regret. We also include the median result and statistical test in the appendix.
2D RostRasiiigin The Rastrigin function is a non-convex function used as a performance test problem for optimization algorithms. It was first proposed by RASTRIGIN (1974) and used as a popular benchmark dataset for evaluating Gaussian process regression algorithms (Cully et al., 2018). Formally, the 2D Rastrigin function is
f(x) 10d Σ ot 10cos(2 d=2
For convenience of comparison, we take the -f(x) as the objective value to make the optimization tasks a maximization task. The neural network is pretrained on 100 data points. As illustrated by figure 8a. CoFLO and DW CoFLO could quickly reach the (near-) optimal region, while the baselines generally suffer a bigger simple regret even after an excessive number of iterations.
Feynman 111.9.52 Equation Growing datasets have motivated pure data-backed analysis in physics. The dataset of 100 equations from the Feynman Lecture on Physics for the symbolic regression tasks in physics (Udrescu and Tegmark, 2020) could play the role as test set for data- back analysis algorithms in physics. The 111.9.52 we choose to test the optimization algorithms is
EaE,t C.EEftsi(((es) sin((w wp) P. h/27 (nwwwa)t/2)
The equations have 6 variables as inputs and are reported to require at least 10 data for the regression task. The neural network is randomly initialized at the beginning. As illustrated by figure 8b, in the first 100 iterations, CoFLO and DW CoFLO behaves similarly to random selection. After the first training at iteration 100, CoFLO and DW CoFLO approach the optimum at much faster pace compared to the baselines; among them, DW CoFLO shows a faster reduction in simple regret.
8

---

Under review as a conference paper at ICLR 2021
Supernova Our first real-world task is to perform maximum likelihood inference on cosmological parameters, the Hubble constant Hy E (60, 80), the dark matter fraction 9M E (0, and the dark energy fraction 42 (0, 1). The likelihood is given by the Roberson -allerr metric, which requires a one-dimensional numerical integration for each point in the dataset from Davis et al. (2007). The neural network is pretrained on hundred data points. As illustrated by figure 8c, the simple regret SE LSO has a faster drop at the beginning, while later remained relatively stable and eventually ends a similar level LSO. These results demonstrate the efficiency of SE LSO when finding sub-optimal. However, without collision reduction, SE LSO couldn't 't outperform the LSO in the long rum, where both reach their limitation. And the CoFLO and DW CoFLO demonstrate its robustness when close to the optimal as both constantly approach the optimal. Among them, DW CoFLO slightly outperform CoFL..
Redshift Distribution The challenges in designing and optimizing cosmological experiments grow commensurately with their scale and complexity. Careful accounting of all the requirements and features of these experiments becomes increasingly necessary achieve the goals of given cosmic survey. SPOKES (SPectrOscopic KEn Simulation) is an end-to-end framework that can simulate all the operations and key decisions of cosmic survey (Nord al., 2016). It can be used for the design, optimization, and forecasting of any cosmic experiment. For example, some cosmic survey campaigns endeavor observe populations of galaxies that exist a specific range of redshifts (distances) from us. In this work, we use SPOKES to generate galaxies within specified window of distances from Earth. We then minimize the Hausdorff distance between the desired redshift distribution and the simulation of specific cosmological surveys generated by SPOKES.
In our experiments, the neural network pretrained on 200 data points. As illustrated by figure 8d, the simple regret of SE LSO drops faster at the initial phase. However, when it gets close to the (near-) optimal region where simple regret is approximately 0,15, caught up by both CoFLO and DW CoFLO, and eventually gets slightly outperformed. Such result indicates that the collision problem could have more impact when the algorithm gets close to the optimal region. Notice that the rudimentary BO eventually outperformed the non-regularized LSO, indicate that without mitigation of collision, the learned representation could worsen the performance in the later stage when the algorithm gets close to the optimal. In conclusion, the mitigation of collision like CoFLO is necessary to further improve the later performance of LSO, when collision matters more in the near-optimal areas.
5.3 DISCUSSION
In general, our experimental results consistently demonstrate the robustness of our methods against collisions in the learned latent space. Our method outperforms all baselines; when compared to the sample-efficient LSO, the dynamically-weige steady capability to reach the optimum by explicitly mitigating the collision in the latent space. In contrast, the Sample-efficient LSO might fail due to the collision problem.
LSO performs better in most cases and shows
CONCLUSION
We have proposed novel regularization scheme for latent space based Bayesian optimization. Our algorithm-namely CoFLO-addreesse the collision problem induced by dimensionality reduction, and improves the performance for latent space-based optimization algorithms. The regularization is proved to be effective in mitigating the collision problem in learned latent space, and therefore can boost the performance of the Bayesian optimization in the latent space. We demonstrate strong empirical results for CoFLO on several synthetic and real-world datasets, and show that CoFLO capable of dealing with high-dimensional input that could be highly valuable for real-world experiment design tasks such as cosmological survey scheduling.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pages 39-1, 2012.
M. O. Ahmed, Sharan Vaswani, and M. Schmidt. Combining bayesian optimization and lipschitz optimization. Machine Learning. 109:79-102, 2019.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for parammte optimization. In J. Shawe-Taylon Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 24, pages 2546-2554, Curran Associates, Inc., 2011.
A. Cully, K. Chatzilygeroudis, F. Allocati, and J.-B. Mouret. Limbo: A Flexible High-performance Library for Gaussian Processes modeling and Data-Efficient Optimization. The Journal of Open Source Software, 3(26):545, 2018. doi:
(05//oss..0054
M. Davis, E. Mortsell, Sollerman, A. C. Becker, Blondin, Challis, A. Clocchiatti, A. V. Filippenko, R. J. Foley, P. M. Garnavich, S. Jha, K. Krisciunas, R. P. Kirshner, B. Leibundgut, W. Li, T. Matheson, G. Miknaitis, G. Pignata, A. Rest, A. G. Riess, B. Schmidt, R. C. Smith, J. Spyromilio,, C. W. Stubbs, N. B. Suntzeff, J. L. Tonry, W. M. Wood- Vasey, and Wood-Vasey,
A. Zenteno. Scrutinizing exotic cosmological models using ESSENCE supernova data combined with other cosmological probes. The Astrophysical Journal, 666(2):716-7))) sep 2007. doi: 00.1086/519988 URL https::t////. /doi.iioo/ http:///ooo.oorr1000000000011110
Josip Djolonga, Andreas Krause, and V. Cevher. High-dimensional gaussian process bandits. In NIPS, 2013.
Geoffrey H. Hinton, Oriol Vinyals, J. Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015.
Wenbing Huang, Deli Zhao, Fuchun Sun, Huaping Liu, and Edward Chang. Scalable gaussian process regression using deep neural networks. In Proceedings of the 24th International Conference on Artificial Intelligence, IICAI' pages 3576 - 3582. AAAI Press, 2015. ISBN 9781577357384.
Xiaoyu Lu, Javier Gonzalez, Zhenwen Dai, and Neil Lawrence. Structured variationally auto- encoded optimization. volume 80 of Proceedings of Machine Learning Research, pages 3267- 3275, Stockholm International Fairs, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL
http:/ http:///ooeeeeengs/eeee
html.
Lester James V. Miranda. PySwarms, a research-toolkit for Particle Swarm Optimization in Python. Journal of Open Source Software, 3, 2018. doi: 10.21105/joss004433. URL
https:: httss:/ddoi.rg//// 20..0..111///ooo 00433. 00433
Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014. URL //titt/u///ttbbbbbb com/ fmfn/ BBayes anOpt imi zaticn
B. Nord, A. Amara, A. Réfrégier, La. Gamper, Lu. Gamper, B. Hambrecht, C. Chang. J.F. Forero-Romero, S. Serrano, C. Cunha, O. Coles, A. Nicola, M. Busha, A. Bauer, w. Saunders, S. Jouvel, Kirk. and Wechsler. Spokes: An end-to-end simulation facility for spectroscopic cosmological surveys. Astronomy and Computing, 15:1 15. 2016. ISSN 2213-1337. doi: http://dbboogg//..oot/accc..oopp attp::/wwe scwencedireecee com/ https///ss.aeeeeeerreeeee/////// 6300000
URL
C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006. URL https: : /bit. http::///.bt.t....ttt..... ly/2ttpBxx.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
L. A. RASTRIGIN. Systems of extremal control.
Nauka, 1974.
URL
https: https::/ei..o wtt.//iiiiiiiiiiim.miim.... i.c.. wttp..a.......//////pppppppppp/e jp/naid/1001 3403158/em
10

---

Under review as a conference paper at ICLR 2021
B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148 175, 2016.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, ICML' 15, pages 2171 2180. JMLR.org, 2015.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference International Conference Machine Learning, ICML'10, pages 1015-1022, Madison, WI, USA, 2010. Omnipress. ISBN 978 9781605589077.
WILLIAM R THOMPSON. ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES. Biometrika, 25(3-4):285-294, 12 1933. ISSN 0006-3444. doi: 1.1003/biome//55344228... 3-........8..
URL
attps:///ol oariioorg/oo/ /10. 1093
Austin Tripp, Erik Daxherger, and José Miguel Hernánde7-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. ArXiv, abs/2006.09191 2020.
Silviu-Marial Udrescu and Max Tegmark. Al Feynman: A physics-inspired method for symbolic regression. Science Advances, 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. volume 51 of Proceedings of Machine Learning Research, pages 370-378, Cadiz, Spain, 09-11 May 2016. PMLR.
A REGRET BOUND FOR A LIPSCHITZ-COTTINUUU OBJECTIVE FUNCTION
In this section, we provide the detailed proof for Theorem We first modify Lemma 5.7 and Lemma 5.8 in Srinivas et al. (2010) since are assuming the deterministic Lipschitz-continuity for h. Use the same analysis tool Z. defined as a set of discretization Z, C Z where Z, will be used at time in the analysis.
We choose a discretization Z, of size (T.) so that V2 E Z,
|| [ælella S rd/T:
(4)
where [=], denotes the closest point in Z, 2. Lemma 1. Pick 8 (0, 1) and set 3 2log( ne. 2dllg (Lrdtt)) where E121 Te. 1, n. 0.
Let T. = Lrdt?. Hence then
[h(=") - Pt1 ([= S 3: /3 ( 1/12 V1
holds with probability 2 1 - 5. Here 2" = g(x²).
Proof. Using the Lipschitz-cootiinity and equation 4, we have that V2 E Z, [h(2) h([2)o) s Lrd/Te
By choosing 7: Lrdi², we have |z|| (Irdi2)d and
V2 E Z, [h(z) h<[2],)) < 1/t?
Then using Lemma 5.6 in Srinivas et al. (2010), we reach the expected result.
Base Lemma 5.5 in Srinivas et al. (2010) and Lemma 1, we could have the following result directly.
Lemma 2. Pick d (0, 1) and set 3 2log(2m//8) + 2dlog(Lrdt?), where Σez1 Te
>
0. Then with probability 1 for all t E N, the regret is bounded as follows:
rt 2/3//2 1 1/12
11

---

Under review as a conference paper at ICLR 2021
Proof. Using the union bound of 8/2 in both Lemma 5.5 in Srinivas et al. (2010) and Lemma have that with probability 1 - 8:
r: = h(==)=-----)))) 1((2))
$ 0 01-1(21) 1/t²² + (1-1(z)) h(z))
- 20 0 + 1/t²
which complete the proof.
Now we are ready use the Lemma5.4 in Srinivas al. (2010) and Lemma complete the proof of Theorem 1.
Proof. Using Lemma5.4 Srinivas et al. (2010), we have that with probability 2 1 -
43,07-1 s C13TTT V
By Cauchy-Schward Schwarz:
Σ 20/ 0t 0t- (x+t) S ViirrT V
Finally, substitute with 2²t (since Σ 1/f = x2/6). Theorem follows.
12

---

Under review as a conference paper at ICLR 2021
B VISUALIZATION OF THE COLLISION EFFECT IN LATENT SPACE
Non-regulorized
150
300
Olcarved Optimam
Oprmam
30
Labant
Collision Ratio
0.50070
0.50015
0.00003
Lurdy m
-0.6
-02
60
52
04
0.6
Lakent Velue
Hean space
25000
Lantoda Mean
12000
Figure 4: Illustrate the collision and quantified measurement of the collision. Here propose two quantity measurement of the collision. For the second graph the y axis the is the ratio exceeding [y1 V/2 > 121 22|= L. And for the third graph, the axis of the third column is the mean of 91 V2l /2//121 22|-
We demonstrate the collision effect in the latent space. We trained the same neural network on Feynman dataset with 101 data points which demonstrate the latent space after two retrains with the retrain interval set to be 50 data points. The regularized one employed DW CoFLO, with the regularization parameter p = 1e², penalty parameter = 1e-2, retrain interval Т, weighting parameter 7 le and the base kernel set be square exponential kernel. The non-regularized one employed LSO.
C SUPPLEMENTAL MATERIALS ON ALGORITHMIC DETAILS
C.1 ALGORITHMIC DETAILS ON NEURAL NETWORK ARCHITECTURE
As the main goal of our paper was to showcase the performance of novel collision-free regularizer, we picked our network architectures to be basic multi-layer dense neural network:
For SPOKES, we used a 5-layer dense neural network. Its hidden layers consist of neurons with Leaky Relu nonlinearities, 8 neurons with Sigmoid nonlincarities, 4 neurons with Sigmoid nonlincarities, and 2 neurons with Sigmoid nonlinearities respectively. Each hidden layer also applies 0.2 dropout rate. The output layer applies Leaky Relu nonlinearity.
For SuperNova, Feynman, and Rastrigin 2D, we used a 4-layer dense neural network. Its hidden layers consist of 8 neurons with Sigmoid nonlinearities, 4 neurons with Leaky Relu nonlinearities, and neurons with Leaky Relu nonlincarities respectively. Each hidden layer applies a 0.2 dropout rate. The output layer also applies Leaky Relu nonlinearity.
13

---

Under review as a conference paper at ICLR 2021
Carter hrom
Caline
9-
6.5 0.35
2.4
675
(a) 1-D regularized latent space
(b) I-D non-regularized latent space
Figure 5: Illustrate the 1-D latent space of Feynman III.9.52 dataset. 5a shows regularized latent space with few observable collisions. 5b shows a non-regularized latent space with bumps of collisions especially around the maxima among collisions in the latent space contribute to the optimization through improving the learned Gaussian process. We observed in this comparison that the next point selected by the acquisition function of the regularized version is approaching the global optima, while the next point in the non-regularized version is trying to solve the uncertainty brought by the severe collision near the currently observed maxima.
observed data points. Besides, having fewer
The neural networks are trained using ADAM with learning rate of 1e
1 hidden layer
L hidden layer
input layer
output layer
(L)
.50
(L.
vi
(1)
21
y1
(L+1)
Yo
TD
(1) M2
Figure 6: Network graph of (L + 1y-layer dense network with input units and 1 output units. In our experiments L is set to be 4 for Rastrigin 2D, Feynman II.9.52, Supernova, and 5 for SPOKES.
C.2 PARAMETER CHOICES
We further investigate the robustness of parameter choices of both the regularization parameter and the penalty parameter on SPOKES dataset. We show the result in the figures below.
14

---

Under review as a conference paper at ICLR 2021
1-0 01
033
10000.0
A=0
025
1000003
100
60 80
300
(a) Performance different X
(b) Performance of different p
Figure 7: Simple regrets under different parameter settings the SPOKES dataset. 7b shows that regularization parameter too big could distract the training process and downgrade the performance. And we choose p leb in practice regression loss of GP in equation 3. 7a shows that a relatively small value could do good job. We believe that's because the wide range of objective values of the tested dataset needs be mitigated. The curves demonstrate the decent performance of CoFLO as long as the parameters are not set to be too large.
it maintains the collision penalty in the same order of the
D ADDITIONAL EXPERIMENTAL RESULTS
We added both the detailed median curves and the p-values of the Welch's t-tests of the experiments discussed in section
D.1
MEDIAN CURVE
The median curves demonstrate similar trends to the mean curves. In the four experiments, DW CoFLO consistently demonstrates a superior performance over the baselines.
Simple reghet
Simple regret
16
Simple ragee
201D
25
Iterations
Iarattonns
Iterations
Iterations
(a) Rastrigin 2D
(b) Feynman 111.9.52
(c) Supernova
(d) SPOKES
D.2 P-VALUES
The table shows the p-values of Welch's t-tests of the experiments. demonstrates the significance of the improvement brought by DW CoFLO over the baselines.
Data
BO
RO
TPE
LSO
SE-LSO
CoFLO
Rastrigin-2D
1.07e-4
3.88e-b
1.0le
6.38e
-.10
4.23e
Supernova
3.24e
3.61e
3.18c-2 2
3.43c-
-
2.62e
Feynman
1.73e-1
1.52e-0P
8.20e
288
6.37e-1
2.25c
SPOKES
4.62e
9.90e
2.64e
-
2.87e 3
-1
15


Reviews:
[{"review_id": "z03CteKbat", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes (1) a new regularization strategy for the latent space-based BO, (2) an optimization aware dynamic weighting for adjusting the collison penalty to improve BO, (3) theoretical analysis for the BO on the latent space. The idea for the regularization is to take in pairs of data points and penalizes those too close in the latent space compared to their target space distance\n\nThe paper makes an interesting observation that the learned representation (for BO to deal with complex object or high-dimension) often leads to collision in the latent space: two points with significant different observations get too close in the learned latent space. Collisions could be regarded as additional noise introduced by the traditional neural network, leading to degraded optimization performance\n\nThe mapping by neural network to learn g: D->Z is typically considered as the regression problem in which the neural network should learn the property that similar input should have similar output.\n\nA pair loss is integrated in learning the neural network. The dynamic weight improves the learned latent space by focusing on the potential high-value region.\n\nThe idea of using constraint in the latent space has also been studied in [1]. \n\nDespite of the good motivation, the paper execution is not yet demonstrated the effectiveness of the proposed approach for three main reasons:\n\n(1) The experiments using 4 settings are quite simple and havenot yet satistisfactorily convinced why the proposed approach performs intuitively better. It can be improved further by demonstrating the collison-effect in more challenging task, such as automatic chemical design [2].\n\n(2) The theoretical analysis follows and extends from Srinivas et al 2010.\n\n(3) Fig 1 demonstrates the collision in 1d using the non-regularized latent space? It will be useful if you can add another figure in the same setting using the regularized latent space.\n\nThe writing and presentation can be improved more.\n\n\nTypo: \nSection 5.1: “promotse\"\nRemark: why “Choosing” is capitalized in the middle of the sentence?\n“UCB use the upper…” => “UCB uses the upper….”\n\n[1] Kusner, M. J., Paige, B., & Hernández-Lobato, J. M. (2017, June). Grammar Variational Autoencoder. In Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017 (Vol. 70, pp. 1945-1954). ACM.\n\n[2] Griffiths, Ryan-Rhys, and José Miguel Hernández-Lobato. \"Constrained Bayesian optimization for automatic chemical design using variational autoencoders.\" Chemical science 11.2 (2020): 577-586.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nvdK58LFQA8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to avoid collision for the latent space based Bayesian optimization method. The main idea is to add a regularization term into the training process. Theoretical analysis is also conducted to understand the performance of the proposed method. \n\nAlthough the idea of the proposed method is somewhat interesting, I do have many concerns for the paper. \n\n1) The writing is not good, so it makes it hard to understand the work. In particular, the English of the paper is frequently bad (wrong grammar, typos, unfinished sentences). The maths notations are occasionally not consistent. For example sometimes, the penalty is defined as p_[i,j}, sometimes it is denoted as p_{ij}.\n2) Section 4.2 is too ambiguous. What are z_i, z_j in the equation in Section 4.2? Based on the notation of the latent space Z, I can guess z_i, z_j are the values in the latent space, but this should be clearly mentioned in the paper. Also, what does \\lambda represent? And how to set it in practice? I went through the 2nd paragraph in Section 6.2 and still feel unclear how to set this hyperparameter in practice.\n3) Section 4.3 is also not clear. What is the intuition behind the weight \\omega_{ij}? What do \\gamma and \\rho represent? How to set them? And what does GP_{Kt}(M_t(x_i)) (in Eq. (1)) denote? \n4) Regarding the theoretical analysis, unless I miss something, it is just the standard theorem as in Srinivas et al. (2010), but replace the assumption of the objective function f being a sample path from the GP, by the assumption of the latent space function h being a sample path from the GP? In which cases this assumption is satisfied? And what does it mean by “comparing to Theorem 2 in Srinivas et al. (2010), the second part of the regret bound doesn’t rely on \\delta\"? As much as I understand, the regret bound in Theorem 1 is the same as the one in Theorem 2 in Srinivas et al. (2010).\n5) Regarding the experiments, the experiments are only conducted on low-dimensional problems (2D, 6D, 3D, …), which is contradict with the motivation of the work (BO for high dimensional inputs). Besides, what does it mean when the neural networks are pretrained on a number of data points? Do we know the corresponding function values of these data points in advance? If yes, for the baseline methods the paper compares with, are these data points employed in these baseline optimization procedures?", "extra_scores": {}, "extra_text": {}}, {"review_id": "KngE3GoSDF3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a regularization technique in training a latent variable model so that points with different functions are pushed apart. It’s demonstrated that the proposed technique can boost regret bound and empirical performance. \n\nOverall, I think it’s a nice paper, but I don’t think the current presentation is good enough for publication at ICLR. \n\ncomments & questions:\n\n1. It’s a natural idea to add a Lipchitz-like regularization loss to mitigate “collision”.  The theoretical result seems a straightforward derivative of the Srinivas et al. (2010), but I don’t really see the novelty of the theoretical result, since Lipchitz continuity is implicitly determined by the kernel function?  \n\n2. the proposed method needs to pretrain the neural network with 100 or 200 points. It’s not clear to me what it means by “pre-train”. Is it supervised or unsupervised? Which 100 points are chosen for pretraining? if it’s supervised, did you count them in the optimization budget? that means if you pre-train on 100 labeled points, then perform 100 BO iterations, a fair comparison to standard BO would grant it a budget of 200 function evaluations. \n\n3. there are several parameters, such as $\\lambda$, $\\gamma$, how are they chosen exactly? how sensitive are these parameters? \nWhat exactly is the “standard BO” algorithm from Nogueira (2014)? Is it UCB? EI? \n\n4. Seems all the benchmark functions have continuous domain with already low dimensions, e.g., the Rastrigin 2D only has 2 dimensions. Do you further reduce the dimension to one with the neural network? It would be great if you could plot the function on latent space. Same for other benchmarks, since they are not very high dimensional. \n\n5. From the experiments, I don’t really see if it’s true that the baselines lose because they have collision problems. Is it possible to design some experiment to demonstrate that?\n\n6. To me seems the work could be more motivated by input domains such as graphs or other discrete structures, at least for the benchmarks in the experiments I don’t see why they need this method despite the claimed superior performance. For your reference, some notable work on Bayesian optimization in latent space for discrete objects:   \n* Kusner et al. (ICML 2017), grammar VAE\n* Jin et al. (ICML 2018), JT-VAE\n* Zhang et al. (NeurIPS 2019), D-VAE\n* ...\n\nMinor:\nthe formula for posterior GP mean and covariance assumes zero prior mean, which was not explicitly pointed out. \nin 3.1, most popular acquisition should definitely include expected improvement \n\nthere are many typos:\nin Abstract: significant different -> significantly different \nin Related Work: taskss -> tasks\nin Related Work third paragraph: smooth(of… -> add space (and many other places)\nin 3.1: ”wiggles” first quote wrong direction \nIn 3.1: “the acquisition function $\\alpha$ … use it -> an acquisition function … uses it\nin 3.1: “then use the sample as the acquisition function …, need to add period \nin 4.1: base on  -> based on \nin 5.1: “promotse” -> promotes? \n...\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7WoqVjMwpjS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n# summary\n\nThis paper proposed a method that can penalize collisions in latent space. To\nbe more concrete, for a model which combines a neural network and a Gaussian\nProcess, e.g. deep kernel learning, the learned latent features for two\ndifferent inputs can be very close. In the following GP modeling, these two\nsimilar latent features will cause difficulties since the covariance between\nthem will be large although these two inputs are quite different.\n\n\n# cons\n\n1.  The general idea presented in this work is very interesting. This is also a\n    very realistic treatment, i.e. collisions in the latent space.\n2.  Although not a new technique, e.g. siamese network, triplet loss, I like\n    the idea of penalizing close points in latent space combined with a GP. It\n    implicitly incorporate prior knowledge in modeling the GP, which usually\n    boost the performance of a GP.\n\n\n# pros\n\n1.  I am doubtful about the correctness of eq(1). Without a treatment of\n    stochastic variational inference, the marginal likelihood of GP cannot be\n    factorized into a product of per data points. This means the batched update\n    of GP in eq(1) will not produce a correct GP model, if I understand eq(1)\n    correctly. Can the authors explain this batched update?\n2.  I think experimental results should be extended to include a comparison\n    with SMAC and TPE, which are two strong baselines. Although this work focus\n    on GP based BO, empirical results of SMAC and TPE **without** considering\n    collisions will make this work more convincing.\n3.  The experimental settings used in this work are not detailed, e.g. how many\n    units in each layer in the neural network, etc. Empirical results are also not sufficient. \n4. In Figure3, the line is the mean curve instead of median of at least 10 experiments. However, without a statistical test, it is hard to tell whether the proposed approach is better than other competing methods.\n\n\n# questions\n\n1.  It is not clear to me why the retrain interval $\\tilde{T}$ is set to be 100\n    for 3c, 3a and 3d in Figure 3. In Algorithm 1, the latent model is updated\n    every $\\tilde{T}$ iterations. In Figure 3, the total iteration numbers for\n    3c, 3a and 3d are 100, 200 and 100 respectively. This means for 3c and 3d,\n    the latent model is updated only once and this update happens at the end of\n    BO. After the update, the model will never be used. Can the authors comment\n    on this?\n\nOverall speaking, I am afraid this paper doesn't contain necessary details and\nthe theoretical results are not strong enough.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers liked the overall idea presented in this paper. Although the idea as well as relevant tooling for incorporating constraints in the latent space has been studied a lot in the past, the authors differentiate their work by applying it in a new interesting problem. At the same time, some confusions about relation to prior work remain after rebuttal. Firstly, the theoretical additions to prior work (Srinivas et al. 2010) are still unclear in terms of significance - they feel more like observations made on top of an existing theorem rather than fresh significant insights. Furthermore, even if prior work has not considered exactly the same set-up, it would still be needed to understand what the performance would be when considering prior models or prior datasets used in similar domains (e.g. suggestions by R2, R3). The latter would be desirable especially since the experimental set-up used in this paper is deemed by the reviewers too simple (while the motivation of the paper is to solve an issue essentially manifesting in complex scenarios).
Decision: Reject