Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 

# summary

This paper proposed a method that can penalize collisions in latent space. To
be more concrete, for a model which combines a neural network and a Gaussian
Process, e.g. deep kernel learning, the learned latent features for two
different inputs can be very close. In the following GP modeling, these two
similar latent features will cause difficulties since the covariance between
them will be large although these two inputs are quite different.


# cons

1.  The general idea presented in this work is very interesting. This is also a
    very realistic treatment, i.e. collisions in the latent space.
2.  Although not a new technique, e.g. siamese network, triplet loss, I like
    the idea of penalizing close points in latent space combined with a GP. It
    implicitly incorporate prior knowledge in modeling the GP, which usually
    boost the performance of a GP.


# pros

1.  I am doubtful about the correctness of eq(1). Without a treatment of
    stochastic variational inference, the marginal likelihood of GP cannot be
    factorized into a product of per data points. This means the batched update
    of GP in eq(1) will not produce a correct GP model, if I understand eq(1)
    correctly. Can the authors explain this batched update?
2.  I think experimental results should be extended to include a comparison
    with SMAC and TPE, which are two strong baselines. Although this work focus
    on GP based BO, empirical results of SMAC and TPE **without** considering
    collisions will make this work more convincing.
3.  The experimental settings used in this work are not detailed, e.g. how many
    units in each layer in the neural network, etc. Empirical results are also not sufficient. 
4. In Figure3, the line is the mean curve instead of median of at least 10 experiments. However, without a statistical test, it is hard to tell whether the proposed approach is better than other competing methods.


# questions

1.  It is not clear to me why the retrain interval $\tilde{T}$ is set to be 100
    for 3c, 3a and 3d in Figure 3. In Algorithm 1, the latent model is updated
    every $\tilde{T}$ iterations. In Figure 3, the total iteration numbers for
    3c, 3a and 3d are 100, 200 and 100 respectively. This means for 3c and 3d,
    the latent model is updated only once and this update happens at the end of
    BO. After the update, the model will never be used. Can the authors comment
    on this?

Overall speaking, I am afraid this paper doesn't contain necessary details and
the theoretical results are not strong enough.


review_title: Official Blind Review #1

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a regularization technique in training a latent variable model so that points with different functions are pushed apart. It’s demonstrated that the proposed technique can boost regret bound and empirical performance. 

Overall, I think it’s a nice paper, but I don’t think the current presentation is good enough for publication at ICLR. 

comments & questions:

1. It’s a natural idea to add a Lipchitz-like regularization loss to mitigate “collision”.  The theoretical result seems a straightforward derivative of the Srinivas et al. (2010), but I don’t really see the novelty of the theoretical result, since Lipchitz continuity is implicitly determined by the kernel function?  

2. the proposed method needs to pretrain the neural network with 100 or 200 points. It’s not clear to me what it means by “pre-train”. Is it supervised or unsupervised? Which 100 points are chosen for pretraining? if it’s supervised, did you count them in the optimization budget? that means if you pre-train on 100 labeled points, then perform 100 BO iterations, a fair comparison to standard BO would grant it a budget of 200 function evaluations. 

3. there are several parameters, such as $\lambda$, $\gamma$, how are they chosen exactly? how sensitive are these parameters? 
What exactly is the “standard BO” algorithm from Nogueira (2014)? Is it UCB? EI? 

4. Seems all the benchmark functions have continuous domain with already low dimensions, e.g., the Rastrigin 2D only has 2 dimensions. Do you further reduce the dimension to one with the neural network? It would be great if you could plot the function on latent space. Same for other benchmarks, since they are not very high dimensional. 

5. From the experiments, I don’t really see if it’s true that the baselines lose because they have collision problems. Is it possible to design some experiment to demonstrate that?

6. To me seems the work could be more motivated by input domains such as graphs or other discrete structures, at least for the benchmarks in the experiments I don’t see why they need this method despite the claimed superior performance. For your reference, some notable work on Bayesian optimization in latent space for discrete objects:   
* Kusner et al. (ICML 2017), grammar VAE
* Jin et al. (ICML 2018), JT-VAE
* Zhang et al. (NeurIPS 2019), D-VAE
* ...

Minor:
the formula for posterior GP mean and covariance assumes zero prior mean, which was not explicitly pointed out. 
in 3.1, most popular acquisition should definitely include expected improvement 

there are many typos:
in Abstract: significant different -> significantly different 
in Related Work: taskss -> tasks
in Related Work third paragraph: smooth(of… -> add space (and many other places)
in 3.1: ”wiggles” first quote wrong direction 
In 3.1: “the acquisition function $\alpha$ … use it -> an acquisition function … uses it
in 3.1: “then use the sample as the acquisition function …, need to add period 
in 4.1: base on  -> based on 
in 5.1: “promotse” -> promotes? 
...

review_title: could be a nice paper if applied on the right problems and demonstrated clearly

final_score: rating: 3: Clear rejection
review_content: The paper proposes a method to avoid collision for the latent space based Bayesian optimization method. The main idea is to add a regularization term into the training process. Theoretical analysis is also conducted to understand the performance of the proposed method. 

Although the idea of the proposed method is somewhat interesting, I do have many concerns for the paper. 

1) The writing is not good, so it makes it hard to understand the work. In particular, the English of the paper is frequently bad (wrong grammar, typos, unfinished sentences). The maths notations are occasionally not consistent. For example sometimes, the penalty is defined as p_[i,j}, sometimes it is denoted as p_{ij}.
2) Section 4.2 is too ambiguous. What are z_i, z_j in the equation in Section 4.2? Based on the notation of the latent space Z, I can guess z_i, z_j are the values in the latent space, but this should be clearly mentioned in the paper. Also, what does \lambda represent? And how to set it in practice? I went through the 2nd paragraph in Section 6.2 and still feel unclear how to set this hyperparameter in practice.
3) Section 4.3 is also not clear. What is the intuition behind the weight \omega_{ij}? What do \gamma and \rho represent? How to set them? And what does GP_{Kt}(M_t(x_i)) (in Eq. (1)) denote? 
4) Regarding the theoretical analysis, unless I miss something, it is just the standard theorem as in Srinivas et al. (2010), but replace the assumption of the objective function f being a sample path from the GP, by the assumption of the latent space function h being a sample path from the GP? In which cases this assumption is satisfied? And what does it mean by “comparing to Theorem 2 in Srinivas et al. (2010), the second part of the regret bound doesn’t rely on \delta"? As much as I understand, the regret bound in Theorem 1 is the same as the one in Theorem 2 in Srinivas et al. (2010).
5) Regarding the experiments, the experiments are only conducted on low-dimensional problems (2D, 6D, 3D, …), which is contradict with the motivation of the work (BO for high dimensional inputs). Besides, what does it mean when the neural networks are pretrained on a number of data points? Do we know the corresponding function values of these data points in advance? If yes, for the baseline methods the paper compares with, are these data points employed in these baseline optimization procedures?
review_title: The paper has some contributions, however, it is not ready for publish yet.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes (1) a new regularization strategy for the latent space-based BO, (2) an optimization aware dynamic weighting for adjusting the collison penalty to improve BO, (3) theoretical analysis for the BO on the latent space. The idea for the regularization is to take in pairs of data points and penalizes those too close in the latent space compared to their target space distance

The paper makes an interesting observation that the learned representation (for BO to deal with complex object or high-dimension) often leads to collision in the latent space: two points with significant different observations get too close in the learned latent space. Collisions could be regarded as additional noise introduced by the traditional neural network, leading to degraded optimization performance

The mapping by neural network to learn g: D->Z is typically considered as the regression problem in which the neural network should learn the property that similar input should have similar output.

A pair loss is integrated in learning the neural network. The dynamic weight improves the learned latent space by focusing on the potential high-value region.

The idea of using constraint in the latent space has also been studied in [1]. 

Despite of the good motivation, the paper execution is not yet demonstrated the effectiveness of the proposed approach for three main reasons:

(1) The experiments using 4 settings are quite simple and havenot yet satistisfactorily convinced why the proposed approach performs intuitively better. It can be improved further by demonstrating the collison-effect in more challenging task, such as automatic chemical design [2].

(2) The theoretical analysis follows and extends from Srinivas et al 2010.

(3) Fig 1 demonstrates the collision in 1d using the non-regularized latent space? It will be useful if you can add another figure in the same setting using the regularized latent space.

The writing and presentation can be improved more.


Typo: 
Section 5.1: “promotse"
Remark: why “Choosing” is capitalized in the middle of the sentence?
“UCB use the upper…” => “UCB uses the upper….”

[1] Kusner, M. J., Paige, B., & Hernández-Lobato, J. M. (2017, June). Grammar Variational Autoencoder. In Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017 (Vol. 70, pp. 1945-1954). ACM.

[2] Griffiths, Ryan-Rhys, and José Miguel Hernández-Lobato. "Constrained Bayesian optimization for automatic chemical design using variational autoencoders." Chemical science 11.2 (2020): 577-586.

review_title: Official Blind Review #2

Meta-review: The reviewers liked the overall idea presented in this paper. Although the idea as well as relevant tooling for incorporating constraints in the latent space has been studied a lot in the past, the authors differentiate their work by applying it in a new interesting problem. At the same time, some confusions about relation to prior work remain after rebuttal. Firstly, the theoretical additions to prior work (Srinivas et al. 2010) are still unclear in terms of significance - they feel more like observations made on top of an existing theorem rather than fresh significant insights. Furthermore, even if prior work has not considered exactly the same set-up, it would still be needed to understand what the performance would be when considering prior models or prior datasets used in similar domains (e.g. suggestions by R2, R3). The latter would be desirable especially since the experimental set-up used in this paper is deemed by the reviewers too simple (while the motivation of the paper is to solve an issue essentially manifesting in complex scenarios).
Decision: Reject