Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a fused autoencoder/VGP approach which unlocks substantial improvements in Bayesian optimization on high dimensional spaces such as molecule graphs.
strengths_and_weaknesses: Originality
The paper brings together a few lines of work: Most importantly, it fuses trust region bayesopt (TuRBO)+sparse/variational GP with learned index point representations (via VAE), reconciling the trainable representation learning with the trust region approach by jointly training with a semisupervised objective. Secondly, the paper incorporates representation improvements (JTVAE=>SELFIES) which provide substantial posterior/eval speedups and higher chances of emitting valid representations in the discrete molecule space. That said, because "all it does" is bring these lines of work together, this work is more about unlocking performance as opposed to proposing a radically original idea -- it's somewhat incremental to TuRBO, to VGP, to SELFIES.

Quality
Well written, problem well framed. Experiments against multiple Guacamol targets + DRD3 as well as Expressions dataset go beyond toy problem to address interesting industry benchmarks. Ablation study is helpful for disentangling the contributions of joint training (LOL-BO) vs representation change (SELFIES). It is evident that both are contributing substantially. It would be nice to have "best score by iterations" type metrics for JTVAE vs SELFIES. Is this an issue of computation budget (100x more expensive JTVAE prohibitive for bayesopt)?.
L310 spelling SELFIS

Clarity
Problem space, experiments are well explained. Joint variational training approach is well explained. I wonder if there is an error in Eq 2 (see questions below).

Significance
The topic area (bayesopt on high dim discrete spaces) is timely and important, and one which the field has not fully solved to-date.
questions: Equation 2 has identical numerator/denominator terms with $q(u)$ and $\Phi(Z|X)$, which seems incorrect for the variational approximation? (At a glance, should it be p/q and $\frac{\Gamma}{\Phi}$)?

Why not compare SELFIES vs JTVAE in terms of score-achieved-by-iterations, as done w/ TuRBO vs LOL-BO in Table 1 rhs? Adding a row for either TuRBO-JTVAE or LOL-BO-JTVAE to Table 1 right would be helpful to set a clearer baseline.


limitations: Bayesopt on biologics might be a space where a societal impact statement might be in order. De novo biologicals design would constitute a mixed-use technology.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the problem of optimizing black-box functions defined over structured inputs. The solution is situated within the general framework of Bayesian optimization (BO) over latent space where a deep-autoencoder model is learned from unsupervised data and subsequently used for employing standard Bayesian optimization algorithms in the continuous (latent) embedding. One challenge with this general approach is the high-dimensionality of the latent space which converts a black-box optimization problem over combinatorial space to that over high-dimensional continuous space. The paper proposes a trust region based BO algorithm in the latent space as a way to address this challenge. Based on the hypothesis that there is a mismatch between the trust region on the latent space and the corresponding trust region in the original structured space, a joint inference procedure is proposed over the Gaussian Process (GP) surrogate model and the deep autoencoder model i.e., treating the deep autoencoder as sort of a deep kernel for the GP model. Experiments are performed on multiple molecular optimization and arithmetic expression tasks.
strengths_and_weaknesses: - The considered problem of black-box functions defined over structured inputs is very significant with many real world applications including molecular design/drug discovery.

- The idea of trust region Bayesian optimization for high-dimensional spaces is well known now especially with the work of Eriksson et al (TurBO)[1]. Therefore, the novelty of the proposed method is relatively limited but it does improve over the standard application of TurBO in latent space.

- The paper presents empirical analysis for the proposed approach. I found the empirical analysis sound and effective in showing the efficacy of the approach on the overall BO performance. The proposed approach outperforms other baselines on multiple benchmarks and it is especially refreshing to see the discussion around usefulness of objectives like log-p metric in molecular optimization (line 308 to 321).

- I found the literature review to be comprehensive and representative of the field.

In summary, the paper considers an important real-world application with an effective solution that outperforms multiple baselines. Barring some time complexity issues with the proposed inference procedure, it should be a useful method for practical applications.

questions: Please see above.
limitations: There is a good description of limitations in the supplementary section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper deals with solving the high dimensional Bayesian optimization problem with latent space deep autoencoder models (DAE) primarily focusing on structured input. The idea revolves around the high dimensionality issues that persist even after projecting the structured input into the latent with DAE models. The primary novelty lies in efficiently incorporating a trust region in the Bayesian optimization step to deal with this high-dimensionality issue, not explicitly addressed in the earlier research. The empirical performance demonstrates significant improvements over the latest SOTA latent space Bayesian optimization algorithms across various complex real-world benchmarks. The paper tries to emphasize improving the optimization methodologies as well along with the improvement of Deep generative models and shows its effectiveness for latent space BO with a combination of both.

strengths_and_weaknesses: The main strength revolves around addressing the mismatch b/w the trust-region in the latent space and  trust-region in the original structured input space. Although an extremely efficient formulation of trust region-based local optimization has been proposed by Eriksson et. al [2] that doesn't deal with the high-dimensionality with latent space BO. One of the most important contributions of this paper lies in formulating the trust-region for a high dimensional latent space Bayesian optimization problem. This paper points out a very interesting and critical point that even a small change in the latent space generated by BO i.e $||z - z'|| \leq \epsilon$ can be drastically different in the original space since the closeness in DAE is defined w.r.t to the decoder and BO doesn't explicitly incorporates that, hence it can still be that $||x - x'|| \geq \Delta$ which contradicts the purpose of trust-region based BO objective as pointed in Eriksson et. al. The main strength of this paper, according to me lies in the joint inference formulation which allows simultaneous optimization of GP with VAE parameters. The paper proposes an interesting methodology combining several popular LSBO/ Trust-region BO methods with State of the art DAE models in an intelligent framework. The empirical evaluations are quite explicit and detailed. In Figure 1, they show that LOL-BO (SELFIES) outperforms drastically all other benchmarks which are interesting. But what is even more critical is that the LOL-BO (JT-VAE) outperforms other baselines as well.

The primary weakness of this research lies in its uniqueness and novelty since it seemed like combining the best aspects of the latest advancements in the field of LSBO, Trust-region BO, and High dimensional BO in general. For ex: The semi-supervised variational inference was introduced by Kingma et. al and has been used to replace the weighted re-training mechanism popular in the LSBO literature by Miguel et al. Similarly, the paper by Eriksson et. al had already introduced the notion of trust-region in Bayesian optimization but it didn't have the latent space aspect to it. Also, the authors incorporate transformer layers on the recently published SELFIES representation which shows significant improvements. So, it's not extremely clear the exact novelty of this research. Another challenge lies in jointly learning multiple parameters and also using deep generative models should require a larger initial dataset both supervised and unsupervised and a comparison of that is not explicitly given as the major purpose of BO and GP, in general, is to be effective in a low-data regime where the initial supervised information is less. Hence, a comparison is missing with the initial supervised (and unsupervised) data on which the GP is trained? Also, related to this lies the concern in the simultaneous updating of the parameters, which the authors have dealt with various heuristics of periodically training, etc. However, there are no guarantees of convergence provided for the same.

### Post Rebuttal
I am happy with the author's response and increased my score.


References :

[1]. Austin Tripp, Erik A. Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems 33, 2020 

[2]. David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In H. Wallach, H. Larochelle, A. Beygelzimer,  F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/6c990b7aca7bc7058f5e98ea909e924b-Paper.pdf 

[3]. Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander Imani Cowen370 Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, Jan Peters, and Haitham Bou-Ammar. High-dimensional Bayesian optimization with variational autoencoders and deep metric learning. CoRR, abs/2106.03609, 2021. URL https://arxiv.org/abs/2106.03609

[4]. Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581–3589, 2014
questions: 1. What was the initial sample size for both the unsupervised and supervised cases? Can a comparison be provided with the other methods as well? I am specifically interested to see when the initial samples especially supervised information less how the performance compares? 

2. Joint-inferencing and using Transformer architectures will definitely require more initial sets of samples right? Unsupervised information can be available but there should be a proper justification for that. Hence, I am curious to understand the initial information comparison.

3. TuRBO maintains m trust regions simultaneously. It uses a TS type approach to choose the best sample across the trust regions which can be paralleled and results in better efficiency. However, this work doesn't consider multiple trust regions, but still, there is an improvement over TurBo. Is it due to the Sparse GP Augmentation aspect with a global set of learned-inducing variables?
limitations: The authors have addressed the computation challenges to some extent and have carefully addressed the societal challenges which are appreciable. They clearly mention that experts should be heavily involved and that FDA guidelines for drug development and approval should be strictly followed.
review_title: null

Meta-review: metareview: This paper develops a well-engineered approach to black-box optimization of expensive functions over structured spaces (e.g., graphs). The solution falls within the framework of Bayesian optimization (BO) over latent (continuous) space learned from unsupervised structures using deep generative models. One challenge in this framework is the high-dimensionality of the learned latent space. One straight-forward approach to handle this challenge is to apply state-of-the-art high-dimensional BO methods such as trust region based BO. The paper hypothesizes that there is a mismatch between the trust region in the latent space and the corresponding trust region in the structured space, and proposes a joint training/inference method over the Gaussian process surrogate model and the deep generative model to overcome this challenge. Experimental results on multiple molecule design optimization and arithmetic expression tasks show very good results over prior methods and shows improvements due to SELFIES representation for this approach.

Some of the reviewers' questioned the overall novelty of the approach as it combines advances in latent space BO, trust-region BO, and high-dimensional BO, but there is an agreement that overall strengths outweigh this concern. This paper makes a useful contribution to the latent space BO framework. Therefore, I recommend acceptance.
confidence: Certain
Decision: Accept