Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

Abstract: We consider the problem of optimizing combinatorial spaces (e.g., sequences, trees, and graphs) using expensive black-box function evaluations. For example, optimizing molecules for drug design using physical lab experiments. Bayesian optimization (BO) is an efficient framework for solving such problems by intelligently selecting the inputs with high utility guided by a learned surrogate model. A recent BO approach for combinatorial spaces is through a reduction to BO over continuous spaces by learning a latent representation of structures using deep generative models (DGMs). The selected input from the continuous space is decoded into a discrete structure for performing function evaluation. However, the surrogate model over the latent space only uses the information learned by the DGM, which may not have the desired inductive bias to approximate the target black-box function. To overcome this drawback, this paper proposes a principled approach referred as LADDER. The key idea is to define a novel structure-coupled kernel that explicitly integrates the structural information from decoded structures with the learned latent space representation for better surrogate modeling. Our experiments on real-world benchmarks show that LADDER significantly improves over the BO over latent space method, and performs better or similar to state-of-the-art methods.


Reviews and author responses:

This paper proposes a method for Bayesian optimization in combinatorial spaces, i.e. spaces of general objects, e.g. trees and graphs.  Existing methods usually map these objects into a supspace of $\mathbb{R}^n$ , then perform traditional BO in that latent space. However, doing this ignores the structural information in the original space, which can be valuable. The method presented in this work incorporates such structural information into BO.  The end result is a covariance function, which combines information both in the latent space and the original space.  The problem this paper tries to solve is very important, i.e. bayesian optimization for structured objects. Since this paper still use traditional acquisition functions, e.g. expected improvement, which can only handle fixed-dimensional vectors. The focus of this work is more of the GP modeling than bayesian optimization. 

Strengths of this paper:
1. this paper is well-written. Authors express their ideas in a succinct way.
2. the empirical experiments are detailed.
3. this work makes it clear correctly incorporating the structural information can indeed improve the performance. 

Weaknesses of this paper:
1. The derivation in eq(4) is wrong. Correct if I missed something. The derivation from eq(3) to eq(4) means  $L = \sum_{i=1}^{m}  v_i v_i^T$.  Eigenvalues are missing. Besides, the empirical feature mapping is not the correct one. Authors put a footnote in page 5 and claim "core concepts are the same". This is not a rigorous way. 
2. The novelty of the proposed method is  limited.  The core of this work is the covariance function in eq(4). This covariance function can be built through kernel construction rules in a very straightforward way. i.e. $x^T A z$, where $A$ corresponds to $K^{-1/2} L K^{-1/2}$. See [1]
3. It is better to use terminology in BO. For example, In BO literature, "best uncovered value" (L287) is called incumbent.

1.Rasmussen, C.E., Williams, C.K.I.: Gaussian processes for machine learning. MIT Press, Cambridge, Mass (2006).

post-rebuttal
====
I'd like to thank authors' detailed response. However, based on existing empirical results, I am still not convinced that the proposed method is effective. The question of whether the success mostly comes from a good structured kernel is to be answered by more empirical studies. Based on discussions with other reviewers, I increase my score to 5. 



 1. One question I have is whether the improvements come from such a combined kernel or solely come from the use of a proper structural covariance function. For completeness, I suggest an extra experiment where only the structural  covariance function is used. 
2. Authors already pointed out how to select the  structural covariance function is one limitation of this work. This is indeed a big limitation. In Fig4 right, the performance of LADDER is on a par with several existing methods. This indicates a wrongly or sub-optimal choice of the corresponding covariance function can dominate the performance of LADDER. 
3. In Fig4 left, it is possible that the huge success of LADDER solely comes from the string kernel.  This shares the same idea with point #1. An extra experiment is needed.  

### Summary

(+) Bayesian Optimization over Combinatorial Spaces is a relevant topic for the community

(+) The paper is clear, well written and properly structured

(+) The empirical evaluation is well structured and split according to clear questions.

(-) In its current form, the paper makes it difficult  to precisely relate its claimed contributions to previous research

(-) Some simple, generic baselines (e.g., random search) seem to be missing

(-) The introduction of the “Generalized Nystrom” parametrization would deserve more intuition. 
  ### Detailed comments

* To clarify, in the “Challenge #2”, the concern is the fact that d is large (i.e., the dimension of Z)? If this is the case, this could be clarified.

* “ Given kernels l and k with L and K...“: In this sentence, it should be clarified that the kernel l operates in Z x Z while k operates over X x X (at least this is my understanding).

* I am under the impression that the paragraph “Structure-coupled kernel. ” is a bit difficult to follow. I check the refs associated with the “Generalized Nystrom extension idea”. I wonder if the material could be introduced differently, by building up more intuition? (e.g., would there be other alternatives, simpler to grasp but too expensive? What would be a more naive kernel, sitting in between LSBO and the more sophisticated proposal? etc.)

* What’s the sensitivity of the results with respect to the amount of unsupervised pre-training?

* I have two comments/questions for the paragraph “Comparison of surrogate models.”. 
(I) Why MAE is used and not a metric that would also quantify the uncertainty? (II) To further validate the hypothesis (“d is too large”), Fig. 2 should also display what happens if we pretrain the decoder/encoder for various sizes of d. 

* “The discrepancy in LADDER’s performance on the chemical design benchmark can be attributed to the low-flexibility of fingerprint kernel which doesn’t include any hyper-parameters”: Aren’t there options of richer kernels to test this hypothesis?

* A relevant reference to add: Lu et al., Structured Variationally Auto-encoded Optimization (http://proceedings.mlr.press/v80/lu18c.html)

* Why naive generic baselines like random search and a standard genetic algorithm aren’t reported?

* I am not sure it is necessary to dedicate a section about “Reinforcement learning” in the related work. Instead, I would like to see further detailed discussion about the precise connections between the present submission and other works around BO + DGMs. The paper explains well how the proposed approach differs from LSBO but not how it differs from previous works. 

### Minor

* In Sec. 2, second paragraph, some “x”’s, e.g.,  need to be made bold for consistency

* “...large [15, 52, 11, 22]..” → “...large [15, 52, 11, 22].”

* “A GP model defines a prior distribution on functions which is entirely characterized by a kernel defined over a pair of inputs…” → ...and a mean function

* In the sentence “...and v_i is the eigenvector of the kernel…”, I am not sure the i index refers to the same i index for K_{i,j} (in the latter, this is the i-th data point z_i). In my understanding, they mean different things and different letters could be used to avoid confusion.

* “Reinforcement learning” → “Reinforcement learning.”


### UPDATE POST REBUTTAL

I have gone through the reviews and the replies of the authors, whom I'd like to thank for the work put in the rebuttal.

Many of my concerns/questions have been clarified, e.g.,

* Added comparison to random search
* Improved introduction and discussion of the “structure-coupled kernel”
* The authors also promise to develop further their related work section to better discuss the links with previous research (beyond LSBO).

While at this stage, I was willing to improve my score, the discussion with the other reviewers made me aware of important aspects that I had overlooked. 

In particular, the experiments of the paper and those provided in the rebuttal do show that the performance hinges on the used structured kernel. Moreover, the latent information seems to contribute only moderately to the final gains, which, when combined with the expensive cost of pre-training, weakens the argument to use the latent information.

All in all, I am thus going to keep my score.
 Adequately addressed, in the sense that I haven't seen any dedicated paragraph about this topic but the work does not present, to the best of my understanding, potential negative societal impact.

This paper introduces a method that allows one to perform Bayesian optimization over a latent space of a data set, whereby an encoder and decoder are available. The key contribution is the use of a structure-coupled kernel to incorporate the decoded outputs as well as the latent space when performing BO.  The paper is well-written and does a good job at motivating the problem and highlighting potential flaws in previous approaches. 

The main contribution of the paper is the introduction of the structure-coupled kernel to incorporate the output of the decoder as well as the latent space when learning the GP surrogate. All the empirical evidence provided in the paper demonstrates that this is a useful addition and it is therefore difficult to find any issues with the method.

The key comment/issue with this work comes down to whether the advantage comes from simply the structure-coupled kernel or whether there is no reason to include the latent space at all. These are the questions that would help clarify this issue:

* The LADDER approach seems to build a data set of the latent space and the decoded latent space. It is clear from the results that the decoded latent space is important for the surrogate model, however it is not clear from the experiments whether it is possible to directly optimize over the black-box functions by simply using just the sub-sequence kernel or just the fingerprint kernel. It would be helpful to include “direct BO approach” as a comparison for Figure 2.
* As far as I understand it, the purpose of working in the latent space is to make it easier to learn a GP surrogate. It seems that by using the decoded structure, the problem is brought back to its original dimensionality. What are the input-output dimensionalities of the chemical design and the arithmetic expression problem? Does the dimensionality increase of LADDER cause any potential problems.

### After author comments:

Thanks very much for providing that extra experiment and comments. As a result I will stick with my score. N/A

 Thank you for your thorough feedback and pointing out that our paper tackles an important problem and provides a well-written approach with detailed experimental evaluation. We will clarify the concerns and answer the questions raised in the review comments below.

**Q1**. The derivation in eq(4) is wrong. Correct if I missed something. The derivation from eq(3) to eq(4) means eigenvalues are missing. Besides, the empirical feature mapping is not the correct one. Authors put a footnote in page 5 and claim "core concepts are the same". This is not a rigorous way

Equation 4 is correct.  However, $v_i$ in Equation 1 is the eigenvalue-scaled eigenvector of the kernel matrix L, i.e., $V = [v_1, v_2, … v_m] = U \Lambda^{½}$ where $U$ and $\Lambda$ are eigenvectors and eigenvalues of L respectively. We will fix this typo in Line 188. Thank you for pointing it out. Please note that we do not explicitly compute $U$ or $\Lambda$, and use the final expression in Equation 4. 

We will remove the footnote on page 5. It was only added to provide some additional intuition for the proposed structure-coupled kernel idea. Roughly, $k_z$ is a good approximation for the kernel feature map but this mapping doesn’t form an orthonormal system and we need to take a weighted dot product to get the kernel value. However, we do agree that it is not rigorous and will remove the footnote. 


**Q2**. One question I have is whether the improvements come from such a combined kernel or solely come from the use of a proper structural covariance function. For completeness, I suggest an extra experiment where only the structural covariance function is used.

|                            | Iteration: 15 | Iteration: 25 | Iteration: 50 | Iteration: 75 | Iteration: 100 |
|----------------------------|---------------|---------------|---------------|---------------|----------------|
| BO w/  Fingerprints kernel | 2.46 +- 0.32  | 2.80 +- 0.23  | 3.11 +- 0.11  | 3.32 +- 0.23  | 3.36 +- 0.23   |
| LADDER                     | 2.78 +- 0.26  | 2.93 +- 0.23  | 3.18 +- 0.20  | 3.25 +- 0.18  | 3.25 +- 0.18   |


We present results for the ‘direct BO approach i.e. BO with only structured kernel’ (BO with Fingerprints kernel) on the  chemical design task in the table above.The experimental setup is the same as in the main paper. We make two observations:   1) LADDER has better accuracy for a large number of initial BO iterations compared to direct BO approach; and 2)  BO with fingerprints kernel  alone reached slightly better accuracy than LADDER in the end. However, please note that the main contribution of this paper is to improve the performance of latent-space BO methods. Note that if we improve the latent space by training deep generative models on larger amounts of unsupervised structures, LADDER will be able to achieve comparable or better accuracy when compared to the direct BO approach. 

Providing similar results for the arithmetic expressions task requires significantly more time due to the computationally expensive string kernel. We will add the corresponding results in the final paper.

We would like to provide a more general argument to justify the research direction of combining the strengths of latent space and structured kernels. If we have a perfect kernel that captures all the relevant domain-specific characteristics/structure, then there is no utility for latent space BO methods or LADDER. However, generic kernels over combinatorial structures (e.g., strings, trees, and graphs) may not always capture the critical domain-specific characteristics. For example, in optimizing Metal-Organic Framework (MOF) materials for adsorption property, e.g, storage of gases for hydrogen-powered vehicles, we can use graph kernels over the graph representation of MOF materials, but they won’t be able to capture some critical characteristics such as pore size, which is important to predict the adsorption property of the MOF materials. In such scenarios, domain experts and practitioners can either spend time to engineer a hand-designed kernel (e.g, Fingerprints kernel from the chemical sciences community) or use large amounts of unsupervised structures data to learn a latent representation to capture these domain-specific characteristics in an automated manner. Our research takes the first step to synergistically combine the advantages of latent space and generic/domain-specific kernels to automate the overall workflow, which is very advantageous for scientists and engineers who are the real-users of this technology. By no means we claim that LADDER is the optimal algorithm to achieve this goal, but our hope is that LADDER will inspire the BO community to explore this important research direction to uncover better algorithms in the future.


 Thank you for the detailed feedback and pointing out that our paper is clear, well written and properly structured and tackles an important problem. We will clarify the concerns and answer the questions raised in the review comments below.

**Q1**. To clarify, in the “Challenge #2”, the concern is the fact that d is large (i.e., the dimension of Z)? If this is the case, this could be clarified.

Correct. The challenge arises because of the large dimensionality of the latent space  and small-data setting (i.e., availability of a small number of structure and function evaluation pairs). 

**Q2**. “Given kernels l and k with L and K...“: In this sentence, it should be clarified that the kernel l operates in Z x Z while k operates over X x X (at least this is my understanding).

Correct. We will add this notation. 

**Q3**. I am under the impression that the paragraph “Structure-coupled kernel. ” is a bit difficult to follow ... I wonder if the material could be introduced differently, by building up more intuition? … What would be a more naive kernel, sitting in between LSBO and the more sophisticated proposal? etc.)

As the name suggests, one way to interpret  the proposed structure-coupled kernel idea is as an extension of the more commonly known Nystrom method, which  allows us to extrapolate eigenfunctions known at a fixed set of points. This interpretation can be seen by setting K = L in equation 4 which gives the exact Nystrom extension equation. 

We could consider alternative methods to combine the kernels over latent space (continuous kernel) and kernels over combinatorial space (structured kernel).  For example, sum and product of the  continuous and structured kernels. However, the resulting acquisition function optimization (AFO) problem will be over a mixed discrete-continuous space which is  presumably much harder than solving the AFO problem over a continuous latent space.


**Q4**. What’s the sensitivity of the results with respect to the amount of unsupervised pre-training? Fig. 2 should also display what happens if we pretrain the decoder/encoder for various sizes of d.


Training these models is very expensive.  Some potential issues include the decrease in valid decodings of candidate inputs in the latent space and different latent space inputs decoding to the same structure in the combinatorial space. [Reference 1, Reference 2]

[Reference 1] Tripp, A., Daxberger, E., & Hernández-Lobato, J. M. (2020). Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33.

[Reference 2] Polykovskiy, D., & Vetrov, D. (2020, June). Deterministic Decoding for Discrete Data in Variational Autoencoders. In International Conference on Artificial Intelligence and Statistics (pp. 3046-3056). PMLR.


**Q5**. The discrepancy in LADDER’s performance on the chemical design benchmark can be attributed to the low-flexibility of the fingerprint kernel which doesn’t include any hyper-parameters”: Aren’t there options of richer kernels to test this hypothesis?

Thank you for this comment. We will also try LADDER with graph kernels  as part of our immediate future work. 

**Q6**. A relevant reference to add: Lu et al., Structured Variationally Auto-encoded Optimization (http://proceedings.mlr.press/v80/lu18c.html)

Thank you for the reference. We will add it in the final paper. 

**Q7**. Why naive generic baselines like random search and a standard genetic algorithm aren’t reported?

Please see the results below (on Chemical design task) where LADDER significantly outperforms random search. We will add these results for both the benchmarks in the final paper. 

|               | Iteration: 15 | Iteration: 25 | Iteration: 50 | Iteration: 75 | Iteration: 100 |
|---------------|---------------|---------------|---------------|---------------|----------------|
| Random Search | 2.49 +- 0.26  | 2.67 +- 0.29  | 2.87 +- 0.19  | 3.02 +- 0.18  | 3.09 +- 0.17   |
| LADDER        | 2.78 +- 0.26  | 2.93 +- 0.23  | 3.18 +- 0.20  | 3.25 +- 0.18  | 3.25 +- 0.18   |

**Q8**. I am not sure it is necessary to dedicate a section about “Reinforcement learning” in the related work. Instead, I would like to see further detailed discussion about the precise connections between the present submission and other works around BO + DGMs. The paper explains well how the proposed approach differs from LSBO but not how it differs from previous works.

 Thank you for your good suggestion. We will incorporate it in the final paper.



 Thank you for the thorough feedback and pointing out that our paper is well-written, provides a well-founded approach with well-chosen experiments indicating a  significant performance benefit.  We will clarify the concerns and answer the questions raised in the review comments below.

**Q1**. “I think the one thing that could greatly strengthen this paper is a small model calibration study comparing the “naive” model that just considers a kernel defined over the latent space, and your LADDER kernel. Doing this even over a toy problem, for which a metric such as RMSE can be easily computed, would help the reader intuitively understand why this approach is better.”

Choosing an appropriate metric for model calibration in the context of our problem setting requires some investigation but we will add a small calibration study with RMSE in the final paper.


**Q2**. “As noted in the experiment section, many combinatorial problems admit a natural kernel over the full problem space. Presumably, many don’t; what if a poor kernel is chosen that does not accurately model the full problem space? This is somewhat noted as a limitation in the conclusion, but it would be nice if the authors provided some more intuition as to what conditions a good kernel for the combinatorial space ought to have.”


Theoretically, the representation power of kernels is studied in terms of properties including “universal” and “characteristic” [Reference 1].  Satisfying these properties is a basic requirement for selecting a kernel. However, since many known kernels are universal, this criterion might not be enough for all problem settings including our small-data setting (small number of structure and function evaluation pairs). Nevertheless, for combinatorial spaces, most of the kernels are roughly defined in terms of counting the number of common substructures in  a given pair of combinatorial structures [Reference 2]. Kernels that capture large sub-structures are generally more powerful in terms of their representational capacity. However, there is a trade-off between representation power and time complexity of kernel computation. For our problem setting of optimizing combinatorial spaces using expensive function evaluations, an instance of small data problem (i.e., with a small number of structure and function evaluation pairs), it is best to select a kernel with the highest representation power and some hyper-parameters that can be optimized in a data-driven manner. Importantly, the inaccuracy of modeling using structured kernel alone can be potentially overcome by the latent space and vice versa. This is the key motivation of our LADDER approach. Our new experimental results on the chemical design task provide partial evidence for this hypothesis.

We would like to provide a more general argument to justify the research direction of combining the strengths of latent space and structured kernels. If we have a perfect kernel that captures all the relevant domain-specific characteristics/structure, then there is no utility for latent space BO methods or LADDER. However, generic kernels over combinatorial structures (e.g., strings, trees, and graphs) may not always capture the critical domain-specific characteristics. For example, in optimizing Metal-Organic Framework (MOF) materials for adsorption property, e.g, storage of gases for hydrogen-powered vehicles, we can use graph kernels over the graph representation of MOF materials, but they won’t be able to capture some critical characteristics such as pore size, which is important to predict the adsorption property of the MOF materials. In such scenarios, domain experts and practitioners can either spend time to engineer a hand-designed kernel (e.g, Fingerprints kernel from the chemical sciences community) or use large amounts of unsupervised structures data to learn a latent representation to capture these domain-specific characteristics in an automated manner. Our research takes the first step to synergistically combine the advantages of latent space and generic/domain-specific kernels to automate the overall workflow, which is very advantageous for scientists and engineers who are the real-users of this technology. By no means we claim that LADDER is the optimal algorithm to achieve this goal, but our hope is that LADDER will inspire the BO community to explore this important research direction to uncover better algorithms in the future.


- [Reference 1] Sriperumbudur, B. K., Fukumizu, K., & Lanckriet, G. R. (2011). Universality, Characteristic Kernels and RKHS Embedding of Measures. Journal of Machine Learning Research, 12(7).

- [Reference 2] Gärtner, T. (2003). A survey of kernels for structured data. ACM SIGKDD explorations newsletter, 5(1), 49-58.


 Thank you for the thorough feedback and pointing out that our paper is well-written and justified with comprehensive experiments. We will clarify the concerns and answer the questions raised in the review comments below.

**Q1**. “…however it is not clear from the experiments whether it is possible to directly optimize over the black-box functions by simply using just the sub-sequence kernel or just the fingerprint kernel. It would be helpful to include “direct BO approach”...”

|                            | Iteration: 15 | Iteration: 25 | Iteration: 50 | Iteration: 75 | Iteration: 100 |
|----------------------------|---------------|---------------|---------------|---------------|----------------|
| BO w/  Fingerprints kernel | 2.46 +- 0.32  | 2.80 +- 0.23  | 3.11 +- 0.11  | 3.32 +- 0.23  | 3.36 +- 0.23   |
| LADDER                     | 2.78 +- 0.26  | 2.93 +- 0.23  | 3.18 +- 0.20  | 3.25 +- 0.18  | 3.25 +- 0.18   |

We present results for the ‘direct BO approach’ (BO with Fingerprints kernel) on the  chemical design task in the table above.The experimental setup is the same as in the main paper. We make two observations:   1) LADDER has better accuracy for a large number of initial BO iterations compared to direct BO approach; and 2)  BO with fingerprints kernel  alone reached slightly better accuracy than LADDER in the end. However, please note that the main contribution of this paper is to improve the performance of latent-space BO methods. Note that if we improve the latent space by training deep generative models on larger amounts of unsupervised structures, LADDER will be able to achieve comparable or better accuracy when compared to the direct BO approach. 

Providing similar results for the arithmetic expressions task requires significantly more time due to the computationally expensive string kernel. We will add the corresponding results in the final paper.

We would like to provide a more general argument to justify the research direction of combining the strengths of latent space and structured kernels. If we have a perfect kernel that captures all the relevant domain-specific characteristics/structure, then there is no utility for latent space BO methods or LADDER. However, generic kernels over combinatorial structures (e.g., strings, trees, and graphs) may not always capture the critical domain-specific characteristics. For example, in optimizing Metal-Organic Framework (MOF) materials for adsorption property, e.g, storage of gases for hydrogen-powered vehicles, we can use graph kernels over the graph representation of MOF materials, but they won’t be able to capture some critical characteristics such as pore size, which is important to predict the adsorption property of the MOF materials. In such scenarios, domain experts and practitioners can either spend time to engineer a hand-designed kernel (e.g, Fingerprints kernel from the chemical sciences community) or use large amounts of unsupervised structures data to learn a latent representation to capture these domain-specific characteristics in an automated manner. Our research takes the first step to synergistically combine the advantages of latent space and generic/domain-specific kernels to automate the overall workflow, which is very advantageous for scientists and engineers who are the real-users of this technology. By no means we claim that LADDER is the optimal algorithm to achieve this goal, but our hope is that LADDER will inspire the BO community to explore this important research direction to uncover better algorithms in the future.

**Q2** “... It seems that by using the decoded structure, the problem is brought back to its original dimensionality. What are the input-output dimensionalities of the chemical design and the arithmetic expression problem? Does the dimensionality increase of LADDER cause any potential problems.”

The latent space dimensions for chemical design and arithmetic expression tasks are 56 and 25 respectively. The dimensionality of the combinatorial space is variable (arithmetic expression strings and chemical graphs of varying size).
Please note that the GP surrogate is still defined over the latent space. It takes into account the decoded structure information but that is required for computing the structure-coupled kernel alone. Therefore, the input and output dimensionality are exactly the same as the naive latent space BO approach (i.e., no increase). For example, the acquisition function optimization procedure is still performed over the continuous latent space which is the exact same search space as the ``naive`` latent space BO approach.

* The authors tackle BO over latent spaces obtained through applications of VAEs, and note that current strategies in the modeling phase do not account for the decoder, thus potentially leading to a model that doesn't’ reject the true objective function. 

* The authors deal with this problem by defining a new kernel that essentially couples interactions of points in the latent space and in the original space, through this Generalized Nystrom Extension idea. 

* A set of well-chosen experiments indicate this approach yields a significant performance benefit. 
  Important problem; well-founded approach; promising results. 

I leave a few additional questions and comments below:

* I think the one thing that could greatly strengthen this paper is a small model calibration study comparing the “naive” model that just considers a kernel defined over the latent space, and your LADDER kernel. Doing this even over a toy problem, for which a metric such as RMSE can be easily computed, would help the reader intuitively understand why this approach is better. 
* As noted in the experiment section, many combinatorial problems admit a natural kernel over the full problem space. Presumably, many don’t; what if a poor kernel is chosen that does not accurately model the full problem space? This is somewhat noted as a limitation in the conclusion, but it would be nice if the authors provided some more intuition as to what conditions a good kernel for the combinatorial space ought to have. 
 Yes

Meta-review: Recent work in Bayesian optimization has proposed latent space optimization, where Bayesian optimization is performed in the latent space of an autoencoder. This paper makes a potentially interesting advancement over the standard approach of using only latent codes as input to the GP. In particular, this paper defines a kernel jointly over a latent code z and the decoded structure \Phi(z). 

Ultimately, I think the simplicity of the authors' approach is its strength. An obvious weakness of latent space optimization is that, because the decoder is not perfect, the decoded structure \Phi(z) may not have the predicted objective value of z, even if the surrogate model were highly accurate. Directly incorporating the decoded structure into the surrogate model is an obvious and natural baseline approach to this, but (crucially) one that I have not seen elsewhere. 

Please incorporate the additional results run during the author feedback period and other reviewer feedback in the final version of the paper.

Decision: accept