Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions:  
In this paper, the challenging problem of using a high-dimensional domain in Bayesian optimization is tackled through gaussian process based variable selection. The method presents an alternative to the current state-of-the-art embedding-based approach. The paper shows that the model improves convergence iterations and time when compared with existing approaches. However, as the number of dimensions in input increases, the variable selection approach becomes less accurate.

overall_review: Strength:

1.	The main strength of the approach is that it tries to overcome the sensitivity of embedding dimensions.
2.	The paper presents a novel variable selection approach that can be expanded into other areas.
3.	The paper is well written, with a clear background distinguishing the contribution from the potential downsides.
 
Weakness:

1.	The paper fails to clarify to the audience the marginal benefits compared with other approaches on high dimensions.
2.	Paper relies on the assumption of separating the important from the non-important, but the robustness and stability 

review_summary:  
The paper is rated borderline leaning accept due to the following reasons:

1.	Issues in running the code to create reproducible results.
2.	The robustness and stability of the approach cannot be quantified.
3.	Unable to distinguish the benefits of the approach since it was noted that it doesn’t perform as well as the embedding-based approach at high dimensions, which is the goal of the work.

actions_required_to_increase_overall_recommendation: Following can be improved/addressed,

1. A step-by-step instruction in the readme file would be helpful to run the project for reproducibility. Also, rerunning the code following the instructions will be helpful so that the reviewer will also be able to validate.

2. Specify the primary benefits of the work over using embedding based BO approach given that the authors have suggested using embedding based approach as first preference.

3. Robustness and stability of the variable selection at high dimensions.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: 1. This paper propose a high-dimensional BO algorithms based on variable selection. 
2. Experimental results show the superior performance of VS-BO.
overall_review: Positive
- The proposed method is easy to understand and the paper is easy to follow.

Negative
- The experimental results seem weak. The statistical significance test may be further strengthen the results.
- The types of real-world problems are limited.

review_summary: This paper is well-written, but the experimental part is weak.
actions_required_to_increase_overall_recommendation: More discussions about the application scenarios of the algorithm and more types of real-world experiments under these problems.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper addresses an important problem in the field of Bayesian optimization (BO): high-dimensional BO. They proposed a new computational efficient BO algorithm that leverages variable selection by performing the following steps: (1) select variables that are considered “important”; (2) recommend experiments using GP models fitted using these variables; (3) get observations and adjust variables deemed “important” every few iterations. The method is compared against other SOTA algorithms for high-dimensional BO problems based on both # iterations and wall/CPU times. 
overall_review: Positive aspects 
- This paper is well organized, with clearly stated framework/algorithm and motivation.
- The proposed algorithm is compared against multiple SOTA algorithms. It seems to achieve comparable or better performance on synthetic and real problems and is computationally efficient.

Negative aspects:
- Some parts of the paper can benefit some clarification and critical analysis.
- There might be some room for improvement for the experiments and the definition of the IS score.
review_summary: This is a well-written and thoughtful paper that can potentially have good impact in the BO and AutoML community. There are some issues in the papers but can be fixed through some additional experiments / clarification texts.
actions_required_to_increase_overall_recommendation: I would like to see the author address the questions/concerned raised in the "Technical Quality And Correctness" and "Clarity" sections.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper addresses high-dimensional Bayesian optimization problems and proposes a new algorithm that leverages variable selection. The idea is to embed high-dimensional space in low-dimensional space with a novel technique. The paper also analyzes the computational complexity and presents the empirical evaluations with results.
overall_review: The proposed algorithm and the idea is simple and clear. The contribution is declared in a good way. However, there are some notations as I mentioned above should be corrected for publishing the paper. Also, in line 128, it is declared that "The algorithm uses different strategies to choose new queries of the variables from two different sets." It is not clear for the rest of the paper.
review_summary: Except for minor corrections, the paper is well-written and contains novelties. There are some parts that are not clear enough as I declared above, and my score will be weak accept for consideration of the other reviewer's opinion.
actions_required_to_increase_overall_recommendation: The definitions could be in another section instead of the Background. Especially, since the equations in the background are unexpected, preliminaries would be better. Mathematical notations need a double check as I mentioned some of them.
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless paper with major impact and strong evaluation, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: &nbsp;

The authors introduce a new method for high-dimensional Bayesian optimisation based on variable selection.

&nbsp;
overall_review: &nbsp;

**__MAJOR POINTS__**

&nbsp;

1. In the section on related work, the literature on VAE-based embedding methods is relevant but is not described. While I believe it is beyond the scope of the paper to compare against VAE-based embedding methods, references such as [2-4] could be briefly discussed. In terms of the literature on high-D BO, TuRBO [5] could also be discussed.

2. In Algorithm 2, line 7, the stopping criterion is defined as the iteration when the difference between successive values of the negative log marginal likelihood is more less than a factor of 10 of the difference between successive values at the previous iteration. This is a hyperparameter of the algorithm and I am wondering if the authors performed any sensitivity analysis on this stopping criterion?

3. For the experiments, why do the authors report the results with 1/8 standard deviation in place of the standard error? The latter may be preferred if the objective is to establish the relative ranking of algorithms. The former may be preferred if the goal is to showcase the variability in performance of the different algorithms.

4. On line 283, in terms of the claim that VSBO outperforms VSBO-mix, would the errorbars be overlapping in the case that the standard error was plotted instead of 1/8 standard deviations? In other words can this claim be definitely supported? If not, perhaps it should be moderated?

&nbsp;

**__MINOR POINTS__**

&nbsp;

1. On line 2, the authors state that the input domain $\mathcal{X}$ is in $[0, 1]^D$. The algorithm introduced, however, is more broadly applicable for bounded input domains in $\mathbb{R}^D$. As such, it may be worth stating that for the purposes of the paper the authors restrict their study "without loss of generality" to input domains in $[0, 1]^D$.

2. Line 23, "maximizes the function f" assumes a maximization problem. While the intention is understood perhaps "optimization" would be more appropriate before the black-box optimization problem is formerly defined as a maximization problem. This is a very minor nit and one which is a matter of taste!

3. On line 28, the reference Griffiths and Hernández-Lobato was published in Chemical Science [6].

4. Line 49, "Letham et al. also points out that when projecting the optimal point from the embedding space back to the original space, there is no guarantee that the resulting point is within X , and thus the algorithm may fail to find an optimum within the input domain." It is worth mentioning here that this problem can be partially addressed (at least in the case of linear embedding methods such as ALEBO) by mapping the input bounds from the original space to the embedding space. In the case of PCA for example, the embedding-based bounds would take the form of linear inequality constraints.

5. In Equation 1, the mean vector is set to zero. This implicitly assumes that the observations have been standardized and this would be worth mentioning.

6. In Equation 5, the motivation for randomly sampling points $x^{(k)}$ as opposed to grid sampling them, in addition to the expense of grid sampling for high dimensions, is presumably so that computation of the IS is robust to uninformative dimensions as in [1] for example?

7. Moriconi et al. Was published in the machine learning journal.

8. BoTorch was published in NeurIPS 2020.

9. Line 203, typo, "Matérn". This is a major crime against the French language.

10. Line 326, in terms of the applications for computational chemistry it would be particularly exciting to see applications of VS-BO to descriptor-based molecular property prediction and optimization [7].

11. In the references there are some missing capitalizations such as "Bayesian", "Gaussian", "Monte Carlo".

12. Line 577, an appropriate citation for L-BFGS could be included [8].

13. In Figure 6b) are the confidence bands really 1/8 standard deviations? The uncertainty bands would be very large if that is the case!

&nbsp;

**__REFERENCES__**

&nbsp;

[1] Bergstra and Bengio, [Random Search for Hyper-parameter Optimization](https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf), JMLR, 2012.

[2] Grosnit et al., [High-Dimensional Bayesian Optimization with Variational Autoencoders and Deep Metric Learning](https://arxiv.org/abs/2106.03609), arXiv, 2021.

[3] Maus et al., [Local latent Space Bayesian Optimization over Structured Inputs](https://proceedings.neurips.cc/paper_files/paper/2022/file/ded98d28f82342a39f371c013dfb3058-Paper-Conference.pdf), NeurIPS 2022.

[4] Notin et al., [Improving black-box optimization in VAE latent space using decoder uncertainty](https://openreview.net/forum?id=F7LYy9FnK2x), NeurIPS 2021.

[5] Eriksson et al., [Scalable Global Optimization via Local Bayesian Optimization](https://proceedings.neurips.cc/paper_files/paper/2019/file/6c990b7aca7bc7058f5e98ea909e924b-Paper.pdf)
, NeurIPS 2019.

[6] Griffiths and Hernández-Lobato, [Constrained Bayesian Optimization for Automatic Chemical Design using Variational Autoencoders](https://pubs.rsc.org/en/content/articlehtml/2019/sc/c9sc04026a), Chemical Science, 2020.

[7] Griffiths et al., [GAUCHE: A Library for Gaussian Processes in Chemistry](https://arxiv.org/abs/2212.04450), arXiv, 2023.

[8] Liu and Nocedal, [On the limited memory BFGS method for large scale optimization](https://link.springer.com/article/10.1007/BF01589116). 330 Math. Program., 45(1-3):503–528, 1989.

&nbsp;
review_summary: &nbsp;

A new algorithm is introduced for high-dimensional Bayesian optimization. The algorithm is simple and straightforward to implement, the empirical and theoretical analyses are thorough and the paper is very well-written. As such I recommend acceptance.

&nbsp;
actions_required_to_increase_overall_recommendation: &nbsp;

I am willing to increase my score by a further point if the minor issues raised above are addressed.

&nbsp;
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: - The introduction of a novel, adaptive variable selection scheme for high-dimensional Bayesian Optimization
- A novel treatment of important versus unimportant variables through the dual use of BO and CMA-ES
- The BO that results from the proposed variable selection is cheap, both in theory and in practice

overall_review: # Positive
- __The relevance is high__: High-dim BO does feature a number of methods which are expensive to the point where even benchmarking against them is painful. Having computationally feasible methods is important, particulairly since the function is likely going to need 100's of evaluations to be optimized, anyway.
- __The idea is good__: Identifying relevant variables and optimizing solely on these is a simple and intuitive idea, which I appreciate. Combining with CMA-ES also seems sensible from a practical perspective. 
- __The proposed algorithm is provably cheap__: Given the conventional BO setup, substeps of the algorithm scales computationally with the dimensionality of $x_{ipt}$ 
- __The evaluation protocol is novel (?) and very practial__: Adding semi- and mini-relevant dimensions to synthetic test functions is a great idea - one which seems substantially more realistic than the current black-and-white regime.
- __Good empirical performance__: Clearly, the method performs well, which is even more impressive given its compute time. However, I _do not_ view this as supremely important. 
- __Clear and well-explained__. The paper is mostly a pleasant read, with a high degree of correctness and clarity. 


# Negative:
- __Hyperparameters of variable selection__: Algo. 2, Line 7 contains the hyperparameters 3 and 10. These are not mentioned as such, nor ablated. I would greatly appreciate if the authors could shed some light on how they arrived at these values, and provide an ablation on how these impact performance. Moreover, I believe they should be parametrized so that their scalar values are not part of the algorithm. 
- __Computational expense & Large number of GPs__: The method involves fitting $D$ GPs, making it scale as $\mathcal{O}(Dn^3)$. To my knowledge, is a uniquely large factor that other methods do not share (rather, they are $\mathcal{O}($D^2 + Dn^2 + n^3)$ as pointed out by the authors. This puts the main claimed benefit of the paper into question, as there should be tasks (pref. those with some multiple on the number of evaluations) where the proposed method is in fact one of the slower ones.
- __Inability to pick up semi-relevant dimensions__: VSBO does not seem to pick up on the semi-important dimensions very well based on the frequency plots. In particular, it struggles with the semi-active dimensions on Styb-Tang, despite the function's extreme steepness along the borders which should simplify matters. Does it pick up any semi-active dimensions (with the current hyperparameter values) for a function with a flatter surface like, say, Hartmann-3? . This could probably be addressed by varying the aforementioned hyperparameters, but I would ultimately want to see that it works as intended out of the box.
- __The algorithm is not digestible__: There is _a lot_ of material, including 5 algorithms totaling over two pages, to go through to get to VS-momentum (i.e. the actual algorithm that is used). To me, this appears to be a design flaw, and _hints_ at over-fitting the algorithm to the problem(s) at hand. Moreover, I want to see the performance of the non-momentum variant.


- __Missing comparison against relevant methods__: Two competing methods from NeurIPS-22, which both work with the (adaptive) axis-aligned subspace assumption, are missing (BaXUS [1],  MCTS-VS [2]). In particular, they are both very relevant with the chosen evaluation protocol. I recognize that these are relatively recent (more than 3 months since publication), but I ultimately do not they are recent enough to warrant exclusion. Since we on the topic of computational efficiency/scalability, TuRBO [3] should probably be in there as well. At the least, these should be included as related work. Having [1] (which uses some of the same benchmarks) and [2] included would improve my perception of the paper.

- __Relation of IS to lengthscales__:The proposed variable selection criterion _seems_ to be very correlated to the lengthscales of the model (except if short lengthscales & little data), and my intuition tells me that the lengthscales should suffice about as well for variable selection. Can the authors provide evidence to the contrary?
- __The MC approximation is not ablated__: The Monte Carlo approximation seems fairly sample-intensive, considering the large dimensionality of the problem. How many samples were used, and does the IS suffer if (relatively) few samples are used? 

# General feedback:
- __Improving the motivation for the method__: I greatly appreciate Fig. 4 and what it communicates (although I think a. and c. are unnecessary). I was not aware that the cost of optimizing the GP in high-dim is that significant, and I think it should be highlighted more. In fact, I think it should serve as a motivating example for the entire paper and be featured in the introduction. 
- __Excessive time-related plots__: Both wall-clock and CPU time seems extreme, and they are (naturally) just about identical. Please remove wall-clock time to make room for Styb-Tang. 
- __Plot the regret__: Please, plot the regret (best possible - current best), especially when you have introduced test new functions where readers may not know the optimal value by heart. 
-__CMA-ES blurs the evaluation: The inclusion of CMA-ES blurs the evaluation, especially under the proposed evaluation protocol. In fact, it appears that CMA-ES (and not the variable selection) is responsible for the optimization of the semi-active dimensions in 2/3 synthetic cases, since VSBO doesn't indenify these dimensions.


# Questions
_No, I don't expect experiments, but would greatly appreciate some form of clarity_!
- How susceptible is the proposed variable selection procedure to observation noise of various magnitudes?


# Minor stuff

- I'm missing the boundary issue (Swersky 2016) as a reason for high-dimensional BO being challenging. Would the authors agree?
- Plotting 1/8 standard __deviation__ seems odd. Why not plot the standard __error of the mean__?
- Fig. 4: The y-axis label "Wall clock time __/__ second" reads "Wall clock time __per__ second" and is very is unconventional, please go with "Wall clock time (seconds)"
- __the__ GP is missing almost everywhere, such as in line 73, 74 & 75.
- Algo 5, row 28: You've written () instead of {} in LaTeX.


1. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. Leonard Papenmeier, Luigi Nardi, Matthias Poloczek. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

2. Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization. Lei Song, Ke Xue, Xiaobin Huang, Chao Qian. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

3. D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek. Scalable Global Optimization via Local Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
review_summary: The idea of variable selection is good, the evaluation protocol is novel and practical, and the paper could potentially fill a need for computationally efficient high-dimensional BO with consistent performance. However, the algorithm does not appear intrinsically scalable $(\mathcal{O}(Dn^3))$ and it is very difficult to gauge how well the proposed variable selection scheme works, given:
- The use of a modified method, VS-momentum, in the results
- The hyperparameters of the method
- The functions that are benchmarked against

but the total list is even longer. Thus, I am not convinced that the proposed variable selection scheme has merit.

I recommend to reject the paper in its current format. I am confident in my assessment given the current evidence, but I am very open to changing my opinion if my critique is addressed.
actions_required_to_increase_overall_recommendation: I am very open to changing my score, and it greatly hinges on further assessing the quality of the variable selection. In an ideal world, that would entail:
- Showing the results (and frequency plots) of the non-momentum method, preferably with & without CMA-ES (doing RS instead)
- Showing how sensitive the (non-momentum) method is to the hyperparameters
- Ablate at which level (i.e. $0.1f, 0.2f, 0.3f$) variables start to become important for Hartmann-6 and Styb-Tang
 
Here, I view the first two as must-haves, and the rest being nice-to-haves. I don't expect it to perfectly pick up all relevant dimensions, but there is currently far too many confounding factors.

Further, I would like to see that the computational expense does not blow up (relative to the competition, given the unique (?) scaling factor) when the number of evaluations is increased by a factor $2-5$ - a common high-dimensional BO budget. I recognize that this is unrealistic for ALEBO and SAASBO.

Moreover, the inclusion of the aforementioned NeurIPS works would be nice to have.
review_title: null

Decision: Accept(Main Track)