Title: Batched Energy-Entropy acquisition for Bayesian Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: bayesian optimization

gaussian processes

acquisition

batch

parallel
Abstract: Bayesian optimization (BO) is an attractive machine learning framework for performing sample-efficient global optimization of black-box functions. The optimization process is guided by an acquisition function that selects points to acquire in each round of BO. In batched BO, when multiple points are acquired in parallel, commonly used acquisition functions are often high-dimensional and intractable, leading to the use of sampling-based alternatives. We propose a statistical physics inspired acquisition function that can natively handle batches. Batched Energy-Entropy acquisition for BO (BEEBO) enables tight control of the explore-exploit trade-off of the optimization process and generalizes to heteroskedastic black-box problems. We demonstrate the applicability of BEEBO on a range of problems, showing competitive performance to existing acquisition functions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new acquisition function BEBBO for batched BO. BEBBO tries to build (negative) free-energies-like acquisition function, enabling gradient-based optimization, tight exploration-exploitation control, and risk-averse BO under heteroskedastic noise. It tries to improve existing parallel acquisition functions in the following ways:\n1.\tuses a hyper-parameter T to directly balance exploration and exploitation by separating these two parts clearly;\n2.\tkeeps the behavior predictable by scaling E and I with batch size;\n3.\tenables the optimization of gradient descent by holding the availability of closed-form expressions for GP.\nThis paper demonstrates several experimental comparisons and shows its effectiveness.\n\nSTRENGTHS:\n1.\tThis paper shows an enlightening acquisition function method inspired by statistical physics. \n\n2.\tThe experimental results show the effectiveness of BEBBO on problems without noise or with heteroskedastic noise.\n\nWEAKNESSES:\n1.\tThe idea is straightforward: simply combine two common components—entropy reduction and the weighted sum of values. In my opinion, the novelty is not very strong. \n\n2.\tThe article doesn't discuss the situation when BEEBO is used with other surrogate models. It seems that if BEEBO is not used with GP, it loses the advantages of closed-form expressions and gradient descent optimization. \n\n3.\tIn control problems shown in Figure A2, the performances of meanBEBBO and maxBEBBO are not outstanding. Especially, these two variants are surpassed by KB in Robot pushing problem.\n\n4.\tAlthough BEEBO performs well on many synthetic test problems, its versatility and effectiveness require more experimental validation in specific applications.\n\n5.\tIn the experiments in main text, the authors only showed the comparison with q-UCB. It would be better to show comparisons with other batched baselines and provide a thorough analysis. The comparison with q-UCB shows the advantage on the balance between exploration and exploitation. But other advantages emphasized in the paper, such as the benefits of gradient descent optimization and the tight control of the explore-exploit trade-off, are not fully demonstrated. I suggest that the authors can re-organize the paper to move the comparison with other baselines to the main paper. \n\n6.\tThe theoretical analysis is not deep. No regret bound is analyzed.\n\nQUESTIONS:\n- How to set the parameter T properly? Do you have some instructions?\n\n- In line 158, why does I(x) also scale linearly with batch size? \n\n- The Equation A5 is not shown.\n\n- What are the advantages of multiplying batch size Q in E(X)?\n\n- How does the behavior of BEBBO change with batch size?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProposing a new acquisition function inspired by statistical physics, which allows explicit control of exploration-exploitation trade-offs in a batch BO setting.\n\nSTRENGTHS:\nDrawing inspiration from statistical physics is a promising direction, as it naturally aligns with Bayesian approaches.\n\nWEAKNESSES:\n**Major Points**\n1. **Lack of Unique Selling Point:** \n   The method does not appear to solve any unique cases that other methods cannot. While the related work section outlines many similar approaches, this work only compares itself to q-UCB. Without a theoretical study, such as convergence rate analysis, there is insufficient motivation to adopt another heuristic approach like this. To demonstrate efficacy, a comprehensive empirical study with extensive comparisons to existing works is necessary, given the unclear advantages.\n\n2. **Review of Claimed Selling Points:**\n   - **Not Based on MC Integration Like BoTorch:** \n     While this is true, it is unclear if it is beneficial. MC approaches are approximations but have convergence guarantees (refer to the BoTorch paper's appendix). This work lacks such guarantees.\n   - **Tight Control of Exploration-Exploitation Trade-off:** \n     The proposed method is not the only solution. UCB theory can bound the feasible region by [max LCB(x), UCB(x)] with 1 - $\\delta$ probability (e.g., see [1]). This region can be controlled by $\\beta$ hyperparameters, corresponding to $\\delta$. Constrained batch BO within this region would yield similar results with theoretical guarantees.\n   - **Heteroskedastic BO:** \n     There are no comparisons with existing methods. UCB variants can address this problem. Vanilla UCB theory does not differentiate between epistemic and aleatoric uncertainty. Therefore, UCB with heteroscedastic GP can serve the same purpose. For example, training a heteroscedastic GP with the observed dataset and replacing the inner GP on noise variance with normal isotropic likelihood variance (pure epistemic uncertainty) will yield similar results as risk-averse batch BO, without needing the change in modelling the acquisition functions regardless of hetero-/homo-scedastic cases like this work.\n\n3. **Setting k in Practice:** \n   Setting  $k$ is not an easy task for users. In UCB, $\\beta$ presents a similar challenge, but there are theoretical guidelines and algorithm to compute this (e.g., [2]).\n\n4. **Constrained Uncertainty Sampling:** \n   This work can be understood as a variant of constrained uncertainty sampling. As [3] explains, variance reduction can be written similarly to the entropy proposed in this paper (see section 2.2). It also shows that the variance-only approach is inferior to UCB both theoretically and empirically. The batch setting may lead to model misspecification, particularly when hallucination (aka fantasization) is applied. The concerns and approach are notably similar to ([49] in your citation number), making a comparison with their method unavoidable.\n\n5. **Data Augmentation Procedure:** \n   The explanation is unclear. Is it fantasization (aka hallucination) or simply observed points? How does this differ from Eq.(3) in [3]?\n\n- [1] Fengxue Zhang, Jialin Song, James C Bowden, Alexan- der Ladd, Yisong Yue, Thomas Desautels, and Yuxin Chen. Learning regions of interest for bayesian optimiza- tion with adaptive level-set estimation. In International Conference on Machine Learning, pages 41579–41595. PMLR, 2023.\n- [2] Kihyuk Hong, Yuhang Li, and Ambuj Tewari. An optimization-based algorithm for non-stationary kernel bandits without prior knowledge. In International Conference on Artificial Intelligence and Statistics, pages 3048–3085. PMLR, 2023.\n- [3] Srinivas, N., Krause, A., Kakade, S., & Seeger, M. (2010). Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design. In Proceedings of the 27th International Conference on Machine Learning (pp. 1015-1022).\n\nQUESTIONS:\nQuestions are written in the above weakness section.\n\nLIMITATIONS:\nLimitations are discussed in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new approach to batch Bayesian optimization that explicitly trades off between exploration and exploitation via energy and entropy terms. The method is able to efficiently generate large batches for evaluation that outperform comparable methods for Bayesian optimization.\n\nSTRENGTHS:\nThe method is novel and well-motivated. \n\nI found the analysis in Appendix B to be especially strong, in which the proposed method BEEBO is compared to other methods for batch Bayesian optimization. This analysis shows the originality of the method and gives strong context for it.\n\nThe paper is clearly written and easy to read. The appendix was especially useful and had many valuable parts.\n\nThe analysis of heteroskedastic noise was great to see and shows a useful and understudied setting where the method is especially valuable.\n\nWEAKNESSES:\nI think the method is interesting and will be of value to the field. However, I do not find that the experimental evaluation of the method provides full support for the claims of the paper.\n\n**Issue 1: Evaluation limited to large-batch setting**\n\nThe major issue is that the paper claims the method is for general batch Bayesian optimization problems, without any qualifiers that I can see. The experiments all use q=100, which is a large batch size. Smaller batch sizes are often of interest too, e.g. batch sizes of 5, 10, and 50 in the GIBBON paper. The setting q=100 used here is also used in the TURBO paper (Eriksson et al. 2019) where it is described as a \"large batch size.\" \n\nGiven the experiment results in the paper, I don't know if this method will perform well for small- or medium-sized batches. Thus, either the experiments need to be expanded to include experiments with batch sizes such as 5 and 10, or the framing of the paper needs to be adjusted to emphasize that the method is specifically for large-batch Bayesian optimization, not general batch BO problems.\n\nThis issue also relates to the choice of baselines. The experiments only explore large-batch settings, where GIBBON (as the paper notes) is known to perform poorly. If the paper wants to claim that it performs better than GIBBON in general, then it needs to make that comparison on batch sizes of 5 and 10. If the paper wants to claim superiority only on large-batch settings, then that's fine to only use q=100, but then it needs to compare to state-of-the-art for large batch. Thompson sampling is a popular method for large-batch settings which is included as a baseline, but to my knowledge the state-of-the-art for large-batch BO is TURBO (Eriksson et al. 2019). In fact the q=100 and the robot pushing and rover trajectory problems are all exactly as in the TURBO paper, so its inclusion as a baseline is pretty obvious and, I think, necessary.\n\n**Issue 2: Lack of statistical significance**\n\nThe results of the experiments do not appear to be statistically significant. The main results given in the main text are tables, and these tables do not have confidence intervals. The only place where uncertainty in the results is show are the figures in the appendix, and there the confidence intervals appear to overlap in most cases. This is due to the use of only 5 replicates. I appreciate that these experiments are costly to run since they are using 1000 iterations, nevertheless the lack of statistical significance in most of the results provides weak support for the claim that BEEBO is actually better, vs. what we're seeing just being noise in the experiments. The paper needs some form of statistical analysis to convince the reader that what we're seeing is not just noise in the experiments. The best way to do this would be to include confidence intervals in tables 2 and 3, and then increase the number of replicates as necessary to achieve statistical significance in the differences in means. I do not feel it appropriate to highlight the method as being \"best value\" when it is possible that the confidence interval for that value contains the values of the other methods.\n\nQUESTIONS:\nThe issues raised above can be addressed by running more experiments (batch sizes 5 and 10; TURBO; more replicates to get reasonable CIs for the results tables). But it will probably require more experiments that can be run in the rebuttal period. Do we expect the method to work well for Q=5 and Q=10?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This work introduces a batched acquisition function that balances exploration and exploitation by using a weighted sum of mutual information and expected value, with the weights defining the trade-off. The discussion links the proposed algorithm to UCB and asserts that it naturally addresses heteroskedastic noise.

STRENGTHS:
1. The proposed acquisition function and its optimization and approximation methods are straightforward and practical. 
2. The paper provides extensive empirical results to illustrate the proposed algorithm's efficiency.

WEAKNESSES:
1. The introduced parameter controlling the trade-off lacks interpretation as in previous methods.
2. The completeness of the related work discussion is concerning. This is potential because the summarization lacks high-level extraction of the design of the algorithm, and the focus of the paper is, to some extent, scattered。

QUESTIONS:
One concrete example of the second aforementioned weakness is the criticism of MC approximation in high-dimensional applications. Recent advancements in applying MCMC address this issue in a principled manner and might be of interest. 

***Reference:***

Yi, Zeji, Yunyue Wei, Chu Xin Cheng, Kaibo He, and Yanan Sui. "Improving sample efficiency of high dimensional Bayesian optimization with MCMC." arXiv preprint arXiv:2401.02650 (2024).

LIMITATIONS:
Discussed above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13