Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper generalises Shannon entropy-based acquisition functions (AF) to a broader class of uncertainty measures. Doing so allows the authors to frame several popular AFs as special cases of the proposed entropy. The proposed AF is also able to provide customised solutions for a number of modified versions of BO setting. The authors study efficient optimization of the AF under certain smoothness conditions. Experimental evaluation compares the proposed method with several baselines.
strengths_and_weaknesses: Post-rebuttal:
I would like to thank the authors for answering my questions. Most of my concerns are addressed, so I will update my score to 6.
__________

The paper is well written and appears to be technically sound. The idea of unifying the large zoo of BO acquisition functions under the same umbrella is neat (even though it was previously mentioned in the literature). It is particularly interesting to see both information-based and decision-theoretic AFs to be reached as special cases of proposed entropy.

The authors mention a diverse set of AFs for custom tasks. However, some of these applications seem somewhat artificially crafted. While the authors do provide a number of references, it’s not always clear on how such problems are derived from these references (e.g., line 221, reference 30 leads to a whole PhD thesis). Furthermore, with such limited description it's not always obvious why these tasks still require BO as an expensive black box optimization, and not some classic optimization techniques. It would be helpful if the authors could shed some more light on the motivation behind these custom tasks.

The experiments are one of the weaker points of the paper. While they do cover a diverse set of tasks, the set of methods is surprisingly small and lacks some “must-have” classic BO baselines, such as PI or GP-UCB. Even more surprisingly, EI, which is mentioned in the main paper as a special case of proposed entropy, is missing in the experimental section.

questions: A few questions in addition to the concerns raised in the previous section:

Line 115: How does the coin example help to motivate that this is a reasonable measure of uncertainty?

Experiments section: How many trials did the authors average results over for the experiments?

How realistic is Eq 7 in general?

Have the authors considered whether probability of improvement (the simplest BO AF one could imagine) can be obtained under the proposed framework?

limitations: -
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors derive a generic acquisition function design strategy (EHIG) based upon a user-specified loss functions and action set applicable to a variety of custom tasks involving Bayesian optimization, viz., Top-k optimization with diversity, Multi-level set estimation and Sequence search.
EHIG includes as special cases both information-based acquisition schemes such as entropy search (ES) and decision theoretic acquisition schemes such as knowledge gradient (KG) and expected improvement (EI).
Empirical results on multiple datasets indicate the superiority of EHIG on custom tasks considered.
strengths_and_weaknesses: Strengths
1. The approach is clearly presented and well-motivated.

2. The proofs in the appendix seem to convincingly demonstrate how EHIG yields multiple well-known special cases such as ES, KG and EI.

3. Experimental validation is performed on a wide range of datasets.

Weaknesses

1. In the experimental comparison, only EHIG is tuned to the custom task and competing approaches are tuned to the conventional task of black box optimization (POM may be an exception in that it may be considered tuned for single level set estimation.)
So it appears to be a foregone conclusion that EHIG would outperform competitors.
It is unclear if any Bayesian Optimization approaches already exist for the custom tasks.

questions: 1. What distance function was considered for the diversity term in the experiments in Sec. 7 ? Was it Euclidean distance ? Did a parameter control the strength of the diversity penalty in eq. (4) ?

2. Did the authors only consider two levels (Multihills) and one level (Pennsylvania Night Light) for the multi-level set estimation experiments ?
Or were larger number of levels also considered ?

limitations: The authors discuss avenues for future work addressing implicit limitations in Sec 8.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces a family of acquisition functions (AFs) for Bayesian Optimization (BO) based on the decision-theoretic entropies, $H_{l,A}$-$\textit{entropy}$, which is a generalized version of Shannon entropy. The AFs can be tailed to select queries that maximize the reduction of uncertainty in $H_{l,A}$-$\textit{entropy}$, which is defined as $\textit{expected}$ $H_{l,A}$-$\textit{information gain}$ (EHIG). The EHIG is a general form of AFs and can be reduced to information-based AFs or decision-theoretic AFs by choosing the parameters, $l$ and $A$. A framework is provided for the applications of EHIG on several categories of problems. Moreover, a gradient-based acquisition optimization method is proposed. Finally, evaluations of the method are made on examples datasets.
strengths_and_weaknesses: Strengths:
1. This paper is well-organized and easy to read. 
2. This article proposes a brand new general AF which unifies two branches, information-based and decision-theoretic AFs, for BO. Following the framework to carefully construct the the EHIG AF for each task and using the proposed gradient-based optimization method, the proposed $H_{l,A}$-$\textit{Entropy Search}$ (HES) procedure has advantage over other baseline methods.

Weaknesses:
1. The experiments are carried out on tasks which no AF has been developed in other work. However, to make this paper more significant, there should be examples that compare the result of the proposed HES method to other traditional BO method with AF on well-studied tasks. In this way the readers can have a better idea on the performance of HES.


questions: 1. Does the Random Search (RS) method optimize any function? In top row of Fig.2 the negative loss of RS slowly increases comparing to HES, but from the description of RS (line 305) the samples are drawn randomly from the full domain, therefore, it is hard to understand why the negative loss increases. Is there an explanation?

2. In Sequence Search task only the route of HES is shown and discussed. Bases on the left of the Fig. 4 at least the Uncertainty Sampling has comparable performance on the negative loss to HES. It would be better if all routes can be shown on the right of Fig. 4 for a thorough and clearer comparison. 

3. Some of the plots are too small to see the details and axes, eg. bottom rows of Fig. 2 and Fig. 3.
limitations: There is no societal impact from this work. The authors should state and summarize the limitations clearly to the readers.
review_title: null

Meta-review: metareview: The paper proposed a novel acquisition function for BO, based on a generalization of Shannon entropy that enables one to incorporate problem-specific loss functions corresponding to a downstream task. The authors show that the proposed acquisition criterion generalizes a number of well-known BO acquisition functions, including EI/KG/ES/PES. A detailed training procedure for optimizing the acquisition function was discussed in the paper, and experimental results show that the proposed acquisition function with the optimization procedure performs well over a diverse set of tasks.

All reviewers agree that this paper is well written, and the idea of unifying a collection of “classical” BO acquisition functions is interesting. There were a few concerns about the sufficiency/significance of the experiments, mainly due to the (lack of) baselines considered in the tasks. The authors clarified the concerns by including preliminary runs of several new experiments, and highlighting that the proposed approaches were targeting novel tasks that went beyond the vanilla optimization tasks. There were no other critical concerns in the reviews. The authors are strongly encouraged to address the questions raised in the reviews when preparing a revision of this paper.

confidence: Certain
Decision: Accept