Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper focuses on the research problem of self-adjusting hyper-parameters of Bayesian Optimization (BO) to balance the exploration and exploitation. 
The contibutions of this paper are: 
1. Clearly formulation of the problem, i.e., when and how should we adjust the hyper-parameters. 

2. The proposed method can self-adjust the important hyper-parameter of acquisition function.
overall_review: Pros

1. This paper is well-motivated and well-written.

Cons
1. The advantages of the current method is not significant enough. For example, the final rank of the proposed method is over 4 (of 11) on two types of test-beds.

2. More issues please refer to "Actions Required To Increase Overall Recommendation".
review_summary: This paper is well-motivated and well-written. The research problem is important. However, the proposed method is not novel and the experimental results are weak.
actions_required_to_increase_overall_recommendation: 1) More experiments on some non-synthetic tasks would further improve this work. 

2) More compared baselines, such as Multi-armed bandits.

3) I notice that the authors have leave the comparison with DAC (RL-based method) as the future work. However, RL is a popular and promosing method in recent work in this research area, and shows the better performance than the heuristic-based methods, especially in some black-box optimization methods [1, 2]. I believe just compare one simple method based on RL (e.g., DQN with action that increase, decrease, or keep $\alpha$) will significantlly strengthen the results.

[1] Learning Step-Size Adaptation in CMA-ES. PPSN, 2020.

[2] Multi-agent Dynamic Algorithm Configuration. NeurIPS, 2022.

4) Some theoretical analysis will also significantly strenghthen the results.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This work proposes an adaptive strategy to adjust the hyper-parameter in weighted Expected Improvement (WEI). The strategy is simple yet efficient.
overall_review: Strength:

++The proposed SAWEI is simple yet efficient.

Weakness:

-- I have one concern about generalizability. This method aims to adjust the hyper-parameter of the weighted EI acquisition function adaptively, and it can degrade to eighter pure exploration, EI, or PI. I am afraid that this method cannot integrate other acquisition functions, such as UCB, TS, Knowledge gradient, etc.

-- I have concerns about the settings of the hyper-parameter. 

1. Why did the authors fix the $\Delta \alpha$? How about adjusting it according to the UBR?

2. According to Fig. 6(a), it seems that $\epsilon=0.05$ has the best performance, then why the authors set  $\epsilon=0.1$ by default (in Line 166)?


review_summary: I am afraid that this method should be further refined so as to integrate more rich acquisition functions. Moreover, the settings of $\epsilon$ and $\Delta \alpha$ might be kind of groundless.
actions_required_to_increase_overall_recommendation: Please see the weakness.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This work suggests a new acquisition function, self-adjusting weighted expected improvement, in Bayesian optimization.  To control an ability to explore and exploit, it adds a new parameter to balance both exploration and exploitation.  In particular, such a parameter is adaptively adjusted.  The authors show some experimental results of diverse acquisition functions.
overall_review: I think that it proposes a novel acquisition function, which is very interesting.  However, the proposed acquisition function is somewhat heuristic.  I think that more detailed justification is required.

* I think that the authors need to do more exhaustive analysis on the proposed method.

* Baselines are unclear.  Please elaborate them.  In particular, please add a description of EI -> PI* (Linear) and PI* -> EI (linear).

* According to the previous work, the upper bound regret is used to determine when we terminate a Bayesian optimization round.  Why can it be employed to adjust exploration and exploitation?

* Could you explain how you implement Line 10 of Algorithm1?

* I think that Figures 3(a) and 3(c) are redundant.  If a space for contents is tight, it is okay to move them to the appendix.

* It can be a preference, but the y-axes of Figures 2, 4, and 5 should be flipped, i.e., higher ranks are upper.
review_summary: Please see the text boxes above.
actions_required_to_increase_overall_recommendation: I am on a positive side now.  I would like to hear how you will revise the paper and justify the proposed method.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper proposes a way to set a trade-off parameter $\alpha$ in a Bayesian optimization (BayesOpt) policy proposed in previous work.
Specifically, this parameter $\alpha$ controls the behavior of the resulting policy on the exploration/exploitation spectrum of Expected Improvement (EI) -- Probability of Improvement (PI), and the paper proposes to adjust the value of $\alpha$ based on optimization history in the following way.
First, we keep track of a metric called the Upper Bound Regret (UBR), which is the difference between a pessimistic estimate of the incumbent value and a lower bound on the lowest function value.
If this metric grows smaller as optimization progresses, it is an indication that optimization with the current setting is converging, and we might benefit by adjusting the value of $\alpha$.

To update the value of this parameter, we inspect the trade-off between exploration and exploitation in the most recent queries.
If recent queries have favored exploration, we adjust the parameter to bias exploitation and vice versa.
This strategy aims to counteract getting stuck at a local optimum from how we are currently balancing exploration and exploitation.

The authors then present comprehensive experiment results comparing the proposed algorithm against a wide range of benchmarks.
overall_review: The paper is well-structured and has a nice flow.
The proposed algorithm is natural and easy to understand: it involves adjusting the trade-off parameter in a direction opposite to recent behavior to avoid getting stuck at local optima.

The paper extends a previously proposed BayesOpt policy by designing a way to automatically set its trade-off parameter.
As such, I worry that the contributions are somewhat incremental.

As noted in the **Technical Quality And Correctness** portion, some of the details about the algorithm are not clear or missing from the manuscript.
This makes appreciating the proposed algorithm more difficult.
It's also somewhat disappointing that the paper doesn't mention theoretical guarantees for the proposed policy.
Given its connection to PI and EI, both of which have their own guarantees, I was hoping some results could be derived for SAWEI.

Finally, I believe the experiment section should include other commonly used BayesOpt policies to ensure that there's value in adopting the current approach.
review_summary: I recommend weak rejection.
While I think this paper proposes an interesting approach to the problem of controlling the behavior of WEI, I fear the contributions are limited and some of the crucial details are missing to make this work a strong submission.
actions_required_to_increase_overall_recommendation: I'd like to understand the exploration term in the acquisition function of SAWEI better, so some discussion on this will be appreciated.
Crucially, more details about how to compare recent exploration vs. exploitation and how the UBR is computed and tuned (by setting $\beta_t$) is needed.

Establishing some theoretical guarantee would help justifying the algorithmic design.
Including stronger baselines such as entropy search policies will make the experiment section more well-rounded.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper proposed a self-adjusting acquisition function for BO, based on EI and PI. It compares to a collection of other ways to combine EI and PI on both synthetic and HPO problems.
overall_review: The strengths of the paper are that it proposes a self-adjusting acquisition function, which they compare to a collection of related benchmarks. It is mostly clearly written, tested on a benchmark relevant to the AUTOML community, and engaged with the interesting question of what acquisition function to use. 

That being said, there are some important elements that are unclear, and which require fixing. It uses two different versions of WEI, which seems to make the baselines less comparable, and lacks details in how the attitude is tracked. The ablation study and the related work start out with claims of what they do that don’t match up with the rest of the paragraph.

Overall, the paper seems to be almost ready: the plots need some work to be more accessible and the writing needs to be clarified in parts. I expect the authors to be able to remedy most, if not all of these concerns in the rebuttal. And in that case the review will of course increase.

review_summary: It is an interesting paper, relevant to the AUTOML community, but which in its current form is unclear, even misleading in parts, and some details are missing. These parts can be remedied by clarifying the writing and plotting, and I expect to increase the score once this has been done.
actions_required_to_increase_overall_recommendation: These are listed in the above sections. In particular, clearing up the difference between the different types of WEI and addressing the concerns in clarity and accessibility.
review_title: null

Decision: Accept(Main Track)