Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper provides a termination method for BO and shows that its works well for a variety of benchmark problems
overall_review: This work is nice because the derivation is clear and the conclusion is simple.
The weakness lies beside. The conclusion and method are somehow intuitive and a bit simple, in hindsight.
review_summary: I have done some work on BO and I find this paper interesting. If the presented method is solid and can be applied widely as it claims, it is a quite helpful work.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper introduces a new approach of early stopping Hyperparameter Optimizer with Bayesian Optimization. Specifically, the authors first estimate the upper bound of the optimization regret and then compare it with a threshold that is computed with the empirical variance of the sample variance of cross-validation. Empirical experiments show that the proposed method is able to early stop at the points that are sufficiently close to the global optima without overfitting the validation loss.
overall_review: Overall the idea of this paper is quite interesting. The first part of this paper is well established, the author starts by bounding the gap in generalization performance between the candidate. Then the authors show that the regret can be estimated with a surrogate model. Finally, the authors propose their termination condition for BO. All these parts are well established. However, the main drawbacks exist in the experiments and clarification parts (see my above comments), which makes the conclusion of this convincible.
review_summary: Overall the early-stopping idea introduced in this paper is quite interesting. The first part of this paper is well written and well-motivated However, the experiment parts are not clear enough to me. I hope that the authors could clarify some of my doubts.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper investigates an automated approach for stopping hyperparameter optimization. The idea is that when the statistical error in the empirical loss overwhelms the loss itself, then it doesn't make sense to continue optimizing because it becomes impossible to separate signal from noise.

To do this, the paper bounds the true regret in terms of the statistical error and the regret wrt the validation loss. The error can be estimated via cross validation while the validation regret can be bounded by a combination of UCB and LCB (confidence bounds).

For a given regret tolerance, the approach is shown to minimize regret within this tolerance 80-90% of the time, while avoiding spurious evaluations, thus saving computation.
overall_review: Overall I think this is an elegant and principled way to approach the problem of when to stop HPO. The paper was nice to read, especially the method/theory section.

However, practically, I think that the adaptivity of the method is not really captured in the experiments. The Conv-N baselines seem really strong. In particular, I think that increasing N to 75 or 100 may match the proposed approach across most if not all of the benchmark tasks. The paper would be much more compelling if there were HPO benchmarks where the approach worked well with a single threshold across the benchmarks, while the Conv-N baseline required lots of tuning. Otherwise, I am concerned that this may be too involved for people to use generally unless each trial was indeed extremely expensive.

I also think that the cross-validation point still stands. At least in the deep learning world, which is one of the biggest customers of HPO, cross-validation is not used very often because the models/datasets are typically so big and expensive that slightly better risk estimation does not justify the increase in computation.
review_summary: Very nice and elegant approach, well written, but perhaps there are some practical limitations that would impede adoption.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The paper proposes a new stopping criterion for Bayesian Optimization based on an estimation of the BO model's error and a termination threshold. For estimating the model error, their approach relies on the upper and lower confidence bounds derived in previous work, which is shown to be accurate with high probability. The termination threshold is automatically defined based on cross-validation variance where possible, or defined by a user otherwise. Experiments with multiple benchmarks and baselines are performed to show that the approach achieves a better trade-off between final solution performance and optimization time. As an additional contribution, the paper proposes a discussion on the potential overfitting potential of AutoML solutions.
overall_review: The paper proposes a neat new automatic stopping criterion for BO, especially for applications where cross-validation is feasible. The problem tackled is relevant and well-motivated. The paper is clear, well-structured, and the building blocks and final solution are well-presented. 

Additionally, there are some interesting results in the experiments section. Namely, the method is capable of achieving similar performance to a non-early stopping scenario, while also saving some compute time, with an easy-to-define (sometimes automatic) stopping criterion. When the threshold is user-defined threshold, the method often achieves the desired accuracy level.

My main concern about this paper lies on the experimental issues presented in the technical quality section. Most notably, it is not fully clear how the baselines were chosen for the experiments and how significant the results are, this makes it hard to judge how relevant the approach really is. At the very least, there needs to be a more thorough discussion on the relevance and significance of the results compared to the baselines. It would also be interesting to see more benchmarks where the CV automatic threshold is used.

At last, the paper lacks a more thorough related works discussion. 

review_summary: The paper is well-written and the proposed approach is interesting, this paper has potential, but I would like to see some of my concerns with the empirical evaluation addressed before it is accepted. I'm recommending that the paper is not accepted in the current state, but I'm willing to increase the score if the concerns are addressed.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper proposes an automatic early termination strategy for hyperparameter optimization.  Unlike a general setting of Bayesian optimization, hyperparameter optimization is able to access to a validation dataset as well as a test dataset.  Thus, this paper employs a validation dataset, which is used to measure a validation loss, in determining the automatic termination.  The key insight of this work is that the discrepancy between the true objective and the computable target suggests when we stop the optimization procedure.  The authors provide the empirical results on real-world hyperparameter optimization. Moreover, interestingly the authors find that overfitting may occur in the context of hyperparameter optimization.
overall_review: The authors argue that overfitting may occur in hyperparameter optimization in the last part of the paper, by providing the numerical results.  However, I am a bit convinced of these results and they should be more highlighted in the introduction.

Apart from this, I fully agree with the need for automatic early termination and the strategy proposed in this work is quite promising.  Although I am somewhat worried about using the sample variance of cross-validation in the termination, due to its computation cost and the implication of the sample variance of cross-validation, it is still a reasonable choice.  I would like to listen to thoughts on this concern in the rebuttal.  Please provide thoughts beyond the cases with manual thresholds.

Could I ask the authors how this strategy can be applied in general Bayesian optimization formulation?  If possible, this work is more impactful than the current version which focuses on hyperparameter optimization.

It is a minor issue, but in Figure 1 ``min`` and ``ucb`` should be regular fonts, not italic fonts.
review_summary: By utilizing available resources in hyperparameter optimization, it suggests an interesting and novel technique on automatic termination of hyperparameter optimization.  Therefore, I recommend the decision of this paper as acceptance.
review_title: null

Meta-review: metareview: 


This paper proposes a new automatic stopping criterion for Bayesian optimization (BO). This methods holds for applications where cross-validation is feasible. It is a problem that is relevant and well-motivated and the paper is well-presented. 

The referees had in the very first round of refereeing some questions on the experimental side section and other main question. Yet, the answers of the authors where quite promising and most referees were willing to increase their valuation of the review rating. 

In the end the referees all have a point 5 review rating (4 times a point 5 review rating) and one time a 4 review rating. This makes an 4.8 review rating and therefore close to the best possible. Also for the potential impact, the technical quality and the clarity rating they are all between the 3 and the 4 mean rating. 

My own reading matches the referees comments. I leave to the Senior Area Chairs the decisions on whether to accept with a highlight or without a highlight the paper, but I would insist in accept the paper in the conference proceedings.
 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept