Title: “Why Not Looking backward?” A Robust Two-Step Method to Automatically Terminate Bayesian Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian Optimization

Termination Criterion

Looking Backward
Abstract: Bayesian Optimization (BO) is a powerful method for tackling expensive black-box optimization problems. As a sequential model-based optimization strategy, BO iteratively explores promising solutions until a predetermined budget, either iterations or time, is exhausted. The decision on when to terminate BO significantly influences both the quality of solutions and its computational efficiency. In this paper, we propose a simple, yet theoretically grounded, two-step method for automatically terminating BO. Our core concept is to proactively identify if the search is within a convex region by examining previously observed samples. BO is halted once the local regret within this convex region falls below a predetermined threshold. To enhance numerical stability, we propose an approximation method for calculating the termination indicator by solving a bilevel optimization problem. We conduct extensive empirical studies on diverse benchmark problems, including synthetic functions, reinforcement learning, and hyperparameter optimization. Experimental results demonstrate that our proposed method saves up to $\approx 80\%$ computational budget yet is with an order of magnitude smaller performance degradation, comparing against the other peer methods. In addition, our proposed termination method is robust in terms of the setting of its termination criterion.

'Why Not Looking backward?' A Robust Two-Step Method to Automatically Terminate Bayesian Optimization

Abstract

Bayesian Optimization (BO) is a powerful method for tackling expensive blackbox optimization problems. As a sequential model-based optimization strategy, BO iteratively explores promising solutions until a predetermined budget, either iterations or time, is exhausted. The decision on when to terminate BO significantly influences both the quality of solutions and its computational efficiency. In this paper, we propose a simple, yet theoretically grounded, two-step method for automatically terminating BO. Our core concept is to proactively identify if the search is within a convex region by examining previously observed samples. BO is halted once the local regret within this convex region falls below a predetermined threshold. To enhance numerical stability, we propose an approximation method for calculating the termination indicator by solving a bilevel optimization problem. We conduct extensive empirical studies on diverse benchmark problems, including synthetic functions, reinforcement learning, and hyperparameter optimization. Experimental results demonstrate that our proposed method saves up to ≈ 80% computational budget yet is with an order of magnitude smaller performance degradation, comparing against the other peer methods. In addition, our proposed termination method is robust in terms of the setting of its termination criterion.

1 Introduction

' Nature does not hurry, yet everything is accomplished. ' -Lao Tzu

In this paper, we consider the black-box optimization problem (BBOP) defined as follows:

1

4

7

10

13

16

19

0

1

2

3

4

5

·

10

-

2

Number of FEs

κ

EI

1

4

7

10

13

16

19

0

0.3

0.6

0.9

1.2

1.5

Number of FEs

κ

diff

Ackley

1

4

7

10

13

16

19

0

0.1

0.2

0.3

0.4

0.5

Number of FEs

κ

PI

where x = ( x 1 , · · · , x n ) ⊤ is a decision vector (variable), Ω = [ x L i , x U i ] n i =1 ⊂ R n represents the search space, and f : Ω → R corresponds to the attainable set in the objective space. In realworld scenarios, function evaluations (FEs) of f ( x ) can be costly, giving rise to expensive BBOPs. Bayesian optimization (BO) has emerged as one of the most effective methods for addressing expensive BBOPs. BO is a sequential model-based optimization technique consisting of two iterative steps: i ) employing limited expensive FEs to construct a surrogate model of the physical objective function, such as a Gaussian process (GP) model [35]; and ii ) selecting the next point of interest for costly FE by optimizing an acquisition function, e.g., probability of improvement (PI) [18], expected improvement (EI) [16], and upper confidence bound (UCB) [31]. Numerous theoretical and methodological advancements have been made in BO. Interested readers can refer to comprehensive survey papers [29, 11] and a recent textbook [13] for further information.

Nevertheless, the question of when to terminate the search process of BO remains a largely underexplored area in the literature. At present, the most prevalent termination criterion is a pre-specified budget, such as the number of FEs or wall-clock time. Though intuitive, this approach neglects the search dynamics inherent to different BBOPs. As a result, this strategy is rigid while it does not offer a general rule for determining an appropriate budget across various problem settings. If the budget is too small, BO may terminate prematurely, yielding a suboptimal solution. On the contrary, an excessive budget may lead to wasted computational resources. Another simple termination method involves stopping BO if the current best solution remains unchanged for a predetermined number of consecutive FEs. However, as highlighted by [24], this strategy also fails to consider the observed data during the sequential model-based optimization process and relies on a pre-defined threshold.

Beyond the aforementioned 'naïve' approaches, a limited number of dedicated efforts have been made to address the termination of BO. One notable method involves monitoring the progress of BO by termination indicators, such as the maximum of EI [28, 16] or PI [22]. In this approach, BO is terminated when the corresponding termination indicator falls below a pre-specified threshold. Very recently, Makarova et al. proposed using the difference between the minimal of the lower confidence bound (LCB) and UCB as the termination indicator. As illustrated in Figure 1, we observe that all criteria used in these termination approaches exhibit significant oscillation during the optimization process. This can be attributed to: i ) the stochastic nature of BO itself, and ii ) numerical errors arising from the non-convex optimization of acquisition functions. Furthermore, as shown in Figures 1(a) and (b), the variation range of the same criterion can differ substantially when addressing problems with distinct fitness landscapes. These factors make determining a universally applicable threshold in practice challenging, resulting in fragile and less intuitive termination criteria compared to simply establishing a budget. Additionally, we find that these termination criteria are 'myopic', as decision-making is based solely on the observations at the current step, leading to a lagged termination. For instance, consider the selected samples shown in Figure 2; it is difficult, if not impossible, to determine when to terminate BO until t = 20 . However, if we look backward to t = 5 , it becomes evident that BO is likely to converge by t = 10 .

Our contributions. In light of the aforementioned challenges, this paper proposes a novel termination method for BO that proactively detects whether the search is located in a convex region of -f ( x ) by examining previously observed samples. BO is terminated if the local regret within this convex region falls below a predetermined threshold. To improve numerical stability, we introduce an approx-

-

20

-

10

0

10

20

-

20

-

15

-

10

-

5

0

5

x

(

t

= 20)

-

20

-

10

0

10

20

-

20

-

15

-

10

-

5

0

5

x

(

t

= 10)

-

20

-

10

0

10

20

-

20

-

15

-

10

-

5

0

5

x

(

t

= 5)

-

f

(

x

)

Predictive by GP

True function

imation method for calculating the termination indicator by solving a bilevel optimization problem. Our proposed termination method is simple, yet it offers theoretical guarantees. To demonstrate its effectiveness, we compare the performance of our proposed method against four peer methods on a variety of benchmark problems, encompassing synthetic functions, reinforcement learning, and hyperparameter optimization.

2 Proposed Method

This section starts with a gentle tutorial of vanilla BO. Then, we delineate the implementation of our proposed termination method, followed by a theoretical analysis at the end.

2.1 Vanilla Bayesian Optimization

As a gradient-free optimization method, BO comprises two major steps. The first step involves constructing a surrogate model based on GP to approximate the expensive objective function. Given a set of training data D = {⟨ x i , f ( x i ) ⟩} N i =1 , GP learns a latent function g ( x ) , such that ∀ x ∈ D , we have f ( x ) = g ( x ) + ϵ , where ϵ ∼ N (0 , σ 2 ϵ ) is an i.i.d. Gaussian noise. For each testing input vector z ∗ ∈ Ω , the mean and variance of the target f ( z ∗ ) are predicted as follows:

where X = ( x 1 , · · · , x N ) ⊤ and f = ( f ( x 1 ) , · · · , f ( x N )) ⊤ . k ∗ is the covariance vector between X and z ∗ , and K is the covariance matrix of X . In this paper, we use the Matérn 5 / 2 kernel as the covariance function to measure the similarity between a pair of data points. The second step consists of an infill criterion based on the optimization of an acquisition function, which determines the next point of merit ˜ x ∗ to be evaluated by the actual expensive objective function:

where f acq ( x ) = µ ( x ) + ωσ ( x ) is the widely used UCB [31] to facilitate our theoretical analysis. Specifically, the parameter ω > 0 , determined according to the confidence level set as 0 . 95 in this paper, controls the trade-off between exploration and exploitation. Subsequently, the next point of merit ˜ x ∗ is used to update the training dataset as D = D ⋃ { ˜ x ∗ } , and BO iterates between the two aforementioned steps sequentially until a termination criterion is met. The convergence of BO can be evaluated by regret:

where x ⋆ represents the ground truth global optimum and ˜ x ⋆ = argmax x ∈D f ( x ) denotes the current best-found solution.

2.2 Proposed Termination Criterion

Inspired by the observations illustrated in Figure 2, we propose a termination method that involves 'looking back' at the last τ > 1 observed points in the dataset D , and storing these in a temporary archive, denoted as ˜ D . The termination criterion we propose is predicated on two primary conditions.

Condition 1. The BO search process is deemed to have converged within a convex hull ˜ Ω if the following condition is satisfied:

where ✶ ( · ) denotes the indicator function, returning 1 if the argument holds true and 0 otherwise. x and x ′ are points selected randomly and distinctively from ˜ D . The convex hull, ˜ Ω = [˜ x L i , ˜ x U i ] n i =1 , is a subset of Ω , where ˜ x L i = argmin x ∈ ˜ D x i and ˜ x U i = argmax x ∈ ˜ D x i .

Condition 2. Assuming Condition 1 is satisfied, and ˜ x denotes the most recently observed point in D , we calculate the local regret ˜ r as follows:

where ˙ x = argmax x ∈ ˜ Ω µ ( x ) and ¨ x = argmax x ∈ ˜ Ω σ 2 ( x ) . The BO process terminates if the following inequality is satisfied:

where ˜ r ωσ ϵ is used as the termination indicator, denoted as κ lb , and η lb is a predetermined threshold.

Remark 1. The inequality within the indicator function ✶ ( · ) in equation (5) is derived from Jensen's inequality [4], which yields a convex function:

where α ∈ [0 , 1] and x , x ′ ∈ ˜ Ω . In order to avoid the necessity of additional function evaluations when computing f ( x + x ′ 2 ) , we substitute µ ( x + x ′ 2 ) into equation (5).

Remark 2. In equation (6), we employ the widely-used L-BFGS algorithm [6] to compute ˙ x and ¨ x . To ensure numerical stability, we suggest the following strategies for initializing the algorithm and defining its termination criterion:

For ˙ x , L-BFGS is initialized at a point randomly selected from ˜ Ω . The algorithm terminates when ∥ ▽ µ ( x ) ∥ 2 ≤ λ . In our work, we set λ = 10 -6 , following Proposition 1.

For ¨ x , L-BFGS is initialized at the point argmax x ∈ ˜ Ω σ 2 ( x ) , where σ 2 ( x ) denotes the lower

bound of σ 2 ( x ) over ˜ Ω . The termination criterion is ∥▽ σ 2 ( x ) ∥ 2 ≤ λ , as per Proposition 2.

Remark 4. When the GP model is overfitting, BO tends to converge within the local region of the current best solution. In this case, both Condition 1 and Condition 2 are easily met while BO will be terminated prematurely. On the other hand, when the model is underfitting, BO will explore Ω in a random manner. In this case, satisfying Condition 1 becomes challenging, and BO will face the risk of failing to be terminated. Therefore, we designed three mitigation strategies: 1) restrict the lengthscale to [0 . 05 , 200] during GP training to prevent lengthscales from becoming excessively large or small; 2) normalize the input of training data to [0 , 1] ; and 3) standardize the output of the training data by centering it on the mean and scaling it by the variance.

Remark 3. Considering equation (7), given that µ ( ˙ x ) -µ (¨ x ) ωσ ϵ ≥ 0 and σ ( ˙ x )+ σ (¨ x ) σ ϵ ≥ 2 , we deduce that η lb ≥ 2 . The upper bound of η lb is empirically determined, as detailed in Section 4.1.

Proposition 1. Consider ∀ x ∈ ˜ Ω , where -µ ( x ) represents a convex function. If ∥ ▽ µ ( x ) ∥ 2 ≤ λ , we can establish:

where λ = (2 m 1 ξ ) 1 / 2 , ξ is a positive constant, and m 1 denotes the strong convexity parameter of -µ ( x ) [4].

Lemma 1. Assume the GP employs a stationary kernel k ( · , · ) . For ∀ x ∈ ˜ Ω , the lower bound of σ 2 ( x ) is given by:

where c < 0 is a constant and x i ∈ D for i ∈ { 1 , · · · , |D|} .

Lemma 2. Given Lemma 1, determining argmax x ∈ ˜ Ω σ 2 ( x ) is equivalent to solving the following bilevel optimization problem:

̸

where ˆ Ω = [ˆ x L i , ˆ x U i ] n i =1 ⊂ ˜ Ω , ˆ x L i = min( x 1 i , x 2 i ) and ˆ x U i = max( x 1 i , x 2 i ) . Given that the lower-level optimization can be addressed via exhaustive search, the analytical solution of (11) is given by ˆ x = (ˆ x L 1 + ˆ x U 1 -ˆ x L 1 2 , · · · , ˆ x L n + ˆ x U n -ˆ x L n 2 ) ⊤ .

Proposition 2. Leveraging Lemma 2, suppose minimize x ∈ ˜ Ω -σ 2 ( x ) exhibits convexity in its local optimal regions, the following inequality is satisfied when ∥ ▽ σ 2 ( x ) ∥ 2 ≤ λ :

where λ = (2 m 2 ξ ) 1 / 2 , ξ > 0 , m 2 > 0 represents the strong convexity parameter of -σ 2 ( x ) in its local optimal regions [4], and β is constrained by 0 ≤ β ≤ σ 2 (¨ x ) -σ 2 (ˆ x ) .

2.3 Theoretical Analysis of the Proposed Termination Criterion

In this subsection, we delve into the theoretical underpinnings of the proposed termination method, focusing on the convergence of BO when the UCB is utilized as the acquisition function.

Lemma 3. As per Srinivas et al. , the optimization process in BO can be conceptualized as a sampling process from a GP. Hence, for any x ∈ Ω , we have:

where δ > 0 signifies the confidence level adhered to by the UCB.

Corollary 1. Based on Lemma 3 and Condition 2, we deduce that:

where ε is a numerical error when optimizing the acquisition function, ˜ x ⋆ = argmax x ∈ Ω f acq ( x ) , and x ⋆ represents the true global optimum. Furthermore,

where ˙ x , ¨ x , and ˜ x ⋆ are elements of ˜ Ω , while δ > 0 denotes the confidence level of the UCB.

Theorem 1. Leveraging Corollary 1, when employing the termination method proposed in this paper, we deduce that the global regret bound of BO as:

where δ > 0 signifies the confidence level associated with the UCB.

Theorem 2. Building upon Condition 1 and Condition 2, and employing the termination method proposed in this paper, we establish the local regret bound of BO as:

where x ∈ ˜ Ω , x ⋆ denotes the true global optimum in ˜ Ω , and δ > 0 is the confidence level of the UCB.

Remark 5. Drawing from Theorem 1 and Theorem 2, we observe that if ε can be considered negligible when ˜ x ⋆ is accurately determined by optimizing the UCB, ˜ r subsequently represents the upper bound of BO regret within the domain Ω . Conversely, if ε cannot be disregarded, ˜ r is posited as the upper bound of BO regret within the restricted domain ˜ Ω .

3 Experimental Settings

In this section, we present the experimental setup for our empirical study, which encompasses the benchmark test problems, the peer algorithms, and the performance metrics used for evaluation.

3.1 Benchmark Problems

We evaluate the performance of our proposed method on three types of benchmark problems.

Synthetic functions: We consider Ackley, Levy, and Schwefel functions [33] with n ∈ { 2 , 5 , 10 } . The objective function f ( x ) is contaminated by Gaussian noise ζ ∼ N (0 . 0 , 0 . 2) . The maximal number of FEs is set to N FE = 50 n , with 5 n allocated to initialization.

Reinforcement learning (RL): We examine two RL tasks chosen from OpenAI Gym [5]: Lunar Lander with n = 12 and Swimmer with n = 16 . We set N FE = 50 n , with 5 n FEs allocated to initialization.

Hyperparameter optimization (HPO): We consider 5 HPO tasks picked up from the HPOBench [9] for tuning support vector machine (SVM) with n = 2 , multi-layer perceptron (MLP) with n = 5 , random forest with n = 4 and XGBoost with n = 8 . The computational budget is set the same as in the RL tasks.

Note that, due to the use of termination criteria, it may not be necessary to exhaust the entire allocated computational budget to terminate BO. To ensure statistical significance, each experiment is independently conducted 21 times with different random seeds.

3.2 Peer Algorithms

As discussed in Section 1, the termination criterion for BO is an understudied topic in the literature. In our experiments, we compare our proposed method with the following four termination methods.

Naïve method: This method ceases BO when ˜ x ⋆ stays unchanged for κ n consecutive iterations. Here, κ n is also the termination indicator. In our experiments, we test three settings of the thresholds η n as 150 , 337 and 524 , respectively.

Nguyen's method [28]: In each iteration of BO, the optimization of acquisition function produces the current optimal EI. By using this as the termination indicator, denoted as κ EI , the Nguyen's method terminates BO when it falls below a predetermined threshold η EI . In our experiments, we consider three settings of η EI as 0 . 01 , 0 . 04 and 0 . 06 , respectively.

Lorenz's method [22]: Analogous to the Nguyen's method, the Lorenz's method replaces the EI with PI as the termination indicator, denoted as κ PI . In our experiments, the termination threshold η PI is set as 0 . 07 , 0 . 2 and 0 . 33 , respectively.

Makarova's method [24]: Similar to the previous two methods, the Makarova's method uses the difference between the lower and upper confidence bounds as the termination indicator, denoted as κ diff . It terminates BO when κ diff ≤ η diff , a predetermined threshold and is set as 0 . 26 , 0 . 62 and 0 . 97 , respectively, in our experiments.

Our proposed method: According to Condition 1 and Condition 2, our proposed method terminates BO when κ lb falls below a predetermined threshold η lb , which is set as 2 . 02 , 2 . 05 and 2 . 08 , respectively. Furthermore, we introduce a hyperparameter τ to control the number of observed samples being looked backward, which is set to τ = 10 in our experiments. The code is available at https://github.com/COLA-Laboratory/OptimalStoping_ NeurIPS2023 .

According to the aforementioned settings, it is evident that the naïve method tends to delay termination when a large η n is used. On the other hand, other methods may incur a delayed termination if a small threshold is used. Note that the choices of the corresponding termination thresholds and the sensitivity of τ are empirically examined in Sections 4.1 and 4.2.

3.3 Performance Metrics

In our experiments, we consider the following three performance metrics to measure the effectiveness of a termination method.

Empirical cumulative probability of a termination indicator:

where ˜ κ i = κ + (¯ κ -κ ) × i N FE × 21 , and i ∈ { 0 , · · · , N FE × 21 } . For a given termination method, κ represents its termination indicator as outlined in Section 3.2. The minimum and maximum

1.4

1.6

1.8

2

2.2

0

0.1

0.2

0.3

0.4

0.5

˜

κi

of (a) method

I

cdf

Ackley

Schwefel

Swimmer

XGBoost

Random Forest

SVM

MLP

-7

-6

-5

-4

-3

-2

-1

0

0

0.1

0.2

0.3

0.4

0.5

˜

κi

100 of (b) method

0

0.03

0.06

0.09

0.12

0

0.1

0.2

0.3

0.4

0.5

˜

κi

of (c) method

0

0.1

0.2

0.3

0.4

0.5

0

0.1

0.2

0.3

0.4

0.5

˜

κi

of (d) method

0

0.5

1

1.5

2

2.5

0

0.1

0.2

0.3

0.4

0.5

˜

κi

of (e) method

×-

(a) Proposed method

1

1.02

1.04

1.06

1.08

˜

κ

i

˜

κ

min

i

0.05

0.1

0.2

0.3

0.4

0.5

(b) Na¨ ıve method

3

4

5

6

7

0

2

4

6

10

2

10

9

10

16

10

23

10

2

10

9

10

16

10

23

values of κ , represented by κ and ¯ κ respectively, are determined across all 21 repeated experiments on each benchmark problem. If I cdf exhibits consistency across a range of benchmark problems, it implies that the threshold choice for the corresponding termination method is consistent and not dependent on the specific problem.

The relative computational cost:

where ˜ N FE is the number of FEs used by a termination criterion when early stopping occurs. A lower value of I cost indicates a higher degree of computational budget saving.

The relative performance degradation incurred by early stopping:

where ¯ x and x are the best and the worst solutions found by BO when consuming all N FE FEs. ˜ x ⋆ signifies the best solution found when early stopping is prompted by a termination criterion. A smaller I perf value indicates less performance degradation resulting from the application of the corresponding termination criterion.

4 Empirical Studies

In this section, our experiments 2 aim to investigate three aspects: i ) the robustness of the termination threshold for different termination methods; ii ) the trade-off between the computational budget saving versus the performance degradation; and iii ) the sensitivity of τ in our proposed termination method.

4.1 Robustness of the Selection of Termination Threshold

In this subsection, we use the I cdf metric to scrutinize the threshold choice of various termination methods across different problems. As per equation (18), it is evident that I cdf ∝ ˜ κ i . As discussed earlier in Section 3.2, a large ˜ κ i can lead to premature early stopping. Consequently, we confine our analysis to instances where I cdf ≤ 0 . 5 . As shown in Figure 3, the trajectories of I cdf for our proposed method appear to converge, whereas those for the other methods diverge with different magnitudes. More specifically, as shown in Figure 3(a), ˜ κ i = 2 can be regarded as a transition point where I cdf ≥ 0 . 95 if ˜ κ i ≥ 2 . This empirical observation corroborates the theoretical result derived

25

50

75

100

2

3

4

5

Number of FEs

κ

lb

(a) Proposed method

Ackley

SVM

25

50

75

100

0

20

40

60

80

Number of FEs

κ

n

(b) Na¨ ıve method

25

50

75

100

0

0.1

0.2

0.3

0.4

0.5

Number of FEs

κ

EI

25

50

75

100

0

0.1

0.2

0.3

0.4

0.5

Number of FEs

κ

PI

25

50

75

100

0

2

4

6

8

Number of FEs

κ

diff

(a)

(b)

(c)

(d)

(e)

-

0.1

0

0.1

0.2

0.3

I

perf

(a)

(b)

(c)

(d)

(e)

-

0.2

0.2

0.6

1

1.4

I

cost

η

1

η

2

η

3

1

20

40

60

80

100

0

3

6

9

12

Number of FEs

r

Ackley

Proposed mehod

Na¨ ıve method

Nguyen's method

Lorenz's method

1

100 200 300 400 500 600

0

50

100

150

200

250

Number of FEs

1

50

100

150

200

250

0

0.1

0.2

0.3

0.4

Number of FEs

MLP

1

200

400

600

800

0

100

200

300

Number of FEs

Swimmer

1

100

200

300

400

0

0.1

0.2

0.3

0.4

Number of FEs

XGBoost

in Condition 2. In contrast, there do not exist a consistent lower bound for the other termination methods. To further elucidate these observations, we plot the distributions of ˜ κ i when I cdf ranges from 0 . 05 to 0 . 5 in Figure 4. It is clear that the bar charts exhibit the least variation for our proposed method. For the naïve method, ˜ κ i increases as I cdf grows. However, the bars for the other three methods show significant fluctuations, particularly for the Nguyen's and the Lorenz's methods. These observations are further substantiated by the trajectories of the termination indicators throughout the BO process, as shown in Figure 5. We present results for the Ackley and HPO for SVM problems here, while complete results are available in the supplementary document. These plots reveal that the trajectories for our proposed method converge to a certain threshold, while those for the other methods not only diverge but also differ significantly on different problems. Based on this discussion, we use I cdf = 0 . 05 as the capping point to guide the selection of the termination threshold for different termination methods: η lb ∈ [2 , 2 . 1] , η n ∈ [57 , 617] , η EI ∈ [3 . 8 × 10 -24 , 0 . 08] , η PI ∈ [2 × 10 -21 , 0 . 39] , η diff ∈ [0 . 09 , 1 . 15] . In our experiments, we apply the Latin hypercube design method [26] to choose three settings as listed in Section 3.2.

4.2 Computational budget saving versus performance degradation

There is a trade-off when early terminating BO, i.e., the performance of BO can be compromised when using less FEs. In this subsection, we employ I cost and I perf to characterize such trade-off. From the comparison results shown in Figure 6 and Table 1, we can see that although the naïve method achieves the best I perf , it consumes almost all FEs. In contrast, our proposed method saves up to ≈ 80% computation budget while the performance degradation is up to a order of magnitude smaller than the other three termination methods. As the trajectories of the regret of BO versus the number of FEs shown in Figure 7, we can see that the other three termination methods suffer from a premature early stopping.

† denotes the performance of our proposed method is significantly better than the other peers according to the Wilcoxon's rank sum test at a 0 . 05 significance level; ‡ denotes the opposite case.

I

perf

2

4

6

8

10

12

14

16

18

0

0.1

0.2

0.3

0.4

I

cost

Proposed method

2

4

6

8

without considering Condition 1

10

12

14

16

18

0.1

0.05

0

-0.05

4.3 Parameter Sensitivity Study

In this subsection, we investigate the sensitivity of our proposed termination method with respect to the parameter τ . We consider various settings of τ ∈ { 2 i } 9 i =1 and repeat the experiments on all benchmark problems introduced in Section 3.1. The aggregated comparison results for I cost and I perf are illustrated as bar charts with error bars in Figure 8. Specifically, we present the results for η lb = 2 . 05 , while the complete results can be found in the supplementary document. The plots show that the choice of τ has minimal impact on the results, except for cases with τ = 2 and τ = 4 . This is reasonable, as the termination method may not utilize sufficient previous information when only considering a few observed samples. Additionally, we examine the scenario where the equality constraint in Condition 1, i.e., equation (5) is relaxed. The comparison results in Figure 8 reveal similar observations regarding the settings of τ . However, we also notice a slight performance degradation and more aggressive early stopping in this case. These findings demonstrate that the Condition 1 helps mitigate the risk of premature early stopping.

5 Other Partially Related Works

Despite the limited number of dedicated studies on termination criteria for BO, various efforts have been made to explore early stopping strategies in different contexts.

The first category primarily focuses on detecting change points in sequential processes [34], with applications spanning various fields such as financial analysis [20], bioinformatics [7], and network traffic data analysis [23], among others. However, modeling the automatic termination of BO as a change point detection (CPD) problem may present several challenges. These include: 1) the absence of suitable stopping metrics that can provide signals for CPD in the optimization process of BO; 2) the unknown and uncertain nature of signal distribution, the number of change points, and change point consistency; 3) limited data available for CPD; and 4) the necessity to further evaluate change points in order to determine an appropriate moment for terminating BO.

The second category primarily focuses on determining the statistically optimal stopping moment for generalized sequential decision-making processes [14, 13]. For instance, in the classical secretary problem, termination criteria are developed to identify the maximum of an unknown distribution with minimal cost through sequential search [12]. They typically establish relationships between the costs and rewards of decision-making using cost coefficients [8, 2, 3], unknown observation costs [15, 37, 30, 25] or discount factors [36], subsequently deriving statistically optimal stopping conditions. However, quantifying the relationship between the improvement of the fitness and the

cost of BO remains challenging. Furthermore, these criteria do not leverage the information provided by the surrogate model, which is crucial in BO.

The third category primarily aims to balance exploration and exploitation in the optimization process. Among them, heuristic methods, exemplified by simulated annealing, are widely employed to halt the local search step of optimization algorithms [17, 1, 21]. However, such methods' hyperparameters lack interpretability and must be fine-tuned according to different problem characteristics. Additionally, McLeod et al. propose a regret-based strategy for switching between local and global optimization. Although promising for complex functions, this approach has certain limitations, including reliance on the authors' proposed regret reduction acquisition function and the potential need for additional computational resources to approximate intractable integrals. Furthermore, Eriksson et al. developed a trust-region-based BO that balances exploitation and exploration. This algorithm terminates local search when the trust region size is reduced to zero. However, the termination criteria lack theoretical guarantees and are bound to the proposed trust region maintenance mechanism.

6 Conclusion

In this paper, we developed a simple yet theoretically grounded two-step method for automatically terminating BO. The key insight is to proactively detect the local convex region and it terminates BO whenever the termination indicator built upon the local regret therein falls below a predetermined threshold. Our proposed termination method naturally strikes a balance between the quality of solution found by BO versus its computational efficiency. The proposed termination method is supported by robust theoretical underpinnings, and we have additionally introduced an approximation method to enhance the numerical stability by solving a bilevel optimization problem. Our extensive empirical studies, conducted across a variety of benchmark problems, including synthetic functions, reinforcement learning, and hyperparameter optimization, consistently demonstrated the better performance of our proposed method compared to other state-of-the-art techniques.

Besides, experimental results also show that the termination criterion of our proposed method is robust across different problems. This property paves an additional opportunity for our proposed termination method to go beyond automatically terminate BO, but to a broader range of applications, such as early stopping to avoid overfitting in neural network traing, change point or anomaly detection in data stream, and even a new perspective to strike the balance between exploitation and exploration under a bandit setting. The primary limitation of the proposed termination criterion is that it requires a predefined termination threshold, which needs to be determined based on prior knowledge or empirical observations. Although a recommended threshold selection range is given here, finding an optimal threshold that suits a wide range of optimization problems remains a challenge.

Author Contributions

SL implemented the theoretical derivations and experiments, as well as drafted the manuscript; KL piloted the idea and re-wrote the manuscript; WL proofread the manuscript.

References

Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures. In ICML'18: Proc. of the International Conference on Machine Learning , pages 462-471. PMLR, 2018.

Bruno Betrò and Fabio Schoen. Sequential stopping rules for the multistart algorithm in global optimisation. Mathematical Programming , 38(3):271-286, 1987.

Bruno Betro and Fabio Schoen. Optimal and sub-optimal stopping rules for the multistart algorithm in global optimization. Mathematical Programming , 57(1):445-458, 1992.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. Retrieved January 20, 2023, from https://github. com/openai/gym .

Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing , 16(5):1190-1208, 1995.

Souhil Chakar, E Lebarbier, Céline Lévy-Leduc, and Stéphane Robin. A robust approach for estimating change-points in the mean of an ar (1) process. Bernoulli , 23(2):1408-1447, 2017.

Herman Chernoff. Sequential design of experiments. The Annals of Mathematical Statistics , 30(3):755-770, 1959.

Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In NeurIPS'21: Proc. of the Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems , 32, 2019.

Peter I. Frazier. A tutorial on bayesian optimization. CoRR , abs/1807.02811, 2018.

PR Freeman. The secretary problem and its extensions: A review. International Statistical Review/Revue Internationale de Statistique , pages 189-206, 1983.

Roman Garnett. Bayesian Optimization . Cambridge University Press, 2023.

Daniel G Goldstein, R Preston McAfee, Siddharth Suri, and James R Wright. Learning when to stop searching. Management Science , 66(3):1375-1394, 2020.

Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In LION'11: Proc. of the Fifth International Conference on Learning and Intelligent Optimization , pages 507-523. Springer, 2011.

Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. J. Glob. Optim. , 13(4):455-492, 1998.

Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. science , 220(4598):671-680, 1983.

H. J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. , 86(1):97-106, 1964.

P. Langley. Crafting papers on machine learning. In Pat Langley, editor, ICML'00: Proc. of the 17th International Conference on Machine Learning , pages 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.

Marc Lavielle and Gilles Teyssiere. Adaptive detection of multiple change-points in asset price volatility. In Long memory in economics , pages 129-156. Springer, 2007.

Daniel James Lizotte. Practical bayesian optimization. 2008.

Romy Lorenz, Ricardo P Monti, Ines R Violante, Aldo A Faisal, Christoforos Anagnostopoulos, Robert Leech, and Giovanni Montana. Stopping criteria for boosting automatic experimental design using real-time fmri with Bayesian optimization, 2016.

Alexandre Lung-Yut-Fong, Céline Lévy-Leduc, and Olivier Cappé. Distributed detection/localization of change-points in high-dimensional network traffic data. Statistics and Computing , 22(2):485-496, 2012.

Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, Andreas Krause, Matthias W. Seeger, and Cédric Archambeau. Automatic termination for hyperparameter optimization. In AutoML'22: Proc. of 2022 International Conference on Automated Machine Learning , volume 188 of Proceedings of Machine Learning Research , pages 7/1-21. PMLR, 2022.

Gustavo Malkomes, Charles Schaff, and Roman Garnett. Bayesian optimization for automated model selection. pages 2892-2900, 2016.

Michael D. McKay, Richard J. Beckman, and William J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics , 42(1):55-61, 2000.

Mark McLeod, Stephen Roberts, and Michael A Osborne. Optimization, fast and slow: optimally switching between local and bayesian optimization. In ICML'18: Proc. of the International Conference on Machine Learning , pages 3443-3452. PMLR, 2018.

Vu Nguyen, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. Regret for expected improvement over the best-observed value and stopping condition. In ACML'17: Proc. of The 9th Asian Conference on Machine Learning , volume 77 of Proceedings of Machine Learning Research , pages 279-294. PMLR, 2017.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proc. IEEE , 104(1):148-175, 2016.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NeurIPS'12: Proc. of the Twenty-sixth Conference on Neural Information Processing Systems , pages 2951-2959, 2012.

Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML'10: Proc. of the 27th International Conference on Machine Learning , pages 1015-1022. Omnipress, 2010.

Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Informationtheoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory , 58(5):3250-3265, 2012.

S. Surjanovic and D. Bingham. Virtual library of simulation experiments: Test functions and datasets. Retrieved January 20, 2023, from http://www.sfu.ca/~ssurjano .

Charles Truong, Laurent Oudre, and Nicolas Vayatis. Selective review of offline change point detection methods. Signal Processing , 167:107299, 2020.

Christopher Williams and Carl Edward Rasmussen. Gaussian processes for machine learning . MIT press Cambridge, MA, 2006.

Tianyi Zhang, Daniel Russo, and Assaf Zeevi. Learning to stop with surprisingly few samples. In COLT'21: Proc. of the Conference on Learning Theory , pages 3887-3888. PMLR, 2021.

Shlomo Zilberstein. Using anytime algorithms in intelligent systems. AI magazine , 17(3):73-83, 1996.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper propose a new stopping criterion for Bayesian optimization. The idea is to consider the last designs visited during optimization to compute a local regret involving another optimization problem. A theoretical analysis is provided as well as an empirical comparison with some alternative termination criteria.\n\nSTRENGTHS:\n- a theoretical analysis of the proposed criterion is given.\n- the empirical results are promising.\n\nWEAKNESSES:\n- stopping criteria have been proposed since the early works on BO, e.g., the discussion by Jones et al. It would be worth considering this option (optimization of EI with convergence guarantees). Another relevant work to compare with is Queipo, N. V., Pintos, S., & Nava, E. (2013). Setting targets for surrogate-based optimization. Journal of Global Optimization, 55, 857-875.\n- the description of the BO setup is vague: what is used for the acquisition function search? Are the hyperparameters optimized? Plus it is typical to increase omega in UCB and decrease lengthscales over time.\n- The selection of the threshold \\eta and the chosen metrics are not easy to interpret (see more detailed questions below).\n\n** The rebuttal and discussion have been lengthy. My main criticisms remains and can be summarized in two points:\n- the link between the general idea and the actual criterion is not clear, and it is not obvious how to use it to control the level of convergence or accuracy in practice.\n- the comparison with existing works lacks at least the works by Jones et al., to test the robustness against model mis-specification and quality of the acquisition function optimization.\n\nQUESTIONS:\n- Could you give a high level description of the conditions ?\n- Could you give an interpretation of the threshold? For instance, criteria on expected improvement and probability of improvement are easy to choose. It would be more natural to select \\tilde{r} as threshold and then do the empirical comparison based on it (this is partially done in Fig. 7).\n- Similarly, in Section 3.2 the values considered seem quite arbitrary, how are they chosen?\n- What happens on multi-modal test functions such as Branin, where local minima are also global ?\n- Could you compare with McLeod et al., that also consider a convexity test?\n- On figure 2: are the hyperparameters fixed? The lengthscale is clearly too large.\n- Could you provide a pseudo-code of the computation of the stopping criterion?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new termination criteria for Bayesian Optimization (BO). Early stopping is an important, and understudied problem in BO, as function evaluations tend to be expensive and therefore terminating the optimization (once a good enough solution has been reached) can save a large amount of resources.\n\nThe criteria is based on two stages: in the first stage we check whether the search process can be said to have converged to a convex hull. This is checked by testing the convexity of the black-box function in the area around the last $\\tau$ observations, where the mean of the GP is used to approximate $f$ where we do not have observations. If this condition is fulfilled, we then calculate the local regret and determine if it falls below a pre-specified threshold. \n\nThe calculation of local regret requires finding the values of $\\arg\\max_{x \\in \\tilde{\\Omega}} \\mu(x) $ and $\\arg\\max_{x \\in \\tilde{\\Omega}} \\sigma^2(x) $. This may lead to numerically unstable results if done naively, so the authors propose using L-BFGS on the convex hull with termination criteria: $|| \\nabla \\mu(x) ||_2 \\leq \\lambda $ and $|| \\sigma^2 ||_2 \\leq \\lambda $ and with specific initializations. Propositions and Lemmas 1 and 2 back-up their recommendations and show we should expect numerically stable results.\n\nThere is then a theoretical analysis of the termination criteria, showing (through Theorems 1 and 2) that if the acquisition function is properly optimized, then their calculation of local regret represents an upper bound on global regret. Otherwise, if there is numerical error when optimizing the acquisition function, then at worst their calculation of local regret will be an upper-bound on the regret restricted to the convex hull $\\tilde{\\Omega}$.\n\nExperiment results are carried out against 4 benchmarks: (a) a naive method that stops the optimization once there is no improvement in observations for a specific amount of observations, (b) Nguyen's method that considers the maximum expected improvement, (c) Lorenz's method that considers the maximum probability of improvement, and (d) Makarova's method that considers the difference between the lower and upper confidence bounds as the termination indicator. They further compare on 3 synthetic benchmarks, 2 reinforcement learning tasks, and 5 hyper-parameter tuning problems.\n\nThe metrics used to compare are: (i) the empirical cumulative probability which explores how consistent the termination criteria is across a range of tasks, (ii) the relative computational cost, which calculates which percentage of the budget is used during the optimization, and (iii) the relative performance degradation, which measures the regret sacrificed by early stopping against letting the optimization continue. \n\nThe results show the following: the proposed method is the most consistent across tasks, having the same termination threshold leads to similar performance no matter the task (something other methods fail at doing). In addition, the proposed method has the smallest performance degradation for all methods except the naive one (which mostly uses all the budget), with the proposed method allowing us to save up to 80% of the budget. Finally, an empirical study shows that the method is robust to the choice of $\\tau$, given $\\tau > 4$.\n\nSTRENGTHS:\nOriginality: The proposed algorithm is novel condition very different from any I have seen before. The idea of testing for convexity and comparing local regret is excellent, and the theoretical study to justify the choice of criteria complements the paper very well. \n\nQuality: The ideas used in the paper are well justified, and explored thoroughly. Previous literature is considered, and compared against fully, and the empirical results test in a large variety of tasks, and further explore the behavior of the algorithm for different hyper-parameter settings. \n\nClarity: The paper has a difficult task of explaining many different things it is introducing, including previous work, theoretical results, and evaluation metrics. It does a great job at many of these things: it explains previous work, gives an _intuition_ of the meaning of the performance metrics, and the figures showcasing the results are excellent. Nonetheless, there are parts of the paper that lack clarity and perhaps can be improved (see later).\n\nSignificance: Given that Bayesian Optimization has seen a surge of interest in the physical sciences where experiments can be very costly, it is very important to have strong termination criteria to save costs as much as possible. The same holds true for most applications of Bayesian Optimization. As the authors suggest, this has been an understudied area, so papers like this one should be very useful to the larger scientific community.\n\nWEAKNESSES:\n- I found a few vital parts of the paper difficult to follow, and perhaps some improvements in the clarity of exposition could improve it. In particular it was difficult to (i) follow some of the proofs in the appendix, in particular the proofs of Theorems 1 and 2 could be expanded upon as it required me to write down a few equations manually to understand them. I was unable to understand the proof of Lemma 2. (ii) understand the intuition behind some of the choices made by the authors, and (iii) it is still unclear to me exactly why $I_{cdf}$ is constructed in such a way. I leave the details to the 'questions' section. \n\n- The naive baseline seems to be implemented _too naively_, indeed the termination thresholds in L176 initially gave me the impression of being too large, and this was confirmed by Figure 6. It would be better if a more diverse set of thresholds were investigated, including smaller values and scaling with the dimension of the problem (indeed for 2d problems a small amount of non-improvements could lead to good results, but early termination on high dimensions). Indeed, most of the budget is used in all benchmarks, showing it is a poor baseline to compare against. \n\nMinor weaknesses:\n\n- Figure 1 was initially a little confusing. Perhaps it would be a good idea to include the algorithm's explanations of section 3.2 into the introduction.\n\nWhile I list important weaknesses, which have an important impact on the score, I want to clarify that I thoroughly enjoyed the paper and I think it is great work. The low score stems from reluctance to fully recommend acceptance when I feel I do not understand important parts of the paper, but I look forward to discussing with the authors and other reviewers and having many of my doubts clarified.\n\nQUESTIONS:\n- What is the intuition behind initializing the L-BFGS for $\\arg\\max_{x \\in \\tilde{\\Omega}} \\sigma^2(x)$ at the maximum of the lower bound?\n\n- In the proof of Lemma 2, the authors mention \"give a stationary, isotropic kernel $k(\\cdot, \\cdot)$, we have: $k^2(x, x^i) \\propto - ||x - x^i||^2_2$\". Is the \"$\\propto$\" symbol meant to represent 'proportional to', if so, why is it the case that the statement holds? It does not seem to hold for the RBF kernel. After a few readings, I think it means that the LHS if an increasing function of the RHS, but usually I have seen the symbol to mean direct linear proportionality so maybe it should be clarified.\n\n- Furthermore, how is it that equation (19) follow from equation (18) _in the appendix_, it is not clear to me (relating to the proof of Lemma 2).\n\n- Is there a more intuitive explanation of $I_{cdf}$? I found it difficult to understand what the CDF variable is. Indeed, $\\tilde{\\kappa}_i$ seems to be fully defined in L200, but it is treated as a variable in Figure 3. I am also failing to understand how the maximum and minimum values of $\\kappa$ are calculated, is it empirically from the test runs? If so, what variable can we vary? This made understanding Figures 3 and 4 difficult. Figure 5 was clear though, and backs up the claims of the authors. \n\n- The termination criteria in equation (7) seems to depend on the level of noise in the black-box function. What would be the effect of varying levels of noise? \n\n- It is my understanding (from reading the appendix) that $\\varepsilon$ in equation (14) is the numerical error when optimizing the acquisition function. Is this correct? If so, it would be good to include in the main paper. \n\n- In my experience of working with early termination BO, I sometimes found that model misspecification can lead to early termination. For example, by underestimating the EI. How would you expect model misspecification to affect your algorithm? \n\n- What is the computational overhead of calculating the stopping criteria for your method? For other methods it seems to be very cheap / free.\n\nLIMITATIONS:\nThe authors do not address any limitations. It would be good for the authors, who know the work best, to try and mention the main ones.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a two-step termination method for Bayesian optimization. The proposed termination method first identifies if the optimization has converged within convex region, then it would stop the optimization procedure when estimated local regret is below predefined threshold. The authors compute estimated the local regret by solving a bilevel optimization problem, and theoretically prove that the regret is upper bounded by the defined local regret. Experiment shows that the proposed methods is more rubust and sample-efficient than other termination methods.

STRENGTHS:
1. The idea is novel and easy to implemented to various problems.

2. The experiment is comprehensive for various benchmarks and evaluation metrics.

WEAKNESSES:
1. During the local regret estimation, the proposed method assumes convexity of the variance function, which is hard to satisfy even if the Condition 1 is satisfied.

2. The paper writing and presentation need to be improved, which I found difficult to follow. For example, in section 2, some intuition or explanation should be provided after the lemma or theorem is proposed. It would be better if the proposed method is highlighted in the result figures.

QUESTIONS:
1. As mentioned in weakness part, is the assumption of convex variance function easy to satisfied on the benchmarks used in this paper?

2. In Lemma 2,  why does the equation need to satisfy $\hat \Omega \cap D = \emptyset$ ?

3. What is the error bar in result figures?

LIMITATIONS:
The author addressed the limilations. I did not see potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04