SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Control and Simulation Center, Harbin Institute of Technology, China. National Key Laboratory of Modeling and Simulation for Complex Systems, China. ShuangLi.hit@outlook.com
Ke Li
Department of Computer Science University of Exeter, EX4 4RN, Exeter, UK. k.li@exeter.ac.uk
Wei Li*
Control and Simulation Center, Harbin Institute of Technology, China. National Key Laboratory of Modeling and Simulation for Complex Systems, China. frank@hit.edu.cn
Bayesian Optimization (BO) is a powerful method for tackling expensive black-box optimization problems. As a sequential model-based optimization strategy, BO iteratively explores promising solutions until a predetermined budget, either iterations or time, is exhausted. The decision on when to terminate BO significantly influences both the quality of solutions and its computational efficiency. In this paper, we propose a simple, yet theoretically grounded, two-step method for automatically terminating BO. Our core concept is to proactively identify if the search is within a convex region by examining previously observed samples. BO is halted once the local regret within this convex region falls below a predetermined threshold. To enhance numerical stability, we propose an approximation method for calculating the termination indicator by solving a bilevel optimization problem. We conduct extensive empirical studies on diverse benchmark problems, including synthetic functions, reinforcement learning, and hyperparameter optimization. Experimental results demonstrate that our proposed method saves up to  $\approx 80\%$  computational budget yet is with an order of magnitude smaller performance degradation, comparing against the other peer methods. In addition, our proposed termination method is robust in terms of the setting of its termination criterion.
"Nature does not hurry, yet everything is accomplished." — Lao Tzu
In this paper, we consider the black-box optimization problem (BBOP) defined as follows:
$$
\underset {\mathbf {x} \in \Omega} {\text {m a x i m i z e}} f (\mathbf {x}), \tag {1}
$$
*Wei Li is the corresponding author of this paper.
Figure 1: Trajectories of termination criteria used in [22], [28] and [24] on Ackley and Levy function where  $n = 1$ . Results are collected from 21 independent runs of vanilla BO while the mean value of termination indicator of each termination criterion is plotted as the solid line associated with the confidence interval. Please refer to Section 3.2 for a description of these termination criteria, as well as the meaning of  $\kappa_{\mathrm{PI}}$ ,  $\kappa_{\mathrm{EI}}$  and  $\kappa_{\mathrm{diff}}$ .
where  $\mathbf{x} = (x_{1},\dots ,x_{n})^{\top}$  is a decision vector (variable),  $\Omega = [x_i^{\mathrm{L}},x_i^{\mathrm{U}}]_{i = 1}^n\subset \mathbb{R}^n$  represents the search space, and  $f:\Omega \to \mathbb{R}$  corresponds to the attainable set in the objective space. In real-world scenarios, function evaluations (FEs) of  $f(\mathbf{x})$  can be costly, giving rise to expensive BBOPs. Bayesian optimization (BO) has emerged as one of the most effective methods for addressing expensive BBOPs. BO is a sequential model-based optimization technique consisting of two iterative steps: i) employing limited expensive FEs to construct a surrogate model of the physical objective function, such as a Gaussian process (GP) model [35]; and ii) selecting the next point of interest for costly FE by optimizing an acquisition function, e.g., probability of improvement (PI) [18], expected improvement (EI) [16], and upper confidence bound (UCB) [31]. Numerous theoretical and methodological advancements have been made in BO. Interested readers can refer to comprehensive survey papers [29, 11] and a recent textbook [13] for further information.
Nevertheless, the question of when to terminate the search process of BO remains a largely underexplored area in the literature. At present, the most prevalent termination criterion is a pre-specified budget, such as the number of FEs or wall-clock time. Though intuitive, this approach neglects the search dynamics inherent to different BBOPs. As a result, this strategy is rigid while it does not offer a general rule for determining an appropriate budget across various problem settings. If the budget is too small, BO may terminate prematurely, yielding a suboptimal solution. On the contrary, an excessive budget may lead to wasted computational resources. Another simple termination method involves stopping BO if the current best solution remains unchanged for a predetermined number of consecutive FEs. However, as highlighted by [24], this strategy also fails to consider the observed data during the sequential model-based optimization process and relies on a pre-defined threshold.
Beyond the aforementioned 'naive' approaches, a limited number of dedicated efforts have been made to address the termination of BO. One notable method involves monitoring the progress of BO by termination indicators, such as the maximum of EI [28, 16] or PI [22]. In this approach, BO is terminated when the corresponding termination indicator falls below a pre-specified threshold. Very recently, Makarova et al. proposed using the difference between the minimal of the lower confidence bound (LCB) and UCB as the termination indicator. As illustrated in Figure 1, we observe that all criteria used in these termination approaches exhibit significant oscillation during the optimization process. This can be attributed to:  $i$ ) the stochastic nature of BO itself, and  $ii$ ) numerical errors arising from the non-convex optimization of acquisition functions. Furthermore, as shown in Figures 1(a) and (b), the variation range of the same criterion can differ substantially when addressing problems with distinct fitness landscapes. These factors make determining a universally applicable threshold in practice challenging, resulting in fragile and less intuitive termination criteria compared to simply establishing a budget. Additionally, we find that these termination criteria are 'myopic', as decision-making is based solely on the observations at the current step, leading to a lagged termination. For instance, consider the selected samples shown in Figure 2; it is difficult, if not impossible, to determine when to terminate BO until  $t = 20$ . However, if we look backward to  $t = 5$ , it becomes evident that BO is likely to converge by  $t = 10$ .
Our contributions. In light of the aforementioned challenges, this paper proposes a novel termination method for BO that proactively detects whether the search is located in a convex region of  $-f(\mathbf{x})$  by examining previously observed samples. BO is terminated if the local regret within this convex region falls below a predetermined threshold. To improve numerical stability, we introduce an approx-
Figure 2: Search dynamics of vanilla BO on the Ackley function  $(n = 1)$  at different time steps after the initialization. In particular,  $t = 5$  indicates five new samples are collected after the initialization.
imation method for calculating the termination indicator by solving a bilevel optimization problem. Our proposed termination method is simple, yet it offers theoretical guarantees. To demonstrate its effectiveness, we compare the performance of our proposed method against four peer methods on a variety of benchmark problems, encompassing synthetic functions, reinforcement learning, and hyperparameter optimization.
This section starts with a gentle tutorial of vanilla BO. Then, we delineate the implementation of our proposed termination method, followed by a theoretical analysis at the end.
2.1 Vanilla Bayesian Optimization
As a gradient-free optimization method, BO comprises two major steps. The first step involves constructing a surrogate model based on GP to approximate the expensive objective function. Given a set of training data  $\mathcal{D} = \{\langle \mathbf{x}^i,f(\mathbf{x}^i)\rangle \}_{i = 1}^N$ , GP learns a latent function  $g(\mathbf{x})$ , such that  $\forall \mathbf{x}\in \mathcal{D}$ , we have  $f(\mathbf{x}) = g(\mathbf{x}) + \epsilon$ , where  $\epsilon \sim \mathcal{N}(0,\sigma_{\epsilon}^{2})$  is an i.i.d. Gaussian noise. For each testing input vector  $\mathbf{z}^{*}\in \Omega$ , the mean and variance of the target  $f(\mathbf{z}^{*})$  are predicted as follows:
$$
\mu (\mathbf {z} ^ {*}) = \mathbf {k} ^ {* ^ {\top}} (K + \sigma_ {\epsilon} ^ {2} I) ^ {- 1} \mathbf {f},
$$
$$
\sigma^ {2} \left(\mathbf {z} ^ {*}\right) = \mathbf {k} \left(\mathbf {z} ^ {*}, \mathbf {z} ^ {*}\right) - \mathbf {k} ^ {* ^ {\top}} \left(K + \sigma_ {\epsilon} ^ {2} I\right) ^ {- 1} \mathbf {k} ^ {*}, \tag {2}
$$
where  $X = (\mathbf{x}^1, \dots, \mathbf{x}^N)^\top$  and  $\mathbf{f} = (f(\mathbf{x}^1), \dots, f(\mathbf{x}^N))^\top$ .  $\mathbf{k}^*$  is the covariance vector between  $X$  and  $\mathbf{z}^*$ , and  $K$  is the covariance matrix of  $X$ . In this paper, we use the Matérn 5/2 kernel as the covariance function to measure the similarity between a pair of data points. The second step consists of an infill criterion based on the optimization of an acquisition function, which determines the next point of merit  $\tilde{\mathbf{x}}^*$  to be evaluated by the actual expensive objective function:
$$
\tilde {\mathbf {x}} ^ {*} = \underset {\mathbf {x} \in \Omega} {\operatorname {a r g m a x}} f ^ {\mathrm {a c q}} (\mathbf {x}). \tag {3}
$$
where  $f^{\mathrm{acq}}(\mathbf{x}) = \mu (\mathbf{x}) + \omega \sigma (\mathbf{x})$  is the widely used UCB [31] to facilitate our theoretical analysis. Specifically, the parameter  $\omega >0$ , determined according to the confidence level set as 0.95 in this paper, controls the trade-off between exploration and exploitation. Subsequently, the next point of merit  $\tilde{\mathbf{x}}^{*}$  is used to update the training dataset as  $\mathcal{D} = \mathcal{D}\bigcup \{\tilde{\mathbf{x}}^{*}\}$ , and BO iterates between the two aforementioned steps sequentially until a termination criterion is met. The convergence of BO can be evaluated by regret:
$$
r = f \left(\mathbf {x} ^ {\star}\right) - f \left(\tilde {\mathbf {x}} ^ {\star}\right), \tag {4}
$$
where  $\mathbf{x}^{\star}$  represents the ground truth global optimum and  $\tilde{\mathbf{x}}^{\star} = \operatorname*{argmax}_{\mathbf{x}\in \mathcal{D}}f(\mathbf{x})$  denotes the current best-found solution.
2.2 Proposed Termination Criterion
Inspired by the observations illustrated in Figure 2, we propose a termination method that involves 'looking back' at the last  $\tau > 1$  observed points in the dataset  $\mathcal{D}$ , and storing these in a temporary archive, denoted as  $\tilde{\mathcal{D}}$ . The termination criterion we propose is predicated on two primary conditions.
Condition 1. The BO search process is deemed to have converged within a convex hull  $\tilde{\Omega}$  if the following condition is satisfied:
$$
\sum_ {j = 1} ^ {\binom {\tau + 1} {2}} \mathbb {1} \left(\mu \left(\frac {\mathbf {x} + \mathbf {x} ^ {\prime}}{2}\right) \geq \frac {f (\mathbf {x}) + f \left(\mathbf {x} ^ {\prime}\right)}{2}\right) = \binom {\tau + 1} {2}, \tag {5}
$$
where  $\mathbb{1}(\cdot)$  denotes the indicator function, returning 1 if the argument holds true and 0 otherwise.  $\mathbf{x}$  and  $\mathbf{x}'$  are points selected randomly and distinctively from  $\tilde{\mathcal{D}}$ . The convex hull,  $\tilde{\Omega} = [\tilde{x}_i^{\mathrm{L}},\tilde{x}_i^{\mathrm{U}}]_{i = 1}^n$ , is a subset of  $\Omega$ , where  $\tilde{x}_i^{\mathrm{L}} = \operatorname*{argmin}_{\mathbf{x}\in \tilde{\mathcal{D}}}x_i$  and  $\tilde{x}_i^{\mathrm{U}} = \operatorname*{argmax}_{\mathbf{x}\in \tilde{\mathcal{D}}}x_i$ .
Condition 2. Assuming Condition 1 is satisfied, and  $\tilde{\mathbf{x}}$  denotes the most recently observed point in  $\mathcal{D}$ , we calculate the local regret  $\tilde{r}$  as follows:
$$
\tilde {r} = \mu (\dot {\mathbf {x}}) - \mu (\tilde {\mathbf {x}}) + \omega (\sigma (\dot {\mathbf {x}}) + \sigma (\tilde {\mathbf {x}})), \tag {6}
$$
where  $\dot{\mathbf{x}} = \operatorname*{argmax}_{\mathbf{x}\in \tilde{\Omega}}\mu (\mathbf{x})$  and  $\ddot{\mathbf{x}} = \operatorname*{argmax}_{\mathbf{x}\in \tilde{\Omega}}\sigma^2 (\mathbf{x})$ . The BO process terminates if the following inequality is satisfied:
$$
\frac {\tilde {r}}{\omega \sigma_ {\epsilon}} \leq \eta_ {\mathrm {l b}}, \tag {7}
$$
where  $\frac{\tilde{r}}{\omega\sigma_{e}}$  is used as the termination indicator, denoted as  $\kappa_{\mathrm{lb}}$ , and  $\eta_{\mathrm{lb}}$  is a predetermined threshold.
Remark 1. The inequality within the indicator function  $\mathbb{1}(\cdot)$  in equation (5) is derived from Jensen's inequality [4], which yields a convex function:
$$
- f \left(\alpha \mathbf {x} + (1 - \alpha) \mathbf {x} ^ {\prime}\right) \leq - \alpha f (\mathbf {x}) - (1 - \alpha) f \left(\mathbf {x} ^ {\prime}\right), \tag {8}
$$
where  $\alpha \in [0,1]$  and  $\mathbf{x},\mathbf{x}' \in \tilde{\Omega}$ . In order to avoid the necessity of additional function evaluations when computing  $f\left(\frac{\mathbf{x} + \mathbf{x}'}{2}\right)$ , we substitute  $\mu\left(\frac{\mathbf{x} + \mathbf{x}'}{2}\right)$  into equation (5).
Remark 2. In equation (6), we employ the widely-used L-BFGS algorithm [6] to compute  $\dot{\mathbf{x}}$  and  $\ddot{\mathbf{x}}$ . To ensure numerical stability, we suggest the following strategies for initializing the algorithm and defining its termination criterion:
1. For  $\dot{\mathbf{x}}$ ,  $L$ -BFGS is initialized at a point randomly selected from  $\tilde{\Omega}$ . The algorithm terminates when  $\| \bigtriangledown \mu (\mathbf{x})\| _2\leq \lambda$ . In our work, we set  $\lambda = 10^{-6}$ , following Proposition 1.
2. For  $\ddot{\mathbf{x}}$ ,  $L$ -BFGS is initialized at the point  $\underset {\mathbf{x}\in \tilde{\Omega}}{\operatorname{argmax}}\underline{\sigma}^2 (\mathbf{x})$ , where  $\underline{\sigma}^2 (\mathbf{x})$  denotes the lower bound of  $\sigma^2 (\mathbf{x})$  over  $\tilde{\Omega}$ . The termination criterion is  $\| \bigtriangledown \sigma^2 (\mathbf{x})\| _2\leq \lambda$ , as per Proposition 2.
Remark 3. Considering equation (7), given that  $\frac{\mu(\dot{\mathbf{x}}) - \mu(\ddot{\mathbf{x}})}{\omega\sigma_{\epsilon}}\geq 0$  and  $\frac{\sigma(\dot{\mathbf{x}}) + \sigma(\ddot{\mathbf{x}})}{\sigma_{\epsilon}}\geq 2$ , we deduce that  $\eta_{\mathrm{lb}}\geq 2$ . The upper bound of  $\eta_{\mathrm{lb}}$  is empirically determined, as detailed in Section 4.1.
Remark 4. When the GP model is overfitting, BO tends to converge within the local region of the current best solution. In this case, both Condition 1 and Condition 2 are easily met while BO will be terminated prematurely. On the other hand, when the model is underfitting, BO will explore  $\Omega$  in a random manner. In this case, satisfying Condition 1 becomes challenging, and BO will face the risk of failing to be terminated. Therefore, we designed three mitigation strategies: 1) restrict the lengthscale to [0.05, 200] during GP training to prevent lengthscales from becoming excessively large or small; 2) normalize the input of training data to [0, 1]; and 3) standardize the output of the training data by centering it on the mean and scaling it by the variance.
Proposition 1. Consider  $\forall \mathbf{x} \in \tilde{\Omega}$ , where  $-\mu(\mathbf{x})$  represents a convex function. If  $\|\nabla \mu(\mathbf{x})\|_2 \leq \lambda$ , we can establish:
$$
\mu (\dot {\mathbf {x}}) - \mu (\mathbf {x}) \leq \xi , \tag {9}
$$
where  $\lambda = (2m_1\xi)^{1 / 2}$ ,  $\xi$  is a positive constant, and  $m_{1}$  denotes the strong convexity parameter of  $-\mu (\mathbf{x})$  [4].
Lemma 1. Assume the  $GP$  employs a stationary kernel  $k(\cdot, \cdot)$ . For  $\forall \mathbf{x} \in \tilde{\Omega}$ , the lower bound of  $\sigma^2(\mathbf{x})$  is given by:
$$
\underline {{\sigma}} ^ {2} (\mathbf {x}) = k (\mathbf {x}, \mathbf {x}) + c \sum_ {i = 1} ^ {| \mathcal {D} |} k ^ {2} \left(\mathbf {x}, \mathbf {x} ^ {i}\right), \tag {10}
$$
where  $c < 0$  is a constant and  $\mathbf{x}^i\in \mathcal{D}$  for  $i\in \{1,\dots ,|\mathcal{D}|\}$ .
Lemma 2. Given Lemma 1, determining  $\underset{\mathbf{x} \in \tilde{\Omega}}{\operatorname{argmax}} \underline{\sigma}^2(\mathbf{x})$  is equivalent to solving the following bilevel optimization problem:
$$
\begin{array}{l} \underset {\mathbf {x} \in \tilde {\Omega}} {\mathrm {m i n i m i z e}} d (\mathbf {x}, \mathbf {x} ^ {1}, \mathbf {x} ^ {2}) = \| \mathbf {x} - \mathbf {x} ^ {1} \| _ {2} ^ {2} + \| \mathbf {x} - \mathbf {x} ^ {2} \| _ {2} ^ {2} \\ \text {s u b j e c t} \quad \left\{\mathbf {x} ^ {1}, \mathbf {x} ^ {2} \right\} \quad = \quad \operatorname {a r g m a x} \quad \| \mathbf {x} ^ {1} - \mathbf {x} ^ {2} \| _ {2} ^ {2}, \tag {11} \\ \mathbf {x} ^ {1}, \mathbf {x} ^ {2} \in \mathcal {D} \cap \tilde {\Omega} \\ \mathbf {x} ^ {1} \neq \mathbf {x} ^ {2}, \hat {\Omega} \cap \mathcal {D} = \emptyset \\ \end{array}
$$
where  $\hat{\Omega} = [\hat{x}_i^{\mathrm{L}},\hat{x}_i^{\mathrm{U}}]_{i = 1}^n\subset \tilde{\Omega}$ ,  $\hat{x}_i^{\mathrm{L}} = \min (x_i^1,x_i^2)$  and  $\hat{x}_i^{\mathrm{U}} = \max (x_i^1,x_i^2)$ . Given that the lower-level optimization can be addressed via exhaustive search, the analytical solution of (11) is given by  $\hat{\mathbf{x}} = (\hat{x}_1^{\mathrm{L}} + \frac{\hat{x}_1^{\mathrm{U}} - \hat{x}_1^{\mathrm{L}}}{2},\dots ,\hat{x}_n^{\mathrm{L}} + \frac{\hat{x}_n^{\mathrm{U}} - \hat{x}_n^{\mathrm{L}}}{2})^\top$
Proposition 2. Leveraging Lemma 2, suppose  $\underset{\mathbf{x} \in \bar{\Omega}}{\mathrm{minimize}} - \sigma^2(\mathbf{x})$  exhibits convexity in its local optimal regions, the following inequality is satisfied when  $\| \nabla \sigma^2(\mathbf{x}) \|_2 \leq \lambda$ :
$$
\sigma^ {2} (\ddot {\mathbf {x}}) - \sigma^ {2} (\mathbf {x}) \leq \beta + \xi , \tag {12}
$$
where  $\lambda = (2m_2\xi)^{1 / 2}$ ,  $\xi >0$ ,  $m_{2} > 0$  represents the strong convexity parameter of  $-\sigma^2 (\mathbf{x})$  in its local optimal regions [4], and  $\beta$  is constrained by  $0\leq \beta \leq \sigma^{2}(\ddot{\mathbf{x}}) - \sigma^{2}(\hat{\mathbf{x}})$ .
2.3 Theoretical Analysis of the Proposed Termination Criterion
In this subsection, we delve into the theoretical underpinnings of the proposed termination method, focusing on the convergence of BO when the UCB is utilized as the acquisition function.
Lemma 3. As per Srinivas et al., the optimization process in BO can be conceptualized as a sampling process from a GP. Hence, for any  $\mathbf{x} \in \Omega$ , we have:
$$
\Pr \left(| f (\mathbf {x}) - \mu (\mathbf {x}) | \leq \omega \sigma (\mathbf {x})\right) > \delta , \tag {13}
$$
where  $\delta > 0$  signifies the confidence level adhered to by the UCB.
Corollary 1. Based on Lemma 3 and Condition 2, we deduce that:
$$
\Pr \left(f ^ {\mathrm {a c q}} \left(\tilde {\mathbf {x}} ^ {\star}\right) + \varepsilon \geq f \left(\mathbf {x} ^ {\star}\right)\right) > \delta , \tag {14}
$$
where  $\varepsilon$  is a numerical error when optimizing the acquisition function,  $\tilde{\mathbf{x}}^{\star} = \underset {\mathbf{x}\in \Omega}{\mathrm{argmax}}f^{\mathrm{acq}}(\mathbf{x})$ , and  $\mathbf{x}^{\star}$  represents the true global optimum. Furthermore,
$$
0 \leq \varepsilon \leq \mu (\dot {\mathbf {x}}) + \omega \sigma (\ddot {\mathbf {x}}) - f ^ {\mathrm {a c q}} (\tilde {\mathbf {x}} ^ {\star}), \tag {15}
$$
where  $\dot{\mathbf{x}},\ddot{\mathbf{x}}$  and  $\tilde{\mathbf{x}}^{\star}$  are elements of  $\tilde{\Omega}$ , while  $\delta >0$  denotes the confidence level of the UCB.
Theorem 1. Leveraging Corollary 1, when employing the termination method proposed in this paper, we deduce that the global regret bound of BO as:
$$
\Pr (r \leq 2 \omega \sigma (\tilde {\mathbf {x}} ^ {\star}) + \varepsilon) > \delta , \tag {16}
$$
where  $\delta > 0$  signifies the confidence level associated with the UCB.
Theorem 2. Building upon Condition 1 and Condition 2, and employing the termination method proposed in this paper, we establish the local regret bound of BO as:
$$
\Pr (f (\mathbf {x} ^ {\star}) - f (\mathbf {x}) \leq \tilde {r}) > \delta , \tag {17}
$$
where  $\mathbf{x} \in \tilde{\Omega}$ ,  $\mathbf{x}^{\star}$  denotes the true global optimum in  $\tilde{\Omega}$ , and  $\delta > 0$  is the confidence level of the UCB.
Remark 5. Drawing from Theorem 1 and Theorem 2, we observe that if  $\varepsilon$  can be considered negligible when  $\tilde{\mathbf{x}}^{\star}$  is accurately determined by optimizing the UCB,  $\tilde{r}$  subsequently represents the upper bound of BO regret within the domain  $\Omega$ . Conversely, if  $\varepsilon$  cannot be disregarded,  $\tilde{r}$  is posited as the upper bound of BO regret within the restricted domain  $\tilde{\Omega}$ .
In this section, we present the experimental setup for our empirical study, which encompasses the benchmark test problems, the peer algorithms, and the performance metrics used for evaluation.
3.1 Benchmark Problems
We evaluate the performance of our proposed method on three types of benchmark problems.
- Synthetic functions: We consider Ackley, Levy, and Schwefel functions [33] with  $n \in \{2, 5, 10\}$ . The objective function  $f(\mathbf{x})$  is contaminated by Gaussian noise  $\zeta \sim \mathcal{N}(0.0, 0.2)$ . The maximal number of FEs is set to  $N_{\mathrm{FE}} = 50n$ , with  $5n$  allocated to initialization.
- Reinforcement learning (RL): We examine two RL tasks chosen from OpenAI Gym [5]: Lunar Lander with  $n = 12$  and Swimmer with  $n = 16$ . We set  $N_{\mathrm{FE}} = 50n$ , with  $5n$  FEs allocated to initialization.
- Hyperparameter optimization (HPO): We consider 5 HPO tasks picked up from the HPOBench [9] for tuning support vector machine (SVM) with  $n = 2$ , multi-layer perceptron (MLP) with  $n = 5$ , random forest with  $n = 4$  and XGBoost with  $n = 8$ . The computational budget is set the same as in the RL tasks.
Note that, due to the use of termination criteria, it may not be necessary to exhaust the entire allocated computational budget to terminate BO. To ensure statistical significance, each experiment is independently conducted 21 times with different random seeds.
3.2 Peer Algorithms
As discussed in Section 1, the termination criterion for BO is an understudied topic in the literature. In our experiments, we compare our proposed method with the following four termination methods.