SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Naive method: This method ceases BO when  $\tilde{\mathbf{x}}^{\star}$  stays unchanged for  $\kappa_{\mathrm{n}}$  consecutive iterations. Here,  $\kappa_{\mathrm{n}}$  is also the termination indicator. In our experiments, we test three settings of the thresholds  $\eta_{\mathrm{n}}$  as 150, 337 and 524, respectively.
- Nguyen's method [28]: In each iteration of BO, the optimization of acquisition function produces the current optimal EI. By using this as the termination indicator, denoted as  $\kappa_{\mathrm{EI}}$ , the Nguyen's method terminates BO when it falls below a predetermined threshold  $\eta_{\mathrm{EI}}$ . In our experiments, we consider three settings of  $\eta_{\mathrm{EI}}$  as 0.01, 0.04 and 0.06, respectively.
- Lorenz's method [22]: Analogous to the Nguyen's method, the Lorenz's method replaces the EI with PI as the termination indicator, denoted as  $\kappa_{\mathrm{PI}}$ . In our experiments, the termination threshold  $\eta_{\mathrm{PI}}$  is set as 0.07, 0.2 and 0.33, respectively.
- Makarova's method [24]: Similar to the previous two methods, the Makarova's method uses the difference between the lower and upper confidence bounds as the termination indicator, denoted as  $\kappa_{\mathrm{diff}}$ . It terminates BO when  $\kappa_{\mathrm{diff}} \leq \eta_{\mathrm{diff}}$ , a predetermined threshold and is set as 0.26, 0.62 and 0.97, respectively, in our experiments.
- Our proposed method: According to Condition 1 and Condition 2, our proposed method terminates BO when  $\kappa_{\mathrm{lb}}$  falls below a predetermined threshold  $\eta_{\mathrm{lb}}$ , which is set as 2.02, 2.05 and 2.08, respectively. Furthermore, we introduce a hyperparameter  $\tau$  to control the number of observed samples being looked backward, which is set to  $\tau = 10$  in our experiments. The code is available at https://github.com/COLA-Laboratory/OptimalStoping_NeurIPS2023.
According to the aforementioned settings, it is evident that the naive method tends to delay termination when a large  $\eta_{\mathrm{n}}$  is used. On the other hand, other methods may incur a delayed termination if a small threshold is used. Note that the choices of the corresponding termination thresholds and the sensitivity of  $\tau$  are empirically examined in Sections 4.1 and 4.2.
3.3 Performance Metrics
In our experiments, we consider the following three performance metrics to measure the effectiveness of a termination method.
- Empirical cumulative probability of a termination indicator:
$$
I _ {\mathrm {c d f}} = \frac {1}{N _ {\mathrm {F E}} \times 2 1} \sum_ {i = 0} ^ {N _ {\mathrm {F E}} \times 2 1} \mathbb {1} (\kappa \leq \tilde {\kappa} _ {i}), \tag {18}
$$
where  $\tilde{\kappa}_i = \underline{\kappa} +\frac{(\bar{\kappa} - \underline{\kappa})\times i}{N_{\mathrm{FE}}\times 21}$ , and  $i\in \{0,\dots ,N_{\mathrm{FE}}\times 21\}$ . For a given termination method,  $\kappa$  represents its termination indicator as outlined in Section 3.2. The minimum and maximum
Figure 3: Trajectories of  $\mathrm{I}_{\mathrm{cdf}}$  collected on different benchmark problems. Here we only show some results without loss of generality, while full results can be found in the supplementary document. Different subplots are (a) our proposed method, (b) Naive method, (c) Nguyen's method, (d) Lorenz's method, and (e) Makarova's method, respectively.
Figure 4: Bar charts with error bars of normalized  $\tilde{\kappa}_i$  for different termination methods when  $\mathrm{I}_{\mathrm{cdf}}$  is set as 0.05, 0.1, 0.2, 0.3, 0.4, and 0.5 respectively.
values of  $\kappa$ , represented by  $\underline{\kappa}$  and  $\bar{\kappa}$  respectively, are determined across all 21 repeated experiments on each benchmark problem. If  $\mathrm{I}_{\mathrm{cdf}}$  exhibits consistency across a range of benchmark problems, it implies that the threshold choice for the corresponding termination method is consistent and not dependent on the specific problem.
- The relative computational cost:
$$
I _ {\text {c o s t}} = \frac {\tilde {N} _ {\mathrm {F E}}}{N _ {\mathrm {F E}}}, \tag {19}
$$
where  $\tilde{N}_{\mathrm{FE}}$  is the number of FEs used by a termination criterion when early stopping occurs. A lower value of  $\mathrm{I_{cost}}$  indicates a higher degree of computational budget saving.
- The relative performance degradation incurred by early stopping:
$$
I _ {\text {p e r f}} = \frac {f (\bar {\mathbf {x}}) - f (\tilde {\mathbf {x}} ^ {\star})}{f (\bar {\mathbf {x}}) - f (\underline {{\mathbf {x}}})}, \tag {20}
$$
where  $\bar{\mathbf{x}}$  and  $\underline{\mathbf{x}}$  are the best and the worst solutions found by BO when consuming all  $N_{\mathrm{FE}}$  FEs.  $\tilde{\mathbf{x}}^{\star}$  signifies the best solution found when early stopping is prompted by a termination criterion. A smaller  $\mathrm{I}_{\mathrm{perf}}$  value indicates less performance degradation resulting from the application of the corresponding termination criterion.
4 Empirical Studies
In this section, our experiments<sup>2</sup> aim to investigate three aspects:  $i$  ) the robustness of the termination threshold for different termination methods;  $ii$  ) the trade-off between the computational budget saving versus the performance degradation; and  $iii$  ) the sensitivity of  $\tau$  in our proposed termination method.
4.1 Robustness of the Selection of Termination Threshold
In this subsection, we use the  $\mathrm{I}_{\mathrm{cdf}}$  metric to scrutinize the threshold choice of various termination methods across different problems. As per equation (18), it is evident that  $\mathrm{I}_{\mathrm{cdf}} \propto \tilde{\kappa}_i$ . As discussed earlier in Section 3.2, a large  $\tilde{\kappa}_i$  can lead to premature early stopping. Consequently, we confine our analysis to instances where  $\mathrm{I}_{\mathrm{cdf}} \leq 0.5$ . As shown in Figure 3, the trajectories of  $\mathrm{I}_{\mathrm{cdf}}$  for our proposed method appear to converge, whereas those for the other methods diverge with different magnitudes. More specifically, as shown in Figure 3(a),  $\tilde{\kappa}_i = 2$  can be regarded as a transition point where  $\mathrm{I}_{\mathrm{cdf}} \geq 0.95$  if  $\tilde{\kappa}_i \geq 2$ . This empirical observation corroborates the theoretical result derived
Due to page limits, additional ablation experiments can be found in the supplementary document.
(a) Proposed method
Number of FEs
(b) Naive method
(c) Nguyen's method
(d) Lorenz's method
(e) Makarova's method
Figure 5: Trajectories of different termination indicators versus the number of FEs during the BO process on Ackley  $(n = 2)$  and HPO for SVM.
(a)
(a)
(b)
(d) (e)
Figure 6: Bar charts with error bars of  $\mathrm{I}_{\mathrm{cost}}$  and  $\mathrm{I}_{\mathrm{perf}}$  obtained by using different settings of termination threshold suggested in Section 3.2, denoted as  $\eta_1$ ,  $\eta_2$  and  $\eta_3$  respectively. Subplots (a) to (e) correspond to our proposed, Naïve, Nguyen's, Lorenz's, and Makarova's methods respectively.
Number of FEs
Number of FEs
Figure 7: Trajectories of the regret of BO versus the number of FEs during the BO process on five selected problems. Full results can be found in the supplementary document.
in Condition 2. In contrast, there do not exist a consistent lower bound for the other termination methods. To further elucidate these observations, we plot the distributions of  $\tilde{\kappa}_i$  when  $\mathrm{I}_{\mathrm{cdf}}$  ranges from 0.05 to 0.5 in Figure 4. It is clear that the bar charts exhibit the least variation for our proposed method. For the naive method,  $\tilde{\kappa}_i$  increases as  $\mathrm{I}_{\mathrm{cdf}}$  grows. However, the bars for the other three methods show significant fluctuations, particularly for the Nguyen's and the Lorenz's methods. These observations are further substantiated by the trajectories of the termination indicators throughout the BO process, as shown in Figure 5. We present results for the Ackley and HPO for SVM problems here, while complete results are available in the supplementary document. These plots reveal that the trajectories for our proposed method converge to a certain threshold, while those for the other methods not only diverge but also differ significantly on different problems. Based on this discussion, we use  $\mathrm{I}_{\mathrm{cdf}} = 0.05$  as the capping point to guide the selection of the termination threshold for different termination methods:  $\eta_{\mathrm{lb}} \in [2, 2.1]$ ,  $\eta_{\mathrm{n}} \in [57, 617]$ ,  $\eta_{\mathrm{EI}} \in [3.8 \times 10^{-24}, 0.08]$ ,  $\eta_{\mathrm{PI}} \in [2 \times 10^{-21}, 0.39]$ ,  $\eta_{\mathrm{diff}} \in [0.09, 1.15]$ . In our experiments, we apply the Latin hypercube design method [26] to choose three settings as listed in Section 3.2.
4.2 Computational budget saving versus performance degradation
There is a trade-off when early terminating BO, i.e., the performance of BO can be compromised when using less FEs. In this subsection, we employ  $\mathrm{I}_{\mathrm{cost}}$  and  $\mathrm{I}_{\mathrm{perf}}$  to characterize such trade-off. From the comparison results shown in Figure 6 and Table 1, we can see that although the naïve method achieves the best  $\mathrm{I}_{\mathrm{perf}}$ , it consumes almost all FEs. In contrast, our proposed method saves up to  $\approx 80\%$  computation budget while the performance degradation is up to a order of magnitude smaller than the other three termination methods. As the trajectories of the regret of BO versus the number of FEs shown in Figure 7, we can see that the other three termination methods suffer from a premature early stopping.
Table 1: The statistical comparison results of different termination methods on  ${\mathrm{I}}_{\text{cost }}$  and  ${\mathrm{I}}_{\text{perf }}$  .
Metrics	Thresholds	Naïve method	Nguyen's method	Lorenz's method	Makarova's method	Proposed method
Icost	η1	1(0)†	0.1313(4.48E-1)‡	0.1244(4.48E-1)‡	0.7856(4.17E-1)†	0.6206(3.17E-1)
η2	1(0)†	0.1082(3.5E-2)‡	0.1053(1.47E-2)‡	0.1414(3.89E-2)‡	0.3012(1.94E-1)
η3	0.8343(2.51E-1)†	0.1048(9.01E-3)‡	0.1044(4.20E-3)‡	0.1313(3.33E-2)‡	0.2209(1.12E-1)
Ip erf	η1	0(0)‡	0.0077(7.28E-2)†	0.0067(7.71E-2)†	0(1.56E-2)†	0(6.81E-3)
η2	0(0)‡	0.0614(1.08E-1)†	0.0721(1.13E-1)†	0.0167 (6.51E-2)†	0(3.35E-2)
η3	0(0)‡	0.0704(1.14E-1)†	0.0978(1.18E-1)†	0.0355 (8.08E-2)†	0.0028(4.17E-2)
$^\dagger$  denotes the performance of our proposed method is significantly better than the other peers according to the Wilcoxon's rank sum test at a 0.05 significance level;  $^\ddagger$  denotes the opposite case.
Figure 8: Bar charts with error bars of  $\mathrm{I}_{\mathrm{cost}}$  and  $\mathrm{I}_{\mathrm{perf}}$  when using  $\tau \in \{2i\}_{i=1}^{9}$  in our proposed termination method.
4.3 Parameter Sensitivity Study
In this subsection, we investigate the sensitivity of our proposed termination method with respect to the parameter  $\tau$ . We consider various settings of  $\tau \in \{2i\}_{i=1}^{9}$  and repeat the experiments on all benchmark problems introduced in Section 3.1. The aggregated comparison results for  $\mathrm{I_{cost}}$  and  $\mathrm{I_{perf}}$  are illustrated as bar charts with error bars in Figure 8. Specifically, we present the results for  $\eta_{\mathrm{lb}} = 2.05$ , while the complete results can be found in the supplementary document. The plots show that the choice of  $\tau$  has minimal impact on the results, except for cases with  $\tau = 2$  and  $\tau = 4$ . This is reasonable, as the termination method may not utilize sufficient previous information when only considering a few observed samples. Additionally, we examine the scenario where the equality constraint in Condition 1, i.e., equation (5) is relaxed. The comparison results in Figure 8 reveal similar observations regarding the settings of  $\tau$ . However, we also notice a slight performance degradation and more aggressive early stopping in this case. These findings demonstrate that the Condition 1 helps mitigate the risk of premature early stopping.
5 Other Partially Related Works
Despite the limited number of dedicated studies on termination criteria for BO, various efforts have been made to explore early stopping strategies in different contexts.
The first category primarily focuses on detecting change points in sequential processes [34], with applications spanning various fields such as financial analysis [20], bioinformatics [7], and network traffic data analysis [23], among others. However, modeling the automatic termination of BO as a change point detection (CPD) problem may present several challenges. These include: 1) the absence of suitable stopping metrics that can provide signals for CPD in the optimization process of BO; 2) the unknown and uncertain nature of signal distribution, the number of change points, and change point consistency; 3) limited data available for CPD; and 4) the necessity to further evaluate change points in order to determine an appropriate moment for terminating BO.
The second category primarily focuses on determining the statistically optimal stopping moment for generalized sequential decision-making processes [14, 13]. For instance, in the classical secretary problem, termination criteria are developed to identify the maximum of an unknown distribution with minimal cost through sequential search [12]. They typically establish relationships between the costs and rewards of decision-making using cost coefficients [8, 2, 3], unknown observation costs [15, 37, 30, 25] or discount factors [36], subsequently deriving statistically optimal stopping conditions. However, quantifying the relationship between the improvement of the fitness and the
cost of BO remains challenging. Furthermore, these criteria do not leverage the information provided by the surrogate model, which is crucial in BO.
The third category primarily aims to balance exploration and exploitation in the optimization process. Among them, heuristic methods, exemplified by simulated annealing, are widely employed to halt the local search step of optimization algorithms [17, 1, 21]. However, such methods' hyperparameters lack interpretability and must be fine-tuned according to different problem characteristics. Additionally, McLeod et al. propose a regret-based strategy for switching between local and global optimization. Although promising for complex functions, this approach has certain limitations, including reliance on the authors' proposed regret reduction acquisition function and the potential need for additional computational resources to approximate intractable integrals. Furthermore, Eriksson et al. developed a trust-region-based BO that balances exploitation and exploration. This algorithm terminates local search when the trust region size is reduced to zero. However, the termination criteria lack theoretical guarantees and are bound to the proposed trust region maintenance mechanism.
In this paper, we developed a simple yet theoretically grounded two-step method for automatically terminating BO. The key insight is to proactively detect the local convex region and it terminates BO whenever the termination indicator built upon the local regret therein falls below a predetermined threshold. Our proposed termination method naturally strikes a balance between the quality of solution found by BO versus its computational efficiency. The proposed termination method is supported by robust theoretical underpinnings, and we have additionally introduced an approximation method to enhance the numerical stability by solving a bilevel optimization problem. Our extensive empirical studies, conducted across a variety of benchmark problems, including synthetic functions, reinforcement learning, and hyperparameter optimization, consistently demonstrated the better performance of our proposed method compared to other state-of-the-art techniques.
Besides, experimental results also show that the termination criterion of our proposed method is robust across different problems. This property paves an additional opportunity for our proposed termination method to go beyond automatically terminate BO, but to a broader range of applications, such as early stopping to avoid overfitting in neural network training, change point or anomaly detection in data stream, and even a new perspective to strike the balance between exploitation and exploration under a bandit setting. The primary limitation of the proposed termination criterion is that it requires a predefined termination threshold, which needs to be determined based on prior knowledge or empirical observations. Although a recommended threshold selection range is given here, finding an optimal threshold that suits a wide range of optimization problems remains a challenge.
SL implemented the theoretical derivations and experiments, as well as drafted the manuscript; KL piloted the idea and re-wrote the manuscript; WL proofread the manuscript.
This work was supported in part by the UKRI Future Leaders Fellowship under Grant MR/S017062/1 and MR/X011135/1; NSFC under Grant 62376056 and 62076056; the Royal Society under Grant IES/R2/212077; the Kan Tong Po Fellowship (KTP/R1/231017); the EPSRC under Grant 2404317; the Amazon Research Award and Alan Turing Fellowship; and the National Natural Science Foundation of China under Grant 62273119.
[1] Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures. In ICML'18: Proc. of the International Conference on Machine Learning, pages 462-471. PMLR, 2018.
[2] Bruno Betro and Fabio Schoen. Sequential stopping rules for the multistart algorithm in global optimisation. Mathematical Programming, 38(3):271-286, 1987.
[23] Alexandre Lung-Yut-Fong, Céline Lévy-Leduc, and Olivier Cappé. Distributed detection/localization of change-points in high-dimensional network traffic data. Statistics and Computing, 22(2):485-496, 2012.
[24] Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, Andreas Krause, Matthias W. Seeger, and Cedric Archambeau. Automatic termination for hyperparameter optimization. In AutoML'22: Proc. of 2022 International Conference on Automated Machine Learning, volume 188 of Proceedings of Machine Learning Research, pages 7/1-21. PMLR, 2022.
[25] Gustavo Malkomes, Charles Schaff, and Roman Garnett. Bayesian optimization for automated model selection. pages 2892-2900, 2016.
[26] Michael D. McKay, Richard J. Beckman, and William J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 42(1):55-61, 2000.
[27] Mark McLeod, Stephen Roberts, and Michael A Osborne. Optimization, fast and slow: optimally switching between local and bayesian optimization. In ICML'18: Proc. of the International Conference on Machine Learning, pages 3443-3452. PMLR, 2018.
[28] Vu Nguyen, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. Regret for expected improvement over the best-observed value and stopping condition. In ACML '17: Proc. of The 9th Asian Conference on Machine Learning, volume 77 of Proceedings of Machine Learning Research, pages 279–294. PMLR, 2017.
[29] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proc. IEEE, 104(1):148-175, 2016.
[30] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NeurIPS'12: Proc. of the Twenty-sixth Conference on Neural Information Processing Systems, pages 2951-2959, 2012.
[31] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML'10: Proc. of the 27th International Conference on Machine Learning, pages 1015-1022. Omnipress, 2010.
[32] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory, 58(5):3250-3265, 2012.
[33] S. Surjanovic and D. Bingham. Virtual library of simulation experiments: Test functions and datasets. Retrieved January 20, 2023, from http://www.sfu.ca/~ssurjano.
[34] Charles Truong, Laurent Oudre, and Nicolas Vayatis. Selective review of offline change point detection methods. Signal Processing, 167:107299, 2020.
[35] Christopher Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
[36] Tianyi Zhang, Daniel Russo, and Assaf Zeevi. Learning to stop with surprisingly few samples. In COLT'21: Proc. of the Conference on Learning Theory, pages 3887-3888. PMLR, 2021.
[37] Shlomo Zilberstein. Using anytime algorithms in intelligent systems. AI magazine, 17(3):73-83, 1996.