SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider two proper statistical distances, Wasserstein distance and Hellinger distance. In contrast to BQBC, the statistical distance between the normally distributed conditionals and the marginal posterior predictive distribution (which is a Gaussian mixture), is not available in closed-form. We propose two approaches: estimating the distances using MC, which we outline for both distances in App E.1, and estimation using moment matching (MM), which we outline below.
Approximation through Moment Matching We propose to fully utilize the closed-form expressions of the involved distances for Gaussians, and approximate the full posterior mixture  $p(y_{\mathbf{x}}|\mathcal{D})$  with a Gaussian distribution using moment matching (MM) for the first and second moment. While a Gaussian mixture is not generally well approximated by a Normal distribution, we show empirically in App. E that the distance between the conditionals and the approximate posterior is small. In the moment matching approach, the conditional posterior  $p(y_{\mathbf{x}}|\pmb {\theta},\ast ,\mathcal{D})$  utilizes a lower bound on the change in the posterior induced by conditioning on  $\ast$ , as derived in GIBBON [42], which conveniently involves a second moment matching step of the extended skew Gaussian [45]  $p(y_{\mathbf{x}}|\pmb {\theta},\ast ,\mathcal{D})$ . This naive approach circumvents a quadratic cost  $\mathcal{O}(N^2 M^2)$  in the number of samples of each pass through the acquisition function, and yields comparable performance to the MC estimation procedures proposed in App. E.1. In App. E, we qualitatively assess the accuracy of the MM approach for both distances, and display its ability to preserve the shape of the acquisition function.
In this section we showcase the performance of the SAL and SCoreBO acquisition functions on a variety of tasks. For active learning, SAL shows state-of-the-art performance on a majority of benchmarks, and is more robust than the baselines. For the optimization tasks, SCoreBO more efficiently learns the model hyperparameters, and outperforms prominent Bayesian optimization acquisition functions on a variety of tasks. All experiments are implemented in BoTorch [2] $^{1}$ . We use the same  $\mathcal{LN}(0,3)^2$  hyperparameter priors as Riis et al. [50] unless specified otherwise. SCoreBO and all baselines utilize fully Bayesian treatment of the hyperparameters. The complete experimental setup is presented in detail in Appendix B, and our code is publicly available at https://github.com/hvarfner/scorebo.git. We utilize the moment matching approximation of the statistical distance. Experiments for the MC variant of SCoreBO are found in App. E.2.
4.1 Active Learning Tasks
To evaluate the performance of SAL, we compare it with BALD, BQBC and QBMGP on the same six functions used by Riis et al. [50]: Gramacy (1D) has a periodicity that is hard to distinguish from noise, Higdon and Gramacy (2D) varies in characteristics in different regions, whereas Branin, Hartmann-6 and Ishigami have a generally nonlinear structure. We display both the Wasserstein and Hellinger distance versions of SAL, denoted as SAL-WS and SAL-HR, respectively. We evaluate each method on their predictive power, measured by the negative Marginal Log Likelihood (MLL) of the model predictions over a large set of validation points. MLL emphasizes calibration (accurate uncertainty estimates) in prediction over an accurate predictive mean. In Fig. 11, we show how the average validation set MLL changes with increasing training data. SAL-HR is the top-performing acquisition function on three out of six tasks, and rivals BALD for stability in predictive performance. This is particularly evident on the Ishigami function, where most methods fluctuate in the quality of their predictions. This can be attributed to emphasis on rapid hyperparameter learning, which is visualized in detail in App. D, Fig. 15. In the rightmost plot, the real-time average per-seed ranking of acquisition function performance is displayed as a function of the fraction of budget expended. SAL-HR performs best, followed by BQBC and BALD. SAL-WS, however, does not display similarly consistent predictive quality as SAL-HR. The ability of SAL-HR to correctly estimate hyperparameters ensures calibrated uncertainty estimates, which makes it the better candidate for BO. In App. C.1, Fig. 11, we show the evolution of the average Root Mean Squared Error (RMSE) of the same tasks, where SAL-WS performs best and SAL-HR lags behind, which demonstrates the viability of various distance metrics on different tasks.
4.2 Bayesian Optimization Tasks
For the BO tasks, we use the Hellinger distance for its proficiency in prediction calibration and hyperparameter learning. We compare against several state-of-the-art baselines from the BO literature: NEI for noisy experiments [36], as well as JES [29], the MES approach GIBBON [42] and PES [24]. As an additional reference, we include NEI for noisy experiments [36] using MAP estimation.
1https://botorch.org/(v0.8.4)
2All Normal and LogNormal distributions are parametrized by the mean and variance.
Figure 4: Negative Marginal Log Likelihood (MLL) on six active learning functions and the (smoothed) relative rankings throughout each run for QBMGP, BQBC, BALD and SAL using Wasserstein and Hellinger distance. We plot mean and one standard error for 25 repetitions.. SAL-HR is the top performing method, placing first in relative rankings. On Ishigami, only SAL-HR and BALD produces stable results.
Figure 6: Average log inference regret and (smoothed) relative ranking across 50 repetitions between the acquisition functions for SCoreBO, JES, MES and NEI on six synthetic test functions. SCoreBO produces the best final regret on 4 out of 6 tasks, and has a substantially lower average ranking by the end of each run.
Efficiently learning the hyperparameters To showcase SCoreBO's ability to find the correct model hyperparameters, we run all relevant acquisition functions on samples from the 8-dimensional GP in Fig. 1. We exploit that for GP samples, the objectively true hyperparameters are known (in contrast to typical synthetic test functions). We utilize the same priors as in Fig. 1 on all the hyperparameters and compare SCoreBO to NEI to assess the ability of each acquisition function to work independently of the choice of prior. In Fig. 5, for each acquisition function, we plot the average log regret over 20 different 8-dimensional instances of this task. The tasks at hand have lengthscales that vary substantially between dimensions, as detailed in App B. The explanation for the good performance of SCoreBO can be see in Fig. 17 in App. D, where SCoreBO converges substantially faster towards the correct hyperparameter values than NEI for both types of priors.
Synthetic test functions We run SCoreBO on a number of commonly used synthetic test functions for  $25|\theta|$  iterations, and present how the log inference regret evolves over the iterations in Fig. 6. All benchmarks are perturbed by Gaussian noise. We evaluate inference regret, i.e., the current best guess of the optimal location  $\arg \max_{\boldsymbol{x}} \mu(\boldsymbol{x})$ , which is conventional for nonmyopic acquisition functions [22, 24, 29]. SCoreBO yields the best final regret on four of the six tasks. In the relative rankings (rightmost plot), SCoreBO ranks poorly initially, but once hyperparameters are learned approximately halfway through the run, it substantially outperforms the competition. On Rosenbrock (4D), the relatively poor performance can be explained by the appar
ent non-stationarity of the task, detailed in Fig. D.3, which makes hyperparameters diverge over time. This exposes a weakness of SCoreB0: When the modeling assumptions (such as stationarity) do not align with the task, optimization performance may suffer due to perpetual disagreement in the posterior. In App. C.2, we display the performance of SCoreB0-KL and SCoreB0-WS on the same set of benchmarks, where both display highly competitive performance.
Figure 5: Regret for NEI and SCoreBO on the 8-dimensional GP sample for two different types of hyperparameter priors. Mean and standard deviation are plotted for all hyperparameter samples across 20 repetitions.
Figure 7: Final loss using SAASBO priors on the noisy embedded Ackley-4, embedded Hartmann-6, the DNA classification and the SVM HPO task, mean and one standard error. SCoreBO identifies the important dimensions rapidly, and successfully optimizes the tasks. The optimal value is marked with a dashed line.
Figure 8: Hyperparameter convergence on the  $25D$ -embedded  $4D$  Ackley function with a SAASBO HP prior for SCoreBO, NEI and BALD. Log HP mean and 1 standard deviation is plotted per iteration. SCoreBO identifies  $\ell_1, \ldots, \ell_4$  as important (short lengthscales,  $\ell_i \approx 10^{-1}$ ) with low uncertainty and  $\ell_5, \ldots, \ell_{25}$  as dummy dimensions ( $\ell_i \gg 10^1$ ). NEI fails to identify any important lengthscales, whereas SCoreBO correctly identifies active dimensions with high certainty.. Notably, SCoreBO finds accurate hyperparameters even faster than BALD, a pure active learning approach. Reference HP values (where available) are marked with a dashed line.
4.3 A Practical Need for Self-correction
Lastly, we evaluate the performance of SCoreBO on three atypical tasks with increased emphasis on the surrogate model: (1) high-dimensional BO through sparse adaptive axis-aligned priors (SAASBO) [15], (2) BO with additively decomposable structure (AddGPs) [19, 32] and (3) nonstationary, heteroskedastic modelling with HEBO [13]. Eriksson & Jankowiak [15] consider their proposed method for noiseless tasks, where active variables easily distinguish from their non-active counterparts. However, SAASBO is not restricted to noiseless tasks. For AddGPs, data cross-covariance, and lack thereof, is similarly difficult to infer in the presence of noise.
In Fig. 7, we visualize the performance of SCoreBO and competing acquisition functions with SAASBO priors on two noisy benchmarks, Ackley-4 and Hartmann-6, with dummy dimensions added, as well as two real-world benchmarks: fitting a weighted Lasso model in 180 dimensions [53], and the tuning of all 385 lengthscales and three regularization parameters of an SVM [12], a task also considered by Eriksson & Jankowiak [15]. On these benchmarks, where finding the correct hyperparameters is crucial for performance, SCoreBO clearly outperforms traditional methods. To further exemplify how SCoreBO identifies the relevant dimensions, in Fig. 8, we show how the hyperparameters evolve on the 25D-embedded Ackley (4D) task. SCoreBO quickly finds the correct lengthscales and outputscale with high certainty, whereas NEI remains uncertain of which dimensions are active throughout the optimization procedure. Impressively, SCoreBO finds accurate hyperparameters even faster than BALD, despite the latter being a pure active learning approach.
Secondly, we demonstrate the ability of SCoreB0 to self-correct on uncertainty in kernel design, by considering AddGP tasks. We utilize the approach of Gardner et al. [19], where additive decompositions are marginalized over. Ideally, a sufficiently accurate decomposition is found quickly, which rapidly speeds up optimization through accurate cross-correlation of data. Fig. 9 demonstrates SCoreB0's performance on two GP sample tasks and a real-world task estimating cosmological constants (leftmost 3 plots) and its ability to find the correct additive decompositions (right). We observe that SCoreB0 identifies correct decompositions substantially better than NEI. Final performance, however, is only marginally better, as substantial resources are expended finding the right decompositions. Notably, the Cosmological Constants task does not display additive decomposability. As such, SCoreB0 unsuccessfully expends resources attempting to reduce disagreement over additive structures, which hampers performance. This demonstrates that while SCoreB0 learns the problem structure at increased rates, improved BO performance does not automatically follow.
Lastly, we apply SCoreBO to the HEBO [13] GP model, the winner of the NeurIPS 2020 Black-box optimization challenge [62]. The model employs input [56] and output warpings, the former of
Figure 9: Final value of using AddGPs on 6D and 10D GP sample functions, fully decomposable in groups of two, and the Cosmological Constants tasks. SCoreBO achieves better final performance (left, middle) with low uncertainty, and successfully finds the additive components of the 6D task (right).
which are learnable to account for the heteroskedasticity that is prevalent in real-world optimization, and particularly HPO [13, 56], tasks. The complex model provides additional degrees of freedom in learning the objective. We evaluate SCoreB0 and all baselines on three 4D deep learning HPO tasks: two involving large language models, and one from computer vision, from the PD1 [66] benchmarking suite. Fig. 10 displays that SCoreB0 obtains the best final accuracy on 2 out of 3 tasks, suggesting that self-correction is warranted for optimization of deep learning pipelines.
Figure 10: Performance on the PD1 deep learning tasks over 20 repetitions using the warpings from HEBO [13]. SCoreBO obtains the best final accuracy on 2 out of 3 tasks, placing second on the third.
The hyperparameters of Gaussian processes play an integral role in the efficiency of both Bayesian optimization and active learning applications. In this paper, we propose Statistical distance-based Active Learning (SAL) and Self-Correcting Bayesian Optimization (SCoreBO), two acquisition functions that explicitly consider hyperparameter-induced disagreement in the posterior distribution when selecting which points to query. We achieve high-end performance on both active learning and Bayesian optimization tasks, and successfully learn hyperparameters and kernel designs at improved rates compared to conventional methods. SCoreBO breaks ground for new methods in the space of joint active learning and optimization of black-box functions, which allows it to excel in high-dimensional BO, where learning important dimensions are vital. Moreover, the potential downside of self-correction is displayed when the model structure does not support the task at hand, or when self-correction is not required to solve the task. For future work, we will explore additional domains in which SAL and SCoreBO can allow for increased model complexity in BO applications.
SCoreBO displays the ability to increase optimization efficiency on complex tasks that necessitate accurate modeling. However, SCoreBO's efficiency is ultimately contingent on the intrinsic ability of the GP to model the task at hand. Appendix 19 demonstrates this issue for the Rosenbrock (4D) function, where SCoreBO performs worse relative to other acquisition functions. There, the hyperparameter values increase over time instead of converge, which suggests that the objective is not part of the class of functions defined by the kernel. Thus, the self-correction effort is less helpful towards optimization. Moreover, increasing the model capacity, such as in Sec. 4.3, comes with increasing resources allocated towards self-correction. In highly constrained-budget applications, such resource allocation may not yield the best result, especially if increased model complexity is unwarranted. This is evident from the synthetic AddGP tasks, where despite accurately identifying the additive components, SCoreBO does not provide substantial performance gains over NEI. Lastly, SCoreBO's reliance on fully Bayesian hyperparameter treatment makes it more computationally demanding than MAP-based alternatives, limiting its use in high-throughput applications.
We thank the anonymous reviewers for their valuable contributions related SAL-KL and and its relationship with BALD, as well as their general feedback on the clarity of the paper and how our method was conveyed. We also thank Eytan Bakshy for their helpful feedback on earlier versions of this paper. Luigi Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project — Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Carl Hvarfner, Erik Hellsten and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348. Frank Hutter acknowledges support through TAILOR, a project funded by the EU Horizon 2020 research and innovation programme under GA No 952215, by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828, by the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG, and by the European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765). The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC partially funded by the Swedish Research Council through grant agreement no. 2018-05973. Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
[62] Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., and Guyon, I. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Escalante, H. J. and Hofmann, K. (eds.), Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pp. 3-26. PMLR, 06-12 Dec 2021. URL https://proceedings.mlr.press/v133/turner21a.html.
[63] Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10663-10674. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/wan21b.html.
[64] Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. International Conference on Machine Learning (ICML) 38, 2021.
[65] Wang, Z. and Jegelka, S. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning (ICML), 2017.
[66] Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Nado, Z., Gilmer, J., Snoek, J., and Ghahramani, Z. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08215, 2023.
[67] Wilson, J. T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. P. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning, 2020. URL https://arxiv.org/abs/2002.09309.
[68] Wu, J., Poloczek, M., Wilson, A. G., and Frazier, P. Bayesian optimization with gradients. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/64a08e5f1e6c39faeb90108c430eb120-Paper.pdf.
[69] Wynne, G., Briol, F.-X., and Girolami, M. Convergence guarantees for gaussian process means with misspecified likelihoods and smoothness. J. Mach. Learn. Res., 22(1), jan 2021. ISSN 1532-4435.
[70] Yao, Y., Vehtari, A., and Gelman, A. Stacking for non-mixing bayesian computations: The curse and blessing of multimodal posteriors, 2020.
[71] Zhang, B., Cole, D. A., and Gramacy, R. B. Distance-distributed design for gaussian process surrogates, 2019.
We review approaches that address model accuracy in Bayesian optimization.
Expanded Space of Models [38, 39] expands model uncertainty to kernel design in order to find the most accurate model possible within an expanded class of functions, whereas [56] expands model uncertainty to involve warpings of the input space. Since the expanded class of functions offers additional modelling flexibility, BO can ideally be conducted on a more accurate model than under a vanilla setting. As demonstrated with AddGPs in Sec. 4.3, SCoreBO can work in conjunction with kernel search to find such models at an accelerated rate.
Simplified Modelling A contrasting line of work involves methods which restrict modeling by reducing the level of detail [6] or scope of the optimization [16, 44, 63]. This line of work acknowledges that modeling may not, or should not, be globally accurate or granular in order to conduct optimization efficiently within the allocated budget. As such, these lines of work address the issue of model accuracy by diametrically opposed philosophies. SCoreBO offers an orthogonal approach to model accuracy by accelerating the convergence of the model at hand, regardless of its level of complexity.
B Supplementary Material on Experiments
All the relevant methods are implemented as acquisition functions in BoTorch. For both the active learning and BO experiments, we run NUTS [25] in Pyro [5] to draw samples from the GP posterior over hyperparameters. Tab. 3 displays the parameters of the MCMC in detail, as well as other relevant parameters of various MC estimations throughout the article. For the active learning experiments, we mimic the experimental setup used in Riis et al. [50], and put a log-normal distribution  $\mathcal{LN}(0,3)$  on the lengthscales, outputscale variance and noise variance. Furthermore, we consider the mean constant  $c$  as a learnable parameter in the BO experiments, with a conventional  $\mathcal{N}(0,1)$  prior on the standardized inputs. When referring to the BoTorch priors, the priors are  $\Gamma(3,6)$ ,  $\Gamma(2,0.15)$ , and  $\Gamma(1.1,0.05)$  for the lengthscales, outputscale, and noise variance, respectively, with the same prior on the learnable constant mean  $c$ .
8D Gaussian Process sample task For the  $8D$  GP sample task, we utilize the lognormal prior from Sec. B on the hyperparameters. The hyperparameters of the sampled objective functions are outlined in Tab. 1. As such, Tab. 1 displays the true hyperparameters of the task we are trying to optimize. These hyperparameters are referenced in Fig. 17 and Fig. 18
Task	σ2	σ2ε	l	Kernel
GP sample - 8D	1	0.1	exp 10{[-1, -0.5, -0.5, 0, 0, 0, 1.5, 1.5, 1.5]}	Matérn
Table 1: Hyperparameters of the  ${8D}$  GP sample task.
SAASBO experiments For the SAASBO experiments, we utilize  $\mathrm{Ax}^3$ , which runs the BoTorch implementation of SAASBO with the deafult prior on the hyperparameters as per BoTorch version 0.8.4, which differs slightly from the paper by Eriksson & Jankowiak [15]. The lengthscale parameters are given a hierarchical prior as  $\tau^2 \sim \mathcal{HC}(\alpha)$ ,  $\kappa_i^2 \sim \mathcal{HC}(1)$  and  $\ell_i = \frac{1}{\kappa_i\tau}$ . We retain the default value of  $\alpha = 0.1$ . The noise variance, outputscale and mean constant are given the priors  $\sigma_{\varepsilon}^{2} \sim \Gamma(0.9, 10)$ ,  $\sigma_f^2 \sim \Gamma(2, 0.15)$  and  $c \sim \mathcal{N}(0, 1)$ .
Additive Gaussian Process experiments The Additive GP setup closely resembles that of [19]. An additive partitioning is sampled, and the marginal likelihood of the model is maximized with regard to  $\theta = \{\ell, \sigma_f, \sigma_\varepsilon\}$ . We utilize a slightly adapted proposal distribution, and fix a maximal number of additive partitions  $g_{max}$ . Moreover, each dimension  $d$  belongs to one distinct additive decomposition
3https://github.com/facebook/Ax
4https://github.com/pytorch/botorch