SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$g_{d}$ , where  $g_{d} \in [1, \ldots, g_{max}]$ . At each iteration of the MCMC scheme, two dimensions  $i, j = \{(i, j) \in [1, \ldots, D], i \neq j\}$ , are sampled uniformly at random and assigned the same new group index  $g$ , where  $g \sim \mathcal{U}(1, g_{max})$ . Setting  $g_{i} = g, g_{j} = g$  thus proposes that dimensions  $i$  and  $j$  belong to the same additive decomposition. Under this proposal distribution, the proposed number of additive decompositions can never surpass  $g_{max}$ , but a lesser number of additive groups can be proposed. We utilize the same warm-starting mechanism as described in Gardner et al. [19], where at each iteration, the final accepted sample from the previous iteration acts as the initial proposal. The substitution of proposal distribution from the (unavailable) original implementation [19] was made to simplify a batch GP implementation in GPyTorch [20]. We choose to employ a BoTorch prior on the hyperparameters for all Additive GP tasks and use a squared exponential kernel, as described in [19].
Task	gmax
GP sample - (2 + 2 + 2)D	3
GP sample - (2 + 2 + 2 + 2 + 2)D	5
Cosmological constants	4
HEBO Experiments For the warped GP experiments, we adapt slightly more conservative priors due to the large number of hyperparameters of the model. Specifically, we set  $\mathcal{LN}(0,1)$  priors on the on lengthscales, outputscale variance and noise variance. For the input warpings, we employ a Kuwaraswamy distribution, a differentiable-CDF alternative to the input warpings proposed by Snoek et al. [56]. For each input dimension  $j$ , the untransformed input  $x_{j}$  has a transformation applied as
$$
z _ {j} = \left(1 - x _ {j} ^ {\alpha}\right) ^ {\beta} \tag {11}
$$
where  $z_{j}$  is the resulting, transformed input. The warping parameters  $\alpha$  and  $\beta$  are both given a  $\mathcal{LN}(0,0.1)$  prior. We note that the HEBO experiments are susceptible to the number of hyperparameter sets. As such, we do not recommend running the experiments than fewer number of hyperparameter sets than that outlined by Tab. 3, as substantially fewer did not yield substantial empirical gains relative to vanilla BO.
Table 2: Additive tasks and their respective maximal number of maximal additive decompositions.
Task	Warmup	Thinning	No. hyperparameter sets	No. optima	No. RFFs
Active Learning	256	16	16	N/A	N/A
BO - Synthetic	256	16	16	8	8192
BO - GP samples	256	16	16	8	2048
BO - SAASBO	128	8	16	8	8192
BO - Additive GPs	32	4	12	8	2048
BO - Warped GPs	64	6	32	8	2048
Table 3: MCMC hyperparameters for all experiments. For the AddGP experiments, each hyperparameter set involves the sampled additive decomposition and its associated MAP-trained hyperparameters. The total number of hyperparameter sets drawn are Warmup + Thinning * No. Hyperparameter sets.
B.2 Benchmarks
For the active learning benchmarks, we follow Riis et al. [50] in the types of benchmarks and noise levels used. Each benchmark, as well as its search space, dimensionality and noise level is described in Tab. 4 and Tab. 5 for AL and BO, respectively. The noise level for all of the BO synthetic test functions were set to  $\sigma_{\varepsilon} = 0.5$ , except the Rosenbrock benchmarks, where the noise standard deviation was set to  $\sigma_{\varepsilon} = 2.5$  due to the extremely large output range of the function. A smaller noise level would consequently bring the signal-to-noisy ratio under the permitted threshold supported by BoTorch.
Compute resources. All experiments are carried out on Intel Xeon Gold 6130 CPUs. Each repetition of the tasks in Sec. 4.1 and Sec. 4.2 are run on 4 cores, and the tasks in Sec. 4.3 are run on 8 cores. Approximately 1,000 core hours are used for each of the AL synthetic tasks, 2000 for the BO synthetic tasks, and 5000 for each task in Sec. 4.3.
Task	Dimensionality	σε	Search space
Gramacy	1	0.1	[0.5, 2.5]
Higdon	1	0.1	[0, 20]
Gramacy	2	0.05	[-2, 6]D
Branin	2	11.32	[-5, 10] × [0, 15]
Ishigami	3	0.187	[-π, π]D
Hartmann-6	6	0.0192	[0, 1]D
Table 4: Benchmarks used for the active learning experiments.
Task	Dimensionality	σε	Search space
Branin	2	0.5	[−5,10] × [0,15]
Rosenbrock-2	2	2.5	[−1.5,1.5]D
Hartmann-3	6	0.5	[0,1]D
Rosenbrock-4	4	2.5	[−1.5,1.5]D
Hartmann-4	4	0.5	[0,1]D
Hartmann-6	6	0.5	[0,1]D
Table 5: Benchmarks used for the Bayesian optimization experiments.
We display the RMSE performance of each of the SAL variants, a comparison of the MC and MM variants of SAL and SCoreBO.
C.1 AL RMSE Performance
Fig. 11 displays the performance of both SAL variants and benchmark AL acquisition functions for the same set of tasks as in Sec. 4.1. We observe that SAL-WS consistently displays top performance, whereas SAL-HR lags behind substantially. This showcases SAL-HR's emphasis on hyperparameter learning as opposed to global exploration. By accurately assessing hyperparameters while sacrificing global exploration, SAL-HR ensures predictions with calibrated uncertainty, while sacrificing the accuracy in predictive mean that follows from exploring the search space. SAL-WS offers a compromise which performs well under both metrics, sacrificing hyperparameter learning and calibration for accuracy in predictive mean.
Figure 11: Root Mean Squared Error (RMSE) on six active learning functions and the (smoothed) relative rankings throughout each run for QBMGP, BQBC, BALD and SAL using Wasserstein and Hellinger distance. We plot mean and one standard error for 25 repetitions.. SAL-WS is the top performing method, placing first in relative rankings. On Ishigami, only SAL-HR and BALD produces stable results.
C.2 SCoreBO Distance Measure Ablation Analysis
In Fig. 13, We compare the Hellinger and Wasserstein variants of SCoreB0, both utilizing the MC approximation of the statistical distance. Since the MC approximation is asymptotically exact, we can better assess the performance of each distance metric, without having to consider the confounding factor that the MM approximation introduces. We note that SCoreB0-WS outperforms SCoreB0-HR
on two tasks, but SCoreB0-HR is the overall more consistent approach. We hypothesize that the relative failure of SCoreB0-WS on Rosenbrock (4D) is caused by the objective's non-stationarity, which likely causes exceedingly large exploration of the hyperparameter space. This is supported by Fig. D.3, where the hyperparameters diverge over time on the Rosenbrock function.
Furthermore, in Fig. 12, we compare the KL and Hellinger variants of SCoreB0, both utilizing the moment matching estimation of the posterior. Notably, SCoreB0-KL is the natural extension of BALD to the self-correcting framework, in line with Prop. 1. SCoreB0-HR performs marginally better than SCoreB0-KL, winning 4 out of 6 tasks. However, the two variants are relatively close.
Figure 12: Log Regret of the Hellinger and Wasserstein MC-variants of SCoreB0. Both variants are competitive on all benchmarks, except for Wasserstein on Rosenbrock (4D) which lags behind slightly. Overall, Hellinger is more consistent, and wins 4 out of 6 benchmarks.
Figure 13: Log Regret of the Hellinger and KL moment matching variants of SCoreB0. Both variants are competitive on all benchmarks. Overall, Hellinger is marginally better, and wins 3 out of 6 benchmarks with an approximate tie on Branin.
C.3 SAASBO Noise Ablations
In Fig. 13, We measure the performance of SCoreBO and NEI on the embedded 25-dimensional Ackley (4D) and Hartmann (6D) tasks with varying noise levels using the SAAS [15] prior. We run three noise levels for each task: Noiseless (solid line), low (dashed line), where the noise standard deviation corresponds to  $3\%$  of total output range for Hartmann (6D), and  $1.3\%$  for Ackley, and high (dotted line,  $13.3\% / 4\%$ ). With increasing levels of noise, the difficulty of inferring active dimensions is expected to increase substantially, which should in turn hamper BO performance. In Fig. 14, we see that NEI and SCoreBO perform comparably on noiseless tasks (solid line) finding close-to-optimal solutions within 100 iterations. Moreover, for small levels of noise), the performance is still comparable. However, we observe a drastic fall-off in performance for NEI at the highest noise level, whereas SCoreBO's degrades gracefully. Notably, SCoreBO almost retains the performance of the noiseless at the highest noise level for Ackley.
D Hyperparameter convergence
In Figures 15, 16, 17, 18, and 19 We demonstrate examples of hyperparameter convergence in AL and BO, as well as an example of hyperparameter divergence on the Rosenbrock  $(4D)$  function, where SCoreBO performs marginally worse.
Figure 14: Best observed value for varying levels of observation noise for NEI and SCoreBO using SAAS priors on the 25D-embedded lower-dimensional test functions. For Low levels of noise, the performance of NEI and SCoreBO are comparable, but SCoreBO retains performance substantially better for higher levels for noise.
D.1 Active Learning Tasks
We display the hyperparameter convergence of SAL-WS, SAL-HR and the baseline active learning acquisition functions in Fig. 15. Both variants display accelerated hyperparameter learning compared to BQBC. SAL-HR in particular achieves low-variance hyperparameter uncertainty on Ishigami and the higher-dimensional Hartmann-6, where other methods struggle. We obtain approximately correct hyperparameters for these tasks by randomly sampling 300 points on the noiseless benchmark, thereafter performing MCMC and averaging the sampled hyperparameter estimates in logspace. The noise level is known a priori. and estimates the other hyperparameters with substantially greater certainty than other methods. We note that there is a drift in the hyperparameters as the number of observations increase, where output- and lengthscales trade off to reduce model complexity. As such, we provide an approximately stationary alternative in Fig. 16, where the outputscale is removed to avoid drift. In both cases, SAL-HR displays superior hyperparameter convergence, obtaining accurate hyperparameters in far fewer iterations, and with substantially less uncertainty than the alternatives.
Figure 15: Hyperparameter convergence on the Ishigami test function with outputscale. While no acquisition converges, SAL-HR and BALD display substantially more stable hyperparameters than other approaches.
Figure 16: Hyperparameter convergence on the Ishigami test function without outputscale. SAL-HR and BALD display stable hyperparameter convergence, and are the only acquisition function to accurately estimate all parameters.
D.2 GP sample tasks
We display the convergence of SCoreBO and NEI with a wide lognormal and BoTorch prior on the GP sample task. We observe that the lognormal prior is well-aligned for most hyperparameters, whereas BoTorch prior is misaligned. This is evidenced by the unimportant dimensions  $\ell_6, \ell_7$ , and  $\ell_8$ , which have suggested lengthscales that are incorrect by more than an order of magnitude. Nevertheless, SCoreBO suggests lengthscales that are approximately twice as long ( $10^{0.25} \approx 1.8$ ) as NEI ( $10^{-0.05} \approx$
0.9), and thus avoids unnecessary exploration along these dimensions. Moreover, SCoreBO correctly identifies the most important dimensions  $\ell_1, \ell_2$ , and  $\ell_3$  with good accuracy quickly, whereas NEI struggles to identify  $\ell_1$ . SCoreBO slightly overestimates the importance of dimensions 2 and 3, likely to compensate for the inability to accurately estimate the importance of other hyperparameters.
Figure 17: Hyperparameter convergence on the 8-dimensional GP sample for the broad log-normal prior. The black dashed line indicates true hyperparameter values. Mean and standard deviation are plotted across 20 repetitions, and a 3 iteration moving average of the plotted moments is applied to increase readability. Lengthscales  $\ell_d$  ordered smallest (most important) to largest (least important). SCoreBO finds accurate hyperparameters faster, has the most accurate values for all hyperparameters, and has substantially lower variance for all important (i.e. not  $\ell_6$ ,  $\ell_7$ , and  $\ell_8$ ) hyperparameters except for the noise variance.
Figure 18: Hyperparameter convergence on the 8-dimensional GP sample for the BoTorch priors. The black dashed line indicates true hyperparameter values. Mean and standard deviation are plotted across 20 repetitions, and a 3 iteration moving average of the plotted moments is applied to increase readability. Lengthscales  $\ell_d$  ordered smallest (most important) to largest (least important). SCoReBO finds accurate hyperparameters faster, has the most accurate values for all hyperparameters, and has substantially lower variance for all important (i.e. not  $\ell_6$ ,  $\ell_7$ , and  $\ell_8$ ) hyperparameters except for the noise variance.
D.3 Hyperparameter Divergence on Synthetic BO tasks
We highlight additional examples on synthetic BO test functions where hyperparameters diverge. Due to the non-stationary structure of Rosenbrock in particular (and to a lesser extent, Branin), hyperparameters values diverge as the number of observations increase. In particular, the extreme steepness along the edges suggests an exceedingly large outputscale. With increasing observations, a lengthscale-outputscale trade-off occurs, where both hyperparameters grow seemingly indefinitely. Notably, this behavior is consistent regardless of the acquisition function (BO, AL, SOBOL). Due to the restricted hyperparameter set employed in the AL tasks, this problem is distinct to the BO tasks.
Figure 19: Hyperparameter divergence for SCoreBO and NEI on Rosenbrock (4D). The outputscale grows larger with increasing iterations, and the lengthscales grow similarly large as a countermeasure.
E Approximation Strategies
We display the quality of the moment matching approximation for both the Hellinger and Wasserstein distance. Moreover, we compare the performances of the MM and MC approaches.
E.1 MC approximation of SAL and SCoreBO
Using Monte Carlo, different distances are most efficiently estimated in different manners. To approximate the Wasserstein distance, we utilize quasi-Monte Carlo. From the definition of the distance in one dimension, we obtain
$$
W ^ {2} (p, q) = \int_ {0} ^ {1} | Q (u) - P (u) | ^ {2} d u \approx \sum_ {\ell = 1} ^ {L} | Q (u _ {\ell}) - P (u _ {\ell}) | ^ {2}, \tag {12}
$$
where  $u_{\ell} \sim \mathcal{U}(0,1)$ , and  $P(x)$  and  $Q(x)$  are the respective cumulative distributions for  $p(x)$  and  $q(x)$ . To approximate the Hellinger distance, we obtain
$$
H ^ {2} (p, q) = 1 - \int_ {\mathcal {X}} \sqrt {\frac {q (x)}{p (x)}} p (x) d x \approx 1 - \sum_ {\ell = 1} ^ {L} \sqrt {\frac {q \left(x _ {\ell}\right)}{p \left(x _ {\ell}\right)}}, \tag {13}
$$
where  $x_{\ell} \sim p(x)$  is sampled using MC. In SCoreB0,  $p(x)$  is the marginal  $p(y_{\pmb{x}}|\mathcal{D})$ , and  $q(x)$  each of the various conditionals  $p(y_{\pmb{x}}|*,\pmb{\theta},\mathcal{D})$ .
E.2 Performance of Monte Carlo
We display the performance of the MC variants of SAL-WS and SCoreB0-HR compared to their MM counterparts. Overall, performances are comparable, as each variant slightly exceeds the other on a couple of benchmarks. On the most complex benchmarks (Ishigami, Hartmann-4, Hartmann-6), the MC variant outperforms MM slightly, which suggests that MC is increasingly justified as disagreement in the posterior gets larger.
Figure 20: Negative marginal log likelihood (MLL) of the SAL MC (blue) and MM (red) variants on the active learning benchmarks. Overall performance is comparable, with three effectively tied benchmarks. MC outperforms slightly Hartmann-4 and Hartmann-6, and MM on Hartmann-3.
Figure 21: Log regret of SCoreB0-MM and SCoreB0-MC on the synthetic BO benchmarks. Overall performance is comparable, with MM outperforming marginally on 4 out of 6 tasks. MC notably outperforms slightly on the difficult Ishigami test function.
E.3 Hellinger Distance Approximation
We display the accuracy of the moment matching approximation, and the sensitivity of the MC approximation to the number of samples  $L$ . In Fig. 24 and Fig. 25, we highlight two examples of the moment matching approximation in comparison to a large-scale, asymptotically exact variant of the MC approximation with 2048 samples. In Fig. 24, the MM approximation struggles to capture the sharp, multimodal surfaces in (blue), and consistently overestimates the distance in (orange). In Fig. 25, the included conditional posteriors are substantially more similar, and as such, the moment matching approximation is more accurate. The shape of the acquisition function is captured almost perfectly, and the magnitude is only marginally overestimated, most prominently in (green). We display the
Figure 22: Example of the per-sample Hellinger distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation mostly retains the shape of the asymptotically exact variant. However, it does not perfectly capture the multi.modality in (blue), and overestimates the distance in the low-variance region at the right edge of (orange). The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
Figure 23: Example of the per-sample Hellinger distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation captures the shape of the asymptotically exact variant well, but overestimates the distance slightly in (green). The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
E.4 Wasserstein Distance Approximation
Figure 24: Example of the per-sample Wasserstein distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation mostly retains the shape of the asymptotically exact variant. The shape of the acquisition function is generally well captured, but high-variance regions have their distance underestimated by the moment matching approach, and low-variance regions have their distance over-estimated, leading to a biased approximation. The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
Figure 25: Example of the per-sample Wasserstein distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation captures the shape of the asymptotically exact variant well, and only marginally over- and underestimates the distance. The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
F Equivalence to BALD
We show that SAL, using the KL divergence as the metric  $d$ , is equivalent to BALD.
$$
\begin{array}{l} B A L D = I (y; \boldsymbol {\theta}) = H (p (y)) - E _ {\boldsymbol {\theta}} [ H (p (y | \boldsymbol {\theta})) ] = \\ = - \int_ {- \infty} ^ {\infty} \int_ {\boldsymbol {\theta}} p (\boldsymbol {\theta}) p (y | \boldsymbol {\theta}) \log [ p (y) ] d \boldsymbol {\theta} d y + \int_ {\boldsymbol {\theta}} p (\boldsymbol {\theta}) \int_ {- \infty} ^ {\infty} p (y | \boldsymbol {\theta}) \log [ p (y | \boldsymbol {\theta}) ] d y d \boldsymbol {\theta} = \\ = \int_ {\boldsymbol {\theta}} p (\boldsymbol {\theta}) \int_ {- \infty} ^ {\infty} p (y | \boldsymbol {\theta}) \log \left[ \frac {p (y | \boldsymbol {\theta})}{p (y)} \right] d y d \boldsymbol {\theta} = \int_ {\boldsymbol {\theta}} p (\boldsymbol {\theta}) K L (p (y | \boldsymbol {\theta}) | | p (y)) d \boldsymbol {\theta} = \\ = E _ {\boldsymbol {\theta}} \left[ K L (p (y | \boldsymbol {\theta}) | | p (y)) \right] = S A L. K L \\ \end{array}
$$
G Additional Background
We provide additional background on fully Bayesian hyperparameter treatment in Gaussian processes and details regarding statistical distance metrics.
G.1 Fully Bayesian Treatment
The posterior probability of observing a value  $y_{x}$  for a point  $x$  is given as:
$$
\begin{array}{l} p \left(y _ {\boldsymbol {x}} \mid \mathcal {D}\right) = \int_ {\boldsymbol {\theta}} \int_ {f} p \left(y _ {\boldsymbol {x}} \mid f, \boldsymbol {\theta}\right) p (f \mid \boldsymbol {\theta}, \mathcal {D}) p (\boldsymbol {\theta} \mid \mathcal {D}) d f d \boldsymbol {\theta} \\ = \int_ {\boldsymbol {\theta}} \int_ {f} p (y _ {\boldsymbol {x}} | f, \boldsymbol {\theta}) p (f | \boldsymbol {\theta}, \boldsymbol {x}, \mathcal {D}) p (\boldsymbol {\theta} | \mathcal {D}) d f) d \boldsymbol {\theta}, \\ \end{array}
$$
where  $f$  are the noiseless, latent function values as  $x$  and  $\mathcal{D}$  is the observed data. The inner integral is equal to the GP predictive posterior,
$$
\int_ {f} p (y _ {\boldsymbol {x}} | f, \boldsymbol {\theta}) p (f | \boldsymbol {\theta}, \mathcal {D}) d f = p (y _ {\boldsymbol {x}} | \mathcal {D}, \boldsymbol {\theta}).
$$
However, the outer integral is intractable and is estimated using Markov Chain Monte Carlo (MCMC) methods. The resulting posterior prediction
$$
p (y _ {\boldsymbol {x}} | \mathcal {D}) = \int_ {\boldsymbol {\theta}} p (y _ {\boldsymbol {x}} | \mathcal {D}, \boldsymbol {\theta}) p (\boldsymbol {\theta} | \mathcal {D}) d \boldsymbol {\theta} \approx \frac {1}{M} \sum_ {j = 1} ^ {M} p (y _ {\boldsymbol {x}} | \mathcal {D}, \boldsymbol {\theta} _ {j}), \boldsymbol {\theta} _ {j} \sim p (\boldsymbol {\theta} | \mathcal {D}),
$$
is a Gaussian Mixture Model (GMM).
Within BAL and BO, the fully Bayesian treatment is often extended to involve the acquisition function, such that the acquisition function  $\alpha$  is computed as an expectation over the hyperparameters [46, 55]
$$
\alpha (\boldsymbol {x} | \mathcal {D}) = \mathbb {E} _ {\boldsymbol {\theta}} [ \alpha (\boldsymbol {x} | \boldsymbol {\theta}, \mathcal {D}) ] \approx \frac {1}{M} \sum_ {j = 1} ^ {M} \alpha (\boldsymbol {x} | \boldsymbol {\theta} _ {j}, \mathcal {D}) \boldsymbol {\theta} _ {j} \sim p (\boldsymbol {\theta} | \mathcal {D}).
$$
G.2 Statistical Distance Details
The Hellinger distance is a similarity measure between two probability distributions which has previously been employed in the context of BO-driven automated model selection by Malkomes et al. [39]. For two probability distributions  $p$  and  $q$ , it is defined as
$$
H ^ {2} (p, q) = \frac {1}{2} \int_ {\mathcal {X}} \left(\sqrt {p (x)} - \sqrt {q (x)}\right) ^ {2} \lambda d x, \tag {14}
$$
with some auxiliary measure  $\lambda$  with which both  $p$  and  $q$  are absolutely continuous. Specifically, for two normally distributed variables  $z_{1} \sim \mathcal{N}(\mu_{1}, \sigma_{1}^{2})$ ,  $z_{2} \sim \mathcal{N}(\mu_{2}, \sigma_{2}^{2})$ ,
$$
H ^ {2} \left(z _ {1}, z _ {2}\right) = 1 - \sqrt {\frac {2 \sigma_ {1} \sigma_ {2}}{\sigma_ {1} ^ {2} + \sigma_ {2} ^ {2}}} \exp \left[ - \frac {1}{4} \frac {\left(\mu_ {1} - \mu_ {2}\right) ^ {2}}{\sigma_ {1} ^ {2} + \sigma_ {2} ^ {2}} \right]. \tag {15}
$$
The Hellinger distance seeks to minimize the ratio between difference in mean and the sum of variances, which punishes outlier predictive distributions of high confidence. Similar to KL, initial queries have a tendency to be axis-aligned to attain selective length scale information.
The Wasserstein distance is the average distance needed to move the probability mass of one distribution to morph into the other. The Wasserstein- $k$  distance is defined as
$$
W _ {k} (p, q) = \left(\int_ {0} ^ {1} \left| F _ {q} (x) - F _ {p} (x) \right| ^ {k} d x\right) ^ {1 / k} \tag {16}
$$
For the normal distributions  $z_{1}$  and  $z_{2}$ , the Wasserstein-2 distance is defined as
$$
W _ {2} \left(z _ {1}, z _ {2}\right) = \sqrt {\left(\mu_ {1} - \mu_ {2}\right) ^ {2} + \left(\sigma_ {1} - \sigma_ {2}\right) ^ {2}}. \tag {17}
$$
In practice,  $W_{2}$  places a premium on matching large-variance regions, leading to higher global exploration which can be detrimental for global optimization.