SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use a separate GP (denoted as  $\mathcal{GP}^{\prime}$ ) to model the negative noise variance function  $g(\cdot) = -\sigma^{2}(\cdot)$  and use it to build a high-probability upper bound  $U_{t}^{\sigma^{2}}(\cdot)$  on the noise variance function  $\sigma^{2}(\cdot)$ .<sup>2</sup> After this, we can modify the criteria for selecting  $n_{t}^{(b)}$  (i.e., line 6 of Algo. 1) to be  $n_{t}^{(b)} = \lceil U_{t}^{\sigma^{2}}(\boldsymbol{x}_{t}^{(b)}) / R^{2}\rceil$ , which ensures that  $U_{t}^{\sigma^{2}}(\boldsymbol{x}_{t}^{(b)}) / n_{t}^{(b)} \leq R^{2}$ . As a result, the condition of  $\sigma^{2}(\boldsymbol{x}_{t}^{(b)}) / n_{t}^{(b)} \leq R^{2}$  is still satisfied (with high probability), which implies that the observed empirical mean at every queried  $\boldsymbol{x}_{t}^{(b)}$  is still  $R$ -sub-Gaussian (Sec. 3.1.1) and theoretical guarantee of Theorem 3.1 is preserved. To construct  $\mathcal{GP}^{\prime}$ , we use the (negated) unbiased empirical noise variance  $\widetilde{y}_{t}^{(b)}$  as the noisy observation:
$$
\widetilde {y} _ {t} ^ {(b)} = - 1 / \left(n _ {t} ^ {(b)} - 1\right) \sum_ {n = 1} ^ {n _ {t} ^ {(b)}} \left(y _ {t, n} ^ {(b)} - y _ {t} ^ {(b)}\right) ^ {2} = g \left(\boldsymbol {x} _ {t} ^ {(b)}\right) + \epsilon^ {\prime} \tag {1}
$$
where  $g(\pmb{x}_t^{(b)}) = -\sigma^2(\pmb{x}_t^{(b)})$  is the negative noise variance at  $\pmb{x}_t^{(b)}$ , and  $\epsilon'$  is the noise. In BTS-RED-Unknown, we use pairs of  $\{(\pmb{x}_t^{(b)}, \widetilde{y}_t^{(b)})\}$  to update the posterior of  $\mathcal{GP}'$ . We impose a minimum number of replications  $n_{\mathrm{min}} \geq 2$  for every queried input to ensure reliable estimations of  $\widetilde{y}_t^{(b)}$ .
3.2.2 Upper Bound on Noise Variance Function
Assumptions. Similar to Theorem 3.1, we assume that  $g$  lies in an RKHS associated with an SE kernel  $k' \colon \| g \|_{\mathcal{H}_{k'}} \leq B'$  for some  $B' > 0$ , which intuitively assumes that the (negative) noise variance
To simplify the derivations, we have replaced the term  $\lceil \sigma_{\mathrm{max}}^2 /R^2\rceil$  by its upper bound  $\sigma_{\mathrm{max}}^2 /R^2 +1$ , after which the resulting regret upper bound is still valid.
Here we have modeled  $-\sigma^2(\cdot)$  (instead of  $\log \sigma^2(\cdot)$  as done by some previous works) because it allows us to naturally derive our theoretical guarantees, and as we show in our experiments (Sec. 5), it indeed allows our algorithms to achieve compelling empirical performances. We will explore modelling  $\log \sigma^2(\cdot)$  in future work to see if it leads to further empirical performance gains.
varies smoothly across the domain  $\mathcal{X}$ . We also assume that the noise  $\epsilon'$  is  $R'$ -sub-Gaussian and justify this below by showing that  $\epsilon'$  is bounded (with high probability).
$\epsilon'$  is  $R'$ -sub-Gaussian. Since the empirical variance of a Gaussian distribution (1) follows a Chi-squared distribution, we can use the concentration of Chi-squared distributions to show that with probability of  $\geq 1 - \alpha$ ,  $\epsilon'$  is bounded within  $[L_{\alpha}, U_{\alpha}]$ , where  $L_{\alpha} = \sigma_{\min}^{2}(\chi_{n_{\min} - 1, \alpha / 2}^{2} / (n_{\min} - 1) - 1)$ ,  $U_{\alpha} = \sigma_{\max}^{2}(\chi_{n_{\min} - 1, 1 - \alpha / 2}^{2} / (n_{\min} - 1) - 1)$ . Here  $\chi_{n_{\min} - 1, \eta}^{2}$  denotes  $\eta^{\mathrm{th}}$ -quantile of the Chi-squared distribution with  $n_{\min} - 1$  degrees of freedom ( $\eta = \alpha / 2$  or  $1 - \alpha / 2$ ). By choosing  $\alpha = \delta / (4T\mathbb{B})$  ( $\delta$  is from Theorem 3.1), we can ensure that with probability of  $\geq 1 - \delta / 4$ ,  $\epsilon'$  is bounded within  $[L_{\alpha}, U_{\alpha}]$  for all  $\boldsymbol{x}_t^{[b]}$ . In other words, with probability of  $\geq 1 - \delta / 4$ , the noise  $\epsilon'$  in (1) is zero-mean and bounded within  $[L_{\alpha}, U_{\alpha}]$ , which indicates that  $\epsilon'$  is  $R'$ -sub-Gaussian with  $R' = (U_{\alpha} - L_{\alpha}) / 2$ . More details are given in Appendix E. Note that the value of  $R'$  derived here is expected to be overly pessimistic, so, we expect smaller values of  $R'$  to be applicable in practice.
Upper Bound Construction. With the assumptions of  $\| g\|_{\mathcal{H}_{k'}}\leq B'$  and  $\epsilon^\prime$  is  $R^{\prime}$ -sub-Gaussian, we can construct the upper bound  $U_{t}^{\sigma^{2}}(\cdot)$ . Denote by  $\Gamma_{\tau_{t - 1}}^{\prime}$  the maximum information gain about  $g$  from any  $\tau_{t - 1} = \sum_{t' = 1}^{t - 1}b_{t'}$  observations, define  $\beta_t^\prime \triangleq B^\prime +R^\prime \sqrt{2(\Gamma_{\tau_{t - 1}}^\prime + 1 + \log(4 / \delta))}$ , and represent the GP posterior mean and standard deviation for  $\mathcal{GP}'$  as  $\mu_{t - 1}^{\prime}(\cdot)$  and  $\sigma_{t - 1}^{\prime}(\cdot)$ . Then we have that
$$
\left| \mu_ {t - 1} ^ {\prime} (\boldsymbol {x}) - g (\boldsymbol {x}) \right| \leq \beta_ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x}), \quad \forall \boldsymbol {x} \in \mathcal {X}, t \in [ T ] \tag {2}
$$
with probability of  $\geq 1 - \delta /2$ . The error probabilities come from applying Theorem 2 of [8] ( $\delta /4$ ) and assuming that  $\epsilon^{\prime}$  is  $R^{\prime}$ -sub-Gaussian ( $\delta /4$ ). This implies that  $-\sigma^{2}(\pmb {x}) = g(\pmb {x})\geq \mu_{t - 1}^{\prime}(\pmb {x})-\beta_{t}^{\prime}\sigma_{t - 1}^{\prime}(\pmb {x})$ , and hence  $\sigma^2 (\pmb {x})\leq -\mu_{t - 1}^{\prime}(\pmb {x}) + \beta_{t}^{\prime}\sigma_{t - 1}^{\prime}(\pmb {x}),\forall \pmb {x}\in \mathcal{X},t\in [T]$ . Therefore, we can choose the upper bound on the noise variance (Sec. 3.2.1) as  $U_{t}^{\sigma^{2}}(\pmb {x}) = -\mu_{t - 1}^{\prime}(\pmb {x}) + \beta_{t}^{\prime}\sigma_{t - 1}^{\prime}(\pmb {x})$
BTS-RED-Unknown Algorithm. To summarize, we can obtain BTS-RED-Unknown (Algo. 3, Appendix F) by modifying the selection criterion of  $n_t$  (line 6 of Algo. 1) to be  $n_t^{(b)} = \lceil (-\mu_{t - 1}'(\pmb{x}_t^{(b)}) + \beta_t'\sigma_{t - 1}'(\pmb{x}_t^{(b)})) / R^2\rceil$ . As a result, BTS-RED-Unknown enjoys the same regret upper bound as Theorem 3.1 (after replacing  $\delta$  in Theorem 3.1 by  $\delta /2$ ). Intuitively, using an upper bound  $U_{t}^{\sigma^{2}}(\pmb{x}_{t}^{(b)})$  in the selection of  $n_t$  implies that if we are uncertain about the noise variance at some input location  $\pmb{x}_t^{(b)}$  (i.e., if  $\sigma_{t - 1}'(\pmb{x}_t^{(b)})$  is large), we choose to be conservative and use a large number of replications  $n_t$ .
4 Mean-Var-BTS-RED
We extend BTS-RED-Unknown (Sec. 3.2) to maximize the mean-variance objective function:  $h_{\omega}(\pmb{x}) = \omega f(\pmb{x}) - (1 - \omega)\sigma^{2}(\pmb{x})$ , to introduce Mean-Var-BTS-RED (Algo. 2). In contrast to BTS-RED-Unknown, Mean-Var-BTS-RED chooses every input query  $x_{t}^{(b)}$  by maximizing the weighted combination of two functions sampled from, respectively, the posteriors of  $\mathcal{GP}$  and  $\mathcal{GP}'$  (lines 5-6 of Algo. 2), while  $n_t$  is chosen (line 7 of Algo. 2) in the same way as BTS-RED-Unknown. This naturally induces a preference for inputs with both large values of  $f$  and small values of  $\sigma^2(\cdot)$ , and hence allows us to derive an upper bound on  $R_T^{\mathrm{MV}}$  (proof in Appendix G):
Theorem 4.1 (Mean-Var-BTS-RED). With probability of at least  $1 - \delta$
$$
R _ {T} ^ {M V} = \widetilde {\mathcal {O}} \Big (\frac {e ^ {C} \sqrt {T}}{\sqrt {\mathbb {B} / \lceil \frac {\sigma_ {\mathrm {m a x}} ^ {2}}{R ^ {2}} \rceil - 1}} \Big [ \omega R \sqrt {\Gamma_ {T \mathbb {B}}} (\sqrt {\Gamma_ {T \mathbb {B}}} + \sqrt {C}) + (1 - \omega) R ^ {\prime} \sqrt {\Gamma_ {T \mathbb {B}} ^ {\prime}} (\sqrt {\Gamma_ {T \mathbb {B}} ^ {\prime}} + \sqrt {C}) \Big ] \Big).
$$
$C$  is a constant s.t.  $\max_{A\subset \mathcal{X},|A|\leq \mathbb{B}}\mathbb{I}\left(f;\pmb {y}_A|\pmb{y}_{1:t - 1}\right)\leq C,\max_{A\subset \mathcal{X},|A|\leq \mathbb{B}}\mathbb{I}\left(g;\widetilde{\pmb{y}}_A|\widetilde{\pmb{y}}_{1:t - 1}\right)\leq C.$
Note that  $\Gamma_{T\mathbb{B}}$  and  $\Gamma_{T\mathbb{B}}^{\prime}$  may differ since the SE kernels  $k$  and  $k^{\prime}$ , which are used to model  $f$  and  $g$  respectively, may be different. Similar to Theorem 3.1, if we run uncertainty sampling for a finite number (independent of  $T$ ) of initial iterations using either  $k$  or  $k^{\prime}$  (depending on whose lengthscale is smaller), then  $C$  can be chosen to be a constant independent of  $\mathbb{B}$  and  $T$ . Refer to Lemma G.6 (Appendix G) for more details. As a result, the regret upper bound in Theorem 4.1 is also sub-linear since both  $k$  and  $k^{\prime}$  are SE kernels and hence  $\Gamma_{T\mathbb{B}} = \mathcal{O}(\log^{d + 1}(T))$  and  $\Gamma_{T\mathbb{B}}^{\prime} = \mathcal{O}(\log^{d + 1}(T))$ . The regret upper bound can be viewed as a weighted combination of the regrets associated with  $f$  and  $g$ . Intuitively, if  $\omega$  is larger (i.e., if we place more emphasis on maximizing  $f$  than  $g$ ), then a larger proportion of the regrets is incurred due to our attempt to maximize the function  $f$ .
Algorithm 2 Mean-Var-BTS-RED.
1: for  $t = 1,2,\ldots ,T$  do  
2:  $b = 0,n_{t}^{(0)} = 0$   
3: while  $\sum_{b^{\prime} = 0}^{b}n_{t}^{(b^{\prime})} < \mathbb{B}$  do  
4:  $b\gets b + 1$   
5: Sample  $f_{t}^{(b)}$  from  $\mathcal{GP}(\mu_{t - 1}(\cdot),\beta_t^2\sigma_{t - 1}^2 (\cdot ,\cdot))$  and  $g_{t}^{(b)}$  from  $\mathcal{GP}'(\mu_{t - 1}'(\cdot),\beta_t'^2\sigma_{t - 1}'^2 (\cdot ,\cdot))$   
6:  $\pmb{x}_{t}^{(b)} = \arg \max_{\pmb {x}\in \mathcal{X}}[\omega f_{t}^{(b)}(\pmb {x}) + (1 - \omega)g_{t}^{(b)}(\pmb {x})]$   
7:  $n_t^{(b)} = \lceil (-\mu_{t - 1}'(\pmb {x}_t^{(b)}) + \beta_t'\sigma_{t - 1}'(\pmb {x}_t^{(b)})) / R^2\rceil$   
8:  $b_{t} = b - 1$   
9: for  $b\in [b_t]$ , query  $\pmb{x}_{t}^{(b)}$  with  $n_t^{(b)}$  parallel processes  
10: for  $b\in [b_t]$ , observe  $\{y_{t,n}^{(b)}\}_{n\in [n_t^{(b)}]}$ . Calculate their mean  $y_{t}^{(b)}$  and (negated) variance  $\widetilde{y}_t^{(b)}$  (1)  
11: Use  $\{(x_{t}^{(b)},y_{t}^{(b)})\}_{b\in [b_t]}$  to update posterior  $\mathcal{GP}$ ,  $\{(x_{t}^{(b)},\widetilde{y}_{t}^{(b)})\}_{b\in [b_{t}]}$  to update posterior  $\mathcal{GP}'$
For BTS-RED-Known and BTS-RED-Unknown which only aim to maximize the objective function  $f$ , we set  $n_{\mathrm{max}} = \mathbb{B} / 2$  in the first  $T / 2$  iterations and  $n_{\mathrm{max}} = \mathbb{B}$  subsequently (see Sec. 3.1.1 for more details), and set  $n_{\mathrm{max}} = \mathbb{B}$  in all iterations for Mean-Var-BTS-RED. We set  $n_{\mathrm{min}} = 2$  unless specified otherwise, however, it is recommended to make  $n_{\mathrm{min}}$  larger in experiments where the overall noise variance is large (e.g., we let  $n_{\mathrm{min}} = 5$  in Sec. 5.2). We use random search to select the initial inputs instead of the uncertainty sampling initialization method indicated by our theoretical results (Sec. 3.1.2) because previous work [28] and our empirical results show that they lead to similar performances (Fig. 8 in App. H.1). We choose the effective noise variance  $R^2$  by following our theoretical guideline in Sec. 3.1.2, i.e.,  $R^2 = \sigma_{\mathrm{max}}^2 (\sqrt{\mathbb{B}} + 1) / (\mathbb{B} - 1)$  which minimizes the regret upper bound in Theorem 3.1. $^3$  However, in practice, this choice may not be optimal because we derived it by minimizing an upper bound which is potentially loose (e.g., we have ignored all log factors). So, we introduce a tunable parameter  $\kappa > 0$  and choose  $R^2$  as  $R^2 = \kappa \sigma_{\mathrm{max}}^2 (\sqrt{\mathbb{B}} + 1) / (\mathbb{B} - 1)$ . As a result, we both enjoy the flexibility of tuning our preference for the overall number of replications (i.e., a smaller  $\kappa$  leads to larger  $n_t$ 's in general) and preserve the ability to automatically adapt to the total budget (via  $\mathbb{B}$ ) and the overall noise level (via  $\sigma_{\mathrm{max}}^2$ ). When the noise variance is unknown (i.e.,  $\sigma_{\mathrm{max}}^2$  is unknown), we approximate  $\sigma_{\mathrm{max}}^2$  by the maximum observed empirical noise variance and update our approximation after every iteration. To demonstrate the robustness of our methods, we only use two values of  $\kappa = 0.2$  and  $\kappa = 0.3$  in all experiments. Of note, our methods with  $\kappa = 0.3$  perform the best in almost all experiments (i.e., green curves in all figures), and  $\kappa = 0.2$  also consistently performs well.
Following the common practice of BO [11, 20, 28, 32, 35], we plot the (batch) simple regret or the best observed function value up to an iteration. In all experiments, we compare with the most natural baseline of batch TS with a fixed number of replications. For mean optimization problems (i.e., maximize  $f$ ), we also compare with standard sequential BO algorithms such as GP-UCB and GP-TS, but they are significantly outperformed by both our algorithms and batch TS which are able to exploit batch evaluations (Secs. 5.1 and 5.2). Therefore, we do not expect existing sequential algorithms to achieve comparable performances to our algorithms due to their inability to exploit batch evaluations. For mean-variance optimization, we additionally compare with the recently introduced Risk-Averse Heteroscedastic BO (RAHBO) [33] (Sec. 6), which is the state-of-the-art method for risk-averse BO with replications. Some experimental details are postponed to Appendix H.
5.1 Synthetic Experiments
We sample two functions from two different GPs with the SE kernel (defined on a discrete 1-D domain within  $[0,1]$ ) and use them as  $f(\cdot)$  and  $\sigma^2 (\cdot)$ , respectively. We use  $\mathbb{B} = 50$ .
Mean Optimization. The mean and noise variance functions used here are visualized in Fig. 1a. This synthetic experiment is used to simulate real-world scenarios where practitioners are risk-neutral and hence only aim to select an input with a large mean function value. After every iteration (batch)  $t$ , an
For simplicity, we also follow this guideline from Sec. 3.1.2 to choose  $R^2$  for Mean-Var-BTS-RED.
(b)
Figure 1: (a) Synthetic function for mean optimization (Sec. 5.1). (b) Average number of replications  $n_t$  for BTS-RED-Unknown. Results for (c) mean and (d) mean-variance optimization.
algorithm reports the selected input with the largest empirical mean from its observation history, and we evaluate the simple regret at iteration  $t$  as the difference between the objective function values at the global maximum  $x^{*}$  and at the reported input. To demonstrate the consistency of our performance, we also tested an alternative reporting criteria which reports the input with the larger LCB value in every iteration, and the results (Fig. 7b in Appendix H.1) are consistent with our main results (Fig. 1c). Fig. 1b plots the average  $n_t$  (vertical axis) chosen by BTS-RED-Unknown for every queried input (horizontal axis), which shows that larger  $n_t$ 's are selected for inputs with larger noise variances in general and that a smaller  $\kappa = 0.2$  indeed increases our preference for larger  $n_t$ 's.
The results (simple regrets) are shown in Fig. 1c. As can be seen from the figure, for Batch TS with a fixed  $n_t$ , smaller values of  $n_t$  such as  $n_t = 5$  usually lead to faster convergence initially due to the ability to quickly explore more unique inputs, however, their performances deteriorate significantly in the long run due to inaccurate estimations; in contrast, larger  $n_t$ 's such as  $n_t = 20$  result in slower convergence initially yet lead to better performances (than small fixed  $n_t$ 's) in later stages. Of note, Batch TS with  $n_t = 1$  (gray curve) represents standard batch TS ( $\mathbb{B} = 50$ ) without replications [28], which underperforms significantly and hence highlights the importance of replications in experiments with large noise variance. Moreover, our BTS-RED-Known and BTS-RED-Unknown (especially with  $\kappa = 0.3$ ) consistently outperform Batch TS with fixed  $n_t$ . We also demonstrate our robustness against  $\kappa$  in this experiment by showing that our performances are consistent for a wide range of  $\kappa$ 's (Fig. 7a in App. H.1). In addition, we show that sequential BO algorithms (i.e., GP-TS, GP-UCB, and GP-UCB with heteroscedastic GP) which cannot exploit batch evaluations fail to achieve comparable performances to batch TS, BTS-RED and BTS-RED-Unknown (Fig. 5 in App. H.1).
Mean-variance Optimization. Here we evaluate our Mean-Var-BTS-RED. We simulate this scenario with the synthetic function in Fig. 6a (App. H.1), for which the global maximums of the mean and mean-variance  $(\omega = 0.3)$  objective functions are different (Fig. 6b). After every iteration (batch)  $t$ , we report the selected input with the largest empirical mean-variance value (i.e., weighted combination of the empirical mean and variance), and evaluate the mean-variance simple regret at iteration  $t$  as the difference between the values of the mean-variance objective function  $h_{\omega}$  at the global maximum  $x_{\omega}^{*}$  and at the reported input. The results (Fig. 1d) show that our Mean-Var-BTS-RED (again especially with  $\kappa = 0.3$ ) outperforms other baselines. Since RAHBO is sequential and uses a fixed number of replications, we use  $\mathbb{B} = 50$  replications for every query for a fair comparison. RAHBO underperforms here which is likely due to its inability to leverage batch evaluations.
5.2 Real-world Experiments on Precision Agriculture
Plant biologists often need to optimize the growing conditions of plants (e.g., the amount of different nutrients) to increase their yield. The common practice of manually tuning one nutrient at a time is considerably inefficient and hence calls for the use of the sample-efficient method of BO. Unfortunately, plant growths are usually (a) time-consuming and (b) associated with large and heteroscedastic noise. So, according to plant biologists, in real lab experiments, (a) multiple growing conditions are usually tested in parallel and (b) every condition is replicated multiple times to get a reliable outcome [31]. This naturally induces a trade-off between evaluating more unique growing conditions vs. replicating every condition more times, and is hence an ideal application for our algorithms. We tune the pH value (in [2.5, 6.5]) and ammonium concentration (denoted as NH3, in [0, 30000] uM), in order to maximize the leaf area and minimize the tipburn area after harvest. We perform real lab experiments using the input conditions from a regular grid within the 2-D domain, and then use the collected data to learn two separate heteroscedastic GPs for, respectively, leaf area and tipburn area. Each learned GP can output the predicted mean and variance (for leaf area or tipburn area) at every input in the 2-D domain, and can hence be used as the groundtruth mean  $f(\cdot)$  and noise variance  $\sigma^2 (\cdot)$  functions. We perform two sets of experiments, with the goal of maximizing (a) the
Figure 2: (a) Mean and (b) mean-variance optimization for the leaf area. (c) Mean and (d) mean-variance optimization for the weighted combination of leaf area and negative tipburn area.
(a)
(b)
Figure 3: (a) Mean and (b) noise variance functions for leaf area, with some selected queries (stars) and their  $n_t$ 's. (c) Mean and (d) mean-variance optimization for hyper. tuning of SVM (Sec. 5.3).
leaf area and (b) a weighted combination of the leaf area  $(\times 0.8)$  and negative tipburn area  $(\times 0.2)$ . For both experiments, we run BTS-RED-Unknown and Mean-Var-BTS-RED to maximize the mean and mean-variance objectives  $(\omega = 0.975)$ , respectively. We set  $\mathbb{B} = 50$ ,  $n_{\mathrm{min}} = 5$  and  $n_{\mathrm{max}} = 50$ .
Fig. 2 shows the results for maximizing the leaf area (a,b) and weighted combination of leaf area and negative tipburn area (c,d). Our BTS-RED-Unknown and Mean-Var-BTS-RED with  $\kappa = 0.2$  and  $\kappa = 0.3$  consistently outperform Batch TS, as well as RAHBO in Figs. 2b and d. For mean optimization, we also compare with sequential BO methods (Fig. 10 in Appendix H.2), which again are unable to perform comparably with other algorithms that exploit batch evaluations. Figs. 3a and b visualize the groundtruth mean and noise variance functions for the leaf area, including the locations of some queried inputs (the selected inputs after every 4 iterations) and their corresponding  $n_t$ 's. Similarly, Figs. 9a and b (Appendix H.2) show the queried inputs and the  $n_t$ 's of Mean-Var-BTS-RED  $(\omega = 0.975)$ , illustrated on heat maps of the mean-variance objective (a) and noise variance functions (b). These figures demonstrate that most of our input queries fall into regions with large (either mean or mean-variance) objective function values (Figs. 3a and 9a) and that  $n_t$  is in general larger at those input locations with larger noise variance (Figs. 3b and 9b). We have included GIF animations for Figs. 3 and 9 in the supplementary material. Our results here showcase the capability of our algorithms to improve the efficiency of real-world experimental design problems.
5.3 Real-World Experiments on AutoML
Reproducibility is an important desiderata in AutoML problems such as hyperparameter tuning [25], because the performance of a hyperparameter configuration may vary due to a number of factors such as different datasets, parameter initializations, etc. For example, some practitioners may prefer hyperparameter configurations that consistently produce well-performing ML models for different datasets. We adopt the EMNIST dataset which is widely used in multi-task learning [10, 15]. EMNIST consists of images of hand-written characters from different individuals, and each individual corresponds to a separate image classification task. Here we tune two SVM hyperparameters: the penalty and RBF kernel parameters, both within [0.0001, 2]. We firstly construct a uniform 2-D grid of the two hyperparameters and then evaluate every input on the grid using 100 tasks (i.e., image classification for 100 different individuals) to record the observed mean and variance as the groundtruth mean and variance. Refer to Figs. 11a, b and c (Appendix H.3) for the constructed mean, variance and mean-variance  $(\omega = 0.2)$  functions. Fig. 3c and d plot the results  $(\mathbb{B} = 50)$  for mean (c) and mean-variance (d) optimization. Our BTS-RED-Unknown and Mean-Var-BTS-RED with both  $\kappa = 0.2$  and 0.3 perform competitively (again especially  $\kappa = 0.3$ ), which shows their potential to improve the efficiency and reproducibility of AutoML. RAHBO underperforms significantly (hence omitted from Fig. 3d), which is likely due to the small noise variance (Fig. 11b) which favors methods with small  $n_t$ 's. Specifically, methods with small  $n_t$ 's can obtain reliable estimations (due to small
noise variance) while enjoying the advantage of evaluating a large number  $b_{t}$  of unique inputs in every iteration. This makes RAHBO unfavorable since it is a sequential algorithm with  $b_{t} = 1$ .
Experiments Using Different Budgets B. Here we test the performances of our algorithms with different budgets (i.e., different from the  $\mathbb{B} = 50$  used in the main experiments above) using the AutoML experiment. The results (Fig. 12 in App. H.3) show that the performance advantages of our algorithms (again especially with  $\kappa = 0.3$ ) are still consistent with a larger or smaller budget.
Additional Experiments with Higher-Dimensional Inputs. To further verify the practicality of our proposed algorithms, here we adopt two additional experiments with higher-dimensional continuous input domains. Specifically, we tune  $d = 12$  and  $d = 14$  parameters of a controller for a Lunar-Lander task and a robot pushing task, respectively, and both experiments have widely used by previous works on high-dimensional BO [16, 20] (more details in App. H.4). In both experiments, the heteroscedastic noises arise from random environmental factors. The results (Fig. 13 in App. H.4) show that our algorithms, again especially with  $\kappa = 0.3$ , still consistently achieve compelling performances.