SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongxiang Dai<sup>1</sup>, Quoc Phong Nguyen<sup>2</sup>, Sebastian Shenghong Tay<sup>1,4</sup>, Daisuke Urano<sup>5</sup>, Richalynn Leong<sup>5</sup>, Bryan Kian Hsiang Low<sup>1</sup>, Patrick Jaillet<sup>2,3</sup>
$^{1}$ Department of Computer Science, National University of Singapore
$^{2}$ LIDS and  $^{3}$ EECS, Massachusetts Institute of Technology
$^{4}$ Institute for Infocomm Research (I2R), A*STAR, Singapore
5Temasek Life Sciences Laboratory, Singapore
dzx@nus.edu.sg, qphongmp@gmail.com, sebastian.tay@u.nus.edu,
{daisuke, richalynn}@tll.org.sg, lowkh@comp.nus.edu.sg, jaillet@mit.edu
Many real-world experimental design problems  $(a)$  evaluate multiple experimental conditions in parallel and  $(b)$  replicate each condition multiple times due to large and heteroscedastic observation noise. Given a fixed total budget, this naturally induces a trade-off between evaluating more unique conditions while replicating each of them fewer times vs. evaluating fewer unique conditions and replicating each more times. Moreover, in these problems, practitioners may be risk-averse and hence prefer an input with both good average performance and small variability. To tackle both challenges, we propose the Batch Thompson Sampling for Replicable Experimental Design (BTS-RED) framework, which encompasses three algorithms. Our BTS-RED-Known and BTS-RED-Unknown algorithms, for, respectively, known and unknown noise variance, choose the number of replications adaptively rather than deterministically such that an input with a larger noise variance is replicated more times. As a result, despite the noise heteroscedasticity, both algorithms enjoy a theoretical guarantee and are asymptotically no-regret. Our Mean-Var-BTS-RED algorithm aims at risk-averse optimization and is also asymptotically no-regret. We also show the effectiveness of our algorithms in two practical real-world applications: precision agriculture and AutoML.
Bayesian optimization (BO), which is a sequential algorithm for optimizing black-box and expensive-to-evaluate functions [14, 15], has found application in a wide range of experimental design problems [21]. Many such applications which use BO to accelerate the scientific discovery process [27] fall under the umbrella of AI for science (AI4Science). Many real-world experimental design problems, such as precision agriculture, share two inherent characteristics: (a) multiple experimental conditions are usually evaluated in parallel to take full advantage of the available experimental budget; (b) the evaluation of every experimental condition is usually replicated multiple times [31] because every experiment may be associated with a large and heteroscedastic (i.e., input-dependent) observation noise, in which case replication usually leads to better performances [2, 33, 43]. Replicating each evaluated experimental condition is also a natural choice in experimental design problems in which it incurs considerable setup costs to test every new experimental condition. This naturally induces an interesting challenge regarding the trade-off between input selection and replication: in every iteration of BO where we are given a fixed total experimental budget, should we evaluate more unique experimental conditions and replicate each of them fewer times or evaluate fewer unique conditions and replicate each more times? Interestingly, this trade-off is also commonly found in
other applications such as automated machine learning (AutoML), in which parallel evaluations are often adopted to exploit all available resources [28] and heteroscedasticity is a prevalent issue [12]. Furthermore, these experimental design problems with large and heteroscedastic noise are often faced with another recurring challenge: instead of an input experimental condition (e.g., a hyperparameter configuration for an ML model) that produces a good performance (e.g., large validation accuracy) on average, some practitioners may be risk-averse and instead prefer an input that both yields a good average performance and has small variability. As a result, instead of only maximizing the mean of the black-box function, these risk-averse practitioners may instead look for inputs with both a large mean function value and a small noise variance [26, 33].
In this work, we provide solutions to both challenges in a principled way by proposing the framework of Batch Thompson Sampling for Replicable Experimental Design (BTS-RED). The first challenge regarding the trade-off between input selection and replication is tackled by the first two incarnations of our framework: the BTS-RED-Known (Sec. 3.1) and BTS-RED-Unknown (Sec. 3.2) algorithms, which are applicable to scenarios where the noise variance function is known or unknown, respectively. For batch selection, we adopt the Thompson sampling (TS) strategy because its inherent randomness makes it particularly simple to select a batch of inputs [28]. Moreover, previous works on BO have shown that the use of TS both allows for the derivation of theoretical guarantees [16, 28] and leads to strong empirical performances [16, 20]. For replication selection, instead of the common practice of replicating every queried input a fixed number of times, we choose the number of replications adaptively depending on the observation noise. Specifically, in every iteration, both algorithms repeat the following two steps until the total budget is exhausted:  $(a)$  choose an input query following the TS strategy, and then  $(b)$  adaptively choose the number of replications for the selected input such that an input with a larger noise variance is replicated more times. Of note, in spite of the noise heteroscedasticity, our principled approach to choosing the number of replications ensures that the effective noise variance  $R^2$  of every queried input is the same (Sec. 3.1). This allows us to derive an upper bound on their cumulative regret and show that they are asymptotically no-regret. Our theoretical guarantee formalizes the impact of the properties of the experiments, i.e., our regret upper bound becomes better if the total budget is increased or if the overall noise level is reduced. Importantly, our theoretical result provides a guideline on the choice of the effective noise variance parameter  $R^2$ , which is achieved by minimizing the regret upper bound and allows  $R^2$  to automatically adapt to the budgets and noise levels of different experiments (Sec. 3.1.2).
To handle the second challenge of risk-averse optimization, we propose the third variant of our BTS-RED framework named Mean-Var-BTS-RED (Sec. 4), which is a natural extension of BTS-RED-Unknown. Mean-Var-BTS-RED aims to maximize the mean-variance objective function, which is a weighted combination of the mean objective function and negative noise variance function (Sec. 2). We prove an upper bound on the mean-variance cumulative regret of Mean-Var-BTS-RED (Sec. 4) and show that it is also asymptotically no-regret.
In addition to our theoretical contributions, we also demonstrate the practical efficacy of our algorithms in two real-world problems (Sec. 5). Firstly, in real-world precision agriculture experiments, plant biologists usually  $(a)$  evaluate multiple growing conditions in parallel, and  $(b)$  replicate each condition multiple times to get a reliable outcome [31]. Moreover, plant biologists often prefer more replicable conditions, i.e., inputs with small noise variances. This is hence an ideal application for our algorithms. So, we conduct an experiment using real-world data on plant growths, to show the effectiveness of our algorithms in precision agriculture (Sec. 5.2). Next, we also apply our algorithms to AutoML to find hyperparameter configurations with competitive and reproducible results across different AutoML tasks (Sec. 5.3). The efficacy of our algorithms demonstrates their capability to improve the reproducibility of AutoML tasks which is an important issue in AutoML [25].
We denote by  $f: \mathcal{X} \to \mathbb{R}$  the objective function we wish to maximize, and by  $\sigma^2: \mathcal{X} \to \mathbb{R}^+$  the input-dependent noise variance function. We denote the minimum and maximum noise variance as  $\sigma_{\min}^2$  and  $\sigma_{\max}^2$ . For simplicity, we assume that the domain  $\mathcal{X}$  is finite, since extension to compact domains can be easily achieved via suitable discretizations [5]. After querying an input  $x \in \mathcal{X}$ , we observe a noisy output  $y = f(x) + \epsilon$  where  $\epsilon \sim \mathcal{N}(0, \sigma^2(x))$ . In every iteration  $t$ , we select a batch of  $b_t \geq 1$  inputs  $\{\pmb{x}_t^{(b)}\}_{b=1,\dots,b_t}$ , and query every  $\pmb{x}_t^{(b)}$  with  $n_t^{(b)} \geq 1$  parallel processes. We denote the total budget as  $\mathbb{B}$  such that  $\sum_{b=1}^{b_t} n_t^{(b)} \leq \mathbb{B}, \forall t \geq 1$ . We model the function  $f$
using a Gaussian process (GP) [40]:  $\mathcal{GP}(\mu(\cdot), k(\cdot, \cdot))$ , where  $\mu(\cdot)$  is a mean function which we assume w.l.o.g.  $\mu(\boldsymbol{x}) = 0$  and  $k(\cdot, \cdot)$  is a kernel function for which we focus on the commonly used squared exponential (SE) kernel. In iteration  $t$ , we use the observation history in the first  $t-1$  iterations (batches) to calculate the GP posterior  $\mathcal{GP}(\mu_{t-1}(\cdot), \sigma_{t-1}^2(\cdot, \cdot))$ , in which  $\mu_{t-1}(\cdot)$  and  $\sigma_{t-1}^2(\cdot, \cdot)$  represent the GP posterior mean and covariance functions (details in Appendix A). For BTS-RED-Unknown and Mean-Var-BTS-RED (i.e., when  $\sigma^2(\cdot)$  is unknown), we use another GP, denoted as  $\mathcal{GP}'$ , to model  $-\sigma^2(\cdot)$  (Sec. 3.2), and denote its posterior as  $\mathcal{GP}'(\mu_{t-1}'(\cdot), \sigma_{t-1}'^2(\cdot, \cdot))$ .
In our theoretical analysis of BTS-RED-Known and BTS-RED-Unknown where we aim to maximize  $f$ , we follow previous works on batch BO [11, 18, 35] and derive an upper bound on the batch cumulative regret  $R_{T} = \sum_{t=1}^{T} \min_{b \in [b_{t}]} [f(\boldsymbol{x}^{*}) - f(\boldsymbol{x}_{t}^{(b)})]$ , in which  $\boldsymbol{x}^{*} \in \arg \max_{\boldsymbol{x} \in \mathcal{X}} f(\boldsymbol{x})$  and we have used  $[b_{t}]$  to denote  $\{1, \ldots, b_{t}\}$ . We show (Sec. 3) that both BTS-RED-Known and BTS-RED-Unknown enjoy a sub-linear upper bound on  $R_{T}$ , which suggests that as  $T$  increases, a global optimum  $\boldsymbol{x}^{*}$  is guaranteed to be queried since the batch simple regret  $S_{T} = \min_{t \in [T]} \min_{b \in [b_{t}]} [f(\boldsymbol{x}^{*}) - f(\boldsymbol{x}_{t}^{(b)})] \leq R_{T} / T$  goes to 0 asymptotically. We analyze the batch cumulative regret because it allows us to show the benefit of batch evaluations, and our analysis can also be modified to give an upper on the sequential cumulative regret of  $R_{T}' = \sum_{t=1}^{T} \sum_{b=1}^{b_{t}} [f(\boldsymbol{x}^{*}) - f(\boldsymbol{x}_{t}^{(b)})]$  (Appendix D). Our Mean-Var-BTS-RED aims to maximize the mean-variance objective function  $h_{\omega}(\boldsymbol{x}) = \omega f(\boldsymbol{x}) - (1 - \omega) \sigma^{2}(\boldsymbol{x}) = \omega f(\boldsymbol{x}) + (1 - \omega) g(\boldsymbol{x})$ , in which we have defined  $g(\boldsymbol{x}) \triangleq -\sigma^{2}(\boldsymbol{x}), \forall \boldsymbol{x} \in \mathcal{X}$ . The user-specified weight parameter  $\omega \in [0,1]$  reflects our relative preference for larger mean function values or smaller noise variances. Define the mean-variance batch cumulative regret as  $R_{T}^{\mathrm{MV}} = \sum_{t=1}^{T} \min_{b \in [b_{t}]} [h_{\omega}(\boldsymbol{x}_{\omega}^{*}) - h_{\omega}(\boldsymbol{x}_{t}^{(b)})]$  where  $\boldsymbol{x}_{\omega}^{*} \in \arg \max_{\boldsymbol{x} \in \mathcal{X}} h_{\omega}(\boldsymbol{x})$ . We also prove a sub-linear upper bound on  $R_{T}^{\mathrm{MV}}$  for Mean-Var-BTS-RED (Sec. 4).
3 BTS-RED-Known and BTS-RED-Unknown
Here, we firstly introduce BTS-RED-Known and its theoretical guarantees (Sec. 3.1), and then discuss how it can be extended to derive BTS-RED-Unknown (Sec. 3.2).
3.1 BTS-RED with Known Noise Variance Function
3.1.1 BTS-RED-Known
Algorithm 1 BTS-RED-Known.
1: for  $t = 1,2,\ldots ,T$  do  
2:  $b = 0,n_{t}^{(0)} = 0$   
3: while  $\sum_{b^{\prime} = 0}^{b}n_{t}^{(b^{\prime})} < \mathbb{B}$  do  
4:  $b\gets b + 1$   
5: Sample a function  $f_{t}^{(b)}$  from the GP posterior of  $\mathcal{GP}(\mu_{t - 1}(\cdot),\beta_t^2\sigma_{t - 1}^2 (\cdot ,\cdot))$  (Sec. 2)  
6: Choose  $\pmb{x}_{t}^{(b)} = \arg \max_{\pmb {x}\in \mathcal{X}}f_{t}^{(b)}(\pmb {x})$  and  $n_t^{(b)} = [\sigma^2 (\pmb {x}_t^{(b)}) / R^2 ]$   
7:  $b_{t} = b - 1$   
8: for  $b\in [b_t]$ , query  $\pmb{x}_{t}^{(b)}$  with  $n_t^{(b)}$  parallel processes  
9: for  $b\in [b_t]$ , observe  $\{y_{t,n}^{(b)}\}_{n\in [n_t^{(b)}]}$ . Calculate their empirical mean  $y_{t}^{(b)} = (1 / n_{t}^{(b)})\sum_{n = 1}^{n_{t}^{(b)}}y_{t,n}^{(b)}$   
10: Use  $\{(x_{t}^{(b)},y_{t}^{(b)})\}_{b\in [b_{t}]}$  to update posterior of  $\mathcal{GP}$
BTS-RED-Known (Algo. 1) assumes that  $\sigma^2 (\cdot)$  is known. In every iteration  $t$ , to sequentially select every  $\boldsymbol{x}_t^{(b)}$  and its corresponding  $n_t^{(b)}$ , we repeat the following process until the total number of replications has consumed the total budget  $\mathbb{B}$  (i.e., until  $\sum_{b' = 1}^{b}n_{t}^{(b')}\geq \mathbb{B}$ , line 3 of Algo. 1):
- line 5: sample a function  $f_{t}^{(b)}$  from  $\mathcal{GP}(\mu_{t - 1}(\cdot),\beta_t^2\sigma_{t - 1}^2 (\cdot ,\cdot))$  ( $\beta_{t}$  will be defined in Theorem 3.1);
- line 6: choose  $\boldsymbol{x}_t^{(b)} = \arg \max_{\boldsymbol{x} \in \mathcal{X}} f_t^{(b)}(\boldsymbol{x})$  by maximizing the sampled function  $f_t^{(b)}$ , and choose  $n_t^{(b)} = \lceil \sigma^2 (\boldsymbol{x}_t^{(b)}) / R^2 \rceil$ , where  $\lceil \cdot \rceil$  is the ceiling operator and  $R^2$  is the effective noise variance.
After the entire batch of  $b_{t}$  inputs have been selected, every  $x_{t}^{(b)}$  is queried with  $n_{t}^{(b)}$  parallel processes (line 8), and the empirical mean  $y_{t}^{(b)}$  of these  $n_{t}^{(b)}$  observations is calculated (line 9).
Finally,  $\{(\pmb{x}_t^{(b)},y_t^{(b)})\}_{b\in [b_t]}$  are used to update the posterior of  $\mathcal{GP}$  (line 10). Of note, since the observation noise is assumed to be Gaussian-distributed with a variance of  $\sigma^2 (\pmb{x}_t^{(b)})$  (Sec. 2), after querying  $\pmb{x}_t^{(b)}$  independently for  $n_t^{(b)}$  times, the empirical mean  $y_{t}^{(b)}$  follows a Gaussian distribution with noise variance  $\sigma^2 (\pmb{x}_t^{(b)}) / n_t^{(b)}$ . Next, since we select  $n_t^{(b)}$  by  $n_t^{(b)} = \lceil \sigma^2 (\pmb{x}_t^{(b)}) / R^2\rceil$  (line 6),  $\sigma^2 (\pmb{x}_t^{(b)}) / n_t^{(b)}$  is guaranteed to be upper-bounded by  $R^2$ . In other words, every observed empirical mean  $y_{t}^{(b)}$  follows a Gaussian distribution with a noise variance that is upper-bounded by the effective noise variance  $R^2$ . This is crucial for our theoretical analysis since it ensures that the effective noise variance is  $R$ -sub-Gaussian and thus preserves the validity of the GP-based confidence bound [8].
In practice, since our BTS-RED-Known algorithm only aims to maximize the objective function  $f$  (i.e., we are not concerned about learning the noise variance function), some replications may be wasted on undesirable input queries (i.e., those with small values of  $f(\pmb{x})$ ) especially in the initial stage when our algorithm favours exploration. To take this into account, we adopt a simple heuristic: we impose a maximum number of replications denoted as  $n_{\mathrm{max}}$ , and set  $n_{\mathrm{max}} = \mathbb{B} / 2$  in the first  $T / 2$  iterations and  $n_{\mathrm{max}} = \mathbb{B}$  afterwards. This corresponds to favouring exploration of more inputs (each with less replications) initially and preferring exploitation in later stages. This technique is also used for BTS-RED-Unknown (Sec. 3.2) yet not adopted for Mean-Var-BTS-RED (Sec. 4) since in mean-variance optimization, we also aim to learn (and minimize) the noise variance function.
Due to our stopping criterion for batch selection (line 3), in practice, some budgets may be unused in an iteration. E.g., when  $\mathbb{B} = 50$ , if  $\sum_{b' = 1}^{b - 1}n_t^{(b')} = 43$  after the first  $b - 1$  selected queries and the newly selected  $n_t$  for the  $b^{\mathrm{th}}$  query  $\pmb{x}_t^{(b)}$  is  $n_t^{(b)} = 12$ , then the termination criterion is met (i.e.,  $\sum_{b' = 1}^{b}n_{t}^{(b')} \geq \mathbb{B}$ ) and only  $43/50$  of the budgets are used. So, we adopt a simple technique: in the example above, we firstly evaluate the last selected  $\pmb{x}_t^{(b)}$  for 7 times, and in the next iteration  $t + 1$ , we start by completing the unfinished evaluation of  $\pmb{x}_t^{(b)}$  by allocating  $12 - 7 = 5$  replications to  $\pmb{x}_t^{(b)}$ . Next, we run iteration  $t + 1$  with the remaining budget, i.e., we let  $\mathbb{B} = 50 - 5 = 45$  in iteration  $t + 1$ .
3.1.2 Theoretical Analysis of BTS-RED-Known
Following the common practice in BO [8], we assume  $f$  lies in a reproducing kernel Hilbert space (RKHS) induced by an SE kernel  $k$ :  $\| f \|_{\mathcal{H}_k} \leq B$  for some  $B > 0$  where  $\| \cdot \|_{\mathcal{H}_k}$  denotes the RKHS norm. Theorem 3.1 below gives a regret upper bound of BTS-RED-Known (proof in Appendix B).
Theorem 3.1 (BTS-RED-Known). Choose  $\delta \in (0,1)$ . Define  $\tau_{t - 1} \triangleq \sum_{t' = 1}^{t - 1} b_{t'}$ , and define  $\beta_t \triangleq B + R \sqrt{2(\Gamma_{\tau_{t - 1}} + 1 + \log(2 / \delta))}$  where  $\Gamma_{\tau_{t - 1}}$  denotes the maximum information gain about  $f$  from any  $\tau_{t - 1}$  observations. With probability of at least  $1 - \delta$  ( $\widetilde{\mathcal{O}}$  ignores all log factors),
$$
R _ {T} = \widetilde {\mathcal {O}} \Big (e ^ {C} \sqrt {R ^ {2} / \big (\mathbb {B} / \lceil \frac {\sigma_ {\max} ^ {2}}{R ^ {2}} \rceil - 1 \big)} \sqrt {T \Gamma_ {T \mathbb {B}}} (\sqrt {C} + \sqrt {\Gamma_ {T \mathbb {B}}}) \Big).
$$
$C$  is a constant s.t.  $\max_{A\subset \mathcal{X},|A|\leq \mathbb{B}}\mathbb{I}\left(f;\pmb {y}_A|\pmb{y}_{1:t - 1}\right)\leq C,\forall t\geq 1.$ $\mathbb{I}\left(f;\pmb {y}_A|\pmb {y}_{1:t - 1}\right)$  is the information gain from observations  $\pmb{y}_A$  at inputs  $A$  given observations  $\pmb{y}_{1:t - 1}$  in the first  $t - 1$  iterations.
It has been shown by [19] that by running uncertainty sampling (i.e., choosing the initial inputs by sequentially maximizing the GP posterior variance) as the initialization phase for a finite number (independent of  $T$ ) of iterations,  $C$  can be chosen to be a constant independent of  $\mathbb{B}$  and  $T$ . As a result, the regret upper bound from Theorem 3.1 can be simplified into  $R_{T} = \tilde{\mathcal{O}}\big(\sqrt{R^{2} / (\mathbb{B} / \lceil\frac{\sigma_{\max}^{2}}{R^{2}}\rceil - 1)}\sqrt{T\Gamma_{T\mathbb{B}}}(1 + \sqrt{\Gamma_{T\mathbb{B}}})\big)$ . Therefore, for the SE kernel for which  $\Gamma_{T\mathbb{B}} = \mathcal{O}(\log^{d + 1}(T\mathbb{B}))$ , our regret upper bound is sub-linear, which indicates that our BTS-RED-Known is asymptotically no-regret. Moreover, the benefit of a larger total budget  $\mathbb{B}$  is also reflected from our regret upper bound since it depends on the total budget  $\mathbb{B}$  via  $\tilde{\mathcal{O}}((\log^{d + 1}(T\mathbb{B}) + \log^{(d + 1) / 2}(T\mathbb{B})) / \sqrt{\mathbb{B}})$ , which is decreasing as the total budget  $\mathbb{B}$  increases. In addition, the regret upper bound is decreased if  $\sigma_{\max}^2$  becomes smaller, which implies that the performance of our algorithm is improved if the overall noise level is reduced. Therefore, Theorem 3.1 formalizes the impacts of the experimental properties (i.e., the total budget and the overall noise level) on the performance of BTS-RED-Known.
Homoscedastic Noise. In the special case of homoscedastic noise, i.e.,  $\sigma^2 (\pmb {x}) = \sigma_{\mathrm{const}}^2,\forall \pmb {x}\in \mathcal{X}$  then  $n_t = \lceil \sigma_{\mathrm{const}}^2 /R^2\rceil \triangleq n_{\mathrm{const}}$  and  $b_{t} = \lfloor \mathbb{B} / n_{\mathrm{const}}\rfloor \triangleq b_{0},\forall t\in [T]$ . That is, our algorithm reduces
to standard (synchronous) batch TS proposed in [28] where the batch size is  $b_{0}$  and every query is replicated  $n_{\mathrm{const}}$  times. In this case, the regret upper bound becomes:  $R_{T} = \widetilde{\mathcal{O}}\left(R\frac{1}{\sqrt{b_{0}}}\sqrt{T\Gamma_{T}b_{0}}\left(1 + \sqrt{\Gamma_{T}b_{0}}\right)\right)$  (Appendix B.1).
Theoretical Guideline on the Choice of  $R^2$ . Theorem 3.1 also provides an interesting insight on the choice of the effective noise variance  $R^2$ . In particular, our regret upper bound depends on  $R^2$  through the term  $\sqrt{R^2 / [\mathbb{B} / (\sigma_{\max}^2 / R^2 + 1) - 1]}$ . By taking the derivative of this term w.r.t.  $R^2$ , we have shown (Appendix C) that the value of  $R^2$  that minimizes this term is obtained at  $R^2 = \sigma_{\max}^2(\sqrt{\mathbb{B}} + 1) / (\mathbb{B} - 1)$ . In other words,  $R^2$  should be chosen as a fraction of  $\sigma_{\max}^2$  (assuming  $\mathbb{B} > 4$  s.t.  $(\sqrt{\mathbb{B}} + 1) / (\mathbb{B} - 1) < 1$ ). In this case, increasing the total budget  $\mathbb{B}$  naturally encourages more replications. Specifically, increasing  $\mathbb{B}$  reduces  $(\sqrt{\mathbb{B}} + 1) / (\mathbb{B} - 1)$  and hence decreases the value of  $R^2$ , which consequently encourages the use of larger  $n_t$ 's (line 6 of Algo. 1) and allows every selected input to be replicated more times. For example, when the total budget is  $\mathbb{B} = 16$ ,  $R^2$  should be chosen as  $R^2 = \sigma_{\max}^2 / 3$ ; when  $\mathbb{B} = 100$ , then we have  $R^2 = \sigma_{\max}^2 / 9$ . We will follow this theory-inspired choice of  $R^2$  in our experiments in Sec. 5 (with slight modifications).
Improvement over Uniform Sample Allocation. For the naive baseline of uniform sample allocation (i.e., replicating every input a fixed number  $n_0 \leq \mathbb{B}$  of times), the resulting effective observation noise would be  $(\sigma_{\mathrm{max}} / \sqrt{n_0})$ -sub-Gaussian. This would result in a regret upper bound which can be obtained by replacing the term  $\sqrt{R^2 / (\mathbb{B} / \lceil \frac{\sigma_{\mathrm{max}}^2}{R^2} \rceil - 1)}$  (Theorem 3.1) by  $\sigma_{\mathrm{max}} / \sqrt{n_0}$  (for simplicity, we have ignored the non-integer conditions, i.e., the ceiling operators). Also note that with our optimal choice of  $R^2$  (the paragraph above), it can be easily verified that the term  $\sqrt{R^2 / (\mathbb{B} / \lceil \frac{\sigma_{\mathrm{max}}^2}{R^2} \rceil - 1)}$  (Theorem 3.1) can be simplified to  $\sigma_{\mathrm{max}} / \sqrt{\mathbb{B}}$ . Therefore, given that  $n_0 \leq \mathbb{B}$ , our regret upper bound (with the scaling of  $\sigma_{\mathrm{max}} / \sqrt{\mathbb{B}}$ ) is guaranteed to be no worse than that of uniform sample allocation (with the scaling of  $\sigma_{\mathrm{max}} / \sqrt{n_0}$ ).
3.2 BTS-RED with Unknown Noise Variance Function
Here we consider the more common scenario where the noise variance function  $\sigma^2 (\cdot)$  is unknown by extending BTS-RED-Known while preserving its theoretical guarantee.
3.2.1 Modeling of Noise Variance Function