SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \min  _ {b \in [ b _ {t} ]} r _ {t} ^ {[ b ]} \mid \mathcal {F} _ {t - 1} \right] \leq \mathbb {E} \left[ \frac {1}{b _ {t}} \sum_ {b = 1} ^ {b _ {t}} \left[ c _ {t} \left(2 \sigma_ {t - 1} (\overline {{\boldsymbol {x}}} _ {t}) + \sigma_ {t - 1} (\boldsymbol {x} _ {t} ^ {[ b ]})\right) \right| \mathcal {F} _ {t - 1} \right] + 2 B \mathbb {P} \left[ \overline {{E ^ {f _ {t}} (t)}} | \mathcal {F} _ {t - 1} \right] \\ \leq \mathbb {E} \left[ \frac {1}{b _ {t}} \sum_ {b = 1} ^ {b _ {t}} \left[ \frac {2 c _ {t}}{p - 1 / t ^ {2}} \sigma_ {t - 1} (\boldsymbol {x} _ {t} ^ {[ b ]}) + c _ {t} \sigma_ {t - 1} (\boldsymbol {x} _ {t} ^ {[ b ]}) \right] \Big | \mathcal {F} _ {t - 1} \right] + \frac {2 B}{t ^ {2}} \\ \leq c _ {t} \left(1 + \frac {2}{p - 1 / t ^ {2}}\right) \mathbb {E} \left[ \frac {1}{b _ {t}} \sum_ {b = 1} ^ {b _ {t}} \sigma_ {t - 1} \left(\boldsymbol {x} _ {t} ^ {[ b ]}\right) \Big | \mathcal {F} _ {t - 1} \right] + \frac {2 B}{t ^ {2}} \\ \leq c _ {t} e ^ {C} \left(1 + \frac {2}{p - 1 / t ^ {2}}\right) \mathbb {E} \left[ \frac {1}{b _ {t}} \sqrt {C _ {2} b _ {t} \mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} \Big | \mathcal {F} _ {t - 1} \right] + \frac {2 B}{t ^ {2}} \\ \leq c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \mathbb {E} \left[ \sqrt {\frac {1}{b _ {t}} C _ {2} \mathbb {I} \left(f ; \boldsymbol {y} _ {t} \mid \boldsymbol {y} _ {1 : t - 1}\right)} \Big | \mathcal {F} _ {t - 1} \right] + \frac {2 B}{t ^ {2}}, \tag {18} \\ \end{array}
$$
where the second inequality follows from equation (16), the fourth inequality results from Lemma B.6, and the last inequality follows since  $\frac{2}{p - 1 / t^2} \leq \frac{10}{p}$ .
Definition B.8. Define  $Y_0 = 0$ , and for all  $t = 1, \dots, T$
$$
\begin{array}{l} \bar {r} _ {t} = \mathbb {I} \{E ^ {f} (t) \} \min _ {b \in [ b _ {t} ]} r _ {t} ^ {[ b ]}, \\ X _ {t} = \bar {r} _ {t} - c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {1}{b _ {t}} C _ {2} \mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} - \frac {2 B}{t ^ {2}} \\ Y _ {t} = \sum_ {s = 1} ^ {t} X _ {s}. \\ \end{array}
$$
Lemma B.9. Conditioned on Lemma B.7 (i.e., with probability of  $\geq 1 - \delta /2$ ),  $(Y_{t}:t = 0,\dots ,T)$  is a super-martingale with respect to the filtration  $\mathcal{F}_t$
$$
\begin{array}{l} \mathbb {E} \left[ Y _ {t} - Y _ {t - 1} \mid \mathcal {F} _ {t - 1} \right] = \mathbb {E} \left[ X _ {t} \mid \mathcal {F} _ {t - 1} \right] \\ = \mathbb {E} \left[ \bar {r} _ {t} - c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {1}{b _ {t}} C _ {2} \mathbb {I} \left(f ; \boldsymbol {y} _ {t} \mid \boldsymbol {y} _ {1 : t - 1}\right)} - \frac {2 B}{t ^ {2}} \mid \mathcal {F} _ {t - 1} \right] \\ = \mathbb {E} \left[ \bar {r} _ {t} \mid \mathcal {F} _ {t - 1} \right] - \mathbb {E} \left[ c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {1}{b _ {t}} C _ {2} \mathbb {I} \left(f ; \boldsymbol {y} _ {t} \mid \boldsymbol {y} _ {1 : t - 1}\right)} + \frac {2 B}{t ^ {2}} \mid \mathcal {F} _ {t - 1} \right] \leq 0. \tag {19} \\ \end{array}
$$
When the event  $E^{f}(t)$  holds, then  $\overline{r}_t = \min_{b\in [b_t]}r_t^{[b]}$  and the inequality follows from Lemma B.7; when  $E^{f}(t)$  does not hold,  $\overline{r}_t = 0$  and hence the inequality trivially holds.
Lemma B.10. Define  $C_0 \triangleq \frac{1}{\mathbb{B} / \lceil \frac{\sigma_{\max}^2}{R^2} \rceil - 1}$ . Given  $\delta \in (0,1)$ , then with probability of at least  $1 - \delta$
$$
R _ {T} \leq e ^ {C} \left(1 + \frac {1 0}{p}\right) c _ {T} \sqrt {C _ {0} C _ {2} \Gamma_ {T} T} + \frac {B \pi^ {2}}{3} + \left(4 B + c _ {T} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {C C _ {0} C _ {2}}\right) \sqrt {2 \log (2 / \delta) T}.
$$
Proof. To begin with, let's derive a lower bound on  $b_{t}$ . Firstly, note that  $n_t \leq \left\lceil \frac{\sigma_{\max}^2}{R^2} \right\rceil$ . This implies that  $b_{t} \geq \mathbb{B} / \left\lceil \frac{\sigma_{\max}^2}{R^2} \right\rceil - 1$ . Next, we need to derive an upper bound on  $|Y_{t} - Y_{t-1}|$  which will be used when we apply the Azuma-Hoeffding's inequality:
$$
\begin{array}{l} \left| Y _ {t} - Y _ {t - 1} \right| = \left| X _ {t} \right| \leq \left| \bar {r} _ {t} \right| + c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {1}{b _ {t}} C _ {2} \mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} + \frac {2 B}{t ^ {2}} \\ \leq 2 B + c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {C C _ {2}}{b _ {t}}} + 2 B \tag {20} \\ \leq 4 B + c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {C C _ {2}}{b _ {t}}}, \\ \end{array}
$$
where the second inequality follows because
$$
\mathbb {I} \left(f; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1: t - 1}\right) \leq \max  _ {A \subset \mathcal {X}, | A | \leq b _ {t}} \mathbb {I} \left(f; \boldsymbol {y} _ {A} | \boldsymbol {y} _ {1: t - 1}\right) \leq \max  _ {A \subset \mathcal {X}, | A | \leq \mathbb {B}} \mathbb {I} \left(f; \boldsymbol {y} _ {A} | \boldsymbol {y} _ {1: t - 1}\right) \leq C, \forall t \geq 1.
$$
Next, we will need the following result to connect the sum of condition information gains to the maximum information gain:
$$
\sum_ {t = 1} ^ {T} \sqrt {\mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} \leq \sqrt {T \sum_ {t = 1} ^ {T} \mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} = \sqrt {T \mathbb {I} (f ; \boldsymbol {y} _ {1 : T})} \leq \sqrt {T \Gamma_ {T \mathbb {B}}}, \tag {21}
$$
in which the first inequality results from the Cauchy-Schwarz inequality, the second inequality follows from the chain rule of conditional information gain, and the last inequality makes use of the
fact that  $\tau_T \leq T\mathbb{B}$ . Subsequently, we are ready to upper-bound the batch cumulative regret:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \bar {r} _ {t} \leq \sum_ {t = 1} ^ {T} c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {1}{b _ {t}} C _ {2} \mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} + \sum_ {t = 1} ^ {T} \frac {2 B}{t ^ {2}} + \\ \sqrt {2 \log (2 / \delta) \sum_ {t = 1} ^ {T} \left(4 B + c _ {t} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {C C _ {2}}{b _ {t}}}\right) ^ {2}} \\ \leq e ^ {C} \left(1 + \frac {1 0}{p}\right) c _ {T} \sqrt {\frac {C _ {2}}{\mathbb {B} / \lceil \frac {\sigma_ {\max} ^ {2}}{R ^ {2}} \rceil - 1}} \sum_ {t = 1} ^ {T} \sqrt {\mathbb {I} (f ; \boldsymbol {y} _ {t} | \boldsymbol {y} _ {1 : t - 1})} + \frac {B \pi^ {2}}{3} + \\ \left(4 B + c _ {T} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {\frac {C C _ {2}}{\mathbb {B} / \lceil \frac {\sigma_ {\max} ^ {2}}{R ^ {2}} \rceil - 1}}\right) \sqrt {2 \log (2 / \delta) T} \\ \leq e ^ {C} \left(1 + \frac {1 0}{p}\right) c _ {T} \sqrt {C _ {0} C _ {2}} \sqrt {T \Gamma_ {T \mathbb {B}}} + \frac {B \pi^ {2}}{3} + \left(4 B + c _ {T} e ^ {C} \left(1 + \frac {1 0}{p}\right) \sqrt {C C _ {0} C _ {2}}\right) \sqrt {2 \log (2 / \delta) T}. \tag {22} \\ \end{array}
$$
The first inequality results from the Azuma-Hoeffding's inequality, the second inequality makes use of the lower bound on  $b_{t}$ :  $b_{t} \geq \mathbb{B} / \left\lceil \frac{\sigma_{\max}^{2}}{R^{2}} \right\rceil - 1$ , and the last inequality follows from equation (21). Equation (22) holds with probability of  $\geq 1 - \delta / 2$  according to the Azuma-Hoeffding's inequality. Then, we also have that  $r_{t} = \overline{r}_{t}, \forall t \geq 1$  with probability of  $\geq 1 - \delta / 2$  according to Lemma B.1. This completes the proof.
Note that  $c_{T} = \widetilde{\mathcal{O}}(R\sqrt{\Gamma_{T\mathbb{B}}})$ , which allows us to simplify the regret upper bound into an asymptotic expression:
$$
R _ {T} = \widetilde {\mathcal {O}} \left(e ^ {C} R \frac {1}{\sqrt {\mathbb {B} / \lceil \frac {\sigma_ {\operatorname* {m a x}} ^ {2}}{R ^ {2}} \rceil - 1}} \sqrt {T \Gamma_ {T \mathbb {B}}} \left(\sqrt {C} + \sqrt {\Gamma_ {T \mathbb {B}}}\right)\right). \tag {23}
$$
B.1 Simplified Regret Upper Bound for Constant Noise Variance
In the special case where the noise variance is fixed throughout the entire domain, i.e., when  $\sigma^2 (\pmb {x}) = \sigma_{\mathrm{const}}^2,\forall \pmb {x}\in \mathcal{X}$ , we have that  $n_t = \lceil \sigma_{\mathrm{const}}^2 /R^2\rceil = n_{\mathrm{const}}$  and  $b_{t} = \lfloor \mathbb{B} / n_{\mathrm{const}}\rfloor = b_{0},\forall t\in [T]$ . In this case, instead of making use of the lower bound on  $b_{t}$  (i.e.,  $b_{t}\geq \mathbb{B} / \left\lceil \frac{\sigma_{\mathrm{max}}^{2}}{R^{2}}\right\rceil -1$ ) as done in the proof of Lemma B.10, we can simply replace  $b_{t}$  with  $b_{0}$  in the proof. As a result, the term of  $\frac{1}{\sqrt{\mathbb{B} / \left\lceil\frac{\sigma_{\mathrm{max}}^2}{R^2}\right\rceil - 1}}$
in the regret upper bound proved in Lemma B.10 can be simply replaced by  $\frac{1}{\sqrt{b_0}}$ , and hence the regret upper bound can be further simplified as:
$$
R _ {T} = \widetilde {\mathcal {O}} \left(e ^ {C} R \frac {1}{\sqrt {b _ {0}}} \sqrt {T \Gamma_ {T b _ {0}}} \left(\sqrt {C} + \sqrt {\Gamma_ {T b _ {0}}}\right)\right). \tag {24}
$$
As another special case where  $\sigma^2 (\pmb {x}) = \sigma_{\mathrm{const}}^2 = R^2$ , every input  $\pmb{x}$  will be evaluated only once (i.e.,  $n_{\mathrm{const}} = 1$ ) and  $b_{0} = \mathbb{B}$ . In this case, our algorithm reduces to the standard batch TS with a batch size of  $\mathbb{B}$ , and the regret upper bound becomes
$$
R _ {T} = \tilde {\mathcal {O}} \left(e ^ {C} R \frac {1}{\sqrt {\mathbb {B}}} \sqrt {T \Gamma_ {T \mathbb {B}}} \left(\sqrt {C} + \sqrt {\Gamma_ {T \mathbb {B}}}\right)\right). \tag {25}
$$
C Choice of  $R^2$  by Minimizing the Regret Upper Bound in Theorem 3.1
The regret bound in Theorem 3.1 depends on the parameter  $R$  through the term  $g \triangleq \sqrt{\frac{R^2}{\mathbb{B} / (\sigma_{\max}^2 / R^2 + 1) - 1}}$ , in which we have replaced the term  $\lceil \sigma_{\max}^2 / R^2 \rceil$  by  $\sigma_{\max}^2 / R^2 + 1$  such that the
resulting regret upper bound is still valid and the subsequent derivations become simplified. Taking the derivative of  $g^2$  w.r.t.  $R^2$  gives us
$$
\frac {\mathrm {d} g ^ {2}}{\mathrm {d} R ^ {2}} = \frac {(\mathbb {B} - 1) (R ^ {2}) ^ {2} - 2 \sigma_ {\max } ^ {2} R ^ {2} - \sigma_ {\max } ^ {4}}{\left[ ((\mathbb {B} - 1) R ^ {2} - \sigma_ {\max } ^ {2}) \right] ^ {2}}. \tag {26}
$$
Setting the above derivative to 0, we have that the value of  $R^2$  that minimizes  $g^2$  and  $g$  (hence minimizing the regret upper bound) is obtained at
$$
R ^ {2} = \sigma_ {\max } ^ {2} \frac {\sqrt {\mathbb {B}} + 1}{\mathbb {B} - 1}. \tag {27}
$$
D Upper Bound on Sequential Cumulative Regret
E Confidence Bound for the Noise Variance Function
Here, as discussed in Sec. 3.2 in the main text, we leverage the concentration of the Chi-squared distribution to show that we can interpret the unbiased estimate of the noise variance in equation (1) as a noisy observation corrupted by a sub-Gaussian noise. For every queried input, we use the unbiased empirical variance (1) as the observation to update the GP posterior for the noise variance. First of all, when an input  $\boldsymbol{x}_t^{[b]}$  is queried, denote the unbiased empirical variance as  $\widetilde{\sigma^2}(\boldsymbol{x}_t^{[b]}) = \sigma^2(\boldsymbol{x}_t^{[b]}) + \epsilon'$ , and we will show next that  $\epsilon'$  is (with high probability) a sub-Gaussian noise. Denote as  $n_t$  the number replications that we use to query  $\boldsymbol{x}_t^{[b]}$ . Since the unbiased empirical variance of a Gaussian random variable follows a Chi-squared distribution, we have that  $\widetilde{\sigma^2}(\boldsymbol{x}_t^{[b]}) \in \left[\frac{\sigma^2(\boldsymbol{x}_t^{[b]})\chi_{n_t - 1,\alpha / 2}^2}{n_t - 1}, \frac{\sigma^2(\boldsymbol{x}_t^{[b]})\chi_{n_t - 1,1 - \alpha / 2}^2}{n_t - 1}\right]$  with probability of  $\geq 1 - \alpha$ , in which  $\chi_{n_t - 1,\alpha / 2}^2 (\chi_{n_t - 1,1 - \alpha / 2}^2)$  denotes the  $(\alpha / 2)^{\mathrm{th}}$ -quantile  $((1 - \alpha / 2)^{\mathrm{th}}$ -quantile) of the Chi-square distribution with  $n_t - 1$  degrees of freedom. This allows us to show that
$$
\begin{array}{l} \epsilon^ {\prime} \in \left[ \sigma^ {2} \left(\boldsymbol {x} _ {t} ^ {[ b ]}\right) \left(\frac {\chi_ {n _ {t} - 1 , \alpha / 2} ^ {2}}{n _ {t} - 1} - 1\right), \sigma^ {2} \left(\boldsymbol {x} _ {t} ^ {[ b ]}\right) \left(\frac {\chi_ {n _ {t} - 1 , 1 - \alpha / 2} ^ {2}}{n _ {t} - 1} - 1\right) \right] \tag {28} \\ \in \left[ \sigma_ {\min} ^ {2} \left(\frac {\chi_ {n _ {\min} - 1 , \alpha / 2} ^ {2}}{n _ {\min} - 1} - 1\right), \sigma_ {\max} ^ {2} \left(\frac {\chi_ {n _ {\min} - 1 , 1 - \alpha / 2} ^ {2}}{n _ {\min} - 1} - 1\right) \right] \triangleq [ L _ {\alpha}, U _ {\alpha} ], \\ \end{array}
$$
which holds with probability  $\geq 1 - \alpha$ .  $n_{\mathrm{min}}$  is the minimum number of repetitions we impose on every queried  $\boldsymbol{x}_t$ . Note that the discussion above on the unbiasedness and boundedness of the noise  $\epsilon'$  still holds after we negate the noise variance (i.e.,  $g(\cdot) = -\sigma^2(\cdot)$ ) to analyze  $\hat{y}_t^{[b]} = g(\boldsymbol{x}_t^{[b]}) + \epsilon'$ . Then the discussion in the main text (Sec. 3.2) follows.
F The BTS-RED-Unknown Algorithm
Our BTS-RED-Unknown algorithm is shown in Algorithm 3.
Our proof here follows a similar structure to the proof in Appendix B, but non-trivial changes need to be made to account for the change of objective function here. Here, for a given  $\omega$ , we attempt to upper-bound the mean-variance cumulative regret  $R_T^{\mathrm{MV}} = \sum_{t=1}^{T} \min_{b \in [b_t]} [h_\omega(\pmb{x}_\omega^*) - h_\omega(\pmb{x}_t^{[b]})]$ , where  $h_\omega(\pmb{x}) = \omega f(\pmb{x}) + (1 - \omega) g(\pmb{x}) = \omega f(\pmb{x}) - (1 - \omega) \sigma^2(\pmb{x})$  and  $\pmb{x}_\omega^* \in \arg \max_{\pmb{x} \in \mathcal{X}} [\omega f(\pmb{x}) + (1 - \omega) g(\pmb{x})]$ . Throughout the analysis here, we assumed a pre-defined  $\omega$  and hence omit any dependence on  $\omega$  for simplicity. That is, we use  $\pmb{x}^*$  in place of  $\pmb{x}_\omega^*$  and use  $h(\cdot)$  to represent  $h_\omega(\cdot)$ .
Algorithm 3 BTS-RED-Unknown.
1: for  $t = 1,2,\ldots ,T$  do  
2:  $b = 0,n_{t}^{(0)} = 0$   
3: while  $\sum_{b^{\prime} = 0}^{b}n_{t}^{(b^{\prime})} < \mathbb{B}$  do  
4:  $b\gets b + 1$   
5: Sample  $f_{t}^{(b)}$  from  $\mathcal{GP}(\mu_{t - 1}(\cdot),\beta_t^2\sigma_{t - 1}^2 (\cdot ,\cdot))$   
6: Choose  $\pmb{x}_{t}^{(b)} = \arg \max_{\pmb {x}\in \mathcal{X}}f_{t}^{(b)}(\pmb {x})$   
7: Choose  $n_t^{(b)} = \lceil (-\mu_{t - 1}'(\pmb {x}_t^{(b)}) + \beta_t'\sigma_{t - 1}'(\pmb {x}_t^{(b)})) / R^2\rceil$   
8:  $b_{t} = b - 1$   
9: for  $b\in [b_t]$ , query  $\pmb{x}_{t}^{(b)}$  with  $n_t^{(b)}$  parallel processes  
10: for  $b\in [b_t]$ , observe  $\{y_{t,n}^{(b)}\}_{n\in [n_t^{(b)}]}$ , then calculate  $y_{t}^{(b)} = (1 / n_{t}^{(b)})\sum_{n = 1}^{n_{t}^{(b)}}y_{t,n}^{(b)}$  and  $\widetilde{y}_t^{(b)} = -1 / (n_t^{(b)} - 1)\sum_{n = 1}^{n_t^{(b)}}\left(y_{t,n}^{(b)} - y_t^{(b)}\right)^2$   
11: Use  $\{(\pmb{x}_t^{(b)},y_t^{(b)})\}_{b\in [b_t]}$  to update posterior of  $\mathcal{GP}$   
12: Use  $\{(\pmb{x}_t^{(b)},\widetilde{y}_t^{(b)})\}_{b\in [b_t]}$  to update posterior of  $\mathcal{GP'}$
Here, same as the main text, we use  $g:\mathcal{X}\to \mathbb{R}^-$  to denote the function  $-\sigma^2:\mathcal{X}\rightarrow \mathbb{R}^-$ . We use  $\mu_{t - 1}$  and  $\sigma_{t - 1}$  to denote the GP posterior mean and standard deviation of the GP for  $f$  conditioned on all  $\tau_{t - 1}$  observations up to (and including) iteration  $t - 1$ , and use  $\mu_{t - 1}^{\prime}$  and  $\sigma_{t - 1}^{\prime}$  to denote the GP posterior mean and standard deviation of the GP for the negative noise variance  $-\sigma^2$ . Denote  $h_t^{[b]}(\pmb {x}) = \omega f_t^{[b]}(\pmb {x}) + (1 - \omega)g_t^{[b]}(\pmb {x})$ , such that  $\pmb{x}_t^{[b]}\in \arg \max_{\pmb {x}\in \mathcal{X}}h_t^{[b]}$ . Denote by  $\mathcal{F}_{t - 1}^{\prime}$  the history of observed pairs of input and empirical noise variance up to iteration  $t - 1$ .
Define  $\beta_{t} \triangleq B + R\sqrt{2(\Gamma_{\tau_{t - 1}} + 1 + \log(3 / \delta))}$  and  $c_{t} \triangleq \beta_{t}(1 + \sqrt{2\log(2\mathbb{B}|\mathcal{X}|t^{2}))}$ . Also define  $\beta_{t}' \triangleq B' + R'\sqrt{2(\Gamma_{\tau_{t - 1}}' + 1 + \log(3 / \delta))}$  and  $c_{t}' \triangleq \beta_{t}'(1 + \sqrt{2\log(2\mathbb{B}|\mathcal{X}|t^{2}))}$ .
Lemma G.1. Let  $\delta \in (0,1)$ . Define  $E^g(t)$  as the event that  $|\mu_{t-1}'(\pmb{x}) - g(\pmb{x})| \leq \beta_t' \sigma_{t-1}'(\pmb{x})$  for all  $\pmb{x} \in \mathcal{X}$ . We have that  $\mathbb{P}\left[E^g(t)\right] \geq 1 - \delta/3$  for all  $t \geq 1$ .
The validity of Lemma G.1 follows from the discussion in Sec. 3.2 (i.e., equation (G.1)), after replacing the error probability of  $\delta / 2$  by  $\delta / 3$ . We also need Lemma B.1 to hold, and since we have replaced the error probability of  $\delta / 2$  in  $\beta_{t}$  from Lemma B.1 by  $\delta / 3$  in our definition of  $\beta_{t}$  above, we have that the event  $E^{f}(t)$  in Lemma B.1 holds with probability of  $\geq 1 - \delta / 3$  here.
Lemma G.2. Define  $E^{g_t}(t)$  as the event:  $|g_t^{[b]}(\pmb {x}) - \mu_{t - 1}'(\pmb {x})|\leq \beta_t'\sqrt{2\log(2\mathbb{B}|\mathcal{X}|t^2)}\sigma_{t - 1}'(\pmb {x}),$ $\forall b\in [b_t]$ . We have that  $\mathbb{P}\left[E^{g_t}(t)|\mathcal{F}_{t - 1}\right]\geq 1 - 1 / (2t^2)$  for any possible filtration  $\mathcal{F}_{t - 1}^{\prime}$ .
Similarly, we will also need Lemma B.2 to hold, but replace the error probability of  $1 / t^2$  by  $1 / (2t^2)$ . Of note, we have also correspondingly changed the value of  $c_{t}$  from Appendix B by replacing  $t^2$  by  $2t^2$  in our definition of  $c_{t}$  above.
The definition of saturated points also needs to be modified:
Definition G.3. Define the set of saturated points at iteration  $t$  as
$$
S _ {t} ^ {\prime} = \left\{\boldsymbol {x} \in \mathcal {X}: \Delta (\boldsymbol {x}) > \omega c _ {t} \sigma_ {t - 1} (\boldsymbol {x}) + (1 - \omega) c _ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x}) \right\},
$$
in which  $\Delta (\pmb {x}) = h(\pmb{x}^{*}) - h(\pmb {x})$  and  $\pmb{x}^{*}\in \arg \max_{\pmb {x}\in \mathcal{X}}h(\pmb {x})$
The following lemma is a counterpart to Lemma B.4 in Appendix B, and the proof here makes use of the same techniques.
Lemma G.4. For any  $\mathcal{F}_{t - 1}^{\prime}$ , conditioned on the events  $E^{g}(t)$ , we have that  $\forall \boldsymbol {x}\in \mathcal{X},b\in [b_t]$
$$
\mathbb {P} \left(g _ {t} ^ {[ b ]} (\boldsymbol {x}) > g (\boldsymbol {x}) \mid \mathcal {F} _ {t - 1} ^ {\prime}\right) \geq p, \tag {29}
$$
$$
\begin{array}{l} \mathbb {P} \left(g _ {t} ^ {[ b ]} (\boldsymbol {x}) > g (\boldsymbol {x}) | \mathcal {F} _ {t - 1} ^ {\prime}\right) = \mathbb {P} \left(\frac {g _ {t} ^ {[ b ]} (\boldsymbol {x}) - \mu_ {t - 1} ^ {\prime} (\boldsymbol {x})}{\beta_ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x})} > \frac {g (\boldsymbol {x}) - \mu_ {t - 1} ^ {\prime} (\boldsymbol {x})}{\beta_ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x})} \Big | \mathcal {F} _ {t - 1} ^ {\prime}\right) \\ \geq \mathbb {P} \left(\frac {g _ {t} ^ {[ b ]} (\boldsymbol {x}) - \mu_ {t - 1} ^ {\prime} (\boldsymbol {x})}{\beta_ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x})} > \frac {| g (\boldsymbol {x}) - \mu_ {t - 1} ^ {\prime} (\boldsymbol {x}) |}{\beta_ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x})} \mid \mathcal {F} _ {t - 1} ^ {\prime}\right) \tag {30} \\ \geq \mathbb {P} \left(\frac {g _ {t} ^ {[ b ]} (\boldsymbol {x}) - \mu_ {t - 1} ^ {\prime} (\boldsymbol {x})}{\beta_ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x})} > 1 \mid \mathcal {F} _ {t - 1} ^ {\prime}\right) \\ \geq \frac {e ^ {- 1}}{4 \sqrt {\pi}}. \\ \end{array}
$$
The next Lemma is the counterpart to Lemma B.5 in Appendix B, but additional challenges need to be carefully handled here.
Lemma G.5. For any  $\mathcal{F}_{t - 1}$  and  $\mathcal{F}_{t - 1}^{\prime}$ , conditioned on the events  $E^{f}(t)$  and  $E^{g}(t)$ , we have that
$$
\mathbb {P} \left(\boldsymbol {x} _ {t} ^ {[ b ]} \in \mathcal {X} \setminus S _ {t} ^ {\prime} | \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right) \geq p ^ {2} - 1 / t ^ {2}, \quad \forall b \in [ b _ {t} ].
$$
$$
\mathbb {P} \left(\boldsymbol {x} _ {t} ^ {[ b ]} \in \mathcal {X} \backslash S _ {t} ^ {\prime} | \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right) \geq \mathbb {P} \left(h _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) > h _ {t} ^ {[ b ]} (\boldsymbol {x}), \forall \boldsymbol {x} \in S _ {t} | \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right), \tag {31}
$$
which holds  $\forall b\in [b_t]$ . This inequality follows from noting that  $\pmb{x}^*$  is always unsaturated according to Definition G.3, and that  $\pmb{x}_t^{[b]} = \arg \max_{\pmb {x}\in \mathcal{X}}h_t^{[b]}(\pmb {x}) = \arg \max_{\pmb {x}\in \mathcal{X}}(\omega f_t^{[b]}(\pmb {x}) + (1 - \omega)g_t^{[b]}(\pmb {x}))$
Next, we assume that the events  $E^{f}(t), E^{f_{t}}(t), E^{g}(t)$  and  $E^{g_{t}}(t)$  all hold, which allows us to derive an upper bound on  $h_t^{[b]}(\pmb{x})$  for all  $\pmb{x} \in S_t'$  and for all  $b \in [b_t]$ :
$$
\begin{array}{l} h _ {t} ^ {[ b ]} (\boldsymbol {x}) = \omega f _ {t} ^ {[ b ]} (\boldsymbol {x}) + (1 - \omega) g _ {t} ^ {[ b ]} (\boldsymbol {x}) \\ \leq \omega (f (\boldsymbol {x}) + c _ {t} \sigma_ {t - 1} (\boldsymbol {x})) + (1 - \omega) (g (\boldsymbol {x}) + c _ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x})) \\ = \omega f (\boldsymbol {x}) + (1 - \omega) g (\boldsymbol {x}) + \omega c _ {t} \sigma_ {t - 1} (\boldsymbol {x}) + (1 - \omega) c _ {t} ^ {\prime} \sigma_ {t - 1} ^ {\prime} (\boldsymbol {x}) \tag {32} \\ \leq h (\boldsymbol {x}) + \Delta (\boldsymbol {x}) \\ = h (\boldsymbol {x}) + h \left(\boldsymbol {x} ^ {*}\right) - h (\boldsymbol {x}) = h \left(\boldsymbol {x} ^ {*}\right), \\ \end{array}
$$
where the first inequality follows from Lemmas B.1, B.2, G.1 and G.2, and the second inequality makes use of the definition of  $h(\pmb{x})$  as well as Definition G.3.
Therefore, (32) implies that for every  $b \in [b_t]$ , if the events  $E^{f}(t), E^{f_{t}}(t), E^{g}(t)$  and  $E^{gt}(t)$  all hold, then we have that
$$
\begin{array}{l} \mathbb {P} \left(h _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) > h _ {t} ^ {[ b ]} (\boldsymbol {x}), \forall \boldsymbol {x} \in S _ {t} \mid \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right) \geq \mathbb {P} \left(h _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) > h \left(\boldsymbol {x} ^ {*}\right) \mid \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right) \\ = \mathbb {P} \left(\omega f _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) + (1 - \omega) g _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) > \omega f \left(\boldsymbol {x} ^ {*}\right) + (1 - \omega) g \left(\boldsymbol {x} ^ {*}\right) \mid \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right) \\ \geq \mathbb {P} \left(f _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) > f \left(\boldsymbol {x} ^ {*}\right) \text {a n d} g _ {t} ^ {[ b ]} \left(\boldsymbol {x} ^ {*}\right) > g \left(\boldsymbol {x} ^ {*}\right) \mid \mathcal {F} _ {t - 1}, \mathcal {F} _ {t - 1} ^ {\prime}\right) \\ \geq p ^ {2}. \tag {33} \\ \end{array}
$$
The second inequality results since the event in the third line implies the event in the line above, and the last inequality follows from Lemmas B.4 and G.4.
Next, for every  $b \in [b_t]$ , we can show that