SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that the definition of  $\Sigma_p$  is consistent with the definition of  $\Sigma_{\infty}$  introduced in Theorem 3
4Lemma 2 holds for all choices of  $p\in [1, + \infty ]$  . In particular, it is still true in the case where  $p = +\infty$  , as it can be understood by taking the limit  $p\to +\infty$  in the inequalities:  $0\leq \mathrm{UCB}_t(\sigma_g)^2 -\sigma_g^2\leq$ $\frac{4C_T}{\sqrt{n_{g,t}}}\left(\sigma_g + \frac{2C_T}{\sqrt{n_{g,t}}}\right)$
Lemma 3. Variance-UCB collects a vector of samples  $\mathbf{n}$  such that for all  $g\in [G]$ , with probability at least  $1 - \tilde{O} (T^{-2})$
$$
n _ {g, T} - n _ {g, T} ^ {*} \leq 3 + \frac {4 C _ {T} p}{\Sigma_ {p} (p + 1)} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , T} ^ {*}}}\right) ^ {\frac {p - 1}{p + 1}} \sqrt {n _ {g , T} ^ {*}} = \Theta (\sqrt {T}).
$$
To understand the meaning of Lemma 3, we note that the feedback-dependent structure of the algorithm makes the numbers of samples  $n_{g,T}$  correlated across groups and over time, since the algorithm attempts to sample regularly from all groups. A key technical challenge is to decouple  $n_{g,t}$  across groups and derive an instance dependent upper bound on  $\pmb{n}_T - \pmb{n}_T^*$ . This challenge does not arise in the classic multi-armed bandits setting, where the decision-maker's goal is to repeatedly pull only the best arm.
**Curvature properties of  $R_p$  around the optimal value  $n_T^*$ :** To answer the second question, we exploit the smoothness of  $R_p(\cdot)$  around  $n_T^*$  to approximate it with a polynomial function of  $n$ . Since  $n_T^*$  is the minimizer of  $R_p(\cdot)$  (subject to  $\sum_{g \in [G]} n_g = T$ ) and  $R_p(\cdot)$  is differentiable around  $n_T^*$ , the first order of the Taylor approximation of  $R_p$  vanishes as  $T$  grows large, formalized in Lemma 4.
Lemma 4. Let  $p < +\infty$  and  $\mathbf{n}' \in \mathbb{R}_+^G$  such that  $\sum_{g \in [G]} n_g' = T$ . Then,
$$
\left| \frac {R _ {p} (\boldsymbol {n} ^ {\prime}) - R _ {p} (\boldsymbol {n} _ {T} ^ {*})}{R _ {p} (\boldsymbol {n} ^ {*})} - \frac {p + 1}{2} \sum_ {g \in [ G ]} \frac {(n _ {g} ^ {\prime} - n _ {g , T} ^ {*}) ^ {2}}{T n _ {g , T} ^ {*}} \right| \leq \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\min} ^ {2}} \max  _ {g} \left(\frac {n _ {g , T} ^ {*}}{n _ {g} ^ {\prime}}\right) ^ {3 p + 3} \frac {\| \boldsymbol {n} ^ {\prime} - \boldsymbol {n} _ {T} ^ {*} \| _ {\infty} ^ {3}}{T ^ {3}}.
$$
We note that the bound in Lemma 4 holds regardless of the choice of the vector  $\mathbf{n}^{\prime}$ , including those generated by Variance-UCB. One can interpret the upper bound in Lemma 4 as follows: the term  $\frac{p + 1}{2}\sum_{g\in [G]}\frac{(n_g' - n_{g,T}^*)^2}{Tn_{g,T}^*}$  represents the exact Taylor first-order approximation of  $\frac{R_p(\boldsymbol{n}') - R_p(\boldsymbol{n}_T^*)}{R_p(\boldsymbol{n}^*)}$ , and the right hand side represents an upper bound on this approximation. In particular, assuming an error  $\epsilon = \boldsymbol{n}' - \boldsymbol{n}^*$ , Lemma 4 can be restated as,
$$
\frac {R _ {p} (\boldsymbol {n} ^ {\prime}) - R _ {p} (\boldsymbol {n} _ {T} ^ {*})}{R _ {p} (\boldsymbol {n} ^ {*})} = \Theta \left(\frac {\| \boldsymbol {\epsilon} \| ^ {2}}{T ^ {2}}\right) + o \left(\frac {\| \boldsymbol {\epsilon} \| ^ {2}}{T ^ {2}}\right).
$$
Putting everything together: The rest of the proof consists of applying Lemmas 3 and 4 to the regret expression in Eq. (7). By Lemma 3, with high probability,  $\| \pmb{n}_T - \pmb{n}_T^*\|_\infty = \tilde{O}\left(\sqrt{T}\right)$ . Applying this to Lemma 4 gives that  $R_{p}(\pmb{n}_{T}) - R_{p}(\pmb{n}_{T}^{*}) = \| \pmb{n}_{T} - \pmb{n}_{T}^{*}\|_{\infty}^{2}R_{p}^{*}\cdot O\left(T^{-2}\right)$ . By Lemma 1,  $R_{p}^{*} = \Theta (T^{-1})$ , and therefore with high probability,  $R_{p}(\pmb{n}_{T}) - R_{p}(\pmb{n}_{T}^{*}) = \tilde{O}(T^{-2})$ . With additional work, we show that the equality also holds in expectation, which implies from Eq. (7) that  $\mathrm{Regret}_{p,T}(\mathrm{Variance-UCB},\mathcal{D}) = \mathbb{E}_{\pi}[R_p(\pmb {n}) - R_p(\pmb {n}_T^*)] = \tilde{O} (T^{-2})$ .
4.1 The case  $p = +\infty$
Even though Lemmas 1, 2, and 3 hold for  $p = +\infty$ , the approximation guarantee given in Lemma 4 does not hold because  $R_{\infty}$  is not differentiable at  $\boldsymbol{n}_T^*$ . As an alternative, we derive a first order upper bound, which we state in Lemma 5.
Lemma 5. Let  $\sigma \in \mathbb{R}_+^G$  and  $\pmb{n}^{\prime}\in \mathbb{R}_{+}^{G}$  such that  $\sum_{g\in [G]}n_g' = T$ . Then,
$$
\frac {R _ {\infty} \left(\boldsymbol {n} _ {T} ^ {\prime}\right) - R _ {\infty} \left(\boldsymbol {n} _ {T} ^ {*}\right)}{R _ {\infty} \left(\boldsymbol {n} _ {T} ^ {*}\right)} \leq - \min  _ {g} \left(\frac {n _ {g} ^ {\prime}}{n _ {g , T} ^ {*}} - 1\right) + \frac {1}{4} \max  _ {g} \left(\frac {n _ {g} ^ {\prime}}{n _ {g , T} ^ {*}} - 1\right) ^ {2} \max  _ {g} \left(\frac {n _ {g , T} ^ {*}}{n _ {g} ^ {\prime}}\right) ^ {3}
$$
Similar to Lemma 4, we interpret the upper bound above by decomposing it into a main term,  $-\min_{g}\left(\frac{n_{g}^{\prime}}{n_{g,T}^{*}} -1\right)$ , and an error term,  $\frac{1}{4}\max_{g}\left(\frac{n_{g}^{\prime}}{n_{g,T}^{*}} -1\right)^{2}\max_{g}\left(\frac{n_{g,T}^{*}}{n_{g}^{\prime}}\right)^{3}$ . As opposed to the case where  $p < +\infty$ ,  $R_{\infty}$  is not differentiable in  $\boldsymbol{n}^{*}$ , thus the inequality above does not stem from a Taylor approximation. Adapting the bound in Lemma 5 in the same steps as in Step 3 gives a regret bound of  $\tilde{O}(T^{-1.5})$ .
5 Deriving the lower bounds
In this section, we provide the key ideas behind the proof of Theorem 2, with the full proof given in Appendix B. Given any policy  $\pi$ , the idea is to pick  $\mathcal{D}_{\pi}$  from two constructed instances  $\mathcal{D}^a = \mathcal{D}_1^a \times \dots \times \mathcal{D}_G^a$  and  $\mathcal{D}^b = \mathcal{D}_1^b \times \dots \times \mathcal{D}_G^b$ . The interaction of  $\mathcal{D}^a$  (resp.  $\mathcal{D}^b$ ) with  $\pi$  yields a random vector of the number of collected samples, denoted by  $n^a$  (resp.  $n^b$ ).  $\mathcal{D}^a$  and  $\mathcal{D}^b$  must satisfy the following two conflicting properties:
1.  $\mathcal{D}^a$  and  $\mathcal{D}^b$  are sufficiently similar, in the sense that the distributions of  $n^a$  and  $n^b$  are close, so that  $\pi$  would have a hard time distinguishing between  $\mathcal{D}^a$  and  $\mathcal{D}^b$ . This notion of similarity is captured by the KL-divergence of  $\mathcal{D}^a$  and  $\mathcal{D}^b$ .
2.  $\mathcal{D}^a$  and  $\mathcal{D}^b$  are also sufficiently dissimilar in the sense that they induce distinct optimal allocation rules  $n_a^*$  and  $n_b^*$ . Specifically, any allocation  $n$  cannot be simultaneously close to both  $n_a^*$  and  $n_b^*$ , and  $n$  incurs a high regret under at least one of the two instances. This notion of dissimilarity is captured below.
Let  $\sigma^a\in \mathbb{R}_+^G$  be the vector of standard deviations of all groups under  $\pmb{\mathcal{D}}^a$ , and let  $S_{\epsilon}^{a}$  be the set of allocations  $\pmb{n}$  such that  $R_{p}(\pmb {n};\pmb{\sigma}^{a})$  is within  $\epsilon$  of  $R_{p}^{*}(\pmb{\sigma}^{a})$  for any  $\pmb {n}\in S_{\epsilon}^{a}$ . Formally,
$$
S _ {\epsilon} ^ {a} := \left\{\boldsymbol {n} \in \mathbb {N} ^ {G} \Big | \sum_ {g} n _ {g} = T, \quad R _ {p} (\boldsymbol {n}, \boldsymbol {\sigma} ^ {a}) - R _ {p} ^ {*} (\boldsymbol {\sigma} ^ {a}) \leq \epsilon \right\}.
$$
Let  $\sigma^b$  be a second vector and define  $S_{\epsilon}^{b}$  similarly. We define the dissimilarity  $d(\pmb{\sigma}^a, \pmb{\sigma}^b) \coloneqq \inf_{\epsilon \geq 0} \{ S_{\epsilon}^a \cap S_{\epsilon}^b \neq \emptyset \}$ , which is the smallest  $\epsilon$  so that these sets of allocations are disjoint. There is a tension between the two requirements on  $\mathcal{D}^a$  and  $\mathcal{D}^b$ , which is formalized next in Lemma 6.
Lemma 6. Let  $\pi$  be a fixed policy and  $\mathcal{D}^a$ ,  $\mathcal{D}^b$  be two instances with standard deviation vectors  $\sigma^a, \sigma^b$ , respectively. Then,
$\max \{Regret_{p,T}(\pmb {\pi},\pmb{\mathcal{D}}^a),Regret_{p,T}(\pmb {\pi},\pmb{\mathcal{D}}^b)\} \geq d(\pmb {\sigma}^a,\pmb {\sigma}^b)\exp \left(-\sum_{g\in [G]}\mathbb{E}_{\pmb {\pi},\pmb{\mathcal{D}}^a}[n_{g,t}]KL(\mathcal{D}_g^a ||\mathcal{D}_g^b)\right).$
The proof follows from an adapted version of LeCam's method [39]. To understand the implication of Lemma 6, we examine the two terms involved in the right hand side of the inequality. The first term,  $d(\sigma^a,\sigma^b)$ , increases the further  $\sigma^a$  and  $\sigma^b$  are from each other. The second term has the opposite monotonicity, as it decreases with the  $KL$  divergence of the two instances. These conflicting terms capture the trade-off of loss minimization versus information gain, and deriving the lower bound will come from constructing two instances for which this trade-off is maximised.
Specifically,  $\mathcal{D}^a$  is chosen such that all groups have data distributed according to  $\mathcal{D}_g^a\sim \mathcal{N}\left(0,\sigma^2\right)$ ; we denote  $\sigma^a\coloneqq \sigma (1,\ldots ,1)$  as the vector of standard deviations of each group. Even though instance  $a$  has all identical groups, the policy  $\pi$  may treat groups differently, and the interaction of  $\pi$  with  $\mathcal{D}^a$  may yield different expected numbers of samples from each group. We pick  $h\in [G]$  such that  $\mathbb{E}_{\pi ,\mathcal{D}^a}[n_{h,T}]$  is minimized over  $[G]$ , and construct  $\mathcal{D}^b$  as follows:
$$
\boldsymbol {\mathcal {D}} ^ {b}: \left\{ \begin{array}{l} \mathcal {D} _ {h} ^ {b} \sim \mathcal {N} \left(0, \sigma^ {2} (1 + \sqrt {\frac {G}{T}})\right) \\ \mathcal {D} _ {g} ^ {b} \sim \mathcal {N} \left(0, \sigma^ {2}\right), \quad \forall g \neq h \end{array} \right..
$$
On the one hand, notice that  $\mathcal{D}^a$  and  $\mathcal{D}^b$  only differ in their coordinate  $h$ , so that  $KL(\mathcal{D}_g^a ||\mathcal{D}_g^b) = \mathbb{1}(g = h)\Theta (GT^{-1})$ , and by minimality of  $h$ ,  $\mathbb{E}_{\pi ,\mathcal{D}^a}[n_{h,t}]\leq \frac{T}{G}$ , so that
$$
\sum_ {g \in [ G ]} \mathbb {E} _ {\boldsymbol {\pi}, \boldsymbol {\mathcal {D}} ^ {a}} \left[ n _ {g, t} \right] K L \left(\mathcal {D} _ {g} ^ {a} | | \mathcal {D} _ {g} ^ {b}\right) = \mathbb {E} _ {\boldsymbol {\pi}, \boldsymbol {\mathcal {D}} ^ {a}} \left[ n _ {h, t} \right] K L \left(\mathcal {D} _ {h} ^ {a} | | \mathcal {D} _ {h} ^ {b}\right) \leq \frac {T}{G} \Theta (G T ^ {- 1}) = \Theta (1).
$$
Second, we show that  $d(\pmb{\sigma}^a,\pmb{\sigma}^b) = \Theta (T^{-2})$ . We do this by measuring the distance between the sets  $S_{\epsilon}^{a}$  and  $S_{\epsilon}^{b}$  for a fixed  $\epsilon$ , and then estimate the smallest  $\epsilon$  for which this distance is 0. This smallest  $\epsilon$  corresponds to  $d(\pmb{\sigma}^a,\pmb{\sigma}^b)$ . Combining both in Lemma 6 yields  $\max \{\mathrm{Regret}_{p,T}(\pmb {\pi},\pmb {\mathcal{D}}^a),\mathrm{Regret}_{p,T}(\pmb {\pi},\pmb {\mathcal{D}}^b)\} \geq \Theta (T^{-2})$ .
Remark 1. For  $p = +\infty$ , the proof remains the same except that in this case  $d(\pmb{\sigma}^a, \pmb{\sigma}^b) = \Theta (T^{-3 / 2})$ , resulting in an overall lower bound of  $\Theta (T^{-3 / 2})$ . See Appendix C.3 for the complete proof.
6 Numerical study
In this section, we present experimental results on the empirical performance of Variance-UCB. We explore the impact of varying each important parameter of the problem: the time horizon  $T$ , norm parameter  $p$ , number of groups  $G$ , distributions  $\mathcal{D}_g$ , and the sub-Gaussianity parameters  $c_1, c_2$ . In all the experiments  $\mathcal{D}_g$  follow Gaussian distributions. Except where they are varied, the default parameter settings are  $T = 10^5$ ,  $p = 2$ ,  $G = 2$ , with the respective data distributions of groups 1 and 2 as  $\mathcal{N}(1,1)$  and  $\mathcal{N}(2,2.5)$ , satisfying  $c_1 = c_2 = 5$ . For each parameter setting evaluated, we run Variance-UCB 500 times and report average regret over all 500 runs. For convenience, time and regret are presented on logarithmic scales.
Figure 1: Impact of the  $p$ -norm on the regret.
First, we vary the choice of the parameter  $p \in \{1, 2, 10, 25, +\infty\}$  and observe its effect on the regret. Figure 1 shows that the convergence rates for each  $p$  precisely match those predicted by Theorems 1 and 3: a slope of -2 for the finite values of  $p$  ( $\{1, 2, 10, 25\}$ ), and a slope of -1.5 for  $p = \infty$ .
Figure 2: Impact of mis-estimating  $C_T$  on the regret
Next, we study how mis-specifying the sub-Gaussianity parameters  $c_{1}$  and  $c_{2}$  affect the regret of Algorithm 1. The parameter values  $(c_{1}, c_{2})$  only affect the algorithm's behavior through  $C_{T}$ , so it is more natural to directly study the impact of errors in  $C_{T}$ . For the parameters used in our experimental setup, the value of  $C_{T}$  prescribed in Equation (6) is  $\sim 5$ ; we consider both underestimating and overestimating  $C_{T}$ , and evaluate regret when instead plugging in values of  $C_{T} \in \{0.001, 5, 1000\}$  in the UCB update step defined in Equation (5). Figure 2 shows that choosing an overestimate of  $C_{T}$  incurs an increased regret. This is not surprising since choosing a larger  $C_{T}$  decreases the confidence of the algorithm in its estimates, and forces over-exploration. Choosing a low  $C_{T}$  also incurs a higher regret which can more severely impact the performance, as the algorithm under-explores and can get stuck in sub-optimal behavior. However, with a long enough time horizon, the algorithm will eventually estimate  $\sigma$  accurately, regardless of the small choice of  $C_{T}$ . As observed in Figure 2, the smallest value of  $C_{T} = 0.001$  initially has the highest regret, corresponding to incorrect estimates and insufficient exploration, and then finally converges very late to have the lowest regret. Even for very large  $T$ , this curve has higher variance due to the noise in the estimates.
Figure 3: The impact of  $\sigma_{\mathrm{min}}$  on the regret
Next, we study the effect of varying the lowest variance  $\sigma_{\mathrm{min}}^2 \in \{0.05, 0.1, 0.5, 1\}$  while holding all other variances fixed, for each  $p \in \{1, 2, +\infty\}$ . First, Figure 3 shows that varying  $\sigma_{\mathrm{min}}$  has no effect on the regret when  $p = +\infty$ . This matches a result of [13], where they prove that when  $p = +\infty$ , their UCB-style algorithm (very similar to Variance-UCB) is not affected by the lowest variance when the feedback is Gaussian. However, our experimental results show that this phenomenon does not persist when  $p$  is finite, as illustrated in Figure 3, where we observe that regret decreases when the lowest variance  $\sigma_{\mathrm{min}}$  increases. This is surprising since increasing the lowest variance makes the feedback more volatile, and one would expect an increase in regret as a result.
Figure 4: Impact of the number of groups  $G$  on regret
Finally, we vary the number of groups  $G \in \{2, 10, 50\}$ . For the additional groups, we generate their data from Gaussian  $\mathcal{D}_g$ , with means  $\mu_g \sim \mathcal{U}([-1, 1])$  and standard deviations  $\sigma_g \sim \mathcal{U}([2, 4])$  independently for each group. From Figure 4, we observe that the regret increases in the number of groups, as expected. When the number of groups is small ( $G = 2$ ), Variance-UCB quickly enters a regime where regret decreases quickly. However, as the number of groups grows ( $G \in \{10, 50\}$ ), the necessary time to enter the decay regime increases ( $\sim 30,000$  for  $G = 10$  and  $\sim 90,000$  for  $G = 50$ ). Initially the algorithm samples (on average) uniformly across all groups due to the UCB term outweighing the sample variance estimates, and each group must wait to be sampled enough times for the algorithm to estimate its optimal sampling rate. This delay will naturally increase with the number of groups. Once the confidence bounds are small enough, the algorithm samples the highest variances first. This causes abrupt variation (especially in the case where  $p$  is small) because the objective function is very sensitive to changes in one coordinate.
The authors gratefully acknowledge the support of National Science Foundation grants IIS-2147361. The authors would also like to thank the reviewers for their valuable feedback which helped improve the paper.
[34] Mitchell Nicholas Sarkies, K-A Bowles, EH Skinner, D Mitchell, Romi Haas, M Ho, Karen Salter, Kerry May, Donna Markham, L O'Brien, et al. Data collection methods in health services research. Applied clinical informatics, 6(01):96-109, 2015.
[35] Burr Settles. Active learning literature survey. 2009.
[36] Aleksandris Slivkins. Contextual bandits with similarity information. In Sham M. Kakade and Ulrike von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory, volume 19 of Proceedings of Machine Learning Research, pages 679-702, Budapest, Hungary, 09-11 Jun 2011. PMLR.
[37] Aleksandris Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning, 12(1-2):1-286, 2019.
[38] Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 27, 2014.
[39] Bin Yu. Assouad, Fano, and Le Cam, pages 423-435. Springer New York, New York, NY, 1997.
[40] Liang Yu, Shuqi Qin, Meng Zhang, Chao Shen, Tao Jiang, and Xiaohong Guan. A review of deep reinforcement learning for smart building energy management. IEEE Internet of Things Journal, 8(15):12046-12063, 2021.
The proof of Lemma 1 consists of using optimality conditions on  $r_p$ :
$$
R _ {p} ^ {*} (\boldsymbol {\sigma}) = R _ {p} (\boldsymbol {n} _ {T} ^ {*}) = \frac {1}{T} R _ {p} (\boldsymbol {n} _ {1} ^ {*}).
$$
Proof. The case of  $p = +\infty$  is discussed in [4, 13]. We focus on when  $p < +\infty$ . Recall the definition of  $R_p^*$  from Eq. (3):
$$
R _ {p} ^ {*} := \min  _ {\boldsymbol {n} \in \mathbb {R} _ {+} ^ {G}} R _ {p} (\boldsymbol {n}) \quad s. t. \quad \sum_ {g \in [ G ]} n _ {g} = T.
$$
Since the function  $x \to x^p$  is increasing in  $\mathbb{R}_+$ , we can replace the objective by  $[R_p(\boldsymbol{n})]^p = \sum_{g \in [G]} \frac{\sigma_g^{2p}}{n_g^p}$ . Any feasible point  $\boldsymbol{n}$  with a zero coordinate would have  $R_p(\boldsymbol{n}) = +\infty$ , therefore any argmin to the optimization program  $\boldsymbol{n}^*$  above must have positive coordinates, and satisfies the KKT conditions
$$
\begin{array}{l} \forall g \in [ G ], \quad \frac {\partial}{\partial n _ {g}} R _ {p} (\boldsymbol {n}) ^ {p} - \frac {\partial}{\partial n _ {g}} \lambda \left(\sum_ {h \in [ G ]} n _ {h} - T\right) = 0, \\ \lambda \in \mathbb {R}. \\ \end{array}
$$
For each  $g \in [G]$ , the first line of the system above is equivalent to  $-p\frac{\sigma_g^{2p}}{n_g^{p + 1}} - \lambda = 0$ . Therefore the KKT conditions imply that
$$
\frac {\sigma_ {1} ^ {2 p}}{n _ {1} ^ {p + 1}} = \ldots = \frac {\sigma_ {G} ^ {2 p}}{n _ {G} ^ {p + 1}}, \quad \text {o r e q u i v a l e n t l y}, \quad \frac {\sigma_ {1} ^ {\frac {2 p}{p + 1}}}{n _ {1}} = \ldots = \frac {\sigma_ {G} ^ {\frac {2 p}{p + 1}}}{n _ {G}} = \frac {\sum_ {g \in [ G ]} \sigma_ {g} ^ {\frac {2 p}{p + 1}}}{\sum_ {g \in [ G ]} n _ {g}} = \frac {\sum_ {g \in [ G ]} \sigma_ {g} ^ {\frac {2 p}{p + 1}}}{T}.
$$
Hence the unique minimizer is the vector
$$
\boldsymbol {n} _ {T} ^ {*} = \frac {T}{\sum_ {g \in [ G ]} \sigma_ {g} ^ {\frac {2 p}{p + 1}}} \boldsymbol {\sigma} ^ {\frac {2 p}{p + 1}}.
$$
Therefore,  $R_{p}^{*} = R_{p}(\pmb{n}_{T}^{*})$ . Moreover,  $R_{p}(\pmb{n}_{T}^{*}) = R_{p}(T\pmb{n}_{1}^{*}) = \frac{1}{T} R_{p}(\pmb{n}_{1}^{*})$ , where the last equality follows from the homogeneity of the norm.
A.2 Proof of Lemma 2
In this section, we establish a key property of  $\mathrm{UCB}_t(\sigma_g)$ , which relies on Assumption 1.
$$
0 \leq U C B _ {t} (\sigma_ {g}) ^ {\frac {2 p}{p + 1}} - \sigma_ {g} ^ {\frac {2 p}{p + 1}} \leq \frac {4 C _ {T}}{\sqrt {n _ {g , t}}} \frac {p}{p + 1} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , t}}}\right) ^ {\frac {p - 1}{p + 1}}.
$$
Proof. Following the notations of Section 3, we introduce the following event:
$$
\mathcal {A} _ {T} := \bigcap_ {g \in [ G ], 2 \leq t \leq T} \left\{\left| \hat {\sigma} _ {g, t} - \sigma_ {g} \right| \leq \frac {C _ {T}}{\sqrt {t}} \right\}.
$$
Based on corollary 1 of [13], we have
$$
\left(\mathrm {i}\right) \mathbb {P} _ {\pi} \left(\mathcal {A} _ {T}\right) \geq 1 - 2 G T ^ {- 2. 5},
$$
(ii) conditionally on  $\mathcal{A}_T$
$$
\forall g \in [ G ], t \geq 2 G, | \hat {\sigma} _ {g, t} - \sigma_ {g} | \leq \frac {C _ {T}}{\sqrt {n _ {g , t}}},
$$
so that conditionally on  $\mathcal{A}_T$ , we have:
$$
\begin{array}{l} \mathbf {U C B} _ {t} (\sigma_ {g}) = \hat {\sigma} _ {g, t} + \frac {C _ {T}}{\sqrt {n _ {g , t}}} \\ = \sigma_ {g} + \frac {C _ {T}}{\sqrt {n _ {g , t}}} + (\hat {\sigma} _ {g, t} - \sigma_ {g}) \\ \in \sigma_ {g} + \frac {C _ {T}}{\sqrt {n _ {g , t}}} + \left[ - \frac {C _ {T}}{\sqrt {n _ {g , t}}}, \frac {C _ {T}}{\sqrt {n _ {g , t}}} \right] \\ = \sigma_ {g} + \left[ 0, \frac {2 C _ {T}}{\sqrt {n _ {g , t}}} \right], \\ \end{array}
$$
where the first Eq. is due to the definition of  $\mathrm{UCB}_t$  introduced in (5), and the bounding is due to the definition of  $\mathcal{A}_T$ . Next, notice that the function  $x \to (\sigma_g + x)^{\frac{2p}{p+1}}$  is increasing, which implies that
$$
\sigma_ {g} ^ {\frac {2 p}{p + 1}} \leq \mathrm {U C B} _ {t} \left(\sigma_ {g}\right) ^ {\frac {2 p}{p + 1}} \leq \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , t}}}\right) ^ {\frac {2 p}{p + 1}}, \tag {8}
$$
which proves the leftmost inequality in Lemma 2. To prove the rightmost inequality, notice that  $x \to (\sigma_g + x)^{\frac{2p}{p+1}}$  is also convex, therefore by Jensen's inequality,
$$
\left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , t}}}\right) ^ {\frac {2 p}{p + 1}} - \sigma_ {g} ^ {\frac {2 p}{p + 1}} \leq \frac {2 C _ {T}}{\sqrt {n _ {g , t}}} \frac {2 p}{p + 1} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , t}}}\right) ^ {\frac {2 p}{p + 1} - 1} = \frac {4 C _ {T}}{\sqrt {n _ {g , t}}} \frac {p}{p + 1} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , t}}}\right) ^ {\frac {p - 1}{p + 1}},
$$
A.3 Proof of Lemma 3
The goal of this section is to prove Lemma 3, which consists of bounding with high probability  $\pmb{n} - \pmb{n}^*$ . To do so, we will design an alternative sequence  $\tilde{\pmb{n}}$ , that is simultaneously easy to analyze, and upper bounds  $\pmb{n}$  with high probability. The motivation for the choice of  $\tilde{\pmb{n}}$  comes from the relaxation of the choice made at every time step by Variance-UCB.
We assume through the whole section what the event  $\mathcal{A}_T$  is realized. For convenience, we view the right hand side of Lemma 2 as a quantity of its own, and introduce the width function
$$
w _ {g}: x > 0 \to w _ {g} (x) := \frac {4 C _ {T}}{\Sigma_ {p} \sqrt {x}} \frac {p}{p + 1} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {x}}\right) ^ {\frac {p - 1}{p + 1}},
$$
so that the inequality in Lemma 2 can be rewritten as
$$
0 \leq \mathbf {U C B} _ {t} \left(\sigma_ {g}\right) ^ {\frac {2 p}{p + 1}} - \sigma_ {g} ^ {\frac {2 p}{p + 1}} \leq \Sigma_ {p} w _ {g} (n _ {g, t}). \tag {9}
$$
We start by proving the following lemma, which follows from the decisions Variance-UCB makes:
Lemma 7. For  $t \geq 2G$ , we have:
$$
n _ {X _ {t + 1}, t} - t w _ {X _ {t + 1}} \left(n _ {X _ {t + 1}, t}\right) \leq n _ {X _ {t + 1}, t} ^ {*}
$$
Proof. The proof exploits the greedy property of the algorithm. A group needs to be sampled exactly twice to have a finite UCB. Therefore, for  $t = 1, \ldots, 2G$ , Variance-UCB samples every group twice (in an arbitrary order), so that at  $n_{1,2G} = \ldots = n_{G,2G} = 2$  and  $\mathrm{UCB}_{2G}(\sigma_1)^{\frac{2p}{p+1}}, \ldots, \mathrm{UCB}_{2G}(\sigma_G)^{\frac{2p}{p+1}} < +\infty$ . By choice of  $X_{t+1}$ , the following inequality holds: