SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abdellah Aznag Rachel Cummings Adam N. Elmachtoub
Department of Industrial Engineering and Operational Research
{aa4683, rac2239, ae2516} @columbia.edu
We consider a problem with multiple groups whose data distributions are unknown, and an analyst would like to learn the mean of each group. We consider an active learning framework to sequentially collect  $T$  samples with bandit feedback, each period observing a sample from a chosen group. After observing a sample, the analyst may update their estimate of the mean and variance of that group and choose the next group accordingly. The analyst's objective is to dynamically collect samples to minimize the  $p$ -norm of the vector of variances of the mean estimators after  $T$  rounds. We propose an algorithm, Variance-UCB, that selects groups according to an upper bound on the variance estimate adjusted to the  $p$ -norm chosen. We show that the regret of Variance-UCB is  $\tilde{O}(T^{-2})$  for finite  $p$ , and prove that no algorithm can do better. When  $p$  is infinite, we recover the  $\tilde{O}(T^{-1.5})$  regret bound obtained in [4, 13] with improved dependence on the remaining parameters and provide a new lower bound showing that no algorithm can do better.
Obtaining accurate estimates from limited labeled data is a fundamental challenge. To tackle this issue, active learning has emerged as a promising solution framework where a decision-maker strategically selects one sample at a time [35, 15, 22, 40, 24]. The estimation task becomes more complex when faced with a large population of different groups, where it is important that all groups are represented in the estimation procedure. If this allocation is not done correctly, the decision might incur structural bias against some groups [32, 21, 33, 29]. If allowed to adjust the sampling allocation strategy, the analyst can address their biases and re-allocate their sampling resources. A key challenge is to dynamically collect data from different groups while maintaining a reasonable representation of all the groups, which is the problem we tackle in this paper. We provide an active learning framework for dynamic data collection with bandit feedback for estimating the means of multiple groups in a representative manner.
We consider a population partitioned into multiple groups, each with data points drawn from an unknown data distribution, and an analyst would like to learn the mean of each group. Each group's distribution has an unknown mean and unknown variance. At each time period, the analyst collects one sample from a group of their choice and observes a sample from that group. Since only one group is observed, this is also known as bandit feedback. Upon observation, the analyst exploits their new knowledge to optimize their choice of the next group to observe. At the end of the time horizon  $T$ , the analyst forms a mean estimate for each group. The objective of the analyst is to return a vector of mean estimates with smallest  $p$ -norm of the variance vector of the mean estimates. The choice of  $p$ -norm is motivated by its ability to capture different aspects of multi-group estimation performance [10, 23, 28].
By considering various values of  $p$ , we gain insights into different dimensions of estimation accuracy, ranging from the overall spread of the estimates ( $p = 2$ , Euclidean norm) to the worst case deviation from the true mean ( $p = +\infty$ , infinity norm), a case that is studied in [4, 13]. This approach allows us to assess the estimation quality from multiple perspectives, providing a more nuanced understanding of how the diverse population is represented in the final estimation of the group means. The analyst can choose  $p$  in a way to capture notions of fairness in the data collection process.
In this work, we first present an active learning framework that captures the trade-off between representation and accuracy in the task of multi-group estimation. Our framework encapsulates a large class of norms ( $p$ -norms), which bring various insights into different dimensions of estimation accuracy. In Section 3, we present Variance-UCB, an algorithm that selects groups according to an upper bound on the variance estimate adjusted to the  $p$ -norm.
We provide a norm-dependent regret bound on Variance-UCB under the two assumptions sub-Gaussian feedback and positive variances. Specifically, we show that the regret is  $\tilde{O}(T^{-2})$  for finite  $p$  norms (Theorem 1). For the case of the infinite norm, we show that the regret is  $\tilde{O}(T^{-1.5})$  (Theorem 3). While bounds at this rate are already established in [4, 13], our bound provides tighter dependencies in the problem parameters  $G$  and  $\sigma_{\min}$ . In the case of the infinite norm, we improve the coefficient of the  $T^{-1.5}$  term in the regret from  $\sigma_{\min}^{-2} G^{2.5}$  to  $G^{1.5}$ . This improvement not only tightens the dependency in the number of groups, but it also partially solves one of the open questions left in [13] regarding the role of  $\sigma_{\min}$  in the regret bound. While we prove that  $\sigma_{\min}$  can still impact the regret, it appears only in lower order terms with respect to  $T$ .
The analysis and proofs of these upper bounds is presented in Section 4. Our proof technique differs from proofs of similar upper bounds in the related literature. Instead of studying the regret function directly, we first consider its Taylor expansion, and focus our analysis on its dominant term. This technique has two advantages. First, the dominant term is much simpler to analyze (linear in the case of the infinite norm, and quadratic in the case of finite  $p$  norms) than the full regret function. Second, the resulting upper bound is tighter in its dominant term, in that it does not suffer from large numerical constants, as is common in the existing literature on regret analysis of bandit algorithms (e.g., [4, 13]).
We also provide new matching lower bounds in  $T$  for both finite (Theorem 2) and infinite  $p$ -norms (Theorem 4), establishing that Variance-UCB achieves the best possible regret in both regimes. These bounds are summarized in Table 1. Prior to our work and to the best of our knowledge, no lower bounds were known for this problem. The analysis of these results is presented in Section 5.
Table 1: Summary of main results
Norm	Variance-UCB Regret	Lower Bound on Regret for Any Policy
p<∞	O(T-2)(Theorem 1)	Ω(T-2)(Theorem 2)
p=∞	O(T-1.5)(Theorem 3)	Ω(T-1.5)(Theorem 4)
We empirically validate our findings by numerical experiments presented in Section 6, which show that our theoretical regret bounds match empirical convergence rates in both cases of finite and infinite  $p$ -norms. We also provide examples showing that for finite  $p$ -norms, the smallest variance affects the regret, even when the feedback is Gaussian. This is in contrast to the case of the infinite norm, where it is proven [13] that under Gaussian feedback, the algorithm is not affected by the smallest variance.
Our motivation stems from growing attention to data collection methods [34, 2, 18, 20]. We focus on the problem of mean estimation and dynamically collecting data to achieve this goal. While there is a substantial body of literature on data acquisition [9, 26], specifically in the presence of privacy concerns and associated costs [19, 30, 31, 12, 17, 16], our approach differs as we do not consider the costs of sharing data. Instead, we concentrate on the data collection process itself, rather than the costs to data providers.
Our work is related to multi-armed bandits problems [5, 37], in the sense that each group can be seen as an arm, and choosing a group to sample from at each time step corresponds to choosing which arm to pull. However, the performance criterion for multi-armed bandits is measured by the difference between the mean of the chosen arm and the best arm [3, 14, 38]. In our framework, the means of the chosen arms do not impact the performance. It is their variances that matter in the optimal solution, as we measure the performance by considering the  $p$ -norm of the variance of the estimator, which can be non-convex. Because of this, and to the best of our knowledge, usual bandits algorithms and proof techniques [25] do not apply. Instead, we propose Variance-UCB, an algorithm that uses high probability upper bounds on the variance estimates adjusted to the chosen norm.
Considering data acquisition from the perspective of active learning is a natural approach [6, 7, 27], and in that sense our work is also related to active learning [15, 22, 24, 40]. In [1], the authors address the optimal data acquisition problem under the assumption of additive objective functions. They formulate the problem as an online learning problem and leverage well-understood tools from online convex optimization [8, 11, 36]. However, their ideas do not apply to our setting due to the non-additive nature of our decisions over time.
The case where the chosen norm is  $\| \cdot \|_{\infty}$  is introduced in [4], where the authors devise the GAFS-MAX algorithm and show that for bounded feedback with known upper bounds, it achieves regret  $\tilde{O}(T^{-1.5})$ . This case is also studied in [13], where the authors extend the feedback to sub-Gaussian, and devise the B-AS algorithm, which also has  $\tilde{O}(T^{-1.5})$  regret. One property that emerges in their study is a regret bound deteriorates with  $\sigma_{\min}^{-1}$ , where  $\sigma_{\min}^2$  is the smallest variance across groups. This is counter-intuitive, as smaller variances in general make the learning simpler. The authors pose as an open question whether any algorithm can have performance independent of  $\sigma_{\min}$  and show that in the special case of exactly Gaussian feedback, one can derive a  $\sigma_{\min}$  free bound. We partially answer this question, by showing that the leading term of the regret bound (with respect to  $T$ ) does not depend on  $\sigma_{\min}$ . In other words, while  $\sigma_{\min}$  can still impact the regret, its effect appears only in lower order terms. Our regret analysis of Variance-UCB for  $\| \cdot \|_{\infty}$  recovers the same dependence in  $T$  as in [13], but also has improved dependence in the number of groups  $G$  and the variance vector  $\sigma$ . While [4, 13] serve as a starting point, most of their findings are only applicable to the special case  $p = +\infty$ . In particular, they cannot be expanded to other choices of norm. We show that for finite  $p$  norms, we prove the regret bound is  $\tilde{O}(T^{-2})$ , which is fundamentally different than the previous  $\tilde{O}(T^{-1.5})$  bound.
2 Model
We consider a population partitioned into  $G$  disjoint groups. Each group is represented by an index  $g$  from  $[G] := \{1, \dots, G\}$ . Each individual in the population holds a real-valued data point; data from each group  $g \in [G]$  are distributed according to an unknown distribution  $\mathcal{D}_g$ , with unknown mean  $\mu_g$  and unknown variance  $\sigma_g^2$ . We denote  $\sigma_{\min} := \min_{g \in [G]} \sigma_g$  and let  $\mathcal{D} := \mathcal{D}_1 \times \dots \times \mathcal{D}_G$ . The analyst wishes to compute an unbiased estimate of the population mean for each group over  $T$  times of data collection, sampling only one group at a time. The set of feasible policies is defined as
$$
\Pi := \left\{\boldsymbol {\pi} = \left\{\pi_ {t} \right\} _ {t \in [ T ]} \mid \pi_ {t} \in G ^ {t - 1} \times \mathbb {R} ^ {t - 1} \rightarrow \Delta (G), \forall t \in [ T ] \right\},
$$
where  $\Delta (G)$  is the set of measures supported on  $[G]$ . Let  $n_{g,T}$  denote the number of collected samples from group  $g$  at the end of time  $T$ , and let  $\hat{\mu}_{g,T}$  be the sample mean estimator of  $\mu_g$  for  $n_{g,T}$  collected samples. Once all data have been collected at the end of the time horizon  $T$ , the analyst will compute the sample mean of each group, i.e.,
$$
\hat {\mu} _ {g, T} = \frac {1}{\sum_ {t = 1} ^ {T} \mathbb {1} _ {X _ {t} = g}} \sum_ {t = 1} ^ {T} \mathbb {1} _ {X _ {t} = g} Y _ {t}.
$$
Note that the vector  $\hat{\mu}_T$  is always an unbiased estimator of the vector  $\pmb{\mu}$ , as long as the policy  $\pi$  samples at least once from each group.
The variance of  $\hat{\mu}_{g,T}$  is  $\frac{\sigma_g^2}{n_{g,T}}$ . The  $p$ -norm of the vector of variances of  $n_{g,T}$  is denoted as
$$
R _ {p} \left(\boldsymbol {n} _ {T}\right) := \left\| \left\{\frac {\sigma_ {g} ^ {2}}{n _ {g , T}} \right\} _ {g = 1} ^ {G} \right\| _ {p}. \tag {1}
$$
The analyst wishes to minimize  $\mathbb{E}[R_p(\pmb{n}_T)]$ .
When choosing a policy, the analyst does not have access to the true standard deviation vector  $\sigma := (\sigma_{1},\dots,\sigma_{G})$ , which is needed to compute the value  $R_{p}(\boldsymbol{n}_{T})$ . Therefore the analyst must learn  $\sigma$  through their decisions. We benchmark the performance of a policy against the best possible performance in a complete information setting where  $\sigma$  is known, that is,
$$
\min  _ {\boldsymbol {n} \in \mathbb {N} ^ {G}} R _ {p} (\boldsymbol {n}) \quad s. t. \quad \sum_ {g \in [ G ]} n _ {g} = T. \tag {2}
$$
The optimization program (2) can be difficult to solve and analyze due to the integer constraints. Instead of using the solution to (2) as a benchmark, we use the solution to its continuous relaxation (a lower bound on (2)), which we denote as,
$$
R _ {p} ^ {*} := \min  _ {\boldsymbol {n} \in \mathbb {R} _ {+} ^ {G}} R _ {p} (\boldsymbol {n}) \quad s. t. \quad \sum_ {g \in [ G ]} n _ {g} = T. \tag {3}
$$
Thus, we can define the regret of a policy as
$$
\operatorname {R e g r e t} _ {p, T} (\boldsymbol {\pi}, \boldsymbol {\mathcal {D}}) := \mathbb {E} _ {\boldsymbol {\pi}} \left[ R _ {p} \left(\boldsymbol {n} _ {T}\right) \right] - R _ {p} ^ {*}. \tag {4}
$$
For our analysis, we assume that the distributions  $\mathcal{D}_g$  are sub-Gaussian, as stated in Assumption 1, with corresponding constants  $c_{1}, c_{2}$  known to the analyst.
Assumption 1 (Sub-Gaussianity). For each  $g \in [G]$ ,  $\mathcal{D}_g$  is sub-Gaussian. That is, there exist universal constants  $c_1, c_2 > 0$  such that for any  $\epsilon > 0$ ,
$$
\mathbb {P} _ {Y \sim \mathcal {D} _ {g}} \left(\left| Y - \mu_ {g} \right| \geq \epsilon\right) \leq c _ {1} \exp \left(- \epsilon^ {2} / c _ {2}\right).
$$
We assume that such  $c_{1}, c_{2}$  are known to the analyst.
Moreover, we assume that all groups have some variation in their data, as stated in Assumption 2.
Assumption 2 (Positive Variance). The minimum group variance is positive; i.e.,  $\sigma_{\min} = \min_{g \in [G]} \sigma_g > 0$ .
3 Our algorithm: Variance-UCB
Our algorithm, Variance-UCB, builds an increasingly accurate upper confidence bound for each  $\sigma_g$ , which we denote  $\mathrm{UCB}_t(\sigma_g)$ . Recall that  $n_{g,t}$  denotes the number of collected samples from group  $g$  through time  $t$ . At each time  $t$ ,  $\sigma_g$  can be estimated via the sample standard deviation
$$
\hat {\sigma} _ {g, t} := \sqrt {\frac {1}{n _ {g , t} - 1} \sum_ {s \leq t : X _ {s} = g} \left(Y _ {s} - \hat {\mu} _ {g , t}\right) ^ {2}}.
$$
For convention, we set  $\hat{\sigma}_{g,0} = \hat{\sigma}_{g,1} = +\infty$ , indicating that at least two samples are required to obtain a meaningful estimate of  $\sigma_{g}$ . We can then define
$$
\mathrm {U C B} _ {t} \left(\sigma_ {g}\right) := \hat {\sigma} _ {g, t} + \frac {C _ {T}}{\sqrt {n _ {g , t}}}, \tag {5}
$$
$$
C _ {T} := 2 \sqrt {2 \log (2 T ^ {4}) c _ {1} \log (c _ {2} T ^ {4})} + \frac {2 \sqrt {c _ {1} \log (2 T ^ {4}) (1 + c _ {2} + \log (c _ {2} T ^ {4}))}}{(1 - T ^ {- 4}) \sqrt {2 \log (2 T ^ {4})}} T ^ {- 2}. \tag {6}
$$
$C_T$  was introduced in [13], and captures the trade-off between accuracy of the upper confidence bound and confidence in the estimate, and is a polylogarithmic factor in  $T$ . In particular,  $C_T$  can be constructed by the analyst since it depends only on  $c_1, c_2$ , and  $T$ , which are known.
At time  $t + 1$ , the algorithm chooses the next group  $X_{t + 1}$  using the rule:
$$
X _ {t + 1} = \arg \max _ {g \in [ G ]} \frac {\mathrm {U C B} _ {t} (\sigma_ {g}) ^ {\frac {2 p}{p + 1}}}{n _ {g , t}},
$$
Algorithm 1 Variance-UCB  $(p,T,G,c_1,c_2)$
Input: norm parameter  $p$ , time horizon  $T$ , number of groups  $G$  and subgaussian parameters  $c_{1}, c_{2}$ .  
1: Initialize  $n_{g,0} = 0$ ,  $\hat{\sigma}_{g,0} = \hat{\sigma}_{g,1} = +\infty$ ,  $\forall g \in [G]$   
2: Compute  $C_T$  according to (6).  
3: for  $t = 0, \ldots, T - 1$  do  
4: Compute  $\mathrm{UCB}_t(\sigma_g)$ :  $\mathrm{UCB}_t(\sigma_g) = \hat{\sigma}_{g,t} + \frac{C_T}{\sqrt{n_{g,t}}}$ ,  $\forall g \in [G]$   
5: Select group  $X_{t+1} = \operatorname{argmax}_g \frac{\mathrm{UCB}_t(\sigma_g)^{\frac{2p}{p+1}}}{n_{g,t}}$   
6: Observe feedback  $Y_{t+1} \sim \mathcal{D}_{X_{t+1}}$   
7: Update the number of samples:  $n_{g,t+1} = n_{g,t} + \mathbb{I}_{X_{t+1}=g}$ ,  $\forall g \in [G]$   
8: Update the mean estimates:  $\hat{\mu}_{g,t+1} = \frac{1}{n_{g,t+1}} \sum_{s=1}^{t+1} \mathbb{1}_{X_s=g} Y_s$ ,  $\forall g \in [G]$   
9: Update the standard deviation estimates:  $\hat{\sigma}_{g,t+1} := \sqrt{\frac{1}{n_{g,t+1}-1} \sum_{s \leq t+1: X_s=g} (Y_s - \hat{\mu}_{g,t+1})^2}$ ,  $\forall g \in [G]$   
10: end for  
Output:  $\hat{\mu}_{g,T} = \frac{1}{n_{g,T}} \sum_{s=1}^{T} \mathbb{1}_{X_s=g} Y_s$ ,  $\forall g \in [G]$
where where ties in the argmax can be broken arbitrarily. The analyst then observes a new sample  $Y_{t + 1}$  from the chosen group  $X_{t + 1}$  and updates the upper confidence bounds accordingly. Variance-UCB is presented formally in Algorithm 1.
Variance-UCB takes as input the time horizon  $T$ , the norm  $p$ , the groups  $[G]$ , and the sub-Gaussian parameters  $c_{1}, c_{2}$ . The algorithm initializes the upper confidence bound for every group to be infinity, which reflects the absence of knowledge of  $\{\sigma_g\}$ .
In the special case  $p = +\infty$ , Variance-UCB (instantiated with a different choice of  $C_T$ ) coincides with the B-AS algorithm of [13].<sup>2</sup>
3.1 Regret guarantees
Our first main result gives theoretical bounds on the performance of Algorithm 1 for finite  $p \in [1, \infty)$ . We show in Theorem 1 that when  $p$  is finite, Variance-UCB incurs regret of  $\tilde{O}(T^{-2})$ .
Theorem 1. For any  $\mathcal{D}$  that satisfies Assumptions 1 and 2 and for any finite  $p$ , the regret of Variance-UCB is at most  $\tilde{O}(T^{-2})$ . That is,
$$
\operatorname {R e g r e t} _ {p, T} \left(\operatorname {V a r i a n c e - U C B}, \boldsymbol {\mathcal {D}}\right) = \tilde {O} \left(T ^ {- 2}\right).
$$
Our second main result, Theorem 2, provides a matching lower bound for our problem. Thus showing that the performance of Variance-UCB and its analysis is the best possible in terms of  $T$ .
Theorem 2. Let  $p$  be finite and  $\kappa$  be a universal constant. For any online policy  $\pi$ , there exists an instance  $D_{\pi}$  such that for any  $T \geq 1$ ,
$$
R e g r e t _ {p, T} (\boldsymbol {\pi}, \boldsymbol {\mathcal {D}} _ {\boldsymbol {\pi}}) \geq \kappa (p + 1) T ^ {- 2} + O \left(T ^ {- 2. 5}\right) = \Theta (T ^ {- 2}).
$$
Our analysis and proof techniques can be extended naturally to the case where  $p = +\infty$ . However, the  $\tilde{O}(T^{-2})$  regret no longer holds, and the convergence rate jumps to  $\tilde{O}(T^{-3/2})$ .
Theorem 3. Let  $\Sigma_{\infty} := \sum_{g \in [G]} \sigma_g^2$ . For any  $\mathcal{D}$  that satisfies Assumptions 1 and 2,
$$
\operatorname {R e g r e t} _ {\infty , T} (\operatorname {V a r i a n c e - U C B}, \boldsymbol {\mathcal {D}}) \leq \left(C _ {T} \sqrt {\Sigma_ {\infty}} + C _ {T} ^ {2}\right) G ^ {1. 5} T ^ {- 1. 5} + o (T ^ {- 1. 5}) = \tilde {O} (T ^ {- 1. 5}).
$$
Note that a similar bound for Theorem 3 has already been established in [13]:
$$
\operatorname {R e g r e t} _ {\infty , T} (\mathrm {B - A S}, \mathcal {D}) \leq \frac {2 8 2 3 0 (c _ {1} ((c _ {2} + 2) ^ {2} + 1) \log^ {2} (T) \Sigma_ {\infty} G ^ {2 . 5}}{\sigma_ {\min } ^ {2} T ^ {1 . 5}} + o (T ^ {- 1. 5}).
$$
<sup>1</sup>In Section 6, we empirically evaluate the impact of mis-estimating  $c_{1}$  and  $c_{2}$ .
2Our results still hold under this modified parameter setting for any  $C_T$  satisfying Lemma 2.
Our result in Theorem 3 improves the existing regret in all the problem parameters, i.e.,  $\Sigma_{\infty}$  to  $\sqrt{\Sigma_{\infty}}$  and from  $G^{2.5}$  to  $G^{1.5}$ . The most significant improvement lies in removing the dependence on  $\sigma_{\mathrm{min}}$  in the main term of the regret. While  $\sigma_{\mathrm{min}}$  still appears in the negligible term  $o(T^{-1.5})$ , this term will be asymptotically dominated for even moderate  $T$ . This improved bound gives a better understanding on how  $\sigma_{\mathrm{min}}$  impacts the performance of the algorithm, which was left as an open question in [13].
Finally, we give a matching lower bound for the case when  $p = \infty$ , showing that the analysis of Variance-UCB is tight in  $T$  when  $p = +\infty$ . To the best of our knowledge, no lower bound for this problem was previously known.
Theorem 4. For any online policy  $\pi$ , there exists an instance  $\mathcal{D}_{\pi}$  such that for any  $T \geq 1$ ,
$$
R e g r e t _ {\infty , T} (\boldsymbol {\pi}, \boldsymbol {\mathcal {D}} _ {\boldsymbol {\pi}}) \geq \frac {1}{2} G ^ {1. 5} T ^ {- 1. 5}.
$$
In Sections 4 and 5, we give outlines of the proofs of Theorems 1 and 2, respectively. While we briefly mention why both results change at  $p = +\infty$ , the full proofs for Theorems 3 and 4 is deferred to Appendix C.
4 Deriving the upper bounds
In this section, we give an overview of the main steps to prove Theorem 1. A complete proof is given in Appendix A. We first show in Lemma 1 that the optimal  $R_{p}^{*}(\sigma)$  is achieved for an optimal static policy  $\boldsymbol{n}_{T}^{*}(p) = \{n_{g,T}^{*}(p)\}_{g\in [G]}$  that assumes knowledge of  $\sigma$  and samples each group  $g$  proportionally to  $\sigma_{g}^{\frac{2p}{p + 1}}$ .
Lemma 1. [Benchmark analysis] For each  $t \in \mathbb{N}^*$  and  $p \in [1, +\infty]$ , let  $n_{g,t}^* = \frac{\sigma_g^{\frac{2p}{p+1}} t}{\sum_{h \in [G]} \sigma_h^{\frac{2p}{p+1}}}$ . Then,
$$
R _ {p} ^ {*} (\boldsymbol {\sigma}) = R _ {p} \left(\boldsymbol {n} _ {T} ^ {*}\right) = \frac {1}{T} R _ {p} \left(\boldsymbol {n} _ {1} ^ {*}\right).
$$
The proof of Lemma 1 utilizes the KKT conditions of the optimization program (3) that defines  $R_{p}^{*}$ . For convenience, we introduce  $\Sigma_{p} := \sum_{g \in [G]} \sigma_{g}^{\frac{2p}{p+1}}$  for each  $p \in [1, +\infty]$  so that we can simplify  $n_{g,t}^{*} = \frac{t}{\Sigma_{p}} \sigma_{g}^{\frac{2p}{p+1}}$ . Using Equation (4), the expression of regret can be simplified to
$$
\operatorname {R e g r e t} _ {p, T} (\boldsymbol {\pi}, \boldsymbol {\mathcal {D}}) = \mathbb {E} _ {\boldsymbol {\pi}} \left[ R _ {p} \left(\boldsymbol {n} _ {T}\right) - R _ {p} \left(\boldsymbol {n} _ {T} ^ {*}\right) \right]. \tag {7}
$$
From Eq. (7), we can understand the behavior of the regret by answering the following questions.
1. How close is the random variable  $\pmb{n}_T$  to the optimal sampling vector  $\pmb{n}_T^*$ ?
2. How does the curvature of  $R_{p}(.)$  affect the difference  $R_{p}(\boldsymbol{n}_{T}) - R_{p}(\boldsymbol{n}_{T}^{*})$ ?
Upper bounding the error  $n_T - n_T^*$ : Before we answer the first question, we show in Lemma 2 that  $\mathrm{UCB}_t(\sigma_g)$  is an increasingly accurate upper bound for  $\sigma_g$ .
Lemma 2. For all  $g \in [G]$ , with probability at least  $1 - \tilde{O}(T^{-2})$
$$
0 \leq U C B _ {t} \left(\sigma_ {g}\right) ^ {\frac {2 p}{p + 1}} - \sigma_ {g} ^ {\frac {2 p}{p + 1}} \leq \frac {4 C _ {T}}{\sqrt {n _ {g , t}}} \frac {p}{p + 1} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , t}}}\right) ^ {\frac {p - 1}{p + 1}}.
$$
The proof of Lemma 2 utilizes Assumption 1 and the choice of  $\mathrm{UCB}_t(\sigma_g)^4$ , as defined in Eq. (5).
Using Lemma 2, the difference between the number of samples Variance-UCB collects and the optimal number of samples can be bounded with high probability, which we state in Lemma 3.