Title: An active learning framework for multi-group mean estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Active learning

mean estimation

bandit feedback

data acquisition
Abstract: We consider a fundamental problem where there are multiple groups whose data distributions are unknown, and an analyst would like to learn the mean of each group. We consider an active learning framework to sequentially collect $T$ samples with bandit, each period observing a sample from a chosen group. After observing a sample, the analyst may update their estimate of the mean and variance of that group and choose the next group accordingly.  The objective is to dynamically collect samples to minimize the $p$-norm of the vector of variances of our mean estimators after $T$ rounds. We propose an algorithm, Variance-UCB, that selects groups according to a an upper bound on the variance estimate adjusted to the $p$-norm chosen. We show that the regret of Variance-UCB is $O(T^{-2})$ for finite $p$, and prove that no algorithm can do better. When $p$ is infinite, we recover the $O(T^{-1.5})$ obtained in \cite{activelearning, carpentier2011upper} and provide a new lower bound showing that no algorithm can do better.

An active learning framework for multi-group mean estimation

Abstract

We consider a problem with multiple groups whose data distributions are unknown, and an analyst would like to learn the mean of each group. We consider an active learning framework to sequentially collect T samples with bandit feedback, each period observing a sample from a chosen group. After observing a sample, the analyst may update their estimate of the mean and variance of that group and choose the next group accordingly. The analyst's objective is to dynamically collect samples to minimize the p -norm of the vector of variances of the mean estimators after T rounds. We propose an algorithm, Variance-UCB, that selects groups according to a an upper bound on the variance estimate adjusted to the p -norm chosen. We show that the regret of Variance-UCB is ˜ O ( T -2 ) for finite p , and prove that no algorithm can do better. When p is infinite, we recover the ˜ O ( T -1 . 5 ) regret bound obtained in [4, 13] with improved dependence on the remaining parameters and provide a new lower bound showing that no algorithm can do better.

1 Introduction

Obtaining accurate estimates from limited labeled data is a fundamental challenge. To tackle this issue, active learning has emerged as a promising solution framework where a decision-maker strategically selects one sample at a time [35, 15, 22, 40, 24]. The estimation task becomes more complex when faced with a large population of different groups, where it is important that all groups are represented in the estimation procedure. If this allocation is not done correctly, the decision might incur structural bias against some groups [32, 21, 33, 29]. If allowed to adjust the sampling allocation strategy, the analyst can address their biases and re-allocate their sampling resources. A key challenge is to dynamically collect data from different groups while maintaining a reasonable representation of all the groups, which is the problem we tackle in this paper. We provide an active learning framework for dynamic data collection with bandit feedback for estimating the means of multiple groups in a representative manner.

We consider a population partitioned into multiple groups, each with data points drawn from an unknown data distribution, and an analyst would like to learn the mean of each group. Each group's distribution has an unknown mean and unknown variance. At each time period, the analyst collects one sample from a group of their choice and observes a sample from that group. Since only one group is observed, this is also known as bandit feedback. Upon observation, the analyst exploits their new knowledge to optimize their choice of the next group to observe. At the end of the time horizon T , the analyst forms a mean estimate for each group. The objective of the analyst is to return a vector of mean estimates with smallest p -norm of the variance vector of the mean estimates. The choice of p -norm is motivated by its ability to capture different aspects of multi-group estimation performance [10, 23, 28].

By considering various values of p , we gain insights into different dimensions of estimation accuracy, ranging from the overall spread of the estimates ( p = 2 , Euclidean norm) to the worst case deviation from the true mean ( p = + ∞ , infinity norm), a case that is studied in [4, 13]. This approach allows us to assess the estimation quality from multiple perspectives, providing a more nuanced understanding of how the diverse population is represented in the final estimation of the group means. The analyst can choose p in a way to capture notions of fairness in the data collection process.

1.1 Summary of contributions

In this work, we first present an active learning framework that captures the trade-off between representation and accuracy in the task of multi-group estimation. Our framework encapsulates a large class of norms ( p -norms), which bring various insights into different dimensions of estimation accuracy. In Section 3, we present Variance-UCB, an algorithm that selects groups according to an upper bound on the variance estimate adjusted to the p -norm.

We provide a norm-dependent regret bound on Variance-UCB under the two assumptions subGaussian feedback and positive variances. Specifically, we show that the regret is ˜ O ( T -2 ) for finite p norms (Theorem 1). For the case of the infinite norm, we show that the regret is ˜ O ( T -1 . 5 ) (Theorem 3). While bounds at this rate are already established in [4, 13], our bound provides tighter dependencies in the problem parameters G and σ min . In the case of the infinite norm, we improve the coefficient of the T -1 . 5 term in the regret from σ -2 min G 2 . 5 to G 1 . 5 . This improvement not only tightens the dependency in the number of groups, but it also partially solves one of the open questions left in [13] regarding the role of σ min in the regret bound. While we prove that σ min can still impact the regret, it appears only in lower order terms with respect to T .

The analysis and proofs of these upper bounds is presented in Section 4. Our proof technique differs from proofs of similar upper bounds in the related literature. Instead of studying the regret function directly, we first consider its Taylor expansion, and focus our analysis on its dominant term. This technique has two advantages. First, the dominant term is much simpler to analyze (linear in the case of the infinite norm, and quadratic in the case of finite p norms) than the full regret function. Second, the resulting upper bound is tighter in its dominant term, in that it does not suffer from large numerical constants, as is common in the existing literature on regret analysis of bandit algorithms (e.g., [4, 13]).

We also provide new matching lower bounds in T for both finite (Theorem 2) and infinite p -norms (Theorem 4), establishing that Variance-UCB achieves the best possible regret in both regimes. These bounds are summarized in Table 1. Prior to our work and to the best of our knowledge, no lower bounds were known for this problem. The analysis of these results is presented in Section 5.

We empirically validate our findings by numerical experiments presented in Section 6, which show that our theoretical regret bounds match empirical convergence rates in both cases of finite and infinite p -norms. We also provide examples showing that for finite p -norms, the smallest variance affects the regret, even when the feedback is Gaussian. This is in contrast to the case of the infinite norm, where it is proven [13] that under Gaussian feedback, the algorithm is not affected by the smallest variance.

1.2 Related work

Our motivation stems from growing attention to data collection methods [34, 2, 18, 20]. We focus on the problem of mean estimation and dynamically collecting data to achieve this goal. While there is a substantial body of literature on data acquisition [9, 26], specifically in the presence of privacy concerns and associated costs [19, 30, 31, 12, 17, 16], our approach differs as we do not consider the costs of sharing data. Instead, we concentrate on the data collection process itself, rather than the costs to data providers.

Our work is related to multi-armed bandits problems [5, 37], in the sense that each group can be seen as an arm, and choosing a group to sample from at each time step corresponds to choosing which arm to pull. However, the performance criterion for multi-armed bandits is measured by the difference between the mean of the chosen arm and the best arm [3, 14, 38]. In our framework, the means of the chosen arms do not impact the performance. It is their variances that matter in the optimal solution, as we measure the performance by considering the p -norm of the variance of the estimator, which can be non-convex. Because of this, and to the best of our knowledge, usual bandits algorithms and proof techniques [25] do not apply. Instead, we propose Variance-UCB, an algorithm that uses high probability upper bounds on the variance estimates adjusted to the chosen norm.

Considering data acquisition from the perspective of active learning is a natural approach [6, 7, 27], and in that sense our work is also related to active learning [15, 22, 24, 40]. In [1], the authors address the optimal data acquisition problem under the assumption of additive objective functions. They formulate the problem as an online learning problem and leverage well-understood tools from online convex optimization [8, 11, 36]. However, their ideas do not apply to our setting due to the non-additive nature of our decisions over time.

The case where the chosen norm is ∥·∥ ∞ is introduced in [4], where the authors devise the GAFS-MAX algorithm and show that for bounded feedback with known upper bounds, it achieves regret ˜ O ( T -1 . 5 ) . This case is also studied in [13], where the authors extend the feedback to sub-Gaussian, and devise the B-AS algorithm, which also has ˜ O ( T -1 . 5 ) regret. One property that emerges in their study is a regret bound deteriorates with σ -1 min , where σ 2 min is the smallest variance across groups. This is counter-intuitive, as smaller variances in general make the learning simpler. The authors pose as an open question whether any algorithm can have performance independent of σ min and show that in the special case of exactly Gaussian feedback, one can derive a σ min free bound. We partially answer this question, by showing that the leading term of the regret bound (with respect to T ) does not depend on σ min . In other words, while σ min can still impact the regret, its effect appears only in lower order terms. Our regret analysis of Variance-UCB for ∥ · ∥ ∞ recovers the same dependence in T as in [13], but also has improved dependence in the number of groups G and the variance vector σ . While [4, 13] serve as a starting point, most of their findings are only applicable to the special case p = + ∞ . In particular, they cannot be expanded to other choices of norm. We show that for finite p norms, we prove the regret bound is ˜ O ( T -2 ) , which is fundamentally different than the previous ˜ O ( T -1 . 5 ) bound.

2 Model

We consider a population partitioned into G disjoint groups. Each group is represented by an index g from [ G ] := { 1 , . . . , G } . Each individual in the population holds a real-valued data point; data from each group g ∈ [ G ] are distributed according to an unknown distribution D g , with unknown mean µ g and unknown variance σ 2 g . We denote σ min := min g ∈ [ G ] σ g and let D := D 1 × . . . ×D G . The analyst wishes to compute an unbiased estimate of the population mean for each group over T times of data collection, sampling only one group at a time. The set of feasible policies is defined as

where ∆( G ) is the set of measures supported on [ G ] . Let n g,T denote the number of collected samples from group g at the end of time T , and let ˆ µ g,T be the sample mean estimator of µ g for n g,T collected samples. Once all data have been collected at the end of the time horizon T , the analyst will compute the sample mean of each group, i.e.,

✶ Note that the vector ˆ µ T is always an unbiased estimator of the vector µ , as long as the policy π samples at least once from each group.

The variance of ˆ µ g,T is σ 2 g n g,T . The p -norm of the vector of variances of n g,T is denoted as

The analyst wishes to minimize E [ R p ( n T )] .

When choosing a policy, the analyst does not have access to the true standard deviation vector σ := ( σ 1 , . . . , σ G ) , which is needed to compute the value R p ( n T ) . Therefore the analyst must learn σ through their decisions. We benchmark the performance of a policy against the best possible performance in a complete information setting where σ is known, that is,

The optimization program (2) can be difficult to solve and analyze due to the integer constraints. Instead of using the solution to (2) as a benchmark, we use the solution to its continuous relaxation (a lower bound on (2)) , which we denote as,

Thus, we can define the regret of a policy as

For our analysis, we assume that the distributions D g are sub-Gaussian, as stated in Assumption 1, with corresponding constants c 1 , c 2 known to the analyst. 1

Assumption 1 (Sub-Gaussianity) . For each g ∈ [ G ] , D g is sub-Gaussian. That is, there exist universal constants c 1 , c 2 > 0 such that for any ϵ > 0 ,

We assume that such c 1 , c 2 are known to the analyst.

Moreover, we assume that all groups have some variation in their data, as stated in Assumption 2.

Assumption 2 (Positive Variance) . The minimum group variance is positive; i.e., σ min = min g ∈ [ G ] σ g > 0 .

3 Our algorithm: Variance-UCB

Our algorithm, Variance-UCB, builds an increasingly accurate upper confidence bound for each σ g , which we denote UCB t ( σ g ) . Recall that n g,t denotes the number of collected samples from group g through time t . At each time t , σ g can be estimated via the sample standard deviation

For convention, we set ˆ σ g, 0 = ˆ σ g, 1 = + ∞ , indicating that at least two samples are required to obtain a meaningful estimate of σ g . We can then define

where

C T was introduced in [13], and captures the trade-off between accuracy of the upper confidence bound and confidence in the estimate, and is a polylogarithmic factor in T . In particular, C T can be constructed by the analyst since it depends only on c 1 , c 2 , and T , which are known.

At time t +1 , the algorithm chooses the next group X t +1 using the rule:

Algorithm 1 Variance-UCB ( p, T, G, c 1 , c 2 ) Input: norm parameter p , time horizon T , number of groups G and subgaussian parameters c 1 , c 2 . 1: Initialize n g, 0 = 0 , ˆ σ g, 0 = ˆ σ g, 1 = + ∞ , ∀ g ∈ [ G ] 2: Compute C T according to (6). 3: for t = 0 , . . . , T -1 do 4: Compute UCB t ( σ g ) : UCB t ( σ g ) = ˆ σ g,t + C T √ n g,t , ∀ g ∈ [ G ] 5: Select group X t +1 = argmax g UCB t ( σ g ) 2 p p +1 n g,t 6: Observe feedback Y t +1 ∼ D X t +1 7: Update the number of samples: n g,t +1 = n g,t + I X t +1 = g , ∀ g ∈ [ G ] 8: Update the mean estimates: ˆ µ g,t +1 = 1 n g,t +1 ∑ t +1 s =1 ✶ X s = g Y s , ∀ g ∈ [ G ] 9: Update the standard deviation estimates: ˆ σ g,t +1 := √ 1 n g,t +1 -1 ∑ s ≤ t +1: X s = g ( Y s -ˆ µ g,t +1 ) 2 , ∀ g ∈ [ G ] 10: end for Output: ˆ µ g,T = 1 n g,T ∑ T s =1 ✶ X s = g Y s , ∀ g ∈ [ G ]

where where ties in the argmax can be broken arbitrarily. The analyst then observes a new sample Y t +1 from the chosen group X t +1 and updates the upper confidence bounds accordingly. Variance-UCB is presented formally in Algorithm 1.

Variance-UCB takes as input the time horizon T , the norm p , the groups [ G ] , and the sub-Gaussian parameters c 1 , c 2 . The algorithm initializes the upper confidence bound for every group to be infinity, which reflects the absence of knowledge of { σ g } .

In the special case p = + ∞ , Variance-UCB (instantiated with a different choice of C T ) coincides with the B-AS algorithm of [13]. 2

3.1 Regret guarantees

Our first main result gives theoretical bounds on the performance of Algorithm 1 for finite p ∈ [1 , ∞ ) . We show in Theorem 1 that when p is finite, Variance-UCB incurs regret of ˜ O ( T -2 ) .

Theorem 1. For any D that satisfies Assumptions 1 and 2 and for any finite p , the regret of VarianceUCB is at most ˜ O ( T -2 ) . That is,

Our second main result, Theorem 2, provides a matching lower bound for our problem. Thus showing that the performance of Variance-UCB and its analysis is the best possible in terms of T .

Theorem 2. Let p be finite and κ be a universal constant. For any online policy π , there exists an instance D π such that for any T ≥ 1 ,

Our analysis and proof techniques can be extended naturally to the case where p = + ∞ . However, the ˜ O ( T -2 ) regret no longer holds, and the convergence rate jumps to ˜ O ( T -3 / 2 ) .

Theorem 3. Let Σ ∞ := ∑ g ∈ [ G ] σ 2 g . For any D that satisfies Assumptions 1 and 2,

Note that a similar bound for Theorem 3 has already been established in [13]:

Our result in Theorem 3 improves the existing regret in all the problem parameters, i.e., Σ ∞ to √ Σ ∞ , and from G 2 . 5 to G 1 . 5 ). The most significant improvement lies in removing the dependence on σ min in the main term of the regret. While σ min still appears in the negligible term o ( T -1 . 5 ) , this term will be asymptotically dominated for even moderate T . This improved bound gives a better understanding on how σ min impacts the performance of the algorithm, which was left as an open question in [13].

Finally, we give a matching lower bound for the case when p = ∞ , showing that the analysis of Variance-UCB is tight in T when p = + ∞ . To the best of our knowledge, no lower bound for this problem was previously known.

Theorem 4. For any online policy π , there exists an instance D π such that for any T ≥ 1 ,

In Sections 4 and 5, we give outlines of the proofs of Theorems 1 and 2, respectively. While we briefly mention why both results change at p = + ∞ , the full proofs for Theorems 3 and 4 is deferred to Appendix C.

4 Deriving the upper bounds

In this section, we give an overview of the main steps to prove Theorem 1. A complete proof is given in Appendix A. We first show in Lemma 1 that the optimal R ∗ p ( σ ) is achieved for an optimal static policy n ∗ T ( p ) = { n ∗ g,T ( p ) } g ∈ [ G ] that assumes knowledge of σ and samples each group g proportionally to σ 2 p p +1 g .

Lemma 1. [Benchmark analysis] For each t ∈ N ∗ and p ∈ [1 , + ∞ ] , let n ∗ g,t = σ 2 p p +1 g t ∑ h ∈ [ G ] σ 2 p p +1 h . Then,

The proof of Lemma 1 utilizes the KKT conditions of the optimization program (3) that defines R ∗ p . For convenience, we introduce 3 Σ p := ∑ g ∈ [ G ] σ 2 p p +1 g for each p ∈ [1 , + ∞ ] so that we can simplify n ∗ g,t = t Σ p σ 2 p p +1 g . Using Equation (4), the expression of regret can be simplified to

From Eq. (7), we can understand the behavior of the regret by answering the following questions.

How close is the random variable n T to the optimal sampling vector n ∗ T ?

How does the curvature of R p ( . ) affect the difference R p ( n T ) -R p ( n ∗ T ) ?

Upper bounding the error n T -n ∗ T : Before we answer the first question, we show in Lemma 2 that UCB t ( σ g ) is an increasingly accurate upper bound for σ g .

Lemma 2. For all g ∈ [ G ] , with probability at least 1 -˜ O ( T -2 ) ,

The proof of Lemma 2 utilizes Assumption 1 and the choice of UCB t ( σ g ) 4 , as defined in Eq. (5).

Using Lemma 2, the difference betwen the number of samples Variance-UCB collects and the optimal number of samples can be bounded with high probability, which we state in Lemma 3.

Lemma 3. Variance-UCB collects a vector of samples n such that for all g ∈ [ G ] , with probability at least 1 -˜ O ( T -2 ) ,

To understand the meaning of Lemma 3, we note that the feedback-dependent structure of the algorithm makes the numbers of samples n g,T correlated across groups and over time, since the algorithm attempts to sample regularly from all groups. A key technical challenge is to decouple n g,t across groups and derive an instance dependent upper bound on n T -n ∗ T . This challenge does not arise in the classic multi-armed bandits setting, where the decision-maker's goal is to repeatedly pull only the best arm.

Curvature properties of R p around the optimal value n ∗ T : To answer the second question, we exploit the smoothness of R p ( · ) around n ∗ T to approximate it with a polynomial function of n . Since n ∗ T is the minimizer of R p ( · ) (subject to ∑ g ∈ [ G ] n g = T ) and R p ( · ) is differentiable around n ∗ T , the first order of the Taylor approximation of R p vanishes as T grows large, formalized in Lemma 4.

Lemma 4. Let p < + ∞ and n ′ ∈ R G + such that ∑ g ∈ [ G ] n ′ g = T . Then,

We note that the bound in Lemma 4 holds regardless of the choice of the vector n ′ , including those generated by Variance-UCB. One can interpret the upper bound in Lemma 4 as follows: the term p +1 2 ∑ g ∈ [ G ] ( n ′ g -n ∗ g,T ) 2 Tn ∗ g,T represents the exact Taylor first-order approximation of R p ( n ′ ) -R p ( n ∗ T ) R p ( n ∗ ) , and the right hand side represents an upper bound on this approximation. In particular, assuming an error ϵ = n ′ -n ∗ , Lemma 4 can be restated as,

Putting everything together: The rest of the proof consists of applying Lemmas 3 and 4 to the regret expression in Eq. (7). By Lemma 3, with high probability, ∥ n T -n ∗ T ∥ ∞ = ˜ O ( √ T ) . Applying this to Lemma 4 gives that R p ( n T ) -R p ( n ∗ T ) = ∥ n T -n ∗ T ∥ 2 ∞ R ∗ p · O ( T -2 ) . By Lemma 1, R ∗ p = Θ( T -1 ) , and therefore with high probability, R p ( n T ) -R p ( n ∗ T ) = ˜ O ( T -2 ) . With additional work, we show that the equality also holds in expectation, which implies from Eq. (7) that Regret p,T ( Variance-UCB , D ) = E π [ R p ( n ) -R p ( n ∗ T )] = ˜ O ( T -2 ) .

4.1 The case p = + ∞

Even though Lemmas 1, 2, and 3 hold for p = + ∞ , the approximation guarantee given in Lemma 4 does not hold because R ∞ is not differentiable at n ∗ T . As an alternative, we derive a first order upper bound, which we state in Lemma 5.

Lemma 5. Let σ ∈ R G + and n ′ ∈ R G + such that ∑ g ∈ [ G ] n ′ g = T . Then,

Similar to Lemma 4, we interpret the upper bound above by decomposing it into a main term,

-min g ( n ′ g n ∗ g,T -1 ) , and an error term, 1 4 max g ( n ′ g n ∗ g,T -1 ) 2 max g ( n ∗ g,T n ′ g ) 3 . As opposed to the case where p < + ∞ , R ∞ is not differentiable in n ∗ , thus the inequality above does not stem from a Taylor approximation. Adapting the bound in Lemma 5 in the same steps as in Step 3 gives a regret bound of ˜ O ( T -1 . 5 ) .

5 Deriving the lower bounds

In this section, we provide the key ideas behind the proof of Theorem 2, with the full proof given in Appendix B. Given any policy π , the idea is to pick D π from two constructed instances D a = D a 1 ×···×D a G and D b = D b 1 ×···×D b G . The interaction of D a (resp. D b ) with π yields a random vector of the number of collected samples, denoted by n a (resp. n b ). D a and D b must satisfy the following two conflicting properties:

D a and D b are sufficiently similar, in the sense that the distributions of n a and n b are close, so that π would have a hard time distinguishing between D a and D b . This notion of similarity is captured by the KL-divergence of D a and D b .

D a and D b are also sufficiently dissimilar in the sense that they induce distinct optimal allocation rules n ∗ a and n ∗ b . Specifically, any allocation n cannot be simultaneously close to both n ∗ a and n ∗ b , and n incurs a high regret under at least one of the two instances. This notion of dissimilarity is captured below.

Let σ a ∈ R G + be the vector of standard deviations of all groups under D a , and let S a ϵ be the set of allocations n such that R p ( n ; σ a ) is within ϵ of R ∗ p ( σ a ) for any n ∈ S a ϵ . Formally,

̸

Let σ b be a second vector and define S b ϵ similarly. We define the dissimilarity d ( σ a , σ b ) := inf ϵ ≥ 0 { S a ϵ ∩ S b ϵ = ∅} , which is the smallest ϵ so that these sets of allocations are disjoint. There is a tension between the two requirements on D a and D b , which is formalized next in Lemma 6.

Lemma 6. Let π be a fixed policy and D a , D b be two instances with standard deviation vectors σ a , σ b , respectively. Then,

The proof follows from an adapted version of LeCam's method [39]. To understand the implication of Lemma 6, we examine the two terms involved in the right hand side of the inequality. The first term, d ( σ a , σ b ) , increases the further σ a and σ b are from each other. The second term has the opposite monotonicity, as it decreases with the KL divergence of the two instances. These conflicting terms capture the trade-off of loss minimization versus information gain, and deriving the lower bound will come from constructing two instances for which this trade-off is maximised.

Specifically, D a is chosen such that all groups have data distributed according to D a g ∼ N ( 0 , σ 2 ) ; we denote σ a := σ (1 , . . . , 1) as the vector of standard deviations of each group. Even though instance a has all identical groups, the policy π may treat groups differently, and the interaction of π with D a may yield give different expected numbers of samples from each group. We pick h ∈ [ G ] such that E π , D a [ n h,T ] is minimized over [ G ] , and construct D b as follows:

̸

On the one hand, notice that D a and D b only differ in their coordinate h , so that KL ( D a g ||D b g ) = ✶ ( g = h )Θ( GT -1 ) , and by minimality of h , E π , D a [ n h,t ] ≤ T G , so that ∑ g ∈ [ G ] E π , D a [ n g,t ] KL ( D a g ||D b g ) = E π , D a [ n h,t ] KL ( D a h ||D b h ) ≤ T G Θ( GT -1 ) = Θ(1) .

Second, we show that d ( σ a , σ b ) = Θ( T -2 ) . We do this by measuring the distance between the sets S a ϵ and S b ϵ for a fixed ϵ , and then estimate the smallest ϵ for which this distance is 0 . This smallest ϵ corresponds to d ( σ a , σ b ) . Combining both in Lemma 6 yields max { Regret p,T ( π , D a ) , Regret p,T ( π , D b ) } ≥ Θ( T -2 ) .

Remark 1. For p = + ∞ , the proof remains the same except that in this case d ( σ a , σ b ) = Θ( T -3 / 2 ) , resulting in an overall lower bound of Θ( T -3 / 2 ) . See Appendix C.3 for the complete proof.

.

6 Numerical study

In this section, we present experimental results on the empirical performance of Variance-UCB. We explore the impact of varying each important parameter of the problem: the time horizon T , norm parameter p , number of groups G , distributions D g , and the sub-Gaussianity parameters c 1 , c 2 . In all the experiments D g follow Gaussian distributions. Except where they are varied, the default parameter settings are T = 10 5 , p = 2 , G = 2 , with the respective data distributions of groups 1 and 2 as N (1 , 1) and N (2 , 2 . 5) , satisfying c 1 = c 2 = 5 . For each parameter setting evaluated, we run Variance-UCB 500 times and report average regret over all 500 runs. For convenience, time and regret are presented on logarithmic scales.

0

2

4

6

8

10

12

14

-

30

-

20

-

10

0

log

T

Log-Regret

p

= 1

p

= 2

p

= 10

p

= 25

p

= +

∞

(

-

2)

-

line

(

-

1

.

5)

-

line

First, we vary the choice of the parameter p ∈ { 1 , 2 , 10 , 25 , + ∞} and observe its effect on the regret. Figure 1 shows that the convergence rates for each p precisely match those predicted by Theorems 1 and 3: a slope of -2 for the finite values of p ( { 1 , 2 , 10 , 25 } ), and a slope of -1.5 for p = ∞ .

0

2

4

6

8

10

12

14

-

20

-

10

0

log

T

Log-Regret

C

T

= 1000

C

T

= 5

C

T

= 0

.

001

Next, we study how mis-specifying the sub-Gaussianity parameters c 1 and c 2 affect the regret of Algorithm 1. The parameter values ( c 1 , c 2 ) only affect the algorithm's behavior through C T , so it is more natural to directly study the impact of errors in C T . For the parameters used in our experimental setup, the value of C T prescribed in Equation (6) is ∼ 5 ; we consider both underestimating and overestimating C T , and evaluate regret when instead plugging in values of C T ∈ { 0 . 001 , 5 , 1000 } in the UCB update step defined in Equation (5). Figure 2 shows that choosing an overestimate of C T incurs an increased regret. This is not surprising since choosing a larger C T decreases the confidence of the algorithm in its estimates, and forces over-exploration. Choosing a low C T also incurs a higher regret which can more severely impact the performance, as the algorithm under-explores and can get stuck in sub-optimal behavior. However, with a long enough time horizon, the algorithm will eventually estimate σ accurately, regardless of the small choice of C T . As observed in Figure 2, the smallest value of C T = 0 . 001 initially has the highest regret, corresponding to incorrect estimates and insufficient exploration, and then finally converges very late to have the lowest regret. Even for very large T , this curve has higher variance due to the noise in the estimates.

0

5

10

-

20

-

10

0

log

T

Log-Regret

p

= 1

σ

min

= 1

σ

min

= 0

.

5

σ

min

= 0

.

1

σ

min

= 0

.

05

0

5

10

-

20

-

10

0

log

T

p

= 2

σ

min

= 1

σ

min

= 0

.

5

σ

min

= 0

.

1

σ

min

= 0

.

05

0

5

10

-

15

-

10

-

5

0

log

T

p

= +

∞

σ

min

= 1

σ

min

= 0

.

5

σ

min

= 0

.

1

σ

min

= 0

.

05

Next, we study the effect of varying the lowest variance σ 2 min ∈ { 0 . 05 , 0 . 1 , 0 . 5 , 1 } while holding all other variances fixed, for each p ∈ { 1 , 2 , + ∞} . First, Figure 3 shows that varying σ min has no effect on the regret when p = + ∞ . This matches a result of [13], where they prove that when p = + ∞ , their UCB-style algorithm (very similar to Variance-UCB) is not affected by the lowest variance when the feedback is Gaussian. However, our experimental results show that this phenomenon does not persist when p is finite, as illustrated in Figure 3, where we observe that regret decreases when the lowest variance σ min increases. This is surprising since increasing the lowest variance makes the feedback more volatile, and one would expect an increase in regret as a result.

0

2

4

6

8

10

12

14

-

20

-

10

0

10

log

T

Log-Regret

G

= 2

G

= 5

G

= 50

Finally, we vary the number of groups G ∈ { 2 , 10 , 50 } . For the additional groups, we generate their data from Gaussian D g , with means µ g ∼ U ([ -1 , 1]) and standard deviations σ g ∼ U ([2 , 4]) independently for each group. From Figure 4, we observe that the regret increases in the number of groups, as expected. When the number of groups is small ( G = 2 ), Variance-UCB quickly enters a regime where regret decreases quickly. However, as the number of groups grows ( G ∈ { 10 , 50 } ), the necessary time to enter the decay regime increases ( ∼ 30 , 000 for G = 10 and ∼ 90 , 000 for G = 50 ). Initially the algorithm samples (on average) uniformly across all groups due to the UCB term outweighing the sample variance estimates, and each group must wait to be sampled enough times for the algorithm to estimate its optimal sampling rate. This delay will naturally increase with the number of groups. Once the confidence bounds are small enough, the algorithm samples the highest variances first. This causes abrupt variation (especially in the case where p is small) because the objective function is very sensitive to changes in one coordinate.

References

Jacob Abernethy, Yiling Chen, Chien-Ju Ho, and Bo Waggoner. Low-cost learning via active data procurement. In Proceedings of the Sixteenth ACM Conference on Economics and Computation , EC '15, page 619-636, New York, NY, USA, 2015. Association for Computing Machinery.

Daron Acemoglu, Ali Makhdoumi, Azarakhsh Malekian, and Asuman Ozdaglar. Too much data: Prices and inefficiencies in data markets. Working Paper 26296, National Bureau of Economic Research, September 2019.

Shipra Agrawal and Nikhil R Devanur. Bandits with concave rewards and convex knapsacks. In Proceedings of the fifteenth ACM conference on Economics and computation , pages 989-1006, 2014.

András Antos, Varun Grover, and Csaba Szepesvári. Active learning in multi-armed bandits. pages 287-302, 10 2008.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Journal of Computer and System Sciences , 75(1):78-89, 2009. Learning Theory 2006.

Maria-Florina Balcan, Steve Hanneke, and Jennifer Wortman Vaughan. The true sample complexity of active learning. Machine learning , 80:111-139, 2010.

Quentin Berthet and Vianney Perchet. Fast rates for bandit optimization with upper-confidence frank-wolfe. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Nikhil Bhat, Vivek F. Farias, Ciamac C. Moallemi, and Deeksha Sinha. Near-optimal a-b testing. Management Science , 66(10):4477-4495, 2020.

Jack Brimberg and Robert F Love. General considerations on the use of the weighted lp norm as an empirical distance measure. Transportation Science , 27(4):341-349, 1993.

Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Yang Cai, Constantinos Daskalakis, and Christos Papadimitriou. Optimum statistical estimation with strategic data sources. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 280-296, Paris, France, 03-06 Jul 2015. PMLR.

Alexandra Carpentier, Alessandro Lazaric, Mohammad Ghavamzadeh, Rémi Munos, and Peter Auer. Upper-confidence-bound algorithms for active learning in multi-armed bandits. In Algorithmic Learning Theory: 22nd International Conference, ALT 2011, Espoo, Finland, October 5-7, 2011. Proceedings 22 , pages 189-203. Springer, 2011.

Leonardo Cella, Alessandro Lazaric, and Massimiliano Pontil. Meta-learning with stochastic linear bandits. In International Conference on Machine Learning , pages 1360-1370. PMLR, 2020.

David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research , 4:129-145, 1996.

Rachel Cummings, Hadi Elzayn, Vasilis Gkatzelis, Emmanouil Pountourakis, and Juba Ziani. Optimal data acquisition with privacy-aware agents. In Proceedings of the IEEE Conference on Secure and Trustworthy Machine Learning , SaTML '23, 2023.

Rachel Cummings, Vitaly Feldman, Audra McMillan, and Kunal Talwar. Mean estimation with user-level privacy under data heterogeneity. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35 of NeurIPS '22 , pages 29139-29151, 2022.

Edith D De Leeuw. Choosing the method of data collection. 2008.

Arpita Ghosh and Aaron Roth. Selling privacy at auction. In Proceedings of the 12th ACM Conference on Electronic Commerce , EC '11, page 199-208, New York, NY, USA, 2011. Association for Computing Machinery.

Stephen J Grove and Raymond P Fisk. Observational data collection methods for services marketing: An overview. Journal of the Academy of Marketing Science , 20:217-224, 1992.

Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems , pages 1-16, 2019.

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2022.

Marius Kloft, Ulf Brefeld, Sören Sonnenburg, and Alexander Zien. Lp-norm multiple kernel learning. The Journal of Machine Learning Research , 12:953-997, 2011.

Yoshiaki Kuwata, Justin Teo, Gaston Fiore, Sertac Karaman, Emilio Frazzoli, and Jonathan P How. Real-time motion planning with applications to autonomous urban driving. IEEE Transactions on control systems technology , 17(5):1105-1118, 2009.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Shengyuan Liu, Yuxuan Zhao, Zhenzhi Lin, Yi Ding, Yong Yan, Li Yang, Qin Wang, Hao Zhou, and Hongwei Wu. Data-driven condition monitoring of data acquisition for consumers' transformers in actual distribution systems using t-statistics. IEEE Transactions on Power Delivery , 34(4):1578-1587, 2019.

Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative machine teaching. In International Conference on Machine Learning , pages 2149-2158. PMLR, 2017.

Wenbo Liu, Shengnan Liang, and Xiwen Qin. Weighted p-norm distance t kernel svm classification algorithm based on improved polarization. Scientific reports , 12(1):1-16, 2022.

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) , 54(6):1-35, 2021.

Kobbi Nissim, Claudio Orlandi, and Rann Smorodinsky. Privacy-aware mechanism design. In Proceedings of the 13th ACM Conference on Electronic Commerce , EC '12, page 774-789, New York, NY, USA, 2012. Association for Computing Machinery.

Kobbi Nissim, Salil Vadhan, and David Xiao. Redrawing the boundaries on purchasing data from privacy-sensitive individuals. In Proceedings of the 5th Conference on Innovations in Theoretical Computer Science , ITCS '14, page 411-422, New York, NY, USA, 2014. Association for Computing Machinery.

Dana Pessach and Erez Shmueli. A review on fairness in machine learning. ACM Computing Surveys (CSUR) , 55(3):1-44, 2022.

Alvin Rajkomar, Michaela Hardt, Michael D Howell, Greg Corrado, and Marshall H Chin. Ensuring fairness in machine learning to advance health equity. Annals of internal medicine , 169(12):866-872, 2018.

Mitchell Nicholas Sarkies, K-A Bowles, EH Skinner, D Mitchell, Romi Haas, M Ho, Karen Salter, Kerry May, Donna Markham, L O'Brien, et al. Data collection methods in health services research. Applied clinical informatics , 6(01):96-109, 2015.

Burr Settles. Active learning literature survey. 2009.

Aleksandrs Slivkins. Contextual bandits with similarity information. In Sham M. Kakade and Ulrike von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory , volume 19 of Proceedings of Machine Learning Research , pages 679-702, Budapest, Hungary, 09-11 Jun 2011. PMLR.

Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27, 2014.

Bin Yu. Assouad, Fano, and Le Cam , pages 423-435. Springer New York, New York, NY, 1997.

Liang Yu, Shuqi Qin, Meng Zhang, Chao Shen, Tao Jiang, and Xiaohong Guan. A review of deep reinforcement learning for smart building energy management. IEEE Internet of Things Journal , 8(15):12046-12063, 2021.

A Proof of Theorem 1

A.1 Proof of Lemma 1

The proof of Lemma 1 consists of using optimality conditions on r p :

Lemma 1. [Benchmark analysis] For each t ∈ N ∗ and p ∈ [1 , + ∞ ] , let n ∗ g,t = σ 2 p p +1 g t ∑ h ∈ [ G ] σ 2 p p +1 h . Then,

Proof. The case of p = + ∞ is discussed in [4, 13]. We focus on when p < + ∞ . Recall the definition of R ∗ p from Eq. (3):

Since the function x → x p is increasing in R + , we can replace the objective by [ R p ( n )] p = ∑ g ∈ [ G ] σ 2 p g n p g . Any feasible point n with a zero coordinate would have R p ( n ) = + ∞ , therefore any argmin to the optimization program n ∗ above must have positive coordinates, and satisfies the KKT conditions

For each g ∈ [ G ] , the first line of the system above is equivalent to -p σ 2 p g n p +1 g -λ = 0 . Therefore the KKT conditions imply that

Hence the unique minimizer is the vector

Therefore, R ∗ p = R p ( n ∗ T ) . Moreover, R p ( n ∗ T ) = R p ( T n ∗ 1 ) = 1 T R p ( n ∗ 1 ) , where the last equality follows from the homogeneity of the norm.

A.2 Proof of Lemma 2

In this section, we establish a key property of UCB t ( σ g ) , which relies on Assumption 1.

Lemma 2. For all g ∈ [ G ] , with probability at least 1 -˜ O ( T -2 ) ,

Proof. Following the notations of Section 3, we introduce the following event:

Based on corollary 1 of [13], we have

(ii) conditionally on A T ,

so that conditionally on A T , we have:

where the first Eq. is due to the definition of UCB t introduced in (5), and the bounding is due to the definition of A T . Next, notice that the function x → ( σ g + x ) 2 p p +1 is increasing, which implies that

which proves the leftmost inequality in Lemma 2. To prove the rightmost inequality, notice that x → ( σ g + x ) 2 p p +1 is also convex, therefore by Jensen's inequality,

which concludes the proof.

A.3 Proof of Lemma 3

The goal of this section is to prove Lemma 3, which consists of bounding with high probability n -n ∗ . To do so, we will design an alternative sequence ˜ n , that is simultaneously easy to analyze, and upper bounds n with high probability. The motivation for the choice of ˜ n comes from the relaxation of the choice made at every time step by Variance-UCB.

We assume through the whole section what the event A T is realized. For convenience, we view the right hand side of Lemma 2 as a quantity of its own, and introduce the width function

so that the inequality in Lemma 2 can be rewritten as

We start by proving the following lemma, which follows from the decisions Variance-UCB makes: Lemma 7. For t ≥ 2 G , we have:

Proof. The proof exploits the greedy property of the algorithm. A group needs to be sampled exactly twice to have a finite UCB. Therefore, for t = 1 , . . . , 2 G , Variance-UCB samples every group twice (in an arbitrary order), so that at n 1 , 2 G = . . . = n G, 2 G = 2 and UCB 2 G ( σ 1 ) 2 p p +1 , . . . , UCB 2 G ( σ G ) 2 p p +1 < + ∞ . By choice of X t +1 , the following inequality holds:

On the one hand, from the leftmost inequality in (9),

On the other hand, from the rightmost inequality in (9),

Therefore by combining both Inequalities (11) and (12) in Inequality (10),

which implies, after multiplying both sides by n g,t n X t +1 ,t and summing over g ∈ [ G ] ,

Dividing both sides by Σ p > 0 , and using the formula n ∗ g,t = σ 2 p p +1 g Σ p t (see Lemma 1) implies

n

X

t

+1

,t

n

∗

X

t

+1

,t

+

tw

X

t

+1

(

n

X

t

+1

,t

)

Lemma 7 follows by substracting tw X t +1 ( n X t +1 ,t ) from both sides.

Lemma 7 states that the possible excess between the number of samples output by the algorithm and the optimal number of samples is not too big, and can be controlled by the width w . Since the width decreases in the number of samples, the function

must be increasing and has therefore an inverse function that is also increasing, which we denote W t g ( x ) . We introduce the following sequence, which mimics the behavior stated in Lemma 7:

˜ n g,t = n g,t For t = 1 , . . . , 2 G ˜ n g,t +1 = ˜ n g,t + ✶ ( ˜ n g,t ≤ W t g ( n ∗ g,t )) For t ≥ 2 G The sequence is easier to analyze and upper bounds the true number of samples n : Lemma 8. ˜ n ≥ n .

Proof. By construction, the result holds for t = 1 , . . . , 2 G . Assume for the sake of contradiction that the result does not hold for a g ∈ [ G ] and t +1 > 2 G , and take such a t minimal. For such a pair ( g, t ) :

✶ where the first step follows from the definition of n . By minimality of t , n g,t +1 > ˜ n g,t +1 and n g,t = ˜ n g,t , which induces the second step. The third step follows from the definition of ˜ n , and the last step follows once again from the minimality of t .

The strict inequality in the chain of inequalities implies that ✶ ( X t +1 = g ) = 1 and ✶ ( n g,t ≤ W t g ( n ∗ g,t )) = 0 , so that n X t +1 ,t > W t g ( n ∗ X t +1 ,t ) .

By taking the inverse of the increasing function W t g on both sides, the previous inequality can be rewritten as

contradicting Lemma 7. Therefore the assumption is wrong and ˜ n ≥ n , which completes the proof.

.

Lemma 9. For a fixed g ,the sequence { W t g ( n ∗ g,t ) } t ≥ 1 is increasing. Consequently,

Proof. For a fixed x > 0 and t ≥ 1 ,

therefore the sequence of functions { x → x -tw g ( x ) } t ≥ 1 is decreasing in t . Consequentely, the sequence of its inverse functions { x → W t g ( x ) } t ≥ 1 is increasing in t :

Moreover, the function W t g is increasing in R + :

By combining Equations (13) and (14), we obtain

which proves that the sequence { W t g ( n ∗ g,t ) } t ≥ 1 is increasing, thus completing the proof for the first part of Lemma 9.

The second part follows by induction on t ≥ 1 . For t ≤ 2 G , the result holds immediately as ˜ n g,t = n g,t and n g,t ≤ 2 . We assume the result holds for a t ≥ 2 G . We distinguish two cases:

˜ n g,t ≤ W t g ( n ∗ g,t ) : This implies that:

where the first step stems from the definition of ˜ n , the second step stems from the assumption ˜ n g,t ≤ W t g ( n ∗ g,t ) , and the third step stems from the first part of the proof.

˜ n g,t > W t g ( n ∗ g,t ) : This implies that:

where the first step stems from the definition of ˜ n , the second step stems from the induction hypothesis, and the third step stems from the first part of the proof.

Studying both cases concludes the induction and proves the inequality for all t ≥ 1 . This concludes the proof of Lemma 9.

Next, we derive an upper bound on W t g :

Lemma 10. For x > 0 , W t g ( x ) ≤ x 1 -t wg ( x ) x .

Proof. Let x, y > 0 with 0 ≤ y -tw g ( y ) = x , so that y = W t g ( x ) by definition of W t g . we have:

where the last step follows from the function y → w g ( y ) y being decreasing, which concludes the proof.

We are now ready to prove Lemma 3.

Lemma 3. Variance-UCB collects a vector of samples n such that for all g ∈ [ G ] , with probability at least 1 -˜ O ( T -2 ) ,

Proof. Conditionally on A T , all the previous lemmas stated in this section hold. Consequently,

where the first step stems from Lemma 8, the second step stems from Lemma 9, the third step stems from first order approximations and the last step stems from Lemma 10. From Assumption 2, σ > 0 , therefore n ∗ = Θ( T ) and the right hand side is Θ( √ T ) .

A.4 Proof of Lemma 4

The goal of this section is to prove Lemma 4. We will do so by combining the optimality of n ∗ and the curvature properties of R p .

Lemma 4. Let p < + ∞ and n ′ ∈ R G + such that ∑ g ∈ [ G ] n ′ g = T . Then,

Proof. Since R p is defined over the constrained set { n ∈ R G + | ∑ g n g = T } with empty interior, Taylor inequality can't be directly applied. To overcome this, we construct an alternative function that has the same curvature as R p while being defined on a non-empty interior domain. Define

For each vector n ∈ R G + with ∑ T t =1 n g = T , we set for each g ∈ [ G -1] λ g := n g T . We have λ ∈ K . Moreover, we have from (1):

where the last equality follows from the homogeneity property of the p -norm. Eq. (16) motivates the introduction of the re-scaled function r :

so that Eq. (16) can be written as

From the definition of K in (15), the interior of K , denoted K ◦ , is non-empty and is equal to

Moreover, the function r p is C 3 in K ◦ . From Lemma 1, λ ∗ ∈ K ◦ . Therefore from Taylor's theorem, we have for λ ∈ K ◦ :

where ∇ is the gradient operator and H is the hessian operator. We exploit the optimality of λ ∗ to derive simple expressions for ∇ r p ( λ ∗ ) , H ( λ ∗ ) , and ∂ 3 r p ( u ) ∂λ g ∂λ h ∂λ i . First, since λ ∗ is optimal and an interior point of K , the following equality holds:

In particular, ⟨ λ -λ ∗ , ∇ r p ( λ ∗ ) ⟩ = 0 . ⟨H ( λ ∗ )( λ -λ ∗ ) , λ -λ ∗ ⟩ is simpified in the following lemma, which proof is deferred to A.6:

so that Inequality (19) is rewritten as

It remains to bound 1 x ! y ! z ! ∂ 3 r p ( u ) ∂λ g ∂λ h ∂λ i . To do this, the following lemma is applied (the proof is also deferred to A.6):

Lemma 12. The following inequality holds:

which implies after dividing both sides by r p ( λ )

By using the change of variable from λ to n ′ , we have

which implies

A.5 Putting everything together

We are now ready to complete the proof of Theorem 1.

Theorem 1. For any D that satisfies Assumptions 1 and 2 and for any finite p , the regret of VarianceUCB is at most ˜ O ( T -2 ) . That is,

Proof. First, notice that

where the first step stems from the definition of regret introduced in Eq. (4), the second step stems from the law of total expectation, and the third step stems from both P ( A T ) ≤ 1 and R p ( n ) -R ∗ p ≤ ∥ σ 2 ∥ p . It remains to show that each term in the rightmost side is in ˜ O ( T -2 ) . First, P π ( A c T ) ≤ 2 GT -2 . 5 = ˜ O ( T -2 ) . Next, we have conditionally on A T :

where the first inequality stems from Lemma 4, and the second inequality stems from ( n g,T -n ∗ g,T ) 2 ≤ ∥ n -n ∗ ∥ 2 ∞ . Since ∑ g n g,T = T , from Lemma 3,

̸

̸

where the first step stems from ∑ h n h,T -n ∗ h,T = ∑ h n h,T -∑ h n ∗ h,T = T -T = 0 , the second step stems from Lemma 3, and the last steps stem from taking the max over the sum. The last inequality implies

In particular, ∥ n -n ∗ ∥ ∞ = ˜ O ( √ min h n ∗ h,t ) = ˜ O ( √ T ) and

Therefore,

Recall from Lemma 1 that R ∗ p = R p ( n ∗ ) = Θ( T -1 ) . Thus by taking the conditional expectation on A T , we have

The following holds:

Proof. Fix a λ ∈ K ◦ and g, h, i ∈ [ G -1] .

Expression of the gradient: On the one hand,the definition of r p in (17) implies

so that

On the other hand, from the formula ( f p ) ′ = pf ′ f p -1 ,

Combining Equations (24) and (25) yields

so that

Therefore ∇ r p = ( ∂ ∂λ 1 r p , . . . , ∂ ∂λ G -1 r p ) = r 1 -p p ( H 1 , . . . , H G -1 ) = r 1 -p p H , which proves the first Eq. of Lemma 13.

which concludes the proof of Theorem 1.

A.6 Proof of auxiliary lemmas

In this section, we prove Lemmas 11 and 12. To do so, simple expressions for the derivatives of r p are needed. These are established in the following lemma:

Lemma 13. For g, h, i ∈ [ G -1] , we introduce the following functions in K ◦ :

Expression of the Hessian: We have

where the first equality is due to the definition of the Hessian, the second equality is due to the expression of the gradient from Eq. (26), the third equality applies the product rule to the derivative, and the fourth equality applies the definition of G g,h in Lemma 13. This proves the second Equality of Lemma 13.

Third derivatives.

where the first equality is due to the definition of the Hessian, the second inequality is due to the Hessian expression established previously, and the third equality is due to the linearity of derivation. In a similar fashion to the previous case, we derivate each of the products r 1 -2 p p H g H h and r 1 -p p G g,h separately. On the one hand,

where the first equality is due to the derivation product rule, the second equality is due to the definition of G g,h introduced in Lemma 13, and the third equality is due to a reordering of the terms. On the other hand, by following the exact same steps

Replacing the previous two expressions in the formula for ∂ 3 ∂λ g ∂λ h ∂ i r p yields

where the symmetry of G g,h = G h,g is used. This derives the third Eq. of Lemma 13 and concludes the proof.

We are now ready to establish the proof of Lemma 11.

Proof. Setting the value λ ∗ in the value of the Hessian established in Lemma 13 implies that for g, h ∈ [ G ] :

where the first equality stems from the definition of the Hessian, the second equality stems from H ( λ ∗ ) = 0 due to the optimality of λ ∗ , the third equality stems from the definition of G g,h , and the fourth equality stems from r p ( λ ∗ ) = Σ 1 p -1 p . As a consequence,

where the first step follows from the definition of the scalar product, and the second step stems from the expression of H ( λ ∗ ) derived previously in the proof. In the third step, the sum is distributed over the terms 1 λ ∗ G and 1 λ ∗ g ✶ ( g = h ) . In the fourth step, the first sum is factorized, and in the fifth step, the equality ∑ g ∈ [ G -1] ( λ -λ ∗ ) g = (1 -λ G ) -(1 -λ ∗ G ) = λ ∗ G -λ G is used. This completes the proof of Lemma 11.

Next, we establish the proof of Lemma 12.

Lemma 12. The following inequality holds:

Proof. From Lemma 1, the following equality holds for each g ∈ [ G ] :

For a fixed u ∈ [ λ , λ ∗ ] , ∑ g u g = ∑ g λ g = ∑ g λ ∗ g = 1 , thus the coordinate g 0 achieving the maximal λ ∗ g u g must have u g 0 ≤ λ ∗ g 0 . Since u ∈ [ λ , λ ∗ ] this implies that λ g 0 ≤ u g 0 ≤ λ ∗ g 0 and

consequently max g λ ∗ g u g ≤ max g λ ∗ g λ g . The following upper bounds follow:

Moreover, since p ≥ 1 , each j ∈ { 1 -p, 1 -2 p, 1 -3 p } is non-positive, and by minimality of λ ∗ : r p ( u ) j ≤ r p ( λ ∗ ) j , so that:

Each of the previous three expressions can be simplified by using r p ( λ ∗ ) = Σ 1 p -1 p :

Hence by using the expression of the third derivatives established in Lemma 13:

As a consequence, by taking the sup over u, g, h, i, x, y, z ,

which completes the proof of Lemma 12.

B Proof of Theorem 2

In this section, we prove Theorem 2. First, we establish an initial lower bound that captures the trade-off between how hard it is to distinguish two instances and how hard it is to optimize both under the same action (See Appendix B.1). Next, we provide a specific counter example which regret is at least Θ( T -2 ) (See Appendix B.2).

B.1 Proof of Lemma 6

Lemma 6. Let π be a fixed policy and D a , D b be two instances with standard deviation vectors σ a , σ b , respectively. Then,

Proof. Let D a and D b be two instances with standard deviations σ a and σ b , and let δ ≥ d ( σ a , σ b ) . Let X be a random variable over the set { a, b } . The following inequalities hold

where the first step stems from the support of X being { a, b } , and the second step stems from the law of total expectation. Let ˆ x be the following (random) classifier:

̸

Since δ ≥ d ( σ a , σ b ) , ˆ x is well defined. Moreover, P π .X ( R p ( n ; σ X ) -R ∗ p ( σ X ) > δ ) ≥ P X (ˆ x = X ) ≥ inf ˆ x P X (ˆ x = X ) , where the infinimum is taken over all the classifiers of { a, b } . Moreover, by Pinsker's inequality, inf ˆ x P X (ˆ x = X ) ≥ exp ( -KL ( D a ||D b ) ) . Moreover, from bandits feedback divergence properties (see [25]), KL ( D a ||D b ) = ∑ g ∈ [ G ] E π , D a [ n g,t ] KL ( D a g ||D b g ) , which concludes the proof of Lemma 6.

B.2 The counter-examples

We introduce the following instances for each g ∈ [ G ] ,

̸

̸

We start by upper bounding the KL -divergence between the two instances:

Proof. For convenience, we set ν := √ 1 T < 1 . The formula for the KL -divergence of two univariate normal distributions of zero mean implies

̸

̸

The taylor expansion of the expression above can be derived by combining the expansions of both the functions x → log(1 + x ) and x → 1 1+ x in the domain (0 , 1) :

Summing over all coordinates implies

where the inequality follows from n T ≤ T . This concludes the proof.

Next, we derive a simpler form for d ( σ a , σ b ) . The simplification exploits the symmetries in σ a , σ b : Lemma 15. Let u denote the unit vector (1 , . . . , 1) T . The following equality holds:

where the function r p is introduced in Appendix A.

Proof. For x ∈ { a, b } , let S x ϵ := { ϵ > 0 | r p ( λ ; σ x ) -r ∗ p ( σ x ) ≤ ϵ } . By definition of d ,

̸

We prove d ( σ a , σ b ) = r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) by proving each of the inequalities d ( σ a , σ b ) ≤ r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) and d ( σ a , σ b ) ≥ r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) .

̸

First, we prove d ( σ a , σ b ) ≤ r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) . Since σ a can be obtained by swapping the first two coordinates of σ b , the symmetry of r p implies r ∗ p ( σ a ) = r ∗ p ( σ b ) . Moreover, for each ( λ 1 , λ 2 , λ ′ ) ∈ K , r p (( λ 1 , λ 2 , λ 3: G ); σ a ) = r p (( λ 2 , λ 1 , λ 3: G ); σ b ) . As a consequence, r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) = r p ( 1 G u ; σ b ) -r ∗ p ( σ b ) , and any ϵ > 0 satisfying ϵ ≥ r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) must also satisfy S a ϵ ∩ S b ϵ = ∅ . In particular, d ( σ a , σ b ) ≤ r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) .

To derive d ( σ a , σ b ) ≥ r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) , we use the following lemma, which proof is deferred later in the section:

Lemma 16. If λ = ( λ 1 , . . . , λ G ) ∈ S a ϵ ∩ S b ϵ , and τ a permutation of [ G ] . λ τ := ( λ τ (1) , . . . , λ τ ( G ) ) is also in S a ϵ ∩ S b ϵ .

̸

Let ϵ ≥ 0 satisfying S a ϵ ∩ S b ϵ = ∅ , and let λ ∈ S a ϵ ∩ S b ϵ . For each permutation τ , λ τ is also in S a ϵ ∩ S b ϵ . Each of S a ϵ and S b ϵ is convex and therefore the intersection S a ϵ ∩ S b ϵ is also convex, which implies that

which in turn implies that ϵ ≥ r p ( 1 G u , σ a ) -r ∗ ( σ a ) . By taking the inf we get d p ( σ a , σ b ) ≥ r p ( 1 G u , σ a ) -r ∗ p ( σ a ) , which completes the proof.

We now state the proof of Lemma 16:

̸

Proof. Let λ = ( λ 1 , . . . , λ G ) ∈ S a ϵ ∩ S b ϵ . Since every permutation can be written as a composition of transpositions (2-cycles), it suffices to prove the result for transpositions. Let τ = ( g, h ) with g = h . We distinguish 3 cases:

g, h ≥ 3 : Since σ a 3: G = σ b 3: G , r p ( λ ; σ x ) = r p ( λ τ ; σ x ) for each x ∈ { a, b } . The implication λ τ ∈ S a ϵ ∩ S b ϵ follows.

g, h ≤ 2 : By symmetry of the problem, σ a 1 = σ b 2 = σ b τ (1) . Similarly, σ a 2 = σ b τ (2) . Therefore, λ ∈ S a ϵ implies λ τ ∈ S b ϵ . Similarly, λ ∈ S b ϵ implies λ τ ∈ S a ϵ . The implication λ τ ∈ S a ϵ ∩ S b ϵ follows.

g ∈ { 1 , 2 } and h ≥ 3 . Without loss of generality, assume that g = 1 and h = 3 . Since D b 1 ∼ D b 3 , the equality σ b 1 = σ b 3 holds and therefore r p ( λ ; σ b ) = r p ( λ τ ; σ b ) , therefore λ τ ∈ S b ϵ . Moreover, r p ( λ τ ; σ a ) = r p ( λ ; σ b ) , therefore λ τ ∈ S a ϵ . The implication λ τ ∈ S a ϵ ∩ S b ϵ follows.

The three cases cover all possible 2-cycles, which completes the proof.

Weare now ready to state the proof of Theorem 2. It remains to show that d p ( σ a , σ b ) = r p ( 1 2 u , σ a ) -r ∗ p ( σ a ) = Θ( T -2 ) . We are ready to state the proof of Theorem 2.

Theorem 2. Let p be finite and κ be a universal constant. For any online policy π , there exists an instance D π such that for any T ≥ 1 ,

Proof. We set D π := argmax { Regret p ( π , D a ) , Regret p ( π , D b )} . The following inequalities hold:

where the first step stems from the definition of D π , the second step stems from Lemma 6, and the third step stems from a combination of both Lemma 19 and Lemma 15. The rest of the proof consists in deriving a lower bound on r p ( 1 G u ; σ a ) -r ∗ p ( σ a ) . Since we will now solely focus on σ a , we can drop the dependencies in σ . A direct consequence of Inequality (19) is

We will lower bound each of the terms p +1 2 ∑ g ∈ [ G ] ( 1 G -λ ∗ g ) 2 λ ∗ g and

-7( p +2) 2 (min g λ ∗ g ) 2 max g ( λ ∗ g 1 /G ) 3 p +3 ∥ u /G -λ ∗ ∥ 3 ∞ . For convenience, we set

Since the first group has the highest variance, λ ∗ 1 should also be the highest. In particular, λ ∗ 1 ≥ 1 G and f T ≥ 0 . By symmetry of σ a , λ ∗ 2 = . . . = λ ∗ G = 1 G -f T G -1 . Therefore,

Next, notice that λ ∗ ≤ 1 , which implies that max g ( λ ∗ g 1 /G ) 3 p +3 ≤ G 3 p +3 , and that ∥ u /G -λ ∗ ∥ ∞ = max( f T , f T / ( G -1)) = f T , and min g λ ∗ g = 1 G -f T G -1 . Therefore,

It remains to bound f T . We do so by deriving the first terms of its taylor expansion in and T :

where the first equality stems from Lemma 1, the second equality stems from the binomial Taylor expansion, and the third equality stems from the Taylor expansion of x → 1 1 -x . The last step implies that there exists universal constants q 1 , q 2 such that for T ≥ 1 ,

Replacing the previous bounding of f T in Inequality (29) yields

which in turn implies

where we used r ∗ p = Θ( T -1 ) and from Lemma 1 that

By setting κ = q 2 1 4 , we get Regret T ( π , D π ) ≥ κ p +1 T 2 + O ( 1 T 2 √ T ) , which completes the proof of Theorem 4.

C Upper and lower bounds when p = ∞

The proof for Theorem 3 (upper bound when p = ∞ ) follows the same high-level steps as the proof of Theorem 1 (upper bound when p ∈ R ). However, some adjustments of the proofs are necessary. Table 2 summarizes the changes that are required.

In Appendix C.1, we introduce and prove Lemma 5, the replacement for Lemma 4. In Appendix C.3, we prove Theorem 4.

C.1 Curvature of R ∞

The upper bound Lemma 4 goes to + ∞ as p = + ∞ and is no longer insightful. In this section, we provide a suitable bound:

Lemma 5. Let σ ∈ R G + and n ′ ∈ R G + such that ∑ g ∈ [ G ] n ′ g = T . Then,

Proof.

From Lemma 1, so that for g ∈ [ G ] and n ′ ,

where the first step follows from R ∗ ∞ = σ 2 g n ∗ g,T , the second step follows from Taylor's inequality applied on the function x → 1 x on n ∗ g,T , and the first step follows from the definition of the infinite norm. By dividing both sides by R ∗ ∞ , rearranging the terms, and taking the max, we obtain

which completes the proof of Lemma 1.

C.2 Proof of Theorem 3

Theorem 3. Let Σ ∞ := ∑ g ∈ [ G ] σ 2 g . For any D that satisfies Assumptions 1 and 2,

Proof. Similarly to Appendix A.5, the total-expectation bounding still holds:

We now upper bound each of the two terms in (30). First, similarly to the case where p < + ∞ presented in Appendix A, P π ( A c T ) ≤ 2 GT -2 . 5 , so that

Next, we apply the final lower bound derived in Section B.2 from [13]:

Lemma 17. Conditionally on A T ,

By taking the min over g ∈ [ G ] in Lemma 17, the following inequality follows conditionally on A T :

Moreover, since C T = ˜ O (1) , the right hand side of the inequality above is in ˜ O ( T ) . Next, from Inequality (23), and from min g n ∗ g,T = T min g σ 2 g Σ ∞ ,

and

Combining the previous two inequalities yields

Recall that R ∗ ∞ = Σ ∞ T . Applying Lemma 5 on n g,T and using the upper bounds (32) and (33) implies that conditionally on A T

Since the right hand side is deterministic, the inequality above extends by taking the conditional expected value on A T . Moreover, the second, third, and fourth term are all in o ( T -1 . 5 ) , so that:

Finally, combining both inequalities (31) and (34) in Inequality (30) yields

which completes the proof of Theorem 3.

C.3 Proof of Theorem 4

Lemma 18. Let σ a , σ b be two vectors with Σ a ∞ := ∑ g ( σ a g ) 2 , Σ b ∞ := ∑ g ( σ b g ) 2 and Σ a ∞ ≥ Σ b ∞ . There exists a Σ a,b ∞ ∈ [Σ b ∞ , Σ a ∞ ] such that

Proof. Let σ a , σ b be two vectors with Σ a ∞ := ∑ g ( σ a g ) 2 and Σ b ∞ := ∑ g ( σ b g ) 2 , ϵ > 0 , and λ ∈ [0 , 1] G such that ∑ g λ g = 1 . We have:

therefore,

Hence, the allocation T λ is simultaneously in S a ϵ and S b ϵ if and only if

In particular, S a ϵ ∩ S b ϵ is the polytope

which is non-empty if and only if

so that d ( σ a , σ b ) is the (unique) solution to the equation ∑ g max ( λ ∗ g ( σ a ) Tϵ Σ a ∞ +1 , λ ∗ g ( σ b ) Tϵ Σ b ∞ +1 ) = 1 . To better understand the form of this solution, we introduce the following three decreasing functions:

First, notice that d ( σ a , σ b ) = f -1 (1) . Next, by assuming (WLOG) that Σ a ∞ ≥ Σ b ∞ , we have f a ≥ f ≥ f b , so that d ( σ a , σ b ) = f -1 (1) ∈ [( f b ) -1 (1) , ( f a ) -1 (1)] . Moreover,

where the first step stems from the definition of f a , the second step stems from max( x, y ) = x + y + | x -y | 2 , the third step stems from ∑ g λ ∗ g = 1 , and the fourth step stems from the defintion of the 1 -norm. The last equation implies

Similarly, f -1 b (1) = Σ b ∞ 2 T ∥ λ ∗ ( σ a ) -λ ∗ ( σ b ) ∥ 1 . Therefore:

which concludes the proof.

Theorem 4. For any online policy π , there exists an instance D π such that for any T ≥ 1 ,

Proof. The proof consists of constructing two instances that optimize the trade-off stated in Lemma 6. The first instance, which is denoted D a , consists of G groups of standard normal distributions. Next, select a group h such that E π, D a [ n g,T ] is minimal. Notice that by minimality of E π, D a [ n h,T ] ,

We have

The second instance, which is denotes D b , is defined as follows:

so that Σ a ∞ = G , Σ b ∞ = G + ν and

̸

We introduce the following Lemma, which proof is deferred to later in this section.

Lemma 19. The following inequality holds: ∑ g ∈ [ G ] E π , D a [ n g,T ] KL ( D a g ||D b g ) ≤ Tν 2 2 G .

By combining both Lemmas 18 and 19 in the inequality stated in Lemma 6, we obtain

By setting ν = √ G T ≤ 1 and D π := argmax { Regret ∞ ( π , D a ) , Regret ∞ ( π , D b ) } , the previous inequality implies

Regret ∞ ( π , D π ) = max { Regret ∞ ( π , D a ) , Regret ∞ ( π , D b ) } ≥ e -1 / 2 2 G 1 . 5 T 1 . 5 1 + ( G -1) 2 G ( G +1) ≥ 1 2 G 1 . 5 T -1 . 5 which completes the proof of Theorem 4, .

Lemma 19. The following inequality holds: ∑ g ∈ [ G ] E π , D a [ n g,T ] KL ( D a g ||D b g ) ≤ Tν 2 2 G .

Proof. For convenience, we set ν := √ G T < 1 . The formula for the KL -divergence of two univariate normal distributions of zero mean implies

The taylor expansion of the expression above can be derived by combining the expansions of both the functions x → log(1 + x ) and x → 1 1+ x in the domain (0 , 1) :

where the first step stems from the inequality n T ≤ T , the second step stems from using the previous two equalities, the third step stems from simplifying the infinite sum term-wise, the fourth step stems from the expansion of a geometric serie, and the fifth step stems from µ = 1 √ T ≥ 0 . Since D a and D b have the same distribution at all coordinates except coordinate h , we have:

where the inequality follows from the minimality of E π , D a [ n h,T ] and ∑ g E π , D a [ n g,T ] = T . This concludes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript studies a special type of bandit problem: instead of maximizing the rewards, the learner aims to minimize the L_p norm of the variance vector for the mean estimators of each arm. The motivation of this problem is multi-group mean estimation, where a small total variance is desired. \n\nThe authors proposed a variance-UCB algorithm, which maintains an upper confidence bound for the variance of each arm, and then chooses the arm which maximizes a certain quantity determined by its current UCB and number of pulls. This quantity is motivated by the expression of optimal allocation in hindsight. This algorithm achieves a regret of O(T^{-2}) for all finite p, which is optimal by matching lower bounds. For p = infinity, this algorithm recovers the \\Theta(T^{-1.5}) optimal regret obtained by [Antos et al. 2008, Carpentier et al. 2011].\n\nSTRENGTHS:\nThe problem setting is practically relevant, and the authors proposed a novel form of UCB-type algorithm to solve it. The resulting regret bound is also tight.\n\nWEAKNESSES:\nI have several major concerns about this work: \n\n1. The target L_p norm in Eqn. (1) is not properly motivated. The authors should at least describe a scenario where minimizing this specific quantity is meaningful. For example, is p = 1 or infty the only interesting scenario? Can the current result be extended to some value of p < 1, say p = 1/2 (I believe this is relevant when one uses absolute estimation error).\n\n2. Although the algorithm is a new UCB-type algorithm, both the intuition and analysis are very straightforward. By the KKT condition for the objective, the optimal allocation should give the same value to all arms for a certain quantity; so the algorithm simply pulls the arm with the largest quantity in order to reduce it. The analysis is then standard, possibly with the exception of Lemma 8 (the key part of the new UCB algorithm). However, this contribution alone does not reach the bar for a NeurIPS publication. \n\n3. Very importantly, the current manuscript does not have a tight regret dependence on the number of arms G. Obtaining so would require a more careful analysis in both the upper and lower bounds. In particular, the lack of tight dependence on G makes the current two-point lower bound much less interesting. \n\n4. There seems to be an issue in the upper bound analysis. Note that Lemma 3 only proves a one-sided inequality, i.e. an upper bound on n - n^*. However, when the authors applied Lemma 4, a two-sided upper bound on |n - n^*| is needed, and this requires additional explanation. I understand this seems to be fine because the sum of n_g is always T, but arguing in this way would have an additional factor of G which I don't know if that's necessary, and in my opinion (see above) the right dependence on G is important. \n\nAlso, if one would only like to obtain an upper bound on n - n^*, the complicated arguments below Lemma 8 seem unnecessary. It seems that applying Lemma 8 to the last time a certain group g is chosen should be enough. \n\nAdditional comments: \n\nPage 3: \"the optimization program (3) can be NP-hard to prove\". Please add a justification. \n\nPage 4, expression of C_T: the precise form is not interesting at all. Just say C_T = C * log(T) for a large enough constant C. \n\nPage 18, Eqn. (21): where does Sigma_p come from? Lemma 2 does not involve this term.\n\nQUESTIONS:\nI'll likely increase my rating if the authors could work out the tight regret dependence on G.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper focuses on the active learning algorithm for multi-group mean estimation. The authors focus on minimizing the $l_p$-norm of the variance vector. This paper proposes the variance-UCB algorithm to actively select which group to sample in each round. The sample complexities for $p<\infty$ and $p=\infty$ are provided, and the tightness of proposed algorithm on $T$ is also verified.

STRENGTHS:
This paper focuses on the novel problem of estimating the mean values of multiple groups with minimal variance. The authors adopt $l_p$-norm to measure the variances of group mean estimates. The variance-UCB algorithm is designed and analyzed for different values of $p$. In addition, the lower bound is established to justify the tightness of the proposed algorithm on the dependency of $T$. The simulation results are provided to corroborate the theoretical findings.

WEAKNESSES:
1. The authors mainly focus on the dependency of $T$. However, the variance of each group also influences the sample complexity. It will be helpful to derive the upper bound that reflects these instance-dependent quantities.

2. The limitation part is missing in the paper.

QUESTIONS:
The questions are provided in the weaknesses section.

LIMITATIONS:
The authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
This paper propose the Variance-UCB algorithm to sequentially learn the mean in a multigroup setting in order to minimize the variance over all mean estimates, and prove the regret of the algorithm is optimal for both finite and infinite p values.

STRENGTHS:
1. The Variance-UCB algorithm in this paper automatically achieves optimal regret for both cases when $p$ is finite and $p=\infty$, and provide solid theoretical proofs for both the general lower bound and the matching regret for the algorithm.

2. The authors support their statements with empirical results by varying different parameters in time horizon $T$, norm parameter $p$, group numbers $G$ and sub-Gaussian parameters.

3. The paper is in general well-written.

WEAKNESSES:
The experiment results lack the comparison with other benchmark results in the literature. The authors only mention that the varying lowest variance has no effect on the regret results when p is infinite which is a known result in the literature.

One typo: line 65 "it is thekkir variances..." should it be "their"?

QUESTIONS:
As stated in the weakness, can the authors compare the Variance-UCB algorithm to other results in the literature? For example, the authors mention in the paper that this work lies in the frame work of a multi-arm bandits setting, but other bandit works in this line did not take care of the variances of the mean estimates. It would be good to empirically show that the proposed algorithm outperforms these bandit algorithms in the literature in this task.

LIMITATIONS:
Lack of comparison to other benchmark algorithms in the literature.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the mean estimation problem under the multi-armed bandit setting. Here, we have a group of populations (random variables) with unknown mean and standard deviations. The goal is to estimate the mean of each group on the fly and optimize the regret (measured by different kinds of norms). Their major contribution is a group of confidence interval based online learning algorithms and showing the optimality of these algorithms.\n\nSTRENGTHS:\nI think many working on multi-armed bandits and statistical learning have naturally wondered this question. I am glad to see this result and that existing UCB type algorithms still work in the new setting.\n\nWEAKNESSES:\nThe text in intro feels a bit strange to me (e.g., quite a bit of empty sells and and “analyst” also sounds odd to me). I think most people understand the importance of this problem. \n\nIt would be helpful if the authors could relate this result to Stein estimator type research and stratified sampling. They feel related.\n\nQUESTIONS:\nI have a few questions: \n\n1. The baseline considers a relaxed version of the original problem because the latter is NP-Hard. Nevertheless, the regrets are tight. Does that imply a certain kind of integrality gap bound between the NP-hard original problem and the relaxed problem? \n2. How is the regret related to the number of populations, and is this also tight? \n3. Is it possible to comment on your result with the long line of research related to Stein estimators? It seems that those statistics results also aim to estimate population mean via shrinkage methods. \n4. Is there a reason to call a random variable a population, e.g., they have finite populations, or they are inconsequential naming? \n5. Are there speculations and educated guesses on the generalization of the problem, e.g., what if my goal is to estimate the standard deviation/second moment of each population?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a general bandit-based active learning approach to the problem of learning the means of several disjoint groups so as to optimize for the variance of the resulting estimates as measured by the p-norm of the variance vector. An algorithm is proposed which samples based on a certain upper confidence bound designed for the variance p-norm, and its regret guarantees are studied. The authors identify and carefully study two regimes: p < infty and p = infty, and prove a somewhat surprising, and tight, dichotomy: namely, the regret is only Theta(1/T^2) for p < infty but increases to Theta(1/T^1.5) for p = infty. The tight lower bounds is achieved via a simple and explicit lower-bounding construction, whereas the upper bounds follow from two different appropriate upper bounds on the curvature of the variance vector p-norm. Some experiments are given to confirm the theoretically identified tight rates.\n\nSTRENGTHS:\nThe main strength of the paper is that it presents a very clean and natural UCB-based algorithm which, for reasons that are transparently shown via a sequence of intuitive arguments (such as a simple Taylor-based curvature bound on the p-norm vector), achieves the tight rates for the important problem of identifying group means with small variance using as few samples as possible. I also consider it quite important and strong that the authors were able to locate the corresponding tight lower bounds, which is achieved by a surprisingly clean and simple construction (where two incompatible worlds are provided which are identical except for two groups' distributions that are slightly tweaked). In this manner, this simple case of optimizing for p-norms of the variance vector becomes, in a sense, resolved (up to potential tweaks to the setup which --- as mentioned in the future work section --- could involve constraining the sampling policies etc.)\n\nAnother strength is the lucid and careful presentation of the results --- the paper is generally quite well-written (aside from a few typos that I encourage the authors to locate and fix).\n\nWEAKNESSES:\nIn the theory part, there are in my opinion no big weaknesses --- the case is worked out quite comprehensively. Of course, a (small) weakness of the approach is the elusive nature of the subgaussian parameters that the proof assumes --- some experiments are later run to convince the reader that you don't necessarily need to know these constants exactly, but the intricate ways in which they are present in the bounds make the situation not as pleasant to deal with theoretically (on which note, I would like to ask the authors to --- for full transparency ---- more prominently mention the assumption, e.g. in the contributions section).\n\nAnother relatively small weakness is that the experimental section could be tightened up, in the sense of some plots being expanded to be more informative and some off-the-cuff remarks elaborated on. (See the Questions section below.) Still, no major problems there in my opinion.\n\nQUESTIONS:\nFirst, for the theory part, and for the experimental part alike, I would like to better understand the nature of requiring the smallest variance to be nonzero as well as it featuring in the performance guarantees and --- in fact, as mentioned on line 275 --- somewhat paradoxically leading to higher regret as it decreases. If there is an intuitive explanation or some examples for why this should be so, I'd like to learn more.\n\nSecondly, could the authors plot out some other norms between l2 and linfty? This way the transition effect as p -> infty could be better portrayed and understood. \n\nThirdly, can plots such as Figure 2 be extended to make the convergence of some \"difficult\" curves (such as C=0.001 in that example) more visible?\n\nFinally, do the authors have any insights into why the transition in Figure 4, even though it takes a long time to wait for it, kicks in so rapidly? Thanks!\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}