Title: Near-Exponential Savings for Population Mean Estimation with Active Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adaptive Sampling

Variance-UCB

Disagreement Based Active Learning
Abstract: We study the problem of efficiently estimating the mean of a $k$-class random variable, $Y$, using a limited number of labels, $N$, in settings where the analyst has access to auxiliary information (i.e.: covariates) $X$ that may be informative about $Y$. We propose an active learning algorithm ("PartiBandits") to estimate $\mathbb{E}[Y]$. The algorithm  yields an estimate, $\widehat{\mu}_{\text{PB}}$, such that $\left( \widehat{\mu}_{\text{PB}} - \mathbb{E}[Y]\right)^2$ is $\tilde{\mathcal{O}}\left( \frac{\nu + \exp(c \cdot (-N/\log(N))) }{N} \right)$, where $c > 0$ is a constant and $\nu$ is the risk of the Bayes-optimal classifier. PartiBandits is essentially a two-stage algorithm. In the first stage, it learns a partition of the unlabeled data that shrinks the average conditional variance of $Y$. In the second stage it uses a UCB-style subroutine ("WarmStart-UCB") to request labels from each stratum round-by-round. Both the main algorithm's and the subroutine's convergence rates are minimax optimal in classical settings. PartiBandits bridges the UCB and disagreement-based approaches to active learning despite these two approaches being designed to tackle very different tasks. We illustrate our methods through simulation using nationwide electronic health records. Our methods can be implemented using the \textbf{PartiBandits} package in R.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PartiBandits, an active learning algorithm for estimating the population mean of a binary label with covariates and limited labels. The method combines stratified sampling with an adaptive allocation policy, and achieves a near-exponential convergence in Massart low-noise condition. Matching upper and lower bounds are provided and experiments on semi-synthetic data validate the gains under ideal stratification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nTackles an under-explored but important problem: active learning for population mean estimation.\n\nBoth matching lower bounds and upper bounds are given, making the algorithms minimax optimal.\n\nGiven a pre-defined stratification G, WS-UCB can achieve nearly exponential convergence rate when \\mu is small.\n\nEmpirical validation, though limited, shows clear improvements over SRS.\n\nWeaknesses:\n\nThe main result only considers the hard-margin condition, which is rarely the case in practice.\n\nIt is not clear what is the role of buffer \\tau in the final bound of theorem 3\n\nThe stratification process is under-specified: A^2 is mentioned in theory but not used in experiments. \n\nExperiments have only SRS baseline.\n\nMinor presentation issues: e.g. what are \\rho_{\\leq} and \\rho_{>} in the statements of Theorem 4? Some notations, like R_1(n),  lack definition in the main text.\n\nQUESTIONS:\nWhat is the convergence rate if the assumption is not Massart low noise condition (i.e. hard-margin) but rather more general Tsybakov condtion?\n\nThe authors eliminate the dependence on \\sigma_{min} but introduce another buffer \\tau, what is the specific dependence on \\tau in the final bounds? Is there a trade-off between big \\tau and small \\tau?\n\nWhy is A^2 not used in the evaluation, is there any practical concern, e.g. efficiency?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-15"}

comments: SUMMARY:
This paper proposes a PartiBandits algorithm for estimating the mean of a binary random variable with auxiliary information, which includes two stages: (1) Using a Agnostic Active algorithm to learn a partition of the data to shrink the average conditional variance of the variable; (2) Using a UCB-style subroutine to actively request labels for the learned strata. The paper proves that the algorithm could achieve a near-exponential improvement in estimation accuracy in comparison to Simple Random Sampling, and the convergence rate is minimax optimal. This paper also provides empirical verification of the algorithm's performance using real-world data.

STRENGTHS AND WEAKNESSES:
Strength: This paper shows strong results and proposes algorithms that achieve theoretically-optimal convergence rates. It also bridges a gap between UCB algorithms and disagreement-based methods. This paper provides both theoretical and empirical evidence to support the claims. It is also well-structured and the presentation of the main results are clear.

QUESTIONS:
1. Is it difficult to extend the algorithms and theoretical results to non-binary cases?

2. How is the computational cost of the PartiBandits? Could it be applied to large datasets in practice?

LIMITATIONS:
The Bayes Optimal Classifier is in the Class assumption (Assumption 2) could be too strong for practical settings, despite it is common in theoretical analysis.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel (meta) algorithm for the estimation of the mean of a binary random variable Y that depends on some (informative) X in a scenario where labels are expensive.  The proposed algorithm consists of two parts: first, an active-learning approach learns a partition G of the space X so that the partitions $A_1,\\dots, A_G$ with probability $P_k=P(X\\in A_k)$ minimize Var(Y|A_k). This is done using an online classifier that partitions based on its predictions. Then in a second step a bandit algorithm is applied which selects the partition A_k so that a bound on the weighted variance $P_k^2 Var(Y|A_k)$ is minimized. The first part of the algorithm is a direct application of the existing algorithm $A^2$, while the second part is a slight variation of Variance-UCB.The idea is that the initial split reduces the variance based on how informative X is, while the second part then focuses the majority of the remaining sample budget on the high variance partitions. The work establishes matching lower and upper bounds for each algorithm, where the lower bound is given by example. Finally, two small experiments illustrate the work.\n\nSTRENGTHS AND WEAKNESSES:\nI was not able to conduct a proper review since the work is very far outside my core domain and i only realized very late that the total review load is around 150 pages and since I scheduled this paper last due to its complexity, I was not able to read up on related work. I still do a best-effort review based on the main article without appendix. I am choosing low confidence and opt for weak accept.\n\nThe strength of the paper is a clear theoretical analysis of the problem including matching lower and upper bounds. This is based on the reliance on strong results from prior work, but is still an achievement. \n\nThe main weaknesses i would like to point out is the overall minimalistic approach of the paper\n1. Algorithm 2 seems underexplored and minimalistic.\n2. Not all sources of uncertainty seem to be carried over to the analysis \n3. the experimental part is very short and especially experiment 2 is not understandable to me. The experiment do not provider enough information to establish the theoretical rates and are too minimalistic to properly explore strengths and weaknesses.\n4. In general, bounds and their intuitive meaning are underexplored.\n\nDetails:\n1. From my understanding, Algorithm 2 partitions the input space in just two parts: A_0 and A_1 based on the classification of the hypothesis produced by the A2 algorithm. As a result, the gain of Algorithm 2 is just based on how well the classifier predicts the right class but does not make use of variability of regions within a class. E.g., assuming the true data is generated by logistic regression, we would ideally want to partition in three classes, where the middle class is close to the boundary.\n2. Algorithm 1 assumes that $P_k=P(x\\in A_k)$ is known. This either means that the assumption is that the initial unlabeled dataset is the full distribution, or the values are given by an oracle. I do not think that matters for small number of partitions since we can assume that the amount of unlabeled data is always sufficiently large compared to the amount of labeled data that the additional error can be included in some of the factors, but formally it is not discussed in the theorems.\n3. Experiment 2: Do I understand it correctly, that in each trial from the 10000 drawn people only the ones are selected which fall in the 5% lower and upper intervals, that is we see the results based on easily separable classes? In that case, I think the experiment is almost a \"strawman\" ( for a lack of better word) since in this example it is also incredibly easy to find the classifier that is separating them with just a few samples. In this case, a maximum margin classifier with just 1 sample from each class would have a high probability to be bayes optimal (when both labels are correct). So this experiment shows the extreme case of where the algorithm must perform best. I still do not understand the shown figure, because it appears the result is almost constant, even though i would expect to see 1/N convergence in both algorithms. A larger sample range, especially in the small sample regime would be interesting. Similarly, for experiment 1 only the case of symmetric noise and two same size classes is investigated. I do not find an explanation in the paper why this special case should be sufficient, since in this case both partitions should have identical variance.\n4. I would have loved to read more about the intuitive meaning of the bounds. For Algorithm 2 for example, I do not understand why it should not depend on the number of partitions. I would intuitively expect the bound to become worse as the number of partitions grows. Similarly, for Algorithm 1 i do not understand why the warmstart is necessary, since it does not appear in the bound.\n\n\ntechnicalities:\n1. I was not able to find the warmstart component in Algorithm 1. I think the initialisation of the std-dev in line 1 should do that, but then i would expect some dependency of t on g(not G) to ensure that each g has an area where its variance is maximized.\n2. A number of quantities appear to be undefined in the main article. For me as a non-expert, especially considering the density or the article, it made me search quite a few times because i was not sure whether i missed something. It might be that some are standard notation, but i would like the authors to check the following variables that i mark with the first occurrence i found:\n- $\\tilde{\\mathcal{O}}$ line 65 (is this: $\\mathcal{O}$ ignoring log-terms?\n- $R_1$, line 201\n- $c_1,c_2$ line167. \n- $ \\Sigma_1(\\mathcal{G})$ line 178\n3. Theorems do not clearly state assumptions. I know that you stated assumptions globally, but I think it is crucial to describe concrete assumptions of each analysis as minimal as possible, e.g., Theorem 3 should not depend on Assumptions 1 and 2 since the algorithm assumes $\\mathcal{G}$ is given. At the same time, considering the choice of $c_1,c_2$ in Algorithm 2, it appears that Theorem 3 holds for a much broader set of random variables than binary Y.\n\nQUESTIONS:\n1. Can you provide experimental results with more difficult settings, e..g, harder to separate classes?\n2. Can you provide scaling results for Algorithm 2 as g increases?\n3. Can you sopecify what the significance of tau is in Algorithm 1, and how your results change when tau is set to 0?\n\nLIMITATIONS:\nno. A conclusion and discussion is missing. I mentioned above the missing discussion of the scaling behavior in several cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of estimating the mean of a binary random variable $Y$ using limited number of labelled instances. They assume that they have an access to covariate $X$ that may be informative about $Y$. The authors suggest a PartiBandits algorithm, which takes a hypothesis class $\\mathcal C$, label budget $N$, a confidence level $\\delta$, and a buffer fraction $\\tau$ as an input. The algorithm performs in two steps. First, using the $A^2$ (agnostic active) algorithm as a subroutine, PartiBandits stratifies the covariate space into two sets. After that, it estimates stratified mean using Variance-UCB algorithm [Aznag et al., 2023] with a ''warm-start'' step (WarmStart-UCB). Assuming the Tsybakov noise condition and that the Bayes optimal classifier belongs to the concept class $\\mathcal C$, the authors prove minimax optimal upper and lower bounds on the accuracy of mean estimation. They illustrate advantage of their procedure over stratified random sampling using Health Record Data.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts.\n\n1. According to the numerical experiments, the PartiBandits algorithm significantly outperforms stratified random sampling. \n\n2. The authors prove rigorous guarantees on the PartiBandits performance. They also prove a matching lower bound claiming optimality of their procedure.\n\n\nWeaknesses.\n\n1. The setting is limited to binary outcomes.\n\n2. The assumption that the Bayes classifier must belong to the concept class $\\mathcal C$ (Assumption 2) is rarely satisfied in practice.\n\n3. The number of strata is limited to 2.\n\n4. The proofs of the upper and lower bounds (Theorems 5 and 6) significantly rely on the well-known results from the literature on active learning (for example, Theorem 4 from [Hanneke, 2011]).\n\nQUESTIONS:\n1. How the result of Theorem 5 will change if $Y$ is a (non-Bernoulli) random variable taking its values in $[0, 1]$?\n\n2. Is it possible to further improve the algorithm by increasing the number of strata?\n\n3. In [Puchkin and Zhivotovskiy, 2022], the authors show that exponential improvements in agnostic active learning are possible even if the Bayes optimal classifier does not belong to the concept class $\\mathcal C$. Instead, they require $\\mathcal C$ to have a bounded combinatorial diameter and a finite star number. Is it possible to replace Assumption 2 by similar requirements in your setup?\n\n4. Is stratified random sampling the only existing competitor? Can you compare PartiBandits with other algorithms?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}