Title: Robust Sampling for Active Statistical Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: active statistical inference

data collection

robust sampling

prediction-powered inference
Abstract: Active statistical inference is a new method for inference with AI-assisted data collection. Given a budget on the number of labeled data points that can be collected and assuming access to an AI predictive model, the basic idea is to improve estimation accuracy by prioritizing the collection of labels where the model is most uncertain. The drawback, however, is that inaccurate uncertainty estimates can make active sampling produce highly noisy results, potentially worse than those from naive uniform sampling.
In this work, we present robust sampling strategies for active statistical inference. Robust sampling ensures that the resulting estimator is never worse than the estimator using uniform sampling. Furthermore, with reliable uncertainty estimates, the estimator usually outperforms  standard active inference. This is achieved by optimally interpolating between uniform and active sampling, depending on the quality of the uncertainty scores, and by using ideas from robust optimization. We demonstrate the utility of the method on a series of real datasets from computational social science and survey research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors’ goal is to increase the efficiency of active sampling policies used in the context of active statistical inference: Inaccurate predictive models can give poor uncertainty estimates, which can result in supposedly efficient sampling methods (e.g., uncertainty sampling) having a larger sample efficiency than uniform sampling. To address this, the authors propose a method which smoothly interpolates between an active and uniform sampling policy, where the “optimal” interpolation is identified in a data-driven way.\n\nSTRENGTHS AND WEAKNESSES:\nThe submission tackles a somewhat significant problem: Predictive (surrogate) models are used to complement scarce data in many practical applications, and robustness of policies informed by such models to the potentially unreliable uncertainty estimates they produce is important in such applications. The scope of the paper is limited to the design of robust active sampling policies; active sampling is useful especially in the types of data-poor settings in which predictive-powered inference is applied.\n\nThe submission appears original, technically sound, and the claims well-supported by both theoretical analysis and experimental results. It is exceptionally clearly written and well organized.\n\nMy main concern regards the effectiveness of the authors’ method: I think the paper could benefit from more discussion of the method’s sensitivity to user-specified aspects, in particular the choice of budget-preserving path.\n\nQUESTIONS:\n**Sensitivity to choice of budget-preserving path:** In Figure 4, the linear and Hellinger budget-preserving paths appear to perform about as well as uniform sampling. In general, how sensitive is the method’s performance to the choice of budget-preserving path? In practice, how would a user select a good budget-preserving path?\n\n**Uncertainty set:**\nTheorem 2 requires that $\\hat{\\rho} = \\rho^\\star + o_P(1)$. Does the degree of misspecification of $\\hat{e}$ affect whether this condition is violated or not? How does incorporation of the uncertainty set affect whether/when this condition is violated?\n\nIn section 2.2, it is mentioned that a permissive uncertainty set results in a large value of $\\hat{\\rho}$, which is intuitive: A misspecified predictive model is less likely to produce a reliable active sampling policy. Are there advantages to solving the robust optimization problem instead of, for instance, adding a penalty for small values of $\\rho$ to the original optimization problem (line 131)?\n\nLIMITATIONS:\nYes. While the authors mention the sensitivity of their method to the choice of budget-preserving path, more details on the extent of this sensitivity could be provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds upon the recently introduced area of active statistical inference, in which the goal is to formulate a sampling policy for the collection of a labeled dataset that will be used in conjunction with a predictive model to estimate some statistical quantity of interest. Specifically, the paper formulates a robust active statistical inference problem where the goal is to find a sampling policy that is no worse than two given reference policies in terms of variance. The paper focuses on the case where one of the policies is the naive uniform sampling policy.\n\n\nThe paper's solution to this problem is to consider a path of policies that interpolate between the given policies and to choose the policy along the path that minimizes the estimate of the variance. To account for potential model misspecification, the paper considers a robust version of their estimator that allows for a perturbation of the variance estimator.\n\n\nThe paper goes on to apply their techniques to real-world data, showing that their method outperforms both uniform and non-robust active sampling strategies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This paper considers an interesting problem at the intersection of robustness and active data collection. Although some aspects of the problem were already covered in the original active statistical inference paper (see below), the paper considers a broader family of interpolation techniques along with a robustness modification to the optimization problem.\n\n\n- The paper evaluates the proposed methodology on 5 different datasets, covering a variety of inference tasks and prediction methods. Overall, their methods enjoy better empirical performance than the baseline approaches.\n\nWeaknesses:\n\n- The novel contributions of the paper need to more clearly delineated. As the paper is written, one might think that this paper is the first to consider the concept of finding the minimum variance interpolating sampling policy. However, this idea was already proposed in the original Zrnic and Candes active statistical inference paper, albeit only for the case of linear interpolants and without any robustness constraints. The contributions of this paper seem to be that it (a) expands the set of interpolating paths, (b) that it adds a robustness constraint to the variance estimator, and (c) that it gives a more thorough evaluation of uniform v.s. active v.s. robust active.\n\n\n- It is hard to tell how useful the robustness constraint is. There is one figure comparing the performance with and without the robustness constraint, confusingly shown twice (Figure 4 and Figure 10). However, because the x and y-axes are not the same between the two, I can't really eyeball how useful the robustness constraint is. There should be several figures comparing with and without robustness for the various path interpolants on ALL the datasets, and they should also be on the SAME axes (maybe use dotted and solid lines). It would also be helpful to see what happens as the robustness hyperparameter c is varied.\n\n\n- There is no mention of multiple runs/random seeds in the paper nor are there any confidence intervals/error bars in the plots. Given how much randomness is involved in this paper (initial training set, random label selection according to the policy, etc), it would be good to see how confident we should be in the results of the paper.\n\n\n- As the paper currently stands, there is no guidance on how to choose the interpolating path. There is one experiment showing that the geometric path empirically outperforms the other two proposed paths on the Pew survey dataset, but there's no reasoning given. Moreover, it's worth pointing out that one could actually consider many types of paths simultaneously and choose the best path/interpolating fraction based on minimizing the variance estimator.\n\nQUESTIONS:\n- Empirically, how important is the robustness criterion for the proposed method?\n\n- What do the results of the paper look like across multiple random seeds?\n\n- Do you have any intuition for why one interpolation method should be preferred over another?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a robust sampling strategy for active statistical learning that guarantees performance no worse than uniform sampling. The core idea is to define a budget-preserving interpolation path between an arbitrary active sampling policy and uniform sampling, and to search along this path for a robust choice of the interpolation parameter $\\rho$. The authors evaluate the method empirically on several datasets and demonstrate that it often outperforms both uniform sampling and the original active sampling policy.\n\nWhile the conceptual contribution is interesting, the paper lacks clarity in several important algorithmic aspects and could benefit from a broader experimental evaluation. If these issues are addressed (especially clarification of optimization strategy and broader empirical baselines), the work would be a valuable contribution.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The motivation to robustify active sampling is strong, especially in settings where model misspecification or label noise can lead to degraded performance.\n* The paper search over an interpolation path between uniform and active sampling for better estimators.\n* Empirical results show improvements, supporting the theoretical analysis.\n* The methodology is general and can potentially be applied to various existing active sampling strategies.\n\n**Weakness**\n\nPlease refer to below questions section.\n\nQUESTIONS:\n**Weaknesses and Questions**\n\n1. Algorithmic Clarity\n\n   The process of obtaining $\\rho_{\\text{robust}}$ is not fully specified in Algorithm 1.\n\n   - Do the authors perform grid search or use gradient-based methods to solve the minimax problem?\n   - If grid search is used, how is the grid step chosen?\n   - The paper should address the sensitivity of the solution to the grid step size or local minima, especially since small changes in $\\rho$ can lead to large changes in the objective function.\n\n2.  Misspecification Set $\\mathcal{C}$\n\n    The choice of the misspecification set $\\mathcal{C}$ is not well justified.\n\n     * The authors appear to assume an $\\ell_2$-ball, but other choices (e.g., $\\ell_\\infty$, greedy perturbations, or adaptive constraints) could lead to different behaviors.\n     * There is no discussion or comparison of how different choices of $\\mathcal{C}$ affect robustness or optimization difficulty.\n     * Given the known difficulty of minimax optimization (non-convexity, local optima), how do the authors ensure the optimal solution can be found?\n\n3. Interpolation Path Design\n\n   * In lines 112–113, the paper introduces linear and geometric interpolation paths between $\\pi$ and $\\pi_{unif}$.\n   * However, the choice of these two is not motivated. Are there other potential interpolation strategies (e.g., entropy-based, adaptive weightings)?\n   * A discussion of why only these two paths are chosen and their tradeoffs would strengthen the paper.\n\n4. Experimental Coverage\n\n   * The paper does not clearly specify which active sampling method $\\pi$ is used in the experiments. This is essential for reproducibility and clarity.\n   * Active learning encompasses many strategies (uncertainty sampling, margin sampling etc.). The current evaluation appears to be based on a single $\\pi$.\n   * To convincingly demonstrate the robustness of the proposed method, it would be helpful to evaluate it on multiple active sampling baselines.\n\n\n5. Typos\n\n   * line 76: \"distribution of the unobserved labels\" should be \"observed labels\".\n\nLIMITATIONS:\n* The method relies on a well-estimated error $\\hat{e}^2(\\cdot)$. If this is poorly estimated (e.g., under small data regimes), the robust sampling policy might become unreliable.\n* The minimax formulation adds significant optimization difficulty compared to standard active learning.\n* The lack of theoretical guarantees about convergence or uniqueness of $\\rho_{\\text{robust}}$ is a concern.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a methodology for performing appropriate labeling in situations where labeling each instance is expensive. While classical active learning methods adaptively determine where to perform labeling and place emphasis on prediction, this paper places emphasis on statistical inference of the model associated with data observation. Existing active statistical inference based on sampling from a random distribution suffers from the problem that when the error value is large, i.e., the sampling probability is small, the results are worse than uniform sampling. In contrast, this paper proposes a robust sampling rule that lies between uniform sampling and active statistical inference, bridging the two. Inference based on the proposed method has been shown to have a smaller asymptotic variance than that of uniform sampling and active statistical inference under appropriate assumptions. Additionally, numerical experiments have been conducted to compare the proposed method with uniform sampling and active statistical inference.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of this paper are as follows: \n\n1. It solves the problem of existing active statistical inference, which is that it performs worse than uniform sampling when the error is large, by appropriately linking the two. \n2. It extends to general M-estimators. It also extends to robust estimation by using the admissible set $\\mathcal{C}$ to allow for the effects of misidentification that arise when estimating the error. \n\nOn the other hand, the weaknesses of this paper are as follows:\n\n1. Insufficient reference to related research. The relationship and differences with active testing (Kossen et al., 2021) should also be mentioned. In addition, the problems of statistical inference when using conventional active learning (e.g., selecting from the most uncertain points) should also be mentioned. \n2. Lack of clarity in the problem setting and experimental results. If my understanding is correct, the proposed method seems to allow exceeding the prior budget. Furthermore, the explanation of the strength of the assumption that the estimate $\\hat{e} (\\cdot)$ in the algorithm converges in probability to the error $e (\\cdot)$ is insufficient. In addition, the intervals between the numerical values on the axes of the figures in the numerical experiments are not consistent, making it difficult to determine what these figures represent. As a result, the effectiveness of the proposed method cannot be correctly confirmed by numerical experiments.\n\n[Quality: 3]\nThe proposed method provides a theory on the asymptotic variance of estimators using an algorithm and extends it to general M estimators. Furthermore, it proposes a robust method for the effect of misidentification of the error function. Additionally, it mentions the issues of the proposed method, such as the selection of $\\mathcal{C}$ and its impact.\n\n[Clarity: 2 $\\to$ 3]\nThere are several unclear points regarding the proposed method. First, it is unclear whether the algorithm allows the budget to be exceeded. Second, the explanation of the strictness of the error estimation, i.e., that $\\hat{e} (\\cdot)$ converges in probability to $e (\\cdot)$, is insufficient. Third, the intervals between the axis scales in each figure are not consistent, making it difficult to interpret the figures and, consequently, to determine whether the numerical experimental results are significant.\n* After the rebuttal, the authors addressed all three points of ambiguity, so I raised Clarity to 3.\n\n[Significance: 3]\nThe proposed method offers a new approach to the problem of existing methods, where the results become worse than uniform sampling when the error function is large, and provides several extensions. This contributes new insights to the field of statistical inference for active learning. Additionally, the paper points out that there is room for further research on the selection of $\\mathcal{C}$ and its impact on robustness.\n\n[Originality: 2 $\\to$ 3]\nThe proposed method suggests the possibility of designing a new distribution that connects uniform sampling and active statistical inference, and is original in that it also considers the impact of misidentification in estimation. However, it lacks comparison with several important related studies. For example, active testing (Kossen et al., 2021) also focuses on statistical inference in active learning rather than prediction, and like this paper, it designs a distribution proportional to the error in each instance and makes decisions based on the realized values from that distribution. It should also mention the problems of using conventional active learning for statistical inference. In the current draft, it seems preferable to perform classical active learning, select samples that minimize prediction, and then perform statistical inference on the resulting model.\n* After the rebuttal, the authors clearly explained the relevance of their study to existing studies, so I raised Originality to 3.\n\nBased on the above, I rated the paper as 3 due to room for improvement in clarity and originality.\n\nKossen et al., Active Testing: Sample–Efficient Model Evaluation, ICML 2021\n\nQUESTIONS:\n-Questions\n\nPlease clarify the following unclear points. If these unclear points are appropriately revised, I will raise the score.\n\n1. Since the algorithm randomly decides whether to observe each instance, it is possible that all instances are observed in extreme cases. Does this paper allow the budget $n_b$ to be exceeded? If so, can the same claims as in this paper be made while keeping the total number of observations within $n_b$ by adaptively changing the sampling distribution for each observation, similar to active testing (Kossen et al., 2021)? \n\n2. In lines 9, 101, and 297, is this claim asymptotic? Does it hold for finite samples? \n\n3. The explanation of the inconsistency in the axis scales in all figures, including Figure 1, and how to read the figures should be clarified.\n\n4. References to related research. The differences between active testing and the present setting should be mentioned. In addition, the issues with performing statistical inference after classical active learning should be mentioned. My understanding is that the bias caused by not randomly selecting instances is the problem. Is this correct? If so, related research on statistical inference after decision-making based on statistical models, such as post-selection inference (Lee et al., 2016), should be mentioned to see if the bias of active selection can be corrected. Perhaps this is difficult because the selection of instances by active learning does not fit into the framework of post-selection inference. Is that correct? \n\nLee et al., Exact post-selection inference, with application to the lasso, The Annals of Statistics 2016\n\n\n5. It is assumed that $\\hat{e} (\\cdot)$ converges in probability to $e (\\cdot)$, but I am not sure if this assumption is strong. Are sufficient conditions given?\n\n\n-Comments\n\nThe following points are not necessarily required to be addressed, but they caught my attention. \n\n1. Is $y$ limited to 0 or 1? Can this be extended to general distributions? \n\n2. If $\\mathcal{C}$ is an $l_2$-ball and the radius is reduced to 0 (as $n \\to \\infty$), can the same theorem be derived? \n\n3. I think it would be better to clarify what the expected values refer to, such as line 85 and Equation (2). I think that expected values of $X$ and those of $(X, Y)$ are mixed together.\n\nLIMITATIONS:\nYes. However, as I understand it, it seems that exceeding the budget is permitted, so the possibility that strict budget constraints may not be met should be added to the limitations. On the other hand, this study proposes a new  method for robust statistical inference in active learning, and therefore does not fall under potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a sampling scheme for statistical inference that is a mixture of uniform and uncertainty based sampling schemes. The authors note that the uncertainty based sampling depends on the quality of the model’s uncertainty estimates, and can even perform worse than random sampling when these estimates are poor. Subsequently, they propose a robust scheme that guarantees the performance would be bounded by that of uniform sampling. Results seem to indicate that the proposed scheme performs well with a range of qualities of uncertainty estimates.\n\nSTRENGTHS AND WEAKNESSES:\nThe central idea of using error function estimates along a path connecting the two sampling schemes is simple and elegant. The main theorems are clearly stated and the paper does a good job of building intuition first with a concrete example (mean estimation in section 2). I had a few concerns and observations but I am somewhat inclined for the paper to be accepted:\n\n1. I am not entirely convinced how practical this method might be given its reliance on some hand-picked hyperparameters, namely the set of misspecification (C) and the choice of path. While the choices made in the paper seem to work well, they do not sufficiently convince me how the results might generalize.\n2. Relatedly, in Fig 3, I am not sure how much of the performance difference can be attributed to the overall scheme vs. just the choice of path. Just eyeballing Fig 4 against Fig 3, it appears the choice of path leads to a substantial difference in performance, with a similar magnitude to the performance difference in Fig 3. It would be useful to see where, say, a linear path lands in Fig 3. Essentially some more ablation studies to really understand the benefits of the proposal.\n3. For the choice of C: the l_2 norm constraint doesn’t seem very intuitive. What if the error estimates are off systematically? We might suspect such systematic differences for the regions where the model is uncertain vs. where it is fairly certain. Would this still bound the overall variance? Would it likely just converge to uniform sampling in practice?\n4. Minor: in Fig 2, consider using the same scaling for both figures so it’s easy to quickly compare the two. You could even combine the two figures and just use different formatting for the curves (-- vs .-). In line 214, “we collect all labels Y_i”, I presume you mean just the burn-in labels. In Fig 3, curious why are we using such odd burn-in sizes?\n\nQUESTIONS:\nSee strengths and weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}