Robust Sampling for Active Statistical Inference
Authors: Puheng Li
Tijana Zrnic
Emmanuel Candes
Venue: NeurIPS 2025 poster ()
Keywords: active statistical inference
data collection
robust sampling
prediction-powered inference
Primary area: probabilistic_methods
TL;DR: We present robust sampling strategies for active statistical inference, a new method for inference with AI-assisted data collection.

Abstract: Active statistical inference is a new method for inference with AI-assisted data collection. Given a budget on the number of labeled data points that can be collected and assuming access to an AI predictive model, the basic idea is to improve estimation accuracy by prioritizing the collection of labels where the model is most uncertain. The drawback, however, is that inaccurate uncertainty estimates can make active sampling produce highly noisy results, potentially worse than those from naive uniform sampling.
In this work, we present robust sampling strategies for active statistical inference. Robust sampling ensures that the resulting estimator is never worse than the estimator using uniform sampling. Furthermore, with reliable uncertainty estimates, the estimator usually outperforms  standard active inference. This is achieved by optimally interpolating between uniform and active sampling, depending on the quality of the uncertainty scores, and by using ideas from robust optimization. We demonstrate the utility of the method on a series of real datasets from computational social science and survey research.

Robust Sampling for Active Statistical Inference
Puheng Li Department of Statistics Stanford University Stanford, CA 94305 puhengli@stani ord edu
Tijana Zrnic
Department Statistics and Stanford Data Science
Stanford University
Stanford, CA 94305
tij ana. www.costanc edu
Emmanuel J. Candès
Department of Statistics and Department of Mathematics
Stanford University
Stanford, CA 94305
candes@stanford.ce
Abstract
new method for inference with Al-assisted data collection. Given budget the number of labeled data points that can be collected and assuming access to an Al predictive model, the basic idea is to im- prove estimation accuracy by prioritizing the collection of labels where the model is most uncertain. The drawback, however, is that inaccurate uncertainty esti- mates make active sampling produce highly noisy results, potentially worse than those from naive uniform sampling. In this work, we present robust sam- pling strategies for active statistical inference. Robust sampling ensures that the resulting estimator is never worse than the estimator using uniform sampling. Fur- thermore, with reliable uncertainty estimates, the estimator usually outperforms standard active inference. This is achieved by optimally interpolating between uniform and active sampling, depending on the quality of the uncertainty scores, and by using ideas from robust optimization. We demonstrate the utility of the method on series of real datasets from computational social science and survey research.
Active statistical inference [1
Introduction
Collecting high-quality labeled data remains challenge in data-driven research, especially when each label is costly and time-consuming to obtain. In response, many fields have embraced machine learning as a practical solution for predicting unobserved labels, such as annotating satellite imagery
in remote sensing [46] and predicting protein structures in proteomics (24). Prediction-powert inference M is methodological framework showing how perform valid statistical inference despite the inherent biases in such predicted labels.
Active statistical inference [51] recently introduced to further enhance inference by actively selecting which data points to label. The basic idea is to compute the model's uncertainty scores for all data points and prioritize collecting those labels for which the predictive model is most uncertain. When the uncertainty scores appropriately reflect the model's errors, Zmic and Candès (51] show that active inference can sigmificantly outperform prediction-ooweee inference (which can essen- tially be thought of as active inference with naive uniform sampling), meaning it results in more accurate estimates and narrower confidence intervals. However, when uncertainty scores poor quality, active inference can result in overly noisy estimates and large confidence intervals. This
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

uniform
active
robust active
Effective sample sizee
2285
1.0
1947
Coverage
1860
1415
0.8
1206
0.8
1291
1471
1677
1912
1291
1471
1677
1912
nb
Figure 1: Effective sample size and coverage on Pew post-election survey data. We compare uniform, active, and robust active sampling. for different values of the sampling budget nap- The target of inference is the approval rate of a presidential candidate. We show one standard deviation around the effective sample size.
an important limitation, seeing that there is widespread recognition that measuring model uncer- tainty challenging. Large language models, for example, are often overconfident in their answers [10
47 48). Miscalibrated uncertainty scores also arise when there is distribution shift between the training data and the test domain.
To illustrate the issue empirically, consider the problem of estimating the approval rate of a pres- idential candidate: 8" Y.) where Y {0, 1} is the binary indicator of approval, using Pew post-clection survey data 32 Here, we have demographic covariates XI...., X,, corresponding to people, but do not observe the approval indicator Y; for everyone. Rather, we have budget ny < n on how many people we can survey and collect their Yg. In addition, we have a machine learning model that
obtain cheap prediction f(X;) of Y; from the available covari- ates. Active inference suggests surveying those individuals where f is uncertain. For example, if f(X;) is obtained by thresholding continuous score p(X;) E [0, 1] representing the probability the model assigns to the missing label taking on the value 1, this could mean prioritizing the collection of labels where p(X,) is close 0.5. In Figure [ we show the effective sample size and coverage of prediction-powered powered inference (uniform sampling), standard active inference, and our robust active inference method, for varying values of the budget ny. The effective sample size is formally de- fined in Section 4 it is the number of samples the method that samples uniformly at random would need use to achieve the accuracy of the labeling method under study. To demonstrate challenge for active inference, we train f on small dataset, resulting in poorly estimated uncertainties. We see that active sampling results in a smaller effective sample size than simple uniform sampling. This is because the variance of the active sampling strategy is large. Meanwhile, the robust method outperforms both baselines. This is achieved by estimating the quality of the uncertainty scores and optimally interpolating between uniform and active sampling. All three methods come with provable validity guarantees, as confirmed by the achieved target coverage of 90%.
The source code for all experiments is available at https::/ggt http:///ittuub..o//ppie...e com /pnn//
Robust-Acttre-Sttaiitt..a Inference
1.1 Related work
Our paper builds active statistical inference (51), which itself builds irrdiction-powereed in- ference [I] and, more generally, statistical inference assisted by predictive models [29] 40 45).
There is growing literature in this space, aimed at ensuring robustness against poor predic-
tions 2 16 19 23 30 31,, sample efficiency when there no good pre-trained model f [52)
simplicity and applicability to general estimation problems [25 50), and handling missing
covariates 25 31]. Notably, several works study adaptive label collection strategies 14 17
Zooming out further, at a technical level this line of work relates to semiparametric inference, miss- ing data, and causality [34 35 44 In particular, the predictinn-ooeeee and active inference estimators closely resemble the augmented inverse probability weighting (AIPW) estimator 35

---

Our work also connects with many areas in machine learning and statistics that study adaptive data collection; most notably, active learning [36 39 and adaptive experimental design [3] 2T. We collect data based on model uncertainty, akin to active learning; however, our objective is statistical inference on typically low-dimensional parameters, rather than prediction. Active testing [26] also involves adaptive data collection, but it pursues different objective of high-precision risk estimation for a fixed model and uses a distinct estimator. Our approach can be seen as an adaptive design assisted by a powerful predictive model, with robustness wrapper for improved performance.
More distantly, our work also relates to robust statisties and robust machine learning 41 42 49]. In particular, our method provides safeguard against poor uncertainty estimation by solving robust optimization problem [ 15].
20
33
1.2 Problem setup
We follow the problem setting from [51]. We observe unlabeled instances XI...., Xn drawn i.d.d from distribution Px, but we do not observe their labels Y;. We use P Px x Pylx denote the joint distribution of (X,, Yi). Our goal is to perform inference for parameter 6 that depends on the distribution of the unobserved labels; that is, the parameter is a functional of P. In particular, assume that 8* can be written as:
0 arg min E ((), where (X,Y)~P
Here, to is convex loss function. This is a broad class of estimands, known as M-estimation, and it includes means, medians, linear and logistic regression coefficients, and more. We have a budget n& on the number of labels we can collect in expectation, and typically < n. To assist in imputing the missing labels, we also have a black-box predictive model f at our disposal.
2 Warm-up: robust sampling for mean estimation
Consider the where H* is the label mean, l" E[y]. The active inference estimator for 0 is given by:
Ôー=
Σ (Xx
E
i=1
Here, (-)) is any sampling rule that satisfies [[[((X) average, and E ~ (X))) is the indicator of whether the label Y, is sampled. Since the number of labeled data points is a sum of independent Bernoullis, a standard Hoeffding argument guarantees that the realized labeling rate will closely match the budget with high probability. Specifically, the labeling ratio will not exceed Ra + with probability e, 6 > 0. As shown in (51), the variance of this estimator is
that the budget constraint is met
6, provided that > (og(1/6) for any
Var (Th)
1 VVr(YY) + Y f(X))²
x
1 T (X)
and the optimal sampling rule is Topt (X.) v to upsample where the model ff makes the largest errors.
In other words, it is optimal
The most straightforward sampling rule that satisfies the budget constraint is the uniform rule: unif (X) na/n. However, if we have access to a good measure of model uncertainty that can serve as a proxy for the model error / /E, f(Xxj) then we can obtain a rule that is closer to Topt. For example, we might prompt a large language model for its uncertainty about X; look at the softmax output of a neural network, and upsample where the uncertainty is high. The issue is that if we severely underestimate the model error, then the estimator's variance can blow up: clearly, π (X,) is small when the actual error (Y; - f(X,))² is large, the variance will be large as well. This is the reason why saw poor performance in Figure
Given any initial sampling rule n, our approach is to find an improved, robust sampling rule n robust that is never worse than either or n uuif By that we mean that the resulting active inference estimator will have a variance that is no worse that with either π or unif used for label collection: Var robuss, min Var(O*), ), Var( ouir
}}.
3

---

2.1
Budget-preserving path
Since our goal is to find a sampling rule robust that performs no worse than T and an arbitrary given π. it is natural to consider a path that connects и and π -"nif while preserving the sampling budget along the path.
Definition 1 BBudget-preservng path), We call continuous path T(0), P E [0. 1]. budger- preserving path connecting π and x zumif if T(1) n, (()) nuiff and ([T(()(() [X]] for all p E [0.
Correspondingly, given point p along the path, compute the estimator @rle) obtained as the active inference estimator ① with sampling rule x(e), The following some examples of valid budget-preservin paths.
Example (Lineerr path). T(() = (1 - p) + pm unif.
Example (Geometric path). π (p) x n² p (T The "xx" hides the normalization factor that ensures (X)] = [X)] for all p.
A natural family of ing the definition of geodesic paths. See Appendix B for general definition of geodesic paths, details of how Examples and Caa be recovered as special cases, as well as further examples.
paths can be recovered via the "least-action" principle, yield-
Of course, if we consistently estimate the optimal point p² [0, 1] along the path, we are guaranteed to find an estimator that outperforms naive active inference and uniform sampling. Moreover, the resulting estimator is still asymptotically normal, which permits the construction of valid confidence intervals. We formalize this key result below in which o² n Var(f=<>)
Theorem 1. Suppose T(e) is presee-pneserving path connecting and quilf Let p" arg min Var Var((=()), and suppose ò = pt +ol(1). Then,
Vn (gmlo) (p0(()-0))) (0,00,)))))
-0+)
where o. S mmnn(op.of of}.
Theorem [ shows that consistently estimating p² will result in an estimator that is no worse than either endpoint. If p* is additionally unique and within (0, 1), then the resulting sampling will strictly outperform both active sampling with æ and uniform sampling. The theoretical results in this paper are asymptotic: however, validity in the sinite-sample regime shown empirically in the experiments in Section 4
It remains explain how estimate ý. Recall from 2 that Var ((=())) L e (X)
C, where
(X)
e² (x) (x))) X and C is quantity that has dependence Zt (e)) Therefore, fit A, we fit an error function é² (.) Re e²)) and solve for the p that minimizes the empirical approximation of Var(ōr(o))
(x.)))
ه = arg min Σ
(x.))
We can find the solution by performing a grid search over P E [0. 1]. The error (²) can be fit on historical or held-out data, it can be gradually fine-tuned during the data collection process. Notice that, if the error estimation consistent in the sense that ( unique, then أ = p², as assumed in Theorem Here, the assumption that é converges to e follows from classical arguments of uniform approximation of flexible estimators, and is common in the field of semiparametric inference. For instance, the widely-used doubly robust estimator [18 (5), which is closely related to our estimator, relies on consistent estimation of nuisance functions.
2 0 and if p" is
2.2 Robustness to error function misspeciffiatioo
Given a path (e), the previous discussion suggests finding ý that minimizes an empirical approxi- mation of the variance Vār(@)(")). This empirical approximation relies error estimate é(-). If

---

this function is severely misspecified, then the computed á might be far from p"; more importantly might not even outperform uniform sampling.
To mitigate this concern, we instead consider a robust optimization problem that incorporates the possibility of é being misspecified:
(X,)) + (X,)
Probust = arg min max eEC
(4)
Here, c= (G.....,G) is the misspecificatiin vector and C is the admissible set of misspecifications. This method allows for setting (e)) close to uniform if the misspecification set C is permissive enough. Solving this minimax problem is computationally efficient, long C is convex set. The outer problem can be solved via a one-dimensional grid search, while the inner problem is tractable due to convexity.
Now, the question is how we should set in practice. Our default will be to simply use C {E : ||e||2 I c), for some hyperparameter > 0. Empirically, can be set by cross-validation. Other choices of the set C are possible, such as bounding other norms of e, for example ||<]| < c. Empirically we found the &g norm to work the best, and in illustrative theoretical examples reach the same conclusion; see Appendix [fror details. We also tried relative misspecification, in the sense that 6, 8² (1 n)) and constrained either the & l norm of the relative perturbation 1). We found that this does not perform as well.
Zmic and Candès [5]] briefly discussed robustness proposal with linear interpolation. It assumes access to historical data, and otherwise it selects a default value for the coefficient, which has no guarantee to outperform uniform and active sampling. Our analysis is far more thorough and sys- tematic, expanding the set of interpolating paths, not requiring historical data incorporating a burn-in period, and adding robustness constraint. These all crucial for the practicality and reliability of the method; see Section 4 for details.
There are other potential optimization objectives to take into account robustness constraints. For example, one may penalize small values of p in the objective (3) with regularization, and similarly use cross-validation to choose the penalty parameter. We leave the investigation of such alternatives for future work.
Robust sampling for general M-estimation
Our sampling principle can be directly extended general convex M-estimation, considered in [5T]. We explain this step-by-step for completeness.
Recall that we consider all inferential targets of the form 6* = arg ming E [X, (X,,X) Y)), for convex loss Eg. Denote la te (X,, Yi). C/ = Le (X,, f (X.)). and define Vla. and Vel, similarly. For an active sampling strategy the general active inference estimator is defined as:
0r arg min (*))), where L= = - Σ co, (Lo.-
C/ 0..
& T T(X,)
(5)
As before, & ~ Bern(T(X,)) the indicator of whether the label Y; is sampled. Following [51] we know that the asymptotic covariance matrix of 9 equals:
Hg- Var (vey- + (Vlg- - veg.) E
n
Hgg
((X)
where Hg. the Hessian Hg. = V (X.
paths (0) and tune the parameter p such that we minimize the variance the resulting estimator )r(0) Denote by Eg and Σ, the asymptotic covariance matrices of the active inference estimator (5) using (0) , and क: (1) = unif respectively.
We again consider
Theorem 2. Suppose (() is budget-preserving path connecting and unif Given a coordinate of interest, let p² arg min Nn."" and suppose p" Olp(1). Suppose further that (r(=") P. 0r.
5

---

Then,
vn -00) S (0,
where Eprai S min (Eo.sh E.
The consistency condition j= E 0 is standard; see the corresponding discussion in [51] and 2). For example, it is ensured when L is convex, such as in the case of generalized linear models (GLMs), or when the parameter space is compact.
As in the case of mean estimation, we fit ý by approximating the variance of the estimator D and searching over p. However, here the notion of error (-) we need estimate is different. In particular, given the form of Σ", we let
(X.))
p = arg min
(X;))
0
where (X) aims to approximate e² (x) = (l((V Veg.) [( and h00 is the j-th column of Hg. In the context of generalized linear models (GLMs), this error simplifies to (X) E f(X)) (x xx Therefore, for mean estimation, the problem
sentially reduces to estimating the error B[(Y- - f(x))?|X]. As before, if e" consistently estimates e², then ý consistently estimates p²
Finally, to protect against poorly estimated errors é, we can incorporate an uncertainty set around the error estimates just before (4) Again, the only difference here is that the é (X,X)''s are esti- mating a different notion of model error tailored to the inference problem at hand.
We summarize our general robust active inference algorithm in Algorithm
Algorithm 1: Robust Active Inference
Input:
unlabeled data X1.... X.. labeling budget 71b, predictive model f. initial sampling rule budget-preserving path (P()) error estimator e.(,., robustness constraint C
1 Solve the minimax problem Probust = arg min max 1 E x()((,))
(X.)+
pE[O,1] 6EC
2 Sample labeling decisions according to (Probam) (X,): E, Bern ( i E [n]]
3 Collect labels (V: E 1}
Output: estimator f (Ptobun)) arg min I (robumt)) defined in Eq. 5
4
Experiments
We turn to evaluating the performance of our robust sampling approach empirically, Each of the fol- lowing subsections dedicated to different experiment using social science research data. Section 4.1 measures presidential approval, Section 4.2 analyzes US age income patterns, and Section 4.3 applies language models to score text on social attributes such as political bias. On each of these datasets, we use the following methods to collect labels: (1) uniform sampling, which essentially recovers rreiccionn-ooereed inference M: (2) standard uncertainty-baa
active sampling [51]; and (3) our robust active method as per Algorithm [ Each dataset will use different base predictive model f. which we describe therein. We set the target coverage level to be 0.9 throughout.
The main metric used for the comparison is effective sample size. To define this metric formally, consider the baseline estimator that samples uniformly random, i.c. according mnif Its effec- tive sample size is simply its budget ng. For other estimators, we say that the effective sample size is equal Bef the estimator achieves the same variance as the baseline estimator with budget neft- For example, if given budget estimator with double the budget, then the estimator has Netff 200. A larger Neff indicates a more efficient estimator. In the case where the effective sample size falls below the budget, Defff ns, the estimator performs worse than the baseline. We show one standard deviation around the effective sample size in all plots.
100 the estimator achieves the same variance as the baseline

---

uniform
active
robust active
Effective sample size
Training 2000
Training size 5000
2480
2456
2083
1766
1796
1490
1536
1257
1313
1061
1123
1597
1880
1357
1597
188D
nb
Figure 2: Effective sample size on Pew post-election survey data, for different dataset sizes used to train f. We compare uniform, active, and robust active sampling, for different values of the sampling budget Ti. The target of inference is the approval rate of presidential candidate.
We also plot empirical estimates of the methods' coverage. We estimate the coverage by resampling the data, constructing confidence intervals for each resampling, and calculating the proportion of times the true parameter value (approximated by the full-data estimate of the target 0*) falls within the constructed intervals. This approach allows us to assess how reliably each method achieves the target coverage level. We resample 500 times to estimate the coverage. (We note that this approach yields conservative coverage estimates when n& is large, because we have n n, "fresh" labels to approximate 8" From the theory, we know that the coverage should be exactly 0.9 for all baselines.
4.1 Post-election survey research
Following [51]. we evaluate the different methods on survey data collected by the Pew Research Center following the 2020 United States presidential election, aiming gauging people's approval of the presidential candidates' political messaging [32] We aim to estimate the approval rate g" EE where Y E {0, 1} is binary indicator of approval of Biden's political messaging. We use a multilayer perceptron (MLP) as our predictive model f- At the beginning, we have a "burn-in" period where we collect all burn-in labels Y; and we use this burn-in data estimate the error function ê(-). Afterwards, we use the fitted function to run robust active inference, as per Algorithm Naturally, the burn-in period counts towards the overall labeling budget 7ib.
We study three questions: (1) the effect of tuning p along the porating robustness constraint C; (2) the effect of tuning along the path and the robust optimiza- tion over combined; and (3) the performance of different peeeee-pres paths.
path, without incor-
Tuning along the pudget-pressrrvin path. First, conduct an experiment without the robust- ness set C, only tuning the parameter ý along the puddee-preserving path. We choose the geometric path from Example 2 To implement active inference,
(x)) min(f(x), - f(x); in which (x) is the predicted probability that the label takes on the value 1, as considered [51. See
Figure for the results. We consider training dataset sizes used to train allowing to the results for a less accurate (left) and a more accurate one (right). We find that, even without robust optimization but only optimizing along the andgetprreeerving path, robust active inference can lead to noticeable improvements in terms of power compared to naive uncertainty-based active sampling and uniform sampling. The performance of standard active inference crucially depends on the quality of f and its uncertainties. We defer the corresponding coverage plots to Appendix
Incorporating robustness. One strategy proposed by Zrnic and Candès [51 is estimate è and set π proportional to è. With this choice, without the additional step of robust optimization, our robust sampling approach would trivially estimate Appendix Aj. We show that incorporating the robustness constraint resolves this issue when (2x) x ê(z). As in the previous case, we use the geometric path and MLP as the predictive model. The results are shown in Figure 3 Recall, è is estimated from the burn-in data. Thus, the longer the burn- period, the better the fit è. This consistent with the observation that active inference gradually outperforms uniform sampling the burn-in period grows. However, when there is little data to fit é, active sampling leads to significantly higher variance than uniform sampling. Our robust sampling approach is never worse than either baseline, across all burn-in data sizes. This is explained by the
0 (a proof of this claim can be found in

---

uniform
active
robust active
Coverage Effective sample size
Bum-in sice
606
Bum--n tice
1215
Bum-in sice
1818
Burn sice - 2426
BurtHin sice 3639
1437
2041
2574
3060
4100
1216
1840
2397
2903
4001
1029
1659
2231
2754
3904
871
5495
2077
2613
3810
1.0
0.7
1574 1775 2003
2001 2216 2455
2515 2704 2906
2946 3108 3278
3968 4068 4151
Figure 3: Effective sample size (top) and coverage (bottom) on Pew post-election survey data, for varying burn-in dataset sizes with respect to different proportions of the data. We compare uniform, active, and robust active sampling, for different values of the sampling budget nag The target of inference is the approval rate of presidential candidate.
hellinger
geometric
linear
Effective sample size
Without robustness constraint
With robustness constraint
3206
3214
2790
2603
2615
2428
2441
2265
2279
2420
2609
2812
2420
2609
2812
пb
Figure 4: Effective sample size for different budget-preserving paths on Pew post-election sur- vey data, without (left) and with (right) a robustness constraint C. In both cases, the geometric path leads to the largest effective sample size. The target of inference is the same as in Figure
fact that, when the fit is poor, the constraint set C chosen via cross-validation is large, resulting in large Probust, thus pushing the sampling rule closer uniform. In Figure (left), we plot the optimized value Probust for different burn-in sizes. As expected, Probust decreases, which means that the optimal strategy gradually moves from uniform sampling toward standard active sampling as the quality of improves.
Choice of budget-preserving path. We have thus far used the geometric path our budget- preserving path. Figure g we compare three budget-preserving paths: the linear path, the geo- metric path, and the Hellinger path (see Appendix B). On the post-election survey dataset, Figure 4 shows that the geometric path is the best of the three chosen paths, regardless whether or not robust optimization over C is used. Therefore, metric path. has been stress-tested and has consistently demonstrated strong performance in our evaluations. We believe this is good tradeoff between simplicity and performance. For improved performance with better choice of path, the practitioner might want to tune it data-driven way; for example, based on the estimated variance small held-out dataset.
practical default, we recommend using the geo-
4.2 Census data analysis
We study the annual American Community Survey (ACS) Public Use Microdata Sample (PUMS) collected by the US Census Bureau [12]. We are interested in investigating the relationship between

---

uniform
active
robust active
Effective sample size
Burn-in size
size
size
size
Bur-in
3233
3418
3468
3382
3591
2284-
3017
1140-
869
737
638
1060
1348
402
542
725
631
1102
пb
Figure 5: Effective sample size on US Census data, for varying burn-in dataset sizes. We compare uniform, active, and robust active sampling, for different values of the sampling budget ny- The larget of inference is the relationship between age and income, estimated via linear regression.
Post-election survey research
Census data analysis
1.0
0.8
0.4
1000
2000
400
800
800
Sizee of bum-in data
Figure 6: Optimized value Probust along the geometric path function of the size of the burn-in data for the post-election survey data (left) and US Census data (right).
age and income in survey data collected in California in 2019, controlling for sex. We estimate the age coefficient 0 of the linear regression vector when regressing income on age and sex. We use an XGBoost model [9] to predict income Y from available demographic covariates. As in the previous problem, we set (x) & ê(z), as in [51], and fit è using burn-in in data. We use the geometric path and incorporate the robust optimization over C. We show the results in Figure Again, we observe that the robust approach outperforms both standard active sampling and uniform sampling for different qualities the error estimate é(-), corresponding to different burn-in dataset sizes. Standard active inference, on the other hand, is very sensitive to the quality of é. In Figure 5 (right), we plot the optimized value Probust for different burn-in sizes. As the previous example, Probust decreases as the quality of é improves, expected. We include corresponding coverage plots in Appendix F
4.3
Computational social science with language models
We study three text annotation tasks used for computational social science research. In each task, have text instances X; and seek collect labels Y, related the text's sentiment, political leaning, and so on. We wish to use a large language model (L.LM) ʃ to predict the high-quality annotations Yt, which are typically collected through laborious human annotation. A natural way of actively sampling human annotations is according to the confidence of the language model [17 (2)). Tian al. [43] propose prompting LLMs to verbalize their confidence in the provided answer, and they find that this results in fairly calibrated confidence scores. Gligorić et al. [17] find that such scores can be useful in actively sampling human annotations. We use GPT-4o annotations and confidences collected by Gligorić et al. [17]. We apply active inference with (X,) & (1 Ct), where C, is the collected confidence score of the language model for prompt X,. This can brittle strategy, since the scores are often overconfident and thus result in very small sampling probabilities. which can blow up the estimator variance through inverse probability weighting. For robust active inference, we use the geometric path and robust optimization with an & constraint set C, as before.

---

uniform
active
robust active
Effective sample size
Political bias
Politeness
Misinformation
1265
3911
1361
987
2779
872
2343
885
770
1975
892
1685
800
908
1019
1142
1976
2378
2861
1010
1138
1281
пb
Figure 7: Effective sample size on social science text annotation datasets. We compare uniform, active, and robust active sampling, for different values of the sampling budget ni. The targets of inference (left to right) the prevalence of right-leaning political bias, the relationship between hedging and politeness, and the prevalence of misinformation.
Political bias. In the first task, the goal is study the political leaning of media articles, using the data curated by Baly et al. [4]. The labels Y are one of left, centrist, or right. The inferential target is the prevalence of right-leaning articles: 0* [ right]]-
Politeness. The next task is to estimate how certain linguistic devices impact the perceived polite- ness of online requests. We use the dataset of requests from Wikipedia and StackExchange curated by Manescu-Nieulescu--ie
et al. [I]] We study how the presence of hedging in the request, polite, Y {0, 1}. Formally, 0 is this effect
Xhedge E {0, 1}. impacts whether text is estimated via a logistic regression with an intercept: logit (P (Y = 1 X )) = 0g+ Xhedge
Misinformation. Finally, we study the prevalence of misinformation in news headlines, using the dataset collected by Gabrel et al. [15]. The labels Y E (0, 13 indicate whether headline contains misinformation. The inferential target is the prevalence of misinformation, 0" E[Y).
We show the results in Figure 7 Across all tasks, the robust approach is essentially never uniform sampling or active inference, in cases even outperforming both by large margin. Standard active inference often leads large intervals, given that sampling directly according the model's verbalized uncertainty leads to instability through inverse probability weighting. We include the corresponding coverage plots in Appendix
than
5 Conclusion
We presented robust sampling strategies for active inference: a principled hedge between uniform and conventional active sampling. By selecting an optimal tuning parameter p along budget- preserving path, robust active inference ensures performance that is no worse than with standard active sampling, and reduces to near-uniform sampling when uncertainty scores are unreliable. Furthermore, the estimator can even surpass standard active inference given reliable uncertainties.
Many directions remain for future work. For example, it would be valuable to understand how to optimally choose the constraint set C, or at least how to choose between several different constraint sets. As presented, our procedure is sensitive the choice of C and may result in sampling rules that are too close or too far from uniform if this set is chosen poorly. We also leave investigations into the optimal budget-press preeeee-ppeeeviinggg path, and practical heuristics for how a practitioner might effectively choose a good path in a data-driven way, for future work.
Acknowledgement
EJC was supported by the Office of Naval Research grant N00014-24-1-2305, the National Science Foundation grant DMS-2013 DMS-2032014, and the Simons Foundation under award 814641.
10

---

References
[1] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zmic. Prediction-powerdd inference. Science, (6266661):669-666 2023.
[2] Anastasios Angelopoulos, John C Duchi, and Tijana Zmni. PPI++: Efficient prediction-
powered inference. arXiv preprint arXiv:331.00455, 2023.
[3] Ruicheng Ao, Hongyu Chen, and David Simchi-Levi. Indictiionguiden active experiments. arXiv preprint arXiv:2411. /20036 2024.
[4] Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, 2020.
[5] A Ben-Tal. Robust optimization. Princeton University Press, 2:35-53, 2009.
[6] Hans-Georg Beyer and Bernhard Sendhoff. Robust optimization-a comprehensive survey. Computer methods in applied mechanics and engineering, 196(33-34) 2007.
[7] Dmitri Burago, Yuri Burago, and Sergei Ivanov. course in metric geometry, volume 33. American Mathematical Society Providence, 2001.
[8] Davide Cacciarelli, Murat Kulahci, and John Solve Tyssedal. Robust online active learning. Quality and Reliability Engineering International, 40(1):277-226, 2024.
[9] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[10] Prateek Chhikara. Mind the confidence gap: Overconfidence, calibration, and distractor effects large language models. arXiv preprint arXiv:2502.11028, 2025.
[11] Cristian Miissau-iiculescu-MMiii,, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christo- pher Potts. A computational approach to politeness with application to social factors. In 51st Annual Meeting of the Association for Computational Linguistics, pages 250-259. ACL, 2013.
[12] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34:6478-6490,
2021.
[13] Valerii Vadimovich Fedorov. Theory of optimal experiments. Flsevier, 2013.
[14] Adam Fisch, Joshua Maynez, R Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-powered inference for hybrid language model evalua- tion. arXiv preprint arXiv:2406. 042.21446..4221,,, 2024.
[15] Virginie Gabrel, Cécile Murat, and Aurélie Thiele. Recent advances in robust optimization: An overview. European journal of operational research, 235(3)441--48, 2014.
[16] Feng Gan. Wanfeng Liang, and Changliang Zou. Prediction de-correlated inference: A safe approach for post-prediotion inference. Australian & New Zealand Journal Staristics. 66(4): 417-440, 2024.
[17] Kristina Gligorié, Tijana Zrnie, Cinoo Lee, Emmanuel J Candès, and Dan Jurafsky. Can uncon- fident IIm annotations be used for confident conclusions? arXiv preprint arXiv:2408, 15204, 2024.
[18] Adam Glynn and Kevin M Quinn. An introduction to the augmented inverse propensity weighted estimator. Political analysis, 18(1):36-56, 2010.
[19] Jessica Gronsbell, Jianhui Gao, Yaqi Shi, Zachary McCaw, and David Cheng. Another look at inference after prediction. arXiv preprint 1719922411.19908, 2024.

---

[20] Yuejun Guo, Qiang Hu, Maxime Cordy, Mike Papadakis, and Yves Le Traon. Robust active learning: sample-efficient training of robust deep learning models. In Proceedings of the 1st International Conference on Al Engineering: Software Engineering for AI, pages 41-42, 2022.
[21] Jinyong Hahn, Keisuke Hirano, and Dean Karlan. Adaptive experimental design using the propensity score. Journal of Business & Economic Statistics, 29(1):96-108, 2011.
[22] Peter J Huber. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribution, pages 492-518, Springer, 1992.
[23] Wenlong Ji, Lihua Lei, and Tijana Zrnie. Predictions as surrogates: Revisiting surrogate out- comes in the age of ai. arXiv preprint arXiv:2501. 09731, 2025.
[24] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873) 2021.
[25] Dan M Kluger, Kerri Lu, Tijana Zmic, Sherrie Wang, and Stephen Bates. Prediction- powered inference with imputed covariates and nonuniform sampling. arXiv preprint arXiv:2501. 18577, 2025.
[26] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, pages 5753- 5763. PMLR, 2021.
[27] Minzhi Li, Taiwei Shi, Caleb Ziems, Min- Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. Coannotating: Sincertainty-uudeeee work allocation between human and large language models for data annotation. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 1487-1505, 2023.
[28] Matthias Liero, Alexander Mielke, and Giuseppe Savaré. Optimal transport in competition with reaction: The hellinger-kannorovich distance and geodesic curves. SIAM Journal on Mathematical Analysis, 48(4):2869-291 2016.
[29] Zachary R McCaw, Sheila M Gaynor, Ryan Sun, and Xihong Lin. Leveraging surrogate outcome to improve inference partially missing target outcome. Biometrics, 1472) 1484, 2023.
[30] Jiacheng Miao and Qiongshi Lu. aasksagnostic machine-leaminn-assee inference. arXiv preprint arXiv:2405.20033 2024.
[31] Jiacheng Miao, Xinran Miao, Yixuan Wu, Jiwei Zhao, and Qiongshi Lu. Assumption- and data-adaptive post-prediction inference. arXiv preprint arXiv:2311. 2023.
[32] Pew. American trends panel (ATP) wave 79, 2020. URL Ut/httpps///ww.peeeeerrr... ocienceedattase aameeaattteeaetiiiiiiieia amerr can trends trends-panel ppanel-wave wwwe-waveeee 79/
[33] Marco Ramoni and Paola Sebastiani. Robust learning with missing data. Machine Learning, 47:147-170, 2001.
[34] James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429):122-129) 1995.
[35] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):846 866, 1994.
[36] Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling esti- mation of error reduction. In ICML, volume 1, page 5. Citeseer, 2001.
[37] Donald B Rubin. Multiple imputation. Flexible imputation of missing data, second edition, pages 29-62. Chapman and Hall/CRC, 2018.
12

---

[38]
Vivien Seguy and Marco Cuturi. Principal geodesic analysis for probability measures under the optimal transport metric. Advances Neural Information Processing Systems, 28, 2015.
[39] Burr Settles. Active learning literature survey. 2009.
[40] Shanshan Song, Yuanyuan Lin, and Yong Zhou. A general M-estimation theory in semi- supervised framework. Journal of the American Statistical Association, 119(546);0065-0775, 2024.
[41] Robert G Staudte and Simon Sheather. Robust estimation and testing. John Wiley & Sons, 2011.
[42] Jacob Steinhardt. Robust learning: Information theory and algorithms. Stanford University, 2018.
[43] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting cal- ibrated confidence scores from language models fine-tuned with human feedback. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433-5443 2023.
[44] Anastasios A Tsiatis. Semiparametric theory and missing data, volume 4. Springer, 2006.
[45] Siruo Wang, Tyler H McCormick, and Jeffrey T Leek. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences, 117(48):30266 30255 2020.
[46] Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Transfer learning from deep features for remote sensing and poverty mapping. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[47] Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu. Calibrating the confidence of large language models by eliciting fi- delity. arXiv preprint arXiv:2404.02655, 2024.
[48] Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. arXiv preprint arXiv:2302.13439, 2023.
[49] Abdelhak M Zoubir, Visa Koivunen, Yacine Chakhchoukh, and Michael Muma. Robust esti- mation in signal processing: A tutorial-style treatment of fundamental concepts. IEEE Signal Processing Magazine, (9(4):11-8, 2012.
[50] Tijana Zmic. A note on the mredccionn-oweeeee bootstrap. arXiv preprint arXiv:2405.18373 2024.
[51] Tijana Zmic and Emmanuel J Candès. Active statistical inference. In Proceedings of the 41st International Conference Machine Learning, pages 23993-63010, 2024.
[52] Tijana Zrnic and Emmanuel J Candès. pross-rreiiciioo--poerrdd inference. Proceedings of the National Academy of Sciences, 121(15):c232208311), 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. The claims made match theoretical and experimental results, and reflect how much the results can be expected generalize to other settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations and future work discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitttions" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asympiotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification:
See Theorems in Section and 3 with proof' provided Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification: Experiments introduced in detail in Section 4 with all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conelusions of the paper. Additional experiments are included in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Open access data and code are provided in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. cc/ htt./uullcccee ppppli//guides//////pppp Cuuee//mddeuubbim SecodeSuumissiooooolo...o. ssionPolccy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiments introduced in detail in Section with all the training and test details necessary to understand the results. Additional experiments are included in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: All the methods introduced in the paper are assessed with confidence inter- vals, and achieved the prescribed statistical coverage level, with results shown in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: was mentioned in Section that the algorithm is computationally efficient and cheap.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive impacts are discussed throughout the paper. There are no negative societal impacts of the work performed.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited all used datasets appropriately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, wwtppppppppehooooo cow com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: Included in supplementary file. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy huttse//beuriigscc//ooteeeeeeee for what should or should not be described.
20

---

A Proofs
A.1 Proof of Theorem
By the definition of galm)
we have
G
10 Σ 1=1
& x(e) (X:)
From the assumption, we have = 0² + op(1). By the continuity of the budget-preserve path x(P), it follows that (X,) = x ) (X,) + o1( for any i E nl.....n). This, as result, gives
(r(m) = -0=10"> + (1) by the continuity (+10)
It follows from Proposition I in [5I] that we have
Vnl (x - 00) S N (0,0%)) og = -Var(0zlo))
Since (z(#) In Ôr(0")
vnl (Brl - 4 N (0,0%))
By the definition of p², p² arg min Var( ) we have
of = Var(@m S min (ar(( Var(O="))) min
This completes the proof.
A.2 sufficient condition for X p" + (1(
Proposition 1. Suppose ex) = (x) + (1(,), and p² is unique. Suppose É(X) is uniformly upper bounded by M > 0. Suppose further that (X) is uniformly lower-bounded by > 0, then we have ý = p² + (1).
Proof. Denote
(x) F f6(x)= (x) : e [0. 11
We first show that F is P-Glivenko-Canttlli class.
Since (P) is continuous, and supported on [0, 1), is uniformly continuous on [0, 1]. Hence for any
5 > O, there exists 17 > 0 such that | (s(()) (X) 7(p2) (x) I m² 5 whenever [p1 Pal I 17. Now, we cover [0, 1] with grid
Pa < P1 <
PK 1, where Pk Pk-1 = n7 for $ K 1.
Then, for any E [PA [Ph-1, PR). we have
(x)
(xxx
2
15p(x) Sm (x) (x)
(0) (x)) -(0=-1))>))))) 5o.
T 1) (x)
m²
Hence [fr, - 8, 1 + 8] is an 26-bracket in L, (P) that contains every fo with p [Pa-1 Px]- So the bracketing number Nn is finite, (2, L. < K S + + < co. We thus conclude from the Blum-DeHard DeHardt theorem that F is P-Glivenko-Cami
class. Consequently, we have
(X.)
E (XX)
E
sup I
(X,,
xX)
This implies that
1 n
82 (Xi)
inf (X))
inf
P
PE(0,
x| (X,)
eE
x(p((
21

---

Ei=1 (xx)) By definition, أ arg min 1 a xx,(
Denote S arg min E (x) (x) Then, by continuity
of (X)) have d(p, 2 for d(p, inf(ll Now, for any / E S, we have
101 -nf p- E ES..
(X)
(X) + o1(()
E n (X)
(X)
(X)
x (X) +pp(1) (x)
(X) + (1) + O1((1) x(0)(x)
1
(X)
(x) + 0
(())(X)
[(()(()) (X
(X)
(X) + +0((1).
Since
(X)
Var = =E
+ +C,
(x)
where a constant independent of P. we have
Var (0 (7))
3 Var(ônln>) + +Op(1).
On the other hand, by the definition of p"
Var(@n ) 2 Var(Oulo')
also holds. Whence Var((=((") I Var(@rlo")
Since p² the unique minimizer Var( ), ir Ln p² by continuity, Since d(p,s) 2, 0 and 6 is an arbitrary element in S, we immediately conclude that
p P p:.
A.3 Proof of Theorem 2
By the definition of gr())
we have
g
1 arg min - Σ
el 0,6
Ei + (l0:: (lo.)) (l).) T(p) (X,)
We assume à = p² + o1) By the continuity of the budget-preserving path nt (()), it follows that for any i [I....,n.. This, result, gives @z(1) ( +1p(1)
N (X,) = =X))
& by the continuity of e/, + Co.s 1/.) (X,) with respect to A.
Given the assumption that ga('n)) Lg 8", from Theorem in 51 we have
vn (gxl0) - -8r) 2 (O,E,))
where Σp. -Hg. Var (veg + veg.,) ) H Since (=())) Pg gr)0n)
Vni (p)(0) -00) $ (0(00,)))))
22

---

The definition p² = arg min Dng yields
min Ej "3j") min f Eo E1s1]-
This completes the proof.
A.4 Setting स x è leads trivial choice of ý 0 when not incorporating robustness constraint
Starting from the variance estimate used in the optimization objective
n
(Xx))
β arg min
n
π
(X,)
by the Cauchy-Schward Snequauiiyyu for any such that constraint), Ei=1 (x,)) (E ((X.)) (E a(x.))
Di=1
(X:) no (i.e. satisfying the budget
The equality holds when æ x é, which
T(X;) x
n
corresponds to (e) with p = O.
B A natural family of budget-preserving paths
Among the diverse set of possible paths 28 38 it is natural to consider geodesic paths, which a family of "shortest paths."
- M from an interval C R to a metric space M with metric is geodesic if there a constant v 2 0 such that for any & there a neighborhood
Definition (Geodesic 77). A curve
of in such that for any P1. E J we have
( (01), (P2)) = v P1 - PP2--
We revisit the examples from Section 2 and provide more geodesic paths.
In all the following examples, we assume P and Q have the same support.
Example (Linear path). The linear path, x(p) (1 p)T + pm prrumif the geodesic path with respect to d) = |P QlI with T ==nif Here, || is any norm.
Example 4 (Geometric path). The geometric path, (e) x π x--0 " ye, is the geodesic path with respect to d(P, Q) = || log p log Q|I with u = || log म - log x uni Here, log is taken element- wise.
Example 5 (Hellinger path). The Hellinger path, x(e) x ((1 p) pv T unif
is the geodesic Here, the square root is path with respect to d(P,Q) |VP vQI with lI umiff taken element-wiss. wise.
Note (more examples), Some distance metrics may not have an analytical characterization for their corresponding geodesic path, such as the Wasserstein and Lonsen-Shanoon distances. However, it is computationally tractable to solve for geodesic path numerically up to a tolerance margin for many well-defined distance metrics. For example, when computing the geodesic for the Censen-Shannon distance, we can discretize the interval [o, 1] into N segments so that Po = P and PN = Q. and we define series of intermediate distributions Pt, P....., PN- The task is then cast as an optimization problem: we minimize the total path length computed as the sum of the square roots of the Jensen-Shannon divergences between successive distributions, i.e., NN /JS(P,, Here, QS(P)IQ) M) + [(Q])),) where M ((P + Q). This is constrained optimization problem and can be solved by standard gradient-based methods.
B.1 Uniqueness of
In Section 2 we saw that the uniqueness the optimal p* and the consistency of é are sufficient conditions for the consistency of p. In the case of all three budget-preseving paths from the previous section, it can easily verified by computing the second derivative of (ar(@r(*) that this variance is strictly convex and thus p² is unique. We include the corresponding proofs for completeness.
23

---

Linear path. equivalent to
We have (X) (1 - (X) + = The problem of minimizing Var (m) is
(Y - )(x)) (1 (1- + Pm
arg min
Denoting s(p) = E
(XX)
we have
(1 (1-p)=(X)+p
d'()) = - (#1 - -())
-))-)) +p )²)
and
(0) = 2(Y - f(x)) (nt - xxx)) 27 ((1-P) + p pm)
Clearly, (0) > 0, which means that ( is convex. Hence, there is unique optimal value of p in [0, 1].
Notice that (1) Th [(Y [(X) (x()) }) Hence, if E [(Y - [(X))3x(X)] >
E [Y - [( then g' g'(1) > O, which implies that the optimal p lies in [0, 1).
Geometric path. Consider the path (X) & P( unu yi in particular, 7 x(p((X)
=(X)==e []
(V-S(x)))
Y ((x)" EE [x(())=_]]
Similar to the last example, we denote g(p) E. Then, have
AE
(()
nb x(X)
(Y- - f(X))²
(Y - 2(x))
g'(p)
n -E E
x(X)1-p log T (X)) E [-]
"E E
n
[T(X²² log ग (x)]
x(X)1-P
11b
and
(Y- - F(X))
(Y f(x))
(p(p)
E
( log² π (X)
E [[T(X)'-
+ n
E [x((X) ~log" xxx)
x(X)1-p
T/b
nb
[Y - -(X))²
2 EE
T(X)1-P 1-p log क (X) E [x))) log (X)]
7bb
Since (Y- f(x))) 2 0, x(X) > and log² T(X)= 2 O, we have that
(Y - f(X))²
(Y - )(X))
E
x (X)1-PP log xXx) E [[T((X)=-]
E
E [x(X) log (X)]
(Y (Y-1(X))2
(Y 7 ((x)2
22
T(X)1-p log² T T(X)
E [F
[[T(X)]-> log2 π T(X)]
x(x)=pP
(Y - -(()))
(Y - )(X))²
log TX) x(X)1-p
E
E [[x(X)2-p) [ log क (X)]
x(X)1=p
122 (Y- (x-)2xx)2
22
log T (X)
E2 (xx)-- log (X)]
x(X]1-p
(Y f(X))2
T(X)=pp log (X) ([X
24

---

The last inequality follows from the Cauchy-Schwarz inequality. Therefore, we have g" 2 0. Further, if क (x) * uuiif the inequality strict, which means g convex. Thus, there is a unique optimal value of p 10.
Hellinger path. Suppose P and Q are two discrete distributions. The Hellinger distance between P and Q is H(PQ)) = V2 IVp VQl22. The geodesic connecting π (X) and x unif is:
xlr)
(sin((1 p).3) sin 3
(X)
sin (p3) sin B
1bb n
where 3 arccos
Lim1
x(Xx) Yib
n
Similarly as above, minimizing the variance Var( amounts to minimizing the function
(Y f(X))²
g(p) = EE
sinn([I-n) ()))
(
sin(p8)
TF
2
sin
sin 8
over p. The derivative g'(p) is given by
2E (Y - f(X))²
(sin((1- P)3) sin
(X) +
sin (p3)
-3
cos( (1 p)33
V (X)) +
cos(p3) sin 3
71b
sin B
sin
while the second (p) is given by
sin((1 p)3)
sin(pB)
1 ( cos((1 p)13) (x(
E
(Y f(x))²
sin B
(X)
sin 3
V
6
sin B
I1 >0. 0.
c3) sin B
'nb + 23
[ii[(11p]pp - p)B)
sin(p
nbb
+
V
sin 3
(X)
sin
V n
Therefore, g(p) strictly convex, and there unique optimal value of p in [0, 1].
C Perturbed model errors after robust optimization
It is natural choose the constraint C by upper-bounding the norm ef Our default choice is the l2 norm, i.e. ||e||2 I c. The le norm can be roughly thought of as controlling the variance of the enrors in e², In particular, imagine (X:) can be viewed noisy version of ex.)): (x.)) (X,) + &is where the (X,, Ex) pairs are i.i.d. and & have mean zero. Then, by concentration, |e = Σ, Var(S.).
In Figure g we illustrate how robust optimization over the Eg set C recovers errors (²(,,) + that are much closer to (X::) than simply using (X()).
D A toy example: choice of C
A simple & norm constraint may not always be the most powerful choice of C. Zooming out, our method can in principle combined with any choice of C, including one where we learn regions of the space where scores systematically overconfident undercontident. At high level, our method (1) learns C' (in our experiment, the "learning" is a simple fitting of a through cross-validation), and (2) solves robust optimization problem with C' in place. Your suggestion is an interesting choice of step (1).
We developed dataset featuring a central "hard" region ([X] < 2) flanked by two "easy" regions (2 < |X| < 5). In the easy regions, error data was sampled from A22, 0.05). the hard region, error was drawn from (1, 0.25). The estimator of error, é(X), is designed to underestimate the
25

---

Truo
Robust emor (12)
Estimated
Index
Figure 8: Perturbed errors e² (X;) + c, naive errors e² (X;;) with la constraint C. We consider a regime where we underestimate the true error (for example, due to the model being overconfident). We let e(X,) (5, 0.25) and ē(X,) ~ N/,,, 0.25), and π(0) the linear path with p = 0.5. The robustness constraint is C { ||e|l I 50}. Each index i corresponds to sample X,. The robust error (green bar) is the error after perturbation, (X;) + G, and the estimated error (blue bar) the error before perturbation, 6² (X;). The robust errors much closer to the estimated errors.
error in the hard region and overestimate the error in the easy region. Specifically, ē(X) 0.5 for [X| $ 2, and ē(X) = 2.5 otherwise.
Subsequently, we trained meta-classifier, a gradient boost classifier, (XX), to identify these regions solely based on the performance of ((X), without prior knowledge of the region boundaries.
This approach proved highly effective, with the meta-classifier achieving over 99% accuracy in identifying the regions. This demonstrates our success in learning the error regions and enables us to separate the constraint set C' based on these distinctions. For instance, C can be defined and ||eaad]|2 < Chard for the hard region ([X < 2). Or even simpler, we can only optimize over hard regions, i.e. Geasy = O. While these regions' dimensions are fixed and depend on X, this presents no practical difficulties because we have complete information about X.
||eeasy| 2 Censy for the easy region (2 |X]
Next, compared this structured constraint with the global constraint. Here, for the structured constraint, we only optimize over the hard region. The following table shows the result when n 7000, n4 1400, and x è
Method
ESS
ESS Gain (%)
Uniform
1400
0.00%
Active
1213
-13.3%
Robust active (global)
1491
6.5%
Robust active (structured)
1495
6.8%
We found that incorporating the structured constraint provided a slight gain in ESS over the global constraint while reducing the constraint size (Cglohal 85 vs. Chard 75). This suggests that a more focused perturbation can be beneficial when we have strong knowledge of confident regions. How- ever, we note that the global constraint remains simple and practical approach given the limited gain.
26

---

E
Additional experimental results
E.1 Plots with coverage
In this subsection, we provide figures corresponding to the figures in the main text, where in addition to the effective sample size we also plot coverage.
uniform
active
robust active
Coverage Effective sample size
Training size=2000
Training size=5000
2448
2423
1773
1803
1509
1555
1285
1341
1094
1156
931
997
1.0
0.9
0.8
0.6
1386
1666
2003
1356
1596
1880
Figure 9: Effective sample size and coverage on Pew post-election survey data, for different dataset sizes used to train f. We compare uniform, active, and robust active sampling, for different values of the sampling budget nis The target of inference is the approval rate of presidential candidate.
hellinger
geometric
linear
Coverage Effective sample size
Without robustness constraint
With robustness constraint
3206
3214
2991
3000
2790
2801
2603
2615
2428
2441
2265
2279
2113
2128
1.0
0.9
0.8
2246
2420
2609
2812
2420
2609
2812
Пр
Figure 10: Efffetiiee sample size and coverage for different budget-preserving paths on Pew post-election survey data, without (left) and with (right) robustness constraint C. In both cases, the geometric path leads the largest effective sample size. The target of inference the same as in Figure
27

---

uniform
active
robust active
Coverage Effective sample size
Bum-in size
140
Bum--in size 280
Burn-in size
420
Bum-in size 560
B size 840
3233
3418
3446
3382
2284
2515
2657
2676
3017
1614
1850
2048
2118
2467
1140
1361
1579
1676
2017
569
737
938
1060
1348
1.0
0.9
0.8
0.7
1728 1958
2218
1443 1744
2109
1578 1874
2230
1478 1773
1746 2036 2375
Пb
Figure 11: Effective sample size and coverage on US Census data, for varying burn-in dataset sizes. We compare uniform, active, and robust active sampling, for different values of the sampling budget ng. The target of inference is the relationship between age and income, estimated via linear regression.
uniform
active
robust active
Coverage Effective sample size
Political bias
Politeness
Misinformation
1265
3911
1381
1117
3297
1238
987
2779
1110
872
2343
995
770
1975
892
680
1665
800
601
1404
717
1.0
0.9
0.8
810
908
1019
1142
1976
2378
2861
1010
1138
1281
Пр
Figure 12: Effective sample size and coverage on social science text annotation datasets. We compare uniform, active, and robust active sampling, for different values of the sampling budget ng. The targets of inference are (left right) the prevalence of right-leaning political bias, the relationship between hedging and politeness, and the prevalence of misinformation.
28

---

uniform
active
robust active (geometric)
robust active (linear)
Effective sample size
2327
2831
3245
3680
4448
2002
2527
3007
3411
1722
2258
3250
1481
2014
2581
3097
4114
1274
1798
2391
2951
4008
1432
2053
2680
3805
1.0
Coverage
0.6
1417 1578 1767
2047 2191 2346
2417 2636 2808
2917 3062 3194
3986 4094 4204
nb
Figure 13: Effective sample size (top) and coverage (bottom) on Pew post-election survey data, for varying burn-in dataset sizes with respect different proportions of the data. We compare uniform, active, and robust active sampling with geometric and linear paths, for different values of the sampling budget ni. The target of inference the approval of presidential candidate.
E.2 Burn-in size v.s. robustness constraint C
In addition to optimized Probust along the path, we also provided the optimized value c in the ro- bustness constraint C {e ||e||2 S c). As expected, we observe more conservative constraint when the errors are poorly estimated.
Post-diection survey research
500
1000
1500
2000
2500
3000
3500
Size of bum-in data
Figure 14: Optimized value e along the geometric path as a function of the size of the burn-in data for the post-clection survey data.
E.3 Sensitivity step size
we employ a grid search for p in the outer loop. We conducted experiments to explore different step sizes of the grid search and confirmed the robustness of our results to step-size selection, as shown below. The ESS gap between these two estimators minimal, and both significantly outperform uniform and active baselines.
When we solve the optimization problem 4
29

---

uniform
active
robust active (step_sizzer size=0.01)
robust active (step_sif Shep_size=0...
Effective sample size
3430
0.9
3263
Coverage
3104
0.8
2953
2945
3107
3277
2945
3107
3277
Figure 15: Effective sample size on Pew post-election survey data, for different step sizes in grid search for p. We compare uniform, active, and robust active sampling with grid search step sizes of 0.01 and 0.1. The target inference the approcal rate of presidential candidate.
30


Reviews:
[{"review_id": "dk6Onqx15Z", "reviewer": "Reviewer_hhQ5", "summary": "In this paper, the authors propose a sampling scheme for statistical inference that is a mixture of uniform and uncertainty based sampling schemes. The authors note that the uncertainty based sampling depends on the quality of the model’s uncertainty estimates, and can even perform worse than random sampling when these estimates are poor. Subsequently, they propose a robust scheme that guarantees the performance would be bounded by that of uniform sampling. Results seem to indicate that the proposed scheme performs well with a range of qualities of uncertainty estimates.", "questions": "See strengths and weaknesses", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The central idea of using error function estimates along a path connecting the two sampling schemes is simple and elegant. The main theorems are clearly stated and the paper does a good job of building intuition first with a concrete example (mean estimation in section 2). I had a few concerns and observations but I am somewhat inclined for the paper to be accepted:\n\n1. I am not entirely convinced how practical this method might be given its reliance on some hand-picked hyperparameters, namely the set of misspecification (C) and the choice of path. While the choices made in the paper seem to work well, they do not sufficiently convince me how the results might generalize.\n2. Relatedly, in Fig 3, I am not sure how much of the performance difference can be attributed to the overall scheme vs. just the choice of path. Just eyeballing Fig 4 against Fig 3, it appears the choice of path leads to a substantial difference in performance, with a similar magnitude to the performance difference in Fig 3. It would be useful to see where, say, a linear path lands in Fig 3. Essentially some more ablation studies to really understand the benefits of the proposal.\n3. For the choice of C: the l_2 norm constraint doesn’t seem very intuitive. What if the error estimates are off systematically? We might suspect such systematic differences for the regions where the model is uncertain vs. where it is fairly certain. Would this still bound the overall variance? Would it likely just converge to uniform sampling in practice?\n4. Minor: in Fig 2, consider using the same scaling for both figures so it’s easy to quickly compare the two. You could even combine the two figures and just use different formatting for the curves (-- vs .-). In line 214, “we collect all labels Y_i”, I presume you mean just the burn-in labels. In Fig 3, curious why are we using such odd burn-in sizes?", "was_revised": true, "final_justification": "I want to thank the authors for addressing all my questions and running additional experiments to address issues I pointed out as weaknesses. With this additional context, I am raising my score from a \"borderline accept\" to \"accept\".", "rebuttal": "Thank you for your thorough and constructive feedback. See below for our responses to your points, and please let us know if these responses adequately address your suggestions.\n\n--- \n\n>“I am not entirely convinced how practical this method might be given its reliance on some hand-picked hyperparameters, namely the set of misspecification ($\\mathcal{C}$) and the choice of path. While the choices made in the paper seem to work well, they do not sufficiently convince me how the results might generalize.”\n\n\nWe appreciate this comment. We have tried several options for both the path and the misspecification set, and across many datasets (including varying data modalities) and uncertainty measures. We have found that the method is reliably never worse than uniform sampling, and often outperforms uniform sampling and active inference by a substantial margin. We have done our best to stress test the methodology; if you have suggestions for additional experiments for validating these observations, we will gladly run them.\n\nWe would also argue that two parameters (misspecification set and choice of path) is not an impractically large number of user-chosen parameters. In our extensive evaluations we have found the $\\ell_2$-constraint and geometric path to be simple and lead to good performance. This is why, to maximize practicality and ensure running the method is as streamlined as possible, we recommend them as a default.\n\n--- \n\n\n> “Relatedly, in Fig 3, I am not sure how much of the performance difference can be attributed to the overall scheme vs. just the choice of path. Just eyeballing Fig 4 against Fig 3, it appears the choice of path leads to a substantial difference in performance, with a similar magnitude to the performance difference in Fig 3. It would be useful to see where, say, a linear path lands in Fig 3. Essentially some more ablation studies to really understand the benefits of the proposal.”\n\n\nThank you for this insightful suggestion. Our core claim is that our procedure consistently yields an estimator that performs no worse than either the active or uniform estimator. However, we acknowledge that the degree to which it outperforms these baselines is highly variable, depending on the specific datasets, paths, and models used.\n\nTo address your question, we conducted new experiments comparing the geometric path, linear path, and the baselines (active and uniform) in the setting of Figure 3. The results, presented below, demonstrate that while the linear path does not achieve the same performance as the geometric path, it consistently performs no worse than either of the two baselines. This finding supports the above claim. We will add these new results to the revised version of the paper. Due to the restriction on uploading images/plots, the results are presented in a table format.\n\nBurn-in size 606:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  1396    |            1511               |             1434         |     1195   |   1396    |\n| 1518    |                 1652          |              1569         |     1264   |     1518   |\n| 1760    |                  1932         |              1836         |     1422   |     1760   |\n| 2003   |                    2205       |               2097         |    1652   |     2003   |\n\nBurn-in size 1213:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  1807    |            1896               |          1844            |     1744   |   1807    |\n| 2023    |                 2142         |               2073        |     1954   |     2023   |\n| 2239    |                  2393         |               2286        |     2058   |     2239   |\n| 2455   |                    2641       |             2519           |    2218   |     2455   |\n\nBurn-in size 1819:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  2340    |            2418               |            2358           |     2288   |   2340    |\n| 2529    |                 2620          |               2553         |     2481   |     2529   |\n| 2718    |                  2854         |                2765        |     2646   |     2718   |\n| 2907   |                   3066        |             2963           |     2823   |     2907   |\n\nBurn-in size 2426:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  2793    |            2866               |         2840          |     2796   |   2793    |\n| 2955   |               3064          |                3033       |     2982   |     2955   |\n| 3117    |               3265         |              3189        |     3131   |     3117   |\n| 3279   |                3458        |             3364        |     3287  |     3279   |\n\nBurn-in size 3639:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  3881    |            3963               |           3938        |     3923   |   3881    |\n| 3935   |               4024         |              4022         |     4010   |     3935   |\n| 4043    |               4166         |               4147       |     4126   |     4043   |\n| 4151   |                4308        |               4285      |     4251  |     4151  |\n\n\n\n--- \n\n> “For the choice of $\\mathcal{C}$: the $\\ell_2$ norm constraint doesn’t seem very intuitive. What if the error estimates are off systematically? We might suspect such systematic differences for the regions where the model is uncertain vs. where it is fairly certain. Would this still bound the overall variance? Would it likely just converge to uniform sampling in practice?”\n\n\nWe agree that a simple $\\ell_2$ norm constraint may not always be the most powerful choice of $\\mathcal{C}$. Zooming out, our method can in principle be combined with *any* choice of $\\mathcal{C}$, including one where we learn regions of the space where scores are systematically overconfident or underconfident. At a high level, our method (1) learns $\\mathcal{C}$ (in our experiment, the “learning” is a simple fitting of $c$ through cross-validation), and (2) solves a robust optimization problem with $\\mathcal{C}$ in place. Your suggestion is an interesting choice of step (1).\n\nWe developed a dataset featuring a central \"hard\" region ($|X| \\leq 2$) flanked by two \"easy\" regions ($2<|X|<5$). In the easy regions, error data was sampled from $\\mathcal{N}(2, 0.05)$. In the hard region, error was drawn from $\\mathcal{N}(1, 0.25)$. The estimator of error, $\\hat{e}(X)$, is designed to underestimate the error in the hard region and overestimate the error in the easy region. Specifically, $\\hat{e}(X) = 0.5$ for $|X| \\leq 2$, and $\\hat{e}(X)=2.5$ otherwise.\n\nSubsequently, we trained a meta-classifier, a gradient boost classifier, $h(X)$, to identify these regions solely based on the performance of $\\hat{e}(X)$, without prior knowledge of the region boundaries.\n\nThis approach proved highly effective, with the meta-classifier achieving over 99% accuracy in identifying the regions. This demonstrates our success in learning the error regions and enables us to separate the constraint set $\\mathcal{C}$ based on these distinctions. For instance, $\\mathcal{C}$ can be defined as $ \\\\| \\epsilon_{\\mathrm{easy}} \\\\|\\_2 \\leq c_{\\mathrm{easy}} $ for the easy region ($2<|X|<5$) and $\\\\|\\epsilon_{\\mathrm{hard}}\\\\|\\_2 \\leq c_{\\mathrm{hard}} $ for the hard region ($|X| \\leq 2$). Or even simpler, we can only optimize over hard regions, i.e. $c_{\\mathrm{easy}}=0$. While these regions' dimensions are not fixed and depend on $X$, this presents no practical difficulties because we have complete information about $X$.\n\n\nNext, we compared this structured constraint with the global constraint. Here, for the structured constraint, we only optimize over the hard region. The following table shows the result when $n=7000$, $n_b=1400$, and $\\pi \\propto \\hat{e}$. \n| Method | ESS | ESS Gain (%) |\n|-------|---------------------------|------------------------|\n| Uniform | 1400 | 0.00% |\n| Active | 1213 | -13.3% |\n| Robust active (global) | 1491 | 6.5% |\n| Robust active (structured) | 1495 | 6.8% | \n\nWe found that incorporating the structured constraint provided a slight gain in ESS over the global constraint while reducing the constraint size ($c_{\\mathrm{global}} = 85$ vs. $c_{\\mathrm{hard}} = 75$). This suggests that a more focused perturbation can be beneficial when we have strong knowledge of confident regions. However, we note that the global constraint remains a simple and practical approach given the limited gain.\n\n\n\n\n--- \n\n> “Minor: in Fig 2, consider using the same scaling for both figures so it’s easy to quickly compare the two. You could even combine the two figures and just use different formatting for the curves (-- vs .-). In line 214, “we collect all labels $Y_i$”, I presume you mean just the burn-in labels. In Fig 3, curious why are we using such odd burn-in sizes?”\n\n\nThis is a helpful suggestion. We have revised Figure 2 as suggested, aligning the x-axes. Yes, \"we collect all labels $Y\\_i$\" refers to the burn-in labels; we will rephrase this sentence for clarity. The burn-in sizes in Figure 3 are set by the proportion of the burn-in data to the whole data (e.g., 10% for a burn-in size of 606), which explains their seemingly unusual appearance. We will clarify this in the revised version.\n\n---"}, {"review_id": "9cOAhanCtb", "reviewer": "Reviewer_svoR", "summary": "This paper proposes a methodology for performing appropriate labeling in situations where labeling each instance is expensive. While classical active learning methods adaptively determine where to perform labeling and place emphasis on prediction, this paper places emphasis on statistical inference of the model associated with data observation. Existing active statistical inference based on sampling from a random distribution suffers from the problem that when the error value is large, i.e., the sampling probability is small, the results are worse than uniform sampling. In contrast, this paper proposes a robust sampling rule that lies between uniform sampling and active statistical inference, bridging the two. Inference based on the proposed method has been shown to have a smaller asymptotic variance than that of uniform sampling and active statistical inference under appropriate assumptions. Additionally, numerical experiments have been conducted to compare the proposed method with uniform sampling and active statistical inference.", "questions": "-Questions\n\nPlease clarify the following unclear points. If these unclear points are appropriately revised, I will raise the score.\n\n1. Since the algorithm randomly decides whether to observe each instance, it is possible that all instances are observed in extreme cases. Does this paper allow the budget $n_b$ to be exceeded? If so, can the same claims as in this paper be made while keeping the total number of observations within $n_b$ by adaptively changing the sampling distribution for each observation, similar to active testing (Kossen et al., 2021)? \n\n2. In lines 9, 101, and 297, is this claim asymptotic? Does it hold for finite samples? \n\n3. The explanation of the inconsistency in the axis scales in all figures, including Figure 1, and how to read the figures should be clarified.\n\n4. References to related research. The differences between active testing and the present setting should be mentioned. In addition, the issues with performing statistical inference after classical active learning should be mentioned. My understanding is that the bias caused by not randomly selecting instances is the problem. Is this correct? If so, related research on statistical inference after decision-making based on statistical models, such as post-selection inference (Lee et al., 2016), should be mentioned to see if the bias of active selection can be corrected. Perhaps this is difficult because the selection of instances by active learning does not fit into the framework of post-selection inference. Is that correct? \n\nLee et al., Exact post-selection inference, with application to the lasso, The Annals of Statistics 2016\n\n\n5. It is assumed that $\\hat{e} (\\cdot)$ converges in probability to $e (\\cdot)$, but I am not sure if this assumption is strong. Are sufficient conditions given?\n\n\n-Comments\n\nThe following points are not necessarily required to be addressed, but they caught my attention. \n\n1. Is $y$ limited to 0 or 1? Can this be extended to general distributions? \n\n2. If $\\mathcal{C}$ is an $l_2$-ball and the radius is reduced to 0 (as $n \\to \\infty$), can the same theorem be derived? \n\n3. I think it would be better to clarify what the expected values refer to, such as line 85 and Equation (2). I think that expected values of $X$ and those of $(X, Y)$ are mixed together.", "limitations": "Yes. However, as I understand it, it seems that exceeding the budget is permitted, so the possibility that strict budget constraints may not be met should be added to the limitations. On the other hand, this study proposes a new  method for robust statistical inference in active learning, and therefore does not fall under potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper are as follows: \n\n1. It solves the problem of existing active statistical inference, which is that it performs worse than uniform sampling when the error is large, by appropriately linking the two. \n2. It extends to general M-estimators. It also extends to robust estimation by using the admissible set $\\mathcal{C}$ to allow for the effects of misidentification that arise when estimating the error. \n\nOn the other hand, the weaknesses of this paper are as follows:\n\n1. Insufficient reference to related research. The relationship and differences with active testing (Kossen et al., 2021) should also be mentioned. In addition, the problems of statistical inference when using conventional active learning (e.g., selecting from the most uncertain points) should also be mentioned. \n2. Lack of clarity in the problem setting and experimental results. If my understanding is correct, the proposed method seems to allow exceeding the prior budget. Furthermore, the explanation of the strength of the assumption that the estimate $\\hat{e} (\\cdot)$ in the algorithm converges in probability to the error $e (\\cdot)$ is insufficient. In addition, the intervals between the numerical values on the axes of the figures in the numerical experiments are not consistent, making it difficult to determine what these figures represent. As a result, the effectiveness of the proposed method cannot be correctly confirmed by numerical experiments.\n\n[Quality: 3]\nThe proposed method provides a theory on the asymptotic variance of estimators using an algorithm and extends it to general M estimators. Furthermore, it proposes a robust method for the effect of misidentification of the error function. Additionally, it mentions the issues of the proposed method, such as the selection of $\\mathcal{C}$ and its impact.\n\n[Clarity: 2 $\\to$ 3]\nThere are several unclear points regarding the proposed method. First, it is unclear whether the algorithm allows the budget to be exceeded. Second, the explanation of the strictness of the error estimation, i.e., that $\\hat{e} (\\cdot)$ converges in probability to $e (\\cdot)$, is insufficient. Third, the intervals between the axis scales in each figure are not consistent, making it difficult to interpret the figures and, consequently, to determine whether the numerical experimental results are significant.\n* After the rebuttal, the authors addressed all three points of ambiguity, so I raised Clarity to 3.\n\n[Significance: 3]\nThe proposed method offers a new approach to the problem of existing methods, where the results become worse than uniform sampling when the error function is large, and provides several extensions. This contributes new insights to the field of statistical inference for active learning. Additionally, the paper points out that there is room for further research on the selection of $\\mathcal{C}$ and its impact on robustness.\n\n[Originality: 2 $\\to$ 3]\nThe proposed method suggests the possibility of designing a new distribution that connects uniform sampling and active statistical inference, and is original in that it also considers the impact of misidentification in estimation. However, it lacks comparison with several important related studies. For example, active testing (Kossen et al., 2021) also focuses on statistical inference in active learning rather than prediction, and like this paper, it designs a distribution proportional to the error in each instance and makes decisions based on the realized values from that distribution. It should also mention the problems of using conventional active learning for statistical inference. In the current draft, it seems preferable to perform classical active learning, select samples that minimize prediction, and then perform statistical inference on the resulting model.\n* After the rebuttal, the authors clearly explained the relevance of their study to existing studies, so I raised Originality to 3.\n\nBased on the above, I rated the paper as 3 due to room for improvement in clarity and originality.\n\nKossen et al., Active Testing: Sample–Efficient Model Evaluation, ICML 2021", "was_revised": true, "final_justification": "The authors' rebuttal appropriately addressed the criteria for raising the score that I presented in Questions, particularly the resolution of ambiguities and the relevance to existing research. In my opinion, this paper has no clear flaws that would warrant rejection, so I have increased my score to 4. However, I believe that this paper can be improved regarding exceeding the budget $n_b$. In the problem setting, we control the number of observations in expectation, so exceeding the budget $n_b$ is acceptable. While I think this is unavoidable because of designing a sampling distribution, I think it should also be strictly within the budget $n_b$.", "rebuttal": "We appreciate the reviewer's comprehensive summary and helpful feedback. We have addressed each of your points below. Please let us know if these responses adequately address your suggestions.\n\n--- \n\n> “Since the algorithm randomly decides whether to observe each instance, it is possible that all instances are observed in extreme cases. Does this paper allow the budget  to be exceeded? If so, can the same claims as in this paper be made while keeping the total number of observations within by adaptively changing the sampling distribution for each observation, similar to active testing (Kossen et al., 2021)?”\n\n\nLike prior work [1], our algorithm enforces the budget *in expectation*. However, since the number of labeled data points is a sum of independent Bernoullis, a standard Hoeffding argument guarantees that the realized labeling rate will closely match the budget with high probability. Specifically, the labeling ratio will not exceed $\\frac{n_b}{n} + \\epsilon$ with a probability of $1 - \\delta$, provided that $n > \\frac{\\log (1/\\delta)}{2 \\epsilon ^2}$ for any $\\epsilon, \\delta > 0$. (Alternatively, this can be interpreted as: if we run the algorithm with a slightly reduced budget $n_b - \\epsilon$, then with high probability the budget $n_b$ will be met.)\n\nActive testing [2] pursues a different objective (high‑precision risk estimation for a fixed model) and uses a distinct estimator so the program is very different. That said, the active data collection is closely related to the setting of active inference. Thank you for pointing this out–we will add a discussion of the connection between active inference and active testing in the revision. However, our primary contribution lies in *robustifying* the sampling process, rather than proposing a new active approach to statistical inference. We believe that this is a novel contribution.\n\n--- \n\n> “In lines 9, 101, and 297, is this claim asymptotic? Does it hold for finite samples?“\n\n\nYes, our theoretical claim is asymptotic, as supported by Theorems 1 and 2. We will make sure to clarify this. Our empirical findings in the experimental section show this claim holds true in finite samples.\n\n\n--- \n\n > “The explanation of the inconsistency in the axis scales in all figures, including Figure 1, and how to read the figures should be clarified.”\n\n\nThank you for flagging this issue. We acknowledge the presence of multiple axis scales within the paper, specifically for sample size, coverage, and interpolation coefficients.\n\nFor coverage and interpolation coefficients, the scale remains consistent across all figures, ranging from 0 to 1. The scale for the budget $n_b$ varies between figures due to several factors, including dataset size (for example, we set the budget as a percentage of the overall dataset size, e.g. 10%) and burn-in size (which is included in the budget). The y-axis, representing the effective sample size (ESS), is determined by how much the procedure gains over uniform sampling, which is in turn determined by the accuracy of the predictions and uncertainties.\n\nTo accurately interpret the figures, the reader should understand the concepts of *effective sample size* (ESS) and *coverage*.\n\nESS quantifies a method’s efficiency compared to uniform sampling. For instance, if our method utilizes a labeling budget of $n_b = 500$ and achieves an ESS of $n_{\\mathrm{eff}} = 600$, this means that our method with 500 labels is as effective as uniform sampling with 600 labels. A higher ESS relative to the baseline signifies greater efficiency. (The uniform estimator by definition will always have an ESS equal to its budget ($n_{\\text{eff}} = n_b$)). Readers can also refer to the definition of effective sample size provided in Lines 193-200 of Section 4.\n\nCoverage captures how often the constructed confidence intervals cover the quantity of interest. Coverage should be at least 0.9 for the error to be controlled, and ideally it should be as close to 0.9 as possible.\n\n---\n\n> “References to related research. The differences between active testing and the present setting should be mentioned. In addition, the issues with performing statistical inference after classical active learning should be mentioned. My understanding is that the bias caused by not randomly selecting instances is the problem. Is this correct? If so, related research on statistical inference after decision-making based on statistical models, such as post-selection inference (Lee et al., 2016), should be mentioned to see if the bias of active selection can be corrected. Perhaps this is difficult because the selection of instances by active learning does not fit into the framework of post-selection inference. Is that correct?”\n\n\n\nThank you for this suggestion. We will expand on the distinctions between active testing and active inference in the revised version, as discussed above. However, we want to reiterate: our primary contribution is to *robustify* the sampling of active inference. This means our sampling method performs well regardless of the uncertainty scores obtained, a benefit not offered by active testing. It is true that classical active learning's greedy label selection can disrupt inference by introducing bias. Our method, consistent with [1], effectively mitigates this bias using an inverse probability weighting method.\n\nPost-selection inference (POSI) [3] addresses a fundamentally different problem: adaptively selecting hypotheses based on *fixed* data. Our framework, however, focuses on adaptively choosing data based on *fixed* hypotheses. These represent two different sources of bias, each with its own unique challenges. The bias mitigation techniques developed and leveraged in the two lines of work are entirely distinct. \n\n---  \n\n> “It is assumed that $\\hat{e}$ converges in probability to $e$, but I am not sure if this assumption is strong. Are sufficient conditions given?”\n\n\nThank you for your question. The assumption that $\\hat{e}$ converges to $e$ follows from classical arguments. As the size of the training or burn-in dataset increases, more information becomes available to estimate the conditional expectation. Flexible estimators, such as the multilayer perceptron (MLP) and XGBoost models used in the paper's experiments, are universal approximators. With sufficient data, they can learn complex relationships and provide a consistent estimate of the true conditional expectation, $e$, assuming the model is correctly specified and $e$ belongs to a sufficiently smooth or low-complexity class. While the rate of convergence depends on the specific estimator and the complexity of the true function, the convergence itself is a standard result from learning theory.\n\n(More generally, this type of assumption is common in the field of semiparametric inference. For instance, the widely-used doubly robust estimator [4, 5], which is closely related to our estimator, relies on consistent estimation of nuisance functions.)\n\n--- \n\n> “Is $Y$ limited to 0 or 1? Can this be extended to general distributions?”\n\n\nOur methodology does not assume $Y$ to be restricted to 0/1. Indeed, we consider general convex M-estimation problems, such as linear regression. For example, in the experiment on US Census data in Figure 5, $Y$ corresponds to income and is thus a continuous variable.\n\n\n--- \n\n> “If $\\mathcal{C}$ is an $\\ell_2$-ball and the radius is reduced to 0 (as $n \\rightarrow \\infty$), can the same theorem be derived?”\n\n\n\nThis is a good question. Roughly speaking, if the estimated error is consistent and the constraint set $\\mathcal{C}$ vanishes as $n \\rightarrow \\infty$, the consistency of $\\rho_{\\mathrm{robust}}$, the asymptotic normality of its corresponding estimator w.r.t $\\rho_{\\mathrm{robust}}$, and the property of its asymptotic variance can be derived by following the same steps used in the proofs of Theorem 1 and Theorem 2. This supports the asymptotic validity of the robust sampling procedure. However, formalizing this precisely is beyond the scope of the present work.\n\n--- \n\n> “I think it would be better to clarify what the expected values refer to, such as line 85 and Equation (2). I think that expected values of $X$ and those of $(X,Y)$ are mixed together.”\n\n\n\nThe expectation is consistently defined throughout the paper: all expectations $\\mathbb{E}$ average out *all* randomness in the argument, unless a conditional expectation is used, in which case the conditional expectation corresponds to a random variable that is measurable with respect to the variable being conditioned on.\n\n--- \n\n[1] Zrnic, Tijana, and Emmanuel Candes. \"Active Statistical Inference.\" International Conference on Machine Learning. PMLR, 2024.\n\n[2] ​​Kossen, Jannik, et al. \"Active testing: Sample-efficient model evaluation.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Lee, Jason D., et al. \"Exact post-selection inference, with application to the lasso.\" (2016): 907-927.\n\n[4] Glynn, Adam N., and Kevin M. Quinn. \"An introduction to the augmented inverse propensity weighted estimator.\" Political analysis 18.1 (2010): 36-56.\n\n[5] Robins, James M., Andrea Rotnitzky, and Lue Ping Zhao. \"Estimation of regression coefficients when some regressors are not always observed.\" Journal of the American statistical Association 89.427 (1994): 846-866."}, {"review_id": "rm5eMLQOCK", "reviewer": "Reviewer_mbvg", "summary": "This paper proposes a robust sampling strategy for active statistical learning that guarantees performance no worse than uniform sampling. The core idea is to define a budget-preserving interpolation path between an arbitrary active sampling policy and uniform sampling, and to search along this path for a robust choice of the interpolation parameter $\\rho$. The authors evaluate the method empirically on several datasets and demonstrate that it often outperforms both uniform sampling and the original active sampling policy.\n\nWhile the conceptual contribution is interesting, the paper lacks clarity in several important algorithmic aspects and could benefit from a broader experimental evaluation. If these issues are addressed (especially clarification of optimization strategy and broader empirical baselines), the work would be a valuable contribution.", "questions": "**Weaknesses and Questions**\n\n1. Algorithmic Clarity\n\n   The process of obtaining $\\rho_{\\text{robust}}$ is not fully specified in Algorithm 1.\n\n   - Do the authors perform grid search or use gradient-based methods to solve the minimax problem?\n   - If grid search is used, how is the grid step chosen?\n   - The paper should address the sensitivity of the solution to the grid step size or local minima, especially since small changes in $\\rho$ can lead to large changes in the objective function.\n\n2.  Misspecification Set $\\mathcal{C}$\n\n    The choice of the misspecification set $\\mathcal{C}$ is not well justified.\n\n     * The authors appear to assume an $\\ell_2$-ball, but other choices (e.g., $\\ell_\\infty$, greedy perturbations, or adaptive constraints) could lead to different behaviors.\n     * There is no discussion or comparison of how different choices of $\\mathcal{C}$ affect robustness or optimization difficulty.\n     * Given the known difficulty of minimax optimization (non-convexity, local optima), how do the authors ensure the optimal solution can be found?\n\n3. Interpolation Path Design\n\n   * In lines 112–113, the paper introduces linear and geometric interpolation paths between $\\pi$ and $\\pi_{unif}$.\n   * However, the choice of these two is not motivated. Are there other potential interpolation strategies (e.g., entropy-based, adaptive weightings)?\n   * A discussion of why only these two paths are chosen and their tradeoffs would strengthen the paper.\n\n4. Experimental Coverage\n\n   * The paper does not clearly specify which active sampling method $\\pi$ is used in the experiments. This is essential for reproducibility and clarity.\n   * Active learning encompasses many strategies (uncertainty sampling, margin sampling etc.). The current evaluation appears to be based on a single $\\pi$.\n   * To convincingly demonstrate the robustness of the proposed method, it would be helpful to evaluate it on multiple active sampling baselines.\n\n\n5. Typos\n\n   * line 76: \"distribution of the unobserved labels\" should be \"observed labels\".", "limitations": "* The method relies on a well-estimated error $\\hat{e}^2(\\cdot)$. If this is poorly estimated (e.g., under small data regimes), the robust sampling policy might become unreliable.\n* The minimax formulation adds significant optimization difficulty compared to standard active learning.\n* The lack of theoretical guarantees about convergence or uniqueness of $\\rho_{\\text{robust}}$ is a concern.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The motivation to robustify active sampling is strong, especially in settings where model misspecification or label noise can lead to degraded performance.\n* The paper search over an interpolation path between uniform and active sampling for better estimators.\n* Empirical results show improvements, supporting the theoretical analysis.\n* The methodology is general and can potentially be applied to various existing active sampling strategies.\n\n**Weakness**\n\nPlease refer to below questions section.", "was_revised": true, "final_justification": "The authors have addressed the major concerns raised in the initial review, particularly they give empirical results to support their theorem. I would like to raise my score to 4.", "rebuttal": "Thank you for your comprehensive review and your valuable feedback. We detail our response below, and please kindly let us know whether our response addresses your concerns.\n\n--- \n\n> “The process of obtaining $\\rho_{\\mathrm{robust}}$ is not fully specified in Algorithm 1. Do the authors perform grid search or use gradient-based methods to solve the minimax problem? If grid search is used, how is the grid step chosen? The paper should address the sensitivity of the solution to the grid step size or local minima, especially since small changes in \\rho can lead to large changes in the objective function.”\n\n\n\nWe employed a grid search for $\\rho$ in the outer loop, utilizing a step size of 0.01. The inner problem, being convex, is straightforward to solve and even possesses a closed-form solution when the constraint set is simple, such as the $\\ell_2$-ball. If the objective function is Lipschitz, we can always locate an $\\epsilon$-optimal point in terms of objective value, with the grid resolution determining $\\epsilon$.The sensitivity of the results to the grid step size is a good question. We conducted new experiments to explore different step sizes and confirmed the robustness of our results to step-size selection, as shown below. Due to the restriction on uploading images/plots, the results are presented in a table format.\n\nOn the Pew election dataset, our results show that a step size of 0.01 yielded $\\rho_{\\mathrm{robust}} = 0.43$, while a step size of 0.1 resulted in $\\rho_{\\mathrm{robust}} = 0.4$. The ESS gap between these two estimators is minimal, and both significantly outperform uniform and active baselines.\n\n| $n_b$  | Robust active (step_size=0.01) | Robust active (step_size=0.1) | Uniform | Active |\n|------|--------------------------------|-------------------------------|---------|--------|\n| 2794 | 2890                           | 2888                          | 2794    | 2802   |\n| 2875 | 2989                           | 2980                          | 2875    | 2894   |\n| 2956 | 3082                           | 3077                          | 2956    | 2951   |\n| 3037 | 3185                           | 3170                          | 3037    | 3060   |\n| 3118 | 3285                           | 3276                          | 3118    | 3104   |\n| 3199 | 3394                           | 3370                          | 3199    | 3192   |\n| 3280 | 3492                           | 3466                          | 3280    | 3257   |\n\n\n\n--- \n\n> “The choice of the misspecification set C is not well justified. The authors appear to assume an $\\ell_2$-ball, but other choices (e.g., $\\ell_{\\infty}$, greedy perturbations, or adaptive constraints) could lead to different behaviors. There is no discussion or comparison of how different choices of $\\mathcal{C}$ affect robustness or optimization difficulty. Given the known difficulty of minimax optimization (non-convexity, local optima), how do the authors ensure the optimal solution can be found?”\n\n\n\nAs we briefly discussed in Section 2.2, we found that $\\ell_2$ misspecification performs better than alternatives empirically. Further intuition is provided in the Appendix. For example, the $\\ell_1$ norm is ineffective since it concentrates perturbations on a single error entry, leaving the remaining entries unchanged. Conversely, the $\\ell_{\\infty}$ norm uniformly distributes perturbations across all error entries, which is also not ideal.\n\nRegarding minimax optimization, as noted in the paper, the inner optimization becomes a convex problem—and thus easily solvable—when the misspecification set $\\mathcal{C}$ is convex. This is crucial for the practicality of our proposal. The outer loop can then be solved via a simple, one-dimensional grid search over $\\rho$.\n\n\n--- \n\n\n> “In lines 112–113, the paper introduces linear and geometric interpolation paths between $\\pi$ and $\\pi^{\\mathrm{unif}}$. However, the choice of these two is not motivated. Are there other potential interpolation strategies (e.g., entropy-based, adaptive weightings)? A discussion of why only these two paths are chosen and their tradeoffs would strengthen the paper.”\n\n\n\nWe have also included the Hellinger path in our paper, with its definition and validity detailed in Appendix B. Figure 4 illustrates a performance comparison between all three paths. These specific paths were chosen due to their simplicity. More generally, these are all examples of geodesic paths: paths that are roughly defined as “shortest paths.” Any geodesic path can be used in our proposal. We elaborate and give more examples of geodesics in Appendix B.\n\n\n---\n\n> “The paper does not clearly specify which active sampling method $\\pi$ is used in the experiments. This is essential for reproducibility and clarity. Active learning encompasses many strategies (uncertainty sampling, margin sampling etc.). The current evaluation appears to be based on a single $\\pi$. To convincingly demonstrate the robustness of the proposed method, it would be helpful to evaluate it on multiple active sampling baselines.”\n\n\n\nWe agree that stress testing the methodology with different sampling methods $\\pi$ is important. We want to clarify that our paper *does* indeed test different $\\pi$ and define these rules for each experiment. For instance, in Figure 2, $\\pi(X) \\propto \\min \\\\{f(X), 1 - f(X)\\\\}$ (see line 222); in Figure 3, $\\pi(X) \\propto \\hat{e}(X)$ (see line 233); in Figure 7, $\\pi$ is based on verbalized LLM confidence (see line 273). We intentionally choose varying $\\pi$ to demonstrate the robustness of the method.\n\nIt is important to distinguish our framework from active learning despite their shared \"active\" nature, as their objectives are fundamentally different. Active inference aims to perform statistical inference on typically low-dimensional parameters. Consequently, many sampling strategies from active learning are not directly applicable or meaningful within the active inference framework, as it would be unclear how to perform valid statistical inference with those strategies. We will clarify these differences in the revised version. \n\n---  \n\n\n> “Typos. line 76: \"distribution of the unobserved labels\" should be \"observed labels\".”\n\n\n\nThank you for raising this point. This is not a typo; rather, it highlights a crucial motivation behind our work. The true parameter $ \\theta^* $ is influenced by the entire data distribution of $(X,Y)$. However, we a priori only observe unlabeled $X_i$ (line 73), no labels $Y_i$ (line 74).\n\n--- \n\n> “The method relies on a well-estimated error $\\hat{e}^2$. If this is poorly estimated (e.g., under small data regimes), the robust sampling policy might become unreliable.”\n\n\n\nWe want to emphasize that our robust sampling procedure remains effective even with a poor error estimation. A poorly estimated $\\hat{e}$ would typically result in a deficient active estimator. However, with our additional robustness layer, a poorly estimated error leads to a larger constraint set $\\mathcal{C}$, which in turn yields a larger $\\rho_{\\mathrm{robust}}$. This larger value pushes the sampling towards uniform sampling, ensuring the reliability of our method despite imprecise error estimation. Figure 6 illustrates the relationship between the interpolation coefficient and burn-in size: smaller burn-in sizes, leading to poorly estimated errors, result in larger values of $\\rho_{\\mathrm{robust}}$.\n\nRelatedly, we see that $\\mathcal{C}$ is large when the burn-in size is small; see below. The corresponding dataset is the Pew election dataset, and $c$ is the magnitude of the $\\ell_2$ constraint, i.e. $\\mathcal{C} = \\\\{\\epsilon: \\\\|\\epsilon\\\\|\\_2 \\leq c\\\\}$.\n\n| Burn-in size  | $c$    |\n|------|------|\n| 606  | 4.43 |\n| 1213 | 2.78 |\n| 1819 | 1.95 |\n| 2426 | 0.91 |\n| 3639 | 0.49 |\n\nAs expected, we observe a more conservative constraint when the errors are poorly estimated.\n\n--- \n\n> “The lack of theoretical guarantees about convergence or uniqueness of  $\\rho_{robust}$ is a concern.”\n\n\nTo reiterate our previous point about computational efficiency: when the constraint set $\\mathcal{C}$ is convex, the inner problem of the minimax formulation becomes a convex problem. This simplifies optimization and prevents convergence to local minima. The effect of performing a one-dimensional grid search over [0,1] to search over $\\rho$ is minimal, as illustrated above.\n\nFurthermore, in line 767 in the Appendix, we offer a theoretical guarantee: $\\hat{\\rho}$ will converge to $ \\rho^* $ provided $ \\rho^* $ is unique and error consistency is achieved. If a consistent error estimator is available, the misspecification set will asymptotically vanish. Consequently, the consistency of $\\rho_{\\mathrm{robust}}$, the asymptotic normality of its corresponding estimator w.r.t $\\rho_{\\mathrm{robust}}$, and the property of its asymptotic variance can be derived by following the same steps used in the proofs of Theorem 1 and Theorem 2. However, formally working out these details is beyond the scope of the present work.\n\n--- \n\n[1] Zrnic, Tijana, and Emmanuel Candes. \"Active Statistical Inference.\" International Conference on Machine Learning. PMLR, 2024."}, {"review_id": "sBxqBN9foC", "reviewer": "Reviewer_myd5", "summary": "This paper builds upon the recently introduced area of active statistical inference, in which the goal is to formulate a sampling policy for the collection of a labeled dataset that will be used in conjunction with a predictive model to estimate some statistical quantity of interest. Specifically, the paper formulates a robust active statistical inference problem where the goal is to find a sampling policy that is no worse than two given reference policies in terms of variance. The paper focuses on the case where one of the policies is the naive uniform sampling policy.\n\n\nThe paper's solution to this problem is to consider a path of policies that interpolate between the given policies and to choose the policy along the path that minimizes the estimate of the variance. To account for potential model misspecification, the paper considers a robust version of their estimator that allows for a perturbation of the variance estimator.\n\n\nThe paper goes on to apply their techniques to real-world data, showing that their method outperforms both uniform and non-robust active sampling strategies.", "questions": "- Empirically, how important is the robustness criterion for the proposed method?\n\n- What do the results of the paper look like across multiple random seeds?\n\n- Do you have any intuition for why one interpolation method should be preferred over another?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- This paper considers an interesting problem at the intersection of robustness and active data collection. Although some aspects of the problem were already covered in the original active statistical inference paper (see below), the paper considers a broader family of interpolation techniques along with a robustness modification to the optimization problem.\n\n\n- The paper evaluates the proposed methodology on 5 different datasets, covering a variety of inference tasks and prediction methods. Overall, their methods enjoy better empirical performance than the baseline approaches.\n\nWeaknesses:\n\n- The novel contributions of the paper need to more clearly delineated. As the paper is written, one might think that this paper is the first to consider the concept of finding the minimum variance interpolating sampling policy. However, this idea was already proposed in the original Zrnic and Candes active statistical inference paper, albeit only for the case of linear interpolants and without any robustness constraints. The contributions of this paper seem to be that it (a) expands the set of interpolating paths, (b) that it adds a robustness constraint to the variance estimator, and (c) that it gives a more thorough evaluation of uniform v.s. active v.s. robust active.\n\n\n- It is hard to tell how useful the robustness constraint is. There is one figure comparing the performance with and without the robustness constraint, confusingly shown twice (Figure 4 and Figure 10). However, because the x and y-axes are not the same between the two, I can't really eyeball how useful the robustness constraint is. There should be several figures comparing with and without robustness for the various path interpolants on ALL the datasets, and they should also be on the SAME axes (maybe use dotted and solid lines). It would also be helpful to see what happens as the robustness hyperparameter c is varied.\n\n\n- There is no mention of multiple runs/random seeds in the paper nor are there any confidence intervals/error bars in the plots. Given how much randomness is involved in this paper (initial training set, random label selection according to the policy, etc), it would be good to see how confident we should be in the results of the paper.\n\n\n- As the paper currently stands, there is no guidance on how to choose the interpolating path. There is one experiment showing that the geometric path empirically outperforms the other two proposed paths on the Pew survey dataset, but there's no reasoning given. Moreover, it's worth pointing out that one could actually consider many types of paths simultaneously and choose the best path/interpolating fraction based on minimizing the variance estimator.", "was_revised": true, "final_justification": "After reading the other reviews and going through the author rebuttals, I still believe this is a borderline paper.\n\nThe novelty of the paper beyond the original active statistical inference paper is somewhat limited, and the paper in its current form does not do a great job of delineating what is truly novel and what was known before. For the actual novelty of the work, the authors introduce two ideas: alternate types of interpolating paths and a robust version of the variance estimator. On both of these ideas, the paper in its current form feels like it has only performed a surface-level investigation. \n\nFor the alternate types of interpolating paths, the paper presents 3 out of a literally infinite class and then uses a small number of simulations to justify settling on one. There is no attempt to answer the question of why that particular scheme works better than the others.\n\nFor the robust version of the variance estimator, it seems like a few versions were tested out in the writing of the paper, but ultimately just one type is presented with no real justification. Why does that form of robustness seem to work so well for this problem? What types of misspecifications are being fixed here? As my model class gets closer to the ground truth, what is the effect of adding/removing this robustness? Is it a bias issue with the model or a variance issue with its predictions?\n\nI feel that diving into these kinds of fundamental questions would greatly improve the paper.", "rebuttal": "Thank you for your detailed review and helpful feedback. We have addressed each of your points below. Please let us know if these responses adequately address your concerns.\n\n--- \n\n> “The novel contributions of the paper need to more clearly delineated. As the paper is written, one might think that this paper is the first to consider the concept of finding the minimum variance interpolating sampling policy. However, this idea was already proposed in the original Zrnic and Candes active statistical inference paper, albeit only for the case of linear interpolants and without any robustness constraints. The contributions of this paper seem to be that it (a) expands the set of interpolating paths, (b) that it adds a robustness constraint to the variance estimator, and (c) that it gives a more thorough evaluation of uniform v.s. active v.s. robust active.”\n\n\n\nThank you for this observation. We acknowledge that we missed including a discussion of the robustness proposal from [1]. We will make sure to clarify the distinction in the revised version of the paper. The proposal from [1] assumes access to historical data to tune the linear interpolation coefficient. Otherwise, it selects a default value for the coefficient, such as 0.5, which does not have the guarantee of outperforming uniform and active sampling. Our analysis is far more thorough and systematic, expanding the set of interpolating paths, not requiring historical data but incorporating a burn-in period, and adding in a robustness constraint. These are all crucial for the practicality and reliability of the method. For example, the robustness constraint is critical: for natural choices of $\\pi$ proposed in [1], tuning the interpolation coefficient *only* would result in trivial sampling rules (see paragraph starting in line 230).\n\n--- \n\n> “It is hard to tell how useful the robustness constraint is. There is one figure comparing the performance with and without the robustness constraint, confusingly shown twice (Figure 4 and Figure 10). However, because the x and y-axes are not the same between the two, I can't really eyeball how useful the robustness constraint is. There should be several figures comparing with and without robustness for the various path interpolants on ALL the datasets, and they should also be on the SAME axes (maybe use dotted and solid lines). It would also be helpful to see what happens as the robustness hyperparameter c is varied.”\n\n\n\nThis is a very good suggestion. (And thank you for pointing out the repeated figure; Figure 10 includes the coverage plots accompanying Figure 4 due to the space limit of the main paper.) The robustness constraint is indeed crucial. As discussed in Section 4.1 (line 230), if we set $\\pi \\propto \\hat{e}$, without the additional robust optimization step, the method would trivially estimate $\\hat{\\rho} = 0$. This means that in all experiments that rely on this choice of $\\pi$, *the plain active baseline is equivalent to not including the robustness constraint*. To give some numbers, here are some results from Figure 5:\n\n| $n_b$ | With robustness constraint | Without robustness constraint | Active | Uniform  |\n|-------|---------------------------|------------------------|--------|--------|\n|  1498    |            1921               |            1691          |     1691   |   1498    |\n| 1717    |                 2317          |               1944         |     1944   |     1717   |\n| 1936    |                 2689         |                2195        |     2195  |     1936   |\n| 2156   |                   2971       |             2495          |     2495   |     2156   |\n| 2375   |            3233               |             2775           |     2775   |     2375   |\n\nBelow we give an additional example when $\\pi$ is not proportional to $\\hat{e}$, specifically in the setting of Figure 4, where the burn-in size is 1819, and $\\pi \\propto \\min\\\\{f(X), 1-f(X)\\\\}$: \n\n| $n_b$ | With robustness constraint | Without robustness constraint | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  2339    |            2464              |             2406         |     2337   |   2339  |\n| 2433    |              2558         |              2491         |     2439   |     2433   |\n| 2622    |              2753           |              2696         |     2605   |     2622   |\n| 2811   |                 2974       |               2869         |    2803   |     2811   |\n\nWe will include the above results in the revision. Thank you again for this valuable suggestion.\n\n--- \n\n\n> “There is no mention of multiple runs/random seeds in the paper nor are there any confidence intervals/error bars in the plots. Given how much randomness is involved in this paper (initial training set, random label selection according to the policy, etc), it would be good to see how confident we should be in the results of the paper.”\n\n\n\nWe appreciate and agree with this suggestion. We average all metrics over 500 trials. For the coverage plots, given that they are obtained by averaging 500 binary indicators, the standard deviation does not add any information, since it equals $\\sqrt{\\mathrm{coverage}(1-\\mathrm{coverage})}$, and should not be plotted (as per standard convention). However, we will plot the standard deviation of the effective sample size everywhere. Below we include the numbers for the problem of collecting politeness annotations with LLMs.\n\n| $n_b$  | Robust active ESS | Robust active Std | Active ESS | Active Std | Uniform ESS | Uniform Std |\n|------|-------------------|-------------------|------------|------------|-------------|-------------|\n| 1643 | 2263              | 51.9              | 1808       | 109.0      | 1643        | 30.3        |\n| 1887 | 2563              | 63.2              | 2065       | 139.4      | 1887        | 47.7        |\n| 2130 | 2849              | 56.8              | 2319       | 111.7      | 2130        | 48.1        |\n| 2374 | 3132              | 55.6              | 2499       | 118.6      | 2374        | 40.2        |\n| 2617 | 3402              | 49.4              | 2693       | 128.5      | 2617        | 37.3        |\n| 2861 | 3654              | 47.9              | 2891       | 105.8      | 2861        | 39.6        |\n\n\n\n---\n\n> “As the paper currently stands, there is no guidance on how to choose the interpolating path. There is one experiment showing that the geometric path empirically outperforms the other two proposed paths on the Pew survey dataset, but there's no reasoning given. Moreover, it's worth pointing out that one could actually consider many types of paths simultaneously and choose the best path/interpolating fraction based on minimizing the variance estimator.”\n\n\n\nWe have stress tested the geometric path on multiple datasets (with different modalities, uncertainty scores, etc) and found it to be the best choice among the considered paths. While we cannot guarantee that this will be the case everywhere, our main goal is to provide a method that consistently performs no worse than uniform or plain active sampling. Across *all* datasets, interpolating paths, uncertainty measures, we have found this core message to be true. Developing a deeper understanding around what properties of the problem determine the right interpolating path is a non-trivial and exciting future direction. We agree that it would be beneficial to simultaneously consider multiple paths and select the best one automatically. The risk is that this might be somewhat less practical, but it would be worthwhile to study the performance gains and see if they justify the extra effort. For now, given our current findings, we recommend the geometric path as a simple, practical default.\n\n---  \n\n\n[1] Zrnic, Tijana, and Emmanuel Candes. \"Active Statistical Inference.\" International Conference on Machine Learning. PMLR, 2024."}, {"review_id": "6YiHSQYFAX", "reviewer": "Reviewer_usPt", "summary": "The authors’ goal is to increase the efficiency of active sampling policies used in the context of active statistical inference: Inaccurate predictive models can give poor uncertainty estimates, which can result in supposedly efficient sampling methods (e.g., uncertainty sampling) having a larger sample efficiency than uniform sampling. To address this, the authors propose a method which smoothly interpolates between an active and uniform sampling policy, where the “optimal” interpolation is identified in a data-driven way.", "questions": "**Sensitivity to choice of budget-preserving path:** In Figure 4, the linear and Hellinger budget-preserving paths appear to perform about as well as uniform sampling. In general, how sensitive is the method’s performance to the choice of budget-preserving path? In practice, how would a user select a good budget-preserving path?\n\n**Uncertainty set:**\nTheorem 2 requires that $\\hat{\\rho} = \\rho^\\star + o_P(1)$. Does the degree of misspecification of $\\hat{e}$ affect whether this condition is violated or not? How does incorporation of the uncertainty set affect whether/when this condition is violated?\n\nIn section 2.2, it is mentioned that a permissive uncertainty set results in a large value of $\\hat{\\rho}$, which is intuitive: A misspecified predictive model is less likely to produce a reliable active sampling policy. Are there advantages to solving the robust optimization problem instead of, for instance, adding a penalty for small values of $\\rho$ to the original optimization problem (line 131)?", "limitations": "Yes. While the authors mention the sensitivity of their method to the choice of budget-preserving path, more details on the extent of this sensitivity could be provided.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The submission tackles a somewhat significant problem: Predictive (surrogate) models are used to complement scarce data in many practical applications, and robustness of policies informed by such models to the potentially unreliable uncertainty estimates they produce is important in such applications. The scope of the paper is limited to the design of robust active sampling policies; active sampling is useful especially in the types of data-poor settings in which predictive-powered inference is applied.\n\nThe submission appears original, technically sound, and the claims well-supported by both theoretical analysis and experimental results. It is exceptionally clearly written and well organized.\n\nMy main concern regards the effectiveness of the authors’ method: I think the paper could benefit from more discussion of the method’s sensitivity to user-specified aspects, in particular the choice of budget-preserving path.", "was_revised": true, "final_justification": "The submission tackles a somewhat significant problem and appears original and technically sound. My main concern was the effectiveness of the proposed method. The authors' rebuttal included additional discussion and experimental results, which (i) clarified the implications of the theoretical results and range of the method’s behavior, and (ii) provided concrete guidance about how to select the budget-preserving path. I believe including this discussion and these results in the paper will enhance the paper’s contributions.", "rebuttal": "We really appreciate your comprehensive feedback. We have responded to each point individually below, and hope our responses adequately resolve your concerns.\n\n--- \n\n> “Sensitivity to choice of budget-preserving path: In Figure 4, the linear and Hellinger budget-preserving paths appear to perform about as well as uniform sampling. In general, how sensitive is the method’s performance to the choice of budget-preserving path? In practice, how would a user select a good budget-preserving path?”\n\n\nOur central claim is that our procedure yields an estimator that performs no worse than either the uniform or active baseline, and that the gains can in many cases be significant. We cannot guarantee the extent of performance gain for any specific problem.\n\nHowever, to further evaluate the possible path choices, we added a new experiment comparing the geometric and linear paths against the active and uniform baselines in the setting of Figure 3. The findings presented below support our central claim above: the linear path consistently performs no worse than either of the two baselines, even if it does not achieve the same performance as the geometric path.\n\nBurn-in size 606:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  1396    |            1511               |             1434         |     1195   |   1396    |\n| 1518    |                 1652          |              1569         |     1264   |     1518   |\n| 1760    |                  1932         |              1836         |     1422   |     1760   |\n| 2003   |                    2205       |               2097         |    1652   |     2003   |\n\nBurn-in size 1213:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  1807    |            1896               |          1844            |     1744   |   1807    |\n| 2023    |                 2142         |               2073        |     1954   |     2023   |\n| 2239    |                  2393         |               2286        |     2058   |     2239   |\n| 2455   |                    2641       |             2519           |    2218   |     2455   |\n\nBurn-in size 1819:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  2340    |            2418               |            2358           |     2288   |   2340    |\n| 2529    |                 2620          |               2553         |     2481   |     2529   |\n| 2718    |                  2854         |                2765        |     2646   |     2718   |\n| 2907   |                   3066        |             2963           |     2823   |     2907   |\n\nBurn-in size 2426:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  2793    |            2866               |         2840          |     2796   |   2793    |\n| 2955   |               3064          |                3033       |     2982   |     2955   |\n| 3117    |               3265         |              3189        |     3131   |     3117   |\n| 3279   |                3458        |             3364        |     3287  |     3279   |\n\nBurn-in size 3639:\n\n| $n_b$ | Robust active (geometric) | Robust active (linear) | Active | Uniform |\n|-------|---------------------------|------------------------|--------|--------|\n|  3881    |            3963               |           3938        |     3923   |   3881    |\n| 3935   |               4024         |              4022         |     4010   |     3935   |\n| 4043    |               4166         |               4147       |     4126   |     4043   |\n| 4151   |                4308        |               4285      |     4251  |     4151  |\n\n\n\n--- \n\n> “Uncertainty set: Theorem 2 requires that $\\hat{\\rho} = \\rho^*+o_p(1)$. Does the degree of misspecification of $\\hat{e}$ affect whether this condition is violated or not? How does incorporation of the uncertainty set affect whether/when this condition is violated?”\n\n\n\nGood question: the consistency of $\\hat{\\rho}$ is indeed affected by the degree of misspecification in $\\hat{e}$. When $\\hat{e}$ is consistent for $e$, the consistency of $\\hat{\\rho}$ can be shown under mild conditions (as detailed in the Appendix). However, a high degree of misspecification in $\\hat{e}$ leads to $\\hat{\\rho}$ being inconsistent, which necessitates the incorporation of a robustness constraint.\n\nThe size of the constraint set influences $\\rho_{\\mathrm{robust}}$: a small set results in $\\rho_{\\mathrm{robust}}$ being close to 0 (active), while a large set yields $\\rho_{\\mathrm{robust}}$ close to 1 (uniform). If the constraint set is too large, we may lose the guarantee of convergence to $\\rho^*$, but by appropriately choosing the constraint set $\\mathcal{C}$ we still offer improved performance over the uniform baseline, as demonstrated empirically.\n\n\n--- \n\n> \"In section 2.2, it is mentioned that a permissive uncertainty set results in a large value of $\\hat{\\rho}$, which is intuitive: A misspecified predictive model is less likely to produce a reliable active sampling policy. Are there advantages to solving the robust optimization problem instead of, for instance, adding a penalty for small values of \\rho to the original optimization problem (line 131)?\"\n\n\nThis is a very interesting suggestion. Both approaches are interpretable and computationally efficient. We feel that our approach is more interpretable, since it directly incorporates a robustness wrapper around the source of the misspecification, which are the error estimates. The regularization approach does this more implicitly. Tuning the regularization parameter also feels potentially more involved than determining the robustness constraint $c$, which is simply based on measuring the error of $\\hat e$ on a small dataset. We will add a brief discussion of this alternative in the revised version of the manuscript. Thank you for suggesting it!\n\n---"}]

Author rebuttal:

Meta-review: This paper studies a method for sampling data in order to achieve more data efficient statistical estimation. The reviewers appreciated the clear description, the simplicity, elegance, and generality of the proposed method, and good empirical results. However, the reviewers also noted a lack of rigorous analysis (both theoretical and experimental) of error bars, hyperparameter choices, and methodological choices which were mostly addressed during the rebuttal period. Additionally, some aspects of the exposition were unclear such as (1) the (expected) budget constraint, (2) relationship to prior work, and (3) burn-in period of the algorithm (not in the main algorithm box of Algorithm 1), though were clarified during the rebuttal period. Overall, I recommend acceptance, though weakly given the large number of changes from the submission. If this paper is accepted, we expect (prior to camera-ready) the authors to take reviewer recommendations for revision of the paper seriously and integrate all experiments and clarifications mentioned by the authors in the rebuttal.
Decision: Accept (poster)