Balanced Active Inference
Authors: Boyu Chen
Zhixiang Zhou
Liuhua Peng
Zhonglei Wang
Venue: NeurIPS 2025 poster ()
Keywords: asymptotic properties
cube method
label selection
statistical efficiency
Primary area: deep_learning

Abstract: Limited labeling budget severely impedes data-driven research, such as medical analysis, remote sensing and population census, and active inference is a solution to this problem. Prior works utilizing independent sampling have achieved improvements over uniform sampling, but its insufficient usage of available information undermines its statistical efficiency. In this paper, we propose balanced active inference, a novel algorithm that incorporates balanced constraints based on model uncertainty utilizing the cube method for label selection. Under regularity conditions, we establish its asymptotic properties and also prove that the statistical efficiency of the proposed algorithm is higher than its alternatives. Various numerical experiments, including regression and classification in both synthetic setups and real data analysis, demonstrate that the proposed algorithm outperforms its alternatives while guaranteeing nominal coverage.

Balanced Active Inference
Boyu ; Zhou' ; Liuhua Peng"! Zhonglei Wang*;
Schooo of Economics, Xiamen University, Xiamen, China
SShnggai Innovation Institute, Shanghai, China
School of Mathematics and Statistics, The University of Melbourne, Melbourne, Australia
WWang Yanan Institute for Studies in Economics, Xiamen University, Xiamen, China boyuchen@stu. xmu edu. en, zhixiangzhou. swwttoutlook.......oo.o liuhua, petuhua.peggunimmeeeee. edu. au, wangz @xmu. edu cn
Abstract
Limited labeling budget severely impedes data-driven research, such as medical analysis, remote sensing and population census, and active inference is a solu- tion to this problem. Prior works utilizing independent sampling have achieved improvements over uniform sampling, but its insufficient usage of available infor- mation undermines its statistical efficiency. In this paper, we propose balanced active inference, novel algorithm that incorporates balancing constraints based on model uncertainty utilizing the cube method for label selection. Under regularity conditions, establish its asymptotic properties and also prove that the statis- tical efficiency of the proposed algorithm is higher than its alternatives. Various numerical experiments, including regression and classification in both synthetic setups and real data analysis, demonstrate that the proposed algorithm outperforms its alternatives while guarantceing nominal coverage. Our code is available bettps//ggthubt ctt.:niif//////////////aa http:///gggt...ooo////////////// Aetllanee__tttie___eeeeeeee
Introduction
Machine learning has revolutionized data-driven fields, yet its success still hinges access to high-quality labeled data, which is a critical component for reliable inference. This dependency on labeled data is particularly pronounced in precision-sensitive fields such medical diagnostics [I]. financial risk assessment (2). and remote sensing [3 where the accuracy of predictions directly affects decision making. However, labeling remains costly and time-consuming process, resulting in persistent gap between the abundance of unlabeled data and the limitation of annotated resources [4|5).
Conventional methods, including random sampling and heuristic-based selection, lack systematic prioritization of informative instances, leading to inefficient labeling [6] 7
Active learning addresses the labeling bottleneck by iteratively selecting uncertain instances to maximize label efficiency 7 g 9.. Extending to statistical inference, active inference [10] strategically queries labels where the model exhibits high uncertainty using independent sampling, and makes statistical inference, including confidence intervals and hypothesis tests, based on the acquired labels. Combined with prediction-powered inference (11), integrating model predictions with limited labeled instances, active inference outperforms random sampling in terms of statistical efficiency. Nonetheless, its reliance independent sampling induces variance inflation and often yields imbalanced datasets. These issues may degrade estimation performance [12), especially under systematic bias or distribution shifts, ultimately limiting the practical utility of existing methods.
"Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Therefore, critical challenge in active inference is to further improve statistical efficiency under constrained labeling budget. To address the inefficiency introduced by traditional active inference strategies, we propose halanced active inference. This method strategically selects informative instances while maintaining statistical representativeness of the population. The key idea is to enforce structural balance in the selected samples so that they preserve key characteristics of the overall data distribution, thereby improving statistical efficiency without increasing the labeling budget. Our method builds upon the principle of covariate balancing, a classical technique in survey sampling and causal inference that aligns the distribution of sampled instances with population-Ievel summaries of auxiliary variables [3 14].
To enforce covariate balance in active instance selection, we implement balanced active inference using the cube method [15] [6], sampling algorithm that by consecutively updating selection probabilities to satisfy certain balancing constraints. The cube method operates in two phases: flight phase, which iteratively adjusts inclusion probabilities to approximate target distributions, and a landing phase, which finalizes the selection by resolving residual imbalances through constrained optimization.
Compared with existing works, the proposed balanced active inference framework offers three ad- vantages. First, we reconceptualize model uncertainty estimates as dynamic auxiliary variables, enabling simultaneous optimization for informativeness and representativeness during instance selec- tion. Second, we introduce a balancing condition that constrains the weighted sum of uncertainties among the labeled instances match the corresponding population total, effectively preventing oversampling from specific uncertainty regions and promoting more stable estimates. Third, we provide a theoretical guarantee that he proposed balanced active inference framework yields lower asymptotic variance compared to conventional active inference methods.
Our contributions can be summarized as follows.
Innovatively increase the statistical efficiency of active inference through balanced sampling.
Incorporate model uncertainty as an auxiliary covariate in balanced sampling using cube method.
Provide closed-fon expressions for the asymptotic variance, illustrating the variance reduction property of the proposed method.
Demonstrate the broad applicability and superiority of our method through extensive experiments on diverse real-world and synthetic datasets.
Related work. (1) Label Inference. The challenge of drawing valid inferences from partially labeled data has inspired diverse methodological developments. Traditional methods for missing data, such as inverse probability weighting and multiple imputation [17] 18, established foundational
principles for handling label scarcity. Semi-supervised inference methods 199 20 demonstrated
how unlabeled data could improve efficiency in parameter estimation, particularly under smoothness assumptions. A pivotal advancement emerged with cnndictiin-poweere inference (PPI) [[]. which integrates machine learning predictions with small labeled dataset to estimate population quantities. By treating predictions as noisy proxies for missing labels, PPI constructs debiased estimators while maintaining statistical validity. However, its reliance uniform random sampling undermines its statistical efficiency, as it fails to prioritize informative instances for labeling.
(2) Active Leurning. Active learning addresses label efficiency by adaptively selecting instances annotation based on model uncertainty 7 9 21. While classical active learning focuses on optimizing model training [22], recent work extends these principles to statistical inference. Active inference [10] formalizes this paradigm, employing umequal-probabee
sampling to prioritize uncertain instances. By combining actively acquired labels with model predictions via GD (general difference) estimator [23], it incorporates auxiliary information to correct for sampling bias and improve estimation efficiency. Despite its advantages, the independent sampling mechanism inherent to active inference introduces variance inflation, as independent label selections may yield imbalanced instances that poorly represent critical regions of the data distribution.
(3) Balanced Sampling. The use of auxiliary information finite population sampling widely recognized for enhancing estimation precision. Classical methods, such as stratification [12] 24 and probability proportional-to-se sampling 2 26), exploit known auxiliary variables to reduce variance. More advanced balanced sampling techniques impose equality constraints, ensuring that

---

weighted sample summations match population totals, guaranteeing significant variance reduction
27
28
2 Problem setup
Suppose we have access to two datasets: a small labeled dataset D, = [(X,, y,) with m instances, independently and identically distributed (i.i.d.) from distribution IP = Ppx Pylx- and a large unlabeled dataset D, {x,)=_ with instances drawn i.i.d. from Px, where the corresponding labels [Y:) unobserved. Denote X C R the feature space, and y S IR as the label space. The primary goal is to estimate the population mean of the unobserved labels D.. defined as
E(G).
(1)
To leverage feature-label relationships, a predictive model f : + is trained on Dj. Additionally,
labeling budget of ny allows the query of labels for subset of D.. Let & {0, 1} denote an indicator for whether the label Y; is acquired for the instance X; E DD, with IE EE & = Tia- The
challenge lies in designing estimator that optimally combines the predictive power of f(-) with strategically sampled labels to reduce the variance of an estimator of (T).
Active inference employs a machine learning model f(-) to predict labels for unobserved instances, coupled with an adaptive sampling strategy that corrects potential prediction biases. The sampling policy ग + [0, 1] determines label acquisition probabilities through uncertainty quantification. Specifically, let i(X,) represent the model's estimated uncertainty measure for instance i, typically equal YY; f( (x.))- The sampling probabilities then normalized as
(X,)
Thb
i(,,)
I +E ((x))
n
ensuring E E C.) nh through scaling. This allocation prioritizes regions where the model exhibits higher prediction uncertainty.
GD estimator 23 of (I for the inference.
1 Σ f(x)) =
& T(X,)
(3)
which is unbiased regardless of the form of the predictive model f(.). Specifically, prediction-
powered inference [1] emerges as a special case when x <(X,) = n// n, corresponding to uniform random sampling.
3 Balanced active inference
Our framework advances active inference by integrating balanced sampling through the cube method, which enforces structural constraints on auxiliary variables to improve statistical efficiency. Specifi- cally, we impose the balancing condition
Σ ((X)E ((i))
w(X.),
(4)
where ü( ú(X;) quantifies the uncertainty of the predictive model f(X.). The balancing constraint ensures that the selected instances preserve the population structure of model uncertainties, mitigating selection bias. To operationalize this, we employ the cube method, two-phase sampling algorithm that first iteratively adjusts inclusion probabilities to satisfy balancing constraints (flight phase), and then resolves residual imbalances via a landing phase when exact balancing is infcasible [16]; see Section of Supplementary Material for an introduction to the cube method.
Once we obtain sampling indicators (& J2 through the proposed balanced sampling strategy, we still consider GD estimator
n
f(x.) (Y f(x.) +X.)
(5)
3

---

Different from
B the sampling indicators associated with (5) satisfy the balancing constraint 4
Remark It may seem preferable to balance (X;; as well, since f(X.) directly captures predictive information about y, However, enforcing the balancing constraint on f(x)) leads the estimator in (5) to degenerate into the form n." Zi (Y,Ss/ma)) thereby forfeiting any benefits from active sampling. In contrust, balancing on (i))) preserves the correction term in S). (X)( accurately approximates the residual error, i.e., i(X;) = Y; ; f(X.), then the proposed estimator f m m-'E" En Yú, thereby improving the statistical efficiency of the GD estimator.
Uncertainty measures To effectively quantify prediction uncertainty for different task types, we define specific uncertainty measures tailored to regression and classification settings. For regression problems, the uncertainty is captured by the absolute residual IY; -f fX,; ().. For classification tasks, let p(X,) = (pi (X:), .P (X.) represent the predicted class probabilities. The uncertainty measure is defined as uxX,) x- (1 [ (X.))), which attains its maximum when the model is maximally uncertain and decreases zero when the model exhibits high confidence in single class. The results of using other uncertainty quantification are provided in Section S6 in the
Supplementary Material.
Stabilization via mixed sampling As suggested by the traditional active inference literature [0) direct implementation of x xXx;) x ū(X;) risks instability when û( (();) is misspecified. To safeguard against variance inflation from near-zero (X;), we consider the following m-xied rule:
(x)(())) π (7) (Xi) = T プー iXx,) + (1- T0b
(6)
n
where r E [0, 1] controls the trade-off between uncertainty prioritization and robustness. Empiri- cal analysis demonstrates that = 0.5 achieves favorable bias-variance trade-offs across diverse scenarios, aligning with findings in [10. This mixture ensures x (x;) > O universally while
retaining adaptivity to ú(;)). The relevant sensitivity analysis of is discussed Section S5 in the Supplementary Material.
Implementation Algorithm outlines our cube-based balanced active inference procedure. The flight phase enforces the balancing constraint via iterative geometric projections, followed by landing phase that minimizes deviation from target inclusion probabilities. The integration of the cube method with uncertainty-aware sampling distinguishes from conventional balanced sampling, as the auxiliary variable ú(z) directly links to the statistical efficiency of the GD estimator. The computational complexity of the cube method is (n 2 [16), where n is the population size and p the number of balancing covariates. In our implementation, the uncertainty measure u is used as the only auxiliary covariate, and the complexity of balanced sampling reduces to (n)).
This synthesis of balanced sampling and active inference provides a unified framework for semi- supervised mean estimation, where model predictions guide sampling, while balancing constraints safeguard against distributional shifts-a key advancement over existing works.
Extension to M-estimation Consider a general M-estimation problem that, given a class of functions f(x;;)).
0" arg min E [L(X,, 01:
where I(XX,, Y1:0) is a loss function measuring the discrepancy between the true label Yi and the predicted label f(X;; 0), and 0 is the parameter of interest, The goal estimate 0 using the labeled data D, and the unlabeled data D., Suppose that there have been an estimation f(X;; 0) trained on the labeled data D, and a uncertainty estimator ((x)) trained on L(X,, - (X,; 0) 0). sampling scheme {TT(x)) } is derived following (2) given budget nig We use the cube method to generate an assignment (5)) such that the balancing constraint in holds. The proposed estimator for M-estimation is then defined as
& T(X,)
arg min
L(Xi, ((x; +
(7)
7

---

Algorithm Balanced active inference
1: Train a prediction model f()) on labeled training data Dj = {(X,, 1,)
2: Compute residuals ej = fX²)) Y, and train uncertainty model i(X;) R leyl.
3: Sample R4 instances for labeling given the label budget = [B]n//n..
4: for X; E D. do
5: Predict label Y; f(x))
6: Predict uncertainty ú; = i(Xi).
7: Compute active probability Pa,i = būg; where i 1,2 Σ in and uniform probability pe,i = b.
8: Blend with uniform sampling: mi (r) TPo, + (1-T)pe.d
9: end for
10: Apply the cube method on { ( Hi=1 to get {E:
satisfying the balancing constraint
7
11: Compute ① via 5
The first term in the curly braces is the loss function evaluated at the predicted label f(X;; 0), while the second term is the correction term that accounts for the difference between the loss function evaluated at the true label Y; and the predicted label f(X;; 0). When the prediction f(X;; 0) is poor, the correction helps to reduce the variance of the estimator by adaptively adjusting the inclusion probabilities based on the uncertainty of the predictions. By leveraging the cube method, we ensure that the selected instances are balanced with respect to the uncertainty estimates, leading more efficient estimation process.
4 Theoretical properties
Before presenting the main theoretical results, we introduce some generality assumptions under which our analysis is conducted.
Assumption 1. There exists function J, such that
Y f(x,) t
where e; satisfies B(E;[X,) 0.
Assumption 2. Assume E(Y? f(x)) + +x +xx)) co.
Assumption 3. There exists positive constant E (0, 1), such that
P (6: = | X.) E [c, - c).
Assumption 4. For any E N, we have with probability one,
lim sup Eroeoiik
E (E,, -T1,) 1 Xn -.
Assumption 5. The estimator f((X)) and ū(X;) statisfy
E. [[((xx)) - f(x,) -ggn (((x))-((xi) ((xi)) o(1).
where sgn(z) [(x 0} 1f < 0} with sgn 0.
Assumption I states that there exists an underlying regression function f such that the observed outcomes Y; can decomposed into a systematic component / (X.) and zero-mean noise term Ei conditional on X; This is standard assumption in supervised learning and nonparametric regression, ensuring the model is well-specified in the conditional expectation sense. Assumption| 2 imposes moment condition that ensures the second moments of the response variable Y,, the true

---

regression function f (X;). and its estimator f (X,) are all finite almost surely. Assumption 3 is standard in the semi-superrised inference literature and has been adopted in works such as [29]
Assumption 4 follows conjecture from [30], asserting that n - xc, the dependence among inclusion indicators for any fixed subset instances vanishes asymptotically. Assumption 5 states
that sgu( (f [(;;) ( (())))())))))))))) is good estimator of [(X;) - ((X,))((X,), and we impose this to facilitate
analytical tractability.
Theorem (Asymptotic normality for mean estimation). Suppose Assumptions
7 hold,
(a) For the balanced active sampling scheme, the estimator 0 defined in vn(O 8*) 4 NO, Vo),
5 satisfies
where Vo- E(E?/T1) + (ar[f
(b) For the traditional active inference with {Eilimi being independent, if E[f((x)] = 8", the estimator 0 in (3) satisfies:
(0 Vo +E ((x)))
vn 0*) N 0,
((()))
The proof of Theorem 1 provided in Appendix A [ the asymptotic normality of the balanced active inference estimator. Specifically, the asymptotic variance of the proposed method consists of two components, including the scaled noise variance and the intrinsic variance of f(X)). Notably, this variance is reduced compared to that the classical active inference methods, as the additional variability from estimation error in f(x,) )) and non-uniform sampling is mitigated through the balancing constraint. Furthermore, the balanced active inference does not require the
predictive model f(-) to be unbiased, it more robust than existing active inference methods.
Remark 2. Assumption S is introduced to enable the derivation of an explicit expression for the asymptotic variance of the proposed balanced active inference method, but it not strictly necessary in practice. We conjecture that the proposed estimator retains its variance reduction benefits even when Assumption 5 mildly violated. Empirical evidence in Section 3 supports our conjecture.
5
Experiments
In this section, we evaluate the performance of our proposed method through both numerical simula- tions and real data applications. More numerical results about the real data analysis are shown in Section S3 of the Supplementary Material.
Datasets We consider three synthetic experiments, including a linear setup, nonlinear setup and
a Friedman setup [31]. Besides, we also consider real data applications. including six regression
datasets and two classification datasets. Regression datasets include Bike Sharing [32). Communities
and Crime 333 Concrete Compressive Strength (34, Energy Efficiency [35), Life Expectancy (36].
Superconductivity Data [37], and binary classification datasets including Credit Fraud Detection 38
and Post-election Survey Research
Baselines Our method (cube-active) is compared with three baseline methods across all exper- iments. As introduced in Section 2 and Section 3 the baselines include: (i) simple random sampling labeling strategy using sample mean estimator Ô 1 Σラ Y, with no involvement of machine learning models (classical); (ii) prediction-powered inference with GD estimator
i - Σm f(x,) + (Y, f(x,)) & using uniform random labeling strategy (uniform); and (iii) active inference based on independent sampling strategies designed using machine learning model predictions with GD estimator (rraditionalacctive)
For the four methods, we first report their Root Mean Squared Error (RMSE), which directly quantifies the deviation between point estimates and the true population mean. Addi- tionally, leveraging the asymptotic normality of each method and its respective variance estimator, we compute the confidence intervals at fixed confidence level 0.9, and compare their empirical coverage rates with respect to the true population mean. The variance estimators of each method are shown in Section S2 of the Supplementary Material.
Evaluation metrics

---

Experiment setup For all the datasets, the reported results are based on T 10 000 Monte Carlo simulations. Following recommendations from the traditional active inference literature [0], we set т 0.5 in 6 across all experiments. All predictive models are obtained by XGBoost [40] All experiments were conducted on a machine equipped with an Intel@) Xeon@ Gold 5118 CPU @ 2.30GHz, featuring 12 cores and 24 threads.
Protocol The numerical analysis proceeds as follows. First, we generate (X, Y) pairs and randomly split them into training/test sets. Then, an XGBoost regressor f()) trained on the training set, and an uncertainty model i(-) is fitted using XGBoost on if(x) Yil in D:. The specific model hyperparameters used for each dataset are detailed in Section S4 of the Supplementary Material. The details of computational resources and efficiency are provided in Section of the Supplementary Ma- terial for completeness. The cube method is implemented by the R package salancesamplingg
Python's rpy2 interface 41 Other experimental details are provided in Section S4 in Supplementary Material.
5.1 Performance across diverse budgets
In this subsection, we evaluate the proposed method on three representative datasets under different labeling budgets, covering both synthetic and real-worth scenarios. Specifically, we consider synthetic regression dataset generated by the nonlinear model
y = 10 (0ssi((Tr (in())))))))) + 20(23 0.5)² + 10x4 + 5xs +
where the predictors 21..... 210 are uniformly distributed on [0, 1] and follows a standard nor- mal distribution; (ii) real regression dataset (UCI Bike Sharing); and (iii) real classification dataset (Credit Fraud Detection) with severe class imbalance. These datasets together provide comprehensive test bed to evaluate the method's performance across different task types and data characteristics.
The results in Figure reveal that the proposed cube-active method consistently outperforms alterna- tives across all labeling budgets. It achieves the lowest RMSE and the narrowest 90% confidence intervals, demonstrating superior predictive performance and sharper uncertainty quantification. Moreover, the empirical coverage rates closely match the nominal level in all cases, highlighting the validity and robustness of the inference procedure. Notably, even in the imbalanced classification setting, our method maintains stable coverage and tight intervals, highlighting its effectiveness across varying scenarios and labeling budgets.
5.2 Efficiency improvement
Table presents the confidence interval widths (with empirical coverage rates in parentheses) for all methods under a labeling budget of 0.1. Across the majority of datasets, the empirical coverage rates remain closely aligned with the nominal confidence level of 0.9, confirming the validity of the asymptotic normal property and the statistical reliability of the proposed estimator.
Our proposed cube-active method achieves substantial and consistent efficiency gains over all baselines. Compared to traditional active inference based on independent sampling, cube-active reduces confidence interval widths by approximately 25%-85% across both synthetic and real- world datasets. When benchmarked against classical methods including uniform and simple random sampling, the improvement remains significant, with at least 30% narrower intervals all tasks.
These gains can be attributed to the enhanced covariate balancing induced by the cube sampling design, which effecttivly controls estimator variance by aligning the labeled set with the underlying feature distribution. As a result, cube-active achieves sharper inference and more precise uncertainty quantification without sacrificing coverage validity. This improvement is particularly valuable in low-budget regimes, where efficient use of labeled data is crucial for reliable statistical inference.
5.3 Label budget saving
Label budget saving refers to the percentage reduction in the required sample size by our method compared to raditional-active inference under a given estimation accuracy. We establish a precision benchmark using the confidence interval width of traditional-activ inference at label budget 0.2.
7

---

(yyabetic)
(gynthoti)
0.5
0.3
Inserval Wieh
RASSE
0..8
0.5
0.2
0.6
0.4-
0.2
0.0
500
1000
1500
2000
0.0
500
1000
1500
2000
0.0
500
1000
1600
2000
Sample Stzee
Sample Stzte
Sample Stor
Bike Sharing
Bike Sharing (3)
Bike Shariu
1.0
0.9
Interva Walth
0.5
0.4
10
10000
2000
3000
4000
1000
2000
3000
4000
1000
2000
2000
4000
Sample Sizee
Sample Size
Sample Size
Credit Fraud Desection (a)
Crodit Frond Detection (b)
Czedit Fraud Detection (6)
1.0
0. 0000
0.0020
0.0005
0.8
Interved Whahs
0.0015
Coretage
2.00 2.0010
0.0002
0.0005
0.2
0.0001
20000
40020
60000
20000
40000
60002
0.9
20000
60000
Sample
Sample
Sample
Figure Performance comparison across three datasets with varying sample sizes. The top row (synthetic Friedman dataset), middle row (UCI Bike Sharing dataset), and bottom row (Credit Card Fraud Detection dataset) each display: (a) Root mean squared error (RMSE); (b) Average width of the 90% confidence intervals, reflecting inference precision; and (c) Empirical coverage rate, confirming interval estimation validity. The proposed cube-active sampling method consistently achieves superior performance across these datasets.
Table 1: Comparison of confidence interval width with 0.1 label budget across methods on 11 datasets. The bold values indicate the narrowest confidence intervals under valid coverage.
Dataset
classical
uniform
traditional-active
cube-active
Linear (synthetic)
0.3171 (0.8998)
0.1690 (0.9053)
0. 1444 (0.8973)
0.0722 (0.8998)
Nonlinear (synthetic)
0.7786 (0.931))
0.3626 (0.8923)
0.2963 (0 (0.8967)
0.1370 (0.9063)
Friedman (syathetic)
0.6803 (0.8955)
0.3798 (0.9440)
0.3305 (0.8979)
0.1052 (0.9056)
Bike
19. 19.2033 (0.9028)
2.8213 (0.8987)
2.2052 (0.8977)
0.3316 (0.8969)
Communities
0.0722 (0.8912)
0.0516 (0.8942)
0.0509 (0.8900)
0.0466 (0.8885)
Concrete
7.1217 (0.8970)
3.9367 (0.9024)
3.7415 (0 (999)
2.6406 (0.8943)
ceddi-fraau-detecee
0.0011 (0.8811)
0.0007 (0.9300)
0.0006 (0.8985)
0.0005 (0.8700)
Energy
5,1520 (0.8938)
2.0364 (0.8544)
1.8944 (0.9055)
0.4136 (0.8025)
Life
3.0828 (0.8967)
1 1.4509 (0.8633)
1.3050 (0.9036)
0.7265 (0.8959)
Post-election
0.0571 (0.8949)
0.0426 (0.9119)
0.0403 (0.8998)
0.0381 (0.8980)
Superconductor
3.2664 (0.9040)
1.6711 (0.9067)
1.5950 (0.9002)
1.1680 (0.9003)

---

For each dataset, we determine the minimal label budget required by alternative methods to achieve the most similar confidence interval width of the benchmark. Given our experimental grid of label budgets (0.03" 0.45 with 0.01 increments), we employ linear interpolation between adjacent budget points for precision matching. When a method's precision at the minimal tested budget (0.03) exceeds the benchmark, its required budget is conservatively denoted as "< 10%". Conversely, if precision remains below benchmark at the maximal budget (0.45), the required budget is cautiously reported as "> 150%"
Table 2 quantifies the label budget efficiency required by different methods match the precision benchmark of cuditional-active at 0.2 label budget. Our method achieves substantial budget savings across all datasets, requiring less label budget of the benchmark to attain equivalent precision. In several cases, our method attains superior precision with less than 40% of the benchmark budget, demonstrating exceptional statistical efficiency. Compared uniform sampling, our method only needs 5% or less labeled instances to achieve higher precision in real-world applications. These sav- ings stem from our method's optimal balanced sampling that simultaneously maximizes information gain and minimizes distributional discrepancy. The method's s adaptive balancing mechanism proves particularly effective high-dimensional settings where conventional active learning methods exhibit diminishing returns due to covariate mismatch. This systematic budget reduction, coupled with maintained statistical validity, establishes cube-active as a practical solution for label-constrained inference scenarios.
Table Comparison of budget saving with the precision benchmarks across methods on 11 datasets.
Dataset
classical
uniform
traditional-active cube-active
Linear(synthetic)
>150%
127%
100%
37%
Nonlinear(syathenn
>150%
117%
100%
70%
Frredtman(ynthetii)
>100%
123%
100%
33%
Bike
>150%
147%
100%
63%
Communities
150%
103%
100%
90%
Concrete
>150%
110%
100%
67%
Credit-fruudddetection
>150%
110%
100%
63%
Energy
>1.0%%
113%
100%
<10%
Life
>150%
123%
100%
43%
Post-election
>150%
107%
100%
93%
Superconductor >150%
110%
100%
63%
5.4 Performance of M-estimation problems
Figure Summarizes the statistical performance of our method in estimating linear regression coeffi- cients for selected variables across the Linear and Bike Sharing datasets. Owing to the complexity of the underlying estimation procedures. deriving closed-fome form variance expressions for the estimators is intractable. Instead, we report the RMSE under different settings, which serves as a robust indicator of statistical efficiency and allows for clear performance comparisons across methods.
The results in Figure consistentty highlight the advantages of the proposed cube-active sampling strategy. For both the synthetic linear dataset and the real-world Bike Sharing dataset, cube-active sampling yields substantially lower RMSE values compared to all baseline methods, including uniform sampling, classical inference, and traditional active inference. These improvements illustrate the method's ability to provide more accurate coefficient estimation, thereby enhancing the overall statistical efficiency of the inference procedure.
6 Discussion
In this paper, we propose balanced active inference and demonstrate its superior statistical efficiency theoretically and numerically in various synthetic setups and real data analysis; Section (88 and [S9 of Supplementary Material for the limitations and societal impacts. A natural direction for future research is to explore the extension of this method into sequential active inference framework.
Transitioning balanced active inference to a sequential setting introduces intriguing yet challenging methodological considerations, The principal challenge lies in the dynamic balancing requirement.

---

(a) Linear (synthetic)
(b) Bike Sharing
200
150
0.06
RMSE
RMSE
100
0.04
0.02
0
500
1000
1500
2000
1000
2000
3000
4000
Sample Size
Sample Size
clossical
uniform
traditional active
cube active (ours)
Figure 2: Root mean squared error (RMSE) of the least squares estimator in linear regression across two datasets with varying sample sizes. (a) the parameter of x1 in synthetic Linear dataset (b) the parameter of temperature variable in Bike Sharing dataset. The proposed cube-active sampling method consistently achieves superior performance across these datasets.
In contrast to the batch scenario, where auxiliary variables are fixed, sequential balanced sampling involves continuously updating auxiliary variables when new instances become available. An adaptive implementation of the cube method, or similar balanced sampling procedure, needs to integrate seamlessly with evolving model predictions and uncertainties in computationally efficient manner. This dynamical adjustment could further reduce variance and improve representativeness, allowing for more precise allocation of labeling resources.
However, theoretical justification for such sequentially balanced sampling remains open. Extending existing results, such as martingale-based analyses that underpin sequential active inference, to balanced sampling contexts is not straightforward due to dependencies and complexity introduced by continually updated balancing constraints. Establishing rigorous statistical properties, such as unbiasedness, variance reduction, and asymptotic normality, within this sequential balanced framework will likely require novel analytical techniques or approximations.
Empirically, preliminary exploration through synthetic experiments and real data applications would be valuable first step toward understanding sequential balanced active inference. Future studies should investigate various update strategies, examining how frequently and substantially the balancing conditions should be adjusted. Developing heuristics and computationally efficient algorithms capable of handling online updates of the balancing constraints would substantially advance the feasibility of sequential balanced active inference. Such advancements have great potential to further enhance labeling efficiency in practical applications, where data arrives continuously, and timely decision- making is critical.
Acknowledgments and Disclosure of F Funding
The authors would like to thank anonymous reviewers for their valuable comments. Wang's research supported in part by Humanitics and Social Sciences Foundation of the Ministry of Education of China Grant (No. 23YJA910005) NSFC (No. 12571291, 72533007, 71988101, 72033002), Peng was supported by ARC (Grant No. LP240100101).
References
[1] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, 542(7633):115-111 2017.
[2] Wei Bao, Jun Yue, and Yulei Rao. A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PloS one, (27))e0180944: 2017.
10

---

[3] Neal Jean, Marshall Burke, Michael Xie, William Matthew Davis, David B Lobell, and Ste- fano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353(6301):790-794) 2016.
[4] William G Cochran. Sampling Techniques. John Wiley & Sons, 1977.
[5] Michael Traugott. The accuracy of opinion polling and its relation to its future. In The Oxford Handbook of American Public Opinion and the Media, pages 316-331. Oxford University Press, 2011.
[6] Burr Settles and Mark Craven. An analysis of active learning strategies for sequence labeling tasks. In proceedings of the 2008 conference on empirical methods in natural language processing, pages 1070-1079, 2008.
[7] Burr Settles. Active learning literature survey. Technical Report, 2009.
[8] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1183-1192. PMLR, 06-11 Aug 2017.
[9] Dongyuan Li, Zhen Wang, Yankai Chen, Renhe Jiang, Weiping Ding, and Manabu Okumura. A survey on deep active learning: Recent advances and new frontiers. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[10] Tijana Zrmic and Emmanuel Candes. Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62993-6333 63010. PMLR, 21-27 Jul 2024.
[11] Anastasios N. Angelopoulos. Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zmic. Prediction-powert inference. Science, 382(6671).669-67777 ):669-674, 2023.
[12] Jerzy Neyman. On the Two Different Aspects of the Representative Method: the Method of Strutified Sampling and Method of Purposive Selection, pages 123-150. Springer, 1992.
[13] Jaroslav Hájek. Asymptotic theory of rejective sampling with varying probabilities from a finite population. Annals of Mathematical Statistics, 35(4): 149:)113,, 1964.
[14] Richard M Royall and Jay Herson. Robust estimation in finite populations I. Journal of the American Statistical Association, 68(344):880-888 1973.
[15] Jean-Claude Deville and Yves Tillé. Efficient balanced sampling: The cube method. Biometrika, 91(4):893-912) 2004.
[16] Yves Tillé. Ten years of balanced sampling with the cube method: An appraisal. Survey methodology, 37(2)215-222 2011.
[17] Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
[18] Donald B Rubin. Multiple imputation after 18+ years. Journal of the American Statistical Association, 91(434):473-488 489, 1996.
[19] Anru Zhang, Lawrence D Brown, and T Tony Cai. Semi-supervised inference: General theory and estimation of means. Annals of Statistics, 47(5):2538 2019.
[20] David Azriel, Lawrence D Brown, Michael Sklar, Richard Berk, Andreas Buja, and Linda Zhao. Semi-supervised linear regression. Journal of the American Statistical Association, 117(540):223- 22118,2221111111 2022.
[21] Greg Schohn and David Cohn. Less is more: Active learning with support vector machines, In Proceedings of the Seventeenth International Conference on Machine Learning, volume page 839-846, 846, 2000.
[22] Burr Settles. From theories to queries: Active learning in practice. In Active Learning and Experimental Design Workshop in Conjunction with AISTATS 2010, pages 1-18. JMLR Workshop and Conference Proceedings, 2011.

---

[23] Claes M Cassel, Carl E Särndal, and Jan H Wretman. Some results on generalized difference estimation and generalized regression estimation for finite populations. Biometrika, 63(3):615- 620, 12 1976.
[24] Al. A. Tchouproff. On the mathematical expectation of the moments of frequency distributions. Biometrika, 12(112): 140-169, 1918.
[25] Morris H Hansen and William N Hurwitz. On the theory of sampling from finite populations. Annals of Mathematical Statistics, 14(4):333-366 1943.
[26] William G Madow. On the theory of systematic sampling, II. Annals of Mathematical Statistics, 20(3):333-354, 1949.
[27] Frank Yates. A review of recent statistical developments in sampling and sampling surveys. Journal of the Royal Statistical Society, 109(1)::2-43, 1946.
[28] Frank Yates. Sampling methods for censuses and surveys. Charles Griffin & Co., Ltd., 1949.
[29] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983.
[30] Laurent Davezies, Guillaume Hollard, and Pedro Vergara Merino. Revisiting randomization with the cube method. arXiv preprint vrXiv:2407.13613 2024.
[31] Jerome H Friedman. Multivariate adaptive regression splines. Annals of Statistics, 19(1):1-61 1991.
[32] Hadi F and Joao Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 2:113-127, 2014.
[33] Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research, 141(3):660-66 678, 2002.
[34] I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete Research, 28(12):1797-1808 1998.
[35] Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and Buildings, 49:560-
567, 2012.
[36] Abhinav Kumar.
Life expectancy (WHO).
Kaggle,
2020.
[37] Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a
superconductor. Computational Materials Science, 154:346-354, 2018.
[38]
Emmanuel
Lopez.
Credit card fraud detection.
Kaggle, 2019.
[39]
Pew.
American
trends panel (atp) wave 79,
2020.
[40] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '1. page 785-794, New York, NY, USA, 2016. Association for Computing Machinery.
[41] Don L Stevens Jr and Anthony R Olsen. Spatially balanced sampling of natural resources. Journal of the American Statistical Association, (9246))22 2004.
[42] Jiahua Chen and J. N. K. Rao. Asymptotic normality under two-phase sampling designs. Statistica Sinica, 1763):0047-104, 2007.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The formulation of the proposed balanced active inference can be found in
Section Theoretical properties can be found in Section and numerical results can be found in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations our proposed method are discussed Section S8 of
Supplementary material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: All the assumptions can be found in Section and all the detailed proof's can be found in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions and formulations required reproduce the main experimental results, including information on the model, datasets, baselines, and evaluation metrics. Additionally, we present the implementation details in Section S4 of Supplementary Material to offer clearer implementation guidance. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The datasets utiliized in this study publicly accessible via the cited refer- ences, and the implementation code is available upon request.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.cc/pubcbc/uuuceeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main settings can be found in Section 5 and detailed settings and additional experiments are presented in Section SS4 and S3 of Supplementary Material to provide further insights into the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The coverage rates of the proposed estimator and its alternatives are reported in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are clearly stated in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httts::/neurr/ http:////wnnppp...uu/u/////////a
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics and have ensured that all aspects of research fully comply with its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of our proposed method are discussed in Section S9 of the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that our work poses no foresecable risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used in our experiments are properly cited, and source links provided for reference.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method development in this work does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

Appendix
A
Proof
In this section, we provide the detailed proof of Theorem lemmas are presented.
1
Before the formal proof, some technical
A.1 Technical Lemmas
Lemma 1 (Proposition 1 in (00). Suppose Assumptions Cand hold and a cube method is conducted to balance auxiliary information {X; i ,,...nn) n. under sampling scheme { I,...y nt.,
then
X
n Σ x; Op (17)
9
n
T
7
where & (0,1) indicates the i-th instance is selected.
Lemma (Lemma C.2 in (0). Let f and g be two functions such that for f. = (6,(1), x,) and 9: = 9 (1), 5,0)), X,) we have E (f? + 9.) < 0a and Ef X,] = E [g: X,] = 0.
If Assumptions 2 and hold. Then, conditional on (Xi)
f + D:D. 4 N (0,
with Vo = E (291f1 + g m1
Lemma 3 (Theorem in [42 Let Un, Vn be two sequences of random variables and B, be
r-algebra. Assume thut
(1) there exists din > 0 such that oin Vn + N(0,1) in distribution as n + co, and Vn is B, measurable;
(2) E {Un B,} = and Var (Un Br) = o2n such that
sup P P Un St Br)
=
where ((1) is the cumulative distribution function of the standard normal random variable;
(3) 2 = 2 72 in probability as n + cc. Then
U, + Vn
NO,
In o2
in distribution as n + co.
A.2 Proof of Theorem
By definition, we have
f(X.) (v, f(X)) &
0 0"
BY
T(X;))
f(X)) f(xi+ + (yi f(x,))
Σ (f(X.) EY). (8)
T(X,).
20

---

For the first term in
8 we have
n
f(x.)- - 5(Xi) + (Yi- f(x,)) &
T(X.))
71
n
[f(x)) - -((x)) + sgn (f(X.) - s(X.)) i(Xi)
71
(I: - f(X.) +,- - sgg (f(X))-((x.)))
- ((x:)) a(Xi)) &
((x))
1
Σ a(x,) + 1
n2 Σ a(x,) &
x(xi)
n
[[(x))) - f(,,) + sgu
(cc))-f(C (f(x))-f(x)) [(X))] (1- &,
m(xi))
(9)
1
Ei
+ op (v)
Ei (X))
[f(x,) - fXx) sgn (f(X.) f(xi)) ((Xi)] (1xx(t)) )
Denoting
by A and applying
Chebyshev's 's inequality on
(
we have
P
1
Σ
A - EAg 2
1
-
1 n
Var
Σ A
(10)
Vni
1=1
Further, we have Var EE= A.) = Em Var(A:) + Eitt Cov(Ar, Az). For Var(Ar), we have
Var(A.) =B =B (F(X)) ((x.) + sgn (f(X.) - -(xx)) a(x:)) 2
E x(.))
27
E [(((x)) f(x.) - sgn (f(X.) - f(X,)) i(xi)] (1 (1- E,
2
x (X.))
Since TXX)) is bounded from to c, there exists a positive constant C' such that
E (f(X)) -f(x))
f(X,) + sgn (f(X)) - f(X)) a(x.)) (1-
Es T(Xi)
5
27
SC E [((X)) - f(X,) (f(x)) + sgn (f(x,) - f(Xx)) m(xx))
Further, by the assumption that E [f(X.) -((x)) sgn (f(x.) - -((x)) a(x.)] (1))), we
have (A,) = (1). For Cov(Ai, = [A, - E [A, E [Ay], first,
E [ArA]]
B (f(x.)- - ((X.)) + sgu (f(x))- - s(x:) a(x.)) Es (Xx)
*(X:)
(A(x,) f(X,) + sgn (f(X.) - f(X.) b(x,))
E 5- (x) =(X,)
((X,) -f(X) + sgn (f(X.) -((x)) a(X,)
(X,)
f(X,) - f(X,) + sgn ([(X.) - ((X.) ((X,)
T(x))
Xx, xx] } E [[G - ((--m(Xx)) (5, - =(x)))
21

---

By Assumption we have with probability 1. lim (E. x(xi)) - ((x))) X X, X,, 0. Thus, with probability 1,0 IE [A:As] = O. And by similar argument, have [AA]] E [f(x..) - f(X) + sgn
a(X;))
( 0 n co. Thus, it can
x(x,)
be concluded that, with probability 1, Eits Cov(AI, A2) = 0. Therefore, we have Var (2 A.) (1), then (TO) implies that
IP (
1 -
Σ A.
-BA,
RA,
2 Th
= (1)
Thus we have
sgn (1ixx)- [(x)) sgn (f(x:) [ ((x)))((xa)
(1-xxx))
G
0pl 1
By Lemma 2 we have conditional on X1, X2,..., Xn
E1
d N 0,
,; )
T (X:)
Thus, applying Lemma
3 we have
s T
vn y EY
N
O,E E
+ Var
(f(x)))
If a Poisson sampling is used, by a standard CLT, we have
vā(0- 0*) - N((0,
where
V12 Var
(f(x)) ((xx)((xi))) (x)))+x(1))))
x =(x))
E
EE (F(X)) + (X- - / ((1))
E1 T (X,)
{x,,) (x1
Var
+ E Var (F(XI)= + (xi - (Xx1)
E2
7 =(1)) [x1
Var(Yi)) + +1 [(xi - f (x,)) 2 21-=(X1) xXx)
x (X1)
+ (1(z)
- )(x)))
(क()) ( -1) )
1
Var(fi) E
Then, the proof is completed.
22


Reviews:
[{"review_id": "xWPKNfblNb", "reviewer": "Reviewer_f7YV", "summary": "This paper addresses the statistical inefficiency caused by traditional independent sampling methods. While existing active inference approaches improve upon uniform sampling by prioritizing instances with high model uncertainty, their reliance on independent selection can lead to imbalanced labeled sets and inflated variance.\n\nTo solve this, the authors propose Balanced Active Inference, an algorithm that integrates balanced sampling principles into the active learning process. The core idea is to treat the model's uncertainty estimates as auxiliary variables and enforce a balancing constraint during label selection.", "questions": "I have a few concerns on the tractability of this method. Specifically,\n\n- Could you provide some insight into the computational complexity of the cube method as used in your algorithm? How does its runtime scale with the size of the unlabeled pool n? Are there potential scalability challenges when applying this method to massive datasets with millions of unlabeled instances?\n\n- A simpler approach to achieve some form of balance would be to stratify the data based on quantiles of the uncertainty and then sample proportionally from each stratum. How would you compare your method, which achieves exact continuous balancing, to such a stratified sampling approach, both theoretically and empirically?", "limitations": "I have mentioned the limitations I foresee in questions and weaknesses sections above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n- The idea of using model uncertainty as the balancing variable within the cube method framework is interesting. It directly addresses a well-known weakness of uncertainty-based sampling (variance inflation) with a principled statistical solution.\n\n- The paper is exceptionally well-written. The authors do a good job of motivating the problem, explaining the limitations of prior work, and introducing their proposed solution.\n\n- The experimental validation is thorough and convincing. The authors test their method on a wide array of synthetic and real-world datasets, covering both regression and classification tasks. The use of multiple metrics (RMSE, confidence interval width, and coverage rate) provides a complete picture of the method's performance.\n\n\n*Weaknesses*\n\n- The paper leverages the cube method, which may be unfamiliar to parts of the core machine learning audience. While it is a standard technique in another field, a slightly more intuitive explanation of its mechanics in the main text could improve accessibility. Furthermore, the computational complexity of the cube method itself, especially for very large unlabeled datasets, is not discussed, which could be a practical concern for scalability.", "was_revised": true, "final_justification": "My concerns are addressed and I shall maintain the score.", "rebuttal": "# **Response to Reviewer f7YV**\n\nThank you for your recognition of the novelty and methodological rigor of our research, as well as the thorough experimental validation presented. We greatly appreciate your constructive feedback and critiques, which are invaluable for refining our manuscript. Below are point-by-point responses incorporating your suggestions:\n\n## **Responding to Weaknesses**\n\n> ### **Weakness 1. Intuition of Cube Method**\n\nThanks for the valuable suggestion of providing **a more intuitive explanation of the cube method**. We will add the following explanation in the revised version:\n\n- Covariate balancing is a commonly used statistical technique to **improve statistical efficiency** when designing a sampling scheme.\n\n- The cube method (Deville & Tillé, 2004) achieves covariate balancing by consecutively updating selection probabilities to satisfy certain balancing constraints.\n\n- Theoretically, the cube method achieves covariate balancing by sacrificing independence among sampled elements, making it challenging to investigate its asymptotic properties. \n\n- Practically, the cube method is easy to implement and can be seamlessly integrated in active inference.\n\n## **Responding to Questions**\n\n> ### **Q1. Computational Complexity of Cube Method**\n\n- As noted in Tillé (2011), the computational complexity of the cube method is $\\mathcal{O}(N \\times p^2)$, where $N$ is the **population size** and $p$ the **number of balancing covariates**. \n\n- In our implementation, **only one** auxiliary covariate ($\\hat{u}$, the uncertainty measure) is used ($p=1$). Thus, complexity reduces to $\\mathcal{O}(N)$, meaning runtime scales linearly with the unlabeled size $n$. \n\n- This linear scaling enables efficient handling of large datasets. \n\n- A discussion on the computational complexity of the cube method will be added in the main text.\n\n> ### **Q2. Stratified Sampling**\n- Thanks for the thoughtful suggestion of trying a stratified sampling approach. \n- New experiments are implemented including the **baseline** of **stratified sampling**: stratifying by $\\hat{f}(X)$ (predicted labels; `Stratified_Y`) and $\\hat{u}$ (predicted uncertainty; `Stratified_u`) using 5 quantile bins (width=0.2) and sampling proportionally from each stratum.\n\n- Below is a representative subset of our experimental results:\n\n**Table 1 Bike Sharing, RMSE**\n\n| Budget | Sample Size | `Poisson_Active` | `Cube_Active` | `Stratified_Y` | `Stratified_u` |\n|--------|-------------|----------------|-------------|--------------|--------------|\n| 0.1    | 869         | 1.777          | **0.128**       | 1.754        | 1.795        |\n| 0.2    | 1738        | 1.146          | **0.079**       | 1.141        | 1.140        |\n\n**Table 2 Friedman, RMSE** \n\n| Budget | Sample Size | `Poisson_Active` | `Cube_Active` | `Stratified_Y` | `Stratified_u` |\n|--------|-------------|----------------|-------------|--------------|--------------|\n| 0.1    | 500 | 0.093    | **0.046**       | 0.093        | 0.092|\n| 0.2    | 1000        | 0.060     | **0.030**       | 0.061        | 0.060|\n\n**Table 3 Credit Fraud Detection, RMSE**  \n\n| Budget | Sample Size | `Poisson_Active` | `Cube_Active` | `Stratified_Y` | `Stratified_u` |\n|--------|-------------|----------------|-------------|--------------|--------------|\n| 0.15   | 21,360         | 0.000155     | **0.000120**       | 0.000160        | 0.000150|\n| 0.35   | 49,841      | 0.000088     | **0.000066**       | 0.000090        | 0.000087|\n\n- Experiments on the datasets show that cube sampling achieves significantly lower RMSE due to its exact continuous balancing.\n\n- Stratification balances auxiliary covariates **discretely**, whereas the cube method enforces exact population-level balance for **continuous** $\\hat{u}$. This eliminates bias in uncertainty estimation, which is critical for active inference performance.\n\n## References\n- Deville, J. C., & Tillé, Y. (2004). Efficient balanced sampling: the cube method. Biometrika, 91(4), 893-912.\n- Tillé, Y. (2011). Ten years of balanced sampling with the cube method: an appraisal. Survey methodology, 37(2), 215-226."}, {"review_id": "2TveYSYkgf", "reviewer": "Reviewer_WQHA", "summary": "This paper introduces balanced active inference, a method for estimating the label mean when only a small portion of a large unlabelled data pool can be labeld. The proposed method first learns uncertainty scores and then draws balanced samples with the cube method. After the selected points are labelled, they are used to produce the estimated mean. This paper proves asymptotic normality and shows that the proposed estimator’s variance is lower than that of the baselines. Experiments on synthetic datasets demonstrate better mean estimates comparing to previous methods.", "questions": "How does Theorem 1 reflect on the choice of $\\\\pi$ other than simply $\\\\mathbb{E}[\\\\pi_1]$? For example, does Theorem 1 hold for any form of the probability $\\\\pi$ satisfying the balanced sampling constraint? Clarifying this would help the readers to understand how tightly the theoretical benefit is coupled to the particular $\\\\pi$ defined in the paper.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe problem formulation is clearly formulated, the method and results are cleanly presented. The proposed method can be easily implemented, and the experiments support the theoretical claim. In general, the proposed method is conceptually elegant and has the potential to be practically impactful.\n\n**Weaknesses**\n\n* The scope the paper is very narrow, as it only focuses on one-dimensional mean estimations. Although the presentation is clean within the scope, the narrow scope reduces the significance for general ML audience. For example, can the proposed method, both theory and algorithm, be applied to classification tasks? \n* The only theoretical result, Theorem 1, is essentially an application of standard tools, e.g., Chebyshev’s inequality and central limit theorem, on top of existing work (Lemma 1 to 3).  A novel proof technique, or results beyond asymptotic normality, would strengthen the paper given the narrow scope. \n* Related to the above point, the Assumption 5 is very strong. It basically asks for an accurate uncertainty estimator. Is obtaining an accurate uncertainty estimator easier than a similarly accurate label estimator itself? \n* It is mentioned in the paper that the mixing hyperparameter $\\\\tau$ is chosen to be 0.5. It would be interesting to see how this hyperparameter influence the performance both theoretically and empirically. \n* Some of the claims in the paper are not well supported. For example, it is said \"safeguard against distributional shifts,\" but there seems to be no evidence to support this.", "was_revised": true, "final_justification": "As mentioned by the authors, the concerns over the theoretical results are not easy to address. However, the additional experiments make the paper more complete, and thus I have raised my rating.", "rebuttal": "# **Response to Reviewer WQHA**\n\nWe sincerely appreciate your recognition of the novel methodological contribution, compelling experimental validation, and significant practical implications of our work, as well as your valuable feedback and critiques, which are crucial for refining our paper. Below we provide point-by-point responses that incorporate your suggestions.\n\n## **Responding to Weaknesses**\n\n> ### **Weakness 1. Scope of the Paper**\n\nThanks for raising concerns about the scope of the paper. While the generation to M-estimation was briefly discussed in the main text, we will delve deeper by formally extending the Balanced Active Inference framework to the general M-estimation setting, **covering both regression and classification**, and present some experimental results.\n\n#### **(i) Settings of M-estimation**\n\n* Given a family of functions $f(X_i;\\theta)$, the task is to solve $ \\theta^{\\ast}= \\arg\\min_{\\theta} \\mathbb{E} \\left [L\\bigl(X_1, Y_1; \\theta\\bigr)\\right]$.\n* $L(X_1, Y_1; \\theta)$ measures the discrepancy between the true label $Y_1$ and the prediction $f(X_1;\\theta)$.\n* Our goal is to estimate $\\theta^{\\ast}$ using a small labeled set $ \\mathcal{D}_l$ and a larger unlabeled set $ \\mathcal{D}_u $.\n* $ f(X_i;\\hat{\\theta}) $ is an initial estimator trained on $ \\mathcal{D}_l $.\n\n#### **(ii) Uncertainty Measures**\n\n- **Regression.** We use the absolute residual $ |Y_i - f(X_i;\\hat{\\theta})| $ as the uncertainty measure.\n- **Classification.** Let $ p(X_i) = (p_1(X_i),\\ldots,p_K(X_i))$ denote the predicted class probabilities. We define $$u(X_i)=\\frac{K}{K-1}\\Bigl(1-\\max_{j\\in[K]} p_j(X_i)\\Bigr),$$ \nwhich peaks when the model is maximally uncertain (uniform distribution) and drops to zero when the model is highly confident in one class.\n\n#### **(iii) M-Estimation**\n\n* A model $\\hat{u}(\\cdot)$ is trained on $\\mathcal{D}_l$ to predict the uncertainty of points in  $\\mathcal{D}_u $. \n\n* Given a labeling budget $n_b$, adopt the sampling scheme described in Equation (2) of the paper.\n\n* Generate an assignment via the cube method, ensuring the balancing constraint (Equation (4)) is satisfied.\n\n* The proposed estimator is\n\n  $$\\tilde{\\theta}=\\underset{\\theta}{\\arg \\min } \\frac{1}{n} \\sum_{i=1}^n\\{L(X_i, f(X_i ; \\hat{\\theta}) ; \\theta)+[L(X_i, Y_i ; \\theta)-L(X_i, f(X_i ; \\hat{\\theta}) ; \\theta)] \\frac{\\xi_i}{\\\\pi(X_i)}\\},$$\n\n  where $\\pi(X_i)$ is the inclusion probability.\n\n#### **(iv) Experiment Results**\n\n* Experiments are conducted on both synthetic data (Table W1.1) and the Bike Sharing dateset (Table W1.2).\n\n**Table W1.1 Synthetic Linear, RMSE**\n\n| Budget | Sample Size | `cube_active` | `poisson_active` | `uniform` | `classical` |\n| ------ | ----------- | ------------- | ---------------- | --------- | ----------- |\n| 0.1    | 500         | **0.0193**    | 0.0300           | 0.0480    | 0.0464      |\n| 0.2    | 1000        | **0.0128**    | 0.0183           | 0.0309    | 0.0325      |\n\n**Table W1.2 Bike Sharing, RMSE (First Feature: `temp`)**\n\n| Budget | Sample Size | `cube_active` | `poisson_active` | `uniform` | `classical` |\n| ------ | ----------- | ------------- | ---------------- | --------- | ----------- |\n| 0.1    | 869         | **32.014**    | 44.279           | 107.979   | 101.781     |\n| 0.2    | 1738        | **21.340**    | 29.348           | 69.920    | 67.390      |\n\n* The results show that our balanced active inference approach has the best performance in terms of RMSE.\n\nA section on M-estimation will be added in the revised version.\n\n> ### **Weakness 2. Limited novelty in theoretical contribution**\n\n- The theoretical foundations of the cube method are still under development.\n- As noted in the **most recent work** by Davezies et al. (2024), even in simplified settings such as standard survey sampling without model uncertainty, only aysmptotic results are avaliable.\n- Non-asymptotic guarantees remain elusive due to **(1) the complex dependence induced by the balancing step**, and **(2) the lack of general concentration tools for such designs**.\n- Given these challenges, our asymptotic analysis represents a meaningful and necessary step toward understanding the statistical properties of balanced active learning.\n- We view non-asymptotic extensions as an important and ambitious direction for future research, but one that lies beyond the scope of this paper.\n\n> ### **Weakness 3. Strength of Assumption 5**\n\n- We acknowledge that the assumption effectively requires an accurate uncertainty estimator, which may not always hold in practice. \n\n- As noted in **Remark 2** of our manuscript, Assumption 5 is primarily introduced to facilitate theoretical analysis, **particularly to derive an explicit form of the asymptotic variance**.\n  \n- This assumption is **not** a strict requirement for the validity of our method.\n\n- Empirical results consistently show that **the proposed estimator achieves substantial variance reduction even when the uncertainty estimators are imperfect**, suggesting robustness to mild violations of this assumption.\n\n> ### **Weakness 4. Sensitivity Analysis on $\\tau$**\n\nThank you for this important suggestion. We acknowledge that the choice of $\\tau = 0.5$ warrants explicit justification.\n\n- Sensitivity analysis on $\\tau$ across multiple datasets will be added in the revised version. \n- The setup is: fix the sampling budget at 0.2 and compute the RMSE and interval width (analogous to Figure 1 in the main text, but with $\\tau$ ranging from 0 to 1 on the x-axis) for methods.\n- Below is a representative subset of our experimental results:\n\n**Table W4.1 Bike Sharing, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ------- | --------- |\n| 0.00    | $\\quad$ **0.127**   | $\\quad$1.460          | 1.466   | 3.846     |\n| 0.25    | $\\quad$**0.093**   | $\\quad$1.261          | 1.466   | 3.846     |\n| 0.50    | $\\quad$**0.079**   | $\\quad$1.146          | 1.466   | 3.846     |\n| 0.75    | $\\quad$**0.072**   | $\\quad$1.064          | 1.466   | 3.846     |\n| 1.00    | $\\quad$**0.130**   | $\\quad$1.038          | 1.466   | 3.846     |\n\n**Table W4.2 Bike Sharing,  Interval Width**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ------- | --------- |\n| 0.00    | $\\quad$ **0.395**   | $\\quad$ 4.847          | 4.844   | 12.805    |\n| 0.25    | $\\quad$ **0.302**   | $\\quad$ 4.128          | 4.844   | 12.805    |\n| 0.50    | $\\quad$ **0.254**   | $\\quad$ 3.754          | 4.844   | 12.805    |\n| 0.75    | $\\quad$ **0.230**   | $\\quad$ 3.530          | 4.844   | 12.805    |\n| 1.00    | $\\quad$ **0.275**   | $\\quad$ 3.432          | 4.844   | 12.805    |\n\n**Table W4.3 Friedman, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ------- | --------- |\n| 0.00    | $\\quad$ **0.029**   | $\\quad$ 0.069          | 0.068   | 0.137     |\n| 0.25    | $\\quad$ **0.029**   | $\\quad$ 0.063          | 0.068   | 0.137     |\n| 0.50    | $\\quad$ **0.031**   | $\\quad$ 0.060          | 0.068   | 0.137     |\n| 0.75    | $\\quad$ **0.034**   | $\\quad$ 0.060          | 0.068   | 0.137     |\n| 1.00    | $\\quad$ **0.063**   | $\\quad$ 0.079          | 0.068   | 0.137     |\n\n**Table W4.4 Post-election survey research, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ----------- | --------- |\n| 0.00    | $\\quad$ **0.00841** | $\\quad$ 0.00865        | 0.00862     | 0.01165   |\n| 0.25    | $\\quad$ **0.00806** | $\\quad$ 0.00840        | 0.00862     | 0.01165   |\n| 0.50    | $\\quad$ **0.00790** | $\\quad$ 0.00824        | 0.00862     | 0.01165   |\n| 0.75    | $\\quad$ **0.00799** | $\\quad$ 0.00839        | 0.00862     | 0.01165   |\n| 1.00    | $\\quad$ 0.01014     | $\\quad$ 0.01047        | **0.00862** | 0.01165   |\n\n- All coverage rates in the above results closely approximate the target confidence level, confirming the validity of our findings.\n- Across all datasets, the RMSE and interval width **generally maintained the following ordering**: `cube_active < poisson_active < uniform < classical`.\n- At $\\tau = 0$: The performance of `poisson_active` becomes nearly equivalent to `uniform`, as both rely solely on simple random sampling.\n- As $\\tau$ increases, both `cube_active` and `poisson_active` exhibit **precision improvement** (decreasing RMSE and interval width), **reaching lower value** around $\\tau = 0.5$ for different datasets. This demonstrably quantifies the precision gains enabled by active sampling.\n- When near $\\tau = 1$, RMSE and interval width increase on some datasets. This occurs because the model for estimating prediction uncertainty ($\\hat{u}$) is inherently imperfect. When the model erroneously assigns near-zero uncertainty ($\\hat{u} \\approx 0$) to specific data points, it can significantly inflate the estimator variance. These findings are well-aligned with conclusions established in Zrnic & Candès (2024).\n\n> ### **Weakness 5. Claims not well supported**\n\nThanks for pointing out this issue. We will carefully check our claims, and add sufficient support or delete non-supported ones.\n\n## **Responding to Question**\n\n> ### **Q1. $\\pi$ in Theorem 1**\n\nThank you for the insightful question.\n\n- Theorem 1 holds for any $\\\\{\\pi_i\\\\}_{i=1}^n$ as long as Assumption 3 in the paper is satisfied.\n- The balanced sampling constraint (Equation (4)) is enforced by the Cube method after a $\\\\{\\pi_i\\\\}_{i=1}^n$ is given.\n- Therefore, the variance reduction benefits proven in Theorem 1 are not tightly coupled to a specific construction of $\\\\{\\pi_i\\\\}_{i=1}^n$.\n- We will clarify this point in the revision to avoid potential confusion.\n\n## References\n\n- Davezies, L., Hollard, G., & Merino, P. V. (2024). Revisiting randomization with the cube method. *arXiv:2407.13613*.\n- Zrnic, T., & Candès, E. J. (2024, July). Active statistical inference. *ICML*."}, {"review_id": "V64fY03Yod", "reviewer": "Reviewer_oZYC", "summary": "This paper proposes a novel method for maximizing the limited labeling budget, called \"balanced active inference\", where balancing constraints are imposed based on the model uncertainty to guide the labeling process.\nThe goal is to achieve balanced sampling that effectively balances the trade-off between labeling cost and statistical efficiency.", "questions": "Please see and respond to questions in \"Strenghts and Weaknesses\".", "limitations": "The authors provide adequate discussions of the limitations of the current work as well as its potential societal impact in the supplemental material (sections S4 and S5).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The paper tackles an important problem that is of broad interest across diverse fields.\nAs labeling costs tend to be costly in many real-world applications, it is of significant practical and theoretical interest to devise an effective labeling strategy that can minimize the acquisition cost, maximize the efficacy of the acquired (additional) labels, and also ensure statistical efficiency (e.g., keeping the dataset \"balanced\").\n\n2. Overall, the proposed approach is well-motivated and the paper proposes a reasonable solution by leveraging augmented\ninverse propensity weighting (AIPW) estimator and the cube method for balanced sampling.\n\nHowever, the paper would benefit from including a more in-depth discussion on relevant recent advances in active learning, prediction-powered inference, and balanced sampling to provide a clear context to the reader against which its novelty and significance can be judged.\nFor now, the discussion in Sec. 1 \"Related work\" is too short and not comprehensive enough.\nThe main novel and significant contributions made in this paper should be better outlined based on this expanded discussion of relevant work in the field.\n\n\n3. While the paper provides some evidence that supports the potential benefits of the proposed balanced inference scheme, the current evaluation results are insufficient to properly assess its advantages (and any trade-offs made in order to keep the dataset balanced).\n\nEspecially:\n(i) There is no comparison against widely used active learning strategies. It is important to show the pros and cons of \"balanced active inference\" compared to (uncertainty-based) AL strategies, and thereby show the impact of balanced sampling using the proposed balanced active inference scheme.\n(ii) The proposed scheme utilizes an uncertainty model using XGBoost that tries to fit the prediction error. It is important to understand the impact of adopting different \"uncertainty models\". Especially, instead of predicting the error and using it as the uncertainty estimate, what would be the impact of utilizing different UQ (uncertainty quantification) schemes widely utilized in the field?\n(iii) How does the balanced active inference scheme (differently) perform in easy vs. hard regression/classification problems?\n\n4. For the classification problem, the authors utilize logistic regression.\nAs a result, the uncertainty model defined in the \"Protocol\" can be easily adopted without any change.\nHowever, how would one estimate the uncertainty for different types of classifiers?\nIt would be helpful to discuss how the method can generalize.\n\n5. Figure 1 lacks legends for the curves, which makes it difficult to interpret the results.\n\n6. To provide a more comprehensive assessment of the reliability of interval estimation using the proposed technique, the authors should include coverage rates for different confidence levels -- in addition to what is considered in the current study (i.e., 0.9).", "was_revised": true, "final_justification": "The evaluation score has been increased after reviewing the authors' rebuttal to the reviewer's initial feedback.", "rebuttal": "# **Response to Reviewer oZYC**\n\nWe sincerely appreciate your recognition of the problem's significance, the practical relevance of our work, and the soundness of our methodological approach. We also find your constructive feedback and critique extremely valuable for optimizing our manuscript. Below, we provide point-by-point responses to your specific suggestions:\n\n## **Responding to Weaknesses**\n\n> ### **Weakness 3 (i). Active Learning Strategies in Inference**\n\nThanks for your constructive feedback.\n\n- Our proposed \"balanced active inference\" method belongs to the emerging field of active statistical inference , which is fundamentally different from traditional active learning. Consequently, we respectfully argue that a direct comparison to AL strategies would be inappropriate.\n\n- The primary goal of traditional AL is to improve a **model's predictive performance** (e.g., accuracy, AUC) with minimal labeling cost (Deng et al., 2023).\n\n- In contrast, the goal of active inference is to improve the **efficiency** of **statistical inference**, i.e., to obtain narrower confidence intervals and more powerful p-values for population parameters, not to improve the model itself. Our work contributes to the latter.\n\n- AL query strategies typically select samples to maximize the model's information gain to better define its decision boundary.\n\n- Active inference sampling strategies are designed to optimally reduce the **asymptotic variance** of the statistical estimator. As formally derived by Zrnic & Candès (2024), this is a fundamentally different optimization problem tailored for inference, not learning.\n\n- Critically, it has been shown that naively applying AL for statistical estimation introduces significant sampling bias, which invalidates classical statistical procedures (Zhang et al., 2021). \n\n- In summary, comparing our inference-focused method against a learning-focused AL strategy would not be meaningful due to conflicting goals, methodologies, and evaluation metrics. The contribution of our work lies in achieving more efficient and robust statistical inference, and we believe it should be evaluated by those standards.\n\n> ### **Weakness 3 (ii) (iii). Discussion of Uncertainty Quantifications**\n\nThank you for this insightful comment.\n\n- The proposed method belongs to the error-prediction paradigm in active inference. This strategy was established by Yoo & Kweon (2019) and continues to be an active and promising research direction, with recent work such as Sinha & Dongaonkar (2024).\n\n- We specifically chose to predict the **absolute error** (an L1-loss objective) due to its **robustness**. Unlike the L2-loss (mean squared error), the L1-loss makes the resulting uncertainty estimation **less sensitive to outlier** predictions. As noted in Hastie et al. (2009), the robustness of L1-norm-based methods against outliers is a fundamental principle in statistics and machine learning.\n\n- This approach is substantially more efficient than many widely-used UQ schemes. Methods like Deep Ensembles (Lakshminarayanan et al., 2017) require training multiple models, while MC Dropout (Gal & Ghahramani, 2016) necessitates numerous forward passes per sample. Our method requires only a **single** forward pass through a lightweight prediction head, making it highly **scalable and practical**.\n\n> ### **Weakness 4. Classification Problem**\n\nThanks for raising concerns about classification problem. We formally extend the Balanced Active Inference framework to the general M-estimation setting, which cover the classification problem.\n\n#### **(i) Settings of M-estimation**\n\n* Given a family of functions $f(X_i;\\theta)$, the task is to solve $ \\theta^{\\ast}= \\arg\\min_{\\theta} \\mathbb{E} \\left [L\\bigl(X_1, Y_1; \\theta\\bigr)\\right]$.\n* $L(X_1, Y_1; \\theta)$ measures the discrepancy between the true label $Y_1$ and the prediction $f(X_1;\\theta)$.\n* Our goal is to estimate $\\theta^{\\ast}$ using a small labeled set $ \\mathcal{D}_l$ and a larger unlabeled set $ \\mathcal{D}_u $.\n* $ f(X_i;\\hat{\\theta}) $ is an initial estimator trained on $ \\mathcal{D}_l $.\n\n#### **(ii) Uncertainty Measures**\n\n- **Classification.** Let $ p(X_i) = (p_1(X_i),\\ldots,p_K(X_i))$ denote the predicted class probabilities. We define $$u(X_i)=\\frac{K}{K-1}\\Bigl(1-\\max_{j\\in[K]} p_j(X_i)\\Bigr),$$ \nwhich peaks when the model is maximally uncertain (uniform distribution) and drops to zero when the model is highly confident in one class.\n\n#### **(iii) M-Estimation**\n\n* A model $\\hat{u}(\\cdot)$ is trained on $\\mathcal{D}_l$ to predict the uncertainty of points in  $\\mathcal{D}_u $. \n\n* Given a labeling budget $n_b$, adopt the sampling scheme described in Equation (2) of the paper.\n\n* Generate an assignment via the cube method, ensuring the balancing constraint (Equation (4)) is satisfied.\n\n* The proposed estimator is\n\n  $$\\tilde{\\theta}=\\underset{\\theta}{\\arg \\min } \\frac{1}{n} \\sum_{i=1}^n\\{L(X_i, f(X_i ; \\hat{\\theta}) ; \\theta)+[L(X_i, Y_i ; \\theta)-L(X_i, f(X_i ; \\hat{\\theta}) ; \\theta)] \\frac{\\xi_i}{\\\\pi(X_i)}\\},$$\n\n  where $\\pi(X_i)$ is the inclusion probability.\n\n> ### **Weakness 5. Figure 1**\n\nThanks for pointing out this issue. \n- The method `traditional_active` (referring to independent random sampling, also known as Poisson sampling) was inadvertently labeled as `poisson_active` in the original figure.  \n- We will revise the figure accordingly.\n\n> ### **Weakness 6. Coverage Rates for Different Confidence Levels**\n- We have calculated the coverage rate under different confidence levels on multiple datasets. \n- Below are some experimental results with a fixed sampling budget of 0.35:  \n\n|     $\\quad$  $\\quad$ $\\quad$ $\\quad$     Bike             |  Sharing         |     $\\quad$  $\\quad$        Credit Fraud            |  Detection |\n|-------------------------|----------------------|-------------------------|------------------------|\n| Confidence Level        | Coverage Rate        | Confidence Level        | Coverage Rate         |\n| 0.95                    | 0.9447              | 0.95                    | 0.9389                |\n| 0.99                    | 0.9889              | 0.99                    | 0.9836                |\n\n- The results demonstrate that our method is **close** to the preset confidence level in all cases.\n- The results will be included in the Supplement in the revised version.\n\n## References\n\n- Deng, Y., Yuan, Y., Fu, H., & Qu, A. (2023). Query-augmented active metric learning. Journal of the American Statistical Association, 118(543), 1862-1875.\n\n- Zrnic, T., & Candès, E. J. (2024, July). Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning (pp. 62993-63010).\n\n- Farquhar, S., Gal, Y., & Rainforth, T. (2021). On Statistical Bias In Active Learning: How and When to Fix It. In International Conference on Learning Representations.\n\n- Yoo, D., & Kweon, I. S. (2019). Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 93-102).\n\n- Haimovich, D., Karamshuk, D., Linder, F., Tax, N., & Vojnovic, M. (2024). On the convergence of loss and uncertainty-based active learning algorithms. Advances in Neural Information Processing Systems, 37, 122770-122810.\n\n- Hastie, T., Tibshirani, R., Friedman, J. H., & Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction (Vol. 2, pp. 1-758). New York: springer.\n\n- Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.\n\n- Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48 (pp. 1050-1059)."}, {"review_id": "0WdPDwN0vf", "reviewer": "Reviewer_ibeC", "summary": "This paper tackles the problem of active inference — estimating statistical properties of the labels in unlabeled datasets using a machine learning predictor. This is a challenging setting, as the predictor may introduce complex biases that must be carefully accounted for in the estimation process. The authors conduct a thorough theoretical analysis and propose a novel estimator that incorporates balancing constraints to achieve desirable properties, such as variance reduction and bounded asymptotic bias. These contributions are supported by both rigorous theoretical guarantees and empirical validation.", "questions": "Could you explain Remark 1?\nCan you share the ablations that you performed to demonstrate that tau=0.5 is appropriate?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper addresses a problem that I believe is important and relevant to the NeurIPS community. However, as someone who is not an expert in active inference, I found the paper somewhat difficult to follow. While I don’t feel equipped to assess the significance of the proposed method or the theoretical results in detail, I do expect NeurIPS papers to be accessible to an engaged, non-expert reader. In this case, while the mathematics was clear and well-presented it was difficult to distinguish between what aspects of the method were novel versus standard, which made it hard to assess the contributions. Some aspects and discussions were introduced very quickly (e.g. Remark 1 was not clear for me). \n\nThat said, I appreciated the framing of the theoretical results. While I cannot fully assess their depth or novelty, it was clear to me why they were relevant, and the assumptions were well-explained. As a non-expert, I focused more on the empirical evaluation (Section 5). I was concerned by the lack of ablations—for instance, the parameter τ=0.5 is used across all experiments with weak justification. Additionally, the choice of baselines seemed limited: many were discussed in the related work, but only three were used in the experiments. A justification for this selection would strengthen the empirical claims, though I defer to a more field-specific reviewer on this point. Overall, I thought the experimental setup was well-motivated, the repetitions and presentation were sound, and the results were clearly displayed and thoughtfully interpreted.", "was_revised": true, "final_justification": "See my added comment below", "rebuttal": "# **Response to Reviewer ibeC**\n\nWe sincerely thank you for recognizing the novelty of our methodological contribution, the thoroughness of our theoretical analysis, and the compelling empirical validation presented in our work, as well as for your thoughtful and constructive feedback, which is invaluable for refining the manuscript. Below, we address each of your concerns point by point:\n\n## **Responding to Weaknesses**\n\nWe appreciate your concern regarding the accessibility of our paper to non-experts in active inference. We acknowledge that while the mathematics in the paper is presented rigorously, some readers may find it difficult to distinguish the novel aspects of our method from existing techniques.\n\n> **What is novel and what is standard?**\n\n• **Standard components**: Active inference in the sense of Zrnić & Candès (2024) uses independent probability-proportional-to-uncertainty sampling together with an AIPW estimator.\n\n• **Our new contributions**:\n1. We **reinterpret model uncertainty** as a continuous auxiliary variable and impose a population-level balancing constraint on it;\n2. We are, to the best of our knowledge, the first to **integrate the cube method into active inference**.\n3. We provide **closed-form asymptotic variance expressions** and prove that they are **strictly smaller** than those of independent sampling, yielding quantitative efficiency gains.\n\nWe will revise the manuscript in the camera-ready version to make the contributions and methodological innovations more clearly distinguishable from standard components.\n\n## **Responding to Questions**\n\n> ### **Q1. Explanation of Remark 1**\n\nWe appreciate the reviewer’s comment regarding Remark 1. To clarify, Remark 1 states that if $\\\\hat{f}$ is balanced, we have the balance condition:\n$$ \\\\sum_{i=1}^n \\\\frac{\\\\hat{f}\\\\left(X_i\\\\right) \\\\xi_i}{\\\\pi\\\\left(X_i\\\\right)}\\\\approx\\\\sum_{i=1}^n \\\\hat{f}\\\\left(X_i\\\\right).$$\nUnder this condition, the estimator simplifies as follows:\n\\\\begin{equation*}\n    \\\\begin{aligned}\n        \\\\tilde{\\\\theta}&=\\\\frac{1}{n} \\\\sum_{i=1}^n\\\\left[\\\\hat{f}\\\\left(X_i\\\\right)+\\\\left(Y_i-\\\\hat{f}\\\\left(X_i\\\\right)\\\\right) \\\\frac{\\\\xi_i}{\\\\pi\\\\left(X_i\\\\right)}\\\\right]\\\\\\\\\n        &=\\\\frac{1}{n}\\\\sum_{i=1}^nY_i\\\\frac{\\\\xi_i}{\\\\pi(X_i)} + \\\\frac{1}{n}\\\\left[\\\\sum_{i=1}^n \\\\hat{f}\\\\left(X_i\\\\right) - \\\\sum_{i=1}^n \\\\frac{\\\\hat{f}\\\\left(X_i\\\\right) \\\\xi_i}{\\\\pi\\\\left(X_i\\\\right)}\\\\right],\n    \\\\end{aligned}\n\\\\end{equation*}\nwhere the second term vanishes due to the balance condition. This shows that the estimator reduces to the standard inverse propensity weighted (IPW) estimator when $\\\\hat{f}$ is sufficiently balanced.\n\n> ### **Q2. Sensitivity Analysis on $\\tau$**\n\nThank you for this important suggestion. We acknowledge that the choice of $\\tau = 0.5$ warrants explicit justification.\n\n- **Sensitivity analysis** on $\\tau$ across multiple datasets will be added in the revised version. \n- The setup is: fix the sampling budget at 0.2 and compute the RMSE and interval width (analogous to Figure 1 in the main text, but with $\\tau$ ranging from 0 to 1 on the x-axis) for methods.\n- Below is a representative subset of our experimental results:\n\n**Table Q2.1 Bike Sharing, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ------- | --------- |\n| 0.00    | $\\quad$ **0.127**   | $\\quad$1.460          | 1.466   | 3.846     |\n| 0.25    | $\\quad$**0.093**   | $\\quad$1.261          | 1.466   | 3.846     |\n| 0.50    | $\\quad$**0.079**   | $\\quad$1.146          | 1.466   | 3.846     |\n| 0.75    | $\\quad$**0.072**   | $\\quad$1.064          | 1.466   | 3.846     |\n| 1.00    | $\\quad$**0.130**   | $\\quad$1.038          | 1.466   | 3.846     |\n\n**Table Q2.2 Bike Sharing,  Interval Width**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ------- | --------- |\n| 0.00    | $\\quad$ **0.395**   | $\\quad$ 4.847          | 4.844   | 12.805    |\n| 0.25    | $\\quad$ **0.302**   | $\\quad$ 4.128          | 4.844   | 12.805    |\n| 0.50    | $\\quad$ **0.254**   | $\\quad$ 3.754          | 4.844   | 12.805    |\n| 0.75    | $\\quad$ **0.230**   | $\\quad$ 3.530          | 4.844   | 12.805    |\n| 1.00    | $\\quad$ **0.275**   | $\\quad$ 3.432          | 4.844   | 12.805    |\n\n**Table Q2.3 Friedman, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ------- | --------- |\n| 0.00    | $\\quad$ **0.029**   | $\\quad$ 0.069          | 0.068   | 0.137     |\n| 0.25    | $\\quad$ **0.029**   | $\\quad$ 0.063          | 0.068   | 0.137     |\n| 0.50    | $\\quad$ **0.031**   | $\\quad$ 0.060          | 0.068   | 0.137     |\n| 0.75    | $\\quad$ **0.034**   | $\\quad$ 0.060          | 0.068   | 0.137     |\n| 1.00    | $\\quad$ **0.063**   | $\\quad$ 0.079          | 0.068   | 0.137     |\n\n**Table Q2.4 Post-election survey research, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ----------- | --------- |\n| 0.00    | $\\quad$ **0.00841** | $\\quad$ 0.00865        | 0.00862     | 0.01165   |\n| 0.25    | $\\quad$ **0.00806** | $\\quad$ 0.00840        | 0.00862     | 0.01165   |\n| 0.50    | $\\quad$ **0.00790** | $\\quad$ 0.00824        | 0.00862     | 0.01165   |\n| 0.75    | $\\quad$ **0.00799** | $\\quad$ 0.00839        | 0.00862     | 0.01165   |\n| 1.00    | $\\quad$ 0.01014     | $\\quad$ 0.01047        | **0.00862** | 0.01165   |\n\n**Table Q2.5 Credit Fraud Detection, RMSE**\n\n| $\\\\tau$ | `cube_active` | `poisson_active` | `uniform` | `lassical` |\n| ------- | ----------- | -------------- | ----------- | --------- |\n|    0.00 |   $\\quad$ **0.000113** |   $\\quad$    0.000142 | 0.000144 |  0.000219 |\n|    0.25 |   $\\quad$ **0.000101** |   $\\quad$    0.000132 | 0.000144 |  0.000219 |\n|    0.50 |  $\\quad$  **0.000099** |   $\\quad$    0.000129 | 0.000144 |  0.000219 |\n|    0.75 |   $\\quad$ **0.000095** |   $\\quad$    0.000128 | 0.000144 |  0.000219 |\n|    1.00 |  $\\quad$  **0.000096** |   $\\quad$    0.000128 | 0.000144 |  0.000219 |\n\n- All coverage rates in the above results closely approximate the target confidence level, confirming the validity of our findings.\n- Across all datasets, the RMSE and interval width **generally maintained the following ordering**: `cube_active < poisson_active < uniform < classical`.\n- At $\\tau = 0$: The performance of `poisson_active` becomes nearly equivalent to `uniform`, as both rely solely on simple random sampling.\n- As $\\tau$ increases, both `cube_active` and `poisson_active` exhibit **precision improvement** (decreasing RMSE and interval width), **reaching lower value** around $\\tau = 0.5$ for different datasets. This demonstrably quantifies the precision gains enabled by active sampling.\n- When near $\\tau = 1$, RMSE and interval width increase on some datasets. This occurs because the model for estimating prediction uncertainty ($\\hat{u}$) is inherently imperfect. When the model erroneously assigns near-zero uncertainty ($\\hat{u} \\approx 0$) to specific data points, it can significantly inflate the estimator variance. These findings are well-aligned with conclusions established in Zrnic & Candès (2024).\n\n## References\n- Zrnic, T., & Candès, E. J. (2024, July). Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning (pp. 62993-63010)."}]

Author rebuttal:

Meta-review: This article looks at a less studied problem, namely active inference, and proposes to improve upon the standard approach of independent sampling by using the cube method to impose a balancing constraint on queried examples. The proposed algorithm is shown to have a smaller asymptotic variance than independent sampling under the assumption of a nearly accurate estimator of uncertainty. The robustness of the variance reduction benefit to this assumption is demonstrated through experiments on synthetic and real data.

The novelty and the interest of the proposed method were appreciated by the reviewers. It is simple, conceptually elegant, and potentially impactful on the field of active inference. Questions on computational complexity, choice of hyperparameter, generalization to M-estimator framework, and comparison to more baselines were raised and addressed during rebuttal, with the support of supplementary empirical results. One remaining concern is about the presentation. The problematics of active inference should be more carefully introduced, to allow an easier grasp of the motivation and the intuition behind the proposed balanced sampling approach to non-expert readers. A discussion on the relation and the difference with active learning would also help better contextualize the contribution of this work.
Decision: Accept (poster)