Title: Balanced Active Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: asymptotic properties

cube method

label selection

statistical efficiency
Abstract: Limited labeling budget severely impedes data-driven research, such as medical analysis, remote sensing and population census, and active inference is a solution to this problem. Prior works utilizing independent sampling have achieved improvements over uniform sampling, but its insufficient usage of available information undermines its statistical efficiency. In this paper, we propose balanced active inference, a novel algorithm that incorporates balanced constraints based on model uncertainty utilizing the cube method for label selection. Under regularity conditions, we establish its asymptotic properties and also prove that the statistical efficiency of the proposed algorithm is higher than its alternatives. Various numerical experiments, including regression and classification in both synthetic setups and real data analysis, demonstrate that the proposed algorithm outperforms its alternatives while guaranteeing nominal coverage.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of active inference — estimating statistical properties of the labels in unlabeled datasets using a machine learning predictor. This is a challenging setting, as the predictor may introduce complex biases that must be carefully accounted for in the estimation process. The authors conduct a thorough theoretical analysis and propose a novel estimator that incorporates balancing constraints to achieve desirable properties, such as variance reduction and bounded asymptotic bias. These contributions are supported by both rigorous theoretical guarantees and empirical validation.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper addresses a problem that I believe is important and relevant to the NeurIPS community. However, as someone who is not an expert in active inference, I found the paper somewhat difficult to follow. While I don’t feel equipped to assess the significance of the proposed method or the theoretical results in detail, I do expect NeurIPS papers to be accessible to an engaged, non-expert reader. In this case, while the mathematics was clear and well-presented it was difficult to distinguish between what aspects of the method were novel versus standard, which made it hard to assess the contributions. Some aspects and discussions were introduced very quickly (e.g. Remark 1 was not clear for me). \n\nThat said, I appreciated the framing of the theoretical results. While I cannot fully assess their depth or novelty, it was clear to me why they were relevant, and the assumptions were well-explained. As a non-expert, I focused more on the empirical evaluation (Section 5). I was concerned by the lack of ablations—for instance, the parameter τ=0.5 is used across all experiments with weak justification. Additionally, the choice of baselines seemed limited: many were discussed in the related work, but only three were used in the experiments. A justification for this selection would strengthen the empirical claims, though I defer to a more field-specific reviewer on this point. Overall, I thought the experimental setup was well-motivated, the repetitions and presentation were sound, and the results were clearly displayed and thoughtfully interpreted.\n\nQUESTIONS:\nCould you explain Remark 1?\nCan you share the ablations that you performed to demonstrate that tau=0.5 is appropriate?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for maximizing the limited labeling budget, called \"balanced active inference\", where balancing constraints are imposed based on the model uncertainty to guide the labeling process.\nThe goal is to achieve balanced sampling that effectively balances the trade-off between labeling cost and statistical efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n1. The paper tackles an important problem that is of broad interest across diverse fields.\nAs labeling costs tend to be costly in many real-world applications, it is of significant practical and theoretical interest to devise an effective labeling strategy that can minimize the acquisition cost, maximize the efficacy of the acquired (additional) labels, and also ensure statistical efficiency (e.g., keeping the dataset \"balanced\").\n\n2. Overall, the proposed approach is well-motivated and the paper proposes a reasonable solution by leveraging augmented\ninverse propensity weighting (AIPW) estimator and the cube method for balanced sampling.\n\nHowever, the paper would benefit from including a more in-depth discussion on relevant recent advances in active learning, prediction-powered inference, and balanced sampling to provide a clear context to the reader against which its novelty and significance can be judged.\nFor now, the discussion in Sec. 1 \"Related work\" is too short and not comprehensive enough.\nThe main novel and significant contributions made in this paper should be better outlined based on this expanded discussion of relevant work in the field.\n\n\n3. While the paper provides some evidence that supports the potential benefits of the proposed balanced inference scheme, the current evaluation results are insufficient to properly assess its advantages (and any trade-offs made in order to keep the dataset balanced).\n\nEspecially:\n(i) There is no comparison against widely used active learning strategies. It is important to show the pros and cons of \"balanced active inference\" compared to (uncertainty-based) AL strategies, and thereby show the impact of balanced sampling using the proposed balanced active inference scheme.\n(ii) The proposed scheme utilizes an uncertainty model using XGBoost that tries to fit the prediction error. It is important to understand the impact of adopting different \"uncertainty models\". Especially, instead of predicting the error and using it as the uncertainty estimate, what would be the impact of utilizing different UQ (uncertainty quantification) schemes widely utilized in the field?\n(iii) How does the balanced active inference scheme (differently) perform in easy vs. hard regression/classification problems?\n\n4. For the classification problem, the authors utilize logistic regression.\nAs a result, the uncertainty model defined in the \"Protocol\" can be easily adopted without any change.\nHowever, how would one estimate the uncertainty for different types of classifiers?\nIt would be helpful to discuss how the method can generalize.\n\n5. Figure 1 lacks legends for the curves, which makes it difficult to interpret the results.\n\n6. To provide a more comprehensive assessment of the reliability of interval estimation using the proposed technique, the authors should include coverage rates for different confidence levels -- in addition to what is considered in the current study (i.e., 0.9).\n\nQUESTIONS:\nPlease see and respond to questions in \"Strenghts and Weaknesses\".\n\nLIMITATIONS:\nThe authors provide adequate discussions of the limitations of the current work as well as its potential societal impact in the supplemental material (sections S4 and S5).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces balanced active inference, a method for estimating the label mean when only a small portion of a large unlabelled data pool can be labeld. The proposed method first learns uncertainty scores and then draws balanced samples with the cube method. After the selected points are labelled, they are used to produce the estimated mean. This paper proves asymptotic normality and shows that the proposed estimator’s variance is lower than that of the baselines. Experiments on synthetic datasets demonstrate better mean estimates comparing to previous methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe problem formulation is clearly formulated, the method and results are cleanly presented. The proposed method can be easily implemented, and the experiments support the theoretical claim. In general, the proposed method is conceptually elegant and has the potential to be practically impactful.\n\n**Weaknesses**\n\n* The scope the paper is very narrow, as it only focuses on one-dimensional mean estimations. Although the presentation is clean within the scope, the narrow scope reduces the significance for general ML audience. For example, can the proposed method, both theory and algorithm, be applied to classification tasks? \n* The only theoretical result, Theorem 1, is essentially an application of standard tools, e.g., Chebyshev’s inequality and central limit theorem, on top of existing work (Lemma 1 to 3).  A novel proof technique, or results beyond asymptotic normality, would strengthen the paper given the narrow scope. \n* Related to the above point, the Assumption 5 is very strong. It basically asks for an accurate uncertainty estimator. Is obtaining an accurate uncertainty estimator easier than a similarly accurate label estimator itself? \n* It is mentioned in the paper that the mixing hyperparameter $\\\\tau$ is chosen to be 0.5. It would be interesting to see how this hyperparameter influence the performance both theoretically and empirically. \n* Some of the claims in the paper are not well supported. For example, it is said \"safeguard against distributional shifts,\" but there seems to be no evidence to support this.\n\nQUESTIONS:\nHow does Theorem 1 reflect on the choice of $\\\\pi$ other than simply $\\\\mathbb{E}[\\\\pi_1]$? For example, does Theorem 1 hold for any form of the probability $\\\\pi$ satisfying the balanced sampling constraint? Clarifying this would help the readers to understand how tightly the theoretical benefit is coupled to the particular $\\\\pi$ defined in the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the statistical inefficiency caused by traditional independent sampling methods. While existing active inference approaches improve upon uniform sampling by prioritizing instances with high model uncertainty, their reliance on independent selection can lead to imbalanced labeled sets and inflated variance.\n\nTo solve this, the authors propose Balanced Active Inference, an algorithm that integrates balanced sampling principles into the active learning process. The core idea is to treat the model's uncertainty estimates as auxiliary variables and enforce a balancing constraint during label selection.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n- The idea of using model uncertainty as the balancing variable within the cube method framework is interesting. It directly addresses a well-known weakness of uncertainty-based sampling (variance inflation) with a principled statistical solution.\n\n- The paper is exceptionally well-written. The authors do a good job of motivating the problem, explaining the limitations of prior work, and introducing their proposed solution.\n\n- The experimental validation is thorough and convincing. The authors test their method on a wide array of synthetic and real-world datasets, covering both regression and classification tasks. The use of multiple metrics (RMSE, confidence interval width, and coverage rate) provides a complete picture of the method's performance.\n\n\n*Weaknesses*\n\n- The paper leverages the cube method, which may be unfamiliar to parts of the core machine learning audience. While it is a standard technique in another field, a slightly more intuitive explanation of its mechanics in the main text could improve accessibility. Furthermore, the computational complexity of the cube method itself, especially for very large unlabeled datasets, is not discussed, which could be a practical concern for scalability.\n\nQUESTIONS:\nI have a few concerns on the tractability of this method. Specifically,\n\n- Could you provide some insight into the computational complexity of the cube method as used in your algorithm? How does its runtime scale with the size of the unlabeled pool n? Are there potential scalability challenges when applying this method to massive datasets with millions of unlabeled instances?\n\n- A simpler approach to achieve some form of balance would be to stratify the data based on quantiles of the uncertainty and then sample proportionally from each stratum. How would you compare your method, which achieves exact continuous balancing, to such a stratified sampling approach, both theoretically and empirically?\n\nLIMITATIONS:\nI have mentioned the limitations I foresee in questions and weaknesses sections above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-17"}