Title: Active Measurement: Efficient Estimation at Scale
Venue: NeurIPS 2025
Accepted: yes
Keywords: importance sampling

image detection

ai for science

counting

human-in-the-loop

unbiased estimators
Abstract: AI has the potential to transform scientific discovery by analyzing vast datasets with little human effort. However, current workflows often do not provide the accuracy or statistical guarantees that are needed. We introduce \emph{active measurement}, a human-in-the-loop AI framework for scientific measurement. An AI model is used to predict measurements for individual units, which are then sampled for human labeling using importance sampling. With each new set of human labels, the AI model is improved and an unbiased Monte Carlo estimate of the total measurement is refined. Active measurement can provide precise estimates even with an imperfect AI model, and requires little human effort when the AI model is very accurate. We derive novel estimators, weighting schemes, and confidence intervals, and show that active measurement reduces estimation error compared to alternatives in several measurement tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an \"active measurement\" strategy, which aims to provide an iterative framework for improving a measurement task through improving the sampling strategy and efficiently involving humans in the loop.\n\nSTRENGTHS AND WEAKNESSES:\n1. This paper proposes an interesting scheme referred to as \"active measurement\", building on or borrowing inspirations from areas such as active testing, adaptive importance sampling (AIS), and prediction-powered inference.\nThe discussion the authors provide in Sec. 3 \"Related Work\" is interesting, but too brief to provide an adequate and useful context to the readers regarding the proposed active measurement scheme.\nThere should be a more comprehensive (and more in-depth) discussion to motivate and position the current study more clearly.\n\n2. A major concern regarding the current study is that the evaluations are very limited.\nCurrently, the authors consider only two counting tasks: (i) counting birds in two high-resolution images; and (ii) counting roosting birds in weather radar images.\nThere are many scientific datasets that the authors could utilize to strengthen their performance assessment.\nFor example, Cryo-EM particle picking and object counting from satellite images would provide good test cases.\n\n\n3. The study would benefit from a comparison with active learning based strategies, especially the ones that are uncertainty-based.\nConceptually, there are many similarities, and it would be beneficial to discuss their similarities and differences, and what specific advantages the active measurement strategy might offer.\nFurthermore, performance comparison with widely used active learning schemes in Figures 3-6 would be helpful.\n\n\n4. In the evaluation result (e.g., Fig. 3), it would be helpful to show the performance in terms of the # of labeled units, and not only the fraction of the labeled units.\n\n\n5. DISCount is an interesting and recent baseline, but to show the general applicability of the proposed active measurement scheme, the authors are strongly encouraged to consider additional baselines.\nIt would be important to see whether one can gain similar consistent improvements across different schemes.\n\nQUESTIONS:\nPlease see \"Strengths and Weaknesses.\"\n\nLIMITATIONS:\nYes, the authors briefly discuss the limitations of the current work in the Conclusion.\nHowever, the paper does not seem to explicitly discuss the potential negative societal impact in the same section, unlike how they responded to the questionnaire section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper address the problem of how to rapidly train a model with human input in the loop. The paper focuses on training a detector for the purposes of measuring a natural phenomenon, although it seems reasonable that the same technique could be used for other purposes. The approach builds on prior art where data are subsampled for labelling, but extends the prior art by focusing on sampling without replacement, compares different weighting methods in the context of sampling without replacement, and derives a new estimate of the variance. The performance is evaluated on two types of bird detectors.\n\nSTRENGTHS AND WEAKNESSES:\nOverall I quite liked this paper. The problem is clear, the writing is clear and the contribution is clear.\n\nI have a few minor technical concerns, but I do not view these as serious problems. Firstly, the acquisition function assumes the estimates across the data set $f(s_t)$ for $s_t \\in \\Omega \\backslash D_t$ are independent. I note that the authors pay quite a lot of attention to the loss of independence between timesteps, but there is also correlation within the sample set $s_t$, e.g., correlation between different tiles in the image. In the active sensing domain (which is different to this domain) it is common practice to learn a Gaussian process that describes how sample sets within the time step might be correlated.\n\nSecondly, the variance estimates and the acquisition function as written do not incorporate an estimate of the variance from $f(s_t)$ directly. $f(s_t)$ provides only the estimated number of measurements, but it is relatively straightforward to modify $f(s_t)$ to return a variance as well, especially since line 8 of Algorithm 1 requires retraining the AI model. Incorporating the variance estimate from the AI model providing f(s_t) would allow, for instance, the model to avoid asking for $s_t$ estimates (line 7 of Algorithm 2) in regions where the signal is fundamentally ambiguous (e.g., a high noise part of the image).\n\nFinally, providing a variance estimate from the AI model and a GP model of correlations across the domain $\\Omega$ would allow the overall model to predict where measurements are most likely to improve. Asking for labels in a region that is high variance and strongly correlated with a labelled region that is also high variance would most likely be unhelpful. In the active learning setting (which is also related but different to this domain), it is most helpful to ask for labels where the learner performance is predicted to improve.\n\nThe experimental results are good, but are fairly specific. It would have been helpful to perform a similar analysis on a different type of data. I appreciated the error bars on the right hand panel of Figure 1, but it was not clear if these were ground truth error bars averaged over $\\Omega \\backslash D_t$ or if these were the predicted variances. I would also have liked to have seen these more carefully analysed in the experimental results section.\n\nThe limitations section was appropriate.\n\nI liked this paper and would argue for acceptance.\n\nQUESTIONS:\n- Are the error bars on the right hand panel of Figure 1 ground truth error bars or predicted variances. How do they compare? Can you provide them for both domains?\n\n- Is there a reason not to incorporate a variance estimate from the AI model itself?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on efficiently measuring objects at scale. This problem is especially relevant for analyzing datasets in scientific domains. The paper introduces active measurement, a framework that uses a human-in-the-loop pipeline and machine learning models to assist with measurements. Collecting labels for a large number of instances is costly and time-consuming, making it difficult to obtain accurate measurements of items. Inspired by active testing and active learning, this framework iteratively gathers labels from humans and refines the estimates of the items. The work proposes new unbiased estimators, weighting schemes, and confidence intervals for improved estimates. The experiments show that they significantly reduce the estimation errors compared to the baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n- The paper is well written. The paper does a great job of motivating the problem of estimating counts at scale.\n- The paper effectively addresses large-scale counting issues by developing new estimators, weighting methods, and confidence intervals for human-in-the-loop scenarios. \n- The paper makes strong theoretical contributions and provides comprehensive proofs for several propositions in the paper. \n- The experiment setup is also reasonable and fair. The experiments show that the proposed method consistently shows lower fractional errors compared to the baselines.\n\n### Weaknesses\n\n**Experiments**\nThe paper primarily focuses on scientific datasets related to bird counting. While they show strong results on these datasets, the paper would be strengthened if it included results on popular crowd counting datasets [a, b]. \n\n**LURE** \nThe paper improves over LURE weights for a pool-based setting with sampling without replacement. I would like to see if the improvements from Proposition 3 improve active learning frameworks as well. \n\n**Nit**\nLine 23-34: The authors have described the problem of measurement they focus on in detail. Could they cite a reference to prior work that also identifies the same problem? \n\n**References**\n\n[a] Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds. ECCV 2018. \n\n[b] Multi-Source Multi-Scale Counting in Extremely Dense Crowd Images. CVPR 2013.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}

comments: SUMMARY:
This paper explores methods for active measurement (human in the loop) to improve precisions of measurements and provide unbiased MC error estimates in imperfect AI model.  The task is tested for counting birds in 2 different datasets and improved performance is demonstrated.

STRENGTHS AND WEAKNESSES:
The paper is clear and presents the approach for the biological domain.  It describes novel contributions beyond state of the art (DIS).  Active measurement is an under-explored area in the sciences and this study systematically studies augmentations to prior work.  

It would be nice to understand when and why WOR vs AIS gives larger and smaller improvements in the dataset.  The relative contributions vary depending on the dataset.  It would also be nice to demonstrated this on another domain (e.g. cell counting?) instead of 2 similar domain applications.

QUESTIONS:
It would be nice to understand when and why WOR vs AIS gives larger and smaller improvements in the dataset.  The relative contributions vary depending on the dataset.  It would also be nice to demonstrated this on another domain (e.g. cell counting?) instead of 2 similar domain applications.

LIMITATIONS:
Demonstrations on other domains to understand how the approach may vary would make the result more significant and applicable.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06