Active Measurement: Efficient Estimation at Scale
Authors: Max Hamilton
Jinlin Lai
Wenlong Zhao
Subhransu Maji
Daniel Sheldon
Venue: NeurIPS 2025 poster ()
Keywords: importance sampling
image detection
ai for science
counting
human-in-the-loop
unbiased estimators
Primary area: general_machine_learning
TL;DR: We present active measurement, a human-in-the-loop AI framework that combines AI predictions with importance sampling, model adaptation, and human labeling to make accurate scientific measurements.

Abstract: AI has the potential to transform scientific discovery by analyzing vast datasets with little human effort. However, current workflows often do not provide the accuracy or statistical guarantees that are needed. We introduce \emph{active measurement}, a human-in-the-loop AI framework for scientific measurement. An AI model is used to predict measurements for individual units, which are then sampled for human labeling using importance sampling. With each new set of human labels, the AI model is improved and an unbiased Monte Carlo estimate of the total measurement is refined. Active measurement can provide precise estimates even with an imperfect AI model, and requires little human effort when the AI model is very accurate. We derive novel estimators, weighting schemes, and confidence intervals, and show that active measurement reduces estimation error compared to alternatives in several measurement tasks.

Active Measurement: Efficient Estimation at Scale
Max Hamilton' Jinlin Lai"
Wenlong Zhao Subhransu Maji Daniel Sheldon
Manning College of Information & Computer Sciences University of Massachusetts, Amherst
jantamiitooonnnnnnnnl wenlongzhan , sheldon)@es umass edu
Abstract
Al has the potential transform scientific discovery by analyzing datasets with little human effort. However, current workflows often do not provide the accuracy statistical guarantees that needed. We introduce active measurement, human-in-the-loop AI framework for scientific measurement. An Al model is used predict measurements for individual units, which then sampled for human labeling using importance sampling. With each new set of human labels, the AI model improved and an unbiased Monte Carlo estimate of the total measurement is refined. Active measurement can provide precise estimates even with an imperfect Al model, and requires little human effort when the AI model is very accurate. We derive novel estimators, weighting schemes, and confidence intervals, and show that active measurement reduces estimation error compared alternatives in several measurement tasks.
Introduction
Al offers a transformative approach to scientific discovery, empowering scientists to analyze vast datasets in ways that traditional methods cannot achieve [22 38 4T Applications include species identification from images and audio for biodiversity monitoring
19 39], disease diagnosis in medical imaging [2] 37], classifying galaxies in astronomy [23), assessing crop health agri- culture [13 [14. and myriad other applications in fields such as remote sensing, microscopy, and neuroscience. In these applications, the typical goal to make measurements to answer science or policy questions, which therefore must be precise. But Al models are far from perfect: they may introduce bias have unacceptable error rates, and do not offer the statistical guarantees that scientists desire.
As concrete example, consider counting birds in the high-resolution photograph in Fig. of large flock of Tree Swallows Old Lyme, CT in September, 2018. Scientists would like to estimate the population size and track it over time from photographs like this [e.g.. 4 7 [8]. Computer vision is an excellent tool for this task, but current workflows are poorly suited to scientific measurement. scientist might be able to estimate the count with relatively low effort using pre-trained object detector adapted with a few labeled examples. However, substantial additional effort is needed to obtain a model that accurate enough for scientific measurement and gain confidence in its performance. A typical scenario would be to first tune and validate the detector's performance using labeled image tiles. To obtain bounds on count performance, more validation needed. After substantial effort, the scientist may find that the count is 20,358 + 8000 birds, which is not accurate enough to track populations over time. What can be done? Under current practice, the scientist returns to model development. For challenging tasks, may be impossible to achieve precise enough measurements.
Denotes authors with equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with medel: urknown accuracy
2000
1750
Hisheresoltionn image
Sample tiles txing predicted counts
Collect Human tounts
Human- in the loop counting
recluno carce and provida guanntus
Figure I: Active measurement. An Al model predicts counts for each image tile, which are then used to form proposal distribution to selectively sample tiles for human labeling. Ground truth counts are used both to improve the Al model and produce a Monte Carlo estimate of the final measurement.
We introduce new paradigm of active measurement where the scientist uses Al to interactively solve the measurement task (Fig. D. Active measurement is based on Monte Carlo estimation. In each step, the scientist receives an unbiased estimate of the true count and an estimate of error and is asked to provide new labels. These labels are used both to reduce Monte Carlo error of the estimated measurement and to improve the Al model for future rounds of estimation. The process continues until the error is sufficiently low - in this example the estimates are 11,977 = 1,076 after labeling 50 tiles in the image, closely aligning with the true count of 12,486 birds. Unlike existing workflows, arbitrarily low eπrоr is possible with enough labels even with an imperfect model. On the other hand, the model approaches perfect performance, very accurate estimates are possible with a small amount of labeled data.
Active measurement builds on adaptive importance sampling (28) active testing 20 26], and prior work on covariate and brased counting (25 32 but extends each of these works in different directions. We introduce the active measurement framework, derive new estimators, and show that they are unbiased and consistent. We contribute novel approaches for confidence interval construction that are tailored to the active measurement setting, and derive novel weighting schemes to combine estimates obtained in each step to minimize the overall estimation variance. We show empirically that our techniques provide accurate confidence intervals and reduce estimation error compared to prior methods several scientific measurement tasks.
Code for this paper is available at: lnttps://github. cvl http:/cllmmmsssatiiiaa http::///ghaaaaooooooccuuuuu/utr
2 Active measurement
We consider the following scientific measurement task. There is a set Q of N = = individual units (c.g., image tiles), with unknown ground-truth measurement f(s) 2 0 for each s Ω. For S Ω, define F(S)= = E.es fs). We seek to estimate the total measurement F(22) across all units.
Active measurement combines human labeling with AI predictions. Suppose that, time t, the scientist has labeled units D, C S2 and therefore knows the ground-truth value F(D,)= = E.ED, f().. Further, suppose an Al model is available (e.g., trained on D.) to predict g(s) s f(8) for other units. Let qe be the probability distribution proportional to g on Ω2 D, The base estimator for active measurement is:
f(s))
=
8, Gr-
(s.)))
The second term is an importance sampling estimate satisfying E, (f((s) = F(2) D.), from which it follows that E[ F(2). This estimator is inspired by active testing (20). which discuss more below. It is unbiased for any proposal distribution qt. but has minimum variance (of zero) when qe f. which motivates the choice qi g f.
The active measurement framework, shown in Alg. uses this estimator each step a human-in- thop process. The set D, contains initially labeled units, which may be empty. At the start of step I, units DD have been labeled, and an acquisition distribution gi used to sample a new unit for labeling. Our typical acquisition strategy is to train an AI model on D, to generate predictions

---

Algorithm Active measurement
Algorithm 2 Variance estimation
Require: Initially labeled units Di S Q, acquisition distribution 91, weight sequences or,, 8
Require: All variables from Alg.
1: for 1,2,.... do
I: for 1,2,... ..., T do
2: Get mean estimate G.,, F(a-1-F(D,)
2: Sample 8, ~ q: (-) and obtain f(s))
3: for T,..., do Form single variance estimator
3:
Form IS estimate
f(8)) A F(D.)+ ((()))
[(s)
Var
E (r(()
- Gr.
(+(8)
SED 12
4:
Combine estimates as Fi. Faa==t== Cie EG
g-(0r)
f(3r)
5: Get variance estimates Var, 1+=1 using Alg.
Gr. (3)
4r (5r) (-()))
6: Combine variances as Var = Ei=1 G, Var,
7: Update D1+1 = D, (s+
5: end for
8: Update acquisition distribution 4-+1 over
Var+
Σ. B, Var
Ω2 Do+1. e.g.. by updating an Al model using D end for
9: end
g(s) for all units s 22 DD and then choose a: g- In Line 3, the newly labeled unit is used form
an importance sampling estimate with E F.] F(2) using Eq. (T). Line 4 combines the estimators from each step into the estimator Fi with normalized weights that satisfy Σt= a, = 1, which incorporates all of the labeled samples and represents the model's overall estimate at time t. Hereafter, for any weighting scheme, we use a, to denote a sequence of unnormalized weights for S r S N and ä, for their normalized counterparts ä, = a,/ Ei= Oty, with t implicit from context.
Proposition 1. The combined estimator Fix- --== a. Fr is unbiased: E[f..] = F(2).
At step I, we also generate variance estimate from Alg. which is derived in $ 5 Alg. takes (t2)) time naively, but can be improved to (t)) using the streaming algorithm in $ B.2 so that step t of active measurement takes (tt) time overall. In practical settings expect the time to be dominated by labeling and updating an Al model. We next discuss related work before returning to weighting schemes and variance estimation.
3 Related work
Active measurement is closely related to three existing lines of work. Active testing 12 interactively estimates the test loss of an Al model by sampling unlabeled points according to predicted losses from surrogate model and then using the newly acquired labels to form an importance-sampling based estimator the loss and update the surrogate model. The sampling strategy and form of the estimator in Eq. (I) is identical to that used by active testing, with the significant difference that the estimand is different: active measurement seeks to directly estimate a scientific measurement while iteratively refining the AI model itself, while active testing estimates test loss and refines a surrogate model. Compared to active testing, we also contribute novel weighting schemes ($ 4) and show that these reduce error, and novel approaches for variance estimation and confidence interval construction ($5), which was not considered in active testing.
20
Active testing and active measurement are both versions of adaptive importance sampling (AIS) 6 27 28 AIS improves an pooortnncessamliing estimator (x))/q()) )) by iteratively refining the proposal g and forming weighted combination of the estimators made with different proposal distributions. Active measurement modifies AIS to sample without replacement from a finite sample space. which leads to the estimator in Eq. (T. compared to the usual iuportance-samplin
estimator supported on all of Ω. Sampling without replacement leads to novel considerations in the selection of weights ($ 4) and in variance estimation ($ (). The DISCount (detector-baal based importance sampling) method of Perez et al. 32) which builds on (25), estimates the total counts in scientific collections using AI model predictions for the importance sampling proposal distribution. Active measurement shares the same goal, and advances on DISCount by interactively refining the Al model with acquired labels and by sampling without replacement, which both can significantly reduce estimation error.
The concept of combining an model with limited human effort related to semi-supervised learning [8], where the model is trained with using combination of labeled and unlabeled data.
3

---

Most recently, prediction powered inference (PPI) [3 was proposed to give valid statistical inference using a small amount of labeled data. Unlike PPI, which assumes data iid and seeks estimate population parameter, active measurement samples data interactively and non-uniformly aided by Al and seeks to estimate scientific measurement on finite dataset. We show in the experiments that active measurement outperforms a PPI-motivated baseline. In similar spirit, active machine learning has been used to construct large datasets with trowd-sourcing [21] 30 [4]. We have a different objective of achieving accurate estimation with small amount of labeled data.
Weighting schemes
What is an appropriate weight sequence a, to combine the estimators in active measurement? An important observation that the estimators in each step, though not independent, are uncorrelated:
Proposition 2. For any 1 S <T<rS t, Cov (F) F.)= =
Thus, the combined variance is Var[F = = of Var [ar[ and the weighting that achieves minimum variance uses inverse-variance weighting, i.e., ar = 1/ Var[ř+]. However, in practice we don't know [F]] and is difficult to estimate (see $ $ so we first consider fixed weighting approaches that approximate this principle under assumptions about how Var[F,] changes due to: (1) adapting the model, and (2) the shrinking sample space.
Square root law. In AIS, a simple and near-optimal alternative is the square root law SQRT = VF (29]. To derive this law, it is assumed that variance reduces at the rate Var[ x T due to adaption of the proposal distribution, but the rate y E [0. 1] is unknown. The square root law works as a conservative strategy that is within a factor of 9/8 8 of the optimal variance even without knowing the rate y: if the variance with optimal weighting strategy for rate at step t is oar (v, and F ASQRT is the estimator using weights of &QRT (we will use similar notation for other weighting schemes
V 3. However, in our context, the model Var[Er] x T ~M below), then sup121 supge[u.1] (ar
does not account for shrinking variance due to sampling without replacement, which is not optimal.
LURE weights In active testing, a weighting scheme known as LURE was introduced for the pool-based setting with sampling without replacement [12]. If the number of units is |[]] N, the LURE weights are at LURE = (W-r)(N-r+1)) The original motivation was to treat samples equally in the combined estimate, but we can reinterpret these weights as accounting for variance reduction due to the shrinking sample space:
Proposition 3. If there exist constants 0 < A S. B such that for any E Q, A f()) < B and A < ((8) < B, then there exists constant C > 0 such that Var[ř,] < -( - 1).
For brevity, define w, = (N-+)(N-++1)) This proposition indicates that the variance is order 1 1/w, when only considering the sample space reduction, so the LURE weights of w, are well justified by inverse variance weighting. However, in our setting, LURE weights neglect variance reduction due to adaptation of the Al model, which also not optimal.
Combined weights. We propose to combine the above two weight- ing schemes and use of COMB wryF. Fig. 2 demonstrates the growth of different weighting schemes. When r is small, the detector has not been well-tuned so weighting schemes that discourage early estimators, like the square root law, are preferred. As becomes closer N. the sample space reduction becomes more significant, which is addressed by the LURE weights. Our combined weights unify the best of the two, accounting for the two sources of variance reduction. The following result bounds our worst-case estimation error under our variance reduction model, in a way similar the square root law for AIS.
SQRT
COMB
10-2
LURE
10-3
10
10-1
16a
10
102
10'
Figure 2: Normalized weights as functions of t for = 700 and N 1000.
Proposition 4. If Var [+] x TT"W/u,, where E [0, 1] and w, are non-decreasing. then the estimate :.OMB Et= a, 1it
COMB F,, where a COMB w., vT. satisfies

---

Var FCOMB
9
sup sup
(4)
121 vE(0.1)
Var (y,t)
where Vy, the estimation variance when the estimators are weighted proportionally the inverse of the ground-truth variances.
In 7 we show empirically that the combined weighting scheme produces lower estimation error than either of the two components alone.
Inverse variance weighting. In AIS, it has been advised not to approximate inverse variance weighting using estimated variances (29]. due to the unreliability of variance estimates with only a finite number of samples in each round of AIS and the potentially large bias this could introduce. In the next section, we introduce consistent estimates of (conditional) variances and show that these can make inverse variance weighting practical and achieve even better estimation error than combined weighting scheme in some settings.
5 Variance estimation and confidence intervals
Along with an unbiased and consistent estimator Fi:: we typically want a consistent estimate of Var [ar[Fi:e] to understand the error and construct confidence intervals. However, this is complicated by the fact that the estimators F, are not iid. Rather, each depends on the samples drawn in the previous step, and we only have single random draw for each As result, we will be able to form an unbiased but not consistent estimate of Var (ar]) unlike in the iid case (but similar AIS 43)
Martingale convergence. Fortunately, we can turn to martingale arguments to instead estimate conditional variance that is appropriate for confidence intervals. For this argument, consider the centered sequence of estimators F1, = E a, (F, F(I)) where the weights unnormalized. Because each Fe is unbiased, the displacement Fix F1 ar F(2)) has zero mean, making the sequence Fi:: a martingale. A martingale central limit theorem can be used to show that averaging over enough of these zero-mean displacements gives a limiting behavior similar to iid averaging. Stated informally, we have the following:
Proposition (informal). Let V Et= a Var [ be the conditional variance of Fir- FII-P(9)) 2 10,
Under suitable regularity conditions,
Viaa
We provide formal statement and proof in $ A.5 which requires constructing a triangular array and letting the number of active measurement steps T and domain size N go infinity jointly. See [43 and [33] for similar results for AIS, i.., sampling with replacement. This result motivates forming estimators of the conditional variance for in confidence intervals.
Novel conditional variance estimators. We first focus on the conditional variance of single estimator Fr, which is the variance of the importance weights (())/gr (s) where s ~ qr:
f(s) ( F(92) ((()).)) (-(s)
Varl Var[Fr1D,
Σ (s)
(5)
SEN D.
In importance sampling, we typically estimate this variance over iid samples from gr. but in active measurement we have only single sample 8- ~ Gr- We therefore derive novel variance estimator that uses samples 87, 8r41s... to estimate Var[Fr following importance sampling approach similar to that the base estimator in Eq. (T). For 2 T. let
[(() - F(2) F(A)D.)) D.)) + (1r) (x)
f((r) - F(2) p.) E(D)D.)) (x,) (1(3,)
Var
Σ (s)
(6)
BED, D.
The first term of Eq. 6lis the exact sum of the terms from Eq. for s E ( term is an importance sampling estimate using s, ~ or for the sum of the remaining terms. We then mix the variance estimates Var for each to get the combined estimator Var, = Σt 33 Var where Br is another sequence of weights; for simplicity, we will always use the LURE weights 3l
8r-1) 1. and the second
gLURE" wy to model the shrinking sample space.
5

---

Proposition 6. The estimators Var, and Var,,r for S r S t satisfy B D.] E|Var,,-| D-] Var[F+| and El Var=1 B| Var (F).
Further, beacuse it averages over many individual estimates, the error of Var, for estimating the conditional variance converges as N:
Proposition 7. With the same settings as Prop. the variance estimate for Fr weighted by the LURE scheme Var.uRE = Σi= LURE Var satisfies that Var [Var,URE) DD+] E 1-7+1 min (1, (N-1)²
The notation H hides multiplicative constants with respect t. When t < N, we have the usual Monte Carlo rate of (L T 1) (recall that ( - + 1 is the number of samples). On the other hand, when N, faster rate is achieved thanks sampling without replacement. This result contrasts with the usual practice of variance estimation for AIS, where only the samples from one stage are used estimate the conditional variance, and the stagewise estimators do not converge.
Confidence intervals. Several practical steps remain to construct confidence intervals. The full variance estimation procedure is shown include Alg. Previously, we assumed knowledge of F(2) D,) for estimating the conditional variance, but in practice we use the plug-in estimator Gr,, Fi:t-1 - F(D+) R F(S2) Dz) from step of active measurement. The full conditional variance estimator is then Var = Ex=1d a Var, x Ei= & Var D,] Vi. which according to Prop.5]iis the appropriate quantity for the martingale central limit theorem. The naive implementation to compute Var using Alg. 2 takes (t time but we can reduce the complexity to O(t) with confidence interval is constructed as Pire + Ra/2 /ar where Fa/2 is the 1 - a/2 quantile the standard normal distribution.
streaming algorithm;
BB. for details. Finally,
We will also consider confidence intervals formed using the "simple" conditional variance estimator Var lit = D a Fr 2 which is the motivated by the fact the sum of squared deviations 7=1
EL aF 2 of the martingale sequence converges to the conditional variance V2. under the same regularity conditions as Prop. 5 see $ A.5 for details. When needed, we will denote the estimator Var from Alg. as Varcond emphasize that is based on estimating conditional variances for each T.
Conditional inverse variance weighting. Prior AIS literature has advised against using inverse variance weighting with estimated variances [29]. However, our analysis suggests that weighting with estimated conditional variances may be appropriate. First, Prop. 5 can be read as Fi.e PV (FF(I)) Vire), which motivates choosing weight sequences o, minimize the conditional variance V Σ a? Var / [D] the optimal choice being a x 1/ Var[F=|Dr]] Second, Prop. controls the error of the conditional variance estimators Var, R Var [F][D.. Because the error decays with the number of samples t - + estimates for < t will more reliable than those for R3 This motivates proposed scheme, which uses inverse estimated variances for S 7t and continues the sequence using the a COMB weights for yt, where 7 E (0, 1) is a hyperparameter:
1//arr COMB
1 $ yt
INV Og
1/Var COME
The constant factor (1/Var.,)) (a COMB) for 7t ensures that the weight sequence continuous at 7 yt. We show in our experiments that these weights can produce even better estimation results than the a COMB weights for suitable settings of Disadvantages are that Fi: is no longer unbiased when using estimated weights, and that confidence intervals may have poorer coverage.
Experiments
We mainly experiment with two tasks: 1) counting birds in high- high-resolution images of Tree Swallow flocks, and 2) counting roosting birds weather radar images across multiple radar stations and years. The first task is analogous to common counting problems in microscopy or medical imaging, where an object detector trained on a few examples may perform reasonably well. In contrast, the second task is considerably more challenging, as it requires a custom detector-typically the result

---

of substantial community effort involving dataset annotation, model training, and validation. To demonstrate domain generality, we also performed experiments on malaria-infected cell counting and damaged- buuidinguiiiiggg counting from satellite images. For all IS-based methods, we use proposal distributions proportional to predicted counts using an object detector.
Counting birds in high-resolutign images. We aim to estimate the total number of birds in two separate images-"sky" and "reeds"-eat with different difficulty levels. The reeds image is more challenging due to its higher bird density and more complex background (see Fig. [ and Fig. AT] We divide the sky and reeds images into tiles of size x00x200 and 160x 160 pixels, respectively. and manually annotate the birds in each tile using the VGG annotator [T]. This results in 925 tiles for
sky and 1,426 tiles for reeds, which serve as the annotation units in our experiments. In total, the ground truth bird count is 5,682 for the sky image and 12,486 for the reeds image.
To detect birds, we train Faster R-CNN 35 detector with ResNet-50 17 backbone pre-trained on ImageNet [9) using the Detectron2
library. The model is fine-tuned with a single A16 GPU for 400 iterations and a learning rate of 0.001 on the annotated tiles. The detector performs reasonably well, with average error rates of 9.5% and 33.1% when trained on 50 randomly selected tiles. We also experimented with few-shot detection models [4 36) that do not require training, but found that a standard Faster R-CNN detector outperforms them when even a few labeled tiles are available. See D.1 for a comparison.
Counting roosting birds in radar. Another scientific application involves estimating bird counts from weather radar. Birds often congregate overnight in large numbers. Their mass departure from roosting sites in the morning can be detected by weather radar and the roosts leave visible signatures in radar-collected data channels such as reflectivity (2)g.(A2) S I0] Scientists can use these signals to estimate total bird counts, providing valuable data to analyze long-term migration trends.
We adapt the experimental setup of DISCount 322. leveraging expert annotated roosts from the Great Lakes analysis in (5) and [10] as ground truth. Our objective is to automatically estimate the total of the daily bird counts for each of 11 radar stations in the Great Lakes region, for dates between June 1st and October 3stt over the five-year period from 2015 to 2019.
We follow Perez et al. 31 to pretrain roost detector manually labeled training set from multiple radar stations that do not include the Great Lakes stations. It based on a Faster R-CNN architecture with a ResNet-101 backbone and an adapter layer handle radar channels across elevations and time steps. Detections from consecutive timestamps are assembled into tracks and bird counts are estimated from the tracks based on the radar geometry and reflectivity of the birds. We provide details in & C.I The detection task is challenging and the best model only achieves 56% mean average precision. Thus, (5] [0] invested substantial manual screening time for scientific analyses that require high precision. Our method substantially reduces this screening effort in estimating the bird counts.
We adapt the detector to each station with learning rate 10-4 for 3000 iterations on a single A16 GPU. To reduce overfitting, we use a mix of 80% pretraining data and 20% station-specine labeled data. The model is finetuned every 10 samples for the first 40 samples, after which performance saturates.
Counting malaria-infected cells. The Malaria Cell dataset (image set BBBC041vI, available from the Broad Bioimage Benchmark Collection (24) comprises 1,364 images (about 80,000 cells). This dataset contains microscopy images of blood smears used for detecting and classifying malaria parasites, aimed at enabling the development and evaluation of automated methods for parasite detection, quantification, and stage classification. For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. We use the same settings as our sky and reeds image experiments. For our initial model we finetune the default Faster R-CNN network on three randomly selected cell images.
Counting damaged buildings. For damaged building detection we focus on the Palu Tsunami subset of xBD [SS), which contains 1 113 satellite images of the shoreline before and after the Palu Tsunami. We count the number of damaged buildings. which corresponds to a label of "major- damage" or "destroyed"" Only the post disaster images are given to the model. We use the same hyperparameters as before. The initial model is trained on 5 randomly selected images.
7

---

Sky Image
Reeds Image
KCLE
KGRR
100
109
fractional error
10-1
10-1
10 2
N=
N=1426
N=765
N=765
10
10
10
10
10
10
10
10
10
10-1
fraction of labeled units
fraction of labeled units
fraction labeled units
fraction labeled units
MC
DIS
DIS AIS
Raw Detector
MC+WOR
DIS+WOR
DIS + AWO---SOOOROO (Ours)
Radar Task
fractional error
DIS
DIS
DIS+ WOR
AIS+ (Ours)
Raw Det, t=
Method
Frac. Erc.
3.78
0.4
Raw Det, 1=10
1.68
Raw Det, t=20
1.74
0.2
Tb Mb I
Raw Det, 1=30
2.41
Nanl
Raw Det, t=40
2.79
Dave Iim KAPX KBUF KCLE KDLH KDTX KGRB KGRR KIWX KLOT KMKXK KTYX
0.0
Act Mea, t=40
0.23
Act Mea, 1=200
0.06
stations
Figure 3: Estimation error on two measurement tasks. Top: Fractional error of the estimated count as the percentage of labeled tiles increases, averaged over 10,000 runs for the counting birds in the "sky" and "reeds" images, and counting roosting birds KCLE and KGRR radar stations. Bottom: Fractional eтror of the estimated count after 200 labeled days for the 11 radar stations, using different estimators. The bottom-rigee rightt table shows that the geometric average fractional error across stations for the raw detector and active measurement across iterations. We see that both the adaptation and the sampling without replacement are beneficial, and quickly outperform the detector and baselines for both tasks.
Baselines. We compare our work against DISCount 32 which we will denote DIS. We also investigate the impact of the two major extensions individually: sampling without replacement (DIS+WOR) and fine-tuning the detector similar to adaptive importance sampling (DIS+AIS). Active measurement combines all of these, simple Monte Carlo (MC), ie., DIS with uniform q 332. and MC without replacement (MC+WOR) Lastly show the performance of the detector predicted counts after each step of adaptation.
denote it as AIS+WOR. We also compare against
Since averaging over many trials is computationally tuning (+AIS) is involved we approximate the effect of interactive model adaptation using a "fixed checkpoint" approach that uses a fixed sequence of detectors pre-trained on an increasing number of labels sampled according to a fixed strategy. Specifically, we sample annotation units uniformly from those with non-zero counts for different sizes. These fixed model checkpoints are then reused across all trials, allowing to run significantly larger number of evaluations. In real- real-world deployment, a practitioner would typically fine-tune the detector using the same samples on which annotations collected during active measurement. However, find that the estimates produced using this approximation match those obtained via a full end-to-end setup, as discussed in $ D.3
prvensive-particllaryy when detector fine-
Evaluation metrics. We evaluate our methods using fractional error and CI coverage. Specifi- cally, we estimate them using the fractional error 1/M EM [f(m)) F(9))F((() and coverage F(2)) S 7a/2 Var
1/M EM M[ 1:t
(m)) 1/2 where M is the number of trials, F(22) is the
ground truth measurement, and (fim) Var are the estimates from the mith trial.
Results
Main results. Our main results for both tasks are presented in Fig. The top row illustrates that simply using raw detector predictions (i.e., directly interpreting model outputs as estimates) can lead to suboptimal performance, particularly when the detector is biased exhibits irreducible
8

---

1.2
Sky Image
Reeds Image
1.0
KDLH
KDLH
1.2
1.1
DIS+AIS
0.99
0.8
DIS+AIS+WOR
2.0
Varg.1
0.9
SQRR
SORI
0.88
2
Verf.!"
LURE
110 10.5)
LURE
INV (0.5)
coverage
100
0.7
100
200
100
200
radiusigt
Figure 4: Relative crrors compared to COMB weighting. Other fixed weight- ing strategies (a (QRQ a LURE) are worse, but inverse variance weighting (denoted by INV (y = 0.5)) may achieve lower error.
Figure 5: Coverage and radius (relative the ground- truth) of CIs on the roost data for station KDLH as function of (from 5,000 replications), built with either variance estimators from $ The left panel uses the Varior estimator. We achieve the desired coverage with narrower Cls.
errors. In the sky and reeds images, the detector performance improves with additional training but typically saturates, In contrast, our method demonstrates greater robustness to biased predictions and continues improve detector performance improves. Compared to other unbiased estimators, active measurement tiles, and further gains are achieved by sampling without replacement once approximately 10% of tiles have been labeled. These improvements are particularly notable in the challenging reeds image, where the detector's baseline performance leaves considerable room for enhancement.
SSS+DIS+WOR outperforms DISCount (DIS) with as little as 1% of labeled
On the radar task, shown in the top-right panels and the bottom row, the detector performance (summarized in the table) is substantially worse. Fine-tuning station-specific data reduces the frac- tional error from 3.78 to approximately 2, averaged across stations, but performance again saturates. However, active measurement (AIS+DIS+WOR) continues to yield improvements. Improvements are consistent across all radar stations and are substantially better than DISCount (DIS).
While the fractional error of the raw detector saturates quickly, observe that additional fine-tuning continues to improve the proposal distribution, which in turn leads to better AIS performance (see $D.2 for details). In Fig. A4 also compare true end-to-end training with the more computationally efficient fixed checkpoint scheme. We observe that fractional errors between the two methods are not significantly different, especially when fewer tiles labeled. With higher number of labeled tiles, we see greater variance arising from the choice of the specific checkpoint; however, the relative performance remains consistent.
Different weighting schemes. We now examine the performance of differentt weighting schemes in active measurement. According to our theory, we expect that our proposed weights a COMB work consistently well for all t. In Fig. 4 find that this is the case for the counting problem in the high resolution images. When t small, the visual detector works poorly, so weighting schemes that assign equal weights early-on, like a LURE, work worse than a COMB As t increases, the reduction of the sample space is more significant, so weighting schemes that do not consider this, like the SQRT weights, works increasingly worse. For large the performance of LURE weights will eventually converge to COMB weights, but are never better. We also test the inverse variance weighting scheme with different hyperparameters 7. The full results can be found in Fig. A5 In general, a conservative y 0.9 can be detrimental because each individual estimator may not be accurate enough. In the middle, we find that y = 0.5 performs well and can achieve even smaller error than a COMB
the performance gains from fine-tuning saturate,
0.3 brings little benefit, while an aggressive
Confidence intervals. We evaluate variance estimators and confidence intervals by looking at CI coverage and width over 5,000 trials on the radar counting problem. Results from one station are shown in Fig. 5 and the full results can be found in Fig. A7 Overall, we find that the DISCount baseline undercovers and has wider confidence intervals, due its higher estimation error-we expect width to be proportional to error if the coverage is correct. With adaptation, both metrics improve, but are different with different estimators. We compare the two variance estimators Var simp and V in the second panel. In most of our experiments, the coverage with them both improves with more samples and converges to the desired confidence level.

---

10f
Sky Image
107
Roads image
Malaria Cells
Dama Buildings
fracioonal enor
10a
MC
10-1
20-1
MC+INGR
Hr.
DIS+WOR
Actipe lesling Active measurement
Active leel Active messureeent
DIS+AIS
10
2215
10t1
hw113
DIS+AIS-WOR
10"
10
Tracton of tobeles tils
10
10-
15
traction of dflableed tles
18 10
traction of llaelled units
fraction of lsbeled units
Figure 6: Fractional emror compared with other baselines (H is motivated by PPI).
Figure 7: Estimation error additional datasets, av- eraged over 1,000 end-to-end trials.
Comparison with other baselines. We use an experiment to illustrate the differences between active measurement and other baselines. Assume the loss function is If(() g(s) for any s E Ω. Active testing will use an acquisition distribution proportional to the approximate loss to select units to label. Even if we know the ground-truth loss values, this acquisition distribution is worse than active measurement because of the mismatch between the proposal distribution and the estimand. Similarly, another estimator, motivated by PPI, is H, E Sen g(s) N - uniformly sampled labeled data. This estimator is also worse than active measurement because requires fixed detector and does not select the best labels for estimation. This is shown in Fig. 6
(g(s,) f(-)) with
Additional results. Additional experimental results on counting malaria cells and counting dam- aged buildings are in Fig. 7 Here we run active measurement with end-to-end settings and average over 1,000 trials. From the first figure, we observe trends similar to those seen in the image and radar experiments for malaria cell counting. DIS consistently outperforms MC, while AIS shows early reduction in error as the model improves. WOR also demonstrates steady improvement, with its impact becoming more pronounced after roughly 10% of the data is labeled. The results on the second dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. We also note that our performance is lower than the numbers reported in DISCount [32], as they use a detector specifically trained for damaged building detection, whereas use simple ImageNet-pretraiee
backbone fine-tuned on just satellite images. These results show that active measurement is capable of improving scientific estimation tasks across different domains.
8 Conclusions, Limitations, Future Work
We introduced active measurement, framework that interactively leverages AI models achieve precise scientific measurements. In contrast to traditional workflows, which are prone to errors from fully automated methods, active measurement integrates Monte Carlo estimation and model adaptation with iterative human labeling. This approach yields unbiased estimates with calibrated confidence intervals. We formally derive the unbiasedness and consistency of our estimators and propose novel techniques for sequential update weighting and uncertainty quantification. Empirical results on two measurement tasks show that active measurement not only reduces estimation error but also provides reliable uncertainty estimates, outperforming existing methods.
One limitation of our approach is that the estimates may still not be precise enough for all applications; users should carefully validate the safety of any Al method in high-stakes scenarios. Another is that Al measurement has potential to be misused, e.g., to target vulnerable populations: our work does not enable new measurements but can make them more accurate.
There are several promising directions for future work. Beyond simple fine-tuning, unsupervised, semi-supervised, and transductive approaches could improve predictions across the full dataset. The current importance sampling strategy prioritizes units with high counts, but early-stage measurements could adopt more balanced sampling to better support detector training- for example, through active learning. Although model adaptation improves performance, there is a substantial computational overhead; in our experiments, we fine-tuned deep networks for detection tasks on GPUs. To enable interactivity, future systems must support fast and lightweight model updates. Approaches such as few-shot counting models and in-context learning architectures may offer practical solutions for efficient adaptation. Another promising opportunity is to consider correlation among images. As an example, in the reeds image we have observed spatial auto-correlation in the residuals of the Al model. Fitting Gaussian process can exploit this spatial structure to improve predictions for unlabeled tiles.
10

---

Acknowledgments and Disclosure of Funding
We thank Maria T. D. Belotti for helpful discussions and contributing the algorithm to estimate bird counts from radar data. We also thank Jody Dole for kindly granting permission to use the sky and reeds photographs featured in our main experiment. This work supported part by NSF grants #2504073, #2406687, #2329927, and #2210979.
References
[1] Merlin Sound ID. 1t..tttss//heerii..alllaabbrbrrr wwtp://ddd..rg/ooddddoooo idd accessed 13 October 2023.
[2] Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: systematic review and meta meta-analysis. NPJ digital medicine, 4(1):65, 2021.
[3] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zmic. Prediction powered inference. Science, 382(6671) 2023.
[4] Carlos Arteta, Victor Lempitsky, and Andrew Zisserman. Counting in the wild. In European conference on computer vision, pages 483 483.498. Springer, 2016.
[5] Maria Carolina TD Belotti, Yuting Deng, Wenlong Zhao, Victoria F Simons, Zezhou Cheng, Gustavo Perez, Elske Tielens, Subhransu Maji, Daniel Sheldon, Jeffrey F Kelly, et al. Long-term analysis of persistence and size swallow and martin roosts in the US Great Lakes. Remote Sensing in Ecology and Conservation, 2023.
[6] Monica F Bugallo, Victor Elvira, Luca Martino, David Luengo, Joaquin Miguez, and Petar M Djuric. Adaptive importance sampling: The past, the present, and the future. IEEE Signal Processing Magazine, 34(4):60-79, 2017.
[7] Jeffrey Buler, Lori A Randall, Joseph P Fleskes, Wylie C Barrow Jr, Tianna Bogart, and Daria Kluver. Mapping wintering waterfowl distributions using weather surveillance radar. PloS one, 7(7):441571, 2012.
[8] O. Chapelle, B. Scholkopf, and A. Zien, Eds. Semi-supervised learning [book reviews]. IEEE Transactions on Neural Networks, 20(3):542-50) 2009. doi: 10. 1009/NNN2009..01599
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10.1 (00pCCVR..000.520000
[10] Yuting Deng, Maria Carolina TD Belotti, Wenlong Zhao, Zezhou Cheng, Gustavo Perez, Elske Tielens, Victoria Simons, Daniel R Sheldon, Subhransu Maji, Jeffrey F Kelly, et al. Quantifying long-term phenological patterns of aerial insectivores roosting in the Great Lakes region using weather surveillance radar. Global Change Biology, 2965):1007-141,, 2023.
[11] A. Dutta, A. Gupta, and A. Zissermann. VGG image annotator (VIA), 2016. Version: 2.0.12, Accessed: Mar. 2025.
[12] Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[13] Fernando mative technologies digital agriculture: Leveraging internet of things, remote sensing, and artificial intelligence for smart crop management. Journal of Sensor and Actuator Networks, 13 (4):39, 2024.
Karen Gutter, Ricardo Vega, and Gilda Carrasco Silva. Transfor-
[14] Demin Gao, Quan Sun, Bin Hu, and Shuo Zhang. A framework for agricultural pest and disease monitoring based on iffernet-of-things and unmanned aerial vehicles. Sensors, 20(5): 1487, 2020.

---

[15] Ritwik Gupta. Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew Gaston. Creating xBD: A dataset for assessing building damage from satellite imagery. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition workshops. pages 10-17, 2019.
[16] Peter Hall and Christopher Heyde. Martingale limit theory and its upplication. Academic press. 2014.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL http::t//////i...rrrr org/ abs abw///12...... 03385
[18] Jason W Horn and Thomas H Kunz. Analyzing NEXRAD doppler radar images to assess nightly dispersal patterns and population trends in Brazilian free-tailed bats (tadarida brasiliensis). Integrative and Comparative Biology, 48(1):24 39, 2008
[19] Justin Kay, Peter Kulits, Suzanne Stathatos, Siqi Deng. Erik Young, Sara Beery, Grant Van Horn, and Pietro Perona. The Caltech fish counting dataset: A benchmark for multipleobbject tracking and counting. In European Conference on Computer Vision, pages 290-311. Springer, 2022.
[20] Jannik Kossen, Sebastian Farquhar. Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, pages 5753-5763
PMLR, 2021.
[21] Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, Kristen Grauman, et al. Crowdsourcing in computer vision. Foundations and Trends@ in computer gruphics and Vision, 1773):77-223,, 2016.
[22] Nico Lang. Walter Jetz, Konrad Schindler, and Jan Dirk Wegner. A high-resolution canopy height model the earth. Nature Ecology & Evolution, pages 1 1-12, 2023.
[23] Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven Bamford, Daniel Thomas, M Jordan Raddick, Robert C Nichol, Alex Szalay, Dan Andreescu, et al. Galaxy Zoo: morpholo- gies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 389(3): 179-11999991 2008
[24] Vebjorn Ljosa, Katherine L Sokolnicki, and Anne Carpenter. Annotated thtuuhurou microscopy image sets for validation. Nature methods, 97)):33,, 2012.
[25] Chenlin Meng, Enci Liu, Willie Neiswanger, Jiaming Song. Marshall Burke, David Lobell, and Stefano Ermon. IS-Count: Large-scale object counting from satellite images with covariate- based importance sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12042, 2022.
[26) Phuc Nguyen, Deva Ramanan, and Charless Fowlkes. Active testing: An efficient and robust framework for estimating accuracy. In International Conference on Machine Learning, pages 3759-3768. PMLR, 2018.
[27] Man-Suk Oh and James O Berger. Adaptive importance sampling in Monte Carlo integration. Journal of statistical computation and simulation, 41(3-4):1433 168, 1992.
[28] Art B Owen. Monte Carlo theory, methods and examples, 2013.
[29] Art B Owen and Yi Zhou. The square root rule for adaptive importance sampling. ACM Transactions on Modeling and Computer Simulation (TOMACS), 30(2):1-12, 2020.
[30] Genevieve Patterson, Grant Van Horn. Serge Belongie, Pietro Perona, and James Hays. Tropel: Crowdsourcing detectors with minimal training. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 3, pages 150-159, 2015.
[31] Gustavo Perez, Wenlong Zhao, Zezhou Cheng, Maria Carolina T. D. Belotti, Yuting Deng. Victoria Simons, Elske Tielens, Jeffrey Kelly, Kyle G. Horton, Subhransu Maji, and Daniel Sheldon. Using spatio-temporal information in weather radar data to detect and track communal URI http:/wwwwwwwww../////////////tt biorxiv
bird roosts. bioRuiv, 2022. doi: 10. 10..:00.22222..............22.1. httc///ontttttrrteeeeeeeeee httpa////eaaaaae////0///////////
513761
12

---

[32] Gustavo Perez, Subhransu Maji, and Daniel Sheldon. DISCount: counting in large image collections with detector-based importance sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22294 223022, 2024.
[33] François Portier and Bernard Delyon. Asymptotic optimality of adaptive importance sampling. Advances Neural Information Processing Systems, 31, 2018.
[34] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3394-3403, 2021.
[35] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016. URL U.LLippps///axxx.o////bbb////// 01497
[36] Khoi Nguyen Thanh Nguyen, Chau Pham and Minh Hoai. Few-shot Object Counting and Detection. In Proceedings of the European Conference on Computer Vision 2022, 2022.
[37] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting, Large language models in medicine. Nature medicine, 29(3):1930-1940, 2023.
[38] Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank van Langevelde, Tilo Burghardt, et al. Perspectives in machine learning for wildlife conservation. Nature communications, 13(1):792, 2022.
[39] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1769-8778, 2018.
[40] Mike Walmsley, Chris Lintott, Tobias Géron, Sandor Kruk, Coleman Krawezyk, Kyle W Willett, Steven Bamford, Lee S Kelvin, Lucy Fortson, Yarin Gal, William Keel, Karen Masters, Vihang Mehta, Brooke D Simmons, Rebecca Smethurst, Lewis Smith, Elisabeth M Baeten, and Christine Macmillan. Galaxy Zoo DECaLS: Detailed visual morphology measurements from volunteers and deep learning for 314,000 galaxies. Monthly Notices of the Royal Astronomical Society, (09()))3966399)))) 3968-3988, September 2021. ISSN 1365-2966. doi: 10. m0.1999//rra///sa22209.
[41] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu. Payal Chandak. Shengchao Liu, Peter Van Katwyk. Andreea Deac, al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
[42] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan Yen Lo, and Ross Girshick. Detectron2. https://giis http:tttww.oooeccccoooeeeeeeeeee httm///cccbooreeeeer//ettttttttt 2019.
[43] Yi Zhou, Adaptive importance sampling for integration. Phd thesis, Stanford University, 1998.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our derivation of estimators, weighting schemes, and confidence intervals is included in Section 2. We show that method results in lower estimation error in Figure 3. The coverages of our confidence intervals are verified Figure 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Section discusses limitations of our work, such as fine-tuning overhead and suboptimal detector improvement in the early stages of measurement.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We provide a proof for all of our propositions and state the assumptions clearly in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide pseudo code in section 2 for the Active measurement and variance estimation algorithms. In section 4 we provide the details of counting birds in high-resolution images, which uses an off-the-shelf detector, and counting birds radar, which closely follows Perez et al. 32 Finally we link to our code on github at the end of the introduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
15

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes)
Justification: We provide access to our code and documentation on github with a link at the end of the introduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcc/ccccccccccccccooooo httie//gggudeee/eeeSuubbiiiii wwweiisssooPoPoiiiio
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the important details the results section, and more specific information in the appendíx.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, an important contribution of our work is that provides error bars for the measurement task. We include these error bars in our results and evaluate them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the compute hardware used in the experiments section, and provide more detail in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the cc
NeurIPS Code of Ethics
Answer: [Yes]
Justification: Justification: We conform with the NeurIPS Code of Ethics to the best of ability.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss both positive and negative societal impacts in Section g
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper poses no such risks. We do not plan to release pre-trained models, as running our method is focused on the fine-tuning procedure itself, not the final weights of a trained model.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The NEXRAD radar data that use is open to the public and can be used as desired, and the roost and sky images were used with permission from the photographer. We adapt code from Perez al. 31 and will give credit in the publicly released repository.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the balaaets.ppaerssttt paperssii thcode. com con/datasess package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation for the data and code is provided with the code submission. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper's method does not involve LLMs..
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hatt:p///eerrppssccc///oeeerrrre for what should or should not be described.
20

---

A Proof of the theoretical results
In this section, we restate each theoretical result and provide proofs.
A.1 Proof of Proposition
Proposition 1. The combined estimator Fin -E Gr Fr is unbiased: E[Fi.) = F(2).
Proof. We first show that each individual estimator F,, is unbiased.
ED.,,, 8, [Fr] = ED,,,, F(D,) f(*+)
9-
f(r))
ED,
F(D+) + +Ber
[g+()+)
F(Dr) + Σ (3)) f(3,)
EED
4r (8()
SEE D.
ED, [F(D,) + P,)
F()).
Therefore, for the combined estimator
E Fiel = Σ a Fr
ar TE F+]
Σ ar F())
1=1
= F(22).
A.2 Proof of proposition
Proposition 2. For any 5 < Cov P.) 0.
Proof.
Pr)= - F( - F(2))
-ED,1(FF F(?)) E., 1Ř, F(2)]]
(F, F(2)) Σ (r(r)) (((,)) + [(x)
ED,
F((2)
((r)
S,En Dr
Σ (r())) f(sr)
ED,
(F, - F(R))
F(D,) +
F(22
(r(sr)
D.
( F(N)) 0)
ED,
0.
21

---

A.3 Proof of proposition
Proposition 3. If there exists constants A < B such that for any s E S2, A f(s) - and
A S (()) & B, then there exists a constant C' such that Var[P,] - C r) -
Proof. To start, we first bound the sampling distribution implied by the detector mcasurements g(s).
For unit E Ω2 Dr, the sampling distribution is 9- (s) Exee g(ry' By the lower and upper bounds,
g(s)
that (N- (+) S -ー- Also, Ever, D. f(s) with the same technique, which (N- --+++11 = F( - F(D,) & (N-++1)B.
bound F( F(D,)
Next, we write the variance of F, in terms of expectations.
F(D,)+ + f(++) F((2) 27
Var ED.,.-
(r(
f(s,)2 + (F(D,) F(2))² +((((D)) F(2)) f(+))
ED,,,
(s())
17 (8+)
(N - + +1)
+ (N -N+ 1) 2(N - 7 1) A3
S ED,,,
A²
B
(N -N+++1) (B²
A²) 2
A
When < N, we further have
Var [+] = (N 7 1)3 (BB + AP) 2
A
N T + 1 B A 22
(N r) -
A2B
(B" + A²) - 4A5
< (N 7N - 11
A²B
Also note that
F(DN) [(NN) F(S)) (N(SN)
[ar[FN]] DDN
(F(DN) + f(SN) F((2) 27
EDN,ON
=EDx
(F(()) F((2)) O.
This means that for all F S N, Var[f;] C(N - T 11,, where C= 2B² 42)
A=B
A.4 Proof of proposition
Proposition 4. If Var(Fr] x T -/WW where y E [0, 1], and w, are non-decreasing, then the estimate Fromb = Σt= acomb Fr, where gooub o W, /F and Σt o comb 1, satisfy
Var [rcomb) Varopt (v,
sup sup 121 yE[0.1]
g'
where Varopt (y, t) is the estimation variance when the estimators are weighted proportional to inverse of ground-truth variances.
Proof. Denote the variance of combined estimator weighted by af & w by Var (v. t), where [0, 1]. An observation is that Var [rembb) Varo.5(y. t), and Var (y, Var, (v. t). So
22

---

shall bound ((rl. (2ry(
Note that for any E [0. 1|,
Var, t) x 2rm 2 -- 1/////0
E+= W7r
-r= Wy 22 y
Σ W+T
Therefore, let
Σt W (,-22-1)
(Er Wy WA
Var, (v. Var, (y,
(y,
E WyT2
we have
811
Er= Ere W. J 2x 2 (log - log r') 0.
0yy
Er= W7
So ty) is strictly convex over y. Also note that (y, is symmetric around y So (1,tt
$1/2 In the combined estimator chose 0.5, which leads the supremum (1, for any t. Furthermore,
(0, (2.0)) 1/2
0.5 is the best assumption for the mixing weights. The derivation is the same as Owen and Zhou 22 and we omit it here. In such case
(Er-1 wr (Erz) W,T
E
sup 1y,s(y,1) 1o.5(1,1))) vE[o,1]
Dtem Wy70.5
2
2
Dr=
7.5
(E'r w.y)
can plot this specific function for few different choices of w, and I
Empirical Bound of Variance Ratio
1.12
1.10
Variance Ratio
1.08
N=100
N=1000
1.06
N N01000
w,=1
1.04
1.02
1.00
10a
10a
102
103
10
t
The lines labeled N correspond the combined weighting scheme proposed in main method (see Section 4), and w-- = a uniform weighting. We can see empirically that these are all bounded above by 2
To prove this bound, we define the following weights,
Wt
0
8r = (w, w,-1)(t T 1).
Dt= w+
23

---

Since W, are non-decreasing, 0; - 1 2 0 and thus B, 2 0. We can also cancel out terms of telescoping series to see that Σ 8 - "' = 1. Therefore both úù, and [3 are non-negative
weights that sum to Note we define w0 = 0.
Plugging these in, and borrowing a trick from Gabriel's Staircase, we can rewrite as,
Σx -= P
E+=1 WyT
Σ 3r 1-7+1 2y'
Σ ud,70.5)
Er= 3 Er'=r -10.5
! E
0.5
We can bound this expression by considering the supremum over all possible positive weights X,,
Σ 3r --r+I Er'=r* T'
E d Exl'm A, t-t+1 Et'zr +'
sup AER 20
E Br E Ej'zy 710.5)
E Ei' d, b-4 Ey' -10.5
Er= Ar) (Erz) 1-t+1 Et
sup X
E F-T+T Er 700.5
We can simplify by using the closed form expression for sum of consecutive integers and a lower bound for the sum of consecutive square roots from Owen and Zhou (2)]
E Ar) (Er 2_ sup AER
-ー (t+r)(t-++))
E 1-7+1 (4055)2.--1-000000
1.5
E/ Xx)
(Ermi 1t
9
sup AER
(8)
8
Σ
d du ((t + 0.5) - (T -o.5)..5))
i-7
Define this last expression inside the supremum as f(),t) Taking the derivative with respect to each
Ano
(Ita) (t (t- n) (Et=rdA) X-)
af(A.))
BAn
(Dta dy स. (I1 - 0.5)1.5 (t 0))
GIi Ar) (I:- ((+))) ( (1+0.5) (n
-(n-0.))1)) 0.5)
EE Ar Ar (a ((.00)) (s -6))
Setting this equal to and rearranging,
0f(),t) Irn
Eie Σーニ )) (2t + n)
2(Et de (f
(9)
(1 (t (++5, 1.b (n - 0.5)
Ei A (t++0.) 0.5) (r - 0.5)
Note the right hand side does not depend on n, the index of the partial derivative. Thus at a critical point, the left hand side will equal the same value for all n. We can define this left side as a function g(n, t, X).
To simplify, make the following substitutions:
ct A) Σ
d
r''=1 dr. 7)
0.5
u(n) = vn - 0.5
w(t) = Vt+ 0.5
24

---

Since 2 1, t 2 1, and n S t, we can note that u(n) > O, w(t) > O, u(n) < w(t), and 0.5 I ci (t, St 0.5. This alleviates concerns over dividing by zero going forward.
Applying these to g(n, 1, 2),
A)) Ci (1, + 2w(()2 2w())²+um)) (w(1) + uu()))+(m))))))))))) (cI + 2wo(1²² (c(( met)² (t))((()) + u(n)²
+ u(n)²)
w w(t)9~u()>' x(n)
Next we can consider the partial derivative over n and set it equal to zero, noting that u'(n) 2u(n)
ag t,)) u(n)(u(n)² + (t)t))(()2 + (2w(t)) (2 c2 (ut (w(t) + (t, A))
On
2u (u(n)² ((()u(())))))))))))))))))(())) + w u(n)² + 2w(t)u(n)2 (2w(t)² c1 (t, A))u(n) - 2w=(t) + (t, 0.
This is a cubic equation over u(n). Looking at the coefficients, and (w)(1) are both positive. Making substitution and using bound on cı (1)( the last term negative. By Descartes" rule of signs, the coefficients of this polynomial have sign change and thus at most 1 positive real root over u(n). Since u(n) is monotonic over n and positive for all n 1, we see that there exists at most one n such that do(n.
(1, A) 2 2/ + - - 0.5 = 1 1.5 0. Clearly
O, for any given and A. From this we can deduce that the equation g t, A) = h(t, A) for any function h(t, X) that does not depend on has at most solutions over n.
On
Applying this to Eq. 9 and noting that the left hand side is equal to g t, A), we can conclude that this equation can have no more than solutions over n. Thus the partial derivative of f(A, t) with respect to À can have at most 2 components with value 0 at the same time. We can also observe fcx,, t) = f(x, t) for all E IR thus the supremum must occur at some finite point. Since we're re optimizing over the positive orthant, that finite point must have all non-zero components with gradient zero. Thus there can be most non-zero components of À at the supremum. Continuing from Eq.
8. we now have the upper bound.
(Ei Ar) (I: (+(++ (I:
sup
AERt
((550.)) (r
07-008)18))) 0.5)..8)²
(A, (A.+++
(10)
Asicis
( 005) (1 055) + + - ((t 0.5)..3 - 15--0 -..0..)))))
We will denote this function inside the supremum as We can replace A and X, with variables and make similar substitutions as before.
u(i) vi 0.5
v(j = vj - 0.5
w(t) = Vt + 0.5
For more compact notation we will ignore arguments going forward. Making these substitutions yields.
9 (x + (x + u²) + yy + +)))
((,,0,i,jt))
8
x(w 23)
p(1od 13)
n + 2
w
Using computer algebra system, we can expand this as,
(1115 D
2 (Ir,,i,,,4
p= -u-orry - 2u wwwryy--aoo
2n 2 u2wyy x2 00 u 2u2 w
u² 2 + un
2u x² 2 2n un 3 xy + u 4 2
uww'ry 2u w + 2uv
+ 2w02 2w02
n wwzzx + w + v² w
q = (n(w² + ww + u² ) y ( + we + v (w ( + u))²
25

---

By definition 2 0 and q > Rearranging terms,
< u, 0 < w. and B < w. Thus we can see that
p p= u² (2w u) zy + 2n vw (w² - v²) my + u² w² (w² - -²) zy
+u²u² x² (4w² v²) ay + 2u w (w² 0²) xy + v (w² - v²² xy
+2n² v 2u²v" way + u v w x² + 22 w2 2u x fu + 2ur w y² v² w
Since w > u and w > v we can see that is a sum of positive terms so 0. Putting this together
we have > 0. Therefore continuing from Eq. 10
(1 P
sup
2(2, y, 1.j.t)
sup 12 20.
121.m.y20,
This proves our bound. In conclusion, we first show that the variance ratio is bounded by another ratio involving 2 weighted averages, We then use the fact that the weights are non-decreasing to transform the expressions into a weighted average of unweighted averages. Using closed forms and approximations we find simpler upper bound. We consider the supremum of this expression over all possible weights, and observe that the maximum can only occur with at most non-zero weights. Finally we analyze this 2 weight case and conclude it is bounded by 9 Thus we have, g'
Var Fromm)
sup sup f21 vE[0.1] Var (y, t) 8
A.5 Proof of Proposition 5 (martingale CLT)
To construct the formal martingale CLT, we need to let the number of active measurement steps T and the domain size N go infinity jointly. Suppose that for each N, we set the number of active measurement steps to be Tv (I'N < N) where TN is non-decreasing with N. By letting . - oc and Ty + co, we have a martingale array SN. = Σ'= XN,- = Er= ã (F, F(St)), for t < TN- Here ā u./ V T'v a² Var [+]] is normalized using total variances. is direct to check that [S, = O. With addition assumptions, we are able to construct
E[S SS.a] a limiting theorem for active measurement.
Proposition (formal). Assume that (1) ETH o Var F+ [ 1/² as N oo; (2) P(n²
oo) 1 and 2 > 0) 1; (3) TT E N. I[[NN,1 > D.] & for any > 0. We have
two central limit theorems
SNTW
D 10,,))
(11)
EIM X
SNTT
D4 10,1).
(12)
IN of Var Varl
Proof. We have met all the conditions for Theorem 3.3 and Corollary 3.2 in Hall and Heyde (16].
Then we directly have
SN.T D, 1).
UNITX
SN.T D, N()
VN.T
where
TN UNIV
X
r=1
Tw
VN,TN = Σ a Var Fr D.].
26

---

The formal proposition employs a different normalizer from active measurement. useful for bounding the discrepancy between conditional and total variances. For active measurement, we have the following corollary.
Corollary 1. With the same conditions as Prop.
5
we also have that
FITTN F(22)
L N,0 1),
(13)
EE
TN az P(2))2
FLITN FF2))
D4 N(,
(14)
Tx a? D+]
Proof. Basically the equations are the same as Eq. and 12 but with different normalizers. After multiplying VE, o², Var(F,,] both the numerator and denominator, we get Eq. 13 and 14
M 0.'
This leads to our informal version of the martingale CLT. Both Var cond and Var simp can be con- structed using the corollary.
We also demonstrate our assumptions in Prop. S with simple example. If Var[Ar] = of A1T-M/u /ww, and [ar[Fr| Dr] 2 9/w, where y [0, and A2 S \1 then with COMB weights
Ts Σ
M Tw Ag 7
a Var[Pr |Dr]
T=l
2r
Tw
72-V
A2
It is clear that 0 < A2/A1 < co, so the first two assumptions are met. For the third assumption, let B((A-F(()) o
D. < co. The additional
Ty = N/2 and further assume for some > 0, sup,,N
assumption implies the uniform boundedness of (F--)(0)2 which is common in the literature 43 Then note that
of
max, 0.o0
AIWTy T
Tv
aZ0?
EMM d1 11,
\ WTx T
A
T'y
IN Etm -1
Tx
1= Wh
NTI-9
(N Tn + 1)
IN E/ 71-31
NT
+ 0.
(15)
(N Tv 1 - y)
Therefore, for any T,
of(F, F(2))2
((( D.] =
I[[XN.r D.
ΣNH
E of,03
TN
00007
(E, - F(2))²
E
I(IXNr11 > e) Dr
Tx
r//l 02.02
d?
T
27

---

Note that
F(D2)2 o,
(F, F(O))
FF-F(Q)) V Er' az
([[xN,r(ee]]2)
> Dr
E
o
O+
0,
Observe that -( the uniform boundedness, then for any cg > 0, there exists No such that for any N No.
p.
A but
-
- az.
for any according to Eq.
15 By
of
2(22))
E
I II | >c]
< ED.
o
Therefore,
IN
Tv
0
Σ
TN
r'=1 a 0
==1
This shows that the third assumption is also met.
A.6 Proof of proposition 6
Proposition 6. The estimators Var, and Varz.r for B[Var,,r Dr] = Var[Fr| and EE [arr=] = B| = Var[Fr].
I satisfy B[Varr D.]
Proof. We first show the unbiasedness of individual variance estimators Var
Var D+1
= =Ep, Drior
Σ (-(s)
f(s) (8)
- F(()) D.)
(=(r)) f(sr)
- F(())(())))))))) (.)))..)))) ID,
(8,)) (r()))
sED, D.
(+(s) [s))
F(22 D.))
(xp)
[((pp) - -F(S)) D.)) ) Dr
= =ED,\D.
+ +E,,
(4(s))
(r()))
(r,)))
,ED, \ Dr
Σ (s) f(s) F(2) D.))
+ Σ (r(5,) ( [((r)
F(2) ((2) D.)) ID,
= =Ep, \D, Dr
(7(s)
(r(())
,ED, \Dr
*,EN D.
[(s) F(22 D.)) DD,
=Ep,
Σ (s))
(r(8)
SEQ D.
= -Var[Pr1D)
Also, note that
Varp, B[ D.]] Varp, E F(D-) + ((+))
[D=
((5))
[((,)) + f(s7)
Varp,
Σ g(st)
((3,)
*+EN D.
Σ ((37) f(7)
Varp, F(D,) +
((+))
8, D.
VarD,
[F(22]
0.
28

---

Therefore, we further have
BI Dr]] Ep, [ar[Fr]12] D.]
Var[Pr] - Varp, |
Var[F,]]
Since Var, is convex on Var,, with the weights, we also have V| Dr] = Var[F-| D-]
and
E| B[Var,] = Var [F+].
A.7 Proof of proposition
Proposition 7. With the same settings Prop. the variance estimate for F, weighted by the LURE scheme Var.uRE = LLURE Var,, satisfies that Var [Var Dr] 2 1-7+1 min (1, (N-02) (-+T)
Proof. We first derive the normalized LURE weights.
Σ BLURE
Σ
1 (N -r+1 1)
1
N N -+++1
N N -
t-++1
(N - -D)NN -
So the normalized LURE weights satisfy
LURE
(N ()(N - 1
(1-++ 1)(N () -
We then bound the variance of individual variance estimators. Observe that D,]
Var[LE[Var, [ D, V|Var [+]] D.]
VarlVar| F. D+1 Var Var,, D D+]
4r (++)
[(xr) F(22) p.)) 2 D,
E
Var
DD
(x,)
(8,))
9- (s,))
f(8r) F(2) D.)) |D,
$ E
E
Drr
9r (s,))
(87)
E
E
(s,))
8
f(s,)*
+F( D.)²) (F())D)))
|D,
DD,
(s,))
(s,))
[B r 1)2
B +
+ BAN - [0,
D.
IE
.8.
A4(N - -T + 1)2
A1
8B (B* + A4)
(N -r 1) -7 1)2.
A8
Therefore, let D = (B*(B++A)) we can say that Var Var rl Del D(N ((N-r+))(((----))) Next, A=
we show that for every two variance estimators Cov (Var Var D+)= 0ff S < $ N.
Cov Cov(Var Var D7) = [E[(Vrr Var., [Fr D2 Var Var,,
ED,, \ \d, IVVar.,, Var., [F, (eI)D.)))) Es., [(Var Var,, [F,)D|)]
ED,, 1D, [Var Var Dr])
0.
29

---

Therefore (when t < N),
(N - -
LURE Var Var 7
D+] =
T + 1) - -r) -r+ Var Var,,1)
(N - 1) -
(t- + + - r -r D(N - T -r++)
D(N () 1)
(1-++1)2
r)
There are two ways to bound the right hand side. When is far from N, we can bound with
D(N ()N - 1)
t
Var Var-URE
LURE D.]
Σ (N r)²
(+-++1)
D(N +)² -T+ 1)
t
1
(t (T-T + + 1)2
(N t)
D(N T + 1)a
--++1
When is close to N, by the fact that D x²
x² 6.
we have
Var LURLURE D.) s D(N - 1)3(N - 1 1)2
1
(
(N r
D(N - - 1)a
x2(N-1 t)
t-++1
6(t -
The above two inequalities still hold when 1 = N. where LURE will assign infinite weights on the exact variance Var,,N = Var so Var[Var- D+] 0. In summary,
see that Var [ D,]
E min [1, (N-1)²
[(-r+1] ) for E' T²D(N-rt1) Thus
1-7+I
Var |URF. Dr] S -- min (1, (-++))
B Details of variance estimation
B.1 The plug-in mean variance estimation
One practical gap is the incorporation of plug-in mean instead of the ground-truth mean in Eq. 6 show that the error introduced by this trick is controlled. First all, the variance of the plug-in mean could be controlled with the following lemma.
ALURE_ E a LLURE F,, its variance satisfies that there exists constant E Lemma 1. For FIt
Var | L:t
FLURE] < B(N-1)
Proof. By Prop. 3
C(N - TN - 11)
30

---

Then
Varl CaRR
(a (aLURE) 2 [ar[Fr]
(2 1)2
C(N =)( 1
PNN () -T 1)2
CN 1)2
(N - 7)
CN t)²
82
N(N - t)
CN(N- t)
Let E CN, then Var (ar &
ÉLURE
E(N-E)
This directly implies that the plug-in mean constructed from the LURE estimate will not be far from F(2).. Denote the variance estimate with the plug-in mean by
[(s) - Gre) (s)
(1))) ( f(stt) (st) (s())
Gr.1) )
Var,,
(s(s) sED,
Σ
We control the error with the following proposition.
Proposition 8. Given the same assumptions as Prop. 3 with at least probability 1 1- p. there exists
constant H dependent such that (Vrr,,t Var,,ll S H(N + 1) N-1+1
Proof. First, note that
Gr,- - F(O) (+) = F
ALURE F(2).
By Chebyshev's inequality, with at least probability P.
E(N - -t+1) p(t 1)
GG,,- - FF))
ÁLURE F(2) <
Also,
IG,, + F(2) D.)1 <
- F(2 - +F D,)
E(N - 1+
+2B(N - +1)
p(t - 1)
$ I(N +1,
31

---

where I is a constant dependent on We expand the difference directly.
Var,, - Var
Σ
(s(s) 7, (())) (f,) D,))
sED \D
(3)) (x(+t)
(G 7, PP0)
{(ار (G (
F(S)
(r((s)
Σ (r(s)] F(R) D.) + 2ʃ Gr.. + F(N) D.)l
+2 [()))
62 T,t
F(2) D.)²
Gr,t (())
(st)
((((t)
B(N - -
Σ
B
I(N + 1) + 2B
p(t 1)
(N T 1)A
RED \D,
(N - -1+1)
I(N - (N - -(+1)
AR IT
A
IB
- 8
B (N -1-111)
E(N - 1)
+
A
1(t
[I B2
E (N + 1/1/ p
N
A2
1-1
Let H B A
then [Var.
Var,,1| S -(
N-++1 1-1
Comparing the derivation of Prop. 7 and Prop. 8 the L1 error introduced by the plug-in mean goes to O at faster rate than the standard deviation of Var,,, indicating that the error is negligible in evaluation when t is large enough.
B.2
The streaming algorithm of variance estimation
2 has summation for each 1 S r S t, amounting to complexity of O(t)). In practice, we would reuse part of previous computation at step t to accelerate the computation step t. Observe that
Alg.
Var,.
Σ
(8))
f(s)
G+t)
2
(st)
f((t)
Gres)
2
(+))
(x()))
(81)
sED
DD,
f(s)2
+ (8() ( f(1))
Σ
+ (s) - - -2(())GT+)
Gra))
sED, DD (5(8)
(s()))
9 (8:)
= Σ
f(s)
62 T,
Σ (x) - 2G Σ ((s) + (3()
[((t)
Gr.z)
(x)
(e(8))
(r()))
SED, A.D D.
VD.
sED, D.
This is a polynomial with respect to our plug-in mean Gr.- Therefore, we could maintain the three summations in the coefficients E.ED, DD (=)
E.ED. \D, D, (+() and E.ED, AEDD,,,,AA [(s) for each during the algorithm to speed-up the computation of each individual Var,,- In addition, our final estimate is
f(a)²
Eter Br Var
Var, Σ B, Var,,r
Σ1=r Br
The numerator is still a polynomial with respect to the plug-in mean whose coefficients can be written as summations. We would further maintain the three coefficients and Ei= 3, to avoid the loop to compute Var,. With the two techniques, our approach is summarized as Alg.
32

---

Algorithm 3 Streaming variance estimation update
Require: Sample 8t, label f(x)) and estimate P1:c-1
I: Set 3ı = 8/2 = 2 = ar = b, = C = 2e =
2: for r= = 1, ...; L do
3: Get estimated mean Gr, F11-1 F()r)
4:
or + ar + 1:
x+ + f(a.))
5:
br t b, + 3t
Vr (s(())
f(te)
6:
C + c + 3t
2+
g=(se)
ac(ms)
7:
8:
Var, = (At 2h, Gr. + G2..)//
9:
I+ 3+ 4. (2t]
Ka,12
10:
yr y (8t)
2+ 2, + (3))
12: end for
C Experimental settings
C.1 Estimating birds in radar data
Weather radar preliminaries. The NEXRAD radar network contains 159 high-resolution radars and covers most of the U.S. territories. Each radar station typically scans the surrounding atmosphere every 4-10 minutes to collect weather data. This is achieved by rotating the radar antenna around the vertical axis at multiple elevation angles. At each elevation, the radar performs "sweeps" to collect several radar products, such as, reflectivity and radial velocity. These signals also record objects like bird roosts. We render radar data for 300km x 300km regions into 600 x 600 pixel arrays, on which we train a detector model to automatically predict bounding boxes for roosts.
Visualization. Fig. A2 (left) illustrates weather radar scans capturing bird roosts. Fig. A2 (right) visualizes the reflectivity data at 0.5o elevation of radar scan that happened at the KCL- weather radar station on August 19 2015 at 10:31:22 UTC, which is a randomly sampled radar scan that captures bird roosts. The circular and semicircular patterns are massive amounts of birds departing from their overnight roosting locations.
Detector, tracking, and filtering. We follow Perez et al. [31] to pretrain station-agnosio spa- tiotemporal roost detector. We also follow their configuration for deploying the detector on the Great Lakes radar stations, which are excluded from the pretraining data, assembling model predicted boundings boxes into roost tracks, and filtering tracks with too few detected time frames or too low of mean detection confidence score. Since each weather radar collects data every few minutes, the same roost is often captured by multiple consecutive scans of a radar station. To avoid double counting, it is necessary to assemble detections of the same roost into track.
Perez et al. 31 deploys the roost predition system on radar data from 12 radar stations in the Great Lakes region and let human experts screen the system predictions. They focus on the time window from 30 minutes to minutes after the local sunrise for every station-day.
Following Deng et al. [10), we focus on 11 stations, including KAPX, KBUF, KCLE, KDLH, KDTX, KGRB, KGRR, KLOT, KMKX, KTYX, KIWX, since KMQT does not clearly observe roosts in the human screened data. Our goal is to estimate the number of birds in June to October in the 2015-2019 five-year period at these 11 stations. We use screened data from Perez et al. [1 as the ground truth bounding boxes for estimating bird counts. We let our Sppcifc finetuned checkpoints predict bounding boxes for the same time periods on the same station-day for our estimations.
Station-day bird count estimation.
Per-sweep counts. For each predicted bounding box by the detector model in a radar sean in station-day, enumerate over the radar sweeps at all elevation within 5000km height and
33

---

follow Belotti et al. [S] to estimate the bird count of this bounding box geographical region in this sweep.
Per-bounding-box counts. We summarize, for each detection, radar sweeps taken across multiple elevations. In order to prevent double counting of birds in regions sampled twice by consecutive beams, we bin the sweep elevations into 1o bins. We then take the average of sweeps that fall in the same bin. Lastly, we sum counts across all bins.
Per-track counts. We obtain summaries across detections by finding the median count within each track and selecting two scans before and after the median. We then calculate the average count within this set of 5 scans.
Per-day counts. We sum over per-track counts to obtain a per-day bird count at a station-day,
C.2 Additional Finetuning And Sampling Details
To fine-tune our models, we use the Detectron2 library with modified configurations. Specifically, we disable learning rate warmup, set the batch size to 8, and set IILTEREEPPTY.. LEMPT__ANNOTATIONS ANNOTATIONS to False. For the high-resolution image experiments, we use the faster_renn, Laim___SS_ R_50___5F LPN__x.Naaaa configuration with the default ImageNet--retrrained weights.
Fine-tuning the image detector takes approximately 20 minutes on single NVIDIA A16 GPU. We perform fine-tuning times, resulting in a total end-to-end runtime of just under 3 hours per image. For the roost detector, each fine-tuning run takes about 2 hours on a single A16 GPU. We fine-tune 4 times, totaling roughly 8 hours of compute per station for roost counting. In both cases, the time required to compute count and variance estimates is negligible.
In practice, the measurements from the detectors could be as low as O. making the labeling of some units impossible and introducing huge variances to the estimation. our image counting tasks, we set g(x) +- max (g()), 1) to make sure that each tile can be sampled. In our radar counting tasks, we set g(s) + (s) + 1000 to cover every day in every station.
C.3 Labeling Birds in Images
We visualize the Reeds and Sky images in figure A1 images entirely. We use tile size of 200 pixels for sky and 160 pixels for reeds, resulting in a total of 925 and 1426 tiles respectively. To label we randomly group the tiles into batches of 50, and have labelers annotate each batch using VGG annotator. To speed up the process, we label each bird with single point at its center of mass. This results in about 1 second of human effort per labeled point. We only label a bird on the edge a majority of it is visible in the tile. In total, our ground truth count for sky is 5682 birds and 12486 birds for reeds.
To collect ground truth, we manually label both
To convert point annotations into approximate bounding boxes, we use a two-stage heuristic algorithm. First, we apply Otsu thresholding to each tile and compute the average size of a square bounding box by dividing the number of foreground pixels by the number of labeled points. In the second stage, we center these boxes at each labeled point, mask out foreground pixels outside the boxes, and recompute the average box size using the remaining foreground pixels.
Additional experimental results
D.1
Comparison to few-shot models
Figure A3 presents results terms of the Hellinger distance between the predicted proposal dis- tribution P and the optimal sampling distribution Q (proportional to ground truth counts). We use Hellinger distance because it accommodates croobbrbyyyylity entries. It is defined as:
k
((P,Q) = /2
Σ
(VD. - 2
Where Pt, di are the probabilities assigned to unit for and Q, respectively. Unlike raw counts, the Hellinger distance is unaffected by constant-factor over or under prediction and thus gives better idea of how good the model is for the estimation process.
34

---

Figure A1: The Sky image (left) and Reeds image (right) used for detection experiments. Note the resolution has been reduced for this visualization.
Bird roast
Weather radar
Figure A2: (Left) Weather radars detect bird roosts departing from their nighttime roosting locations. (Right) Roosts appear as circular patterns in images rendered from radar products, such as the reflectivity at 0.5% channel visualized.
the left, we compare the proposal distributions sky image raw predictions from our fine- tuned detector and the few-shot FamNet model (34), which estimates object counts from few examples without additional training. FamNet struggles to adapt with more examples, while the fine-tuned model achieves lower Hellinger distance even with single labeled tile.
Few-Shot Model Evaluation On Sky Image
Hellinger Distance Proposal Distributions
0.250
Reeds
0.14
Helliner Didtance
0.13
0.200
0.12
0.11
FamnNet
0.150
0.125
0.10
0.100
0.89
0.075
10 1.5
20
25
30
0.050
10
20
30
40
500
60
Lobeled Tiles
of Labeled Tiles
Figure A3; Left: Comparing the performance of raw predictions from fine-tuned detectors vs. few- shot FamNet(34) model, based on Hellinger distance to ground truth. The FamNet model performs worse and does not improve as much with more labeled tiles. Right: Hellinger distance between the raw predictions of the fine-tuned detector and the optimal sampling distribution for both images, averaged over 100 checkpoints. Unlike the raw counts which saturate quickly, additional training continues to improve the proposal distribution, especially on the harder Reeds image.
35

---

Reeds End End
Sky End End
fractional atror
DIS+WOR
DIS+AIS (end -o--odd)ddd
DI5+A15+WOR (end-to-end)
DIS+AIS+WOR
10a
10
10-2
10at
10-3
10-2
10-1
fraction laablled tile (1426 total)
fraction of labeled tiles (925 total)
Figure A4: Comparing our fixed checkpoint scheme against true end-to-end training, averaged over 1000 trials. While the fixed checkpoint approach introduces slightly higher variance, particularly at higher label counts, it preserves relative trends between methods. This supports its use for our large-scale evaluation.
Sky Image
Sky Image
Reeds Image
Reeds Image
erorl eeeo with COMB
1.2
INV (0.3)
INV (0.7) (0.9)
IMV (0.3)
(0.7)
1.1
(0.5)
(0V9
1.0
0.9
COMB
LURE
INV
(0.5)
LURE
INV (0.5)
0.8
100
200
300
400
100
300
100
200
300
400
200
400
Figure A5: Relative errors compared to a COMB weighting. Other fixed weighting strategies SQRT
CURE) are worse, but inverse variance weighting (denoted by INV (7)) may achieve lower error.
D.2 Fine- Line-Tuning Improves Proposal Distribution
we track the Hellinger distance over the course of finetuning, averaged over 100 checkpoints. While raw detector counts tend to saturate early, the Hellinger distance of the proposal distribution continues to improve, especially for the harder Reeds image, before eventually saturating.
On the right of Figure A3
D.3 True End-to-End Finetuning
To approximate the effect of interactive model adaptation in our main results, we "fixed checkpoint" approach: predefined sequence detectors trained on progressively larger number of labels. This allows us to conduct significantly larger number of trials than would be feasible with full end-to-end training.
In practice, however, a practitioner would retrain the detector as new samples are labeled during active measurement. To assess how well our "fixed checkpoint" approach approximates this more realistic scenario, we compare it to end-to-end training averaged over 1,000 trials (Figure A4).
The results indicate that performance under both approaches is similar, particularly when using fewer tiles, and that the relative trends between methods are preserved. We do observe slightly higher variance in the fixed checkpoint setting, likely due the specific checkpoint chosen.
D.4 Different weighting schemes for all experiments
We compare different weighting schemes in both image counting and radar counting experiments. The results are in Fig. AS and A6] From the first and the third columns, we conclude that the acomB weights work consistently better than assRT and dI.URE. the second and the fourth columns, performances of inverse variance weighting with different are compared. In general, a conservative = 0.3 brings little benefit, while an aggressive 7 = 0.9 can detrimental because each individual estimator may not be accurate enough.
36

---

D.5 Confidence intervals for all experiments
In Fig. A7 we compare the CI coverages for all experiments. most of the experiments, the coverage for active measurement improves as the number of labeled samples increases (except on KIWX and KTYX). This behavior is typical in importance sampling-bass -ased methods. Out of the 11 radar stations, we achieve near perfect coverage (around 0.95) in 6, good coverage (around 0.9) in and fair coverage (around 0.8) in 3. Comparing the two variance estimators Vari+ simp and Var coud
find that the Varie works better, especially stations with bad coverage. We conclude that the Cls in active measurement usually have the desired properties for measuring the uncertainty.
37

---

KAPX
KAPX
KBUF
KBUF
enor erfor with COMB
SORT
COMB
INV (0.3)
INV (0.7)
SQRT
COMB
INV (0.3)
NV (0.7)
1.2
LURE
INV (0.5)
INV (0.5)
INV.(G.9)
1.2
LURE
INV (0.5)
INV (0.5)
INV (0.9)
with
1.0
0.8
0.8
100
200
100
200
100
200
100
150
2000
KCLE
KCLE
KDLH
KDLH
enTO arror with COMB
1.4
(0.3)
(0.7)
errar arror with COMB
1.4
(0.3)
(0.
INV
(0.5)
INV
LURE
INV
INV
INV
1.0
1.0
0.6
0.6
100
200
50
100
150
200
100
200
100
150
200
KDTX
KDTX
KGRB
KGRB
erfor Farror with COMB
SQRT
COMB
INV (0.3)
INV (0.7)
tstar crror with COMB
SQRT
COMB
INV (0.3)
INV (0.7)
1.2
LURE
INV (G.5)
INV (0.5)
INV (0.9)
1.2
LURE
INV (0.5)
INV (0.5)
INV (0.9)
1.0
0.8
0.8
100
200
50
100
150
200
100
200
100
150
200
KGRR
KGRR
KIWX
KIWX
error amror With COMB
1.4
(0.3)
(0.7)
errar cerror with COMB
1.4
(c.
LURE
INV
(0.5)
INV
LURE
INV
INV
INV (0.9)
1.0
1.0
0.6
100
200
50
100
150
200
0.6
100
200
50
100
150
200
KLOT
KLOT
KMKX
KMKX
erorr enor with COMB
1.4
SORT
COMB
INV (0.3)
INV (0.7)
crtar error with COMB
SQRT
COMB
INV (0.3)
INV (0.
1.2
LURE
INV (0.5)
INV (0.5)
INV (0.9)
1.2
LURE
INV (0.5)
INV (0.5)
INV (0.9)
1.0
0.8
0.6
0.6
KTYX
KТYX
erfor ceror with COMB
1.4
(0.3)
1.2
WURE
INV (0.5)
INV (0.5)
INV (0.9)
1.0
0.8
0.6
100
200
50
100
150
200
Figure A6: Relative errors compared to a COMB weighting, for the roost counting problem on all 11 radar stations.
38

---

Sky Image
Sky Image
Reeds Image
Reeds Image
1.0
0.4
coverage
0.3
radius/gt
0.9
0.2
0.1
100
200
100
200
100
200
100
200
KAPX
KAPX
KBUF
KBUF
1.0
1.2
COvero
0.9
0.8
radius/gt
0.8
0.7
100
200
100
200
100
200
100
200
KCLE
KCLE
KDLH
KDLH
1.0
Goveraee
0.9
radius/gt
0.8
0.7
100
200
100
200
100
200
100
200
KDTX
KDTX
KGRB
KGRB
1.0
1.2
0.9
covera
0.8
radius/gt
0.8
0.6
100
200
100
200
100
200
100
200
KGRR
KGRR
KIWX
KIWX
1.0
1.2
0.9
coveros
radius/
0.8
0.4
0.6
0.0
100
200
100
200
100
200
100
200
KLOT
KLOT
KMKX
KMKX
1.0
1.2
0.9
coverad
0.8
radius/gt
0.8
0.4
0.6
0.0
100
200
100
200
100
200
100
200
KΤYX
1.0
1.2
DIS+AIS
%009
0.8
DIS+AIS+WORR Var
0.8
0.4
Vartens
coverage
100
200
100
200
0.0
radius/gt
Figure A7: Full coverage results for counting on the two images and on all radar stations.
39


Reviews:
[{"review_id": "zSJ8mRAfff", "reviewer": "Reviewer_QSbc", "summary": "This paper explores methods for active measurement (human in the loop) to improve precisions of measurements and provide unbiased MC error estimates in imperfect AI model.  The task is tested for counting birds in 2 different datasets and improved performance is demonstrated.", "questions": "It would be nice to understand when and why WOR vs AIS gives larger and smaller improvements in the dataset.  The relative contributions vary depending on the dataset.  It would also be nice to demonstrated this on another domain (e.g. cell counting?) instead of 2 similar domain applications.", "limitations": "Demonstrations on other domains to understand how the approach may vary would make the result more significant and applicable.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is clear and presents the approach for the biological domain.  It describes novel contributions beyond state of the art (DIS).  Active measurement is an under-explored area in the sciences and this study systematically studies augmentations to prior work.  \n\nIt would be nice to understand when and why WOR vs AIS gives larger and smaller improvements in the dataset.  The relative contributions vary depending on the dataset.  It would also be nice to demonstrated this on another domain (e.g. cell counting?) instead of 2 similar domain applications.", "was_revised": false, "final_justification": "", "rebuttal": "# WOR vs AIS Clarification\n*“It would be nice to understand when and why WOR vs AIS gives larger and smaller improvements in the dataset.”*\n\nIn theory, WOR consistently reduces estimation error regardless of the detector, with improvements becoming more pronounced  as more units are labeled. In contrast, AIS yields the greatest benefit when fine-tuning leads to substantial improvements in model predictions. Additional gains arise when our weighting strategy aligns with the (unknown) error rates. That said, our weighting scheme includes guarantees that bound the worst-case error, ensuring robustness even when this alignment is imperfect.\n\n# Domain Generality\n*“It would also be nice to demonstrate this on another domain (e.g., cell counting) instead of two similar domain applications.”*\n\nWe thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [1], and satellite images of the Palu Tsunami from the xBD dataset [2].\n\n**Fractional Error Rate On Malaria Cell Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.1946 | 0.1878  | 0.1932  | 0.1804       |\n| 3         | 0.1254 | 0.1105  | 0.1153  | 0.0996       |\n| 5         | 0.0955 | 0.0838  | 0.0850  | 0.0800       |\n| 10        | 0.0683 | 0.0612  | 0.0594  | 0.0558       |\n| 30        | 0.0407 | 0.0324  | 0.0344  | 0.0277       |\n| 50        | 0.0318 | 0.0239  | 0.0265  | 0.0173       |\n\nThe Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three  randomly selected cell images.\n\nLooking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled. \n\n**Fractional Error Rate On Building Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.9708 | 0.9708  | 0.9863  | 0.9233       |\n| 3         | 0.8707 | 0.8757  | 0.7920  | 0.8413       |\n| 5         | 0.7706 | 0.7804  | 0.6720  | 0.7185       |\n| 10        | 0.5653 | 0.5807  | 0.5517  | 0.5742       |\n| 30        | 0.3961 | 0.3530  | 0.3586  | 0.2931       |\n| 50        | 0.3025 | 0.1910  | 0.2707  | 0.1595       |\n\nFor damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.\n\nThe results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [3], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.\n\n\n[1] Ljosa et al., Nature Methods, 2012 \n\n[2] Gupta, R., et al, \"Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery,\" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.\n\n[3] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302."}, {"review_id": "poC8pGKTf9", "reviewer": "Reviewer_9oDa", "summary": "The paper focuses on efficiently measuring objects at scale. This problem is especially relevant for analyzing datasets in scientific domains. The paper introduces active measurement, a framework that uses a human-in-the-loop pipeline and machine learning models to assist with measurements. Collecting labels for a large number of instances is costly and time-consuming, making it difficult to obtain accurate measurements of items. Inspired by active testing and active learning, this framework iteratively gathers labels from humans and refines the estimates of the items. The work proposes new unbiased estimators, weighting schemes, and confidence intervals for improved estimates. The experiments show that they significantly reduce the estimation errors compared to the baselines.", "questions": "Please see weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths \n\n- The paper is well written. The paper does a great job of motivating the problem of estimating counts at scale.\n- The paper effectively addresses large-scale counting issues by developing new estimators, weighting methods, and confidence intervals for human-in-the-loop scenarios. \n- The paper makes strong theoretical contributions and provides comprehensive proofs for several propositions in the paper. \n- The experiment setup is also reasonable and fair. The experiments show that the proposed method consistently shows lower fractional errors compared to the baselines.\n\n### Weaknesses\n\n**Experiments**\nThe paper primarily focuses on scientific datasets related to bird counting. While they show strong results on these datasets, the paper would be strengthened if it included results on popular crowd counting datasets [a, b]. \n\n**LURE** \nThe paper improves over LURE weights for a pool-based setting with sampling without replacement. I would like to see if the improvements from Proposition 3 improve active learning frameworks as well. \n\n**Nit**\nLine 23-34: The authors have described the problem of measurement they focus on in detail. Could they cite a reference to prior work that also identifies the same problem? \n\n**References**\n\n[a] Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds. ECCV 2018. \n\n[b] Multi-Source Multi-Scale Counting in Extremely Dense Crowd Images. CVPR 2013.", "was_revised": true, "final_justification": "I have been in support of this paper. The authors have agreed to add additional references and experiments on additional datasets in the paper.", "rebuttal": "## Application to Active Learning\n*“Would like to see if Proposition 3 improvements help active learning frameworks.”*\n\nIn the context of active learning, our theoretical analyses mirror two types of biases: (1) bias due to sampling without replacement and (2) bias due to a varying acquisition function. With the biases, the samples are not acquired uniformly and should be weighted. There can be similar results as ours by formalizing the variances against the acquisition functions over the sample spaces. We think they may lead to novel weighting strategies that help active learning frameworks.\n\n## Missing Reference\n*“Could they cite a reference to prior work that identifies the same measurement problem?”*\n\nThanks for pointing out the additional references of counting problems above. We will discuss them in our next version. For this work, we mainly focus on counting problems that have biological and ecological implications. For example, there have been similar problems of counting penguins [1], bats [2], and other birds [3] in those domains.\n\n## Additional Datasets\n*“The paper would be strengthened if it included results on popular crowd counting datasets [ECCV 2018, CVPR 2013].”*\n\nWe thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [4], and satellite images of the Palu Tsunami from the xBD dataset [5].\n\n**Fractional Error Rate On Malaria Cell Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.1946 | 0.1878  | 0.1932  | 0.1804       |\n| 3         | 0.1254 | 0.1105  | 0.1153  | 0.0996       |\n| 5         | 0.0955 | 0.0838  | 0.0850  | 0.0800       |\n| 10        | 0.0683 | 0.0612  | 0.0594  | 0.0558       |\n| 30        | 0.0407 | 0.0324  | 0.0344  | 0.0277       |\n| 50        | 0.0318 | 0.0239  | 0.0265  | 0.0173       |\n\nThe Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three  randomly selected cell images.\n\nLooking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled. \n\n**Fractional Error Rate On Building Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.9708 | 0.9708  | 0.9863  | 0.9233       |\n| 3         | 0.8707 | 0.8757  | 0.7920  | 0.8413       |\n| 5         | 0.7706 | 0.7804  | 0.6720  | 0.7185       |\n| 10        | 0.5653 | 0.5807  | 0.5517  | 0.5742       |\n| 30        | 0.3961 | 0.3530  | 0.3586  | 0.2931       |\n| 50        | 0.3025 | 0.1910  | 0.2707  | 0.1595       |\n\nFor damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.\n\nThe results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [6], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.\n\n[1] Arteta, C., Lempitsky, V., & Zisserman, A. (2016, September). Counting in the wild. In European conference on computer vision (pp. 483-498). Cham: Springer International Publishing.\n\n[2] Horn, J. W., & Kunz, T. H. (2008). Analyzing NEXRAD doppler radar images to assess nightly dispersal patterns and population trends in Brazilian free-tailed bats (Tadarida brasiliensis). Integrative and Comparative Biology, 48(1), 24-39.\n\n[3] Buler, J. J., Randall, L. A., Fleskes, J. P., Barrow Jr, W. C., Bogart, T., & Kluver, D. (2012). Mapping wintering waterfowl distributions using weather surveillance radar. PloS one, 7(7), e41571.\n\n[4] Ljosa et al., Nature Methods, 2012 \n\n[5] Gupta, R., et al, \"Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery,\" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.\n\n[6] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302."}, {"review_id": "xDRryX77wE", "reviewer": "Reviewer_qcFR", "summary": "This paper address the problem of how to rapidly train a model with human input in the loop. The paper focuses on training a detector for the purposes of measuring a natural phenomenon, although it seems reasonable that the same technique could be used for other purposes. The approach builds on prior art where data are subsampled for labelling, but extends the prior art by focusing on sampling without replacement, compares different weighting methods in the context of sampling without replacement, and derives a new estimate of the variance. The performance is evaluated on two types of bird detectors.", "questions": "- Are the error bars on the right hand panel of Figure 1 ground truth error bars or predicted variances. How do they compare? Can you provide them for both domains?\n\n- Is there a reason not to incorporate a variance estimate from the AI model itself?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall I quite liked this paper. The problem is clear, the writing is clear and the contribution is clear.\n\nI have a few minor technical concerns, but I do not view these as serious problems. Firstly, the acquisition function assumes the estimates across the data set $f(s_t)$ for $s_t \\in \\Omega \\backslash D_t$ are independent. I note that the authors pay quite a lot of attention to the loss of independence between timesteps, but there is also correlation within the sample set $s_t$, e.g., correlation between different tiles in the image. In the active sensing domain (which is different to this domain) it is common practice to learn a Gaussian process that describes how sample sets within the time step might be correlated.\n\nSecondly, the variance estimates and the acquisition function as written do not incorporate an estimate of the variance from $f(s_t)$ directly. $f(s_t)$ provides only the estimated number of measurements, but it is relatively straightforward to modify $f(s_t)$ to return a variance as well, especially since line 8 of Algorithm 1 requires retraining the AI model. Incorporating the variance estimate from the AI model providing f(s_t) would allow, for instance, the model to avoid asking for $s_t$ estimates (line 7 of Algorithm 2) in regions where the signal is fundamentally ambiguous (e.g., a high noise part of the image).\n\nFinally, providing a variance estimate from the AI model and a GP model of correlations across the domain $\\Omega$ would allow the overall model to predict where measurements are most likely to improve. Asking for labels in a region that is high variance and strongly correlated with a labelled region that is also high variance would most likely be unhelpful. In the active learning setting (which is also related but different to this domain), it is most helpful to ask for labels where the learner performance is predicted to improve.\n\nThe experimental results are good, but are fairly specific. It would have been helpful to perform a similar analysis on a different type of data. I appreciated the error bars on the right hand panel of Figure 1, but it was not clear if these were ground truth error bars averaged over $\\Omega \\backslash D_t$ or if these were the predicted variances. I would also have liked to have seen these more carefully analysed in the experimental results section.\n\nThe limitations section was appropriate.\n\nI liked this paper and would argue for acceptance.", "was_revised": true, "final_justification": "I read the authors response, and I am still generally supportive of this paper.", "rebuttal": "## Clarification on Error Bars (Fig. 1)\n*“Are the error bars ground truth or predicted variances? Can you provide for both domains?”*\n\nThe error bars are 95% confidence intervals based on predicted variances. Since this figure is from a single trial (unlike the averaged results in the results section), ground truth variances are not available. Thank you for pointing this out, we will clarify it in the next version. More detailed results on predicted variances are included in Figure A7 of the supplemental.\n\n## Model Variance Incorporation\n*“Is there a reason not to incorporate a variance estimate from the AI model itself?”*\n\nWe want to clarify that we focus on the variance of the estimation, which is different from the variance of the AI model. There are two challenges to using the uncertainty of the AI model. First, we sometimes need more advanced methods like ensembling and Bayesian neural networks to estimate the uncertainty of an AI model. Our scope of work is generic but specific improvements can be introduced as extensions of this work. Second, it is indirect to link the variances of the model to the variances of estimation. The predicted measurements are normalized and lie in the denominators in estimation, which by itself introduces interesting but non-trivial statistical problems.\n\n## Sample Correlation Concerns\n*“There is correlation within the sample set, e.g., between tiles in an image; could consider a GP model.”*\n\nThese comments are very astute and touch on directions we are actively considering. A concrete example is the “reeds” image (Figure 1), where birds follow a clear spatial distribution.\n\nFirst, we would like to point out that this correlation does not invalidate any of our analysis. This is because we work within a finite sample framework, where all images and ground truth counts $f(s_t)$ are fixed and non-random. This differs from a typical ML setup, which assumes data are drawn from an underlying distribution. Philosophically, this is because we are _not_ interested in generalizing to future images from the same distribution—we only care about performance on the large but finite set of images. As a consequence, the only randomness in our estimator is sequence $s_t$ of sampled units and any randomness in the training of the AI model at each step. Our bias and variance analyses correctly handle these sources of randomness by considering the bias and variance at each step due to the sample $s_t$ conditional on all prior choices. \n\nThat said, considering correlation among images could be very useful. One promising opportunity is to better predict counts and thus improve proposal distributions to reduce the variance of active measurement. As an example, in the reeds image we have observed spatial auto-correlation in the residuals of the AI model. Fitting a GP can exploit this spatial structure to improve predictions for unlabeled tiles. \n\nAs you point out, a GP can also be used to model uncertainty in an active learning framework to help select samples that will improve the AI model as much as possible. This is an interesting direction for future work (we mentioned it very briefly in Lines 359–362.\n\n## Domain Generality\n*“Experimental results are good but fairly specific; would have been helpful to test on different data types.”*\n\nWe thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [1], and satellite images of the Palu Tsunami from the xBD dataset [2].\n\n**Fractional Error Rate On Malaria Cell Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.1946 | 0.1878  | 0.1932  | 0.1804       |\n| 3         | 0.1254 | 0.1105  | 0.1153  | 0.0996       |\n| 5         | 0.0955 | 0.0838  | 0.0850  | 0.0800       |\n| 10        | 0.0683 | 0.0612  | 0.0594  | 0.0558       |\n| 30        | 0.0407 | 0.0324  | 0.0344  | 0.0277       |\n| 50        | 0.0318 | 0.0239  | 0.0265  | 0.0173       |\n\nThe Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three  randomly selected cell images.\n\nLooking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled. \n\n**Fractional Error Rate On Building Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.9708 | 0.9708  | 0.9863  | 0.9233       |\n| 3         | 0.8707 | 0.8757  | 0.7920  | 0.8413       |\n| 5         | 0.7706 | 0.7804  | 0.6720  | 0.7185       |\n| 10        | 0.5653 | 0.5807  | 0.5517  | 0.5742       |\n| 30        | 0.3961 | 0.3530  | 0.3586  | 0.2931       |\n| 50        | 0.3025 | 0.1910  | 0.2707  | 0.1595       |\n\nFor damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.\n\nThe results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [3], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.\n\n\n[1] Ljosa et al., Nature Methods, 2012 \n\n[2] Gupta, R., et al, \"Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery,\" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.\n\n[3] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302."}, {"review_id": "rKru6Ou9te", "reviewer": "Reviewer_P6dZ", "summary": "This paper introduces an \"active measurement\" strategy, which aims to provide an iterative framework for improving a measurement task through improving the sampling strategy and efficiently involving humans in the loop.", "questions": "Please see \"Strengths and Weaknesses.\"", "limitations": "Yes, the authors briefly discuss the limitations of the current work in the Conclusion.\nHowever, the paper does not seem to explicitly discuss the potential negative societal impact in the same section, unlike how they responded to the questionnaire section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. This paper proposes an interesting scheme referred to as \"active measurement\", building on or borrowing inspirations from areas such as active testing, adaptive importance sampling (AIS), and prediction-powered inference.\nThe discussion the authors provide in Sec. 3 \"Related Work\" is interesting, but too brief to provide an adequate and useful context to the readers regarding the proposed active measurement scheme.\nThere should be a more comprehensive (and more in-depth) discussion to motivate and position the current study more clearly.\n\n2. A major concern regarding the current study is that the evaluations are very limited.\nCurrently, the authors consider only two counting tasks: (i) counting birds in two high-resolution images; and (ii) counting roosting birds in weather radar images.\nThere are many scientific datasets that the authors could utilize to strengthen their performance assessment.\nFor example, Cryo-EM particle picking and object counting from satellite images would provide good test cases.\n\n\n3. The study would benefit from a comparison with active learning based strategies, especially the ones that are uncertainty-based.\nConceptually, there are many similarities, and it would be beneficial to discuss their similarities and differences, and what specific advantages the active measurement strategy might offer.\nFurthermore, performance comparison with widely used active learning schemes in Figures 3-6 would be helpful.\n\n\n4. In the evaluation result (e.g., Fig. 3), it would be helpful to show the performance in terms of the # of labeled units, and not only the fraction of the labeled units.\n\n\n5. DISCount is an interesting and recent baseline, but to show the general applicability of the proposed active measurement scheme, the authors are strongly encouraged to consider additional baselines.\nIt would be important to see whether one can gain similar consistent improvements across different schemes.", "was_revised": true, "final_justification": "Please note that the overall rating has been increased after reviewing the authors' response to some of the questions/suggestions in the original review comments as well as the additional performance evaluation results they have provided in the rebuttal.", "rebuttal": "## Related Work Depth\n*“Related Work discussion is too brief to motivate/position the study clearly.”*\n\nAs another reviewer also suggests, we will expand our related work section to include more discussion of counting and active learning in the literature.\n\n## Comparison with Active Learning Strategies\n*“Comparison with uncertainty-based active learning would be beneficial.”*\n\nWe include a related comparison in Figure 6 to active testing, where the sampling strategy is proportional to the loss—a commonly used active learning heuristic. In this case, we do not observe much improvement in fine-tuning performance, and the mismatch with the estimation objective results in a noticeable drop in overall performance. We will also incorporate sampling based on other active learning criteria, such as uncertainty and expected model change, in the revision. Thank you for the suggestion. \n\n## Evaluation Metrics\n*“In Fig. 3, show # of labeled units, not only fraction.”*\n\nThanks for this feedback, we will update the plot to make the # of labeled units more obvious in the final version.\n\n## Additional Baselines\n*“Encouraged to consider additional baselines beyond DISCount.”*\n\nFor the counting problem, we tested two additional image-counting baselines: one motivated by prediction powered inference (PPI) and the other by active testing. Please see Figure 6 in the main text and Figure A8 in the supplemental material. As mentioned above, we plan to include comparisons with different active learning heuristics for selection in the revision.\n\n## Societal Impacts\n*“The paper does not explicitly discuss potential negative societal impacts”*\n\nThank you for this observation. We briefly touch on this in Section 8, noting that our method is problem-agnostic and aims to improve the accuracy of existing measurement workflows. That said, we agree it would be helpful to more clearly address potential societal impacts, and we will ensure this is clear in the next version.\n\n## Limited Evaluation Scope\n*“Only two counting tasks; consider using Cryo-EM, satellite images.”*\n\nWe thank the reviewers for their suggestions for additional domains to apply our work. In response, we performed experiments on two additional datasets: malaria infected cell counting and damaged building counting from satellite images. Specifically, we used image set BBBC041v1, available from the Broad Bioimage Benchmark Collection [1], and satellite images of the Palu Tsunami from the xBD dataset [2].\n\n**Fractional Error Rate On Malaria Cell Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.1946 | 0.1878  | 0.1932  | 0.1804       |\n| 3         | 0.1254 | 0.1105  | 0.1153  | 0.0996       |\n| 5         | 0.0955 | 0.0838  | 0.0850  | 0.0800       |\n| 10        | 0.0683 | 0.0612  | 0.0594  | 0.0558       |\n| 30        | 0.0407 | 0.0324  | 0.0344  | 0.0277       |\n| 50        | 0.0318 | 0.0239  | 0.0265  | 0.0173       |\n\nThe Malaria Cell dataset comprises 1,364 images (~80,000 cells). For this evaluation, we focus on counting the number of infected cells, which are around 5% of the dataset. Due to time constraints we do not perform any hyperparameter tuning, and use the same settings as our Sky and Reeds image experiments. Results are averaged over 500 trials. For our initial model we finetune the default Faster R-CNN network on three  randomly selected cell images.\n\nLooking at the first table, we see similar trends to the image and radar experiments. AIS reduces error fairly early on as the model improves. WOR also has a consistent improvement, with a larger impact after around 10% of the data is labeled. \n\n**Fractional Error Rate On Building Dataset**\n| % Labeled | DIS    | DIS+WOR | DIS+AIS | DIS+AIS+WOR |\n|-----------|--------|---------|---------|--------------|\n| 1         | 0.9708 | 0.9708  | 0.9863  | 0.9233       |\n| 3         | 0.8707 | 0.8757  | 0.7920  | 0.8413       |\n| 5         | 0.7706 | 0.7804  | 0.6720  | 0.7185       |\n| 10        | 0.5653 | 0.5807  | 0.5517  | 0.5742       |\n| 30        | 0.3961 | 0.3530  | 0.3586  | 0.2931       |\n| 50        | 0.3025 | 0.1910  | 0.2707  | 0.1595       |\n\nFor damaged building detection we focus on the Palu Tsunami subset of xBD which contains 113 satellite images of the shoreline affected by the tsunami. We count the number of damaged buildings, which corresponds to a label of “major-damage” or “destroyed”. Only the post disaster images are given to the model. We use the same hyperparameters as before, and results are averaged over 1,000 trials. The initial model is trained on five randomly selected images.\n\nThe results on this dataset follow similar trends: WOR provides the most benefit when a larger fraction is labeled, while AIS is more helpful early on. There is a bit more noise with this dataset, so we plan to average over more trials in the next version of the paper. We also note that our performance is lower than the numbers reported in DISCount [3], as they use a detector specifically trained for damaged building detection, whereas we use a simple ImageNet-pretrained backbone fine-tuned on just 5 satellite images.\n\n\n[1] Ljosa et al., Nature Methods, 2012 \n\n[2] Gupta, R., et al, \"Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery,\" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.\n\n[3] Perez, G., Maji, S., & Sheldon, D. (2024). DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20), 22294-22302."}]

Author rebuttal:

Meta-review: This paper introduces a human-in-the-loop framework that combines predictions with importance sampling and iterative human labeling. To get unbiased estimates for scientific measurements, the authors propose estimators, weighting schemes, and confidence intervals for sampling without replacement. While the reviewers appreciate the theoretical contributions and potential impact, I encourage the authors to have a more candid discussion of the method's practical limitations. For example, this could include i) computational overhead of iterative model fine-tuning, ii) when precision may still be insufficient for high-stakes applications, iii) methodological compromises in evaluation and algorithm design, e.g., fixed checkpoint approximation, and iv)  candid discussion of the scope of this work.
Decision: Accept (poster)