Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
Authors: Jannik Kossen
Sebastian Farquhar
Yarin Gal
Tom Rainforth
Venue: NeurIPS 2022 Accept (2022)
Keywords: active testing
sample-efficiency
model evaluation
active evaluation
active learning
bayesian active learning
experimental design

Abstract: We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks.


Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
Jannik Kossen
Sebastian Farquhar
Yarin Gal
Tom Rainforth²
OATML, Department of Computer Science, University of Oxford 2 Department of Statistics, University of Oxford DeepMind
Abstract
We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using surrogate- saroogae--based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of state-of-the-an of-the-art when applied challenging model evaluation problems for deep neural networks.
Introduction
Machine learning research routinely assumes access to large, labeled datasets for supervised learning. Practitioners often do not have that luxury. Instead, labeled data might be available initially, and label acquisition often involves significant cost: it might require medical experts to analyze x-ray images, physicists to perform experiments in particle accelerators, or chemists synthesize molecules. In these situations, we should carefully acquire only those labels that are most informative for the task at hand, maximizing performance while minimizing labeling cost [1, 62, 27, 43].
While a wide array of active learning methods [1, 62] have been developed to address the problem of training models when labels expensive, much less attention has been given to the problem of label-efficient model evaluation. Such model evaluation important, e.g. for performance analysis, model selection, hyperparameter optimization. Indeed, Lowell et al. [45] argue that lack of efficient approaches to model evaluation holds back adoption of active learning methods in practice.
Prior work in label-efficient model evaluation has focused on using importance sampling (IS) tech- niques to reduce the variance of Monte Carlo (MC) estimates [38, 59, 70]. particular, recent approaches [38, 18] have iteratively refined the proposal using information from previous evaluations, producing adaptive IS (AIS) estimators with state-of the-artt performance.
Though these methods provide substantial label-efficiency gains over naive sampling, they have some clear weaknesses that suggest improvements should be possible. Firstly, the AIS weights assigned to samples depend on the acquisition proposal used at that iteration. Thus, early samples are often inappropriately weighted, because we are yet learn an effective proposal. This can significantly increase the variance of the resulting estimator. Secondly, MC estimates can be inefficient at dealing with label noise: constraining the estimate to be a weighted sum of observed losses limiting when the losses themselves are noisy. Finally, IS methods can be restrictive on the acquisition strategies used to choose points to label: acquisition must be stochastic with known probabilities.
Coorrespnndence jannjk. kossen .x..u...c..kk
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To address these limitations, we introduce Active Surrogate Estimators (ASEs), a new approach to label-efficient model evaluation that is based on interpolation rather than MC estimation. Specifically, ASEs actively learn surrogate model to directly predict losses at all observed and unobserved points in the pool. Here, the surrogate can take the form of any appropriate machine learning model, and we introduce a general framework for learning effective surrogates. The interpolation-basee nature of ASEs leads to a variety of advantages over MC estimates: ASEs better accommodate noisy label observations, adapt quicker to new information, and provide complete flexibility in how new datapoints are acquired, allowing for careful customizations that are not appropriate in the AIS framework. To exploit this, we propose novel acquisition strategy, XWED, that tailors the acquisition towards points that will be most beneficial to the final estimation task.
To summarize, our main contributions are: (1) We introduce Active Surrogate Estimators; new method for lahel-efficient model evaluation based on actively learning a surrogate model ($3). (2) We introduce XWEDD, a novel acquisition strategy that targets a weighted disagreement that is highly relevant to our final surrogate-basee prediction ($4). (3) We provide theoretical insights into ASEs' errors ($5). (4) We apply ASEs to the evaluation of deep neural networks trained for image classification, where we consistently outperform the current state-of-the-ard ($7).
2 Label-Efficient Model Evaluation
We first describe the general setup of label-efficient model evaluation, also known as active testing [38]. We wish to evaluate the predictive model f : X + y, mapping inputs I E X to labels y E V. Here, model evaluation refers estimating the expected loss of the model predictions, that is, the risk
E (Y),
(I)
where we use capital letters for random variables, C is loss function, and the expectation is over the true test distribution, which can be different to the distribution of the training data for f. We make no assumptions on f or the way in which it has been trained. We only require that we can query its predictions at arbitrary x. Because we wish to estimate the risk of f. we keep f fixed and it does not change during evaluation. Everything here applies to both classification and regression tasks.
Following existing approaches, we study active testing in the pool-based setting. This is the typical scenario experienced in practice [62]: samples x, can often be cheaply collected in the form of unlabeled data, but the density for the data generating process itself is typically unknown. Concretely, one assumes the existence of pool of samples, with indices P = distribution, and for which we know input features x; but not labels yı, i P.
drawn from the test
If we knew the labels at all test pool indices P, we could compute the empirical test risk f = 1/N EiEp [xf(r:), y.). When N is large, this provides an accurate estimate of However, one often cannot afford to evaluate f: the cost associated with label acquisition is too high to acquire labels for all points in P. Instead, active testing methods must decide on subset O C P of points to label, where typically we have that M = 0 < [P] = N.
The simplest approach would be to sample indices uniformly at random. The resulting naive MC estimate, fia = 1/ad Eieo (x.().) yi), represents the predominant practice of using randomly constructed test set instead of actively evaluating models. However, the variance of fna is often unacceptably high when M is small. Methods for active testing seek to provide risk estimates with lower estimation error than fua, without increasing the number of label acquisitions M.
Similar to active learning, it is advantageous in active testing to not select all points O upfront, but instead to select them adaptively by using information from previous evaluations. The mechanism by which points are chosen is known as an acquisition strategy. As with active learning, it typical to assume that the cost of labeling is dominant, and all other costs incurred during model evaluation are typically ignored. Therefore, it can be worth (repeatedly) training surrogate models, despite the resulting expense, if this helps minimize the estimation error for given label budget M.
2.1 Importance Sampling-Base -Baed Active Testing
Importance sampling (IS) [30, 51] is a classic strategy for forming MC estimates with lower variance than naive MC. IS samples points from proposal X ~ q instead of the original reference distribution, which we will denote p. Adaptive importance sampling (AIS) [32, 48, 5] improves IS by refining the proposal g using information from previous iterations. However, traditional IS and AIS are only

---

applicable if we can evaluate the density of the reference distribution, at least up to normalization. This not the case in many relevant active testing scenarios, e.g. when z; are high-dimensionts images.
IS cannot be applied directly in the pool-based setting because of the need to sample without replacement. However, in the context of training objective for active learning, Farquhar et al. [18] introduced LURE, which converts IS to this pool-based setting. Specifically, they compute
1 Σ Um C (f( Vim)) Um = =+ N - M M m=1
LLURE
(2)
N
(N- + 1)
where g(mm) is the probability of choosing to label sample im at iteration Kossen et al. [38] showed that LURE can also be applicd to active testing, for which the optimal proposal is
q" (fm) & (I)(F(X), N,,Y = T.m]
(3)
In practice, 9º is intractable as the distribution Y|X = 2i,, is unknown. However, using ideas from AIS, one can adaptively improve q with information gathered from previous labels. In partic- ular, Kossen et al. [38] introduce strategies to directly approximate (3) (see $6). When g correctly identifies regions in the input where the loss is large, ÁLURE reduces variance compared to fua-
Unlike standard AIS, adaptive LURE conveniently does not require knowledge of an explicit density p(x))-a pool of samples suffices. However, most techniques for reducing the variance of the weights in the AIS literature (see e.g. Bugallo et al. [5]) cannot currently be transferred over to adaptive LURE. In particular, we cannot employ methods like deterministic mixture AIS [50] to update historic weights using future proposal adaptations. This means that poorly-calibrated early weights can persist and significantly increase the variance of the estimator.
3 Active Surrogate Estimators
We now introduce Active Surrogate Estimators (ASEs), our novel method for label-efficient model evaluation. Following the setup in $2, we assume a pool of unlabeled samples from which we iteratively choose points to label. Unlike existing methods that rely on MC estimates, ASEs use an interpolation- iherpolation-based based approach to estimate model risk: a surrogate model directly predicts losses for all points in the test pool. ASEs are further based around an adaptive and iterative procedure:
an active learning approach to carefully select the labels used to train our surrogate. We first discuss the form that the ASE and surrogate take, and later cover our novel acquisition strategy in $4.
3.1 Deriving the ASE
The fundamental idea of ASES is that of interpolation-basee estimation: we wish to build a risk estimator of the form i (Ø) = 1/N Eier g(2,; Φ). where g σ) is a flexible surrogate model with parameters ó that approximates the conditional expected loss, E [C(f(X), Y) = x²).
Though algorithmically quite different, the motivation behind this interpolation based approach is quite similar that of variational inference (VI) [74]: rather than relying on sample- based approx- imation for f, we are converting our estimation into the optimization of a functional approximation. Though, as with VI, this comes the cost of weaker theoretical asymptotic guarantees compared with MC estimation (see $5), it allows us to generalize more effectively when only a small number of labels available. This trade-off is typically worthwhile in the small sample size regime of active testing.
To decide what form g should take, we first note that the theoretically optimal g g--in terms of mean squared error (MSE) for r-is simply the expected loss above, analogously to the optimal proposal for LURE. While this is itself obviously intractable, we can learn an effective g by trying to approximate this optimal g". Though one could directly approximate this using generic regressor, such as a neural network, this would disregard that we already know the loss function L, can evaluate f. and know how both are combined to give the optimal g". In fact, the only unknown is the conditional expectation Y|X = xr for the test distribution.
We therefore introduce auxiliary model 77 T (z; Φ) of only this conditional. Combining t, S, and L, we construct y as g( d) = -y-x(-laxd)) [Z(((x), Y)), where the only learnable components of are the parameters of n, Φ. With introduced, we can finally write the ASE as
i (ASE(O) (0) 1 N E By (x (C((()))
(4)
3

---

where we emphasize that the expectation over Y|X unknown true label conditional. Note that the fact that the ASE includes an explicit model of the label noise is a decisive advantage over prior work (see $8). For classification, we can now directly evaluate the expectation over the labels as a weighted sum over the predictive probabilities E, π (yla Φ) (f(()=)))) For regression, foosed-form solutions of the expectation exist for some losses, and we can always compute unbiased and accurate MC estimates using cheap samples from и.
is now with respect to π and not the
Assembling g from L. f. and π will typically be advantageous modeling y directly: a direct model would need to learn not only about predictions of f. but also how they relate to the test distribution. In contrast, learning models x <yl|:; of conditional outcomes is standard procedure. This also makes it easy to include other prior information. For example, initializing π on the training data can sometimes give the ASE a useful estimate of the risk before observing any test data at all. Lastly, the modular construction of g allows us to quickly adapt the ASE to different models or loss functions.
Often, f. like π, will be probabilistic model of outcomes. However, it will generally be highly inappropriate choice for æ, such that it is important to always learn a separate auxiliary model. This is because we are essentially using decorrelate the errors of π from those of f by training a separate я. Further, we may want to choose from different model class than J, e.g. allow for meaningful incorporation of uncertainties. Lastly, while f is by construction fixed, we can adapt π during evaluation to reflect our updated knowledge of the test label distribution, we explain next.
predict the emrors of f. and we should therefore aim to
3.2 Adaptive Refinement of ASEs
Algorithm 1 Adaptive Refinement of ASEs
ASEs are adaptive: they iteratively improve their estimate by updating the surrogate new labels are acquired. A crucial compo- nent to them doing this effectively is that they actively select the labels to acquire at each iteration using an acquisition strategy.
Inputs:
Test pool {z; JEP, acquisition strategy a, tar- get model f. initial auxiliary model To
1: Initialize O = and U = P
2: for m to M do
3: Select im = a(Tm-1.. (RiJiEP, (utheo)
4: Acquire label y/m. Y Zt.
As such, they can be thought of as perform- ing estimation via active learning of the auxiliary model r, noting that the estimate they produce is, in turn, fully defined by this auxiliary model and the pool set.
5: Update O +- O im and l t- U \ im
6: Update auxiliary model T. (e.g. by retraining)
7:
end for
8: Return ÉASE as per (4) with π = TM
To define this adaptive refinement process more formally, denote with 2 and O the sets of unobserved and observed points in the test pool. These are initialized as l = P and O = 0. At each subsequent iteration of our adaptive refinement process, an acquisition strategy selects an index in E l of an unlabeled datapoint, for which we then acquire a label yi... We remove im from U, and add it O. Then, we obtain an improved ASE by retraining π, e.g. on the total observed test data { y:) : i E O}; this retraining does not necessarily need to be performed at every iteration if doing so would be computationally infeasible. A summary of this approach is given in Algorithm 1.
In SE, we show that the computational complexity of ASES is given by O (M (N + |Durin))) ))) assuming initial training set of size (Duaiuu for the surrogate. This is identical to the computational complexity of LURE- based active testing but more expensive than a naive MC estimate.
In ASEs, our acquisition strategy takes the current auxiliary model nm, the original model f, the pool of samples, and all previous evaluations as input, i.e. we have im a(#m-1: J, (ax (ax)ee. {vs heo). A good acquisition strategy selects points such that the expected ASE error after M iterations is minimal [28]. While ASES can be used with arbitrary acquisition strategies, we next introduce a novel acquisition strategy that we specifically design for use with ASEs.
4 The XWED Acquisition Function
It is important we tailor our acquisition strategy to the problem of active surrogate estimation, rather than apply more generic active learning approaches. We need to acquire labels such that we produce the best ASE estimate given a limited labeling budget. To do this, we propose novel acquisition function called the Expected Weighted Disagreement, or XWED for short, that designed specifically to target improvements of the ASE estimate itself.
4

---

In line with acquisition strategies in active learning, we now require that I takes the form T (z/2;; σ) Eewn(ss)) [ θ o)], where θ represents some model parameters we wish to learn about. In a BNN, θ is the weights and biases of the network, while corresponds to the parameters of the posterior approximation given by previous observations (such that contains all information from previous data). To avoid clutter, we omit Φ in the following presentation of acquisition functions.
For active learning, BALD [27] an established acquisition strategies that acquires points with high epistemic uncertainty [13, 34]. For a predictive model π as above, BALD can be calculated as the disagreement between the total predictive entropy and the expected entropies over posterior draws,
(ALD(x)) (a))
(m)
[- log
(Y|z)] (ewa(()) [y (Bxxa((-o,))) [- log (M,r, O]]]
(5)
However, BALD, and other acquisition functions from active learning, are ill-suited to ASEs because they do not consider the effect the uncertainties in π have on the ASE estimate. In particular, ASEs depend on the loss Lyy, [x)) of the original model f at the pool samples, but neither the form of the loss function C nor the original model f are considered in BALD acquisition.
Acquisition for ASEs should instead target points that are both informative about O and which we expect will contribute significantly our final risk estimate. The basis of XWED is therefore to weight the disagreement terms for particular inputs x using their corresponding loss L(Y, (z))::
XWED(x) = EYy (1(w)) CY, (Y[[²))
(6)
Be-r() [B (1)=0 |- -(Y, f=))) log (Y||,, @)]]
With XWED, we acquire points with large epistemic uncertainty [13, 34] in the auxiliary x about the outcomes only if they are relevant the estimate FASE- To convert this acquisition function to an acquisition strategy, we simply choose the point for which the XWED score is maximal: al a1 S, Jij {vi bEo) - arg maxigti (W:)D(;;). As we show in $7, acquisition from XWED allows for sigmificant improvements compared to BALD and other strategies.
XWED is compatible with a variety of model classes for π, such as Bayesian Neural Networks (BNNs) [46, 21] and deep ensembles [41]. In SD.2, we give a worked through example for the practical computation of XWED when the main model f is a neural network and the surrogate æ deep ensemble, both trained for classification. We emphasize that XWED is an acquisition strategy explicitly designed for active testing; it is not even defined in traditional active learning context, where the distinction between a fixed model f and the adaptive auxiliary model π does not exist.
5 Theoretical Analysis of the ASE Error
We next provide a theoretical analysis of ASEs to understand individual contributions to their error. Specifically, we will break down the errors of ASEs into (4) error from using a finite evaluation pool and (IT) error from imperfection of the surrogate, further breaking the latter down as (IIA) error from any limited expressivity in our surrogate class and (IIB) error from imperfect training.
To start, let be the final parameters obtained from the stochastic training procedure of the surrogate g. such that our learned surrogate is g(;; Here, & is a random variable with the stochasticity originating from (i) sampling the test pool itself, (ii) any stochasticity in the acquisition strategy, (ii)) randomness in the acquired labels, and (iv) stochasticity in the optimization process of the surrogate.
Next, let X, p(z) be a hypothetical additional input sample that is independent from the test pool. We can use this to define G(Ø) = E [g(Xo:)) as the true expectation with respect to the input distribution of a surrogate with parameters d. In turn, we can define o² argmin (G(Ø) - r| as the parameters which give the best approximation of the true expected loss EE [C(f(X). Y))X x]. Note here that 6* is not random as it depends only on the fixed function class of and the fixed true risk r.
Denoting our test pool as P = {X1.... XN }. where X; p(x) are independently sampled, we can write f (b,, P) to make the dependency of the ASE estimator (4) on P and explicit. By using the triangle inequality, we can then break down the error || @ P) r|] to the true model risk
| (b, P)- r1l = || (Φ. P) G(Φ) + G( G(Ø*) + G(d*) -rl
5 || Φ P) - G(+)
+ [[G()) r||
< AASE (T) (
- -G( -(+)) +[0))++((('))10)) -rl [|G - G(
(7)
5

---

where | || is any valid norm, and we note that G()) is independent of Xu and G(Ø²) is deterministic.
We now consider each of the terms in (7):
(7) = [ P) - -G( the error that originates from only evaluating the surrogate on finite set of pool of points. As per standard results for MC estimation, this will generally decrease as O(//NN,, such that it will be small for large test pools. Note here that fASE P) is typically not an unbiased estimator for G(@) as the set of input points available in the pool can influence the distribution of Φ. However, this effect will typically be very small and the bias will diminish as N increases. We can also, if desired, eliminate it completely by evaluating 9 over separate test pool than that used during the acquisition of test labels and updating of g. This variation on the standard ASE estimator can also be useful when the pool is impractically large for performing label acquisition, as we can evaluate the final learned g on a larger pool than used during its training.
( = [[G(+) r| represents the error from the imperfection of the surrogate. For the squared norm, it equals Eg (Exx (Xx; Φ)] - r) such that it is the expected squared difference (over surrogate parameters) between the surrogate's 's expectation over inputs and the true risk. It can itself be further broken down into terms (TTA) and BB as described below. Note that this term demonstrates the critical dependency of ASE's performance on the regressional performance of the surrogate itself.
(LIA) -G(oo) gives the error from limits in the choice of the surrogate function class. It measures the difference between g" and g(=: Whenever our surrogate function class contains the true expected loss, this term will be exactly zero. Even when this is not the case, it should generally be negligible for sensible choices of the surrogate function class. For example, if π is a deep neural network, the limiting factor will almost always be the training error from using finite data (i.e. term (IIB) below), rather than the theoretical expressivity of the network.
(ITB) = [|G[ G(Ø) ]| can be thought of as the error originating from the training of the surrogate: it is the norm of the difference between the true expectation of the surrogate we have learned and the true expectation of the best possible surrogate we could have learned.
5.1 Discussion and Empirical Analysis
In practice, (IIB) will typically dominate the ASE error. If term (1) is significant, the pool size is limiting factor, which can only be the case when our ASE estimation has been successful (assuming M < N), as its implies we have highly accurate estimate of the full empirical test risk f. If term (IIA) is significant, this means we are using an inappropriately weak model class for our surrogate.
To check these assertions hold in practice we empirically estimate (7) and (IIA) for the experiments in $7.1 at M = 1: we find that they respectively make up only 0.00013% and 0.031% of the full error; thus (IIB) is completely dominant here.
Further characterizing the dominating term (IIB) in the fully general setting is unfortunately not feasible: it equates to characterizing the error of a regression, which necessarily can only be done by considering particular class of surrogate. However, for a particular choice of surrogate, existing results from learning theory could be applied to further categorize its convergence. Here work on the convergence of models in active learning [23, 24, 58, 7, 544 will be particularly appropriate, noting that these all make specific assumptions about the model and acquisition strategy. More specifically, for Gaussian process surrogates and acquisition strategies from Bayesian quadrature, the results of Kanagawa & Hennig [31] could be applied. On the other hand, for surrogates based on deep learning architectures, one could use results from the convergence of stochastic gradient descent methods [47], which typically give rates of ((///T) under appropriate assumptions. These will require unbiased gradients in the optimization of the objective function, which we can achieve for ASEs with stochastic acquisition and the de- di-biasing scheme of Farquhar et al. [18]. In summary, for large pool sizes and appropriate surrogate classes, (LIB) will be the dominant contributor to the error, noting that the inequality in (7) becomes tight as one of the errors dominates. As such, the convergence of ASEs essentially reduces to the convergence of the surrogate, and thus, by extension, the convergence of the auxiliary model. Moreover, if we can ensure that this auxiliary model convergences to the true conditional output distribution as the number of label acquisitions M increases (this itself implies that our surrogate function class is sufficient powerful and so (ITA) = 0), we can thus ensure that the ASE estimator itself converges, noting that M - oo also predicates N + oo (as N: > M by construction), such that (4) + 0.

---

Related Work
Active Learning, Active learning (AL) [1. 62, 27, 21,
21, 61] allows for the training of supervised models in a label-efficient manner, and it is a particular instance of optimal experimental design (OED) [44, 6, 60, 20J. OED formalizes the idea of targeted acquisition to maximize the information gained with respect to target objective. Relatedly, transductive AL [71, 68, 36] performs AL of model predictions with respect known set of points. Filstroff et al. [19] propose AL for decision making.
Active Testing. Sawade et al. [39] explore IS for active risk estimation. They rely on vanilla IS which does not respect the pool-based setting [38]. Yilmaz et al. [70] have recently proposed active testing based on Poisson sampling. Both [70] and [59] outperform simple baselines; however, they use non-adaptive acquisitions that cannot adjust to test data.
Of particular relevance is the current ofate-of-the-art approach by Kossen et al. [38]. Building on the LURE estimator, they propose an AIS-style estimate of the model test risk ($2.1). To approximate the optimal acquisition (3), they, like us, introduce an auxiliary model π (ty) 6) that is (usually) adapted as test labels acquired. Given restrictions of the LURE estimator, they always sample from
g(im) x ((=3(()))))) [C(((x=....)))
for label acquisition and do not actively improve the proposal, e.g. through active learning.
[2, 33, 40, 29] explore stratification for efficient evaluation of classifiers: a simple, non-adaptive criterion is used to divide the test pool into strata, from which labels are selected uniformly at random. These approaches can be improved by applying the method considered here within the strata.
Surrogates. The use of surrogate models replace expensive black-box functions is pervasive throughout much of the sciences [53, 37, 10]. Bayesian optimisation (BO) [63, 4] and Bayesian quadrature (BQ) [49, 56, 3] are prominent examples of surrogate- baarodtte-baeedd approaches that actively select samples. BO iteratively finds the maximum of an unknown function, using surrogate model and ideas similar active learning to select sample locations. Unlike active learning, BO does not focus on a globally accurate surrogate. BQ obtains a Bayesian posterior for the integration of a Gaussian process (GP) [55] surrogate against a known data density p(r). Despite biased estimates, BQ can drastically outperform MC in low dimensions. However, BQ does not scale well to high dimensions, where, often, GP surrogates are not appropriate and explicit densities p(z) are unknown and hard to approximate.
Model Evaluation without Labels. Similar to our experiments in $7.3, [22, 57,9, 9, 11, 12, 64, 65] study the evaluation of model test performance without test labels or even without test sets. However, unlike ASEs, these works are based around performing meta analysis and require access to multiple related datasets (for which test labels are available). They rely on regressions from dataset/model summaries to performance scores and cannot be used in the setting considered in this paper, where only one dataset is available. Similarly, meta-learning for hyperparameter optimization, e.g. [15, 67, 16, 72, 26], interpolates test performance across variety of datasets and model hyperparameters.
7 Experimental Results
We next study the performance of ASES for active testing in comparison to relevant baselines. Concretely, we compare to naive MC and the current state-of-the-art LURE-based active testing approach by Kossen et al. [38]. As a general and common application, we consider the important problem of actively evaluating deep neural networks trained on high-dimensional image datasets.
We give full details on the experiments in the appendix. In particular, $B and $C contain additional results and figures, and $D gives further details on the computation of XWED and the baselines.
7.1 Evaluation with Distribution Shift
We first explore model evaluation in a challenging distribution shift scenario. Concretely, we given a fixed model trained on 2000 digits of MNIST [42], where all sevens have been removed from the training set. We then estimate the model risk on test with all classes present in their normal proportions. The test set has size N = 60000 but we can only afford to acquire labels for small subset. Such distribution shift settings are relevant in practice and challenging for active testing: both ASE- and LURE- active testing needs to approximate the expected loss, and this is particularly hard under distribution shift, where the initial æ has yet leam about the test distribution.
7

---

Following [38], we use radial Bayesian Neural Networks [17] for and k, with x initialized on the training set and regularly retrained dur- ing evaluation. For LURE, we acquire labels by sampling from (8). For the ASE, we use the
MC
Squared Error
10
LURE ElLoss
ASE XWED
10-3
XWED acquisition strategy (6). Both ASE and
LURE use the same auxiliary model class π, but
arrive at different network weights, ASSE and OLURE, because their distinct acquisition strate- gies lead to different test observations used for the re-training of x.
2000
4000
No Acquired Points
Figure I: ASE significantly improves over LURE and MC in terms of squared error for the distribu- tion shift experiment. Shown are mean errors and the shading two std. errors over 100 runs. We retrain at steps marked with grey lines.
Main Result. Figure shows the MSE for ASE and baselines when estimating the unknown test pool risk. Note here that we are careful to use the best possible setup of the LURE baseline to construct a challenging benchmark, as shown in the next section. We see that ASE improves on all baselines when used with the XWED acquisition strategy. Further, LURE can struggle with variance, with errors of individual runs occasionally spiking. This occurs when there is a significant mismatch between observed loss and proposal, e.g. when acquired point is ambiguous mislabeled. This does not happen for ASEs, as we do not weight true observations against predictions in the estimate.
7.2 Acquisition Strategies for ASE and LURE
We next perform an ablation to investigate different acquisition strategies besides XWED in the distribution shift scenario: We select points with maximal BALD score, Eq. (5). Further, acquire points by sampling from the expected loss; this is the exact same acquisition strategy used for LURE-based active testing, cf. (8). See $D.1 for further details on how we compute these baselines.
Figure 2 visualizes how often the acquisition functions acquire the missing class as testing progresses, and Figure (a) shows that ASE is able to outperform naive MC for all acquisition strategies, but there are noticeable variations in their performance. The XWED acquisition, that we have proposed specifically with ASES in mind, provides the best performance. BALD ac- quisition does not target those points that lead to the largest improvements the estimate, which
Frequency of Acquiring" 7
1.0
E[Loss]
XWED
0.5
BALD
MC
0.0
leads to much slower decrease in ASE error early on, though it does catch up with XWED later. Expected loss acquisition- wwhich is opti- mal for LURE- wiil sample the missing class (where loss is high) almost exclusively, which
0
500
1000
N Acquired Points
Figure 2: Proportion of acquired points which have
class '7', Results averaged over 100 runs.
(a)
(b)
10
MC
10
LURE E[Loss
Squared Error
ASE XWED
LURE XWED
ASE
LURE BALD
10-1
10
ASE
Loss)
10
10
0
2000
4000
2000
4000
No Acquired Points
N Acquired Points
Figure 3: Different acquisition strategies for ASES and LURE, (a) While XWED performs best, ASE outperforms MC for all investigated acquisition functions. (b) LURE suffers high variance for all acquisition strategies except expected loss. Shown are mean errors, shading is two std. errors over 100 runs. We retrain ı at steps marked with grey lines. The "~"-symbol indicates stochastic acquisition.
8

---

initially works well but later leads to π failing to improve on the other classes. In contrast, XWED focuses on the missing class early on out-once the loss is well characterized for the missing class- explores other areas as well. This behavior is preferable over the baselines, and we continue this discussion in $B.1.
In Figure (b), we explore how LURE behaves under different acquisition strategies, such as sampling from XWED or BALD scores. We find that LURE is noticeably more sensitive to the acquisition strategy than ASE: both XWED and BALD acquisition lead to high-variance estimates that no longer outperform the naive MC baseline. It seems one cannot expect acquisition strategies other than expected loss to perform well with LURE. In contrast, ASEs can directly apply any acquisition scheme without introducing such sensitivity because the acquisition does not affect which points are included in the estimate or how they are weighted.
We provide additional results in the appendix: $B.2 shows that adaptive improvement of the acquisi- tion function is necessary for label-efficient active testing in the distribution shift scenario, and, in $B.3, we study additional choices for ASE acquisition strategies (which do not outperform XWED).
7.3 Evaluation of CNNs for Classification
Next, we study active testing ResNets on more complex image classification datasets. Concretely, we study the efficient evaluation of a Resnet- 18 [25] on and of WideResNet [73] on CIFAR-100. In each case, we train the model be evaluated, f. on training set containing 40 000 points, and then use an evaluation pool of size N 2000. As ground truth, we compare to the empirical risk on a held out set of 20 000 points. For n, we use deep ensembles [41] of the original ResNet models. This experiment design is also used by [38].
[69] and CIFAR-10 [39].
In this experiment, there is no distribution shift between the training data for ʃ and the data on which we evaluate it. At the same time, the training set is large and, therefore, a lot of prior information is available to train the initial æ. Further, the number of test acquisitions is comparatively small (M 50) and unlikely to affect meaningfully, given the models have already seen 40 000 points from the training data. Indeed, we find that updating ı with this additional test data makes no noticeable difference to performance. To minimize computational costs, we therefore report results with fixed throughout this experiment, such that, here, is equal for ASE and LURE, constant æ leads to a con- stant LURE proposal (making the method more akin to IS than AIS) and constant prediction for ASE.
Figure 4 shows that ASEs perform effectively in this setting, significantly outperforming LURE and naive MC for all datasets. We also display the error from the finite pool size, by comparing the empirical risk of all N unobserved test points against our additional ground truth set and that ASE consistently performs similarly to this pool limit. This suggests that the pool size N may be the limit of the performance for ASEs here rather than the quality of the surrogate.
CIFAR-100
CIFAR-10
Difference to True Test Risk
2
LURE E[Loss]
0.5
0.25
ASE (coost.)
0
0.0
0.00
25
50
100
10
Squared Error
10-2
10-2
10
10
100
10
100
10
100
10
Acquired Points
Figure 4: Active Testing of ResNets on CIFAR-100. CIFAR-10, and FashionMNIST without retrain- ing the auxiliary model n. Despite not observing any test labels, ASE again improves upon LURE. We display medians over 1000 random test sets and 10/90 % (top) and 25/75 % (bottom) quantiles (top: too small to be visible for ASE). In grey, we display lower bound on the error from the finite pool.

---

This experiment produces a rather unusual result: as π is fixed throughout, the ASE actually does not make use of any information from the test labels! Its accurate estimates of the test risk are entirely due to information from the training data. Note that LURE here has access to the same и as ASE in addition to up to 50 test labels. Nevertheless, the surrogate prediction of the ASE at all pool points leads to much lower error than LURE estimate. This 'eero-shot' setting opens up interesting avenues for model evaluation when there are no test labels available at all.
However, in practice, one may not know if the test data follow the same distribution as the training data. Figure B.1 (a) shows that both LURE and ASE perform poorly when not retraining т in the distribution shift setting for any labeling budget M. We therefore generally suggest regular adaptation of π for both ASE- and LURE- active testing approaches. And even without distribution shift, the information from test labels will always eventually become relevant as M increases.
ASEs compatible with arbitrary loss functions, cf. (4). For example, for 0-1 1-I-Loss, ASEs can be used to estimate the accuracy of the main model. In $B.4, we demonstrate that ASEs continue to outperform all other baselines for the task of accuracy estimation.
In our theoretical analysis in $5, we argue that, in most practical settings, the error originating from the surrogate will dominate the ASE error. In $B.5, we reduce the size of the training set to 10 000. This makes the problem more challenging for ASEs because it reduces the quality of the surrogate model. Nevertheless, we find that ASEs maintain their significant advantage over the baselines.
8 Discussion
In the previous section, we have seen ASEs outperform MC-based alternatives in practice. Here, we discuss advantages of ASEs that are important to their practical success.
Actively Learning Estimators. ASEs are flexible about which active learning strategy can be used. In contrast, AlS-based strategies must balance competing goals: the acquisition strategy must both find informative data to improve the proposal in future iterations and provide importance-samplinn weights that reduce the estimator variance. For AIS, acquiring the most informative data with high probability can make the estimator worse because the resulting acquisition probabilities are a bad approximation for the optimal 9² Prior work using LURE does not address this trade-off and always acquires with the best approximation of q" available. ASE acquisitions can also be deterministic, while AIS requires full-support stochastic acquisition to avoid bias.
Surrogate Updates. For AIS, improving the proposal distribution only improves future acquisitions, but does not retrospectively improve weights attached to carlier observations. Bad initial acquisition weights have a lasting influence on the variance of AIS. In contrast, ASE makes better retrospective use of information: the iterative refinement of g leads to updated predictions at all pool points.
Noisy Labels. ASES have a further advantage when label distributions are noisy, i.e. observations y are stochastic even for the true label conditional x)) This is common in practice, because of ambigu- ous inputs near glass-boundaries mislabeled examples, or measurement error. ASES directly model the noisy outcome conditionals using If is accurate, the ASE error does not increase due to noisy labels, and the ASE can interpolate beyond the noise. contrast, AIS suffers badly: noisy observa- tions form a direct part of the estimate, leading to increased variance; while the acquisition (8) models the noise, this only affects weights, and noisy loss observations directly enter the estimate (2). Chen & Choe [8] provide detailed analysis of how stochastic observations affect IS estimators negatively.
9 Conclusions
We have introduced Active Surrogate Estimators (ASEs), a surrogate-based method for label-efficient model evaluation. The surrogate is actively learned, and we introduced the XWED acquisition function to directly account for the risk prediction task. Our experiments show that ASEs and XWED acquisition give state-of-the-ar performance for efficient evaluation of deep image classifiers.
Acknowledgments and Disclosure of Funding
We thank Andreas Kirsch and the anonymous reviewers for helpful feedback and interesting discus- sions that have led to numerous improvements of the paper. We acknowledge funding from the New College Ycotown Scholarship (JK) and the Oxford CDT in Cyber Security (SF).
10

---

References
[1] Atlas, L... Cohn, D., and Ladner, R. Training connectionist networks with queries and selective sampling. In Advances in Neural Information Processing Systems, volume pp. 566-573, 1990.
Bennett, P. N. and Carvalho, V. R. Online stratified sampling: evaluating classifiers web-scale. In International conference on Information and knowledge management, volume 19, pp. 1581-1584, 2010.
[3] Briol, F.-X., Oates. J.. Girolami, M., Osborne, M. A.. Sejdinovic, D. Probabilistic integration: role in statistical computation? Statistical Science, 34(1):1 2019.
[4] Brochu, E., Cora, M., and Freitas, N. tutorial on bayesian optimization expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv: 2010.
[5] Bugallo, M. F,, Elvira, V., Martino, L., Luengo, D., Miguez, J., and Djuric, P. M. Adaptive importance sampling: The past, the present, and the future. IEEE Signal Processing Magazine, 34(4):60-3 79, 2017.
[6] Chaloner, K. and Verdinelli, 1. Bayesian experimental design: A review. Statistical Science, pp. 273-304, 1995.
[7] Chaudhuri, K., Kakade, S. M., Netrapalli, P., and Sanghavi, S. Convergence rates of active learning for maximum likelihood estimation. Advances in Neural Information Processing Systems, 28, 2015.
[8] Chen, Y.-C. and Choe. Y. Importance sampling and its optimality for stochastic simulation models. Electronic Journal of Statistics, (00))336---4233 2019.
[9] Corneanu, C. A., Escalera, S., and Martinez, A. M. Computing the testing error without a testing set. In Conference on Computer Vision and Pattern Recognition, 2020.
[10] Cozad, A., Sahinidis, N. V., and Miller. D. C. Learning surrogate models for simulation- based optimization. AIChE Journal, 60(6):221 1-2227, 2014.
[11] Deng, W. and Zheng, L. Are labels always necessary for classifier accuracy evaluation? In Conference on Computer Vision and Pattern Recognition, 2021.
[12] Deng, W., Gould, and Zheng, L. What does rotation prediction tell about classifier accuracy under varying testing environments? In International Conference on Machine Learning, 2021.
[13] Der Kiureghian, and Ditlevsen, O. Aleatory or epistemic? does it matter? Structural safety, 31(2): 105-112, 2009.
[14] DeVries, and Taylor, G. W. Improved regularization convolutional neural networks with cutout. 2017.
[15] Eggensperger, K., Hutter. F., Hoos, H., and Leyton-Brown, Brown, K. Efficient benchmarking of hyperparameter optimizers via surrogates. In Proceedings of the AAAI Conference Artificial Intelligence, 2015.
[16] Eggensperger, K., Lindauer, M., Hoos, H. H., Hutter, and Leyton Leyton-Brown, K. Efficient benchmarking of algorithm configurators via model-based surrogates. Machine Learning, 2018.
[17] Farquhar, S., Osborne, M. A., and Gal, Y. Radial Bayesian neural networks: Beyond discrete support in large-scale Bayesian deep learning. In International Conference Artificial Intelligence and Statistics. volume 23, pp. 1352-1330 1362, 2020.
[18] Farquhar, S., Gal, Y., and Rainforth, T. On statistical bias in active learning: How and when to fix it. In International Conference on Learning Representations, 2021.
[19] Filstroff, L.., Sundin, Mikkola, P., Tiulpin, A., Kylmäoja, J., and Kaski, S. Targeted active learning for bayesian dacision-making. 10Xiv:2106.044 041004 2021.
[20] Foster, A., Ivanova, D. R., Malik, I., and Rainforth, Deep adaptive design: Amortizing sequential bayesian experimental design. In International Conference on Machine Learning, 2021.
[21] Gal, Y., Islam, R., and Ghahramani, Z. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1 192, 2017.
[22] Guerra, S. B., Prudêncio, R. B., and Ludermir, B. Predicting the performance of learning algorithms using support vector machines as meta-regressors. In International Conference on Artificial Neural Networks, 2008

---

[23] Hanneke, S. Adaptive rates of convergence in active learning. In COLT, 2009.
[24] Hanneke, S. Rates of convergence active learning. The Annals of Statistics, pp. 333-361, 2011.
[25] He, K., Zhang, X., Ren, S., Sun. Deep residual learning image recognition. Conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
[26] Hc, X., Zhao, K., and Chu, X. Automl: A survey of the state-of-the-art. Knowledge-Basse Systems, 2021.
[27] Houlsby, N., Huszár, F., Ghahramani, Z.., and Lengyel, M. Bayesian active learning for classification and preference learning. arXiv:1112.5745, 2011.
[28] Imberg, Jonasson, J., and Axelson-Fisk, M. Optimal sampling in unbiased active learning. Artificial Intelligence and Statistics, 23, 2020.
[29] Ji, D.,, Logan, R. Smyth, P., and Steyvers, M. Active bayesian assessment of black-box classifiers. In AAAI Conference on Artificial Intelligence, volume 35, pp. 7935-7944, 2021.
[30] Kahn, H. and Marshall, A. W. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1:263-278, 1953.
[31] Kanagawa, M. and Hennig, P. Convergence guarantees for adaptive hayesian quadrature methods. In Advances neural information processing systems, 2019.
[32] Karamchandani, A., Bjerager, P., and Comelll, Adaptive importance sampling. In Structural Safety and Reliability, pp. 855-862. ASCE, 1989.
[33] Katariya, N., Iyer. A., and Sarawagi, S. Active evaluation of classifiers large datasets. In International Conference on Data Mining, volume 12, pp. 329-338, 2012.
[34] Kendall, and Gal, Y. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? Advances In Neural Information Processing Systems, 2017.
[35] Kingma, D. P. and Ba, Adam: A method for stochastic optimization. arXiv:1412.6988 2014.
[36] Kirsch, A., Rainforth, T., and Gal, Y. Active learning under pool set distribution shift and noisy data. arXiv:2106.11713 2021.
[37] Kleijnen, J. P. Kriging metamodeling in simulation: A review. European journal of operational research, 192(3):707-714 2009.
[38] Kossen, J., Farquhar. S., Gal, Y., and Rainforth, T. Active testing: Sample-eflicient model evaluation. In International Conference on Machine Learning, 2021.
[39] Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images, 2009.
[40] Kumar, A. and Raj, B. Classifier risk estimation under limited labeling resources. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 3-15, 2018
[41] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pp. 1402-6413. 2017.
[42] LeCun, Y., Bottou, L., Bengio. Y., and Haffner, P. Gradient-baat based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
[43] LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 521(7553):436 2015.
[44] Lindley, D. V. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 986-1005, 1956.
[45] Lowell, D., Lipton, Z. C., and Wallace, B. C. Practical Obstacles to Deploying Active Learning. Empirical Methods in Natural Language Processing, November 2019,
[46] MacKay, J. A practical bayesian framework for backpropagation networks. Neural computation, 4(3): 472- 1992.
[47] Moulines, E. and Bach. F. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems, volume 24, 2011.
12

---

[48] Oh, M.-S. and Berger, J. O. Adaptive importance sampling in monte carlo integration. Journal of Statistical Computation and Simulation, 4163-4)1143-668, 1992.
[49] O'Hagan, A. Bayes-hermite quadrature. Journal of statistical planning and inference, 29(3):245-260, 1991.
[50] Owen, A. and Zhou, Y. Safe and effective importance sampling. Journal of the American Statistical Association, 95:135- 143,, 2000.
[51] Owen, A. B. Monte carlo theory, methods and examples, 2013.
[52] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang. E., DeVito, Z.,, Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, F., Fox, E., and Garnett, R.
Advances in Neural Information Processing Systems 32, pp. 8024 8035, 2019.
[53] Qian, Z., Seepersad, C. C., Joseph, V. R., Allen, J. K., and Jeff Wu, C. Building surrogate models based on detailed and approximate simulations. Journal of Mechanical Design, 128(4))668 2005 2005.
[54] Raj, A. and Bach, F. Convergence of uncertainty sampling for active learning. arXiv:2110.15784 2021.
[55] Rasmussen, C. E. Gaussian processes in machine learning. In Summer school on machine learning, pp-
63-71, 2003.
[56] Rasmussen, C. E. and Ghahramani, Z. Bayesian monte carlo. In Advances in neural information processing systems, pp. 505-512, 2003.
[57] Redyuk, S., Schelter, S., Rukat, T., Markl, V., and Biessmann, F. Learning to validate the predictions of black box machine learning models on unseen data. In Proceedings of the Workshop Comaa--I-tthe-aee Data Analytics, 2019.
[58] Sabato, S. and Munos, R. Active regression by stratification. Advances in Neural Information Processing Systems, 27, 2014.
[59] Sawade, C., Landwehr, N., Bickel, S., and Scheffer, T. Active risk estimation. In International Conference on Machine Learning, volume 27, pp. 951-958, 2010.
[60] Sebastiani, and Wynn, H. Maximum entropy sampling and optimal Bayesian experimental design.
Journal of the Royal Statistical Society: Series (Statistical Methodology), 62(1):145-117, 2000.
[61] Sener, and Savarese, S. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
[62] Settles, B. Active Learning Literature Survey. Machine Learning, 2010.
[63]
Shahriari, B., Swersky, K., Wang, Z., Adams, P., and De Freitas. N. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEFE, 104, 2015.
[64] Sun, X., Hou, Y., Li, H., and Zheng, L. Label-free model evaluation with semi-structured dataset representations. arxiv:2112.00694, 2021.
[65] Talagala, S., Li, F., and Kang, Y. Fformpp: Feature-hased forecast model performance prediction. International Journal of Forecasting, 2022.
[66] Van Rossum, G. and Drake, F. L. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
[67] Vanschoren, J. Meta-learning: A survey. arXiv:1810003548, 2018.
[68] Wang. C., Sun, S., and Grosse, Beyond marginal uncertainty: How accurately can bayesian regression models estimate posterior predictive correlations? In International Conference on Artificial Intelligence and Statistics, pp. 2476 27777, 2021.
[69] Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms, 2017.
[70] Yilmaz, E.. Hayes, P. Habib, R., Burgess, J., and Barber, D. Sample efficient model evaluation. 10X1:2109.12043. 2021.
13

---

[71] Yu, K., Bi, J., and Tresp, V. Active learning via transductive experimental design. In International Conference on Machine learning, volume 23, 2006.
[72] Yu, T. and Zhu, H. Hyper-parameter optimization: A review of algorithms and applications. arXiv:2003.0500 05689, 2020.
[73] Zagonuyko. and Komodakis, N. Wide residual networks. In British Machine Vision Conference, pp- 2016.
[74] Zhang, C., Bütepage, J., Kjellström, H., and Mandt, S. Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence, 41(8):2008 2026, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See $5. The ASE error is mainly
bounded by the error the surrogate regressor.
(c) Did you discuss any potential negative socictal impacts your work? [Yes] $F discusses this in detail.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Cf. $5,
(b) Did you include complete proofs of all theoretical results? [Yes] Cf. $5.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Full code for reproducing the experiments is linked to in the appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We give all training details in $7 and $D.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We repeat experiments and report error bars for all experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We report this in $G.
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Yes, we cite the creators of the datasets and code libraries use in SG.
(b) Did you mention the license of the assets? [Yes] Yes, mention licenses in $G.
(c) Did you include any assets either in the supplemental material URL? [Yes] We release code for ASEs in the supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Does not apply-
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Does not apply.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Does not apply.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Does not apply.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Does not apply.
14


Reviews:
[{"review_id": "qOpvOFYeB63", "reviewer": "Reviewer_Aog1", "summary": "This paper proposes a new method for active testing called Active Surrogate Estimators (ASEs). ASEs actively learn a surrogate model to predict the losses and also employ a novel acquisition strategy, called XWED, which is a modification of BALD but puts weights for each data point using its corresponding loss value. The experiments show the proposed method is better than other baselines such as MC and LURE.", "questions": "In Eq (6), why don't we just multiply a factor $\\mathbb{E}_{Y \\sim \\pi(\\cdot|x)} [\\mathcal{L}(Y, f(x))]$ directly into $\\mathrm{BALD}(x)$ in Eq (5) ? Is there any specific reason why we need to multiply $\\mathcal{L}(Y, f(x))$ into each of the expectations in Eq (6) ?", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem considered is important.\n- The proposed methods, ASEs and XWED, are novel and well-motivated.\n- The proposed methods achieve good performance in the experiments.\n- The paper is well-written, although the presentation can be further improved.\n\nWeaknesses:\n- The proposed methods are small modifications of previous methods. Particularly, ASE is a small modification of LURE, while XWED is a small modification of BALD. However, I think the proposed methods are still good contributions since these modifications help improve the performance of the methods in practice.\n- The paper does not give adequate descriptions for several baselines used in the experiments, e.g. LURE ~ $\\mathbb{E}$[Loss], LURE ~ XWED, LURE ~ BALD, ASE BALD, etc. It would be better to explain each of these methods using some formulas. That would help readers understand more about the ways different components of the methods can be used.\n- I think it would also be better for the paper to give some concrete examples where the proposed methods are applied and computed. For example, what would be the specific formulas when the model is a neural network, what would be the model $\\pi$, and how the computation is done? These examples can be given in reference to specific steps in Algorithm 1. I think Section 5 of the paper can be shorten for more space.\n- There is also a related research area that aims to evaluate models without any labels, e.g. see [1, 2] below and other related work. It would be better if the paper also compares their active testing algorithm with these methods.\n\nReferences:\n- [1] Corneanu et al. Computing the Testing Error without a Testing Set. CVPR 2020.\n- [2] Deng & Zheng. Are Labels Always Necessary for Classifier Accuracy Evaluation? CVPR 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "leGyYhcvvDm", "reviewer": "Reviewer_mCJr", "summary": "Extensive research is going on in an active learning area. However, the annotation cost of an evaluation set is overlooked in previous studies. Since active learning assumes expensive labeling costs, the cost of evaluation set annotation is also important. Active testing, the problem addressed in this paper, has been underexplored.\n\nThis work tackles how to minimize a labeling cost for an evaluation set. The paper proposes two main ideas: an estimator function that predicts the expected error of a task model, and an acquisition function to minimize the proposed expected error function. The paper experiments with its method on MNIST dataset with distribution shift scenario, and error estimation without distribution scenario on CIFAR10, CIFAR100, and F-MNIST.", "questions": "1. How is R_ASE sensitive to the accuracy of a surrogate model?\n2. What is the absolute scale of true losses in the experiments? How would the graph look like if graphs are scaled (normalized) over true loss?\n", "limitations": "The supplementary material addresses societal impact and limitations.\nHowever, the most important limitation, the dependency of R_ASE (equation.4) and surrogate model accuracy not analyzed nor addressed in this paper.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n### Novelty\nThe paper proposes a new error estimation function (Active surrogate estimator) and test-data acquisition function.\n\n### Experiment\nThe paper shows its performance with and without a distribution shift scenario, addressing the effect of the method on a scenario with a missing class scenario (an extreme case of class imbalance).\nSection 7.2 shows the performance on CIFAR10, CIFAR100, and F-MNIST which are sufficient to show their effectiveness in terms of experiment size.\nIn both with/without distribution shifts, the proposed method is meaningfully effective over the competing methods.\n\n### Analysis\nSection5 decomposes the error bound of an active surrogate estimator and interprets each term. The interpretation of these terms is meaningful and interesting.\n\n\n## Weakness\n\n### Weak novelty of surrogate estimator and its limitations\n1. weak novelty: Although the paper proposes two parts, an active surrogate estimator and acquisition function of active testing, the surrogate estimator (Equation (4)) idea is not surprising. In my simplified interpretation, equation (4) is a loss prediction of a task model using soft-pseudo labels, i.e., the prediction of a surrogate model. In other words, this objective heavily relies on a surrogate model in hoping that the surrogate model is very accurate as true ground truths. In addition, most of the performance gain of this paper is coming from this surrogate estimator, not the acquisition function (figure2 (a))\n\n2. Limitation: Due to the dependency of equation(4) on the surrogate model, I presume that the performance of the surrogate model is critical for equation (4) to perform well, which is an important limitation not analyzed nor addressed in this paper.\n\n### Experiment with strong task and surrogate models\nThis experiment setting may be due to the limitation of equation (4).\nIn the experiment, I conjecture that the authors use strong task and surrogate models.\n\n1. In section 7.1, the task and surrogate model are initially trained on 2000 data (roughly 3.5% of the MNIST training set. the Training set size of MNIST is 60,000). Typical classification models perform over 95% accuracy with 1% of the MNIST training set. This is not the weakness but I am pointing out that the task and surrogate model accuracy is high.\n\n2. In section 7.2, experiments are conducted on more challenging datasets. In these experiments, both task and surrogate models are trained using 40,000 samples. This is 80% of the training dataset for CIFAR10 and CIFAR 100 which has a training set size of 50,000 samples.\nIn a typical active learning setting, models are evaluated starting from 10% and additionally trained on an actively annotated labeled set of size 20-30%, summing up a total of 30-40% annotated data. I presume the models are trained on a larger set of samples (40,000 / 80%) than the typical setting (18,000~2,4000 / 30-40% ) due to the dependency of equation (4) on surrogate model performance.\n\nMissing analysis and limitation.\nSince surrogate model performance important factor in this study, this should be analyzed and addressed in this paper.\n\n### Missing details or experiment.\n1. Section 7.1 shows only 1 case for the distribution shift analysis. It would be better to show a few more cases for this study.\n2. The graphs in this paper show squared error, the difference between a true expected loss and predicted loss. What is the absolute size of the true expected loss? What would be the critical value for reasonable estimation error for loss expectation? What would the graph look like if you use the ratio between the expected loss and true loss?\n3. Typo? In Equation (7), ‘R’ should be ‘r’?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z3UpeDcg03", "reviewer": "Reviewer_qkQm", "summary": "This paper proposes the ASE, showing a better performance than the Lure approach in active testing. The newly proposed networks for the g and \\pi function and the acquisition function show better performance in the active test. Also, the various aspects of ASE are explored in the distribution shift.          \n", "questions": "I have some questions conserving the active test and algorithms. \n1. The approximation of risk or loss can be valuable. However, the minimum loss cannot ensure maximum accuracy in the classification task. In general, active learning and accuracy is essential. Let me know your answer. \n2. What's the primary reason for reducing the variance of active sampling in the AES. Updating procedures or architectures. Can you provide more solid answers?           \n3. Does the network's capacity for \\pi influence the performance of AES?   \n\n", "limitations": "There is no critical flaw in this paper. The section of theoretical aspects can be improved in the presentation. Experiments are limited, related the selection of \\pi.   ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong points: The proposed algorithm is intuitive and straightforward, performs better, and addresses the active test problem well. Experiments are well designed to show the critical properties of the proposed algorithm. \n\nWeak points: Theoretical analysis of the ASE error is somewhat vague. The decompositions and analysis are valuable. However, I, II, and III properties are not well-explained, including too massive contexts. It seems that the main is the empirical analysis due to the non-existence of any theorem,   \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "2x5sb_qBon", "reviewer": "Reviewer_DGAC", "summary": "The authors proposed a new method, ASE, for label-efficient model evaluation. ASE uses a surrogate-based estimation approach that interpolates the errors of points with unknown labels. ASEs actively learn the surrogate. The authors proposed a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. Experiments show that ASEs offer greater label-efficiency than SOTA. \n", "questions": "1) Could we use the difference between expected accuracy and current accuracy as a metric, besides squared error, as the metric to evaluate different algorithms? If so, how is the performance compared to other active learning algorithms, e.g., BALD and Entropy?\n2) Could we evaluate these algorithms in a batch mode setting? For each algorithm, we can specify the batch size $B$ and select top-$B$ in each round. We believe it is very interesting to extend XWED to batch mode setting, similar to the way in which BatchBALD extends BALD.", "limitations": "could the authors also discuss the computation complexity in the paper for different algorithms?", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1) The authors proposed a novel algorithm based on surrogate estimation for active testing.\n2) The authors maintain a high quality of writing and presentation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept