ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms
Authors: Huong Ha
Sunil Gupta
Santu Rana
Svetha Venkatesh
Venue:  (2021)
Keywords: active learning
bayesian learning
machine learning testing
information theory

Abstract: Machine learning models are being used extensively in many important areas, but there is no guarantee that a model will always perform well or as its developers intended. Understanding the correctness of a model is crucial to prevent potential failures that may have significant detrimental impact in critical application areas. In this paper, we propose a novel framework to efficiently test a machine learning model using only a small amount of labelled test data. The core idea is to efficiently estimate the metrics of interest for a model-under-test using Bayesian neural network. We develop a methodology to efficiently train the Bayesian neural network from the limited number of labelled data. We also devise an entropy-based sampling strategy to sample the data point such that the proposed framework can give accurate estimations for the metrics of interest. Finally, we conduct an extensive set of experiments to test various machine learning models for different types of metrics. Our experiments with multiple datasets show that given a testing budget, the estimation of the metrics by our method is significantly better compared to existing state-of-the-art approaches.

Under review as a conference paper at ICLR 2021
ALT-MAS: A DATA-EFFICIENT FRAMEWORK FOR ACTIVE TESTING OF MACHINE LEARNING ALGO- RITHMS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning models are being used extensively in many important areas, but there no guarantee that a model will always perform well or as its developers intended. Understanding the correctness of a model is crucial to prevent potential failures that may have significant detrimental impact in critical application areas. In this paper, we propose a novel framework to efficiently test machine learn- ing model using only a small amount of labelled test data. The core idea is to efficiently estimate the metrics of interest for model-under-test using Bayesian neural network. We develop a novel methodology to incorporate the information from the model-under test into the Bayesian neural network training process. We also devise an entropy-based sampling strategy to sample the data point such that the proposed framework can give accurate estimations for the metrics of interest. Finally, we conduct an extensive set of experiments to test various machine learn- ing models for different types of metrics. Our experiments with multiple datasets show that given testing budget, the estimation of the metrics by our method is significantly better compared existing of-te-of-thh-art approaches.
INTRODUCTION
Today, supervised machine learning models are employed across sectors to assist humans in making important decisions. Understanding the correctness of a model is thus crucial to avoid potential (and severe) failures. In practice, however, it is not always possible to accurately evaluate the model's ss cor- rectness using the held-out training data in the development process (Sawade et al. 2010)) Consider hospital that buys an automated medical image classification system. The supplier will provide performance assessment, but this evaluation may not hold in this new setting as the supplier and the hospital data distributions may differ. Similarly, an enterprise that develops a business predic- tion system might find that the performance changes significantly over time the input distribution shifts from the original training data. In these cases, the model performance needs to be re-evaluated the assessments provided from the supplier from the development process can be inaccurate. To accurately evaluate the model performance, new labelled data points from the deployment area needed. But the process of labelling is expensive as one would usually need a large number of test instances. Thus the open question is how to test the performance of a machine learning model (model-under--est)) with parsimonious use of labelled data from the deployment area.
This work focuses on addressing this challenge treating the model-under-test as a black-box as in common practice one only has access the model outputs. One previous approach aims estimate a risk score which is a function of the model-under-ted output and the ground- -tuth (akin to metric) using limited labelled data (Sawade et al. 2010). However, the approach has only been shown to be tractable for some specific risk functions (e.g. accuracy). Another approach in (Gopakumar cet al., (0188) suggested to search for the worst case model performance using limited labelled data, however, we posit that using worst case to assess the goodness of a model-under-test is an overkill because the worst case often just an outlier. Recently, (Schelter et al., (020) learns to validate the model without labelled data by generating a synthetic dataset representative of the deployment data. The restrictive assumption is that it requires domain experts to provide a set of data generators, a task usually infeasible in reality.

---

Under review as a conference paper at ICLR 2021
We propose scalable data-efficient framework that assess the performance of black-box model-under-test on any metric (that is applicable for black-box models) without prior knowledge from users. Furthermore, vation for evaluating one or multiple metrics is inspired by the current practice of users who need
framework can estimate multiple metrics simultaneously. The moti- assess the model-under-test a classification system, the user might want to solely check the overall accuracy or simultancously check the overall accuracy, macro-precision (recall) and/or the accuracies of some classes of interest.
varied aspects that important to them. For instance, for
To achieve sample efficiency, we formulate our testing framework as an active learning (AL) prob- lem (Cohn et al. 1996). First, small subset of the test dataset is labelled, and a surrogate model is learned from this subset to predict the ground truth of the unlabelled data points in the test dataset. Second, an acquisition function constructed to decide which data point the test dataset should be chosen for labelling. The data point selected by the acquisition is sent to an external oracle for la- belling, and then added the labelled set. The process is conducted iteratively until the labelling budget is depleted. The metrics of interest are then estimated using the learned surrogate model.
With this framework, one choice is to use a standard AL method to learn surrogate model that accurately predicts the labels of all the data points in the test dataset, however, this choice is not optimal. To efficiently estimate the metrics of interest, the surrogate model should not need to accu- rately predict the labels of all the data points; it only needs to accurately predict the labels of those data points that contribute significantly to the accuracy of the metric estimations. For our active test- ing framework, we first propose method train the surrogate model that can provide high metric estimation accuracy (using limited number of labelled data) by incorporating information from the model-under-test. Second, we derive an entropy-based acquisition function that can select the data points for whom labels should be acquired so as to enable maximal reduction in the estimation un- certainty of the metric of interest. We then use this computed entropy to generalize our framework to be able to work with multiple metrics. Finally, we demonstrate the efficacy of our proposed testing framework using various models-under-test and wide range of metric sets on different datasets. In summary, our main contributions are:
1. ALT-MAS, a data-efficient testing framework that can accurately estimate the performance of machine learning model:
2. A novel approach to train the BNN so as to accurately estimate the metrics of interest;
3. A novel sampling methodology so to estimate the metrics of interest efficiently; and,
4. Demonstration of the empirical effectiveness of our proposed machine learning testing framework on various poodels-under-eee for a wide range of metrics and different datasets.
2 PROBLEM FORMULATION AND BACKGROUND
2.1 PROBLEM FORMULATION
Let us assume we are given a black-box model-under-test A that gives the prediction A(x) for an input x, with A(x) E C = (1,..... ,,, C). Let us also assume we have access (i) an unlabelled test dataset X' = {., J.M and, (ii) an oracle that can provide the label for each input x in X. Given a set of performance metrics { Qx RNN x N - R. the goal is to efficiently estimate the values of these metrics, {Q;}K That is, we aim to estimate,
1K 1== when evaluating the model-under-tee A on the test dataset X.
O (Ax, k= --,,..,,K
(I)
with Ax = (A(x))zxx and Vx = (uz using the minimal number of oracle queries.
In this work, we focus on classifiers because they are common supervised learning models and also the target models of most machine learning testing papers (Zhang et al., 2019). Besides, it is also worth noting that, as we only have access to the outputs of the black-box classifier, the metrics {Qx} must be those that can be computed using solely the classifier outputs Ax and the ground- truth labels V/x of the test dataset X. Examples of Q& include the accuracy, error rate, pla-slass precision/recall, macro precision/recall, F3 score, etc. This to distinguish with the metrics that require information from the classifier internal structure such as the log-loss metric.
2

---

Under review as a conference paper at ICLR 2021
2.2 BACKGROUND
Bayesian neural networks (BNNs) are special neural networks that maintain a distribution over its parameters (MacKay 1992 Neal 1995). Specifically, given the training data Drr {2z, yi J a BNN can provide the posterior distribution p" (wi[tr)) with w being the neural network weights. In practice, performing exact inference obtain p" (w> (wr[)r) is generally intractable, hence we use a variational approximation technique to approximate this posterior. In particular, we employ the MC-dropout method (Gal & Ghahramani 2016) as it known to be both scalable and theoretically guaranteed in terms of inferring the true model posterior distribution p" (w( That is, the MC-
dropout method is equivalent to performing approximate variational inference to find a distribution in tractable family that minimizes the Kullback-Leibler divergence to the true model posterior.
3 ACTIVE TESTING WITH METRIC- SAMPLING STRATEGY
Our active testing framework is summarized as follows. First, a small subset X of the test dataset X' is labelled to construct a labelled set D; = (x,, Jx, where Vx, denotes the labels provided by the oracle for X;; and a BNN B., (with parameter w) is learned from D; to predict the labels of the unlabelled data points in X. Second, an acquisition function is constructed based on the BNN B the characteristies of the metric set, and the model-under-test outputs Ax to decide which data point is to be labelled so as to maximally reduce the uncertainty in the metric estimations. This data point is sent for labelling, and is added to the labelled set Dj. This process is conducted iteratively until the labelling budget is depleted. The metrics of interest are estimated using the BNN B
In this section, we propose method to train BNN that can give accurate metric estimations from a limited number of labelled data (Section (31). a method to estimate the metrics of interest given the BNN (Section 3.2). and method sample the most informative data point maximize the estimation accuracy of a specific metric (Section 3.3 or a set of metrics (Section 3.4.
3.1 BAYESIAN NEURAL NETWORK TRAINING METHODOLOGY
Given the labelled set Dj and the model-under-test outputs Ax. the goal is to train BNN B., such that the corresponding metric estimations are most accurate. Training BNN using solely the labelled set Dj might not result in accurate enough metric estimations. Thus, to improve the metric estimation accuracy, we propose to incorporate the information from the model-under-tes outputs Ax into the BNN training process. In particular, using the labelled set Dj, we also train a binary classifier C, that aims to predict the data points in the test dataset for which the model-under-test agrees with the ground-truth. Using the predictions by the classifier C,,, we then construct an augmented labelled sel S 1'ss,xx, } where Xs are all the data points in the test dataset X' that C, identifies the model-under test predictions are accurate, and Yx. are the corresponding model-under- teederr-esttte outputs of X's. The BNN is then trained using both the labelled set D; and the augmented labelled set St.
To train the binary classifier C,,, we first split the labelled set Dj into two parts: training and vali- dation, and then train C., on the training part whilst tuning the softmax probability threshold using the validation part so that C. achieves the highest precision on the validation part. This is because we want C. to choose a data point only when is most certain that the ground-truth and the model- under-test output of that data point is the same. Besides, as the precision of C,, is rarely 100%, thus, after obtaining the set of data points provided by C.p. we only take N₂ data points from this set with the highest softmax probability. The number N, is computed by multiplying the precision of C, on the validation part with the cardinality of the original predicted set. For example, if the preci- sion of the classifier C, is 50% the validation part and the original predicted set consisting of 100 data points, the final augmented labelled set S) only consists of 50 data points with the highest softmax probability. Finally, the binary classification problem can be imbalanced, particularly when the model-under-test is very accurate or very bad. Hence, when training C,, we employ the over- sampling technique (for the minority class) ensure the training data of the binary classification problem to be balanced, i.c. the cardinalities of the majority and minority classes are equal.
Remark 3.1. With this training methodology, the more accurate the model-une sedee--undr--est, the more accurate the BNN B.. In case when the model-under-test is bad, the augmented labelled set S, does not consist of many elements, thus, the BNN accuracy does not improve much compared to when training solely using the labelled set D, However, in this case, the BNN does not need to have high
3

---

Under review as a conference paper at ICLR 2021
accuracy in order to accurately estimate the metries. Specifically, for any data point for which the model-under-test disagrees with the ground truth, the BNN does not need to accurately predict its label. That is, even when the BNN predicts other labels (except the model-under-tes output label), the metric estimation still accurate (more detailed examples in Section C.3 of the appendix).
Remark 3.2. For simplicity, we suggest to set the architecture of the binary classifier and BNN to be same. For example, if the BNN is a 2-layer MLP, then the binary classifier is also a 2-layer MLP.
3.2 METRIC ESTIMATION WITH BAYESIAN NEURAL NETWORK
For metric Qk. given the model-under-tes outputs Ax and the labelled set D the true value O of this metric generally cannot be computed because Vx... (the corresponding labels of the unlabelled set Xwi is unknown. However, if the true posterior distribution p" (w) of the BNN is known, then Q can be computed as,
Qi = (xx, = Qx Dxx,, Vx..)) = Exx,, 1 ~p(x,, | (Qu (1xx Dx..Dl
(2)
|Q(Axx (Dxs. Dxmme)pp' p (m Di) dus,
where Vx., is random variable representing the possible labels of Xus. Vxuea denotes the predicted labels of Xu using the BNN weights w, i.e. Vxn. (izw) IEE and Ur.w arg max, ma... P(yn = cla.w), Vo E Nui- Substitute p" (w((D) by the MC-dropout variational distribution (D(w)))) (Di) (Gal & Ghahramani 2016), we can approximate Q as.
Qu f /Qx(x,, Dxx Dxanl)go (w) (D) dw EΣ M j=f Qa Dx,, D Vxno,l
(3)
where {0 are M stochastic forward passes from the distribution go(w).
Remark 3.3. worth noting that the value of metric Qs can also be estimated by computing (Ax, where Dx,, are the predictions of the whole test dataset X' using w. However, in this case, the estimation might be biased, i.e. the estimation might not converge to the true value of the metric when the labelled set is the whole test dataset. In fact, this estimation depends on the quality of the M stochastic forward passes {m²}- In contrast, with the approximation proposed in Eqs. (2) and (3). the estimation becomes increasingly less biased and when all the data points in the test dataset labelled, the estimation obtained by our method is equal the true metric value.
3.3 SAMPLING METHODOLOGY FOR A SINGLE METRIC
In the sequel, we use D {x] yx3 and X X X to denote the labelled and unlabelled set obtained after iteration t, respectively. We also use Bt, denote the BNN trained on Dj= and Vx; to denote the labels corresponding to the unlabelled set X For a metric Qk, at iteration t + 1, given the model-under-test outputs Ax and the labelled set D/. the goal of the sampling process is to select a data point x to label so as to maximally increase the metric estimation accuracy. This can be considered equivalent to sampling the data point .; such that knowing its label results in maximal uncertainty reduction for the metric estimation. To identify x; our key idea is to (i) evaluate how much the uncertainty of each data point's label contributes to the uncertainty of the metric estimation, and, (ii) sample the data point causing the highest uncertainty in the metric estimation.
To solve (i), we derive formulation that captures the relation between the uncertainty of data point's 's label and the uncertainty of the metric value. We construct this formulation based on Eq. (2). but we set the data point of interest (e.g. x) variable while fixing other data points (e.g. X (.) as their expected values. Specifically, let us denote Ún as the random variable representing all the possible labels of a data point x, then we define new random variable Q& (z) as follows,
Qx(x)) = ES -E D ~P A. (1, (Qx D'pp. &' " Yor 1e))), Vz E X
(4)
where Dxx,, \or denotes the random variable representing the possible labels of X \ conditioned on DJ_ This random variable represents the possible values of metric Q2 for each data point
DD includes the augmented data set generated using the proposed method in Secion 3.2
4

---

Under review as a conference paper at ICLR 2021
To solve (ii), formulate acquisition function that can suggest which data point that causes the highest uncertainty in the metric estimation. We define it as the mutual information (Houlsby et al. 2011
Gal et al. 2017) between Qx (z) and the BNN parameters w, i...
(x): DJ] H[ (c)) ar Dj) Bawp(w/D)) [H (x)),,
(5)
We use the mutual information as it is one of the most common criteria in deep Bayesian active learning to represent uncertainty (Gal et al. 2017). The first term of the acquisition function in Eq. (5) is the entropy of the metric estimation and the second term an expectation of the entropy of the metric estimation over the posterior of the model parameters. The data point that maximizes w/),, Dj] is the data point for which the model has many possible values for metric estima- tion, i.. the posterior draws have disagreement.
We can approximate [IQ Djl by using the posterior MC-dropout distribution qe (w) First, we show how to compute the first term on the right hand side of Eq. ( Since for each data point x, th is a discrete random variable with C' distinct values, so Q& (x) is also a discrete random variable with at most C' distinct values. Therefore, its entropy HQQx (z) |a,, DJ can be computed as,
(x) (z) = Di), (+) = E, QEQ p(Qu(=)= g/2, Dj) log (2x
(6)
where Q consists of all the possible values of Q (x) when Us E bound, p(Qx(z)= - u| Dj can be expressed as ~ p(u= = h/, Df) where Q (q) is the inverse function that maps the value of (x) to ds- Given M stochastic forward passes {w0 1 from (o( (x))r))) Df] can finally be approximated as,
By using the union
M
( - E,Ee (C
Σ M
j=1 p(ur = h121, (1,) 1,,))/M)
(0)
x log
Σ Ener (1)
j=1 p(ue = h/,, (. wr))\/M)
where (x(x) can be approximated as Qu(x) as EL (uwng(w(Ds) [Qx(Ax: Dx,+ D Dx Now Bel) P (2/ j=l Qk AVA, Dy. DxyAaa,, (el))/M, with Dxc,, tan denoting the predicted labels for X a given the parameter wy. The second term, BuwppaaD ( w]], also be approximated in similar fashion (detailed formula is in Section B.2 of the appendix).
Finally, the data point to be sampled is the one with the highest value of [2)x D/}, i.c.
i = argmax [x) Dj).
Remark 3.4. In the case argmax ut [[ (z, DH contains multiple elements, we select the data point that has the highest classification mutual information [[]0>; wil Dj). where li w Dj] is the standard mutual information for classification problems (BALD) (Gal et al. 2017
3.4 SAMPLING METHODOLOGY FOR MULTIPLE METRICS
{ at iteration + 1. the goal to sample the data point to maximally increase the estimation accuracy of all the metrics. As proposed in Section 3.3 for each metric Qk, we can sample the data point with the highest value of (2) Dj). Extending this idea to multiple metrics,
Given a set of multiple metries K
sample the data point with the highest sum of I[Qu (z); w/x D/},
को+ argmax
Σ
[1Qxxx): D/),
where D1] defined in Eq.
Our proposed framework Active Learning for Testing with Metric-Aware Sampling strategy, ALT- MAS, is summarized in Algorithm]
5

---

Under review as a conference paper at ICLR 2021
Algorithm The Proposed ALT-MAS Algorithm
1: Input: Model-under-tess AA, test dataset X' initial labelled set Dj = {2f, ,/1 (2 E X), the Bayesian neural network B labelling budget T, set of metrics interest {Qx 1 k=1
2: Output: The estimations of the metrics in {Qx] J
3: Initialize X = X xPP
4: for ===, 1, T do
5: /* Generate stochastic forward passes 2, 4a (w] #/
6: Train the binary classifier C using Di-1
7: Generate the augmented labelled set St using Ct-1
8:
Train the Bayesian neural network - from D and st-1
9:
Generate M stochastic forward passes { M from _5_ using MC-dropout
10: /* Compute the distribution random variables (x) for every x: */
11:
for k = 2... ..... K do
12:
For each x E X compute Q& (z) with all Vs {l...., )} using Eq. 4 and { )=
13:
For each x E 2 compute the set Qx that consists of all possible values of Qx (z) end for
15:
/* Find the data point that maximizes the mutual information function */ Σ (1) ( D formulated in Eq. 5
16:
Solve - argmax
Ask the oracle for label of x
Update Dj = DI U {at, }ey} and X X xz
19:
end for
Compute the metric estimations { using Eq. (3
3.5 THEORETICAL TIME COMPLEXITY OF THE PROPOSED ACQUISITION FUNCTIONS
Let us denote O(ax (2)) as the time complexity when computing metric Qs on test dataset X' (e.g. when Qk is the accuracy metric, as is 1). Then the time complexity of ALT-MAS (Eq. 8) is OCC2) + (ax + 2)MCC) where C is the number of classes of the number of Monte Carlo forward passes, and | is the cardinality of X. This time complexity is computed by splitting the computation of ALT-MAS into steps: (I) computing Qx(x) Vr E X when V {I, C.}- time complexity O(ag MC) (2) computing probability distribution of Qx(x) Vo X time complexity O + MCJ and, (3) computing the entropy time complexity ((MC) For set of K metrics, the time complexity of ALT-MAS (Eq. 9 is OKKC* + MCM ΣK 0g MC i.e. is linear in the number of instances in X.
M is the
EXPERIMENTAL RESULTS
We evaluate our active testing framework using various models-under-test and metric sets on the datasets MNIST and CIFAR10. Our experiments aim to answer:
1. Does our active testing framework perform better than traditional machine learning testing approaches? This checks whether testing framework based on AL necessary.
2. Does our active testing framework perform better than existing Al. approaches when es- timating metric values? This is show that suitable BNN training methodology and sampling approach that reduces the uncertainty of the metric estimation are needed.
3. Does our proposed sampling method work well for different sets of metrics and models- under-test? This is to ensure robustness across scenarios.
4. How is the performance of the BNNs? This checks whether a complex BNN is needed for our active testing framework to be successful.
To answer questions (1) and (2), we compare method ALT-MAS with two baselines: (i) Tradi- tion: the traditional method where the metrics are computed using their mathematical formula with all the labelled data up to the current iteration, and the labelled data is picked randomly from the whole test dataset; (ii) BALD: the state-of-the-a deep Bayesian AL method (Gal et al. (017). the metrics are estimated using the predicted label of the BNN trained with BALD. Note that here,

---

Under review as a conference paper at ICLR 2021
Precision_2 2
Recall 2
Accuracy
Acc Prech RecM
Act Prec Rec
30
30
30
30
30
Average relative (Percanaage
1.
20
10
15
80
160
80
160
80
180
Precision
Recall22 2
Accuracy
Acc: PrecM RocM
1o7 Acc--Prr---R
300
300
60
30
200
20
150
100
200
400
600
150
300
450
300
600
900
250
500
750
200
400
600
Precision_2 2
Recall
Accuracy
Acc Pee--PeeeM--cce RecM
Prec Rec
M-NNIST-ES
18-
18
20
16
12
15
10
200 400
600
750
250
500
750
0
0
150
300
4b0
150
300
Namber oracle queries
Number oracle
Number of onacle queries
Number of orscle queries
Number of oracle queries
Tradition
Figure I: Average relative error of the estimated metrics, for each combination of model-under-test (M-MNIST, M-FashionMNISTT & M-MNIST-ES) and metric set. Curves are smoothed over a sliding window of 10 oracle queries (iterations). Plotting mean and standard error over 3 repetitions. Better methods are those that converge faster zero (Best seen in color).
don't 't compare with the traditional AL methods as has been shown that deep Bayesian AL methods outperform traditional AL methods (Settles) 2010: Gal et al. 2017).
To answer questions (3) and (4), for each dataset, we evaluate our proposed machine learning testing framework using various types of models-under-ter and metric sets. Since our testing framework depends solely on the model-under-tes, outputs, we evaluate our proposed framework with different models-under-test that have different levels of accuracy (e.g. good, average, and bad). This is to simulate the reality when the user receives black-box model-under-test and they want to know that model-unnerrtest is good or bad using their criteria (i.e. metrics). The metric sets consist of either a single metric or multiple metrics, and the metrics are of different types: per-class metrics and overall metrics. For each combination of dataset, model-under-test and metric set, we repeat the experiment 3 times. Finally, we investigate the prediction performance of the BNNs used in all these experiments. All the experiments Tesla V100 SXM2 32GB GPUs. All the source codes are implemented in Tensorflow 1.15.0 and will be publicly available after the acceptance of this paper.
running on multiple servers where each server has multiple
4.1 MNIST DATASET
We aim to estimate the performance of various models-under-test using different metric sets on the MNIST test dataset (LeCun et al. 2010). Our experiment setup as follows. Firstly, we use a standard convolutional neural network (CNN) and then train it on the train MNIST dataset to construct a good model-under-test (M-MNIST). Secondly, we either reduce the number of training epochs, retrain the CNN to construct average models-under-test (M-MNIST-ES), Thirdly, the CNN on the train FashionMNIST dataset (Xiao et al. 2017) (completely different from MNIST) to generate bad model-under-tes IM-FahhionMNISTT Details of these models are in Section C.I of the appendix. For each model-underr-est, we aim to estimate various sets of common metrics: i) 3 metric sets containing solely one metric in each set: precision of class recall of class 2 and overall accuracy, and, ii) 2 metric sets consisting of multiple metrics: a set of 3 metrics (aceuracy, macro-precision, macro-recall), and a set of 21 metrics (aceuracy, precision and recall of each class). The BNN and binary classifier are mutilayer perceptrons (MLP) with 2 layers and dropout in each layer (details are in Section C.I of the appendix).
train
In Figure [ we report the average relative errors of the estimated metrics for each combination of model-underr-ess and metric set. First, we can see that the method Tradition not consistent.
The model is implemented in tensorflow but use the official network architecture published on keras.com for MNIST dataset, as there is no official source code on tensorlow.org for MNIST dataset. Class is random choice. Based experiments, ALT--MS outperforms baselines any class.

---

Under review as a conference paper at ICLR 2021
M-CFAR1O
80
Precision 2
Recall 2
Accuracy
Acc-PrecM:BBecM
60
Acc- Prec Rec
40
40
Average relative error (Percentage)
20
20
20
20
0
250
500
750
1200
400
0
400
800
250
500
750
Preciston
Recall
Acc-PrecM-Reck
Acc-Prec-Rec
MSVHN
-6
24
24
75
16
50
25
4000
800 1200
500
1000
1500
0
500
1000
1580
500
1000
1500
400
800
1200
Precision_2 2
Recail 2
Accuracy
Acc-PrecM-RRcM
Acc-Prec-Rec
IN-CIFARID.ES
8.8
15
12
25
8
6
15
250
500 1000 1500
400
500
1000
1500
250
500
Number of oracle queries
Number of onacle queries
Number oracle queries
Number of oracle quenes
Number oracle quenees
BALD
Tredition
ALT-MAS
Figure 2: Average relative error of the estimated metrics, for each combination of model-under-tes (M-CIFARIO, M-SVHN, & M-CIFARIO-FEN (0-ES) and metric set. Curves are smoothed over a sliding window of 10 oracle queries (iterations). Plotting mean and standard error over repetitions. Better methods are those that converge faster to zero (Best seen in color).
Tradition performs very well on M-MNIST but performs badly on M-FashionMNIST. Based on our observation, Tradition performs well on M-MNIST because this model has very high accuracy (99.1%)) which means its outputs and the true labels are very similar. Thus, by randomly picking data points, acquire their labels, and predict all the metric values by the mathematical formula on the current labelled set will provide the metric estimations to be approximately 100%, which are similar to the true values of the metrics. However, for other moodlss-unerr-tee with different accuracies, this technique will not work. Now, we compare the performance between ALT-MAS and BALD. For most of the combinations of model-under-test and metric set, our method ALT-MAS outperforms BALD by high margin, especially on per-class metric and M-MNIST. This is understandable as i) M-MNIST is a good model and thus the BNN trained with ALT-MAS has much higher accuracy compared to that with BALD, and, ii) the optimal sampling strategy achieve accurate estimations for per-class metrics is very different to the optimal sampling strategy to obtain BNN with high accuracy. This clearly demonstrates the superiority of our BNN training and our optimal sampling methodologies. In particular, we can that the BNN benefits the information from the model- under-test, and, the optimal sampling strategy to achieve accurate metric estimation should be the one that tailors to the characteristics of the models-under-tess and the metrics.
4.2 CIFARIO DATASET
We now consider the CIFARIO test dataset (Krizhevsky) 2009). We use the same strategy in Section 4. to generate three different models-under-test: M-CIFARIO (trained on train CIFARIO) with high accuracy, M-CIFARIO-ES (trained on train CIFARIO with early stopping) with average accuracy, and M-SVHN (trained on SVHN a different dataset) with low accuracy. We use the same 5 sets of metrics in Section 4.1 The BNN we in this case is the standard LeNet model (Lecun et al. 1998) with dropout applied before the relu layers. More details about the experiment setup in Section C.2 of the appendix. In Figure 2, we report the average relative errors of the estimated metrics of several combinations of model-under-test and metric set. Similar to the observations on MNIST dataset, the method Tradition is not consistent, i.e. it performs well on the good model-under-teed but performs badly on other models-under-test BALD also performs well on some scenarios but does not perform well when the model-under-tes is good or when the metric is per-class metric. In contrast, ALT-MAS performs well on all of the scenarios.
4.3 THE QUALITY OF THE BNNS
In Section of the appendix, we investigate the performance of the BNNs in the active testing compare the classification accuracies of the BNNs
methods (ALT-MAS& & BALD). In particular, in BALD and those in ALT-MAS (with our augmented data training strategy and optimal sampling strategy). First, we that, for most of the models-under-test and metric sets, the BNNs does not.
8

---

Under review as a conference paper at ICLR 2021
need to have high accuracy for the metric estimation to be accurate. Second, due to augmented data training strategy, the prediction accuracies of the BNNs improve significantly, especially when the wodels-under-tes. are good. Lastly, due proposed sampling strategy, the BNN only needs to learn the labels of the data points contributing significantly to the metric estimation in order to achieve accurate metric estimation. More discussions are in Section C.3 of the appendix.
RELATED WORK
Trustworthy Machine Learning. This line of works aims to assess the confidence of deep neural
networks in making predictions on test data points (Plati, 1999| Zadrozny & Elkan [2002] Niculescu-
Mizil Caruana 2005; Guo et al. 2017 Gal Ghahramani 2016 Lakshminarayanan al. | 2017
Hendrycks & Gimpel 2017 Jiang al. 2018 Corbière al.) 2019). For example, (Gal & Ghahra
mani 2016) and Lakshminarayanaal et al. (0117) estimate the uncertainty of deep neural networks
via Bayesian methods so to return distribution over the predictions. The work in Hendrycks &
Gimpel (017) uses the softmax probability of the network to detect if a test data point is misclas- sified out-of-distrbbution. A new trust score to understand whether classifier's prediction for test data point can be trusted or not is proposed in Jiang et al. (2018). Most recently, (Corbière et al. (2019) assess the confidence of model by proposing new target criterion for model confidence based on the True Class Probability. These methods rely on the training data and/or internal archi- tecture of the network to generate the model confidence score. Our method, in contrast, assumes an already trained and black-box model; our goal is to estimate the performance of the model on various metrics on new lest dataset.
Machine Learning Testing, A comprehensive review of machine learning testing methods can be found in (Zhang et al. 2019). Some recent notable works include (Tian et al. 2018 Sun al. 2018 Zhou et al. 2019). All of these works are based on white-box testing. i.e. the testing framework uses information about the internal structure the model. Our work black-box testing, i.e. our method assesses the performance of the model based solely its outputs. The closest related work to ours are (Sawade et al.) 2010 Schelter et al. 2020 Gopakumar et al. 2018) these methods, their limitations, and how overcome them discussed in Section 1.
Active Learning. These methods aim to train a machine learning model in a data-efficient way by selecting the most informative data points for which labels should be acquired (Settles) 2010 Blundell et al.) 22155) Gal et al., 2017). These AL methods aim to train a model to predict the labels of new data points accurately whilst method aims to train model to estimate specified metric set accurately. We have shown experimentally that for the active testing framework, our proposed method outperforms existing AL methods.
6 CONCLUSION
We propose a novel approach to efficiently evaluate the performance black-box machine learning models. The core idea is to efficiently estimate important metrics of the model being tested based on Bayesian neural network. We develop a novel method for training the BNN to achieve accurate metric estimations. We also devise novel entropy- based sampling strategy to sample data point such that the proposed framework can accurately estimate the metrics of interest simultaneously using minimal number of labelled data. Experimental results show that proposed approach works efficiently for estimating multiple metrics using diverse models and datasets.
REFERENCES
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML), pp. 1613- 1622, 2015.
D.A. Cohn, Z. Ghahramani, and M.I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4(1): 1996.

---

Under review as a conference paper at ICLR 2021
C. Corbière, N. THOME, A. Bar-Hen, M. Cord, and P. Pérez. Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
pp.
Y. Gal and Z. Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning- In Proceedings of The 33rd International Conference on Machine Learning (ICML 2016). pp. 1050-1059, 2016.
Y. Gal, R. Islam, and Z. Ghahramani. Deep bayesian active learning with image data. In Pro- ceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1183-1192, 2017.
S. Gopakumar, S. Gupta, S. Rana, V. Nguyen, and S. Venkatesh. Algorithmic assurance: An ac- tive approach to algorithmic testing using bayesian optimisation. In Proceedings the 32nd International Conference on Neural Information Processing Systems (NeurIPS), pp. 5478, 2018.
C. Guo, G. Pleiss, Y. Sun, and K.Q. Weinberger. On calibration modern neural networks, In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1321-1330, 2017.
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution exam- ples in neural networks. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2017.
N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classification and preference learning. arXiv:1112.5745 Dec 2011.
H. Jiang, B. Kim, M. Guan, and M. Gupta. To trust or not to trust a classifier. In Advances Neural Information Processing Systems (NeurIPS), pp. 5541-5552. 2018.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
B. Lakshminarayanan, A. Pritzel, and Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6402-6413. 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 86(11):2278-2324 1998.
Y. LeCun, C. Cortes, and C.J. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exdb/nnist,, 2, 2010.
D. MacKay. A practical bayesian framework for backpropagation networks. Neural Computation, 4(3):448-477 472, May 1992.
R. Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A.Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
A. Niculescu- Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference Machine Learning (ICML), pp. 625 625-633, 2005.
J.C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. In Advances in Large Margin Classifiers, pp. 61-74. MIT Press, 1999.
C. Sawade, N. Landwehr, S. Bickel, and T. Scheffer. Active risk estimation. In Proceedings of the 27th International Conference on International Conference Machine Learning (ICMI), pp- 951-958, 2010.
S. Schelter, T. Rukat, and F. Biessmann. Learning to validate the predictions of black box classifiers unseen data. In Proceedings of ACM SIGMOD (SIGMOD), 2020.
10

---

Under review as a conference paper at ICLR 2021
B. Settles. Active learning literature survey. Technical report, Computer Science Department, Uni- versity of Wisconsin -adiiso,,,,,,,nn 2010.
J. Snoek, H. Larochelle, and R.P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2951-2959, 2012.
Y. Sun, X. Huang, D. Kroening, J. Sharp. M. Hill. and R. Ashmore. Testing Deep Neural Networks. arXiv e-prints, art. arXiv: 047994 Mar 2018.
Y. Tian, K. Pei, S. Jana, and B. Ray. Deeptest: Automated testing of deep-neural-eewwor--diee autonomous cars, In Proceedings of the 40th International Conference on Software Engineering (ICSE), pp. 303-3314 2018.
H. Xiao, K. Rasul, and Vollgraf. Fashion-mnist a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http://axx...or///bb///
07747
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability es- timates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 694 -69, 2002.
J.M. Zhang, M. Harman, L. Ma, and Y. Liu. Machine Learning Testing: Survey, Landscapes and Horizons. arXiv e-prints, art. arXiv: 10742, Jun 2019.
J. Zhou, F. Li, J. Dong, H. Zhang, and D. Hao. Cost-effective testing of deep learning model through input reduction, 2019. URL https::/ www.nneefffotirooooo S1XCepNYPr
A APPENDIX
B ALGORITHM DETAILS
B.1 BACKGROUND
Bayesian Neural Networks Bayesian neural networks (BNNs) are special neural networks that maintain distribution over its parameters (MacKay 1992; Neal 1995). In practice, performing act inference in BNNs generally intractable, thus, there has been line of works aiming to perform approximate inference. In this work, perform inference as is both both scalable and theoretically guaranteed equivalent to perform- ing approximate variational inference find a distribution tractable family that minimizes the Bullback-Leibler divergence to the true model posterior.
the MC-dropout method (Gal & Ghahramani 2016) to
Bayesian Active Learning by Disagreement (BALD) BALD is sampling methodology in Bayesian active learning, which samples the data point that maximizes the mutual information be- tween the model prediction and the model posterior (Houlsby al. [011]] Gal al. 2017). To be more specific, let us denote DJ { Vxx² J as the labelled set obtained after iteration é and B as the BNN trained on Dj (with parameter w), then the acquisition function of BALD is defined as,
Tly: ar Djj [F][ DJJ (u--p(/lp [Hln) w]],
(10)
where is predicted label for the data point The data point that maximizes BALD is the data point for which the model has many possible predictions, i.c. the posterior draws have disagreement.
B.2 DETAILS THE PROPOSED ACQUISITION FUNCTION
Our proposed acquisition function is defined as the mutual information (Houlsby et al. 2011
Gal
et al. 2017 between (x) and the BNN parameters w. i.c.,
[z)x()): w D H[Q((=))æ x, D/] - Eu-n(w/
(1I)
This mutual information can be approximated using the MC-dropout variational distribution go (w) Next, we show how approximate each tearm the right hand side of Eq.

---

Under review as a conference paper at ICLR 2021
Computing H=Q Di] For each data point 2, 0s is a discrete random variable with C' distinct values, so (x) also a discrete random variable with at most C distinct values. Therefore, (I))(=)) Df] can be computed as,
([Qx(r)) Dil = E, TEQ p((((z) q/a, Di) log logp(Qx(x) g/2, Di),
(12)
where @ consists of all the possible values of Qx when de E (l....CC By using the union bound, HIQu (z, Djj can then be expressed as,
(x)),( Djj = -,
qEQ (Eneor plū. h|r; M)r, log (Ene p(ū. hd,,, Di))
where (q) is the inverse function that maps the value of (x) du- Given M stochastic forward passes { } from the MC-dropout posterior distribution (o(w Df), [x)x(x)) DJ can
finally be approximated as,
(x)x, DI1 ~ -EEe ( ((Ine ((0)
1
M P(ù. h12, wr))/MM j=1
(13)
log E (Eneor (w)
j=l p(ūe h|2, úh (rz))/M)
M
where Qu(x) can approximated as Qu(x) ss Ewnng(l(ps) [Qx(Ax: Dix3, Dxra /a,w Unl) Rd EE
M Qk (V Dy. D (x)ra,,,,))))MM bell)/M, with VxH, Vays denoting the predicted labels for X \
given the parameter ay.
Computing Ewep(WiD (+)),,) Similar as in the above paragraph, given M stochastic forward passes ك from the MC-dropout variational distribution qo (w)
Bu-p(w/ D2) [H Qu (2) wIl can be approximated as,
E D.) [H (2) 12,
(x) 2g]
M
E E ql20)))== = g|2, (5)))
M j=1
1
Σ M
(Evee (Enedr p(u. h/2 (
M 1=
x log (Enco p(u 1121 k12,
(14)
where Q consists of all the possible values of (x(z) when dr (I...,cc. (g) is the inverse function that maps the value of Qx(x)) dz. and Qc(x) can be approximated as (x(x) ss Euwqq(w/D)) (Qx [Dx;,, Dxs, Dx,) ~ (E Qx DY, Pxy (r)))) with Dxs, Va,d, denoting the predicted labels for X \ given the parameter wj.
C EXPERIMENT DETAILS
C.1 MNIST DATASET
Models-under-test The three models-under-tess
to evaluate proposed method are:
(i) M-MNIST: A CNN trained on the train MNIST dataset (LeCun et al. 2010). We implement the model in tensorflow but use the official network architecture published keras.comf as there is no official source code on tensorflow.org. The accuracy of this model on the test MNIST dataset 99.06%;
https://keras. htwp://pwwwwxxx////po/o/o//.ppta examel st__
12

---

Under review as a conference paper at ICLR 2021
iii M-MNIST-ES. A CNN trained on the train MNIST dataset. The model architecture is same as M-MNIST, but we use early stopping to decrease the model performance, i.. we set smaller epochs and higher batch size. The accuracy of this model the test MNIST dataset is 70.67%;
(iii) M-FashionMNIST: CNN trained on the train FashionMNIST dataset (i.e. completely different dataset compared to MNIST) (Xiao et al. 2017). The model is implemented using the official code published on tensorflow.org. The accuracy of this model on the test MNIST dataset is 12.39%.
Metric sets
The metric sets we use to evaluate our proposed method are:
Three sets metrics consisting of only one metric in each set: precision of class 2, recall of class 2, and overall accuracy. These are common metrics used to evaluate performance classifiers, and these metrics cover both per-class metrics and overall metrics.
(ii) Two sets of metrics consisting multiple metrics. We use two sets: one set consisting of 3 metrics (accuracy, macro-precision, macro-recall), and one consisting of 21 metrics (accuracy, precision and recall of each class).
The BNN and the binary classifier architecture The BNN and the binary classifier have the same architecture. The architecture is an MLP with two layers, 256 neurons/layer, and with dropout applied in each layer. The number of MC-dropout samples are 50. The initial labelled set Dy has 100 data points randomly sampled from the test dataset. We tuned 3 hyper-parameters: learning rate, epochs and dropout rate using Bayesian optimization (Snock et al. 2012). We also reinitialize the BNN after each iteration as in (Gal et al. 2017).
C.2 CIFARIO DATASET
Models-under-test The three models-under-test we use to evaluate our proposed method are:
(i) M-CIFARI0: CNN trained on the train CIFARIO dataset (Krizhevsky 2009). We im- plement the model in tensorflow but use the official network architecture published on keras.con as there is no official source code on tensorflow.org. The accuracy of this model the test CIFARIO dataset is 77.43%.
(ii) M-CIFARIO-ES: CNN trained the train CIFARIO I0 dataset. The model architecture is same as M-CIFAR 10, but we use early stopping to decrease the model performance, i.e. we set smaller epochs and higher batch size. The accuracy of this model on the test CIFARIO dataset is 40.39%.
(iii) M-SVHN: A CNN trained on the train SVHN dataset (i.e. completely different dataset compared to CIFARIO) (Netzer et al., 201T). The model architecture is same as M- CIFARIO. The accuracy of this model on the test CIFAR10 dataset is 9.3%.
Metric sets described in Section C.I
The metric sets we use are same as the metrics set used for the MNIST dataset (as
The BNN and the binary classifier architecture The BNN and the binary classifier have the same architecture. The architecture the standard LeNet model (Lecun et al.) 1998) with dropout applied before the relu layers. The number of MC-dropout samples are 50. The initial labelled set Di has 500 data points randomly sampled from the test dataset. We tuned hyper-parameterss learning rate, epochs and dropout rate using Bayesian optimization (Snoek et al. 2012). We also reinitialize the BNN after each iteration as in (Gal et al. 2017).
C.3 THE QUALITY OF THE BNNS
In Figures and we plot the prediction accuracies of the BNNs in BALD and ALT-MAS for all combinations of models-under-tes and metric sets on the MNIST and CIFARIO dataset. Below,
ficttion
httes://eera.iio/evall/////////s
13

---

Under review as a conference paper at ICLR 2021
evaluate how complex the BNN needs to be for the metric estimation to be accurate and how effective our augmented training strategy and our proposed sampling methodology are.
Theoretically, we can see that for the BNN to be useful, it does not need to have high classification accuracy; only needs to accurately predict the data points that contribute to the metric estimation. Being incorrect other data points does not affect the predicted metric values. See example:
True label
Model-under-test
Surrogate model prediction
prediction
0
5
Any prediction except 5 (e.g. 2,2,3, 3, will give accurate metric estimation
5
5
Correct prediction matters for some metrics (e.g. accuracy, preci- sion/recall of class 5), but not for other metrics (e.g. precision/recall of other classes)
In summary, the metric estimation accuracy depends on the model-under-ter and the metric set. This is clearly demonstrated on the performance of BALD. With BALD, we use the predicted labels by the BNNs trained by BALD to estimate the metric values. For MNIST, the BNNs trained with BALD have accuracies ranging from 70- 90%, but for the models-under-ee M-MNIST-ES (average & bad models), the metric estimation accuracies range from 90 100% that is much higher than the BNNs' accuracies. Similarly, for CIFARIO, the BNNs accuracies range from 40 60%. but for the models-under-ter M-SVHN and M-CIFARIO-ES, the metric estimation accuracies range from 90 100%. For the good mandell-under-eee M-MNIST and M-CIFARIO then BALD requires high BNN accuracy in order achieve good metric estimations.
M- FashionMNIST and
For our proposed method ALT-MAS, with the models M- M-FashionMNIST, M-MNIST-ES,, M-SVHN and M-CIFARIO-ES and the overall metrics (accuracy), the observations are similar to BALD That is, the BNNs accuracies are much lower to the metric estimation accuracies. The better thing is that, for the per-class metrics (precision and recall of class 2, the BNNS accuracies do not need to be high (as in the overall metrics) in order to achieve high metric estimation accuracy. In particular, the BNNs accuracies by ALT-MAS are much lower than the BNNs by BALD, but the metric esti- mations by ALT-MAS are much higher than by BALD. This proves the motivation of our sampling approach, that is, the BNN only needs to accurately predict the data points that contribute to the met- ric estimation. For the good models-under-test M-MNIST and M- M-CIFARIO, due to our augmented training strategy, the BNN accuracies improve significantly, and thus, leads to the metric estimations to be much more accurate compared to BALD.
Recall
Accuracy
Acc- Rec-PrecM-RecM
Acc- Prec-Rec
96
96
96
96
M.MNIST
86
88
88
88
Accuracy of the BNN (Percentage)
80
80
80
60
80
100
300
200
160
240
160
240
M-Faahion MN15
96
Precision_2 2
Recall
Accuracy
Acc-PrecM-Rece
Acc-Prec-RR
88
88
88
88
88
80
60
80
72
72
72
72
72
200
600
200
400
600
200
600
200
400
200
600
Precision 2
Recall 2
Accuracy
Acc-PrecM-RRcM
Acc-Prec-Rec
(N-MN/ST-E5
88
90
88
84
80
80
80
78
72
72
200
400
600
200
400
600
200
400
600
0 200
400
600
200
400
600
Number of oracle queries
Number oraule queries
queries
queries
Number queries
BALD
ALT-MAS
Figure 3: The accuracy of the BNN, for each combination of model-under-test (M-MNIST, M- FashionMNIST, & M-MNIST-ES) and metric set. Curves are smoothed over sliding window of oracle queries (iterations). Plotting mean and standard error over 3 repetitions (Best seen in color).
14

---

Under review as a conference paper at ICLR 2021
Accuracy
Acc-PrecMh-Rech
CIFARRO
64
64
56
56
Accurecy the (NN (Pecceningg))
640
250
500
750
750
500
750
250
500
750
250
500
750
250
500
750
Precision_2
Recall
Accurecy
55
Acc-PrecM--RecM
50
50
50
46
40
40
0
250
500 750
250
500 750
250
500
750
250
500
750
250
500
750
Es
Prectsion_2
Recall_2
Accuracy
Acc-PrecM-RRct
Acc-Prec-Rec
M.FIFRLDD
'2)
48
48
8
44
44
44
44
80
500
750
250
500
750
250
500
750
750 Number oracle queries
250
500
750
Number of oracle queries
Number
Number oracle queries
Number oradee quertes
BALD
ALT-MAS
Figure 4: The accuracy of BNN, for each combination model-under-test (M-CIFARIO, SVHN, & M-CIFARIO-ES) and metric set. Curves smoothed over sliding window of 5 oracle queries (iterations). Plotting mean and standard error over repetitions (Best seen in color).
15


Reviews:
[{"review_id": "qGsttwY1clE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is quite clear and is well-motivated by a practical situation of having a classifier that is a black box and that is evaluated by various metrics that one would like to assess as efficiently as possible. The paper demonstrates that the simpler task of using active learning to identify examples for labeling that would most reduce the uncertainty in metrics of interest is more effective than the more general task of using active learning to learn to predict the labels that the original classifier would give and using those results to calculate the metrics of interest.\n\nThe only cons that I see are that the paper lacks some obvious explorations that I think would be quite valuable and informative:\n1. What is the variation in the sequences of points chosen for labeling depending on the metric that is being calculated?\n2. Calculating the ROC curve does not require classifier internal structure. It only requires some continuous output representing class membership rather than just a discrete indication.\n3. Remark 3.2: Simplicity seems an insufficient reason to have the binary classifier and Bayesian Neural Network have the same structure. This should be explored and at least a summary of performances given for variations on this.\n4. To sample the point that is best for multiple metrics, equation (9) is used, which calculates the sum of the mutual informations between the BNN parameters and the metrics. Is it obvious that using the sum is the best way? I wonder if using something different like the point that most increases the minimum mutual information may work better.\n5. In algorithm 1, step 6, note that $\\mathcal{C}_{\\eta}^{t-1}$ is being trained on $\\mathcal{D}_l^{t-1}$.\n6. In algorithm 1, step 8, $\\mathcal{S}_l^{t-1}$ is undefined. Do you mean to refer to $\\mathcal{S}_l$ from step 7?\n7. In figure 1, the average metrics seem to vary quite significantly with the number of oracle queries. Since you are measuring the average, I expected the decrease to be relatively smooth. How do you account for the significant variation?\n8. The Gal and Ghahramani paper is listed twice in the bibliography.", "extra_scores": {}, "extra_text": {}}, {"review_id": "e22i41OEMLo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nThe paper proposes an active testing approach that actively selects test instances to estimate the performance of a (black box) machine learning model. The key idea is to train a Bayesian Neural Networks (BNN) with a small amount of labeled test data and evaluate how well the model-under-test agrees with the BNN on samples for which the BNN has a high confidence. More instances to be labeled by an oracle are selected with active learning, i.e. select the data point that minimizes the uncertainty of the metric prediction.\n\n#### Recommendation:\nI recommend to reject the paper, mostly because the experimental results do not support the claim that the proposed approach performs much better than traditional evaluation or prior works. Furthermore, appropriateness of the experimental setup and novelty of the BNN training are unclear and proper discussion of related works is missing.\n\n#### Strong Points:\n- Active testing is an important research direction since more and more pre-trained models are applied in practice without being fine-tuned.\n- The approach is clearly described. The paper is well-written and easy to understand.\n\n#### Weak Points:\n- The evaluation does not support the claim that ALT-MAS performs substantially better than traditional evaluation or prior works. Out of 30 plots, the proposed method performs best in approx. 3 cases, ties for the first place in approx. 8 cases, and is inferior (wrt. the traditional eval and BALD) in the remaining ~19 cases.\n- The title suggests that the paper addresses machine learning in general. However, only a simple deep neural network for MNIST and CIFAR10 is used in the evaluation. Hence, it remains unclear if the approach can successfully be applied to other machine learning methods and datasets. In general, it would be a strong point of this paper to include many different machine learning models since it is agnostic regarding the learning framework used.\n- I disagree with the general statement 'Better methods are those that converge faster to zero'. It depends a lot on the budget available for labeling new instances and the desired value (and confidence) of the target metrics. A clearly better method would have a lower error for any amount of labeling budged.\n- Active testing is appealing when it can be assumed that the test data has a different distribution than the training data. However, most of the experiments use same train/test distribution. Hence, the approach is not evaluated in an appropriate setting.\n- The paper claims to contribute 'novel approach to train the BNN so as to accurately estimate the metrics of interest'. However, it uses Monte Carlo dropout. It is unclear which part of the training is novel.\n- The main body of the paper does not contain a section that positions the work well in the available literature.\n\n#### Questions:\n1. Do I understand correctly that the 'Traditional' evaluation uses as many samples for evaluation as ALT-MAS and BALD? I am referring to the phrase 'the traditional method where the metrics are computed using their mathematical formula with all the labeled data *up to the current iteration*, and the labeled data is picked randomly from the whole test dataset'.\n2. In Section 4.2, you write that 'ALT-MAS performs well on all of the scenarios'. In Figure 2, first row, I'd say that ALT-MAS is inferior than the traditional evaluation in 4 out of 5 cases and ties for the first place for Recall_2. How do you come to the conclusion that ALT-MAS performs well on all of the scenarios?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cjj_zzj4ZZg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors have proposed using an active learning approach to estimate evaluation metrics for a given model. The approach learns a sampling function that decides which observations need to be labeled, which are then fed to a Bayesian neural network (BNN) that aims to estimate the distribution Y|X. The authors select which observations to sample by maximizing the mutual information between the model evaluation metric and the BNN parameters.\n\nPros:\n+ Active testing of models is a difficult problem in high-dimensions. The proposed problem is highly relevant.\n\nMajor concerns:\n\n1. The authors have primarily focused on comparing against a traditional sampling method that simply takes IID samples and a deep-learning-based active learning algorithm for learning the distribution of Y|X. Both of these comparators are very weak. The former is not doing any active learning. The latter is trying to do a much harder task of learning Y|X rather than estimating the evaluation metrics. I think a more reasonable comparison method is the active testing approach from Sawade 2010 that estimates the distribution p(Y|X;\\theta) using a neural network.\n\n2. There seem to be a number of misconceptions about prior work. The authors argue that previous approaches are specific to particular evaluation metrics and cannot scale to high-dimensional data. However, the active risk estimation approach taken in Sawade 2010 is quite general and can be used to estimate the expected value of any function l that maps the predicted target and the true target to a real-valued number, which is the same type of functions considered in this paper. It is also easy to extend the approach to multi-variate evaluation metrics by, say, replacing the squared error with the squared L2 norm. Finally, you can use any estimator of Y|X in their framework, so one could use a neural network to do active risk estimation for high-dimensional data.\n\n3. The authors do not give a justification for why they used mutual information to prioritize which observations to label. Is there a way to show that it minimizes the estimation error of the evaluation metrics? And similarly, why should one try to prioritize which observations to label using the sum of mutual informations?\n\n4. I am confused by the y-axes in Figures 1 and 2. What is average relative error? The average relative error is very large in certain cases. In this case, do all the methods have unacceptably large errors?\n\n5. When estimating evaluation metrics for classifiers, it is important to characterize the theoretical guarantees of the estimates. Do we know if the method is consistent or (asymptotically) unbiased? Are there confidence intervals associated with this approach? In addition to some discussion of the theoretical properties, please evaluate the bias of the method in simulations (whereas the simulations currently only show relative error).", "extra_scores": {}, "extra_text": {}}, {"review_id": "7pKT1MAhzCo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors proposed ALT-MAS, a data-efficient testing framework that can accurately estimate the performance of a machine learning model, a novel approach to train the BNN to accurately estimate the metrics of interest, and a novel sampling methodology to estimate the metrics of interest efficiently. The performances of proposed methods are demonstrated through the empirical effectiveness of our proposed machine learning testing framework on various models-under-test for a wide range of metrics and different datasets.\n\nThe problem setting addressed by the authors is one of the hot topics. The experimental results on the two data sets show good performance on the proposed method.\n\nExperimental results using MNIST and CIFAR10 show that the proposed method can consistently provide better accuracy than the conventional method.\n\nAlthough the proposed method is based on early stopping, we could not be sure from the paper whether the proposed method can consistently reproduce the same performance on other tasks.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses a well-motivated problem of evaluating the accuracy of a black-box classifier A(x) using actively selected set of labeled examples.  They predict two additional classifiers --- one to predict if an example will be corrected by A(x) and the second a Bayesian NN to assign a distribution over likely labels of unlabeled examples.  The final accuracy estimate is obtained   from the labeled data and the probabilistic labels on the unlabeled data.
Reviewers rightly pointed out that the paper only compares with conventional active learning methods, and skips comparison with methods specifically designed for active evaluation.  A list is attached below.  Also, the overall technical contribution seems limited in terms of both empirical accuracy gains it obtains and novelty of ideas exposed.

Related papers:
A lazy man's approach to benchmarking: Semisupervised classifier evaluation and recalibration P Welinder, M Welling, P Perona - Proceedings of the IEEE …, 2013 - cv-foundation.org

Active evaluation of classifiers on large datasets N Katariya, A Iyer, S Sarawagi - 2012

Adaptive Stratified Sampling for Precision-Recall Estimation. A Sabharwal, Y Xue - UAI, 2018

Online Active Model Selection for Pre-trained Classifiers Mohammad Reza Karimi, Nezihe Merve Gürel, Bojan Karlaš, Johannes Rausch, Ce Zhang, Andreas Krause

Towards Efficient Evaluation of Risk via Herding Z Xu, T Yu, S Sra
Decision: Reject