Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper is quite clear and is well-motivated by a practical situation of having a classifier that is a black box and that is evaluated by various metrics that one would like to assess as efficiently as possible. The paper demonstrates that the simpler task of using active learning to identify examples for labeling that would most reduce the uncertainty in metrics of interest is more effective than the more general task of using active learning to learn to predict the labels that the original classifier would give and using those results to calculate the metrics of interest.

The only cons that I see are that the paper lacks some obvious explorations that I think would be quite valuable and informative:
1. What is the variation in the sequences of points chosen for labeling depending on the metric that is being calculated?
2. Calculating the ROC curve does not require classifier internal structure. It only requires some continuous output representing class membership rather than just a discrete indication.
3. Remark 3.2: Simplicity seems an insufficient reason to have the binary classifier and Bayesian Neural Network have the same structure. This should be explored and at least a summary of performances given for variations on this.
4. To sample the point that is best for multiple metrics, equation (9) is used, which calculates the sum of the mutual informations between the BNN parameters and the metrics. Is it obvious that using the sum is the best way? I wonder if using something different like the point that most increases the minimum mutual information may work better.
5. In algorithm 1, step 6, note that $\mathcal{C}_{\eta}^{t-1}$ is being trained on $\mathcal{D}_l^{t-1}$.
6. In algorithm 1, step 8, $\mathcal{S}_l^{t-1}$ is undefined. Do you mean to refer to $\mathcal{S}_l$ from step 7?
7. In figure 1, the average metrics seem to vary quite significantly with the number of oracle queries. Since you are measuring the average, I expected the decrease to be relatively smooth. How do you account for the significant variation?
8. The Gal and Ghahramani paper is listed twice in the bibliography.
review_title: This paper aims to use active learning to obtain labels for points that allow for the most efficient way of estimating metrics of interest describing the performance of a classifier. The idea is that this would be simpler and more efficient than the typical active learning strategy of trying to learn a model to actually predict what the labels would be. The paper demonstrates this through some experiments.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The authors have proposed using an active learning approach to estimate evaluation metrics for a given model. The approach learns a sampling function that decides which observations need to be labeled, which are then fed to a Bayesian neural network (BNN) that aims to estimate the distribution Y|X. The authors select which observations to sample by maximizing the mutual information between the model evaluation metric and the BNN parameters.

Pros:
+ Active testing of models is a difficult problem in high-dimensions. The proposed problem is highly relevant.

Major concerns:

1. The authors have primarily focused on comparing against a traditional sampling method that simply takes IID samples and a deep-learning-based active learning algorithm for learning the distribution of Y|X. Both of these comparators are very weak. The former is not doing any active learning. The latter is trying to do a much harder task of learning Y|X rather than estimating the evaluation metrics. I think a more reasonable comparison method is the active testing approach from Sawade 2010 that estimates the distribution p(Y|X;\theta) using a neural network.

2. There seem to be a number of misconceptions about prior work. The authors argue that previous approaches are specific to particular evaluation metrics and cannot scale to high-dimensional data. However, the active risk estimation approach taken in Sawade 2010 is quite general and can be used to estimate the expected value of any function l that maps the predicted target and the true target to a real-valued number, which is the same type of functions considered in this paper. It is also easy to extend the approach to multi-variate evaluation metrics by, say, replacing the squared error with the squared L2 norm. Finally, you can use any estimator of Y|X in their framework, so one could use a neural network to do active risk estimation for high-dimensional data.

3. The authors do not give a justification for why they used mutual information to prioritize which observations to label. Is there a way to show that it minimizes the estimation error of the evaluation metrics? And similarly, why should one try to prioritize which observations to label using the sum of mutual informations?

4. I am confused by the y-axes in Figures 1 and 2. What is average relative error? The average relative error is very large in certain cases. In this case, do all the methods have unacceptably large errors?

5. When estimating evaluation metrics for classifiers, it is important to characterize the theoretical guarantees of the estimates. Do we know if the method is consistent or (asymptotically) unbiased? Are there confidence intervals associated with this approach? In addition to some discussion of the theoretical properties, please evaluate the bias of the method in simulations (whereas the simulations currently only show relative error).
review_title: Review of ALT-MAS

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors proposed ALT-MAS, a data-efficient testing framework that can accurately estimate the performance of a machine learning model, a novel approach to train the BNN to accurately estimate the metrics of interest, and a novel sampling methodology to estimate the metrics of interest efficiently. The performances of proposed methods are demonstrated through the empirical effectiveness of our proposed machine learning testing framework on various models-under-test for a wide range of metrics and different datasets.

The problem setting addressed by the authors is one of the hot topics. The experimental results on the two data sets show good performance on the proposed method.

Experimental results using MNIST and CIFAR10 show that the proposed method can consistently provide better accuracy than the conventional method.

Although the proposed method is based on early stopping, we could not be sure from the paper whether the proposed method can consistently reproduce the same performance on other tasks.

review_title: Nice problem setting and good performance.

final_score: rating: 3: Clear rejection
review_content: #### Summary:
The paper proposes an active testing approach that actively selects test instances to estimate the performance of a (black box) machine learning model. The key idea is to train a Bayesian Neural Networks (BNN) with a small amount of labeled test data and evaluate how well the model-under-test agrees with the BNN on samples for which the BNN has a high confidence. More instances to be labeled by an oracle are selected with active learning, i.e. select the data point that minimizes the uncertainty of the metric prediction.

#### Recommendation:
I recommend to reject the paper, mostly because the experimental results do not support the claim that the proposed approach performs much better than traditional evaluation or prior works. Furthermore, appropriateness of the experimental setup and novelty of the BNN training are unclear and proper discussion of related works is missing.

#### Strong Points:
- Active testing is an important research direction since more and more pre-trained models are applied in practice without being fine-tuned.
- The approach is clearly described. The paper is well-written and easy to understand.

#### Weak Points:
- The evaluation does not support the claim that ALT-MAS performs substantially better than traditional evaluation or prior works. Out of 30 plots, the proposed method performs best in approx. 3 cases, ties for the first place in approx. 8 cases, and is inferior (wrt. the traditional eval and BALD) in the remaining ~19 cases.
- The title suggests that the paper addresses machine learning in general. However, only a simple deep neural network for MNIST and CIFAR10 is used in the evaluation. Hence, it remains unclear if the approach can successfully be applied to other machine learning methods and datasets. In general, it would be a strong point of this paper to include many different machine learning models since it is agnostic regarding the learning framework used.
- I disagree with the general statement 'Better methods are those that converge faster to zero'. It depends a lot on the budget available for labeling new instances and the desired value (and confidence) of the target metrics. A clearly better method would have a lower error for any amount of labeling budged.
- Active testing is appealing when it can be assumed that the test data has a different distribution than the training data. However, most of the experiments use same train/test distribution. Hence, the approach is not evaluated in an appropriate setting.
- The paper claims to contribute 'novel approach to train the BNN so as to accurately estimate the metrics of interest'. However, it uses Monte Carlo dropout. It is unclear which part of the training is novel.
- The main body of the paper does not contain a section that positions the work well in the available literature.

#### Questions:
1. Do I understand correctly that the 'Traditional' evaluation uses as many samples for evaluation as ALT-MAS and BALD? I am referring to the phrase 'the traditional method where the metrics are computed using their mathematical formula with all the labeled data *up to the current iteration*, and the labeled data is picked randomly from the whole test dataset'.
2. In Section 4.2, you write that 'ALT-MAS performs well on all of the scenarios'. In Figure 2, first row, I'd say that ALT-MAS is inferior than the traditional evaluation in 4 out of 5 cases and ties for the first place for Recall_2. How do you come to the conclusion that ALT-MAS performs well on all of the scenarios?
review_title: I recommend to reject the paper, mostly because the experimental results do not support the claim that the proposed approach performs much better than traditional evaluation or prior works. Furthermore, appropriateness of the experimental setup and novelty of the BNN training are unclear and proper discussion of related works is missing.

Meta-review: The paper addresses a well-motivated problem of evaluating the accuracy of a black-box classifier A(x) using actively selected set of labeled examples.  They predict two additional classifiers --- one to predict if an example will be corrected by A(x) and the second a Bayesian NN to assign a distribution over likely labels of unlabeled examples.  The final accuracy estimate is obtained   from the labeled data and the probabilistic labels on the unlabeled data.
Reviewers rightly pointed out that the paper only compares with conventional active learning methods, and skips comparison with methods specifically designed for active evaluation.  A list is attached below.  Also, the overall technical contribution seems limited in terms of both empirical accuracy gains it obtains and novelty of ideas exposed.

Related papers:
A lazy man's approach to benchmarking: Semisupervised classifier evaluation and recalibration P Welinder, M Welling, P Perona - Proceedings of the IEEE …, 2013 - cv-foundation.org

Active evaluation of classifiers on large datasets N Katariya, A Iyer, S Sarawagi - 2012

Adaptive Stratified Sampling for Precision-Recall Estimation. A Sabharwal, Y Xue - UAI, 2018

Online Active Model Selection for Pre-trained Classifiers Mohammad Reza Karimi, Nezihe Merve Gürel, Bojan Karlaš, Johannes Rausch, Ce Zhang, Andreas Krause

Towards Efficient Evaluation of Risk via Herding Z Xu, T Yu, S Sra
Decision: Reject