Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Extensive research is going on in an active learning area. However, the annotation cost of an evaluation set is overlooked in previous studies. Since active learning assumes expensive labeling costs, the cost of evaluation set annotation is also important. Active testing, the problem addressed in this paper, has been underexplored.

This work tackles how to minimize a labeling cost for an evaluation set. The paper proposes two main ideas: an estimator function that predicts the expected error of a task model, and an acquisition function to minimize the proposed expected error function. The paper experiments with its method on MNIST dataset with distribution shift scenario, and error estimation without distribution scenario on CIFAR10, CIFAR100, and F-MNIST.
strengths_and_weaknesses: ## Strength

### Novelty
The paper proposes a new error estimation function (Active surrogate estimator) and test-data acquisition function.

### Experiment
The paper shows its performance with and without a distribution shift scenario, addressing the effect of the method on a scenario with a missing class scenario (an extreme case of class imbalance).
Section 7.2 shows the performance on CIFAR10, CIFAR100, and F-MNIST which are sufficient to show their effectiveness in terms of experiment size.
In both with/without distribution shifts, the proposed method is meaningfully effective over the competing methods.

### Analysis
Section5 decomposes the error bound of an active surrogate estimator and interprets each term. The interpretation of these terms is meaningful and interesting.


## Weakness

### Weak novelty of surrogate estimator and its limitations
1. weak novelty: Although the paper proposes two parts, an active surrogate estimator and acquisition function of active testing, the surrogate estimator (Equation (4)) idea is not surprising. In my simplified interpretation, equation (4) is a loss prediction of a task model using soft-pseudo labels, i.e., the prediction of a surrogate model. In other words, this objective heavily relies on a surrogate model in hoping that the surrogate model is very accurate as true ground truths. In addition, most of the performance gain of this paper is coming from this surrogate estimator, not the acquisition function (figure2 (a))

2. Limitation: Due to the dependency of equation(4) on the surrogate model, I presume that the performance of the surrogate model is critical for equation (4) to perform well, which is an important limitation not analyzed nor addressed in this paper.

### Experiment with strong task and surrogate models
This experiment setting may be due to the limitation of equation (4).
In the experiment, I conjecture that the authors use strong task and surrogate models.

1. In section 7.1, the task and surrogate model are initially trained on 2000 data (roughly 3.5% of the MNIST training set. the Training set size of MNIST is 60,000). Typical classification models perform over 95% accuracy with 1% of the MNIST training set. This is not the weakness but I am pointing out that the task and surrogate model accuracy is high.

2. In section 7.2, experiments are conducted on more challenging datasets. In these experiments, both task and surrogate models are trained using 40,000 samples. This is 80% of the training dataset for CIFAR10 and CIFAR 100 which has a training set size of 50,000 samples.
In a typical active learning setting, models are evaluated starting from 10% and additionally trained on an actively annotated labeled set of size 20-30%, summing up a total of 30-40% annotated data. I presume the models are trained on a larger set of samples (40,000 / 80%) than the typical setting (18,000~2,4000 / 30-40% ) due to the dependency of equation (4) on surrogate model performance.

Missing analysis and limitation.
Since surrogate model performance important factor in this study, this should be analyzed and addressed in this paper.

### Missing details or experiment.
1. Section 7.1 shows only 1 case for the distribution shift analysis. It would be better to show a few more cases for this study.
2. The graphs in this paper show squared error, the difference between a true expected loss and predicted loss. What is the absolute size of the true expected loss? What would be the critical value for reasonable estimation error for loss expectation? What would the graph look like if you use the ratio between the expected loss and true loss?
3. Typo? In Equation (7), ‘R’ should be ‘r’?

questions: 1. How is R_ASE sensitive to the accuracy of a surrogate model?
2. What is the absolute scale of true losses in the experiments? How would the graph look like if graphs are scaled (normalized) over true loss?

limitations: The supplementary material addresses societal impact and limitations.
However, the most important limitation, the dependency of R_ASE (equation.4) and surrogate model accuracy not analyzed nor addressed in this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method for active testing called Active Surrogate Estimators (ASEs). ASEs actively learn a surrogate model to predict the losses and also employ a novel acquisition strategy, called XWED, which is a modification of BALD but puts weights for each data point using its corresponding loss value. The experiments show the proposed method is better than other baselines such as MC and LURE.
strengths_and_weaknesses: Strengths:
- The problem considered is important.
- The proposed methods, ASEs and XWED, are novel and well-motivated.
- The proposed methods achieve good performance in the experiments.
- The paper is well-written, although the presentation can be further improved.

Weaknesses:
- The proposed methods are small modifications of previous methods. Particularly, ASE is a small modification of LURE, while XWED is a small modification of BALD. However, I think the proposed methods are still good contributions since these modifications help improve the performance of the methods in practice.
- The paper does not give adequate descriptions for several baselines used in the experiments, e.g. LURE ~ $\mathbb{E}$[Loss], LURE ~ XWED, LURE ~ BALD, ASE BALD, etc. It would be better to explain each of these methods using some formulas. That would help readers understand more about the ways different components of the methods can be used.
- I think it would also be better for the paper to give some concrete examples where the proposed methods are applied and computed. For example, what would be the specific formulas when the model is a neural network, what would be the model $\pi$, and how the computation is done? These examples can be given in reference to specific steps in Algorithm 1. I think Section 5 of the paper can be shorten for more space.
- There is also a related research area that aims to evaluate models without any labels, e.g. see [1, 2] below and other related work. It would be better if the paper also compares their active testing algorithm with these methods.

References:
- [1] Corneanu et al. Computing the Testing Error without a Testing Set. CVPR 2020.
- [2] Deng & Zheng. Are Labels Always Necessary for Classifier Accuracy Evaluation? CVPR 2021.
questions: In Eq (6), why don't we just multiply a factor $\mathbb{E}_{Y \sim \pi(\cdot|x)} [\mathcal{L}(Y, f(x))]$ directly into $\mathrm{BALD}(x)$ in Eq (5) ? Is there any specific reason why we need to multiply $\mathcal{L}(Y, f(x))$ into each of the expectations in Eq (6) ?
limitations: No potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the ASE, showing a better performance than the Lure approach in active testing. The newly proposed networks for the g and \pi function and the acquisition function show better performance in the active test. Also, the various aspects of ASE are explored in the distribution shift.          

strengths_and_weaknesses: Strong points: The proposed algorithm is intuitive and straightforward, performs better, and addresses the active test problem well. Experiments are well designed to show the critical properties of the proposed algorithm. 

Weak points: Theoretical analysis of the ASE error is somewhat vague. The decompositions and analysis are valuable. However, I, II, and III properties are not well-explained, including too massive contexts. It seems that the main is the empirical analysis due to the non-existence of any theorem,   


questions: I have some questions conserving the active test and algorithms. 
1. The approximation of risk or loss can be valuable. However, the minimum loss cannot ensure maximum accuracy in the classification task. In general, active learning and accuracy is essential. Let me know your answer. 
2. What's the primary reason for reducing the variance of active sampling in the AES. Updating procedures or architectures. Can you provide more solid answers?           
3. Does the network's capacity for \pi influence the performance of AES?   


limitations: There is no critical flaw in this paper. The section of theoretical aspects can be improved in the presentation. Experiments are limited, related the selection of \pi.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed a new method, ASE, for label-efficient model evaluation. ASE uses a surrogate-based estimation approach that interpolates the errors of points with unknown labels. ASEs actively learn the surrogate. The authors proposed a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. Experiments show that ASEs offer greater label-efficiency than SOTA. 

strengths_and_weaknesses: 1) The authors proposed a novel algorithm based on surrogate estimation for active testing.
2) The authors maintain a high quality of writing and presentation.

questions: 1) Could we use the difference between expected accuracy and current accuracy as a metric, besides squared error, as the metric to evaluate different algorithms? If so, how is the performance compared to other active learning algorithms, e.g., BALD and Entropy?
2) Could we evaluate these algorithms in a batch mode setting? For each algorithm, we can specify the batch size $B$ and select top-$B$ in each round. We believe it is very interesting to extend XWED to batch mode setting, similar to the way in which BatchBALD extends BALD.
limitations: could the authors also discuss the computation complexity in the paper for different algorithms?
review_title: null

Meta-review: metareview: This paper tackles the active testing problem and proposes a novel estimator (ASE) of the expected loss based on a surrogate function, and a novel acquisition function (EWED) to minimize the error.  SOTA performance is reported, and some ablation study is conducted.

Reviewers raised concerns about novelty, high dependence on the quality of the surrogate, and missing baselines, which the authors have well addressed and convinced the reviewers.  Although the proposed methods are small modifications from existing methods, by using the surrogate, they significantly improve the existing methods in the experiments.
confidence: Certain
Decision: Accept