Title: Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice
Venue: NeurIPS 2023
Accepted: yes
Keywords: active learning

active testing
Abstract: In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, "testing'', which often refers to the model evaluation process of using empirical risk to estimate the intractable true generalization risk, also requires data annotations. The annotation cost for "testing'' (model evaluation) is under-explored. Even in works that study active model evaluation or active testing (AT), the learning and testing ends are disconnected. In this paper, we propose a novel active testing while learning (ATL) framework that integrates active learning with active testing. ATL provides an unbiased sample-efficient estimation of the model risk during active learning. It leverages test samples annotated from different periods of a dynamic active learning process to achieve fair model evaluations based on a theoretically guaranteed optimal integration of different test samples. Periodic testing also enables effective early-stopping to further save the total annotation cost. ATL further integrates an "active feedback'' mechanism, which is inspired by human learning, where the teacher (active tester) provides immediate guidance given by the prior performance of the student (active learner). Our theoretical result reveals that active feedback maintains the label complexity of the integrated learning-testing objective, while improving the model's generalization capability. We study the realistic setting where we maximize the performance gain from choosing "testing'' samples for feedback without sacrificing the risk estimation accuracy. An agnostic-style analysis and empirical evaluations on real-world datasets demonstrate that the ATL framework can effectively improve the annotation efficiency of both active learning and evaluation tasks.

Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice

Abstract

In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, 'testing', which often refers to the model evaluation process of using empirical risk to estimate the intractable true generalization risk, also requires data annotations. The annotation cost for 'testing' (model evaluation) is under-explored. Even in works that study active model evaluation or active testing (AT), the learning and testing ends are disconnected. In this paper, we propose a novel active testing while learning (ATL) framework that integrates active learning with active testing. ATL provides an unbiased sample-efficient estimation of the model risk during active learning. It leverages test samples annotated from different periods of a dynamic active learning process to achieve fair model evaluations based on a theoretically guaranteed optimal integration of different test samples. Periodic testing also enables effective early-stopping to further save the total annotation cost. ATL further integrates an 'active feedback' mechanism, which is inspired by human learning, where the teacher (active tester) provides immediate guidance given by the prior performance of the student (active learner). Our theoretical result reveals that active feedback maintains the label complexity of the integrated learning-testing objective, while improving the model's generalization capability. We study the realistic setting where we maximize the performance gain from choosing 'testing' samples for feedback without sacrificing the risk estimation accuracy. An agnostic-style analysis and empirical evaluations on real-world datasets demonstrate that the ATL framework can effectively improve the annotation efficiency of both active learning and evaluation tasks.

1 Introduction

Labeled data are essential for supervised learning in both model training and evaluation. Active learning (AL) provides a promising direction to reduce the human annotation cost by constructing a smaller but more effective labeled dataset for training purposes [24, 27, 12, 5, 1, 8]. However, AL only partially addresses the human annotation cost as the cost of constructing a labeled dataset for model testing has been overlooked. In reality, the annotation budget could easily be drained by building large test datasets that most modern AL methods rely on to evaluate the model performance and determine the stopping criterion for learning. In contrast to AL, active model evaluation or active testing (AT) focuses on actively selecting the testing data. Few existing efforts develop unbiased risk estimation techniques for label-efficient model evaluation [20, 14]. However, these methods are designed to evaluate fixed models that have been fully trained, making them ill-suited for evaluating an actively learned model, which is constantly updated and inadequately trained during most parts of the learning process. To address these fundamental challenges and realize label-efficient learning in practice, we propose to integrate AT with AL in a novel and efficient learning framework, referred to

as ATL. The ATL framework systematically addresses the unique active testing challenges arising from evaluating an actively learned model that is under-trained and constantly evolving. It leverages an interactive learning-testing-feedback process to better control the overall labeling budget, which not only achieves efficient evaluation of the AL model but also ensures faster convergence in model training. Through periodical testing, ATL gains useful insights to terminate the learning process as early as possible that avoids allocating unnecessary labeling budget.

Incorporating AT into AL is a highly non-trivial task. Unlike AL, AT has two inter-connected objectives: i) design of an unbiased risk estimator to quantify the model performance, and ii) design of a sampling strategy to select informative test samples. Achieving both objectives simultaneously under the AL setting with a constantly evolving and under-trained model is far more challenging than existing active model evaluations that assume a fixed and adequately trained

2

1

0

1

2

3

2

1

0

1

2

3

0

4

8

12

16

20

Test Phase

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Estimation Error

1e

2

ARE

Random Testing

Our Method

model. For the first objective, its challenge stems from the potential biased active test samples brought by the second objective. We need an unbiased risk estimator to ensure the fairness of the evaluation. An unbiased risk estimator can be achieved by importance sampling or its variants [25, 20, 21, 22, 19, 16]. Recent active testing works also employ expectation analysis and propose other weighting mechanisms [10, 14]. However, all these works assume that the model being evaluated is fixed and already well-trained [20, 14]. Therefore, these estimators are not designed to support a dynamic AL setting, where the model continues to evolve when learning from newly labeled samples. Figure 1 shows that both the standard estimator ( i.e., random testing) and direct adaptation of an existing unbiased estimator (ARE) [20] fail to provide an accurate and consistent model evaluation in the more challenging AL setting. As for the second objective, different from sample selections in AL, the testing selection needs to be compatible with the unbiased risk estimator. It is essential to ensure asymptotic convergence that guarantees an unbiased model evaluation while reducing the variance that can improve the convergence speed and provide a stable signal to terminate the learning process as early as possible.

Conducting AT along with AL enables ATL to seamlessly connect the active training and testing sides, which allows them to communicate with and guide each other within a well-integrated learning process. To this end, the proposed ATL framework leverages an active learning-testingfeedback (LTF) loop, which largely resembles real-world human learning. In particular, within the LTF loop, the active testing results are kept after each active learning round, which mimics the quizzes in the human learning setting. We refer to them as active quizzes . As shown in Figure 2, the

Active

Quiz Records

(

)

Testing

Set

Training

Set

Unlabeled

Pool

Oracle

Final Estimate

(

)

Early Stop?

Test Selection

(

)

Quiz Review

AL

Selection

Active

Feedback

Test

Integrate

Active

Tester

active learner acts as a student while the active tester acts as a teacher. Both the student and the teacher select data from the unlabeled pool and get their labels from the oracle. The student attempts to use the labeled data to pass the quizzes provided by the teacher. On the other hand, the teacher provides a fair quiz to evaluate the student's performance. Meanwhile, the teacher also sends back the keys of some quiz questions, which are referred to as active feedback , to help the student learn better.

Given a dynamic AL process, multiple active quizzes will be created at different learning phases. ATL forms an integrated risk estimator by aggregating all the quizzes. Our theoretical analysis guarantees that the integrated risk estimation converges to the true risk asymptotically. The aggregation strategy also minimizes the variance, which ensures faster convergence. Furthermore, through active feedback, the AL model can benefit from training with a small proportion of the testing samples (as feedback) after each quiz while maintaining the asymptotic convergence condition of the risk estimation. Finally, the integrated risk estimator provides an unbiased estimate of the model performance with a small variance, which provides a convenient way to determine whether the AL model has converged.

This can further save the labeling budget by stopping the learning early. We may also combine the estimated risk with other available information from the unlabeled data to create more systematic early-stopping criteria. Our main contributions are summarized as follows:

the first ATL framework that builds an active learning-testing-feedback loop to mimic human learning for achieving label-efficient and dynamic evaluation of actively learned models,

a theoretically sound integrated risk estimator that combines active quiz results and minimizes the variance of the difference between the estimated and true risks,

an innovative active feedback process to further improve learning effectiveness without extra labels,

a systematic early stopping criterion that combines integrated risk estimation with information of unlabeled data to terminate AL process early.

We conduct experiments on both synthetic and real-world datasets to show the improved testing performance of the proposed ATL framework.

2 Related Work

Active model evaluation. Actively testing the model performance in a label efficient way is relatively under-explored. Among the few excising efforts, model testing is treated independently from model training, which is unrealistic in most practical settings. The problem of active risk estimation (ARE) was first formulated in [20], which used importance sampling to construct an unbiased risk estimator. Then, the optimal test sampling distribution is derived from an optimization problem that minimizes the variance of the risk error. A similar process has been used to estimate more complex loss measures [21] or perform active model selection [23]. However, such risk estimator is no longer unbiased nor label-efficient when an AL process is involved that samples from the same unlabeled pool for model training. Farquhar et al. propose two risk estimators to cancel the statistical bias by active data sampling strategy in a data-efficient manner [10]. However, their proposed estimators could not deal with the overfitting bias making them less suitable for model evaluation.

In a more recent work, Kossen et al. propose to use the unbiased risk estimators to perform active testing [14]. The difference from [20] is that the selection order of the testing points is considered, and the weights are defined for selection indices instead of a single testing proposal distribution. In this way, the testing proposal can be evaluated at each round, making the setting more realistic. However, the optimal proposal is still impossible to obtain. To approximate the optimal proposal, a surrogate model is needed, which is similar to the introspective risk model in [20]. The surrogate model idea is further extended in [15], where the unbiased risk estimator is replaced by the predicted loss from the surrogate model. Yilmaz et al. propose to replace the commonly used importance sampling with Poison sampling, which better stabilizes the test samples [29]. A different line of work leverages stratified sampling to first stratify the unlabeled pool and then conduct efficient sampling over strata in an importance sampling manner [13, 3, 30]. However, all those work is limited to offline test sampling in which the test proposal is fixed during the sampling process and could not benefit from prior knowledge such as the previously labeled data.

Early stopping in AL. Early stopping strategies in AL have been sparsely investigated over years [2, 7, 6, 31, 26, 18]. Most methods only consider the learning process and do not consider a fully connected learning-testing loop. Some have shown that the stopping criteria based on unlabeled information are more effective than those based on labeled information. This inspires us to combine the proposed integrated risk estimation with unlabeled data information to form a systemic early stopping criterion to further reduce the labeling budget.

3 The ATL Framework

Problem setting. We formulate the ATL framework under a novel and fully integrated learning and testing-while-it-learns setting. W.l.o.g., we assume that all data are generated from an underlying distribution, and we could obtain the label y through the true labeling function: y = h ( x ) provided by the human annotator (note that unlike the predictive model f θ ( · ) , the true labeling function is not governed by the parameter θ and it might be non-realizable). Similar to a traditional pool-based AL setting, we have a small labeled set S L = { ( x , y ) } N L (whose labels are revealed). As for the unlabeled pool, we only have access to the features: S U = { x } N U . We further assume that the unlabeled pool is sufficiently large so that active sampling will not change the pool distribution since only limited labels will be revealed. That is, p ( x , y ) remains fixed during AL. Our primary task couples two objectives: AL and model evaluation at the same time. In this setting, the goal of AL is to

create a labeled set by selecting samples to label and achieve the best performance. The performance of the model is indicated by the expected error when generalized to the underlying data distribution, which is often noted as the risk R . As discussed earlier, existing works on label-efficient model evaluation assume that the model being evaluated is already well-trained and kept fixed. They are inadequate to handle a dynamic active learning scenario, which is the focus of ATL. Given some loss function as the risk measurement L θ : f θ ( x ) × y → R , our task of testing the model reduces to evaluating the following expectation:

However, directly evaluating (1) is infeasible as the true density function p ( x , y ) is unknown. So empirically, we have to adopt an approximation. In the pool-based learning setting, our approximation resorts to the expected empirical risk over the pool distribution D pool .

3.1 Overview: Active Risk Estimation During AL

In each AL iteration, we select to label x ∗ based on a sampling criterion p sample ( x ) to improve the model. We design the framework to be AL-agnostic, meaning that most standard AL sampling strategies can be applied here without further assumptions. Normally, the evaluation of the AL model is performed on a hold-out test set either at the end or during different to-learn data selection rounds of AL. However, the labeling cost of the test set is often times ignored. To achieve a more efficient evaluation framework, we propose to have the active risk estimation ( i.e., active testing) process intertwined with AL. Specifically, we use time stamps t to denote each stage where we perform active testing. At time t , we use testing proposal q t ( x ) to select samples, Q t = { x (1) t , ..., x ( n t ) t } , then label them to evaluate the model. We name this novel evaluation style active quiz . Active quiz is motivated by the dynamic nature of the AL process, which allows us to have an instant evaluation of the current model during AL. We also show that the quizzes can further be conveniently combined into a final exam for a comprehensive evaluation. In this way, we essentially divide a given budget for evaluation into small batches, which makes it possible to stop the evaluation before reaching the total budget by leveraging the signal from past quizzes.

The purpose of having the active quiz is to achieve a good estimate of the current model performance, which is not realizable in most existing AL frameworks. We use the risk R to denote the performance. R is dependent on the specific model, thus at each step we are trying to estimate the true risk of the model at t : R ( f t ( · )) 2 . Using Q t , we can have an estimate ̂ R Q t ( f t ) . We name the evaluation after the last quiz Q T as the final exam because it includes all the quiz samples: {Q 1 , ..., Q T } . We use ˜ R {Q 1 ,..., Q T } ( f T ( · )) to denote the final performance of active learning model. Note that ˜ R is different from each ̂ R t because: (i) the evaluation sets contain all the previous quiz data, and (ii) it only evaluates the final model, f T . Intuitively, using {Q 1 , ..., Q T } should be more preferable than using a single Q t because more data samples are available to better represent the data distribution. However, the combination is not straightforward because each quiz is sampled using the optimal selection proposal at that time. We propose a principled way to combine multiple quizzes to achieve an unbiased ˜ R with theoretical guarantees. We further formulate an active feedback strategy to integrate the dual objectives of learning and testing and provide a theoretical underpinning to utilize some labeled test samples to improve the model's generalization capability.

3.2 Unbiased Low-variance Estimation of Model Risk

For a passive learning setting, where the model is fixed, the risk can be directly estimated through an importance weighting sampling procedure, such as the ARE scheme developed in [20]. Let q ( x ) denote the test sample selection proposal, then the risk estimate can be defined as

We can use the central limit theorem to show that ̂ R n,q is an unbiased estimation of R . Furthermore, their difference follows a zero-mean Gaussian asymptotically:

When n is large, we also know that n Var [ ̂ R n,q ] n →∞ - - - - → σ 2 q . Then, the optimal q ( x ) minimizes the variance of the estimate. Using variational analysis, the optimal q ( x ) is

which minimizes the expected squared difference between the estimate and the true risk (also the variance of the asymptotic Gaussian). Since the pool distribution remains fixed when only a small number of instances are sampled for AL, we can keep p ( x ) as a uniform distribution over the pool. Since the importance weighting proposals are known up to a normalization factor, we have p ( x ) = 1 N U where N U is the pool size. To optimize model evaluation in an active learning setting, we can evaluate the model at each time stamp t in the following manner: First, we obtain an active testing proposal, denoted as q t , by substituting (4) with f θ t . Next, we obtain the active quizzes denoted as Q t ∼ q t . We then evaluate the model's performance at time t using (2).

However, there are two main limitations to this approach. First, the test proposal given by (4) contains an unknown quantity, the true risk R . Previous studies have attempted to address this issue by predicting the true risk from the pool. However, in section 3.3, we demonstrate that commonly used true risk predictors are not accurate, particularly in the early stages of active learning. To address this, we propose a multi-source true risk predictor. Second, the current test setting does not allow for sharing of data between previous and current quizzes due to the use of different active proposal distributions. This inefficiency not only wastes data, but also reduces the stability of the model evaluation, as the test samples in each quiz may be too small to provide a robust evaluation. Therefore, in section 3.4, we propose an estimator for the active learning setting that can integrate quizzes sampled from different proposal distributions while remaining optimal.

3.3 Intermediate Estimate of the True Risk

In the optimal test sample selection proposal given by (4), a critical component is the true risk, which is unknown and needs to be estimated. A straightforward solution is to leverage the current model [20] or some proxy model [14] to predict the potential risk over the entire pool as R θ . However, the following analysis shows that this is equivalent to the model uncertainty in the classification setting.

Proposition 1. Under the classification setting and when a standard (i) 0 -1 loss or (ii) cross-entropy loss is used, R θ is equivalent to measuring the model uncertainty.

More details of the proposition are provided in Appendix B.2. The above proposition shows that R θ can only capture the model uncertainty instead of the true risk. When the labeled training samples are limited, it is possible that the model uncertainty can not reflect the level of true risk. We propose to combine the current training risk, the model uncertainty, and the current test risk (using Q t that we have) to get an aggregated multi-source estimate of the true risk:

The multi-source estimate can more effectively avoid overestimating the risk when only using the testing information and underestimating the risk when only using training information. By aggregating these two sides of information, neither underfitting or overfitting of the model will harm the estimation to a significant extent. The effectiveness of the multi-source aggregated risk estimation has been demonstrated in our empirical evaluation.

3.4 Active Quiz Integration for Final Risk Estimation

We have shown the optimal way to select test samples to evaluate a fixed model in the passive learning setting. However, we still need to extend to our scenario where there are multiple stages and AL while AT are intertwined. We propose to combine the quiz results by assigning a set of weights v t on them. We will show that the proposed v t is optimal given the fixed quizzes. Let's denote a sequence of independent risk estimates (treated as random variables) as ̂ R = ( ̂ R Q 1 ( f T ) , .. ̂ R Q T ( f T ) ⊤ ) , where each ̂ R Q t ( f T ) asymptotically converges to the true risk R ( f T ) . We should also note that we are using each ̂ R Q t to evaluate the same model f T , not using the actual values of the estimate at time t (which is the evaluation of f t ). That is, we always utilize previously collected sets of quiz samples to

evaluate the current model.

The variance σ 2 t ( f T ) is given by

We have defined the asymptotic Gaussian variance term when we use the quiz set Q t from time t to evaluate the final model f T . Let C t = 1 /σ 2 t ( f T ) denote the model confidence of f T when evaluated on a test set that follows q t . Next, we formally show that if the model f T is more confident about q t , then Q t should take a larger weight in the integrated final evaluation result.

Theorem 1. Given fixed {Q 1 , ..., Q T } , the weighted combination ˜ R = ∑ T t =1 v t ̂ R Q t where v t = C t ∑ T t =1 C t is the optimal estimator constructed by all samples in {Q 1 , ..., Q T } .

Remark. The weight v t depends on how accurate the estimate using each quiz set is. If we assume that the model becomes more accurate throughout the entire process, then we know that ˜ R T -1 should be closer to R than ˜ R T -2 , thus the weight on the most recent one is the largest. However, we still do not know R and have to estimate the weight. Here we adopt a similar way as estimating R θ to estimate the difference terms C t using the expected loss and ̂ R Q t ( f T ) : 1 C t =

1 n t ∑ n t i =1 ∑ y p ( x ( i ) ) q t ( x ( i ) ) [ L ( f T ( x ( i ) ) , y ) -̂ R Q t ( f T )] 2 p ( y | x ( i ) ) . With the optimal v ∗ t , we define the final estimate result as ˜ R = ∑ T t =1 v ∗ t × ̂ R Q t . In practice, if we do not pre-set the length of the entire AL process, we can treat each stage as the final exam. After each AL sampling round, we perform an active testing round as a quiz. Then we use the integration method above to get the final estimate. This way, we will always have the optimal evaluation of the model given all the test samples that have been selected.

3.5 Active Feedback: Improve Model Learning Without Sacrificing Testing Accuracy

The proposed final risk estimation can provide label-efficient evaluation of the model performance. We can further improve the label efficiency on the the AL side since some of these labels can also be used for model training. Assume at the t -th quiz, we have sampled and labeled a set of test data Q t according to the test proposal q t ( · ) . Since Q t is also a labeled dataset, the active learner could leverage it as a source of feedback information to facilitate learning. In the proposed active learning-testing setting, the number of labeled samples for learning/training N L and testing N T both contribute to the limited overall annotation budget. Since learning and testing need to be sample-efficient, it is reasonable to assume that N L and N T are of similar magnitudes instead of having N L ≫ N T as in the existing active model evaluation works. We further denote S FB as the feedback set, which contains samples from the collection of quizzes. The active feedback process can be formally defined as a subset selection problem that optimizes a joint objective of learning and testing.

where C is a parameter to balance the two objectives: ( I ) = R ( f θ | ( S L ∪S FB ) ) for learning and ( II ) = || R -˜ R ( { Q 1 ,...,Q T }\S FB ) || for testing.

Theorem 2. Both the learning and testing tasks have the same label complexity, which is in the order of O (1 / √ n ) . As a result, (1) the joint objective has an overall label complexity of ( O (1 / √ N L + N FB ) + O (1 / √ N T -N FB )) ; (2) the balancing parameter C is in the order O (1) .

Remark. The Theorem reveals that it is possible to achieve an optimal balance between ( I ) and ( II ) by choosing a suitable S FB such that we can further improve the model learning performance while maintaining the risk estimation quality from our quizzes-testing process. It provides a foundation to justify the benefit of active feedback. It is worth to note that the optimal subset selection problem is usually NP-hard, but we can still draw a useful conclusion that an appropriate subset S FB can potentially improve the combined learning-testing objective rather than having S L and S T = { Q 1 , ..., Q T }

always separated. The theorem can also be more intuitively interpreted by drawing analogy with human learning, who automatically incorporate knowledge or partial knowledge at inference time. In contrast, a typical supervised-learned ML model is frozen at evaluation/testing time. Thus, we need to explicitly add feedback so that the model can effectively improve the learning performance by getting new training samples.

Following the above theoretical result, we propose a selection proposal q FB ( · ) for S FB based on the following learning-testing intuitions. In standard AL, we should select samples that are considered to be the most informative and diverse ones. From the informativeness perspective, samples with high losses could be challenging for the current model. From the diversity perspective, the newly added samples should be different from the current labeled training set, which can be achieved using a diversity metric, such as d ( S L , x ) = √ x ⊤ A -1 L x , where A L = ϵ I + ∑ z ∈S L zz ⊤ [11], where ϵ > 0 .From the evaluation perspective, removal of feedback samples should have as little impact to the risk estimation as possible. Since the test samples are chosen according to q ( x ) , it is less harmful to choose a feedback sample with a large q ( x ) as similar testing samples are likely to be selected into the test data. To this end, we propose the following feedback selection strategy:

where η is a scaling parameter. We construct S FB after each active quiz round. The samples are added in S L to re-train the model, and the risk estimation is also re-weighted after the removal.

3.6 Early Stopping with Instant Risk Estimations

In the ATL framework, the unbiased risk estimation after each AL step can be used to construct an effective convergence indicator to support early stopping of AL. In particular, we propose to use the change of a moving average of active risk estimations:

Prior works show that leveraging the unlabeled data works well empirically to stop AL early [2, 6]. To this end, we propose to further augment ∆ ˜ R t with stabilized predictions (SP) from the unlabeled data [6], where SP = 1 -mean x ∈U (∆ y ( x )) .

4 Experiments

The proposed ATL framework is generic and can be applied to any existing active learners. In our experiments, we choose two commonly used models, Gaussian Processes (GP) and neural networks, to demonstrate the effectiveness of the proposed work. The GP is applied to a 2D synthetic dataset to help us understand the important behavior of the proposed test sampling strategy and how active feedback benefits the entire learning process. The neural network model is applied to relatively larger scale image datasets, including MNIST, FashionMNIST and CIFAR10 to demonstrate the practical performance of the proposed framework. As for the AL strategy, ATL is designed to be AL-agnostic. To demonstrate the general applicability, the main results are obtained using uncertainty ( i.e., entropy) based sampling as the AL strategy given its great popularity in many AL models. In Appendix C.2.3, we show that ATL can be easily integrated with a wide range of commonly used AL algorithms.

4.1 Experimental Settings

For synthetic experiments, we generate 2,500 data samples based on a moon-shaped distribution with two smaller inverted moon-shaped clusters at two remote corners. The dataset is shown in Figure 1 (a). We add the smaller clusters because the learning performance is usually closely related to how AL explores the unknown regions in the feature space and the designed shape allows us to visualize testing and feedback samples to evaluate their effectiveness. Moreover, we create two levels of imbalance in the dataset, the intra-class imbalance and the inter-class imbalance, to simulate an imbalanced class distribution. For the real-world experiments, we use the cross-entropy loss for risk evaluation. When we compare with the true risk R , we use the average evaluation results of the model on a large hold-out subset of the dataset (10,000 data samples) to represent R . The hold-out test set does not interact with the AL or AT processes, thus is considered a fair evaluation. In real-world experiments, we adopt the same procedure with the total pool containing 30,000 data samples.

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:3.9e-03

AL Sample

AL Sample

AT Sample

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

(a) AL sampling (quiz 9)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:5.6e-03

AL Sample

AL Sample

AT Sample

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

(b) Random (quiz 9)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:8.4e-04

AL Sample

AL Sample

AT Sample

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

(c) ATL-NF (quiz 9)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:1.2e-03

AL Sample

AL Sample

AT Sample

Feedback Sample

0.16

0.24

0.32

0.40

0.48

0.56

0.64

0.72

(d) ATL (quiz 9)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:4.4e-03

AL Sample

AL Sample

AT Sample

0.24

0.30

0.36

0.42

0.48

0.54

0.60

0.66

0.72

(e) AL sampling (quiz 18)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:-1.2e-04

AL Sample

AL Sample

AT Sample

0.24

0.30

0.36

0.42

0.48

0.54

0.60

0.66

0.72

(f) Random (quiz 18)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:8.4e-08

AL Sample

AL Sample

AT Sample

0.24

0.30

0.36

0.42

0.48

0.54

0.60

0.66

0.72

(g) ATL-NF (quiz 18)

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:2.1e-05

AL Sample

AL Sample

AT Sample

Feedback Sample

0.16

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

(h) ATL (quiz 18)

In each test sampling round, we compute over the entire unlabeled pool according to (4). We use a deterministic way of summing over all classes based on the posterior distribution predicted by the model. The multi-source estimate only utilizes labels of the training samples and previously selected test samples along with the predictions for unlabeled samples. Later in the active feedback stage, if a test sample is selected to be added to the training set, we remove it from the testing set. For test and feedback selections, we sequentially sample n t times to obtain each batch. The testing process does not involve re-training the model, thus the batch mode has no effect.

4.2 Synthetic Data Experiments

In Figure 3, we show the distribution of test samples selected using (1) AL sampling, (2) random sampling, (3) ATL w/o feedback, and (4) ATL at an early stage ( i.e., quiz 9) and later stage ( i.e., quiz 18) of the learning process. We observe that using AL sampling for test point selection has less accurate risk estimation performance due to highly biased sample selection near the decision boundary. Random sampling, on the other hand, follows the true data distribution, so it mainly selects from the central region, where the data is dense. Our proposed test sampling focuses on the most representative regions as random sampling does, but it also covers other interesting regions with relatively low density ( e.g., the right edge where the minority blue class is mainly located), and the model evaluation reaches the highest accuracy in the early phases of learning.

In Figure 3d, we plot the feedback with the same annotation cost to demonstrate its learning-testing trade-off. By providing part of the testing data as effective feedback, the estimation error is slightly increased. However, the sacrifice in model evaluation accuracy pays off from the training side. The feedback helps the AL model build a more representative training set and learn a more reasonable decision boundary than without feedback. It allows the AL model to gain a greater advantage in the early stage of training. Figures 3e-3h show the test data distribution in a later stage at quiz 18. The proposed testing selection is the only one that picks samples to test the critical but low-density regions. More importantly, Figure 3h shows that learning with feedback is most successful as it acquires an informative training dataset and a near-optimal decision boundary in the end.

4.3 Real-Data Experiments

The main results of real-world experiments consist of three parts: (1) risk estimation during standard AL (without active feedback), (2) AL performance and risk estimation with active feedback, and (3) AT-based early stopping. In this section, we focus on showing the results from the first two and leave the details of part (3) in the Appendix. In the AL without feedback case, we compare the estimation error results of ATL-NF (no feedback) with ARE quizzes integrate [20] and the adapted Active Testing [14] and Active Surrogate Estimation [15] (referred to as AT integrate and ASE integrate). We adapt the AT and ASE baseline to use in the AL setting by adjusting the pool size N at each round and including the selected training loss to maintain the unbiased estimator ˜ R LURE . We also need to train a surrogate model using fewer samples than normal (we use a separate NN model re-trained every few rounds). Because the AL model and the surrogate models are both severely under-trained in the early stage of AL, and that the training samples must also be considered in ˜ R LURE , the estimation

error is usually significant in the AL setting. In the active feedback experiments, we compare the proposed ATL with ATL-NF and ATL-RF (random feedback).

AL-AT results. We use squared difference between the estimated risk and the true risk as well as the variance of the squared difference to measure the estimation error instead of the variance of the estimate itself because each time we repeat the experiment the model is different. This is also a difference caused by studying the AL process, instead of evaluating a fixed model. From Table 1, we see that ATL-NF has a much lower error of estimation than single ARE quizzes and AT/ASE integrate. The ARE quizzes put all testing samples without adjusting the importance weights, thus will result in a bigger estimation error. The drawback of the AT method is that the surrogate model is not a very reliable reference and the evaluated model changes over time. Because it selects samples based on a max-score criterion instead of importance sampling, the change of model may affect the unbiased estimate. The ASE integrate has the same issue of relying on a good surrogate model. However, the risk estimation of ASE is also based on the surrogate model predictions, which is consistent with the test sample selection. Thus, it outperforms AT integrate most of the time.

Active feedback results. We compare traditional AL and active learning-testing with feedback. We use both the cross-entropy loss as the metrics for the model performance. In Table 2, we see that the proposed feedback and random feedback can both help the model reach a smaller testing loss than pure AL with the same total number of labels. The initial training set contains 500 labels, while each of the 20 AL rounds adds 500 labels. AT then samples 100 points after each AL round, and the active feedback sends 50 labels back to training. Because of the importance sampling with replacement nature of the process, the feedback approaches actually use fewer total labels in the end. We also study the impact of feedback on the estimation error in Table 3. In the early stage of AL, it is obvious that random feedback is harmful to the estimation error. The proposed approach also suffers from the increase of the estimation error. However, as the AL goes on, our proposed feedback selection can maintain a low level of estimation error, while improving the model performance. Combining these

benefits, we can potentially save the labeling budget for both learning and evaluation purposes. We report further ablation studies on the feedback proposal in Appendix C.

We also provide a study on the size of the feedback set. As mentioned in the proof for Theorem 2, we keep the size of feedback simple in this work. This is to be consistent with our theoretical analysis and the experiments show that the active feedback process is helpful in this generic setting. Further details about extending this have been discussed as part of the future directions. However, even in the simple setting of fixed feedback size, we can see that the learning and testing performances do not consistently and monotonically change with respect to the feedback size. From Tables 4 and 5 below, we can see that in general, model risk (learning performance) is better when we use a larger feedback size, but at the same time the estimation error (testing performance) may become much worse. The model risk on CIFAR10 behaves differently with the feedback size, probably because the model performance is not good enough and adding difficult samples in this stage does not necessarily help with the generalization ability.

5 Conclusion

We address a real-world challenge for sample-efficient learning, where valuable labels from human experts that can be used for testing/evaluation are also scarce. We propose an ATL framework that builds an active learning-testing-feedback loop to achieve a label-efficient evaluation of an AL model on the fly, with the potential to further improve the model performance without extra labels and opening up to the creation of systematic early stopping criteria. We theoretically prove that ATL has an unbiased and label-efficient estimator and provide an analysis that shows how the label complexity dependencies are maintained through active feedback. The experiments show that ATL optimizes the number of human labels needed in learning by simultaneously acting as a fair referee and an educative teacher.

References

Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Journal of Computer and System Sciences , 75(1):78-89, 2009.

Garrett Beatty, Ethan Kochis, and Michael Bloodgood. The use of unlabeled data versus labeled data for stopping active learning for text classification. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC) , pages 287-294. IEEE, 2019.

Paul N Bennett and Vitor R Carvalho. Online stratified sampling: evaluating classifiers at web-scale. In Proceedings of the 19th ACM international conference on Information and knowledge management , pages 1581-1584, 2010.

Andrew C Berry. The accuracy of the gaussian approximation to the sum of independent variates. Transactions of the american mathematical society , 49(1):122-136, 1941.

Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning , pages 49-56, 2009.

Michael Bloodgood and John Grothendieck. Analysis of stopping active learning based on stabilizing predictions. arXiv preprint arXiv:1504.06329 , 2015.

Michael Bloodgood and K Vijay-Shanker. A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. arXiv preprint arXiv:1409.5165 , 2014.

Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Ningshan Zhang. Adaptive region-based active learning. In International Conference on Machine Learning , pages 2144-2153. PMLR, 2020.

Carl-Gustav Esseen. On the liapunov limit error in the theory of probability. Ark. Mat. Astr. Fys. , 28:1-19, 1942.

Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665 , 2021.

Claudio Gentile, Zhilei Wang, and Tong Zhang. Achieving minimax rates in pool-based batch active learning. In International Conference on Machine Learning , pages 7339-7367. PMLR, 2022.

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 , 2011.

Namit Katariya, Arun Iyer, and Sunita Sarawagi. Active evaluation of classifiers on large datasets. In 2012 IEEE 12th International Conference on Data Mining , pages 329-338. IEEE, 2012.

Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sampleefficient model evaluation. In International Conference on Machine Learning , pages 5753-5763. PMLR, 2021.

Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Anurag Kumar and Bhiksha Raj. Classifier risk estimation under limited labeling resources. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages 3-15. Springer, 2018.

Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations , 2020.

Lutz Prechelt. Automatic early stopping using cross validation: quantifying the criteria. Neural networks , 11(4):761-767, 1998.

Christoph Sawade. Active evaluation of predictive models . Universitätsverlag Potsdam, 2013.

Christoph Sawade, Niels Landwehr, Steffen Bickel, and Tobias Scheffer. Active risk estimation. In ICML , 2010.

Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active estimation of f-measures. Advances in Neural Information Processing Systems , 23, 2010.

Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. Advances in neural information processing systems , 25, 2012.

Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012.

Burr Settles. Active learning literature survey. 2009.

Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics , 2(1):54-60, 2010.

Andreas Vlachos. A stopping criterion for active learning. Computer Speech & Language , 22(3):295-312, 2008.

Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neural networks (IJCNN) , pages 112-119. IEEE, 2014.

B. L. Welch and H. W. Peers. On formulae for confidence points based on integrals of weighted likelihoods. Journal of the Royal Statistical Society. Series B (Methodological) , 25(2):318-329, 1963.

Emine Yilmaz, Peter Hayes, Raza Habib, Jordan Burgess, and David Barber. Sample efficient model evaluation. arXiv preprint arXiv:2109.12043 , 2021.

Tiancheng Yu, Xiyu Zhai, and Suvrit Sra. Near optimal stratified sampling. arXiv preprint arXiv:1906.11289 , 2019.

Jingbo Zhu, Huizhen Wang, and Eduard Hovy. Multi-criteria-based strategy to stop active learning for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1 , pages 1129-1136, 2008.

Appendix

Table of Contents

Appendix Overview

Organization. The Appendix is organized as follows. We first provide a summary of notations in Appendix A. Then, we present the details of the theoretical analysis in the main paper in Appendix B. We further provide additional experimental results in Appendix C, including a synthetic visualization for different feedback approaches in Appendix C.1 and more detailed real-world experiments in Appendix C.2. The hardware information is presented in Appendix D. We discuss the limitation, future direction, and social impact of the proposed work in Appendix E. Th algorithm block for the pseudo code and the the link to the source code are presented in Appendix F.

A Summary of Notations

B Proof and Additional Analysis of Main Theoretical Results

B.1 Proof of Theorem 1

Proof. We start by presenting the asymptotic convergence of the active risk estimator and the solution for the optimal testing selection proposal q ∗ ( x ) . From [20], we know that using the risk estimator ˆ R n,q we would get an unbiased estimate of the true risk R because it is essentially an importance sampling based estimator. Then from the central limit theorem, ˆ R 0 n,q = ∑ n i =1 w ( i ) l ( i ) and W n = ∑ n i =1 w ( i ) are asymptotically normally distributed with

Then, with the multivariate delta method, we know that if Y n = ( Y n 1 , ..., Y nk ) is a sequence and √ n ( Y n -µ ) n →∞ - - - - →N (0 , Σ) , then

Here the function is g ( x, y ) = x y with x = 1 n ˆ R 0 n,q and y = 1 n W n . The result is

where σ 2 q = ∫ p ( x ) q ( x ) (∫ [ L ( f ( x ) , y ) -R ( f )] 2 p ( y | x ) dy ) p ( x ) d x .

Then, the optimal test proposal is obtained by minimizing σ 2 q . By introducing a Lagrange multiplier β for the constraint ∫ q ( x ) d x = 1 , we have

Thus, we have q ∗ ( x ) ∝ p ( x ) √ ∫ [ L ( f ( x ) , y ) -R ( f )] 2 p ( y | x ) d y .

Now, we provide the detailed proof for Theorem 1. As shown in Section 3.4, ̂ R satisfies

Next, we apply the multi-variant delta method. Define g : R T → R , g ( ̂ R ) = ∑ T t =1 v t ̂ R Q t . Then, we have ▽ g = ( v 1 , ..., v t ) ⊤ . Given the diagonal covariance matrix, the final variance is:

When we perform the 'final exam' estimation after gathering all quizzes {Q 1 , ..., Q T } , the other factors including testing proposals are fixed. We analyze the optimal solution for v t by constructing the Lagrangian objective σ 2 T + γ ( ∑ t v t -1) (where γ is a Lagrangian multiplier). By taking the derivative w.r.t each v t along with the Lagrangian, we have

which leads to v t = C t ∑ T t =1 C t .

The Corollary below provides an alternative view of Theorem 1.

Corollary 1. If we do not change individual q t but still combine all available test samples, then adjusting their importance weight by w ′ ( i ) t = v t × w ( i ) t gives the optimal estimator.

Proof. In the alternative view, we have:

where w ( t ) i = p ( x ( i ) ) q t ( x ( i ) ) . We can view the final estimate ˜ R as a function of ˜ R 0 and W ′ that has the form f ( X,Y ) = X Y . Then we directly analyze the expectation and variance of ˜ R using the delta method: First we have

where µ X = E [ X ] and µ Y = E [ Y ] . Applying (21) on our estimate, and we get:

where we utilize ∑ T t =1 v t = 1 and E [ ˜ R 0 t W ′ t ] = R . For the variance, we have:

Applying to our estimate leads to

where we utilize f ( X,Y ) = X Y → f ′ X = 1 Y , f ′ Y = -X Y 2 , µ X = R,µ Y = 1 . Note that since we assume q t ( x ) are fixed, we have E [ W ′ ˜ R 0 ] = E p ( y | x ) E q 1 ... E q T [ ∑ T t =1 v t ∑ n i =1 ( w ( i ) t ) 2 l ( x ( t ) t )] = E p ( y | x ) [ ∑ T t =1 v t E q t ∑ n i =1 ( w ( i ) t ) 2 l ( x ( t ) t )] .

B.2 Proposition 1

We show two concrete examples for Proposition 1. In each case, the estimated introspective loss is analogous to an uncertainty measure.

The estimation of 0 -1 loss is:

̸

which is the sum of the predicted probability of all classes other than the most probable class.

The estimation of cross-entropy loss is:

which is the entropy of the predicted probability.

When we use deep learning models, R θ usually largely underestimates the risk over the entire pool. In other works such as [14, 15], the surrogate risk acts in a similar way. For the final risk estimator to be accurate, the introspective risk estimation or the surrogate risk first needs to be accurate, which somewhat beats the purpose of active risk estimation. However, we still try to improve this intermediate step without assuming that we have access to an unrealistically accurate estimation, leading to our proposed R θ in Section 3.3.

B.3 Proof of Theorem 2 and Active Feedback Analysis

In Theorem 2, we formalize the combined learning-testing objective as a joint optimization problem with the variable being a subset S FB that can be transferred from the testing set S T to the learning set S L . We define the process of selecting the subset as the 'active feedback' process, which connects the learning and testing objectives through a balancing parameter C given in (8). Performing exact optimization of the subset along with a parameter C would require more detailed knowledge on the learning model and the AL strategy. We instead provide a general analysis to show that active feedback could indeed provide an optimal solution for the joint optimization problem, where C scales as O (1) . Following our theoretical result, we empirically demonstrate the effectiveness of an intuitive feedback approach in the experimental sections (Section 4.3, Appendix C).

Proof overview. We apply some generic generalization bound ( e.g., [17] for CNN or similar models) to the learning objective (I) in the joint optimization problem given by (8), which gives O (1 / √ n ) . We then leverage the confidence interval to get a high probability bound for the testing objective (II), which also gives O (1 / √ n ) [4, 9, 28]. We use the formalized results on the convergence of the estimate as introduced in [20]. With that, we continue to show that both the learning and testing objectives share the same dependency on n . These common dependencies on n give us the foundation to further analyze the feedback process. We offer an intuitive justification of active feedback as follows. The risk estimators are importance weighted estimates of the true risk. The estimate converges to the true risk asymptotically, so fewer samples might hurt the quality of the estimate (due to a large variance), but does not change the fact that the expected average of the estimate is still the true risk. With the confidence interval conversions, we can see that except for the change of constants, the objective's dependency on the number of samples does not change. (This also provides guidance for the feedback proposal later: if we can keep the change of the estimate to the minimum, meanwhile using the samples discarded from the test set to improve the AL model as much as possible, it would be the ideal use of available labels.) Following these high-level ideas as described above, we present the detailed proof below.

Proof. We first break the joint (I) learning-(II) testing objective into two parts and approach each part separately:

The learning objective. As mentioned earlier, (26) is a common generalization error bound for CNN or similar models. For example, given a training set S L with N L samples, we can draw from the basic bound ( e.g., according to Theorem 2.1 in [17]):

with probability of at least 1 -δ , where C ′ , β ′ , and λ ′ are constants and | θ | is the total number of trainable parameters in the network. In our case, we do not make further assumptions about the constants and | θ | is fixed for evaluating a certain model. Similarly, we can substitute N L with N L + N FB and arrive at:

We notice that in both (28) and (29), we include the expected loss which is slightly different from the best possible AL model risks R ( f ∗ θ | ( S L ∪S FB ) ) and R ( f ∗ θ |S L ) . However, the difference is usually

on a smaller scale than (1 / √ N L -1 / √ N L + N FB ) . In general, we assume that R ( f ∗ θ | ( S L ∪S T ) ) ≲ R ( f ∗ θ | ( S L ∪S FB ) ) ≲ R ( f ∗ θ |S L ) since more labeled samples can benefit learning (we do not need to assume a strictly monotonic case for the sake of this analysis). For most AL strategies, the difference between the expected empirical risk and the optimal risks given the learning set size is on a higher order of dependency on n than the learning bound itself. If we ignore the higher order terms, we can simplify the results as shown in (26). Then, the term and more importantly the change in the learning objective that is related to the assumed feedback S FB is only dependent on N FB through O (1 / √ N L + N FB ) .

The testing objective. The relation in (27) can be further analyzed by taking a probabilistic view. If we assume the risks are bounded in the third moment, w.l.o.g., the two risk-difference terms can both be generalized to a slightly more specific high-probability confidence interval [4, 9, 28] than the plain central limit theorem result itself: with probability of at least 1 -α , we have

where F -1 is the inverse cumulative distribution function of the Studentt distribution and ˜ σ 2 is the empirical variance. For the active feedback analysis, we only care about how N FB affects the testing objective, thus also obtaining an O (1 / √ N T -N FB ) dependency.

The detailed balancing between the two objectives (I) and (II) requires specific knowledge about the constants involved in the bounds. However, if we only focus on terms involving N FB, both dependencies on the sample numbers are on the 1 / √ n level, making it possible to be balanced by a constant factor C . Combining these results, we get the N FB term as O (1 / √ N L + N FB ) + O (1 / √ N T -N FB ) (absorbing O (1) terms that do not depend on N FB). The next key factor is that throughout the entire ATL process, we either keep N FB fixed or only change it at a linear rate (flexible N FB should be an interesting future direction). Combining with our previous assumption that N L and N T are of similar magnitudes, we know that an optimal balance could be achieved between (I) and (II) to minimize the joint learning-testing objective given in (8).

C Additional Experiment Results

In this section, we present the detailed experimental settings and additional experimental results.

C.1 Synthetic Experiment

Figure 4 shows how the proposed feedback strategy helps to encourage exploration. The background color shows the model's predictive distribution. For each quiz, we display all the training samples obtained by an active learner (red and blue circles representing 2 classes) but only the current quiz (triangles) and feedback samples (squares, then added to circles in later AL rounds) from the active tester to make the visualization clear. Figure 4a shows that ATL selects a feedback sample in the bottom right corner because it is not included in the current knowledge base of the AL model. The AL model predicts it poorly in the quiz. In Figure 4b, we see that the AL model is guided by the feedback samples and starts to explore the bottom right corner. Once the AL model collects samples from the bottom right area, ATL stops to provide guidance for that region. In this way, the proposed feedback strategy manages to find the minority cluster at the other corner shortly as shown in Figure 4c.

In Figure 4d, to further demonstrate the effectiveness of the proposed feedback strategy, we compare it with the feedback samples selected using two other baselines: random feedback (in Figure 4f) and AL based feedback (in Figure 4e), when the samples at the bottom right corner are first discovered. First, we notice that those data samples are found by the AL model rather than through the feedback strategies. As a result, it happens at a much later quiz time compared with ATL. Therefore, they result in a less efficient learning process. Second, we observe that when an AL model discovers a

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:2.5e-03

AL Sample

AL Sample

A

T

Sample

Feedback Sample

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

(a) ATL Feedback Quiz 2

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:2.0e-03

AL Sample

AL Sample

A

T

Sample

Feedback Sample

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

(b) ATL Feedback Quiz 3

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:2.0e-05

AL Sample

AL Sample

A

T

Sample

Feedback Sample

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

(c) ATL Feedback Quiz 4

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:4.2e-05

AL Sample

AL Sample

A

T

Sample

Feedback Sample

0.16

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

(d) ATL Feedback Quiz 10

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:6.4e-04

AL Sample

AL Sample

A

T

Sample

Feedback Sample

0.16

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

(e) AL Feedback Quiz 9

2

1

0

1

2

3

2

1

0

1

2

Estimation Error:1.2e-03

AL Sample

AL Sample

A

T

Sample

Feedback Sample

0.16

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.80

0.88

(f) Random Feedback Quiz 10

new area to learn, both baseline feedback strategies fail to provide support even though they have some test samples ( i.e., the red point at the bottom right corner) available in the interesting region. Last, we can see that at around quiz 10, the AL model with the proposed ALT converges to a better decision boundary that captures the entire data distribution while the two other baselines both fail to correctly discover the predictive distribution at the two corners. As a result, ALT leads to a more accurate model (shown in Table 7) while maintaining lower estimation error in the end.

C.2 Real-world Experiments

In this section, we provide more results on the real-world datasets including MNIST, FashionMNIST and CIFAR10, mainly to demonstrate different feedback approaches and how we can implement early stopping in ATL.

C.2.1 Experimental Settings

In all experiments, we use a CNN model and standard data transformation for each dataset. In each AL training round, we run 10 epochs for MNIST and FashionMNIST and 50 epochs for CIFAR10. A threshold of 1 × 10 -5 is used for probability outputs as required for the proposal q ( x ) computation [14] to avoid 0 denominators.

An important detail to note is that for ATL-NF results, we sample 50 additional data points during AL for fair comparison (550 in each round), which is actually very similar to ATL-RF . The results with only 500 data points per round will be shown in the following section C.2.2. Another detail worth mentioning is that although we set the initial budget to be 500 labels and add 500 training

samples plus 100 testing samples in each round, the final total budget is 12 , 450 on average instead of 12 , 500 because we allow replacement while sampling.

C.2.2 Random Hold-out Test and Random Feedback

In this section, we discuss the issues with traditional hold-out validation/testing procedures during AL and compare the results using random sampling for both test sample selection and feedback selection with the proposed learning-testing-feedback process.

In Table 8, random test is referring to randomly sampling 100 test samples after each 550-sample (additional 50 for fair comparison, same as ATL-NF) AL round and simply averaging the loss over these test samples. Random test & feedback is referring to sampling 100 test samples after each 500-sample AL round and then randomly selecting 50 for feedback. Random test & weighted is referring to the same process but the quizzes are weighted by 1 /R t . From Table 8, we can see that in the small-data regime, random sampling may not provide an accurate estimate of the true risk. However, in later AL rounds, the no feedback case (Random Test) can maintain an unbiased estimate, and we do see that some results are comparable with active risk estimation baselines without the ATL-integrate estimator. This is probably because existing active risk estimation baselines (ARE-quiz, AT-integrate, ASE-integrate) do not consider the biased selection and model change through the AL process. The methods that use surrogate models also suffers from the insufficient training of the surrogate model. However, random testing selection does not work well with the active feedback process. For Random Test & Feedback and Random Test & Weighted, we often see much worse estimation due to the feedback process involved.

C.2.3 ATL with Various AL Strategies

In this section, we show additional ATL estimation error results and active feedback results using two different AL strategies: margin sampling [27] and BALD [12].

From Table 9, we can see that the integrated risk estimation performs similarly regardless of the AL strategy. The relative scales of different risk estimations are also similar to the entropy case in the main paper. From Table 10, we do see some different effects. Depending on the AL strategy, random feedback performs unstably. However, our proposed feedback still mostly outperforms the fair comparison baseline. The study on AL-specific feedback strategies could be an interesting future direction.

C.2.4 Additional Active Feedback Comparisons

In this section, we show a more complete comparison between different feedback approaches. The feedback comparison consists of two parts: (1) baseline comparison including no feedback (ATL-NF), random feedback (ATL-RF), entropy-based feedback (ATL-EN) and (2) ablation study including loss-based feedback (ATL-LF), weighted loss-based feedback (ATL-WL) and the proposed weighted loss plus diversity feedback (ATL).

First, we show the hold-out test risk of the AL model throughout AL using different active feedback approaches as the indicator of the model performance. From Table 11, we see that in most occasions, all active feedback approaches can reduce the test risk compared to ATL-NF.

In Table 12, we show a full comparison of the squared error of risk estimation. All estimation results are based on the proposed ATL estimator ˜ R , where ATL-NF, ATL-RF, ATL-EN serve as baselines, meanwhile ATL-LF and ATL-WL serve as ablation studies since the proposed ATL utilizes the weighted loss as well. We see that all feedback approaches suffer from an increased estimation error, especially in the early stage when the number of test samples available is small. We see that

the baseline methods suffer from increased estimation error. However, ATL can usually maintain a similar level of estimation error after 20 AL rounds. For ATL-LF, there is usually a larger variance of the estimation error. The potential reason for the unstable behavior of ATL-LF is that it only selects samples with larger losses in the feedback process. Although the importance mechanism can make up for some of the difference, there is still the potential risk of the estimate being biased. Further combining with the diversity metric, we achieve the best results with ATL.

Concluding from both the risk results and the estimation error results, we show that the proposed feedback approach achieves a good balance in the performance-estimation trade-off. This is because we consider both the loss L and the importance weight q in the selection criterion. Overall, ATL achieves a similar model test risk as ATL-LF/ATL-WL, both of which are much better than ATL-NF and ATL-RF. ATL also achieves a much lower estimation error than ATL-RF, ATL-EN, and ATL-LF.

C.2.5 Single Feedback Round Comparison

In previous experiments, we add additional training points for the no feedback case (ATL-NF) to make fair comparison for the model risk. However, if we look at the risk change before and after a single feedback round, the difference is even more obvious as shown in Table 13.

C.2.6 Early Stopping in AL

In this section, we show how the ATL-based risk estimation can be readily used for early stopping in AL. In the above experiments, we observe a steady decrease of the estimated risk most of the

times. However, we do find the decrease becomes more insignificant near the end of the 20 rounds of learning, especially for the MNIST and Fashion MNIST datasets. We observe that after a certain amount of AL rounds, the risk decrease is significantly small, and the corresponding test accuracy is also stabilized (MNIST around 94% , Fashion MNIST around 80% , CIFAR around 54% ). This gives us the opportunity to apply early stopping in real-life AL applications. We here show the average stopping iteration and model performance (hold-out test accuracy) of the compared methods in Table 14. Following the same threshold value, by augmenting the moving average of active risk estimation given by (10) with stabilized prediction (SP), the combined method can stop at a similar testing accuracy as compared with the SP method, but with much lower variance in test accuracy. Based on the threshold setting, it is also possible to stop AL much earlier, saving the overall labeling budget.

D Details of Hardware for Experiments

All experiments were run on clusters with either NVIDIA A6000 or NVIDIA A100 graphic cards and Intel Xeon Gold 6150 CPU processors. The runtime of the experiments varies depending on the number of repeat runs, but is usually on the scale of a few hours. For example, to get the 5 runs results of one ATL setting for 20 AL rounds on MNIST or Fashion MNIST may take about 6 to 8 hours. The CIFAR10 experiments may take slightly longer.

E Limitation, Future work, and Social Impact

In this section, we first discuss some limitation of the proposed framework and identify some important future direction. We then discuss some potential social impact of our work.

E.1 Limitation and Future Directions

In this paper, we propose an integrated framework that combines active learning and testing. In the interactive framework, the exchange of training and testing information should be carefully guided. Although the proposed testing selection is statistically unbiased and the active feedback is backed by the high-level analysis, we still have room for improving the specification of methods in applicable settings, which we will introduce here as future directions:

From the learning perspective, we can improve upon the general setting in this paper. In this paper, we focus on introducing a general framework and working under the agnostic setting. However, using specific AL strategies can potentially provide advantages in certain use cases. There have been works that analyze AL label complexity bounds using either importance weighting mechanism in stream-based settings [5, 8] or other methods in poolbased settings [11].

Continuing on the results from the feedback size analysis in Section 4.3 and the discussion above, the feedback size is a very important factor in the process, especially if we allow the

size to change during AL. Further investigating the relationship between the sample size and the combined learning-testing objective can potentially improve the framework.

We also propose the ATL framework under an AL-agnostic assumption. Given specific AL strategies, we might be able to also incorporate the learning or testing proposal in the construction of feedback proposal.

E.2 Social Impact

The proposed ATL framework considers the practical challenges of applying active learning in real-world settings, where both model training and evaluation require labeled data. It is a critical step towards realizing label-efficient learning in practice, which can benefit many critical domains where data annotation is highly costly. To this end, the proposed ATL framework has the potential to fundamentally address the data annotation crisis and further broaden the usage of AI to benefit the entire society.

F Algorithm and Source Code

Here we present the pseudo code for the ATL framework using an algorithm block below:

Algorithm 1: Active Testing While Learning (ATL)

Source code . The data and source code for replicating the results are provided in this link: https: //github.com/ritmininglab/ATL.git

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Active Testing while Learning (ATL) framework, which efficiently collects testing samples for online model risk estimation during the active learning process. The online risk estimation enables early termination of active learning. Additionally, ATL establishes a connection between Active Learning (AL) and Active Testing (AT) through an active feedback mechanism that transfers selected testing data points to the training dataset, thereby enhancing model performance.\n\nSTRENGTHS:\n1. The paper tackles two crucial problems in the AL domain: efficient sampling of testing data points and the combination of AL with AT. Both of these problems are highly relevant and significant in AL research. \n2. The paper introduces an unbiased estimator of the model risk, and also proposes a unique method to combine active quizzes as the final exam, maximizing the utilization of testing samples.\n3. The paper provides a comprehensive set of experiments to evaluate the proposed estimator and feedback methods. The experimental results demonstrate the superior performance of the proposed approaches compared to existing techniques.\n\nWEAKNESSES:\nIn the first half, the paper focuses extensively on optimizing the data sampling process to obtain an unbiased estimator of model risk. However, a flaw arises in the feedback mechanism, which moves data points from testing dataset to the training dataset based on the rule described in equations (8) or (9). This feedback undermines the unbiasedness of the risk estimator by introducing dependence in the selection of testing samples. Specifically, when we extract samples from the testing datasets based on some specific rule such as (8) or (9), the remaining samples are no longer independently sampled, thereby introducing bias.\n\nQUESTIONS:\nThe paper needs more discussion on the potential bias introduced by the feedback mechanism.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper considers the active testing procedure to assess the risk. The keys are 1) a proposal of a quiz using importance sampling, 2) combining the historical quizzes to make a test set for the final model test, and 3) a proposal of a feedback algorithm to strengthen the model using the labeled test data. Experiments focus on the performance of risk estimation, and some work concern test accuracy with feedback algorithms.

STRENGTHS:
The proposal of an activel testing procedure can catch realistic problems. The quiz usage is impressive, and the set of quizzes is fully used.

WEAKNESSES:
Maybe we can consider a more straightforward example, such as the explicitly calculated risk. The performance of the proposed algorithms is validated. However, more solid experiment results are limited since the active learning algorithms are various, but only a few algorithms are considered.

QUESTIONS:
I have some questions. 

1) what's the exact AL algorithm used in the experiments? 

2) Is there any effect of different AL algorithms on ALT? 

3) the feedback algorithm seems to be a hybrid of AT and ATL. Therefore, it seems that the benefit of feedback cannot be larger than the pure AL in the same budget (not the same budget in experiments)? 

4) Is the algorithm possibly adaptive to the quiz sets (test sets)? I want to know how to construct the test set in Table 2.

LIMITATIONS:
Moderate explained. The connection between AL and AT seems lower.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a relatively new flavor to recent active learning works, by considering active testing. In particular, they evaluate the learned model on the fly on adaptively acquired set, called quizzes. Model learns based on those quizzes and part of it is added, as a feedback from the quiz. At the end, it is evaluated on the final exam set (cumulative quizzes excluding feedback examples). Statistical convergence guarantees have been provided, and low variance (unbiased) estimators are considered. Proof of concept is provided on test dataset w/ GP model and common classification datasets are considered for the real world setting. The main paper provides the results on how far is true risk (estimated from large number of held out samples) from empirical risk.\n\nSTRENGTHS:\nThe paper provides a relatively new flavor to recent active learning works, by considering active testing. In particular, they evaluate the learned model on the fly on adaptively acquired set, called quizzes. Model learns based on those quizzes and part of it is added, as a feedback from the quiz. At the end, it is evaluated on the final exam set (cumulative quizzes excluding feedback examples). Statistical convergence guarantees have been provided, and low variance (unbiased) estimators are considered. Theory and practical aspect seem interesting to the broad community. It is generally well written.\n\nWEAKNESSES:\nMuch of the weakness in the work are some of my confusions and doubts. I *strongly* appreciate the authors to add an algorithmic block which shows how each and every step is done in the true implementation. Lastly, I haven't verified the correctness of the proof line by line, but the theorems seem sound. \n\nOn AL acquisition functions:\n- I don't see any discussion at all about what AL algorithm is being used in all the experiments. Can authors please add that discussion.\n\nOn theory: \n\n- How is the sampling from optimal $q_t$ done to get the quizzes at each testing round? The expression mentioned in equation (4) involves an integral done over squared error of loss function and true risk and the true underlying posterior distribution over labels. While for the true risk, it is approximated using a multi-source estimate, I am not sure where do we get the labels to even define the distribution. \n- The multi-source estimate is a function of time step, that is ideally equation (5) should've $t$ in the LHS. Secondly, right after the first round, how are we getting $\\hat{R}_{Q_t}$ to multi-source estimate?\n- Wherever the expectation (integrals) are approximated using the empirical estimates in the quizzes or final test, or to get $C_t$, what is the scale of example? I feel that for small number of examples, $C_t$'s estimate can be pretty noisy, and therefore making $\\tilde{R}$ estimate noisy. \n- For the active feedback setting, is only 1 example chosen from the quiz set $\\mathcal{Q}_t$? If not, how is the batch selection done? Equation (9) only seem to be providing information for 1 example. If it is more than that, then I'd appreciate writing this as an algorithm (greedy procedure), or at least some discussion. \n\n\nOn experiments: \n- Can authors provide absolute accuracy numbers, and how does that change over the course of AL rounds? \n- How is the scale of error in Fig 3g so different than the others?\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nI feel that more dataset/arch should be considered to study the real impact of the early stopping method, and in particular it should be compared against (in a fair manner) to fix accuracy based criterion. While I appreciate the provided framework, it needs much more discussion as to how it would fit in the current paradigm, in particular when we have low budget active learning regime, or where we already start with a pretrained backbone (to be sample efficient). Lastly, I feel skeptic about the empirical estimates for the expectations, as usually the examples used for the testing would be small (as it counts in the labeling budget as well).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for integrating active learning and active testing in an online fashion. The proposed algorithm incrementally sample training and testing examples for each batch and can leverage all of the testing examples so far for evaluation. Moreover, the sampling distribution for testing is designed based on an estimate of the variance of the risk. Experiments are conducted for both Gaussian processes and neural networks on multiple datasets.\n\nSTRENGTHS:\nThe paper studies a very important problem and provides an effective solution. I find the method to be novel.\n\nWEAKNESSES:\n1. I am not sure if the claim that the test risk estimate is unbiased. It is indeed unbiased if one knows the true risk, but when using the estimate proposed in section 3.3 to substitute the true risk, is this still unbiased? Moreover, if the feedback set is used in training, the estimator doesn't seem to be unbiased either. I think the authors should play down the claim of the proposed risk estimate is unbiased since it is not when combined with all of the tricks.\n2. In the experiments, the results of random sampling are not shown in the tables.\n3. There are quite a few heuristics proposed in the paper from sections 3.3 to 3.5. Although the authors provided intuitive explanations, the design of the algorithm may have overfitted to the three datasets the authors tested on. For example, the weighting between different components of the intermediate estimate seems quite arbitrary (section 3.3). It is also not clear how C is chose in practice in section 3.5. I strongly encourage the authors to further test their design across a wider range of settings.\n4. The paper assumes the active learning algorithm to sample based on a probability distribution. However, many practical AL algorithms do not. This can cause the test risk estimate to be really high for those algorithms. I strongly encourage the authors to address this problem.\n\nQUESTIONS:\n1. The active learning strategy used in this paper seems to be very vague. What's the algorithm being used?\n2. There are quite a few components to the proposed algorithm. Can you give an algorithm box to make it clearer? Also, can you provide a time complexity analysis?\n3. In [1] appendix , the authors used a water-filling algorithm to make sure the examples sampled up to time t follows roughly from distribution $q_t$, even though $q_1, ..., q_{t-1}$ may differ from $q_t$. I wonder how this technique compares against what's proposed in 3.4.\n4. Please also see the weakness section. I think this work studies an important problem and proposes a cool solution. I am willing to raise my score if the authors sufficiently address (part of) my concerns.\n\n[1] Katz-Samuels, J., Zhang, J., Jain, L., & Jamieson, K. (2021, July). Improved algorithms for agnostic pool-based active classification. In International Conference on Machine Learning (pp. 5334-5344). PMLR.\n\nLIMITATIONS:\nSufficient.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}