Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice
Authors: Dayou Yu
Weishi Shi
Qi Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: active learning
active testing
TL;DR: We propose a novel framework that integrates active learning with active testing, resulting in an unbiased sample-efficient estimation of the model risk during active learning.

Abstract: In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, "testing'', which often refers to the model evaluation process of using empirical risk to estimate the intractable true generalization risk, also requires data annotations. The annotation cost for "testing'' (model evaluation) is under-explored. Even in works that study active model evaluation or active testing (AT), the learning and testing ends are disconnected. In this paper, we propose a novel active testing while learning (ATL) framework that integrates active learning with active testing. ATL provides an unbiased sample-efficient estimation of the model risk during active learning. It leverages test samples annotated from different periods of a dynamic active learning process to achieve fair model evaluations based on a theoretically guaranteed optimal integration of different test samples. Periodic testing also enables effective early-stopping to further save the total annotation cost. ATL further integrates an "active feedback'' mechanism, which is inspired by human learning, where the teacher (active tester) provides immediate guidance given by the prior performance of the student (active learner). Our theoretical result reveals that active feedback maintains the label complexity of the integrated learning-testing objective, while improving the model's generalization capability. We study the realistic setting where we maximize the performance gain from choosing "testing'' samples for feedback without sacrificing the risk estimation accuracy. An agnostic-style analysis and empirical evaluations on real-world datasets demonstrate that the ATL framework can effectively improve the annotation efficiency of both active learning and evaluation tasks.

Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice
Dayou Yu'
Weishi Shi?
Qi Yu!
Rochester Institute of Technology, Rochester, NY 146231 University of North Texas, Denton, TX 76203
(dy2507, yu).,gii.juuirii yuryrui.eeddei weishi shi @unt edu
Abstract
In active learning (AL), we focus reducing the data annotation cost from the model training perspective. However, "testing", which often refers to the model evaluation process of using empirical risk to estimate the intractable true generaliza- tion risk, also requires data annotations. The annotation cost for "testing" (model evaluation) is under-explored. Even in works that study active model evaluation active testing (AT), the learning and testing ends are disconnected. In this paper, we propose novel active testing while learning (ATL) framework that integrates active learning with active testing. ATL provides an unbiased sample-efficient estimation of the model risk during active learning. leverages test samples an- notated from different periods of dynamic active learning process to achieve fair model evaluations based a theoretically guaranteed optimal integration of different test samples. Periodic testing also enables effective early- strly-stopping to further save the total annotation cost. ATL further integrates an "active feedback" mechanism, which is inspired by human learning, where the teacher (active tester) provides immediate guidance given by the prior performance of the student (active learner). Our theoretical result reveals that active feedback maintains the label com- plexity of the integrated learning-testing objective, while improving the model's ss generalization capability. We study the realistic setting where maximize the performance gain from choosing "testing" samples for feedback without sacrificing the risk estimation accuracy. An agnostic-style analysis and empirical evaluations on real-world datasets demonstrate that the ATL framework can effectively improve the annotation efficiency of both active learning and evaluation tasks.
Introduction
Labeled data are essential for supervised learning in both model training and evaluation. Active learning (AL.) provides a promising direction reduce the human annotation cost by constructing
a smaller but more effective labeled dataset for training purposes 24 AL only partially addresses the human annotation cost the cost of constructing labeled dataset for model testing has been overlooked. In reality, the annotation budget could easily be drained by building large test datasets that most modern AL methods rely on evaluate the model performance and determine the stopping criterion for learning. In contrast to AL, active model evaluation or active testing (AT) focuses on actively selecting the testing data. Few existing efforts develop unbiased risk estimation techniques for label-efficient model evaluation [20 [4]. However, these methods are designed to evaluate fixed models that have been fully trained, making them ill-suited for evaluating an actively learned model, which is constantly updated and inadequately trained during most parts of the learning process. To address these fundamental challenges and realize label-efficient learning in practice, we propose to integrate AT with AL in novel and efficient learning framework, referred to
5
g However,
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

as ATL. The ATL framework systematically addresses the unique active testing challenges arising from evaluating an actively learned model that is under-trained and constantly evolving. It leverages an interactive
process to better control the overall labeling budget, which not only achieves efficient evaluation of the AI, model but also ensures faster convergence in model training. Through periodical testing, ATI. gains useful insights to terminate the learning process as early as possible that avoids allocating unnecessary labeling budget.
Incorporating AT into AL is highly non-trivial task. Unlike AL, AT has two inter-connected ob- jectives: i) design of an unbiased risk estimator to quantify the model performance, and ii) de- sign of sampling strategy to select informative test samples. Achieving both objectives simulta- neously under the AL setting with a constantly evolving and under-trained model is far more challenging than existing active model evalua- tions that assume a fixed and adequately trained model. For the first objective, its challenge stems from the potential biased active test samples brought by the second objective. We need an unbiased risk estimator to ensure the faimess of the evaluation. An unbiased risk estimator can be achieved by importance sampling its vari-
Kanzon Herhod
(a) Data distribution (b) Active estimation crror
Figure I: Risk estimation comparison
ants [25] 20 21 22 19 [6]. Recent active testing works also employ expectation analysis and propose other weighting mechanisms [10 [4]. However, all these works assume that the model being evaluated is fixed and already well-trained (20 14). Therefore, these estimators are not designed to support dynamic AL setting, where the model continues evolve when learning from newly labeled samples. Figure I shows that both the standard estimator (i.e., random testing) and direct adaptation of an existing unbiased estimator (ARE) (20) fail to provide an accurate and consistent model evaluation in the more challenging AL setting. As for the second objective, different from sample selections in AL, the testing selection needs be compatible with the unbiased risk estimator. It is essential to ensure asymptotic convergence that guarantees an unbiased model evaluation while reducing the variance that can improve the convergence speed and provide stable signal terminate the learning process as early as possible.
Conducting AT along with AL enables ATL to seamlessly connect the active training and testing sides, which allows them communicate with and guide each other within well-integrated learning process. To this end, the proposed ATL frame- work leverages an active learning-testinn feedback (LTF) loop, which largely resem- partic-
Testing Set
Training Set
Activa Feedback
Integrate
Quic Review
Test Stleetion on
Al Selection
Active
Activa
Quiz Records QQ.-.., Qr)
Final Estimate
(R
bles real-world human learning. ular, within the LTF loop, the active testing results are kept after each active learning round, which mimics the quizzes in the hu- man learning setting. We refer to them as active quizzes. As shown in Figure 2 the active learner acts as a student while the active tester acts as teacher. Both the student and the teacher select data from the unlabeled pool and get their labels from the oracle. The student attempts to use the labeled data to pass the quizzes provided by the teacher. On the other hand, the teacher provides fair quiz to evaluate the student's 's performance. Meanwhile, the teacher also sends back the keys of some quiz questions, which are referred to as active feedback, to help the student learn better.
Early Stop?
Oracie
Unlabeled
Pool
Figure 2: Overview of the ATL Framework
Given dynamic AL process, multiple active quizzes will be created at different learning phases. ATL forms an integrated risk estimator by aggregating all the quizzes. Our theoretical analysis guarantees that the integrated risk estimation converges to the true risk asymptotically. The aggregation strategy also minimizes the variance, which ensures faster convergence. Furthermore, through active feedback, the AL model can benefit from training with a small proportion of the testing samples (as feedback) after each quiz while maintaining the asymptotic convergence condition the risk estimation. Finally, the integrated risk estimator provides unbiased estimate of the model performance with small variance, which provides a convenient way to determine whether the AL model has converged.

---

This can further save the labeling budget by stopping the learning early. We may also combine the estimated risk with other available information from the unlabeled data to create more systematic early-stopping criteria. Our main contributions are summarized as follows:
loop to mimic human learning for achieving label-efficient and dynamic evaluation of actively learned models, a theoretically sound integrated risk estimator that combines active quiz results and minimizes the
the first ATL framework that builds an active Saaninggtesting-feeet
variance of the difference between the estimated and true risks,
an innovative active feedback process to further improve learning effectiveness without extra labels, a systematic early stopping criterion that combines integrated risk estimation with information of unlabeled data to terminate AL. process early.
We conduct experiments on both synthetic and real-world datasets to show the improved testing performance of the proposed ATL framework.
2 Related Work
Active model evaluation. Actively testing the model performance in a label efficient way is relatively under-explored. Among the few excising efforts, model testing is treated independently from model training, which is unrealistic in most practical settings. The problem of active risk estimation (ARE) was first formulated in (0).. which used importance sampling to construct an unbiased risk estimator. Then, the optimal test sampling distribution is derived from an optimization problem that minimizes the variance of the risk error. A similar process has been used to estimate more complex loss measures 21 or perform active model selection [23]. However, such risk estimator is no longer unbiased nor label-efficient when an AL process involved that samples from the same unlabeled pool for model training. Farquhar et al. propose two risk estimators to cancel the statistical bias by active data sampling strategy in a data-efficient manner [10. However, their proposed estimators could not deal with the overfitting bias making them less suitable for model evaluation.
In more recent work, Kossen al. propose to use the unbiased risk estimators to perform active testing [14]. The difference from [20] is that the selection order of the testing points is considered, and the weights are defined for selection indices instead of single testing proposal distribution. In this way. the testing proposal can be evaluated at each round, making the setting more realistic. However. the optimal proposal is still impossible to obtain. To approximate the optimal proposal, a surrogate model is needed, which is similar to the introspective risk model in [20]. The surrogate model idea is further extended in (T5), where the unbiased risk estimator replaced by the predicted loss from the surrogate model. Yilmaz et al. propose to replace the commonly used importance sampling with Poison sampling, which better stabilizes the test samples 29) A different line of work leverages stratified sampling to first stratify the unlabeled pool and then conduct efficient sampling over strata in an importance sampling manner [3] 30 However, all those work limited to
offline test sampling in which the test proposal is fixed during the sampling process and could not benefit from prior knowledge such as the previously labeled data.
Early stopping in AL. Early stopping strategies in AL have been sparsely investigated over years 2 6 31
26 18). Most methods only consider the learning process and do not consider a fully connected learning- teetinggtetingggggo loop. Some have shown that the stopping criteria based on unlabeled information are more effective than those based on labeled information. This inspires us to combine the proposed integrated risk estimation with unlabeled data information to form a systemic early stopping criterion to further reduce the labeling budget.
3 The ATL Framework
Problem setting. We formulate the ATL framework under a novel and fully integrated learning and wastin--wiilei--llaanss setting. W.L.o.g.. we assume that all data are generated from an underlying distribution, and we could obtain the label y through the true labeling function: y = h((x) provided by the human annotator (note that unlike the predictive model fa(-), the true labeling function is not governed by the parameter and it might be realizaliz..le) Similar to a traditional based AL setting, we have small labeled set S₁ = {xx v)} No (whose labels are revealed). As for the unlabeled pool, we only have access to the features: So {x)-Ne. We further assume that the unlabeled pool is sufficiently large so that active sampling will not change the pool distribution since only limited labels will be revealed. That is, p(x, y) remains fixed during AL. Our primary task couples two objectives: AL and model evaluation at the same time. In this setting, the goal of AL is to
3

---

create labeled set by selecting samples to label and achieve the best performance. The performance of the model indicated by the expected error when generalized the underlying data distribution. which is often noted as the risk R. As discussed earlier, existing works on lahel-efficient model evaluation assume that the model being evaluated already well-trained and kept fixed. They are inadequate to handle dynamic active learning scenario, which is the focus of ATL. Given some loss function as the risk measurement Lo fx(x) x y + R. our task of testing the model reduces to evaluating the following expectation:
R
=
E [Co] - ( u))( ())pp(())))(())))))))))))))))))) y) {x.y>~
((fo(())
(1)
However, directly evaluating is infeasible as the true density function p(x, y) is unknown. So empirically, we have to adopt an approximation. In the pool-based learning setting, our approximation resorts to the expected empirical risk over the pool distribution D Dpoot.
3.1 Overview: Active Risk Estimation During AL
In each AL iteration, we select to label x based on sampling criterion Peample (x)) to improve the model. We design the framework to be AL-agnostic, meaning that most standard AL sampling strategies can be applied here without further assumptions. Normally, the evaluation of the AL model is performed on a hold-out test set either at the end or during different to-learn data selection rounds of AL. However, the labeling cost of the test set is often times ignored. To achieve a more efficient evaluation framework, we propose to have the active risk estimation (i.e., active testing) process intertwined with AL. Specifically, we use time stamps to denote each stage where we perform active
(1) x...,."""! x,"'), }, then label them to evaluate the model. We name this novel evaluation style active quiz. Active quiz is motivated by the dynamic nature of the AL process, which allows us to have an instant evaluation of the current model during AL. We also show that the quizzes can further be conveniently combined into a final exam for a comprehensive evaluation. In this way, we essentially divide given budget for evaluation into small batches, which makes possible to stop the evaluation before reaching the total budget by leveraging the signal from past quizzes.
testing. At time f, we use testing proposal (x( to select samples, Q =xx,"
The purpose of having the active quiz is to achieve good estimate of the current model performance. which is not realizable in most existing AL frameworks. We use the risk R to denote the performance. R is dependent on the specific model, thus at each step we are trying to estimate the true risk of the model at l: R(f((.)) Using 2t, we can have an estimate Ro, (Je). We name the evaluation after the last quiz Qr the final exam because it includes all the quiz samples: {Q1...., Qr). We use R (fr(-)) to denote the final performance of active learning model. Note that R is different from each R2 because: (i) the evaluation sets contain all the previous quiz data, and (ii) it only evaluates the final model, fr. Intuitively, using (Q1,.., 2r } should be more preferable than using a single O, because more data samples are available to better represent the data distribution. However, the combination is not straightforward because each quiz sampled using the optimal selection proposal at that time. We propose principled way to combine multiple quizzes to achieve an unbiased R with theoretical guarantees. We further formulate active feedback strategy to integrate the dual objectives of learning and testing and provide a theoretical underpinning to utilize some labeled test samples to improve the model's generalization capability.
3.2 Unbiased Low-variance Estimation of Model Risk
For passive learning setting, where the model is fixed, the risk can be directly estimated through an importance weighting sampling procedure, such the ARE scheme developed in (0) Let gxx) denote the test sample selection proposal, then the risk estimate can be defined as
())
Rm.
(((10) ((Fo(x(0))
(2)
(x)))
We can use the central limit theorem to show that Ren. is an unbiased estimation of R. Furthermore, their difference follows a zero-mean Gaussian asymptotically:
p(x)
(/ICI1(t).0)) R)pplix)dy p(x)dx (3)
V T R R)
2000 \(0,0²) of
((x)
2 'Hrre use fe(-) represent (-) to keep the notation uncluttered.

---

When n is large, we also know that n Var [Rn.g] -0,, oz.- Then, the optimal (xx) minimizes the variance of the estimate. Using variational analysis, the optimal g(x) is
(x(x) p(x) /IC(FO(3))0) -R]pp(xx))
(4)
which minimizes the expected squared difference between the estimate and the true risk (also the variance of the asymptotic Gaussian). Since the pool distribution remains fixed when only small number of instances are sampled for AL, we can keep p(x) as uniform distribution over the pool. Since the importance weighting proposals are known up to a normalization factor, we have p(x) = Ny where Ny is the pool size. To optimize model evaluation in an active learning setting, we can evaluate the model at each time stamp in the following manner: First, we obtain an active testing proposal, denoted as qt, by substituting 4 with fo.. Next, we obtain the active quizzes denoted as Q2 Qt- We then evaluate the model's performance at time using
However, there are two main limitations to this approach. First, the test proposal given by (4) contains an unknown quantity, the true risk R. Previous studies have attempted to address this issue by predicting the true risk from the pool. However, in section 3.3] we demonstrate that commonly used true risk predictors are not accurate, particularly in the early stages of active learning. To address this, we propose multi-source true risk predictor. Second, the current test setting does allow for sharing of data between previous and current quizzes due to the use of different active proposal distributions. This inefficiency not only wastes data, but also reduces the stability of the model evaluation, as the test samples in each quiz may be too small to provide a robust evaluation. Therefore, in section 3.4, we propose an estimator for the active learning setting that can integrate quizzes sampled from different proposal distributions while remaining optimal.
3.3 Intermediate Estimate of the True Risk
In the optimal test sample selection proposal given by critical component is the true risk, which is unknown and needs to be estimated. A straightforward solution is to leverage the current model (20 or some proxy model [4] to predict the potential risk over the entire pool Rg. However, the following analysis shows that this is equivalent to the model uncertainty in the classification setting.
Proposition 1. Under the classification setting and when standard (i) 0 loss or (ii) cross-entropy loss used, Ro equivalent to measuring the model uncertainty.
More details of the proposition are provided in Appendix B.2 The above proposition shows that Rs can only capture the model uncertainty instead of the true risk. When the labeled training samples are limited, it is possible that the model uncertainty can not reflect the level of true risk. We propose to combine the current training risk, the model uncertainty, and the current test risk (using Qr that we have) get an aggregated multi-source estimate of the true risk:
|Sr|Rrriin + |Su Ro nR.
Rmulti
(5)
|St| + |Syl +
The multi-source estimate can more effectively avoid overestimating the risk when only using the testing information and underestimating the risk when only using training information. By aggregating these two sides of information, neither underfitting or overfitting of the model will harm the estimation a significant extent. The effectiveness of the multi-source aggregated risk estimation has been demonstrated in our empirical evaluation.
3.4 Active Quiz Integration for Final Risk Estimation
We have shown the optimal way select test test samples evaluate fixed model in the passive learning setting. However, we still need to extend to our scenario where there are multiple stages and AL while AT are intertwined. We propose combine the quiz results by assigning set of weights v, on them. We will show that the proposed Ve is optimal given the fixed quizzes. Let's denote a sequence of independent risk estimates (treated as random variables) as R = (Ro, (JT),, Roz (JT)T) where each Ro. (ST) asymptotically converges to the true risk R(fr). We should also note that we are using each Ro, to evaluate the same model fr, not using the actual values of the estimate at time (which is the evaluation of ft). That is, always utilize previously collected sets of quiz samples to
5

---

evaluate the current model.
Ro. (fT)
Wilp
1
p(x(1))
((Tr(x()),,)))
-i=1 Wi
2im
p(xto))
(x(x(1)
2=1
miml
q=(x(11)
2=1
where The variance of (ST) is given by
-R(ff))
n4 m4-000
o(
(6)
f p(x) (x(x)
(FIECENOO)N)) - R(p((xx)dy) p(x)dx
(7)
We have defined the asymptotic Gaussian variance term when we use the quiz set Qt from time to evaluate the final model fr. Let C. =1/a denote the model confidence of fr when evaluated on a test set that follows qt- Next, we formally show that if the model fr is more confident about qt. then O should take a larger weight in the integrated final evaluation result.
Theorem 1. Given fixed {Qi...., Qr). the weighted combination R = ΣT= Re. where = is the optimal estimator constructed by all samples in {Q1,... Qr).
E. Ci
Remark. The weight i2 depends on how accurate the estimate using each quiz set is. If we assume that the model becomes more accurate throughout the entire process, then we know that Rr-1 should be closer to R than Ry-2, thus the weight on the most recent one is the largest. However, we still do not know R and have to estimate the weight. Here we adopt similar way estimating Rg to estimate the difference terms C2 using the expected loss and Re, (fr): &
(C(fr(x()) ),9) Re, (fT)) ())) With the optimal uy, we define the final 1 E Σn (x(x ))
estimate result as R = -EE ot x Re.- In practice, if we do not pre-set the length of the entire AL process, we can treat each stage as the final exam. After each AL sampling round, we perform an active testing round as quiz. Then we use the integration method above to get the final estimate. This way, we will always have the optimal evaluation of the model given all the test samples that have been selected.
3.5 Active Feedback: Improve Model Learning Without Sacrificing Testing Accuracy
The proposed final risk estimation can provide label-efficient evaluation of the model performance. We can further improve the label efficiency used for model training. Assume at the t-th quiz, we have sampled and labeled a set of test data Q according to the test proposal q (-). Since Q, is also labeled dataset, the active learner could leverage it as a source of feedback information to facilitate learning. In the proposed active learning-testiin setting, the number of labeled samples for learning/trannin Nr. and testing Ny both contribute to the limited overall annotation budget. Since learning and testing need to be sample-efficient, it is reasonable assume that Nt and Nr are of similar magnitudes instead of having Ny >> Nr as in the existing active model evaluation works. We further denote SyB as the feedback set, which contains samples from the collection of quizzes. The active feedback process can be formally defined as subset selection problem that optimizes joint objective of learning and testing.
the the AL side since some of these labels also be
SiB arg min S.eE(Q..... .r) (R(oe()))s..) + C PeclQ RrHee
(8)
where C is parameter balance the two objectives: (T) R(Je((S.usm)) for learning and (IT) = || ((((Q....QT))sss)) for testing.
Theorem Both the learning and testing tasks have the same label complexity, which in the order of O(1/vn). As result, (1) the joint objective has an overall label complexity of (O(1/VNL + NrB) + O(I//Nr - Nrr)); (2) the balancing parameter C is in the order O(1).
Remark. The Theorem reveals that it is possible to achieve an optimal balance between (I) and (IT) by choosing a suitable Spg such that we can further improve the model learning performance while maintaining the risk estimation quality from our quizzes-testing process. provides a foundation to justify the benefit of active feedback. It is worth to note that the optimal subset selection problem is usually Na-hard, but we can still draw a useful conclusion that an appropriate subset Syp can poten- tially improve the combined dearning-testing objective rather than having St and ST {Q1, .... Qr}

---

always separated. The theorem can also be more intuitively interpreted by drawing analogy with human learning, who automatically incorporate knowledge or partial knowledge at inference time. In contrast, a typical supervised-learen need to explicitly add feedback so that the model can effectively improve the learning performance by getting new training samples.
MI. model is frozen at evaluation/testing time. Thus, we
Following the above theoretical result, we propose selection proposal (FB(.) for SFB based on the following learning-testir to be the most informative and diverse ones. From the informativeness perspective, samples with high losses could be challenging for the current model. From the diversity perspective. the newly added samples should be different from the current labeled training set, which can be achieved using
intuitions. In standard AL, we should select samples that are considered
a diversity metric, such as dlSr,,)) Vx x/xAAp'x where Ar,
where > o.From the evaluation perspective, removal of feedback samples should have as little impact to the risk estimation as possible. Since the test samples are chosen according to q(x). it is less harmful to choose a feedback sample with large ((x) as similar testing samples are likely to be selected into the test data. To this end, we propose the following feedback selection strategy:
+ Ezes, zz
x" arg max QFB (x, 1) arg max (x) + r)d(Sr, (x.))E
(x.y)ER:
where is a scaling parameter. We construct SFB after each active quiz round. The samples are added in St re-train the model, and the risk estimation also re- weighted after the removal.
3.6 Early Stopping with Instant Risk Estimations
In the ATL framework, the unbiased risk estimation after each AL step can be used to construct an effective convergence indicator to support early stopping of AL. In particular, we propose to use the change of a moving average of active risk estimations:
Eisrow U.R,
E 1 U.R,
ARe
(10)
Ei 1/
U.
Prior works show that leveraging the unlabeled data works well empirically to stop AL early 12 6 To this end, we propose to further augment A AR, with stabilized predictions (SP) from the unlabeled data 66, where SP 1 mean (Ay()))
4 Experiments
The proposed ATL framework is generic and can be applied to any existing active learners. In our experiments, we choose two commonly used models, Gaussian Processes (GP) and neural networks, to demonstrate the effectiveness of the proposed work. The GP is applied to 2D synthetic dataset to help us understand the important behavior the proposed test sampling strategy and how active feedback benefits the entire learning process. The neural network model is applied to relatively larger scale image datasets, including MNIST, FashionMNIST and CIFAR to demonstrate the practical performance of the proposed framework. As for the AL strategy, ATL is designed to be AL-agnostic. To demonstrate the general applicability, the main results are obtained using uncertainty (i.e,, entropy) based sampling as the AL strategy given its great popularity in many AL models. In Appendix C.2.3 we show that ATL can be easily integrated with a wide range of commonly used AL algorithms.
4.1 Experimental Settings
For synthetic experiments, we generate 2,500 data samples based on a moon-shaped distribution with two smaller inverted moon-shaped clusters two remote corners. The dataset is shown in [ (a). We add the smaller clusters because the learning performance is usually closely related to how AL explores the unknown regions in the feature space and the designed shape allows to visualize testing and feedback samples to evaluate their effectiveness. Moreover, we create two levels of imbalance in the dataset, the intra-class imbalance and the inter-class imbalance, to simulate an imbalanced class distribution. For the real-world experiments, we use the cross-ent cross-entropy loss for risk evaluation. When we compare with the true risk R, we use the average evaluation results of the model on a large hold-out subset of the dataset (10,000 data samples) to represent R. The hold-out test set does not interact with the AL or AT processes, thus considered fair evaluation. real-world experiments, we adopt the same procedure with the total pool containing 30,000 data samples.
7

---

Deteate Orar's 2003
(a) sampling (quiz 9)
(b) Random (quiz 9)
ATL-NF (quiz 9)
(d) ATL (quiz 9) Deivet 2eas
cntrertion Drar 20:35
Extrratin Evech 4o:33
(e) AL sampling (quiz 18) (I) Random (quiz 18) (g) ATL-NF (quiz 18)
(h) ATL (quiz 18)
Figure 3: Active test samples selected by differrnnt criteria at quiz 9 [(a)-(d),, and 18 (c)-(h)
In each test sampling round, we compute over the entire unlabeled pool according to (4). We use deterministic way of summing over all classes based on the posterior distribution predicted by the model. The multi-source estimate only utilizes labels of the training samples and previously selected test samples along with the predictions for unlabeled samples. Later in the active feedback stage, if a test sample is selected to be added to the training set, we remove it from the testing set. For test and feedback selections. we sequentially sample ny times obtain each batch. The testing process does not involve re-training the model, thus the batch mode has no effect.
4.2 Synthetic Data Experiments
In Figure 3. we show the distribution test samples selected using (1) AL sampling, (2) random sampling, (3) ATL w/o feedback, and (4) ATL at an early stage (i.e., quiz 9) and later stage (i.e., quiz 18) of the learning process. We observe that using AL sampling for test point selection has less accurate risk estimation performance due to highly biased sample selection near the decision boundary. Random sampling, on the other hand, follows the true data distribution, so it mainly selects from the central region, where the data is dense. Our proposed test sampling focuses the most representative regions as random sampling does, but it also covers other interesting regions with relatively low density (e.g., the right edge where the minority blue class is mainly located), and the model evaluation reaches the highest accuracy in the early phases of learning.
In Figure 3d we plot the feedback with the same annotation cost to demonstrate its learning-testine trade-off. By providing part of the testing data as effective feedback, the estimation error is slightly increased. However, the sacrifice in model evaluation accuracy pays off from the training side. The feedback helps the AL model build a more representative training set and learn a more reasonable decision boundary than without feedback. It allows the AL model to gain a greater advantage in the early stage of training. Figures 3e show the test data distribution a later stage at quiz 18. The proposed testing selection is the only that picks samples test the critical but low-density regions. More importantly. Figure (3h shows that learning with feedback is most successful as acquires informative training dataset and near-optimal decision boundary in the end.
4.3 Real-Data Experiments
The main results of real-world experiments consist of three parts: (1) risk estimation during standard AL (without active feedback), (2) AL performance and risk estimation with active feedback, and (3) AT- based early stopping. In this section, we focus on showing the results from the first two and leave details of part (3) the Appendix. In the AL without feedback case, we compare the estimation error results of ATL--F (no feedback) with ARE quizzes integrate 200 and the adapted Active Testing [14 and Active Surrogate Estimation 15) (referred to as AT integrate and ASE integrate). We use in the AL setting by adjusting the pool size N at each round and including the selected training loss maintain the unbiased estimator RLURE. We also need to train a surrogate model using fewer samples than normal (we use a separate NN model re-trained every few rounds). Because the AL model and the surrogate models are both severely under-trained in the early stage of AL, and that the training samples must also be considered in RLURE RE, the estimation
adapt the AT and ASE baseline to
8

---

error is usually significant in the AL setting. In the active feedback experiments, we compare the proposed ATL with ATL-NF and ATL-RF (random feedback).
Table 1: Estimation enror: squared difference between estimate and true risks (x10
AL round
Dataset
12
16
Method
ARE quiz
5.27
6.39 1.54
2.96 + 3.45
8.85 + 4.31
8.31 3.96
MNIST
AT inlegrate
16.3 24.5
32.8
22.1
6.93 + 18.0
8.72 + 3.59
3.11 2.98
ASE integrate
3.76
1.45
1.00
2.17
= 5.06
4.00 2.37
5.88 5.27
ATL-NF
2.37 1.17
0.79 4 1.15
0.17 0.15
0.36 -: 0.30
1.32 = 0.37
ARE quiz
3.01
4.62 0.77
8.63 2.47
5.71 1.87
23.78 : 1.75
Fashion
AT integrate
1.996.1
36.1 1011=30 30.7
34.1 + 31.4
28.0 36,9
22.5 25.7
MINST
ASE integrate
11.1366
3.12 3.53
3.56 016:55.00
5.29 9.78
8,12 2: 6.72
ATL-NI
3.64
1.61
0.67
0.96
0.98
3.04 1.37
ARE quiz
10.1
8.779
13.8 13.0
22.2 14.7
21.9 31.4
14.1 13.4
AT integrate
6.89 + 6.98
12.0
7.18
21.8 + 5.73
12.9 + 9.76
38.9 25.6
CIFARIO
ASE integrate
10.9 3.07
0.51
2.87
7.53
1.46
17.6 2.06
23.2 6.10
ATL-NE
8.83 7.79
3,06 5.04
4,95
1 7.12
7.94 1 5.22
6.20 5.79
Table 2: Hold-out test risk using different feedback criteria over 20 AL rounds
AI. round
Dataset
Method
8
12
16
20
AIL-NF
0.92 0.06
0.55
0.08
0.46
0.32 1
0.22 1 0.02
MNIST
AIL-RF
2.12
0.54
0.02
0.41
0.05
0.20
4 0.03
0.21 = 0.02
ATL
0.88 = 0.07
0.53 + 0.04
0.39 = 0.03
0.26 + 0.01
0.19 = 0.03
Fashion
AIL-NF
0.75 0.03
0.69
0.02
0.61 = 0.02
0.57
+ 0.04
0.56
= 0.03
ATL-RF
0.75
0.68
0.61 0.01
0.58
0.56
MNIST
ATL
0.74 0.03
0.65 = 0.04
0.59 = 0.02
0,56 = 0.03
0,51 = 0.01
ATL-NF
1.91 0.04
1.76 +: 0.05
1.72 = 0.01
1.66 + 0.02
1.55 1 0.03
CIFARIO
AIL-RF
1.91 0.03
1.77 1 0.04
1.69 - 0.03
1.60 1 0.04
1.54 = 0.07
ATL
1.90
1.65
1,58
0.02
Table 3: Estimation error with feedback over 20 AL rounds (x (x10-3)
AL round
Dataset
12
16
20
Method
MNIST
AIL-RF
26.8
21.4 17.0
3.54 + 4.01
5.54
3.21
7.62 + 4.41
ATL
14.6
16.9
13.7
3.19
2.63
4.15
3.20
1.87
1.41
Fashion
ATL-RF
10.2 9.30
4.41 4.11
2.19 1 5.53
5.69
4: 4.52
11.6 + 7,51
MNIST
ATL
2.50 +- 2.93
1.94 4 2.25
1.78 + 1.07
6,32
+ 5.41
5,03 + 4,41
ATL-RE
20.6 17.6
19.1 + 13.7
9.82
= 8.03
33.6
+ 30.5
24.8
= 32.4
CIFARIO
AIL
11.6 13.4
5.11
3,45
8.81
6.51
11,9
16.7
6.37 2: 6.29
AL-AT results. We use squared difference between the estimated risk and the true risk as well as the variance of the squared difference measure the estimation error instead of the variance of the estimate itself because each time we repeat the experiment the model is different. This is also a difference caused by studying the AL process, instead of evaluating a fixed model. From Table [ we see that ATL-NF has a much lower eтror of estimation than single ARE quizzes and AT/ASE integrate. The ARE quizzes put all testing samples without adjusting the importance weights, thus will result in a bigger estimation eиror. The drawback of the AT method is that the surrogate model is not a very reliable reference and the evaluated model changes over time. Because it selects samples based on a max-score criterion instead of importance sampling, the change of model may affect the unbiased estimate. The ASE integrate has the same issue of relying on a good surrogate model. However, the risk estimation of ASE is also based on the surrogate model predictions, which is consistent with the test sample selection. Thus, it outperforms AT integrate most of the time.
Active feedback results. We compare traditional AL and active learning- with feedback. We use both the cross-entropy loss as the metrics for the model performance. In Table we see that the proposed feedback and random feedback can both help the model reach smaller testing loss than pure AL with the same total number of labels. The initial training set contains 500 labels, while each of the 20 AL rounds adds 500 labels. AT then samples 100 points after each AL round, and the active feedback sends 50 labels back training. Because of the importance sampling with replacement nature of the process, the feedback approaches actually use fewer total labels in the end. We also study the impact feedback on the estimation error in Table 3 In the early stage of AL, it obvious that random feedback is harmful to the estimation error. The proposed approach also suffers from the increase of the estimation error. However, as the AI. goes on, our proposed feedback selection can maintain low level of estimation error, while improving the model performance. Combining these

---

benefits, we can potentially save the labeling budget for both learning and evaluation purposes. We report further ablation studies on the feedback proposal in Appendix C
We also provide a study on the size of the feedback set. As mentioned the proof for Theorem we keep the size of feedback simple in this work. This is to be consistent with theoretical ana ysils and the experiments show that the active feedback process is helpful in this generic setting. Further details about extending this have been discussed as part of the future directions. However, even in the simple setting of fixed feedback size, we can see that the learning and testing performances do not consistently and monotonically change with respect to the feedback size. From Tables and below, we can see that in general, model risk (learning performance) is better when we use larger feedback size, but at the same time the estimation eтror (testing performance) may become much worse. The model risk on CIFARIO 10 behaves differently with the feedback size, probably because the model performance is not good enough and adding difficult samples in this stage does not necessarily help with the generalization ability.
Table 4: Hold-out test risk using different feedback criteria over 20 AL rounds
AL. round
Dutaset
Feodback size
8
12
16
20
83%
0.86 + 0.09
0.53 = 0.04
0.40 4 0.08
0.30 4 0.02
0.20 = 0.03
67%
0.87 + 0.08
0.52 + 0.03
0.35 + 0.03
0.30 + 0.02
0.21 = 0.02
MNIST
50%
0.88 + 0.07
0.53 = 0.04
0.39 = 0.03
0.20 0.01
0.19 0.03
25%)
0.94 0.06
0.54 = 0.03
0.42 1 0.08
0.35 4 0.02
0.25 0.02
20%
0.39 1..44-..0 0.04
11.66 + 0.08
10.43 4 0.00
0.38 + 0.01
0.24
0.02
83%
0.74 0.02
0.87 + 0.03
0.60 + 0.03
0.54 4: 0.02
0.51 = 0.03
67%
0.77
0.52
Fashion
MNIST
50%
0.74 1744+110 0.03
0.65 + 0.04
0.59 + 0.02
0.56 + 0.03
0.51
0.01
25%
0.76 0.02
0.70 0.01
0.62 + 0.02
0.59
0.05
0.53 = 0.03
20%
0.77
0.71 0.02
0.61 0.02
0.61
0.00
0.51 0.04
83%
1.92 1.000
1.71
1.07 0.07
1.59 0.001
1.57 0.00
67%
1,96 - 0,00
1.75 1155:00 0.02
1.64 0.04
1.58 0.04
1.58
CIFARIO
50%
1.00
1.76 0.02
1.65 1 0.03
1.58
0.02
1.53 0.02
25%
0.03
1.59
20%
1.91 0.03
1.76 = 0.02
1.73 4 0.03
1.59 4 0.02
1.63
0.02
Table 5: Squared difference between the estimate and the true risk over 20 AL rounds (x 10-33
AL. round
Dataset
Feedback size
8
12
16
20
83%
50.2 = 39.8
21.0 = 24.3
1.36 4 8.44
11.4 + 12.7
7.59
4.45
67%
25.6 +: 23.4
29.3 + 29.7
6.90
8.05
6.24 + 6.71
7.50
5.07
MNIST
50%
14.6 22.1
16.9 13.7
2.63
1.15 3.20
L87
1.41
25%
11.7
10.0 798
9.73
4.76 5.25
1.59
1.66
20%
28.0) 24.4
11.8 + 14.5
5.91
+ 3.82
4.31 + 4.80
1.25
1.36
83%
8.39 8.99
7.52 2 10.4
2.77 4: 3.58
3.87 1: 4.45
11.1
7.02
Fashion
67%
8.59
50%
2.50 + 2.93
1.94
2.25
1.78
1.07
6.32 + 5.41
5.03
4.41
25%
3.04 = 4.00
2.33 - 488
1.54 2 1.18
6.40
8.06
4.13
3.99
20%
2,62 2 1.57
1.56
1.77
2.12 4.52
5.05
4.33
5.22
3.27
83%
$1.55011
113
7.75
56.1
17.0
17.2
31.3
02.2
43.3
67%
21.62250 25.6
36.1
1 20.5
24.1
18.6
30.7
40.8
36.2
CIFAR1O
50%
11.6 13.4
5.11
3.45
8.8
6.51
11.3
16.7
6.57
6.29
25%
5.800
4.48
20%
5.44 4: 6.65
3.65 3.44
11.2 4 11.0
4.21 + 1.36
5.82
3.34
5 Conclusion
We address real-world challenge for sample-efficient learning, where valuable labels from human experts that can be used for testing/evaluation are also scarce. We propose an ATL framework that builds active on the fly, with the potential to further improve the model performance without extra labels and opening up the creation of systematic early stopping criteria. We theoretically prove that ATL has an unbiased and label-efficient estimator and provide an analysis that shows how the label complexity dependencies maintained through active feedback. The experiments show that ATL optimizes the number of human labels needed in learning by simultaneously acting as fair referee and an educative teacher.
loop to achieve label-efficient evaluation of an AL model
10

---

Acknowledgements
This research was supported in part by an NSF IIS award IIS-1814450 and an ONR award N000 14- 18-1-2875. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agency. We would also like to thank the anonymous reviewers for their constructive comments.
References
[1] Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning, Journal of Computer and System Sciences, 75(1):78-88) 89, 2009.
[2] Garrett Beatty, Ethan Kochis, and Michael Bloodgood. The use of unlabeled data versus labeled data for stopping active learning for text classification. 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 287-294. IEEE, 2019.
[3] Paul N Bennett and Vitor R Carvalho. Online stratified sampling: evaluating classifiers at web-scale. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 1581-1584, 2010.
[4] Andrew C Berry. The accuracy of the gaussian approximation the sum of independent variates. Transactions of the american mathematical society, 49(1):122-136, 1941.
[5] Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning, pages 49-56, 2009.
[6] Michael Bloodgood and John Grothendieck. Analysis of stopping active learning based stabilizing predictions. arXiv preprint arXiv:1504.066 06329, 2015.
[7] Michael Bloodgood and K Vijay-Shanker. A method for stopping active learning based on sta- bilizing predictions and the need for user-adjustable stopping. arXiv preprint arXiv: 1409.5165. 2014.
[8] Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Ningshan Zhang. Adap- tive region-based active learning. In International Conference on Machine Learning, pages 2144-215 PMLR, 2020.
[9] Carl-Gustav Esseen. On the liapunov limit error in the theory of probability, Ark. Mat. Astr. Fys., 28:1-19, 1942.
[10] Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXi::2101.11665 2021.
(11) Claudio Gentile, Zhilei Wang, and Tong Zhang. Achieving minimax rates in pool-based batch active learning. International Conference on Machine Learning, pages 7339-7367. PMLR, 2022.
[12] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint / 2011.
[13] Namit Katariya, Arun Iyer, and Sunita Sarawagi. Active evaluation of classifiers on large datasets. In 2012 IEEE 12th International Conference on Data Mining, pages 329-338. IEEE, 2012.
[14] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, pages 5753-5763, PMLR, 2021.
[15] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
11

---

[16] Anurag Kumar and Bhiksha Raj. Classifier risk estimation under limited labeling resources. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 3-15. Springer, 2018.
[17] Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. International Conference on Learning Representations, 2020.
[18] Lutz Prechelt. Automatic early stopping using cross validation: quantifying the criteria. Neural networks, 11(4))61--67, 1998.
[19] Christoph Sawade. Active evaluation of predictive models. Universitätsverlag Potsdam, 2013.
[20] Christoph Sawade, Niels Landwehr, Steffen Bickel, and Tobias Scheffer. Active risk estimation. In ICML, 2010.
[21] Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active estimation of f-measures.
Advances Neural Information Processing Systems, 23, 2010.
[22] Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. Advances in neural information processing systems, 25, 2012.
[23] Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[24] Burr Settles, Active learning literature survey. 2009.
[25] Surya T Tokdar and Robert Kass, Importance sampling: a review. Wiley Interdisciplinary Reviews: Compulational Statistics, 2(1):54-60, 2010.
[26] Andreas Vlachos. A stopping criterion for active learning. Computer Speech & Language, 22(3):295-33) 112 2008
[27] Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neural networks (IJCNN), pages 112-119. IEEE, 2014.
[28] B. Welch and H. W. Peers. On formulae for confidence points based on integrals of weighted likelihoods. Journal of the Royal Statistical Society. Series B (Methodological), 25(2):318-329 1963.
[29] Emine Yilmaz, Peter Hayes, Raza Habib, Jordan Burgess, and David Barber. Sample efficient model evaluation. arXiv preprint arXiv:2109, /2043, 2021.
[30] Tiancheng Yu, Xiyu Zhai, and Suvrit Sra. Near optimal stratified sampling. arXiv preprint A9Xiv 1289, 2019.
[31] Jingbo Zhu, Huizhen Wang, and Eduard Hovy. Multi-criteria-ball based strategy to stop active learn- ing for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics- Volume I, pages 1129-1 136, 2008.
12

---

Appendix
Table of Contents
A Summary of Notations
14
B Proof and Additional Analysis of Main Theoretical Results
B.1 Proof of Theorem
Corollary
B.2 Proposition
B.3 Proof of Theorem and Active Feedback Analysis
C Additional Experiment Results
18
C.I Synthetic Experiment
C.2 Real-world Experiments.
C.2.1 Experimental Settings
C.2.2 Random Hold-out and Random Feedback ................................
C.2.3
ATL with Various AL Strategies
20
C.2.4
Additional Active Feedback Comparisons
20
C.2.5 Single Feedback Round Comparison
C.2.6 Early Stopping in AL
2.9
D Details of Hardware for Experiments
E Limitation, Future work, Social Impact
23
F.1 Limitation and Future Directions
23
Specifying AL Strategies Feedback Size Control and Balancing the Learning- Objective Connecting Saaming-Testingg- Testing-Feeddaael Feedback Proposals
23
24
E.2 Social Impact
F Algorithm and Source Code
24
13

---

Appendix Overview
Organization. The Appendix is organized as follows. We first provide a summary of notations in Appendix A] Then, present the details of the theoretical analysis in the main paper in Appendix B We further provide additional experimental results in Appendix [ including a synthetic visualization for different feedback approaches in Appendix [C. and more detailed real-world experiments in Appendix C.2 The hardware information is presented in Appendix D
We discuss the limitation,
future direction, and social impact of the proposed work in Appendix
Th algorithm block for the
pseudo code and the the link to the source code are presented in Appendix
A Summary of Notations
Table 6: Summary of key notations with definitions
Notation
Definition
(x. u) E &xy
Data points
D(,(x,
Data distribution
Lo
Loss function of model fo()
R
True risk
True risk evaluated on model whose parameter 0 is learned
R(felp)
from dataset D.
St.
Laheled set and unlabeled pool
Qi = (xx)
The f-th quiz
9(x).9
Test sample selection proposal and the optimal proposal
Rg: Ri
Risk estimator indexed by the test proposal q or time step
R
Integrated risk estimator
Model confidence of ft and weight coefficient for time step
Ce, Ve
in final R
Seu
Active feedback set
Nr. Nr, Nya
Number of samples in learning, testing and feedback sets Diversity metric, diversity norm matrix, small positive value to avoid singular issues
((.e),Ar.
Feedback proposal, balancing parameter between the proposal- loss term and the diversity term in the feedback proposal Balancing parameter for the risk estimation in unlabeled- information-combined early stopping criterion
Orn (x),
B Proof and Additional Analysis of Main Theoretical Results
B.1 Proof of Theorem 1
Proof. We start by presenting the asymptotic convergence of the active risk estimator and the solution for the optimal testing selection proposal g (x). From 20), we know that using the risk estimator fi.., we would get an unbiased estimate the true risk R because is essentially an importance sampling
based estimator. Then from the central limit theorem, Re Ei= (00 and W,, = Σm= w are asymptotically normally distributed with
1
Vn
R R) n
n No0 var (1
(11)
Vn
Wn -1
NO, var (1)
(12)
n2
Then, with the multivariate delta method, we know that if Yn = (Yn1. .... Ynk) is a sequence and vnl(n, 1)) a-000, (N 2), then
VTl(((t)) s(p)) m000 (olv) Ev
(13)
14

---

Here the function is g(z, v) = with = 1 Rn and 1 Wn. The result is
vn
R n- N(0,
(14)
W
where o = p(x) 9(अ)
(IICU())) (()))((u))))))))) p(x)dx.
Then, the optimal test proposal is obtained by minimizing ogg. By introducing Lagrange multiplier for the constraint f ((())xx 1, we have
L(g, B) og + ((ox))dx-1)
(15)
0L p(x)² [(C(S(x),)) v) RRO))p(vix)d +
(16)
dq
g(x)2
Thus, have q² (x) p(x) [IC(f(x).)) v) - p(f)))(v/xxddd.
Now, we provide the detailed proof for Theorem 1. As shown in Section 3.4 R satisfies
vn - R1) ~ N (0, diag [o)
(17)
Next, apply the multi-variant delta method. Define R² R. (R) = E v.Ro Then, we have Vg- (UI....,y)) u4) Given the diagonal covariance matrix, the final variance is:
of
of (v1,-.,, 04)
(01,-.,,2)
of
Σ
p(x) (x())
ot
((ICCH(O)))-RR//PPPEEPEE R(fe))P(xxxd p(x)dx
(18)
When we perform the "final exam" estimation after gathering all quizzes {Q), 2r). the other factors including testing proposals are fixed. We analyze the optimal solution for e by constructing the Lagrangian objective of + +7(2, Ur - 1) (where is a Lagrangian multiplier). By taking the derivative w.r.t each in along with the Lagrangian,
have
O 1 (0 + tee 1/T) 0
(19)
bv.
which leads to U Ex C.'
The Corollary below provides alternative view of Theorem
Corollary 1. If we do not change individual qs but still combine all available test samples, then adjusting their importance weight by ((( X W, gives the optimal estimator.
(6)
Proof. In the alternative view, we have:
R2
(2) (1) T Ex Ue
R
(20)
Σ
E VeW
where w/!
9e (x( We can view the final estimate R as a function of R.' and W' that has the form Then we directly analyze the expectation and variance of R using the delta method:
(9)
((X,Y)= T.
First we have
E(f(X,Y)) E[f(ux py) + fy (4x, px) + fl(xx - py) + R
=E[UUX my)) + (XX ux]] -
((x - MY)]
=R|s(ux uY)) + fx (E)(X ux)] - [y (E) - JYY)
-((xx,MY)
(21)
15

---

where yu B[ and jy = E Applying 21 on our estimate, and we get:
he
EL 24 E Rdj
E[R(fr]] =
E i=
Ei=1 14 Wntt
1=
where we utilize Et= 1= U2 = and Wi|
W7
=
R. For the variance, we have:
(X) = (X Y) (X, Y)1
= =I(f(X, Y) - f(xx, )
= MY + (xx, MY) MX) + (x - (x)
(IX -xxx
+2 -XX - (xx) (Y), uy
-xx)²
+ -(rr)) -py)a
fXl + +2f ffxxxxxxxxxx (ux, (x
HY (22)
Applying our estimate leads to
Var [t] 1e var + Var Var[ [2] (CCF)-2RCoV/ 2RCov R?1
R2 - 2 R
2REW
= ΣΣ p(x)
op (f14U)(()..)) R(JT)) p(x)dx
(23)
(x(x)
where we utilize f(X,Y) x + f3 4, 8r xz 1x = R, = 1. Note that since
we assume (x( are fixed, we have F[W' p((())) E.g .....rr 1Ei= 1/ ( (1) Ep(xix) E vr (w) ((x (1))
B.2 Proposition
We show two concrete examples for Proposition 1. In each case, the estimated introspective loss is analogous to an uncertainty measure.
The estimation of 0-1 loss is:
1
Σ ΣΣ [(fo(()) + v)p(v/x:0)
Ro
(24)
Su
XESU
which is the sum of the predicted probability of all classes other than the most probable class.
The estimation of cross-entropy loss is:
Rs
Σ XESu
logiolx) 0))
(25)
SS
which the entropy of the predicted probability.
When we use deep learning models, Rg usually largely underestimates the risk over the entire pool. In other works such as [14 [5], the surrogate risk acts in similar way. For the final risk estimator to be accurate, which somewhat beats the purpose of active risk estimation. However, we still try to improve this intermediate step without assuming that we have access an unrealistically accurate estimation, leading to our proposed Rg in Section 3.3
introspective risk estimation or the surrogate risk first needs to be accurate,
16

---

B.3 Proof of Theorem 2 and Active Feedback Analysis
we formalize the combined learning- testtingttnngg objective as joint optimization problem with the varable being a subset Spy that can be transferred from the testing set Sy to the learning set Sz. We define the process of selecting the subset as the "active feedback" process, which connects the learning and testing objectives through a balancing parameter C' given in (8). Performing exact optimization of the subset along with a parameter C' would require more detailed knowledge on the learning model and the AL strategy. We instead provide general analysis to show that active feedback could indeed provide an optimal solution for the joint optimization problem, where C' scales as (1). Following our theoretical result, empirically demonstrate the effectiveness of an intuitive feedback approach the experimental sections (Section 4.3 Appendix C
In Theorem 2
Proof overview. We apply some generic generalization bound (e.g., 17 for CNN or similar models)
the learning objective (I) in the joint optimization problem given by (8 which gives O(1/V
We then leverage the confidence interval to get a high probability bound for the testing objective (II, which also gives (11/ / m)
28 We use the formalized results on the convergence of the estimate as introduced in 20 With that, we continue to show that both the learning and testing objectives share the same dependency on n. These common dependencies on n give us the foundation to further analyze the feedback process. We offer an intuitive justification of active feedback as follows. The risk estimators are importance weighted estimates of the true risk. The estimate converges to the true risk asymptotically, so fewer samples might hurt the quality of the estimate (due to large variance), but does not change the fact that the expected average the estimate is still the true risk. With the confidence interval conversions, the objective's 's dependency on the number of samples does not change. (This also provides guidance for the feedback proposal later: if we can keep the change of the estimate to the minimum, meanwhile using the samples discarded from the test set to improve the AL model as much as possible, it would be the ideal use of available labels.) Following these high-level ideas described above, present the detailed proof below.
see that except for the change of constants,
Proof. We first break the joint (1) learning-(1)) testing objective into two parts and approach each part separately:
R(Je((22um))
& RCNN( + (1V/L/ N2 + Nen
S (oNN((((s..) (1 (1/VNz NN +
(26)
||- OrNSeilll < || Q R QT SFB)
+
R|
(27)
The learning objective. As mentioned earlier, (26 is a common generalization error bound for CNN similar models. For example, given training set Sy, with N₁, samples, we can draw from the basic bound (e.g., according Theorem 2.1. in
10| Nr
log(1/8) N
R(Jois.) = EDILSSS, (-) S Es, lioi,, (-) +)+ C" C
B'X
(28)
with probability of least 8, where C², 3', and X' are constants and [0] is the total number of trainable parameters the network. In our case, we do not make further assumptions about the constants and [6] is fixed for evaluating certain model. Similarly, Nr + Nrg and arrive at:
substitute Nr. with
R(Jonssusw)) = ED Epllrons (-) S Eis (SSE)))))))))))))))) Mo (-)
10
log (1/)) Nr. + NFB
+
3'X
Nr + NFB V
(29)
We notice that in both (28) and (2). we include the expected loss which slightly different from the best possible AL model risks R(fa R(fāj(s))ss) and R(fā(s). ). However, the difference is usually
17

---

on smaller scale than - 1/VNE=+ NNs)) In general, we assume that (( S R( x R( since more labeled samples can benefit learning (we do not need to assume strictly monotonic case for the sake of this analysis). For most AL strategies, the difference between the expected empirical risk and the optimal risks given the learning set size is on higher order of dependency on than the learning bound itself. If we ignore the higher order terms, we can simplify the results as shown in (26). Then, the term and more importantly the change in the learning objective that is related to assumed feedback STB is only dependent on NrB through O(1/ + Nra)
The testing objective. The relation in (27) can be further analyzed by taking a probabilistic view. If we assume the risks are bounded in the third moment, w.l.o.g., the two risk-difference terms can both be generalized a slightly more specific high-probabiltyy confidence interval 4 9 28 than the
plain central limit theorem result itself: with probability of at least - we have
(TQ ....., Qr1) RT(1Q1: Qr} \ &Sa)ll
2 F (1
a NNr
PNT - (1-2)
--
NN Nip
(30)
2
/NT
Nin
2)
v /NT NFB
F (1
NN
Qrl) R|| S 2
(31)
2
/Nr
where F-1 is the inverse cumulative distribution function of the Student-t distribution and 5² is the empirical variance. For the active feedback analysis, we only care about how NYB affects the testing objective, thus also obtaining an O(1/ NEB) dependency.
The detailed balancing between the two objectives (I) and (II) requires specific knowledge about the constants involved in the bounds. However, if we only focus terms involving NYg, both dependencies on the sample numbers are on the 1/vn level, making it possible to be balanced by constant factor C. Combining these results, we get the NFp term as O(1 + NFa) +
OL/v/Nr - NFB) (absorbing (L) terms that do not depend on NFg). The next key factor that throughout the entire ATL process, we either keep NYB fixed or only change it at a linear rate (flexible Nyu should be an interesting future direction). Combining with our previous assumption that N₂ and Nr are of similar magnitudes, we know that an optimal balance could be achieved between (1) and (II) to minimize the joint learning-teste -eettngggggiiggggg objective given in
C Additional Experiment Results
In this section, we present the detailed experimental settings and additional experimental results.
C.1 Synthetic Experiment
Figure 4 shows how the proposed feedback strategy helps to encourage exploration. The background color shows the model's predictive distribution. For each quiz, we display all the training samples obtained by an active learner (red and blue circles representing 2 classes) but only the current quiz (triangles) and feedback samples (squares, then added to circles later AL rounds) from the active tester to make the visualization clear. Figure 4a| shows that ATL selects feedback sample in the bottom right corner because is not included the current knowledge base of the AL model. The AL model predicts it poorly in the quiz. In Figure 4b we see that the AL model is guided by the feedback samples and starts to explore the bottom right corner. Once the AL model collects samples from the bottom right area, ATL stops to provide guidance for that region. In this way, the proposed feedback strategy manages to find the minority cluster at the other corner shortly as shown in Figure 4c
In Figure 4d. to further demonstrate the effectiveness of the proposed feedback strategy, we compare it with the feedback samples selected using two other baselines: random feedback (in Figure 41) and AL based feedback (in Figure when the samples at the bottom right corner are first discovered. First, we notice that those data samples are found by the AL model rather than through the feedback strategies. As result, it happens at much later quiz time compared with ATI.. Therefore, they result in less efficient learning process. Second, we observe that when an AL model discovers
18

---

Estimation Error:2.5e-03
Estimation Error:2.0e-03
Estimation Error:2.0e-05
ALtarge
redis Sanale
A.3urg rediia Samale
Sack
Al Aaaa
67k
E14
(a) ATL Feedback Quiz 2
(b) ATL Feedback Quiz 3
(c) ATL Feedback Quiz 4
Estimation Erron:4.2-055
Estimation Error:6.4e-0
Estimation Error: 1e-03
xT Sange
AiSanga
Gergle
2.28
6.4
(d) ATT Feedback Quiz 10
(e) AI. Feedback Quiz
(f)
Random Feedhack Quiz
Figure 4: Effectiveness of active feedback for improving model training
Table 7: Test classification accuracy
Feedback Strategy
Quiz 2
Quiz 3
Quiz 4
Quiz
Quiz 10
ATL
0.75
0,87
0.90
0.96
0.97
AL
0.75
0.83
0.84
0.84
0.88
Random
0.74
0.78
0.81
0.83
0.91
ATL NN
0.74
0.76
0.79
0.81
0.86
new area to learn, both baseline feedback strategies fail to provide support even though they have some test samples (i.e., the red point the bottom right corner) available in the interesting region. Last, we can see that at around quiz 10, the AL model with the proposed ALT converges to better decision boundary that captures the entire data distribution while the other baselines both fail to correctly discover the predictive distribution at the two corners. As result, ALT leads to more accurate model (shown in Table 7 while maintaining lower estimation error the end.
C.2 Real-world Experiments
In this section, we provide more results on the real-world datasets including MNIST, FaSH and CIFAR10, 10, mainly to demonstrate different feedback approaches and how we can implement early stopping ATI.
C.2.1 Experimental Settings
In all experiments, we use a CNN model and standard data transformation for each dataset. In each AL training round,
10 epochs for MNIST and FashionMNIST and 50 epochs for
CIFARIO. A threshold of 1 x 10-5 5 is used for probability outputs as required for the proposal (xx) computation [4 to avoid denominators.
An important detail to note that for ATL-NF results, we sample additional data points during AL for fair comparison (550 in each round), which is actually very similar to ATL-RF. The results with only 500 data points per round will be shown in the following section C.2.2 Another detail worth mentioning is that although we set the initial budget to be 500 labels and add 500 training
19

---

samples plus 100 testing samples in each round, the final total budget is 12, 450 on average instead of 12,500 500 because allow replacement while sampling.
C.2.2 Random Hold-out Test and Random Feedback
In this section, discuss the issues with traditional hold-out validation/testingg procedures during AL and compare the results using random sampling for both test sample selection and feedback selection with the proposed poaming-testinng feedbback process.
Table 8: Risk estimation error comparison with random methods
Al. round
Dataset
Method
4
12
16
20
Random Test
7,80 = 13.4
6.61 + 3.72
5.94 + 7.25
3.15 + 5.83
6.18 +: 2.41
Random Test & Feedback
30.4 42.2
16.8 16.1
5.34 + 6.33
11.8 + 10.6
5.62 4: 2.73
MNIST
Random Test Weighted
71.3 = 31.9
19.1
12.3
1222
10.0 11.7
5.64 1.64
ATL-NF
2.57 1.17
0.79 + 1.15
0.17
0.15
0.56 + 0.30
1.32 0.37
Random Test
6,97 = 11.2
5.29 + 3.56
10.9 = 6.55
5.40
- 2.76
8.56 + 7.57
Fashion
Random Test & Feedback
12.7 = 13.4
15.4 16.0
12.7 1 6.33
18.8 20.9
32.7 3: 15.4
MNIST
Random Test Weighted
6.85 = 11.8
3.01
2,80
= 11.1
1.58 + 29.0
9.51 9.13
ATL-NF
3,64 1.61
0.67
0.38
0.96
1 0,16
0,98 1: 0,43
3.04
1.37
Random Test
20.5 = 6.50
15.8
10.3
13.0
10.1
9.99 + 6.58
9.89 + 9.61
Random Test & Feedback
11.7 = 36.4
16.4
15.4
31.0
12.8
12.8
11.7
117210 8.65
55.3 19.0
CIFARIO
Random Test Weighted
13.5 14.8
14.0 11.3
15.1 I1.1
11.2
48.7
ATL-NE
883 7.79
3,06
5.04
4.95 7.12
1.441.22.
6.20 3.79
Syntheric
Random Test
4.17 - 0.78
8:00
0.71
11.2
+ 0.19
5.06 + 0.10
8.96 + 0.04
Random Test & Feedback
678
75.0
= 3.95
35.0 = 1.02
8.29
= 29
0.51 +
Random Test & Weighted
5.26 = 2.04
14.10
= 2.51
10.66 1.85
8.71 + 1.02
9.21 0.07
ATL-NF
3,67 0.11
2.86 0.63
1.48 1 0.14
1.20
0.26 1:26:1000
In Table random test is referring to randomly sampling 100 test samples after each 550-sample (additional 50 for fair comparison, same as ATL-NF) AL round and simply averaging the loss over these test samples. Random test & feedback is referring to sampling 100 test samples after each 500-sample AL round and then randomly selecting 50 for feedback. Random test & weighted is referring to the same process but the quizzes are weighted by 1 Rr. From Table g we can see that in the small-data regime, random sampling may not provide an accurate estimate of the true risk. However, in later AL rounds, the no feedback case (Random Test) can maintain an unbiased estimate, and we see that some results are comparable with active risk estimation baselines without the ATL-integrate estimator. This is probably because existing active risk estimation baselines (ARE-quiz, AT-integrate, ASE-integrate)
not consider the biased selection and model change through the AL process. The methods that use surrogate models also suffers from the insufficient training of the surrogate model. However, random testing selection does not work well with the active feedback process. For Random Test & Feedback and Random Test & Weighted, we often much worse estimation due to the feedback process involved.
C.2.3 ATL with Various AL Strategies
In this section, we show additional ATL estimation error results and active feedback results using two different AL strategies: margin sampling (27] and BALD 12
From Table 19] we can see that the integrated risk estimation performs similarly regardless of the AL strategy. The relative scales of different risk estimations are also similar to the entropy case in the main paper. From Table 10 we do see some different effects. Depending on the AI. strategy, random feedback performs unstably. However, our proposed feedback still mostly outperforms the fair comparison baseline. The study on AL-specific feedback strategies could be an interesting future direction.
C.2.4 Additional Active Feedback Comparisons
In this section, we show a more complete comparison between different feedback approaches. The feedback comparison consists of two parts: (1) baseline comparison including no feedback (ATL--NF), random feedback (ATL-RF), entropy-based feedback (ATL-EN) and (2) ablation study including loss-based feedhack (ATI.-LF), weighted loss-based feedback (ATL--WI.) and the proposed weighted loss plus diversity feedback (ATL).
20

---

Table 9: Squared difference between the estimate and the true risk over 20 AL rounds (x 10-3)
AL round
Dataset
AL strategy
4
8
12
16
20
Method
ARE quiz
16.9
2.30
3.86
6.91
10.1
AT integrate
11.7
18.9
15.3
8.35
2.64
MNIST
Margin
ASE integrate
3.96
3.78
5.04
11.1
17.6
ATL
3.40
3.49
3.94
1.93
2.46
ARE quiz
16.3
16.7
11.5
11.8
9.92
Fashion MNIST
AT integrate
6.29
5.66
6.98
7.04
2.55
Margin
ASE integrate
18.4
12.5
19.9
13.4
5.38
ATI.
1.14
0.95
0.87
0.79
0.62
ARE quiz
11.1
3.31
1.32
1.24
5.64
AT integrate
3.97
7.05
7.48
6.74
8.33
CIFARIO
Margin
ASE integrate
9.15
12.8
3.99
9.50
4.04
ATL
2.49
0.87
1.88
0.55
2.96
ARE quiz
16.99
9.16
5.75
6.91
10.1
AT integrate
8.48
35.9
11.7
28.5
19.1
MNIST
BALD
ASE integrate
16.4
5.52
7.88
2.16
4.81
ATL
3.83
4.41
5.24
1.54
2.82
ARE quiz
11.1
12.2
13.1
9.92
Fashion MNIST
AT integrate
6.09
4.04
7.33
8.86
8.75
BALD
ASE integrate
9.00
10.7
4.73
5.23
5.69
ATL
4.25
3.92
4.50
4.28
4.59
ARE quiz
3.04
8.93
3.11
19.4
13.2
AT integrate
12.5
22.3
21.9
15.6
14.9
CIFAR1O
BALD
ASE integrate
2.47
3.94
5.20
12.0
19.7
ATI.
1.09
3.82
4.65
7.21
10.6
Table 10: Hold-out test risk using different AL strategies and feedback methods over 20 AL rounds
AL round
Dataset
AL strategy
8
12
16
20
Method
ATL-NE
0.97
0.61
0.49
0.29
0.14
MNIST
Margin
ATI.RF
0.89
0.52
0.32
0.22
0.14
ATL
0.88
0.53
0.35
0.17
0.14
ATL-NF
1.30
1.05
0.90
0.82
0.69
Fashion MNIST
Margin
ATL-RF
1.23
1.06
0.88
0.73
0.70
ATL
1.14
0.95
0.87
0.79
0.62
ATL-NF
1.68
1.57
1.50
1.55
1.37
CIFARIO
Margin
ATL-RF
1.61
1.58
1.56
1.57
1.34
ATL
1.62
1.55
1.49
1.54
1.34
ATL-NF
1.03
0.75
0.49
0.36
0.29
MNIST
BALD
ATL-RF
0.91
0.58
0.44
0.35
0.25
ATL
0.89
0.59
0.46
0.35
0.25
ATL-NE
1.18
1.04
0.91
0.89
0.85
Fashion MNIST
BALD
ATL-RF
1.40
1.06
0.95
0.88
0.81
ATL
1.19
1.07
0.90
0.83
0.80
ATI-NE
1.95
1.92
1.78
1.72
1.72
CIFARIO
BALD
ATI-RF
1.93
1.91
1.79
1.75
1.73
ATL
1.88
1.80
1.76
1.56
1.57
First, show the hold-out test risk of the AL model throughout AL using different active feedback approaches as the indicator of the model performance. From Table T1 we see that in most occasions, all active feedback approaches reduce the test risk compared ATL-NE.
In Table [2] we show a full comparison of the squared error of risk estimation. All estimation results are based on the proposed ATL estimator fł, where ATL-NF, ATL-RF, ATL-EN serve as baselines, meanwhile ATL-LF and ATL-WLL serve ablation studies since the proposed ATL utilizes the weighted loss as well. We see that all feedback approaches suffer from an increased estimation error, especially in the early stage when the number of test samples available is small. We see that
21

---

Table 11: Hold-out test risk using different feedback criteria over 20 AL rounds
AL round
Dutaset
Method
4
8
12
20
ATL-NE
0.92
0.06
0.55 4 0.08
+ 0.06
0.04
0.00
AIL-RI
0.92 1 0.12
0.54 1 0.02
0.41 1 0.05
0.29 0.03
0.21 0.002
ATI-EN
0.90 + 0.12
0.55 + 0.06
0.41
+ 0.02
0.34 0.06
0.223 + 0.03
MNISTT
ATLALE
0.89 + 0.10
0.56 4 0.04
0.41
1 0.02
0.32 0.07
0.20 0.02
ATL-WL
0.86 0.06
0.53 + 0.06
0.40
+: 0.05
0.32 0.00
0.22 0.08
ATL
0.88 + 0.07
0.53 + 0.04
0.39 + 0.03
0.26
0.01
0.19 + 0.03
ATL-NF
0.75 0.03
0.69 = 0.02
0.61 = 0.02
0.57 = 0.04
0.56 0.003
AIL-RF
0.75 0.04
0.68 1 0.02
0.61 1: 0.01
0.58 = 0.06
0.56 1 0.04
Fashion
AIL-EN
0.70 0.02
0.67 = 0.05
= 0.02
0.59 000900.00 0.03
0.56
0.00
MNIST
ATL-LT
0.76 0.04
0.65 1 0.03
0.63 0.00
0.02 0.56
0.56
0.04
ATL
0.76 0.03
0.65 + 0.02
0.62 = 0.01
0.56 = 0.02
0.53
1 0.022
ATL
0,74 0.03
0.65 + 0.04
0.59 = 0.02
0,56
1.6600.03 0.03
0.51
AT
1.91 0.04
1.76 0.05
1.72 = 0.01
1.66
10060..00 0.02
1.55 0.03
ATL-RF
1.91
0.03
0.04
1.54 0.007
ATL-EN
1,92 = 0.00
1.76 0.04
1.70 0.03
1.66 = 0.04
1.51 0.00
CIFARIO
ATL-LF
0.04
1.75 + 0.03
1.66 + 0.01
1.59 = 0.03
1.54 0000
ATL-WL
1.94 + 0.04
1.75 + 0.03
1.63 0.01
1.63 0.03
1.54 0.01
ATL
1,90 1 0.05
L76:0.000
1.65 1 0.03
1,58 0.02
1,53 13310.022
Table 12: Squared difference between the estimate and the true risk over 20 AL rounds (x10-3)
AL round
Dataset
Method
ATL-NF
2.57 + 1.17
0.79 1.15
0.17 + 0.15
0.56 1 0.30
1.32 + 0.37
ATL-RI
26.8
21.4 17.0
3.54
400
5.54 3.21
7.62 4.41
ATL-EN
23.6 1 24.8
14.0 15.8
13.8
1 11.7
29.5 = 21.7
21.8 4 12.8
MNIST
AIL-LF
15.6 = 12.6
12.4
36.9
48.5
25.8
15.7 = 14.8
10.9 47.47744 7.44
ATL-WL
16.5 19.4
21.0
24.3
8.44
11.4 = 12.7
7.50 1 4.45
14.6 = 22.1
16.9 =: 13.7
2.63
4.15 3.20
1.87 1.41
ATL-NF
3.64 1.61
0.67
0.16
0.98 0.43
3.01 1.37
ATL-RF
10.2 9.30
4.41 3.77
2.19
1.19955.53 5.53
5.69 = 4.52
1..6
Fashion
ATI-EN
93.2 + 23.4
50.2 + 10.2
78.5 + 32.4
76.2 59.6
85.8 + 25.9
MNIST
AIL-LF
9.36 = 10.2
27.2 26.0
28.6
14.6 12.0
11.0 15.2
ATL-WW
8.39
6.50
7.29
4.45
11.1 7.00
ATL
2,50 = 2.93
1,94 1 2.25
1,78 1: 1.07
6.32 = 5.41
5,03 1 4.41
ATL-NF
8.83 +: 7.79
3.06 4 5.04
4.95 4: 7.12
7.94 = 5.22
6.20 5.79
ATL-RE
20.6 17.6
19.1 +: 13.7
9.82
8.03
33.6 30.5
24.8 32.4
ATI
30.3 17.0
17.4
31.7
27.0
CIFARIO
ATL-LE
35.0 27.9
45.8 1 28.5
20.3
1: 10.1
57.2 33.6
40.5
1 34.0
ATL-WL
22.7 = 19.7
25.0
+ 13.2
12.9
+ 21.5
52.2 45.9
28.7 + 16.3
ATL
11.6 13.4
5.11 + 3.45
8.81
6.51
11.9
16.7
6.57
6.29
the baseline methods suffer from increased estimation error. However, ATL usually maintain similar level of estimation error after 20 AL rounds. For ATL-LF, there is usually a larger variance of the estimation error. The potential reason for the unstable behavior of ATL-LF is that it only selects samples with larger losses in the feedback process. Although the importance mechanism can make up for some of the difference, there is still the potential risk of the estimate being biased. Further combining with the diversity metric, we achieve the best results with ATL.
Concluding from both the risk results and the estimation error results, show that the proposed feedback approach achieves a good balance in the we consider both the loss L and the importance weight g in the selection criterion. Overall, ATL achieves a similar model test risk as ATL-LF/ATTL-WL both of which are much better than ATL-NF and ATL-RF. ATL also achieves much lower estimation error than ATL-RF, ATL-EN, and ATL-LE.
trade-off. This is because
C.2.5 Single Feedback Round Comparison
In previous experiments, we add additional training points for the no feedback case (ATL-NF) to make fair comparison for the model risk. However, if we look at the risk change before and after a single feedback round, the difference is even more obvious as shown in Table [3]
C.2.6 Early Stopping in AL
In this section, we show how the ATL--baed risk estimation can be readily used for early stopping in AL. In the above experiments, we observe a steady decrease of the estimated risk most of the
22

---

Table 13: Hold-out test risk before and after specific feedback round AL round
Dataset
20
Method
ATI-hhefrre
0.91 + 0.09
0.54 + 0.04
0.41 + +0008
0.29 000+-0000 0.02
0.21
0.03
MNIST
AlL-alter
0.88 = 0.07
0.53 + 0.04
0.39 1 0.03
0.26 = 0.01
0.19 = 0.03
Fashion
betore
0.77 1 0.03
0.66 1 0.03
0.61 1 0.02
0.37 1: 0.03
0.53 1 0.03
MNIST
All-alter
0.74 0.03
0.65 0.04
0.59 0.02
0.56 = 0.03
0.51 0.01
ATI-hefore
1.97 + 0.07
1.82 0.05
1.70 0.03
1.67 + 0.03
1.57 +.07
CIFARIO
AlL--ater
1.90 0.05
1.76 + 0.02
1.65 = 0.03
1.58 = 0.02
1.53 0.02
times. However, we do find the decrease becomes more insignificant near the end of the 20 rounds of learning, especially for the MNIST and Fashion MNIST datasets. We observe that after a certain amount of AL rounds, the risk decrease is significantly small, and the corresponding test accuracy is also stabilized (MNIST around 94%, Fashion MNIST around 80%, CIFAR around 54%). This gives us the opportunity to apply early stopping in real-life AL applications. We here show the average stopping iteration and model performance (hold-out test accuracy) of the compared methods in Table [4] Following the same threshold value, by augmenting the moving average of active risk estimation given by (IO) with stabilized prediction (SP). the combined method can stop at a similar testing accuracy as compared with the SP method, but with much lower variance in test accuracy. Based on the threshold setting, it is also possible to stop AL much earlier, saving the overall labeling budget.
Table 14: Average early stopping iteration and final test accuracy comparison (with variance)
Datasct
Method
Iteration
Variance
Test Accuracy
Variance
SP
15
6.8
94.52%
6.0e -
MNIST
Comhined
11
1.2
94.08%
3.1e
Fashion MNIST
SP
16
4.4
81.32%
3.7e
Combined
12.4
L.04
80.12%
2.
SP
12
2.8
53.87%
1
CIFARIO
Combined
12.8
0.10
54.43%
D Details of Hardware for Experiments
All experiments were run on clusters with either NVIDIA A6000 or NVIDIA A100 graphic cards and Intel Xeon Gold 6150 CPU processors. The runtime of the experiments varies depending the number of repeat runs, but is usually on the scale of a few hours. For example, to get the 5 runs results of one ATL setting for 20 AL rounds on MNIST or Fashion MNIST may take about 6 to 8 hours. The CIFAR1O experiments may take slightly longer.
E Limitation, Future work, and Social Impact
In this section, we first discuss some limitation the proposed framework and identify some important future direction. We then discuss some potential social impact of our work.
E.1 Limitation and Future Directions
In this paper, we propose an integrated framework that combines active learning and testing. In the interactive framework, the exchange of training and testing information should be carefully guided. Although the proposed testing selection is statistically unbiased and the active feedback is backed by the high-level analysis, we still have room for improving the specification of methods in applicable settings, which we will introduce here as future directions:
From the learning perspective, we can improve upon the general setting in this paper. In this paper, we focus on introducing a general framework and working under the agnostic setting. However, using specific AL strategies use cases. There have been works that analyze AL label complexity bounds using either importance weighting mechanism in stream-based settings [ other methods in pool- based settings
potentially provide advantages in certain
Continuing on the results from the feedback size analysis Section 4.3 and the discussion above, the feedback size is very important factor in the process, especially if we allow the
23

---

size to change during AL. Further investigating the relationship between the sample size and the combined learning-testing objective can potentially improve the framework.
We also propose the ATL framework under AL-agnostic assumption. Given specific AL strategies, we might be able to also incorporate the learning or testing proposal in the construction of feedback proposal.
E.2 Social Impact
The proposed ATL framework considers the practical challenges of applying active learning in real-world settings, where both model training and evaluation require labeled data. It is critical step towards realizing label-efficient learning in practice, which can benefit many critical domains where data annotation is highly costly. To this end, the proposed ATL framework has the potential to fundamentally address the data annotation crisis and further broaden the usage of Al to benefit the entire society.
F Algorithm and Source Code
Here we present the pseudo code for the ATL framework using an algorithm block below:
Algorithm I: Active Testing While Learning (ATL.
Input Total number of quizzes: T,
Active leaning/tsttigffeedbaet budget: NAL, NAT, NeB Unlabeled pool: Su.
Active learning model at quiz t: fe, (x),
AL sampling strategy: a fo, (x) x Sy + R,
Learning objective L: L: fe, (x) + R.
Annotation method: h x
Output: Quiz samples Q Q1, Q2 Ort.
Quiz results: RQ = {RQ, RQ R
Annotated training dataset: S₁.
Sr. =
// Active Learning Samples
Q = 0
// Quiz Selection Criterion
3 for / = to T do
for to NAL do
= argmax al fe,, St)
// Active Learning Starts
S St fx, hxx))
Su So {x
// /ctivee Learning
Estimate Remullii using eq(5)
Active Testing Starts
Compute test distribution q( using eq(4)
10
Q₁ = (
11
for i = to NAr do
Select test sample x; ~ qxx,, Rgulii Su)
12
13
Q. Q. (x.)
Su \ for i NFR do
15
Choose x* using eq(9)
// Active Feedback Starts
St = St (x*
17
Q Ox
// Active Feedback Ends
Compute Ro. using cq(6)
19
Ro Ro U (Re.}
21
Q = QUQ.
// Active Testing Ends
Source code. The data and source code for replicating the results provided in this link: https //github. www.r/omoomingggoioccccc At..LAAALn.a//
24


Reviews:
[{"review_id": "ai6O4AgwVT", "reviewer": "Reviewer_LqNc", "summary": "This paper proposes a framework for integrating active learning and active testing in an online fashion. The proposed algorithm incrementally sample training and testing examples for each batch and can leverage all of the testing examples so far for evaluation. Moreover, the sampling distribution for testing is designed based on an estimate of the variance of the risk. Experiments are conducted for both Gaussian processes and neural networks on multiple datasets.", "questions": "1. The active learning strategy used in this paper seems to be very vague. What's the algorithm being used?\n2. There are quite a few components to the proposed algorithm. Can you give an algorithm box to make it clearer? Also, can you provide a time complexity analysis?\n3. In [1] appendix , the authors used a water-filling algorithm to make sure the examples sampled up to time t follows roughly from distribution $q_t$, even though $q_1, ..., q_{t-1}$ may differ from $q_t$. I wonder how this technique compares against what's proposed in 3.4.\n4. Please also see the weakness section. I think this work studies an important problem and proposes a cool solution. I am willing to raise my score if the authors sufficiently address (part of) my concerns.\n\n[1] Katz-Samuels, J., Zhang, J., Jain, L., & Jamieson, K. (2021, July). Improved algorithms for agnostic pool-based active classification. In International Conference on Machine Learning (pp. 5334-5344). PMLR.", "limitations": "Sufficient.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper studies a very important problem and provides an effective solution. I find the method to be novel.", "weaknesses": "1. I am not sure if the claim that the test risk estimate is unbiased. It is indeed unbiased if one knows the true risk, but when using the estimate proposed in section 3.3 to substitute the true risk, is this still unbiased? Moreover, if the feedback set is used in training, the estimator doesn't seem to be unbiased either. I think the authors should play down the claim of the proposed risk estimate is unbiased since it is not when combined with all of the tricks.\n2. In the experiments, the results of random sampling are not shown in the tables.\n3. There are quite a few heuristics proposed in the paper from sections 3.3 to 3.5. Although the authors provided intuitive explanations, the design of the algorithm may have overfitted to the three datasets the authors tested on. For example, the weighting between different components of the intermediate estimate seems quite arbitrary (section 3.3). It is also not clear how C is chose in practice in section 3.5. I strongly encourage the authors to further test their design across a wider range of settings.\n4. The paper assumes the active learning algorithm to sample based on a probability distribution. However, many practical AL algorithms do not. This can cause the test risk estimate to be really high for those algorithms. I strongly encourage the authors to address this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The unbiased risk estimate.**\n\nThank you very much for this insightful question! Please see the answer to Q4 and Q5 in the general response. Adding to Sections 3.2 to 3.4 in the paper, we discussed in detail the improved multi-source intermediate estimator (which impacts the variance-optimal solution but does not change the unbiased nature) in Q4, and the effect of feedback (for which we provide a practical solution and empirical studies) in Q5.\n\n**Q2: Random sampling results.**\n\nThank you for raising this concern. We have already included the random feedback results in Table 3. In Appendix D.2.2 Table 6, we also provide the results using random test sampling results and detailed analysis. Please note that this result is quite different from the AT comparisons from previous works without the AL setting. First, the AL changes the proposal distributions and the straightforward ARE/AT/ASE integrates actually introduce biases. Second, the base model is not accurate enough, making the test sampling proposal less optimal. From the results, we can see that random test sampling is comparable to the baselines as it also asymptotically converges to the true risk, but ATL performs better.\n\n**Q3: Analysis from Sections 3.3 to 3.5 and the constant C.**\n\nWe have further explained the problem setting in the general response. More specifically, we have established the quiz by using existing results in Section 3.2, improved upon existing intermediate estimate of the true risk in Section 3.3, theoretically constructed an unbiased low-variance integrate estimator in Section 3.4, and provided further analysis in Section 3.5. Most part of Section 3.4 and 3.5 are strict theoretical analyses using the properties of multi-variate Gaussian and other theoretical results. We have to introduce some heuristics in Section 3.3 and the later part of 3.5 to have practical solutions or discussions. \n\nThe weighting between different components of the intermediate estimate is based on the size of each component at the current stage. Each component (training loss, predictive uncertainty, and test loss) is assigned a weight according to the number of samples we have. The idea is to improve upon only using $R_{\\theta}$ since $R_{\\theta}$ usually underestimates the true pool risk (in a poor-calibrated case, overestimating is also possible). We agree that the current $R^{multi}_{\\theta}$ is still heuristic, but in future works we might further improve the estimate using some theoretical insights (which usually require more assumptions to be made about the model and AL). In Section 3.5, the constant $C$ is not a specific parameter, and is not utilized in the practical solution. It is used for the purpose of theoretical analysis, and the other part that matters is that in general settings it can be inferred to be a constant of $\\mathcal{O}(1)$, meaning that a balance can indeed be achieved between the two components in the joint learning-testing objective.\n\n**Q4: Practical AL algorithm and setting clarification.** \n\nWe would like to clarify that we work in a setting where the unlabeled pool is significantly larger than the labeled training set. Such a setting is commonly adopted that covers many practical scenarios where the labeling budget is limited or the labeling cost is prohibitive.  \nThe proposed risk estimate does not depend on (or make assumptions about) the AL sampling distribution. We agree that practical AL strategies might not have clearly defined sampling distributions, but given the scale of our problem setting, we assume the test risk estimation to be mostly orthogonal to the AL. As we explained about the unbiased risk estimation (see the answer to Q4 in the general response), the estimation error is expected to be low when we use the approximately variance-optimal test sampling approach. \n\n**Q5: AL strategy specification.**\n\nPlease see Q2 in the general response.\n\n**Q6: Algorithm box and time complexity.** \n\nPlease see Q3 in the general response. As for the time complexity, we should study each component separately as the number of samples varies. The AL and training components are standard. In test selection, we first compute $R_{\\theta}$ ($\\mathcal{O}(N_U)$), then evaluates $q(\\bf x)$ for the unlabeled pool ($\\mathcal{O}(N_U)$). The integrated risk estimation takes $\\mathcal{O}(N_UTN_T)$. The feedback might require an additional step of computing the distance ($\\mathcal{O}(N_TN_L)$). The sampling complexity should be comparable to AT/ASE methods, which require additional complexity $\\mathcal{O}(N_LTN_T)$ for training surrogate models. \n\n\n**Q7: Comparison with reference [1].**\n\nThank you for suggesting the reference! The method mentioned is a very interesting approach to achieving optimal sampling in multiple rounds. However, since the scope of the method is only on active learning, full replacement is allowed and the entire sampling serves the single purpose of ensuring the learning performance. In our setting, since our test sampling follows AL sampling and model training, the change of target is much more significant between rounds. As we show in Section 3.3, the sampling in each quiz is theoretically variance-optimal for the current model. However, as the model changes, the goal also changes, and we expect the sampled data instances needed to compensate for the change will be greater than the pure-AL case, meaning more labels might be required to achieve a globally optimal sampling result than the locally optimal one. Unfortunately, due to the vast difference in settings (AL vs ATL, binary vs general, among others), it is not likely we can provide a reasonable comparison between the methods. Our integration in Section 3.5 has a similar effect to adjusting to the global optimal (but using the fixed budget for each round) by adding another layer of weights to the local optimal according to a confidence-based metric. "}, {"review_id": "2kKP2rLP4I", "reviewer": "Reviewer_bQCr", "summary": "The paper provides a relatively new flavor to recent active learning works, by considering active testing. In particular, they evaluate the learned model on the fly on adaptively acquired set, called quizzes. Model learns based on those quizzes and part of it is added, as a feedback from the quiz. At the end, it is evaluated on the final exam set (cumulative quizzes excluding feedback examples). Statistical convergence guarantees have been provided, and low variance (unbiased) estimators are considered. Proof of concept is provided on test dataset w/ GP model and common classification datasets are considered for the real world setting. The main paper provides the results on how far is true risk (estimated from large number of held out samples) from empirical risk.", "questions": "Please refer to the weakness section. ", "limitations": "I feel that more dataset/arch should be considered to study the real impact of the early stopping method, and in particular it should be compared against (in a fair manner) to fix accuracy based criterion. While I appreciate the provided framework, it needs much more discussion as to how it would fit in the current paradigm, in particular when we have low budget active learning regime, or where we already start with a pretrained backbone (to be sample efficient). Lastly, I feel skeptic about the empirical estimates for the expectations, as usually the examples used for the testing would be small (as it counts in the labeling budget as well). ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a relatively new flavor to recent active learning works, by considering active testing. In particular, they evaluate the learned model on the fly on adaptively acquired set, called quizzes. Model learns based on those quizzes and part of it is added, as a feedback from the quiz. At the end, it is evaluated on the final exam set (cumulative quizzes excluding feedback examples). Statistical convergence guarantees have been provided, and low variance (unbiased) estimators are considered. Theory and practical aspect seem interesting to the broad community. It is generally well written.", "weaknesses": "Much of the weakness in the work are some of my confusions and doubts. I *strongly* appreciate the authors to add an algorithmic block which shows how each and every step is done in the true implementation. Lastly, I haven't verified the correctness of the proof line by line, but the theorems seem sound. \n\nOn AL acquisition functions:\n- I don't see any discussion at all about what AL algorithm is being used in all the experiments. Can authors please add that discussion.\n\nOn theory: \n\n- How is the sampling from optimal $q_t$ done to get the quizzes at each testing round? The expression mentioned in equation (4) involves an integral done over squared error of loss function and true risk and the true underlying posterior distribution over labels. While for the true risk, it is approximated using a multi-source estimate, I am not sure where do we get the labels to even define the distribution. \n- The multi-source estimate is a function of time step, that is ideally equation (5) should've $t$ in the LHS. Secondly, right after the first round, how are we getting $\\hat{R}_{Q_t}$ to multi-source estimate?\n- Wherever the expectation (integrals) are approximated using the empirical estimates in the quizzes or final test, or to get $C_t$, what is the scale of example? I feel that for small number of examples, $C_t$'s estimate can be pretty noisy, and therefore making $\\tilde{R}$ estimate noisy. \n- For the active feedback setting, is only 1 example chosen from the quiz set $\\mathcal{Q}_t$? If not, how is the batch selection done? Equation (9) only seem to be providing information for 1 example. If it is more than that, then I'd appreciate writing this as an algorithm (greedy procedure), or at least some discussion. \n\n\nOn experiments: \n- Can authors provide absolute accuracy numbers, and how does that change over the course of AL rounds? \n- How is the scale of error in Fig 3g so different than the others? \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: AL strategy specification.**\n\nPlease see Q2 in the general response.\n\n**Q2: Sampling using $q_t$ and distribution estimation.**\n\nAs we explained in Q1 in the general response, the test sampling is done in a sequential way. In each test sampling round, we compute $q({\\bf x})$ over the entire unlabeled pool according to Eq. (4). We use a deterministic way of summing over all classes based on the posterior distribution predicted by the model. The multi-source estimate only utilizes labels of the training samples and previously selected test samples, and also uses the predictions for unlabeled samples. Later in the active feedback stage, if a test sample is selected to be added to the training set, we remove them from the testing set. Our newly provided ATL algorithm pseudo code also demonstrates the entire process (see our answer to Q3 in the general response).\n\n**Q3: Multi-source risk estimate.**\n\nThank you for the suggestion, we will add $t$ to the terms in Eq.(5). In the first round, $\\tilde{R}$ is not considered because there are no test samples yet. In each round since the first, all the terms in Eq.(5) are based on the current stage of the ATL process. Meaning that at time $t$, we will use the current training loss $R_{train}$, the current test risk $\\tilde{R}$ (we will fix the notation in the paper, cannot add $t$ in openreview), and the currently computed $R_{\\theta}$ (computed in the same way as in ref[19]).\n\n**Q4: Estimate of $C_t$ and scale.**\n\nThank you for the question. As we find there is indeed a notation issue in the description of $C_t$, we would like to clarify that it should be $\\frac{1}{n_t}\\sum_{i=1}^{n_t}\\sum_y\\frac{p(x)}{q_t(x)}[\\mathcal{L}(f_T(x),y)-R]^2p(y|x)$ ($R$ estimated by (R^{multi}{\\theta}) at T). This definition is based on $\\sigma_t(F_T)$ as previously mentioned, which scales with $n_t$. Although we do not consider the final $\\tilde{R}$ noisy because of it, the number of test samples being small can indeed raise the concern of estimation quality, but this aligns with the fundamental challenge studied in this work. We have taken approaches to make the result as accurate as possible, e.g. the estimate of $(R^{multi}_{\\theta})_T$ is done over the remaining unlabeled set. Since $v_t$ is always used after normalization, the key idea is to compare the confidence of quizzes of different $t$ and apply weights accordingly, serving the purpose of achieving the approximately variance-optimal result in the end.\n\n**Q5: Active feedback setting and algorithm box.**\n\nPlease see Q3 in the general response about the algorithm. For test and feedback selections, we sequentially sample $n$ times to obtain each batch. The testing process does not involve re-training the model, thus the batch mode has no effect. For the feedback process, unlike AL where obtaining the label and re-training the model could have a large impact, we already have the labels for both training and testing samples at this moment. Re-training may have an impact, but is probably not worth the cost.\n\n**Q6: Test accuracy results.**\n\nThank you for the suggestion. We here present the test accuracy results corresponding to the tables in the paper:\n\n| Dataset      | AL strategy | Feedback | Iter4  | Iter8  | Iter12  | Iter16  | Iter20  |\n| -------- | --------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| MNIST | Entropy | NF | $83.6\\%$ | $91.2\\%$ | $93.2\\%$ | $94.8\\%$ | $95.9\\%$ |\n| MNIST | Entropy | RF   | $83.3\\%$ | $91.4\\%$ | $94.2\\%$ | $94.6\\%$ | $96.1\\%$ |\n| MNIST | Entropy | ATL | $84.3\\%$ | $92.2\\%$ | $94.3\\%$ | $95.3\\%$ | $96.1\\%$ |\n| FashionMNIST | Entropy | NF   | $69.9\\%$ | $77.0\\%$ | $78.1\\%$ | $81.5\\%$ | $83.2\\%$ |\n| FashionMNIST | Entropy | RF  | $73.0\\%$ | $76.1\\%$ | $78.6\\%$ | $82.0\\%$ | $83.3\\%$ |\n| FashionMNIST | Entropy | ATL   | $72.9\\%$ | $77.5\\%$ | $80.8\\%$ | $82.1\\%$ | $84.3\\%$ |\n| CIFAR10 | Entropy | NF  | $42.5\\%$ | $46.8\\%$ | $52.4\\%$ | $57.0\\%$ | $57.8\\%$ |\n| CIFAR10 | Entropy | RF   | $42.1\\%$ | $50.5\\%$ | $49.1\\%$ | $57.3\\%$ | $58.2\\%$ |\n| CIFAR10 | Entropy | ATL | $43.3\\%$ | $50.4\\%$ | $51.4\\%$ | $56.8\\%$ | $59.5\\%$ |\nAs we can see from the table, the accuracy results are mostly consistent with the true risk (lower loss coincides with higher accuracy) with few exceptions. We will further explain early-stopping related accuracy results in Q8: limitations.\n\n**Q7: Scale of error in Fig 3g.**\n\nThank you for noticing the detail and raising the question. Figure 3 demonstrates the learning-testing-feedback process on a synthetic dataset. Because the dataset is simpler, the true risk of the model is very low, especially at a later stage as in quiz 18. The estimation error is also low in this case, particularly with the no feedback case (ATL-NF) as in Figure 3(g). However, in this case, there is no benefit to model training from the feedback.\n\n**Q8: Limitations**\n\nThank you for the suggestion. We acknowledge that the current early stopping results have limitations as mentioned. However, the fixed accuracy stopping will result in a much higher variance in the stopping iteration. \n\n| Dataset      | Method | Iteration | Variance  | Test Accuracy  | Variance  |\n| -------- | --------- | ----- | ----- | ----- | ----- |\n| Fashion MNIST | Fixed | 15.8 | 1.36 | $81.33\\%$ | $4.4e-5$ |\n| SVHN | Fixed |  16 | 1.6 | $85.20\\%$ | $6.1e-5$ |\n| SVHN | Combined | 15.6 | 0.64 | $84.02\\%$ | $3.2e-4$ |\n\nWe agree that the expectations make the problem difficult. In this paper, we use the integration of quizzes, the improved intermediate estimate, and practical feedback solutions to address these challenges and conduct empirical studies. However, as the reviewer mentioned, there is still much room for improvement in the current framework. Again, we would like to re-state our contribution as proposing the first ATL framework, and presenting a practical solution that improves the learning performance while maintaining the quality of risk estimation given a limited total labeling budget."}, {"review_id": "dYhQ49XReb", "reviewer": "Reviewer_5t4X", "summary": "This paper considers the active testing procedure to assess the risk. The keys are 1) a proposal of a quiz using importance sampling, 2) combining the historical quizzes to make a test set for the final model test, and 3) a proposal of a feedback algorithm to strengthen the model using the labeled test data. Experiments focus on the performance of risk estimation, and some work concern test accuracy with feedback algorithms.", "questions": "I have some questions. \n\n1) what's the exact AL algorithm used in the experiments? \n\n2) Is there any effect of different AL algorithms on ALT? \n\n3) the feedback algorithm seems to be a hybrid of AT and ATL. Therefore, it seems that the benefit of feedback cannot be larger than the pure AL in the same budget (not the same budget in experiments)? \n\n4) Is the algorithm possibly adaptive to the quiz sets (test sets)? I want to know how to construct the test set in Table 2.", "limitations": "Moderate explained. The connection between AL and AT seems lower.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposal of an activel testing procedure can catch realistic problems. The quiz usage is impressive, and the set of quizzes is fully used.", "weaknesses": "Maybe we can consider a more straightforward example, such as the explicitly calculated risk. The performance of the proposed algorithms is validated. However, more solid experiment results are limited since the active learning algorithms are various, but only a few algorithms are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Explicitly calculated risk and limitation from AL algorithm.**\n\nWe would like to clarify that our design of framework is AL-agnostic, and we do not assume any fixed distribution for AL sampling. We will further discuss the AL strategies in Q2 and Q3 below. As for the test risk specification, in the evaluation, we do have the hold-out test risk as the reference true risk (we will further explain in Q5 below).\n\n**Q2\\&Q3: AL strategy specification and various strategies.**\n\nPlease see Q2 in the general response. Since our integrated risk estimator is AL-agnostic, different AL strategies do not have an effect on the risk estimation. However, the choice of AL strategies might affect how much the active feedback approach can improve the training. As we see from the results, the relative comparisons among different risk estimations are similar when using various AL strategies. Although random feedback might not be as effective when using BALD,  our proposed feedback still mostly outperforms the fair comparison baseline.\n\n**Q4: The benefit of feedback cannot be larger than pure AL.**\n\nWe would like to clarify that in our framework, both learning and validation are considered to be sample-efficient and we consider the total labeling budget. As we are not certain about the \"pure AL in the same budget\" setting description, to respond to the comment, we would like to discuss two different cases: 1) pure AL with full budget vs ATL; 2) fair comparison between AL and ATL. (For confusion about the hold-out test set used for evaluation, please see Q5 below.)\n\n1) In the first case, we assume that pure AL uses the entire budget $N_B$ for training, while ATL needs to divide the budget into $N_L$ and $N_T$ ($N_{FB}$ will be taken from $N_T$ and added to $N_L$). Indeed, the benefit of feedback can not be larger than using all labeling budget for AL. However, we are assuming that some validation is needed during the AL process, and there is no way to perform validation if the labeling budget is drained by AL. Thus, our goal is a combined objective of improving the training performance while not sacrificing too much of the estimation quality. \n\n2) In the second case, we compare AL using the fair budget $(N_{L})^*=N_{L}+N_{FB}$. This is the setting we use in Table 2 and Appendix D.2.3-D.2.4, Tables 7 and 8. In this case, whenever we compare the model performance (hold-out test risk), the model is trained using the same number of labels for ATL-NF and all feedback methods. In this case, we do not think a clear conclusion can be drawn about whether the benefit of feedback can be larger than pure AL. On the one hand, we have a limited selection range. On the other hand, we have already obtained labels for all samples within $N_T$, thus could potentially choose better samples. We acknowledge that this could be tightly related to the specific AL strategy, but at least in our experiments using entropy sampling, feedback results can in fact surpass pure AL results in most cases (Table 7). We do provide an additional study on the various percentage of labeling budget being used for active feedback in Tables 9 and 10 in Appendix D.2.4, which shows that the relationship is not trivial. We consider the further study on the trade-off between learning performance and risk estimation quality as an important future direction. \n\n**Q5: Constructing test sets and the evaluation setting.**\n\nFor the first part of the question, we do make an adaptation of active risk estimation approaches to the AL setting, where we formulate the integrated risk estimator from individual quizzes, aiming to minimize the estimation variance. For the second part of the question, the hold-out test set is fixed using a large number of samples to provide the reference true risk. The hold-out test risk is solely used for evaluation purposes. (Similar to AL works that still use the hold-out test accuracy as the evaluation method, we also need to have a hold-out test set to provide the reference true risk. The difference is that in pure AL, the hold-out test set only provides reference for the risk/accuracy performance, while here it is also used to compute the risk estimation error.) This should come naturally to any AL/AT work. "}, {"review_id": "qSMshaAAk7", "reviewer": "Reviewer_2XK2", "summary": "The paper introduces the Active Testing while Learning (ATL) framework, which efficiently collects testing samples for online model risk estimation during the active learning process. The online risk estimation enables early termination of active learning. Additionally, ATL establishes a connection between Active Learning (AL) and Active Testing (AT) through an active feedback mechanism that transfers selected testing data points to the training dataset, thereby enhancing model performance. ", "questions": "The paper needs more discussion on the potential bias introduced by the feedback mechanism. \n", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "1. The paper tackles two crucial problems in the AL domain: efficient sampling of testing data points and the combination of AL with AT. Both of these problems are highly relevant and significant in AL research. \n2. The paper introduces an unbiased estimator of the model risk, and also proposes a unique method to combine active quizzes as the final exam, maximizing the utilization of testing samples.\n3. The paper provides a comprehensive set of experiments to evaluate the proposed estimator and feedback methods. The experimental results demonstrate the superior performance of the proposed approaches compared to existing techniques.", "weaknesses": "In the first half, the paper focuses extensively on optimizing the data sampling process to obtain an unbiased estimator of model risk. However, a flaw arises in the feedback mechanism, which moves data points from testing dataset to the training dataset based on the rule described in equations (8) or (9). This feedback undermines the unbiasedness of the risk estimator by introducing dependence in the selection of testing samples. Specifically, when we extract samples from the testing datasets based on some specific rule such as (8) or (9), the remaining samples are no longer independently sampled, thereby introducing bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Bias introduced by the feedback mechanism.**\n\nThank you very much for raising this insightful point! As we agree with your comment, we here further elaborate on the problem following the answer in Q5. \n\nThe key objective of feedback is to improve model learning without sacrificing the risk estimation too much. In Section 3.5, we investigate active feedback as a way to optimize a (novel) joint objective of learning and testing. Our theoretical result reveals that it is possible to achieve an optimal balance between the two sub-goals (i.e., (I) and (II) as specified in Eq (8) of the paper) by choosing a suitable feedback set such that we can further improve the model learning performance while maintaining the risk estimation quality from our quizzes-testing process. Meanwhile, given the AL-agnostic nature of the framework, no further assumptions on the actual AL algorithms are available, thus leaving a general analysis. \n\nFollowing the general analysis of the combined objective of learning and testing, we focus on proposing a practical solution. To minimize the negative impact of the feedback process on the risk estimation, we propose a practical solution in Eq.(9) that considers the corresponding test sampling importance $q({\\bf x})$ of test samples. It aims to improve the combined objective of learning and testing in active feedback. The empirical results in Table 3 also show that we maintain the risk estimation better than random feedback sampling. \n\nIn the revised paper, we will further clarify the limitation of the feedback process and clearly state the further analysis of the feedback impact as part of future works."}]

Author rebuttal:
In this general response, we address a set of important questions that commonly occur in multiple reviewers' comments, avoiding repeating the same response in each individual rebuttal. 


 
**Q1: Clarification of the problem setting. (To all reviewers)**

In this paper, we aim to construct an integrated framework of active-testing-learning. This problem is a novel but practical one. Unlike many active learning (AL) or few existing active testing (AT) works that focus on efficient sampling designed solely for training or testing, respectively, the proposed work is the first effort that simultaneously considers both active testing and learning using an integrated framework. It tackles a much more challenging but highly practical setting, where the total labeling budget for the entire learning-testing (validation) process is limited. In the scope of the proposed work, we focus on studying risk estimation and the active feedback process that can work seamlessly with a wide range of different AL algorithms (we will elaborate on the AL-agnostic related results below). Following this rationale, our evaluation mainly covers the following perspectives: (1) for the first objective, we compare the estimated risk with a hold-out test risk (representing the true risk); (2) for the second objective, we evaluate how well we can maintain an accurate risk estimation while improving the model performance using active feedback. 

**Q2: AL strategy. (To reviewers LqNc, bQCr, and 5t4X)** 

The ATL framework is designed to be AL-agnostic so that it can be easily integrated with a wide range of commonly used AL algorithms. To demonstrate its general applicability, the main results are obtained using uncertainty (i.e., entropy) based sampling as the AL strategy given its great popularity in many AL models. We will more clearly describe and explain the chosen AL strategy in the revised paper. Additionally, we have gathered risk estimation results using two commonly used AL strategies: *margin sampling* (Wang et al. A new active labeling method for deep learning), *BALD* (Houlsby et al. Bayesian Active Learning for Classification and Preference Learning). 

The results (in PDF) show that the integrated risk estimation performs similarly regardless of the AL strategy. The relative scales of different risk estimations are also similar to the entropy case in the main paper. As for feedback results, we do see some different effects. Depending on the AL strategy, random feedback performs unstably. However, our proposed feedback still mostly outperforms the fair comparison baseline. 





**Q3: Algorithm block. (To reviewers LqNc and bQCr)**

Following the reviewers' suggestion, we have provided an algorithm block (in the attached PDF) that includes and organizes all the key steps in a systematic way.

**Q4: The unbiased risk estimator and the multi-source estimate. (To reviewers LqNc and bQCr)**

The unbiased risk estimation achieved by each quiz is guaranteed asymptotically by Lemma 1 in (Active risk estimation [19]), which we referenced in Eqs. (2)(3). Then, we extended the unbiased quiz-wise risk estimator to the integrated version (Section 3.4) through the multivariate Gaussian analysis before Theorem 2 in our paper. Since the asymptotic relationship is not great when $N$ is small, the need for reducing the variance of the estimation becomes even more important in the AL setting. The key idea of Theorem 1 is to analyze the best variance of the integrated estimator given quizzes. To achieve this, the test sampling proposal in each quiz is kept as in Eq. (4). 

Guided by our theoretical analysis, the multi-source estimate $R_\theta^\text{multi}$ of the true risk is introduced to construct the test sampling proposal distribution $q$ in practice. It is an improvement upon the $R_\theta$ proposed by Christoph et al. (Active risk estimation [19]). Ideally, if $R_\theta$ is close to the true risk $R$, the test sampling proposal will have the lowest variance while unbiasedly converging to the true risk. Similarly, in Active Testing ([13]), if the true risk is known, the optimal test sampling proposal can be directly constructed using the individual losses, resulting in an immediately converging estimator. However, in reality, the labels are unknown for the samples to be selected. Thus, an approximation has to be made and will affect the quality of risk estimation. We would like to clarify (similar to [13, 19]) that since the estimators are weighted (by importance weighting or AT-unbiased weighting) sums, they always asymptotically converge to the true risk (in the same way as random sampling will also converge to the true risk, just slowly). Thus, the quality of the estimation really depends on the variance of the estimator. Again, the true variance can not be known beforehand because we do not have access to the labels, thus the practical solutions are proposed to achieve good testing sampling. In Proposition 1, we show that existing $R_\theta$ is not optimal. By combining the current loss information that we do have access to, including ($R_{train},R_{\theta},\hat{R}_{Q_t}$), it allows us to bring the intermediate estimate closer to the true risk.



**Q5: The effect of feedback on the unbiased risk estimator. (To reviewers LqNc and 2XK2)**


We want to clarify that our paper explicitly claims unbiasedness only for the integrated estimator $\tilde{R}$. The active feedback process may indeed affect the unbiased risk estimator in a negative way. We show that the negative impact could be controlled in a combined objective of learning-testing. In a general analysis, we show that active feedback could indeed achieve an optimal solution for the joint optimization problem. Then we provide a practical solution and empirically verify the theoretical by showing that active feedback can indeed improve model learning without sacrificing the risk estimation too much. Please see the answer to Q5 from reviewer 2XK2 for more details.


Meta-review: This paper introduces the "ATL framework" that integrates active learning with model testing, The goal is to perform unbiased risk estimation. The reviewers noted the novelty and importance of the approach/problem. There is expectation that this will be of interest to the NeurIPS community.
Decision: Accept (poster)