SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dayou Yu<sup>1</sup> Weishi Shi<sup>2</sup> Qi Yu<sup>1*</sup>
Rochester Institute of Technology, Rochester, NY 14623
University of North Texas, Denton, TX 76203
{dy2507,qi.yu}@rit.edu1 weishi.shi@unt.edu2
In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, "testing", which often refers to the model evaluation process of using empirical risk to estimate the intractable true generalization risk, also requires data annotations. The annotation cost for "testing" (model evaluation) is under-explored. Even in works that study active model evaluation or active testing (AT), the learning and testing ends are disconnected. In this paper, we propose a novel active testing while learning (ATL) framework that integrates active learning with active testing. ATL provides an unbiased sample-efficient estimation of the model risk during active learning. It leverages test samples annotated from different periods of a dynamic active learning process to achieve fair model evaluations based on a theoretically guaranteed optimal integration of different test samples. Periodic testing also enables effective early-stopping to further save the total annotation cost. ATL further integrates an "active feedback" mechanism, which is inspired by human learning, where the teacher (active tester) provides immediate guidance given by the prior performance of the student (active learner). Our theoretical result reveals that active feedback maintains the label complexity of the integrated learning-testing objective, while improving the model's generalization capability. We study the realistic setting where we maximize the performance gain from choosing "testing" samples for feedback without sacrificing the risk estimation accuracy. An agnostic-style analysis and empirical evaluations on real-world datasets demonstrate that the ATL framework can effectively improve the annotation efficiency of both active learning and evaluation tasks.
Labeled data are essential for supervised learning in both model training and evaluation. Active learning (AL) provides a promising direction to reduce the human annotation cost by constructing a smaller but more effective labeled dataset for training purposes [24, 27, 12, 5, 1, 8]. However, AL only partially addresses the human annotation cost as the cost of constructing a labeled dataset for model testing has been overlooked. In reality, the annotation budget could easily be drained by building large test datasets that most modern AL methods rely on to evaluate the model performance and determine the stopping criterion for learning. In contrast to AL, active model evaluation or active testing (AT) focuses on actively selecting the testing data. Few existing efforts develop unbiased risk estimation techniques for label-efficient model evaluation [20, 14]. However, these methods are designed to evaluate fixed models that have been fully trained, making them ill-suited for evaluating an actively learned model, which is constantly updated and inadequately trained during most parts of the learning process. To address these fundamental challenges and realize label-efficient learning in practice, we propose to integrate AT with AL in a novel and efficient learning framework, referred to
as ATL. The ATL framework systematically addresses the unique active testing challenges arising from evaluating an actively learned model that is under-trained and constantly evolving. It leverages an interactive learning-testing-feedback process to better control the overall labeling budget, which not only achieves efficient evaluation of the AL model but also ensures faster convergence in model training. Through periodical testing, ATL gains useful insights to terminate the learning process as early as possible that avoids allocating unnecessary labeling budget.
Incorporating AT into AL is a highly non-trivial task. Unlike AL, AT has two inter-connected objectives: i) design of an unbiased risk estimator to quantify the model performance, and ii) design of a sampling strategy to select informative test samples. Achieving both objectives simultaneously under the AL setting with a constantly evolving and under-trained model is far more challenging than existing active model evaluations that assume a fixed and adequately trained
(a) Data distribution
(b) Active estimation error
Figure 1: Risk estimation comparison
model. For the first objective, its challenge stems from the potential biased active test samples brought by the second objective. We need an unbiased risk estimator to ensure the fairness of the evaluation. An unbiased risk estimator can be achieved by importance sampling or its variants [25, 20, 21, 22, 19, 16]. Recent active testing works also employ expectation analysis and propose other weighting mechanisms [10, 14]. However, all these works assume that the model being evaluated is fixed and already well-trained [20, 14]. Therefore, these estimators are not designed to support a dynamic AL setting, where the model continues to evolve when learning from newly labeled samples. Figure 1 shows that both the standard estimator (i.e., random testing) and direct adaptation of an existing unbiased estimator (ARE) [20] fail to provide an accurate and consistent model evaluation in the more challenging AL setting. As for the second objective, different from sample selections in AL, the testing selection needs to be compatible with the unbiased risk estimator. It is essential to ensure asymptotic convergence that guarantees an unbiased model evaluation while reducing the variance that can improve the convergence speed and provide a stable signal to terminate the learning process as early as possible.
Conducting AT along with AL enables ATL to seamlessly connect the active training and testing sides, which allows them to communicate with and guide each other within a well-integrated learning process. To this end, the proposed ATL framework leverages an active learning-testing-feedback (LTF) loop, which largely resembles real-world human learning. In particular, within the LTF loop, the active testing results are kept after each active learning round, which mimics the quizzes in the human learning setting. We refer to them as active quizzes. As shown in Figure 2, the
Figure 2: Overview of the ATL Framework
active learner acts as a student while the active tester acts as a teacher. Both the student and the teacher select data from the unlabeled pool and get their labels from the oracle. The student attempts to use the labeled data to pass the quizzes provided by the teacher. On the other hand, the teacher provides a fair quiz to evaluate the student's performance. Meanwhile, the teacher also sends back the keys of some quiz questions, which are referred to as active feedback, to help the student learn better.
Given a dynamic AL process, multiple active quizzes will be created at different learning phases. ATL forms an integrated risk estimator by aggregating all the quizzes. Our theoretical analysis guarantees that the integrated risk estimation converges to the true risk asymptotically. The aggregation strategy also minimizes the variance, which ensures faster convergence. Furthermore, through active feedback, the AL model can benefit from training with a small proportion of the testing samples (as feedback) after each quiz while maintaining the asymptotic convergence condition of the risk estimation. Finally, the integrated risk estimator provides an unbiased estimate of the model performance with a small variance, which provides a convenient way to determine whether the AL model has converged.
This can further save the labeling budget by stopping the learning early. We may also combine the estimated risk with other available information from the unlabeled data to create more systematic early-stopping criteria. Our main contributions are summarized as follows:
- the first ATL framework that builds an active learning-testing-feedback loop to mimic human learning for achieving label-efficient and dynamic evaluation of actively learned models,
- a theoretically sound integrated risk estimator that combines active quiz results and minimizes the variance of the difference between the estimated and true risks,
- an innovative active feedback process to further improve learning effectiveness without extra labels,
- a systematic early stopping criterion that combines integrated risk estimation with information of unlabeled data to terminate AL process early.
We conduct experiments on both synthetic and real-world datasets to show the improved testing performance of the proposed ATL framework.
Active model evaluation. Actively testing the model performance in a label efficient way is relatively under-explored. Among the few excising efforts, model testing is treated independently from model training, which is unrealistic in most practical settings. The problem of active risk estimation (ARE) was first formulated in [20], which used importance sampling to construct an unbiased risk estimator. Then, the optimal test sampling distribution is derived from an optimization problem that minimizes the variance of the risk error. A similar process has been used to estimate more complex loss measures [21] or perform active model selection [23]. However, such risk estimator is no longer unbiased nor label-efficient when an AL process is involved that samples from the same unlabeled pool for model training. Farquhar et al. propose two risk estimators to cancel the statistical bias by active data sampling strategy in a data-efficient manner [10]. However, their proposed estimators could not deal with the overfitting bias making them less suitable for model evaluation.
In a more recent work, Kossen et al. propose to use the unbiased risk estimators to perform active testing [14]. The difference from [20] is that the selection order of the testing points is considered, and the weights are defined for selection indices instead of a single testing proposal distribution. In this way, the testing proposal can be evaluated at each round, making the setting more realistic. However, the optimal proposal is still impossible to obtain. To approximate the optimal proposal, a surrogate model is needed, which is similar to the introspective risk model in [20]. The surrogate model idea is further extended in [15], where the unbiased risk estimator is replaced by the predicted loss from the surrogate model. Yilmaz et al. propose to replace the commonly used importance sampling with Poison sampling, which better stabilizes the test samples [29]. A different line of work leverages stratified sampling to first stratify the unlabeled pool and then conduct efficient sampling over strata in an importance sampling manner [13, 3, 30]. However, all those work is limited to offline test sampling in which the test proposal is fixed during the sampling process and could not benefit from prior knowledge such as the previously labeled data.
Early stopping in AL. Early stopping strategies in AL have been sparsely investigated over years [2, 7, 6, 31, 26, 18]. Most methods only consider the learning process and do not consider a fully connected learning-testing loop. Some have shown that the stopping criteria based on unlabeled information are more effective than those based on labeled information. This inspires us to combine the proposed integrated risk estimation with unlabeled data information to form a systemic early stopping criterion to further reduce the labeling budget.
3 The ATL Framework
Problem setting. We formulate the ATL framework under a novel and fully integrated learning and testing-while-it-learns setting. W.l.o.g., we assume that all data are generated from an underlying distribution, and we could obtain the label  $y$  through the true labeling function:  $y = h(\mathbf{x})$  provided by the human annotator (note that unlike the predictive model  $f_{\theta}(\cdot)$ , the true labeling function is not governed by the parameter  $\theta$  and it might be non-realizable). Similar to a traditional pool-based AL setting, we have a small labeled set  $S_L = \{(\mathbf{x},y)\}^{N_L}$  (whose labels are revealed). As for the unlabeled pool, we only have access to the features:  $S_U = \{\mathbf{x}\}^{N_U}$ . We further assume that the unlabeled pool is sufficiently large so that active sampling will not change the pool distribution since only limited labels will be revealed. That is,  $p(\mathbf{x},y)$  remains fixed during AL. Our primary task couples two objectives: AL and model evaluation at the same time. In this setting, the goal of AL is to
create a labeled set by selecting samples to label and achieve the best performance. The performance of the model is indicated by the expected error when generalized to the underlying data distribution, which is often noted as the risk  $R$ . As discussed earlier, existing works on label-efficient model evaluation assume that the model being evaluated is already well-trained and kept fixed. They are inadequate to handle a dynamic active learning scenario, which is the focus of ATL. Given some loss function as the risk measurement  $\mathcal{L}_{\theta}: f_{\theta}(\mathbf{x}) \times y \to \mathbb{R}$ , our task of testing the model reduces to evaluating the following expectation:
$$
R = \underset {\{\mathbf {x}, y \} \sim \mathcal {D}} {\mathbb {E}} [ \mathcal {L} _ {\theta} ] = \int \mathcal {L} \left(f _ {\theta} (\mathbf {x}), y\right) p (\mathbf {x}, y) \mathrm {d} (\mathbf {x}, y) \tag {1}
$$
However, directly evaluating (1) is infeasible as the true density function  $p(\mathbf{x},y)$  is unknown. So empirically, we have to adopt an approximation. In the pool-based learning setting, our approximation resorts to the expected empirical risk over the pool distribution  $\mathcal{D}_{pool}$ .
3.1 Overview: Active Risk Estimation During AL
In each AL iteration, we select to label  $\mathbf{x}^*$  based on a sampling criterion  $p_{\text{sample}}(\mathbf{x})$  to improve the model. We design the framework to be AL-agnostic, meaning that most standard AL sampling strategies can be applied here without further assumptions. Normally, the evaluation of the AL model is performed on a hold-out test set either at the end or during different to-learn data selection rounds of AL. However, the labeling cost of the test set is often times ignored. To achieve a more efficient evaluation framework, we propose to have the active risk estimation (i.e., active testing) process intertwined with AL. Specifically, we use time stamps  $t$  to denote each stage where we perform active testing. At time  $t$ , we use testing proposal  $q_t(\mathbf{x})$  to select samples,  $\mathcal{Q}_t = \{\mathbf{x}_t^{(1)}, \dots, \mathbf{x}_t^{(n_t)}\}$ , then label them to evaluate the model. We name this novel evaluation style active quiz. Active quiz is motivated by the dynamic nature of the AL process, which allows us to have an instant evaluation of the current model during AL. We also show that the quizzes can further be conveniently combined into a final exam for a comprehensive evaluation. In this way, we essentially divide a given budget for evaluation into small batches, which makes it possible to stop the evaluation before reaching the total budget by leveraging the signal from past quizzes.
The purpose of having the active quiz is to achieve a good estimate of the current model performance, which is not realizable in most existing AL frameworks. We use the risk  $R$  to denote the performance.  $R$  is dependent on the specific model, thus at each step we are trying to estimate the true risk of the model at  $t$ :  $R(f_{t}(\cdot))^{2}$ . Using  $\mathcal{Q}_t$ , we can have an estimate  $\widehat{R}_{\mathcal{Q}_t}(f_t)$ . We name the evaluation after the last quiz  $\mathcal{Q}_T$  as the final exam because it includes all the quiz samples:  $\{\mathcal{Q}_1,\dots,\mathcal{Q}_T\}$ . We use  $\widetilde{R}_{\{\mathcal{Q}_1,\dots,\mathcal{Q}_T\}}(f_T(\cdot))$  to denote the final performance of active learning model. Note that  $\widetilde{R}$  is different from each  $\widehat{R}_t$  because: (i) the evaluation sets contain all the previous quiz data, and (ii) it only evaluates the final model,  $f_{T}$ . Intuitively, using  $\{\mathcal{Q}_1,\dots,\mathcal{Q}_T\}$  should be more preferable than using a single  $\mathcal{Q}_t$  because more data samples are available to better represent the data distribution. However, the combination is not straightforward because each quiz is sampled using the optimal selection proposal at that time. We propose a principled way to combine multiple quizzes to achieve an unbiased  $\widetilde{R}$  with theoretical guarantees. We further formulate an active feedback strategy to integrate the dual objectives of learning and testing and provide a theoretical underpinning to utilize some labeled test samples to improve the model's generalization capability.
3.2 Unbiased Low-variance Estimation of Model Risk
For a passive learning setting, where the model is fixed, the risk can be directly estimated through an importance weighting sampling procedure, such as the ARE scheme developed in [20]. Let  $q(\mathbf{x})$  denote the test sample selection proposal, then the risk estimate can be defined as
$$
\widehat {R} _ {n, q} = \frac {1}{\sum_ {i = 1} ^ {n} \frac {p \left(\mathbf {x} ^ {(i)}\right)}{q \left(\mathbf {x} ^ {(i)}\right)}} \sum_ {i = 1} ^ {n} \frac {p \left(\mathbf {x} ^ {(i)}\right)}{q \left(\mathbf {x} ^ {(i)}\right)} \mathcal {L} \left(f _ {\theta} \left(\mathbf {x} ^ {(i)}\right), y ^ {(i)}\right) \tag {2}
$$
We can use the central limit theorem to show that  $\widehat{R}_{n,q}$  is an unbiased estimation of  $R$ . Furthermore, their difference follows a zero-mean Gaussian asymptotically:
$$
\sqrt {n} \left(\widehat {R} _ {n, q} - R\right) \xrightarrow {n \rightarrow \infty} \mathcal {N} \left(0, \sigma_ {q} ^ {2}\right), \sigma_ {q} ^ {2} = \int \frac {p (\mathbf {x})}{q (\mathbf {x})} \left(\int [ \mathcal {L} (f _ {\theta} (\mathbf {x}), y) - R ] ^ {2} p (y | \mathbf {x}) d y\right) p (\mathbf {x}) d \mathbf {x} \tag {3}
$$
2Here we use  $f_{t}(\cdot)$  to represent  $f_{\theta_t}(\cdot)$  to keep the notation uncluttered.
When  $n$  is large, we also know that  $n\mathrm{Var}[\widehat{R}_{n,q}] \xrightarrow{n \to \infty} \sigma_q^2$ . Then, the optimal  $q(\mathbf{x})$  minimizes the variance of the estimate. Using variational analysis, the optimal  $q(\mathbf{x})$  is
$$
q ^ {*} (\mathbf {x}) \propto p (\mathbf {x}) \sqrt {\int [ \mathcal {L} (f _ {\theta} (\mathbf {x}) , y) - R ] ^ {2} p (y | \mathbf {x}) \mathrm {d} y} \tag {4}
$$
which minimizes the expected squared difference between the estimate and the true risk (also the variance of the asymptotic Gaussian). Since the pool distribution remains fixed when only a small number of instances are sampled for AL, we can keep  $p(\mathbf{x})$  as a uniform distribution over the pool. Since the importance weighting proposals are known up to a normalization factor, we have  $p(\mathbf{x}) = \frac{1}{N_U}$  where  $N_U$  is the pool size. To optimize model evaluation in an active learning setting, we can evaluate the model at each time stamp  $t$  in the following manner: First, we obtain an active testing proposal, denoted as  $q_t$ , by substituting (4) with  $f_{\theta_t}$ . Next, we obtain the active quizzes denoted as  $Q_t \sim q_t$ . We then evaluate the model's performance at time  $t$  using (2).
However, there are two main limitations to this approach. First, the test proposal given by (4) contains an unknown quantity, the true risk  $R$ . Previous studies have attempted to address this issue by predicting the true risk from the pool. However, in section 3.3, we demonstrate that commonly used true risk predictors are not accurate, particularly in the early stages of active learning. To address this, we propose a multi-source true risk predictor. Second, the current test setting does not allow for sharing of data between previous and current quizzes due to the use of different active proposal distributions. This inefficiency not only wastes data, but also reduces the stability of the model evaluation, as the test samples in each quiz may be too small to provide a robust evaluation. Therefore, in section 3.4, we propose an estimator for the active learning setting that can integrate quizzes sampled from different proposal distributions while remaining optimal.
3.3 Intermediate Estimate of the True Risk
In the optimal test sample selection proposal given by (4), a critical component is the true risk, which is unknown and needs to be estimated. A straightforward solution is to leverage the current model [20] or some proxy model [14] to predict the potential risk over the entire pool as  $R_{\theta}$ . However, the following analysis shows that this is equivalent to the model uncertainty in the classification setting.
Proposition 1. Under the classification setting and when a standard (i)  $0 - 1$  loss or (ii) cross-entropy loss is used,  $R_{\theta}$  is equivalent to measuring the model uncertainty.
More details of the proposition are provided in Appendix B.2. The above proposition shows that  $R_{\theta}$  can only capture the model uncertainty instead of the true risk. When the labeled training samples are limited, it is possible that the model uncertainty can not reflect the level of true risk. We propose to combine the current training risk, the model uncertainty, and the current test risk (using  $\mathcal{Q}_t$  that we have) to get an aggregated multi-source estimate of the true risk:
$$
R _ {\theta , t} ^ {\text {m u l t i}} = \frac {\left| \mathcal {S} _ {L} \right| R _ {\text {t r a i n}} + \left| \mathcal {S} _ {U} \right| R _ {\theta} + n \tilde {R} _ {t}}{\left| \mathcal {S} _ {L} \right| + \left| \mathcal {S} _ {U} \right| + n} \tag {5}
$$
The multi-source estimate can more effectively avoid overestimating the risk when only using the testing information and underestimating the risk when only using training information. By aggregating these two sides of information, neither underfitting or overfitting of the model will harm the estimation to a significant extent. The effectiveness of the multi-source aggregated risk estimation has been demonstrated in our empirical evaluation.
3.4 Active Quiz Integration for Final Risk Estimation
We have shown the optimal way to select test samples to evaluate a fixed model in the passive learning setting. However, we still need to extend to our scenario where there are multiple stages and AL while AT are intertwined. We propose to combine the quiz results by assigning a set of weights  $\mathbf{v}_t$  on them. We will show that the proposed  $\mathbf{v}_t$  is optimal given the fixed quizzes. Let's denote a sequence of independent risk estimates (treated as random variables) as  $\widehat{\mathbf{R}} = (\widehat{R}_{\mathcal{Q}_1}(f_T), \dots, \widehat{R}_{\mathcal{Q}_T}(f_T)^\top)$ , where each  $\widehat{R}_{\mathcal{Q}_t}(f_T)$  asymptotically converges to the true risk  $R(f_T)$ . We should also note that we are using each  $\widehat{R}_{\mathcal{Q}_t}$  to evaluate the same model  $f_T$ , not using the actual values of the estimate at time  $t$  (which is the evaluation of  $f_t$ ). That is, we always utilize previously collected sets of quiz samples to
evaluate the current model.
$$
\widehat {R} _ {\mathcal {Q} _ {t}} (f _ {T}) = \frac {1}{\sum_ {i = 1} ^ {n _ {t}} w _ {i}} \sum_ {i = 1} ^ {n _ {t}} w _ {i} l _ {i} = \frac {1}{\sum_ {i = 1} ^ {n _ {t}} \frac {p (\mathbf {x} ^ {(i)})}{q _ {t} (\mathbf {x} ^ {(i)})}} \sum_ {i = 1} ^ {n _ {t}} \frac {p (\mathbf {x} ^ {(i)})}{q _ {t} (\mathbf {x} ^ {(i)})} \mathcal {L} (f _ {T} (\mathbf {x} ^ {(i)}), y ^ {(i)})
$$
$$
\text {w h e r e} \sqrt {n _ {t}} \left(\widehat {R} _ {\mathcal {Q} _ {t}} \left(f _ {T}\right) - R \left(f _ {T}\right)\right) \xrightarrow {n _ {t} \rightarrow \infty} \mathcal {N} \left(0, \sigma_ {t} ^ {2} \left(f _ {T}\right)\right) \tag {6}
$$
The variance  $\sigma_t^2 (f_T)$  is given by
$$
\int \frac {p (\mathbf {x})}{q _ {t} (\mathbf {x})} \left(\int [ \mathcal {L} (f _ {T} (\mathbf {x}), y) - R ] ^ {2} p (y | \mathbf {x}) \mathrm {d} y\right) p (\mathbf {x}) \mathrm {d} \mathbf {x} \tag {7}
$$