SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Carsten T. Lüth<sup>1,2</sup> Till J. Bungert<sup>1,2</sup> Lukas Klein<sup>1,2,3</sup> Paul F. Jaeger<sup>1,2</sup>
$^{1}$ German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Germany  $^{2}$ Helmholtz Imaging, German Cancer Research Center (DKFZ) Heidelberg, Heidelberg, Germany  $^{3}$ Institute for Machine Learning, ETH Zürich, Switzerland
carsten.lueth@dkfz-heidelberg.de
Active Learning (AL) aims to reduce the labeling burden by interactively selecting the most informative samples from a pool of unlabeled data. While there has been extensive research on improving AL query methods in recent years, some studies have questioned the effectiveness of AL compared to emerging paradigms such as semi-supervised (Semi-SL) and self-supervised learning (Self-SL), or a simple optimization of classifier configurations. Thus, today's AL literature presents an inconsistent and contradictory landscape, leaving practitioners uncertain about whether and how to use AL in their tasks. In this work, we make the case that this inconsistency arises from a lack of systematic and realistic evaluation of AL methods. Specifically, we identify five key pitfalls in the current literature that reflect the delicate considerations required for AL evaluation. Further, we present an evaluation framework that overcomes these pitfalls and thus enables meaningful statements about the performance of AL methods. To demonstrate the relevance of our protocol, we present a large-scale empirical study and benchmark for image classification spanning various data sets, query methods, AL settings, and training paradigms. Our findings clarify the inconsistent picture in the literature and enable us to give hands-on recommendations for practitioners. The benchmark is hosted at https://github.com/IML-DKFZ/realistic-al.
Active Learning (AL) is a popular approach to efficiently label large pools of data by interactively querying the most informative samples for training a classification system. While some parts of the AL community actively propose new query methods (QMs) to advance the field [14, 30], others report AL to be generally outperformed by alternative training paradigms to standard training (ST) such as Semi-Supervised Learning (Semi-SL) [20, 43] and Self-Supervised Learning (Self-SL) [6], or even by well-configured standard baselines [44]. To add to the confusion, further studies have shown that AL can even decrease classification performance in certain settings, a phenomenon referred to as the "cold start problem" [6, 20, 43].
This heterogeneous state of research in AL poses a significant challenge for anyone seeking to efficiently annotate their dataset and facing the questions: On which tasks and datasets is AL beneficial? And how to best employ AL on my dataset?
(b)
Figure 1: (a): The five pitfalls (P1-P5) for meaningful evaluation in the context of the Active Learning loop. Detailed information is provided in Sec. 2. (b): The five pitfalls are highly prevalent in the current literature (green ticks denote successful avoidance of the respective pitfall). A detailed correspondence between individual studies and pitfalls is provided in Appendix B. Our study is the first to avoid all pitfalls and enable trustworthy performance assessment of AL methods.
We argue that the inconsistency arises from a lack of systematic and realistic evaluation of AL methods. AL inherently comes with specific requirements on how methods will be applied in practice. First and foremost, the stated purpose of "reducing the labeling effort on a task" implies that AL methods need to be rolled out to unseen datasets different from the labeled development data. This inherent requirement of cross-task generalization needs to be reflected in method evaluation, posing a need to test methods under diverse settings to identify robust and trustworthy configurations for the subsequent "blind" application on real-life tasks (see "validation paradox", Sec. 2.2). However, such considerations are generally neglected in AL research, as identified in our work by means of five key pitfalls in the current literature, spanning from a lack of tested AL settings and tasks to a lack of appropriate baselines (see Figure 1 and P1-P5 in Sec. 2.3).
To this end, we present an evaluation framework for deep active classification that overcomes the five pitfalls and demonstrate the relevance of this contribution by means of a large-scale empirical study spanning various datasets, QMs, AL settings, and training paradigms. To give one concrete example for the disruptive impact of our work: We address the widespread pitfall of neglecting classifier configuration (see P4 in Figure 1b) by introducing a light-weight protocol for high-quality configuration. The results demonstrate how this simple protocol lets even our random-query baseline exceed the originally reported performance of recent AL methods [1, 30, 35] on the widely-used CIFAR-10/100 datasets, while drastically reducing computational effort compared to the configuration protocol of Munjal et al. [44].
This example showcases, that the novelty of our work does not lie in presenting entirely novel results and insights, but in the fact that our comprehensive and systematic approach is the first to address all five key pitfalls and thus to provide trustworthy insights into the real-life capabilities of current AL methods. By relating these insights to recent studies that have been subject to flawed evaluation, we are able to resolve existing inconsistencies in the field and provide robust guidelines for when and how to apply AL on a given task.
2 Realistic Evaluation in Active Learning
2.1 Active Learning Task Formulation
As depicted in Figure 1a, AL describes a classification task, where a dataset  $\mathcal{D}$  is given that is divided into a labeled set  $\mathcal{L}$  and an unlabeled pool  $\mathcal{U}$ . Initially, only a fraction of the data is labelled ("starting budget"), which is ideally split further into a training set and a validation set for hyperparameter (HP) tuning and performance monitoring. After initial training, the QM is used to generate queries  $\mathcal{Q}_{\mathcal{U}}$  of a certain amount ("query size") that represent the most informative samples from  $\mathcal{U}$  based on the current classifier predictions. Subsequently, queried samples are labeled  $(\mathcal{Q}_{\mathcal{L}})$ , moved from  $\mathcal{U}$  to  $\mathcal{L}$ , and the classifier is re-trained on  $\mathcal{L}$ . This process is repeated until classifier performance is satisfying, as monitored on the validation set.
2.2 Overview over critical concepts in Active Learning evaluation
Evaluating an AL algorithm typically means testing how much classification performance is gained by data samples queried over several training iterations. The QM selecting those samples is considered useful if the performance gains exceed the gains of randomly queried samples. While this process is well-established, it is prone to neglecting critical concepts for evaluation and thus to over-simplification of how AL algorithms are applied in practice.
AL validation paradox. When applying AL on a new, mostly unlabelled dataset, one can not directly validate which QM type or configuration performs best, as this would require labeling individual AL trajectories through the dataset for each setting. This excessive labeling would directly contradict the goal of AL to reduce the labeling effort, a predicament which we refer to as the AL validation paradox. Notably, this is in contrast to standard ML, where fixed labeled training and validation splits generally suffices for model selection and configuration.
Special requirements on evaluation. As the described paradox impedes on-the-spot validation, it forces one to, instead, estimate how well certain QMs will perform on the given task based solely on prior knowledge. The quality of this prior knowledge depends on how extensively the respective QM has been validated prior to application, i.e. on the development data. This implies several critical requirements on evaluation:
- A generalization gap between development and application arises, because the latter typically comes with practical constraints such as on the size of the starting budget, the query size, or a given class imbalance. To avoid AL failure, such constraints need to be anticipated by evaluating QMs on a realistic range of the corresponding design choices.
- Given these inherent generalization requirements in AL, it is crucial to simulate the generalization process by evaluating QMs on roll-out datasets, i.e. tasks that were not part of the configuration process on the development datasets.
- Evaluation needs to acknowledge that AL is an elaborate and costly training paradigm, and thus ensure that methods are tested against simpler and cheaper alternatives. For instance, if the gain of a QM over random sampling vanishes by simply tuning the learning rate of the classifier, or by a single self-supervised pretraining, the QM, and AL in general, provides no practical value.
To effectively describe the current oversight of these requirements in the field, we identified five concrete evaluation pitfalls (P1-P5), which are detailed in Section 2.3.
Cold start problem. Neglecting these pitfalls could render AL ineffective, or even counterproductive, for specific tasks, as it might result in QMs underperforming relative to a random-sampling baseline. Such failures occur predominantly in few-label settings and are a recognized challenge in the research community, often termed as the cold start problem [20].
2.3 Current pitfalls of Active Learning evaluation
The current AL literature features an inconsistent landscape of evaluation protocols, but, as Figure 1b shows, none of them adhere to all requirements for evaluation described above. To study the current oversight of these requirements, we identify five key pitfalls (P1-P5) that need to be overcome for meaningful evaluation of QMs. For a visual overview of the pitfalls and how they integrate into the AL setting see Figure 1a.
P1: Lack of evaluated data distribution settings. To ensure that QMs work out-of-the-box in real-life settings, they need to be evaluated on a broad data distribution. Relevant aspects of a distribution in the AL-context go beyond the data domain and include class distribution, the relative difficulty of separation across classes, as well as a potential mismatch between the frequency and importance of classes. All of these aspects directly affect the functionality of a QM and may lead to real-life failure of AL when not considered in the evaluation.
Current practice: Most current work is limited to evaluating QMs on balanced datasets from one specific domain (e.g. CIFAR-10/100) and under the assumption of equal class importance. To our knowledge, testing the generalizability of a fixed QM setting to new datasets ("roll-out") has not been performed before. There are some experiments conducted on an artificially imbalanced dataset
(CIFAR-LT) [35, 44] suggesting good AL performance. Further, Gal et al. [19] study AL on the ISIC-2016 dataset, but obtain volatile results due to the small dataset size. Atighehchian et al. [3] study AL on the MIO-TCD dataset and reported performance improvements for underrepresented classes. In the large study Beck et al. [4] perform experiments on five mostly balanced datasets datasets and [58] use 13 datasets for mostly standard AL experiments.
$\rightarrow$  Proposed solution: We argue that the underrepresentation of class-imbalanced datasets in the field is one reason for current doubts regarding the general functionality of AL. Real-life settings will most likely not be class balanced providing a natural advantage of AL over random sampling. We propose to consider diverse datasets with real class imbalances as an essential part of AL evaluation and advocate for the inclusion of "roll-out" datasets, as a real-life test of selected and fixed AL settings.
P2: Lack of evaluated starting budgets. There are two reasons for why this parameter is an essential aspect of AL evaluation: 1) Upon application, the budget might be fixed and the QM is required to generalize out-of-the-box to this setting. 2) We are interested in the minimal budget at which the QM works since a too large budget implies inefficient labeling (equivalent to random queries) and a too small budget is likely to cause AL failure (cold start problem). This search needs to be performed prior to an AL application due to the validation paradox.
Current practice: Most recent studies evaluate AL on a single starting budget made of thousands of samples on datasets such as CIFAR-10 and CIFAR-100[30, 44, 54, 55]. Information-theoretic publications commonly use a smaller starting budget [19, 32], but typically on even simpler datasets such as MNIST [38]. Beck et al. [4] compare two starting budgets on MNIST reporting no performance drop. On the other hand, some studies benchmarking AL against Semi-SL and Self-SL compare two [6] or three [20, 43] starting budgets often with the conclusion that smaller starting budgets lead to AL failure. Bengar et al. [6] report that there exists a relationship between the number of classes in a task and the optimal starting budget (the intuition being that class number is a proxy for task complexity).
$\rightarrow$  Proposed solution: To overcome this pitfall and resolve the current contradictions, we evaluate all QMs for three different starting budgets on all datasets. We refer to these settings as the low-, mid-, and high-label regime. Extending on the findings of Bengar et al. [6], adequate budget sizes are determined using heuristics based on the number of classes per task.
P3: Lack of evaluated query sizes. The number of samples queried for labelling in each AL iteration is an essential aspect of QM evaluation. This is because, upon application, this parameter might be predefined by the compute-versus-label cost ratio of the respective task (a smaller query size amounts to higher computational efforts but might enable more informed queries and thus less labeling). Since query size cannot be validated on the task at hand due to the validation paradox, the generalizability of QMs to various settings of this parameter needs to be evaluated beforehand.
Current practice: In current literature, there is a concerning disconnect between theoretical and practical papers regarding what constitutes a reasonable query size. Information-theoretical papers typically select the smallest query size possible and QMs such as BatchBALD [32] are specifically designed to simulate reduced query sizes [19, 46]. In contrast, practically-oriented papers usually employ larger query sizes [30, 43, 44, 51, 55], but only in combination with large starting budgets (P2), where cold start problems generally do not occur. Only a few studies perform limited evaluations of varying query sizes. Beck et al. [4] and Zhan et al. [58] report a negligible effect of varying query sizes, but only evaluate in combination with large starting budgets (1000 samples). In line with this, Munjal et al. [44] conclude that the choice of query size does not matter, but only compared two large values (2500 versus 5000 samples) on a fixed large starting budget (5000 samples). Atighehchian et al. [3] come to a similar conclusion, but also only considered a relatively large starting budget (500) for ImageNet-pretrained models on CIFAR-10, where, again, no cold start problem occurs. Bengar et al. [6] employ varying query sizes without further analysis of the parameter.
$\rightarrow$  Proposed solution: To overcome this pitfall and reliably study the effect of query sizes also in low-label, i.e. high-risk, settings, we evaluate all QMs for three different query sizes in combination with varying starting budgets on all datasets (i.e. as part of the low-, mid-, and high-label regimes). For a specific focus on the effect of query size in the low-label settings, we perform an additional
ablation with varying query sizes on a small fixed starting budget.
P4: Neglection of the classifier configuration. As stated in Sec. 2.2, when aiming to draw conclusions about the performance or usefulness of a QM, it is critical that this evaluation be based on well-configured classifiers. Otherwise, performance gains might be attributed to AL that could have been achieved by simple hyperparameter (HP) modifications. Separating a validation split from the training data is a crucial requirement for sound HP tuning.
Current practice: Most studies in AL literature do not report how HPs are obtained and do not mention the use of validation splits [30, 35, 43, 51, 55]. Typically, reported settings are copied from fully labeled data scenarios. In some cases, even the proposed QMs feature delicate HPs without reporting how they were optimized raising the question of whether these settings generalize to new data [30, 51, 55]. Munjal et al. [44] demonstrate how adequate HP tuning on a validation set allows a random query baseline to outperform current QMs under their originally proposed HP settings. However, they run a full grid search for every QM and AL training iteration, which might not be feasible in practice.
$\rightarrow$  Proposed solution: To overcome this pitfall and enable meaningful performance assessment of AL methods, we define a validation dataset of a size deducted heuristically from the starting budget. Based on this data, a small selection of HPs (learning rate, weight decay and data augmentations [17]) is tuned only once per AL experiment while training on the starting budget. The limited search space and discarding of multiple tuning iterations result in a lightweight and practically feasible protocol for classifier configuration.
P5: Neglection of alternative training paradigms. Analogously to arguments made in P4, meaningful evaluation of AL requires comparison against alternative approaches that address the same problem. Specifically, the training paradigms Self-SL [13, 25] and Semi-SL [39, 52] have shown strong potential to make efficient use of an unlabeled data pool in a classification task thus alleviating the labeling burden. Additionally to benchmarking AL against Self-SL and Semi-SL, the question arises of whether AL can yield performance gains when combined with these paradigms.
Current practice: While most AL studies do not consider Self-SL and Semi-SL, there are a few recent exceptions: Bengar et al. [6] benchmark AL in combination with Self-SL and conclude that AL only yields gains under sufficiently high starting budgets. However, these results suffer from inadequate classifier configuration (P4). Yi et al. [54] propose a QM in combination with Self-SL, but the employed Self-SL strategy is limited by compatibility with the proposed QM. Further, Gao et al. [20] combine Semi-SL with AL and report superior performance compared to ST for CIFAR-10/100 and ImageNet [48], i.e. the datasets on which Semi-SL methods have been developed. Similarly, Mittal et al. [43] evaluate the combination of Semi-SL and AL on CIFAR-10/100, reporting strong improvements compared to ST and find that AL decreases performance for small starting budgets.
$\rightarrow$  Proposed solution: To overcome this pitfall and resolve current inconsistencies, we benchmark all QMs against both Self-SL and Semi-SL, and evaluate the combination of AL with these paradigms. Crucially, we are the first to study these relations as part of a reliable evaluation, i.e. while avoiding all other key pitfalls (P1-P4).
This section describes the design of our empirical study in light of the proposed improvements for AL evaluation (detailed experimental settings can be found in Appendix D). We first address P1 by extending our evaluation to 5 different datasets, containing different label distributions. Specifically, these datasets include CIFAR-10, CIFAR-100, CIFAR-10 LT, ISIC-2019 and MIO-TCD, where the first three are developmental datasets and the latter two are used exclusively for the proposed roll-out evaluation. Further, we address P2 and P3 by defining three different label regimes which we refer to as "low-label", "mid-label" and "high-label" regimes. Starting budgets and query sizes are both set to  $5 \times C$ ,  $25 \times C$  and  $100 \times C$  for the three label regimes, where  $C$  denotes the number of classes. To address P4, we configure our classifiers for all three label regimes based on a validation set five
These deviate for CIFAR-100  $[5 \times C$  (low-),  $10 \times C$  (mid-),  $50 \times C$  (high-label)] due to a smaller ratio of dataset size to number of classes.
times the size of the starting budget. Further, addressing P4 and P5 we use a ResNet-18 [24] as the backbone in all experiments and optimize the essential HPs for each respective training paradigm. At last, we address P5 by comparing randomly initialized models (standard training) against Self-SL pre-trained models and Semi-SL models.
Compared query methods. In this section, we describe the applied QM in more general terms and refer to Appendix A for details. We focus exclusively on QMs which do not alter the classifier and require no configuration of additional HPs except for bayesian QMs which are based on dropout. Generally, QMs can be divided into two categories as they are either based on uncertainty estimation or enforcing exploration. Random: The baseline all QMs are compared against which randomly draws samples from the pool  $\mathcal{U}$ . Core-Set: This explorative QM aims to find the core-set of a convolutional neural network [49] by means of a K-Center Greedy approximation on intermediate representations. Entropy: This uncertainty-based QM selects the samples with the highest entropy across predicted class scores [50]. BALD: This uncertainty-based QM uses the mutual information between the class label and the model parameters with regard to each sample for greedy selection [27], it was introduced with dropout for deep bayesian active learning[19]. BADGE: This QM performs a clustering based on per-sample gradient vectors obtained via proxy labels. This enables a selection that is both diverse and guided by uncertainty [2].
Datasets. The initial datasets for our experiments are CIFAR-10/100 [36]. For further analysis we added CIFAR-10 LT [10], an artificially created dataset built upon CIFAR-10 with a long-tail distribution of classes following an exponential decay for the training split. The imbalance factor  $\rho$  was selected to be 50, following [35]. On all three of these datasets, we use accuracy on the class-balanced test set as the primary performance metric. Finally, we selected ISIC-2019[15, 16, 53] and MIO-TCD[42] as roll-out datasets to verify the generalizability of AL methods to new settings. Both of these datasets feature inherently imbalanced class distributions, and are more likely to be subject to label noise compared to the CIFAR datasets. As such, we deem the two roll-out datasets an essential step toward a realistic evaluation of AL methods. For both MIO-TCD and ISIC-2019, we use balanced accuracy as the primary performance measure (Appendix D).
Active learning setup. We report performance measures for each dataset on identical test splits based on three experiments using different seeded models and different train and validation splits to reduce the possible influence of these parameters on our results. Further, we train the models from scratch on every training step to avoid correlated queries [32].