SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training paradigms. Randomly initialized and supervised-trained models are referred to as ST models. Further, we use the popular contrastive SimCLR [13] training as a basis for Self-SL pretraining. These models are fine-tuned and are referred to as Self-SL models. Self-SL models have a two-layer MLP as a classification head to make better use of the representations (ablation in Appendix E). For ST and Self-SL models, we obtain bayesian models by adding dropout to the final representations before the classification head following [19]. As a Semi-SL method, we use FixMatch [52] which combines the principles of consistency and uncertainty reduction. Due to the long training times (factor 80) compared to ST training, we only ran experiments in the low- and mid-label regime while increasing the query size by a factor of three to reduce training costs.
Hyperparameter selection. The HPs for our models are selected for each label regime before the AL loop is started using the corresponding validation set. For Self-SL and ST models we use a fixed training recipe (Scheduler, Optimizer, Batch Size, Epochs) and only optimize learning rate, weight decay and data augmentations. The data augmentations used are standard augmentations and Randaugment which uses stronger augmentations acting as a regularization [17]. For Semi-SL methods we fix all HPs with the exception of learning rate and weight decay following Sohn et al. [52]. Model selection for ST and Self-SL models is based on the best validation set epoch, while for Semi-SL models the final checkpoint is used. For imbalanced datasets, we use oversampling for ST and Self-SL pre-trained models Buda et al. [8] and use weighted cross-entropy-loss and distribution alignment for FixMatch.
Low-Label query size ablation. To investigate the effect of query size in the low-label regime, we conduct an ablation with Self-SL pre-trained models on CIFAR-100 and ISIC-2019. For CIFAR-100 the query sizes are 50, 500 and 2000, while for ISIC-2019 they are 10, 40 and 160.
Figure 2: Results obtained with our proposed evaluation protocol over five different datasets and the three label regimes. These experiments are to our knowledge the largest conducted study for AL and reveal insights along the lines of the five key parameters as discussed in Sec. 2. The strong performance dip on MIO-TCD and ISIC-2019 is discussed in Sec. 4.
4 Results & Discussion
The results of our empirical study are shown in Figure 2. An in-depth analysis of results for individual datasets and analysis based on the pairwise penalty matrix [2] and area under the budget curve [57] can be found in Appendix F. Here, we will discuss the main findings along the lines of our five identified pitfalls of evaluation (P1-P5). Our findings demonstrate the relevance of the proposed protocol for realistic evaluation and its potential to generate trustworthy insights on when and how AL works. If not otherwise mentioned, all references in this chapter refer to AL studies.
P1 Data distribution. The proposed evaluation over a diverse selection of dataset distributions including specific roll-out datasets proved essential for realistic evaluation of QMs as well as the different training strategies. One main insight is the fact that class distribution is a crucial predictor for the potential performance gains of AL on a dataset: Performance gains of AL are generally higher on imbalanced datasets and occur consistently even for ST models with a small starting budget, which are typically prone to experience cold start problems. This observation is consistent with a few previous studies [30, 35, 54]. Further, our results underpin the importance of the roll-out datasets e.g. when looking at the sub-random performance of BALD (with Self-SL) and Entropy (with ST) on MIO-TCD. Such worst-case failures of AL application (increased compute and labeling effort due to AL) could not have been predicted based on development data where all AL-parameters are optimized. Another example is the lack of generalizability of Semi-SL, where performance in relation to other Self-SL and ST decreases gradually with data complexity (going from CIFAR-10 to CIFAR-100/-10 LT to the two roll-out datasets MIO-TCD and ISIC-2019.)
P2 Starting budget. The comprehensive study of various starting budgets on all datasets reveals that AL methods are more robust with regard to small starting budgets than previously reported
Figure 3: Low-label query size ablation on CIFAR-100 and ISIC-2019. On CIFAR-100, reducing the query size resolves the observed failure mode of BALD. However, no improvement is observed on ISIC-2019, presumably because BALD already shows the best performance. Further, BADGE performs consistently well across all query sizes without failure modes, revealing its robustness also for low-label settings.
[6, 20, 43]. With the exception of Entropy we did not observe cold start problems even for any QM even in combination with notoriously prone ST models. The described robustness is presumably enabled by our thorough classifier configuration (P4) and heuristically adapted query sizes (P3). This finding has great impact potential suggesting that AL can be applied at earlier points in the annotation process thereby further reducing the labeling cost, especially in combination with BADGE which performed consistently better or on par with Random. Similarly for Self-SL models, AL performs well on small starting budgets with the exceptions of CIFAR-100 (BALD, Entropy) and MIO-TCD (BALD, Core-Set, Entropy).
P3 Query size. Based on our evaluation of the query size we can empirically confirm its importance with regard to 1) general AL performance and 2) counteracting the cold start problem. The are, however, surprising findings indicating that the exact interaction between query size and performance remains an open research question. For instance, we observe the cold start problem for BALD with Self-SL on CIFAR-100 ( $\sim$ 50% accuracy at 2k labeled samples for query sizes of 500 (low-label regime) and 1k (mid-label regime). On the other hand, in the high label regime (budget of 5k and query size of 5k) ST and Self-SL models with similar accuracies of 50% and 60%, respectively, benefit from BALD. Since cold start problems are commonly associated with large query sizes, this finding seems counter-intuitive, but has been reported before although without further investigations [6, 20, 43]. To gain a better understanding of this phenomenon and how QMs interact with query size in the low-label regime, we performed a dedicated experiment series for Self-SL training on CIFAR-100 and ISIC-2019 (see Figure 3 and Appendix F.2). Due to a significant improvement of BALD on CIFAR-100 for even smaller query sizes from sub-Random to Random performance and no worsening on ISIC-2019, we can conclude that small query sizes represent an effective countermeasure against the cold-start problem for BALD contrary to the findings of [44, 58] and currently not considered as a solution [6, 20, 43]. This potentially explains the gap between these findings and more theoretical works which advertise using the smallest query size possible [19, 33]. Importantly, while smaller query sizes appear as a potential solution for the observed instabilities of BALD, they seem to have no considerable effect on the other QMs. Moreover, our ablation reveals that even in the low-label settings, BADGE is the overall most reliable of all compared QMs and exhibits no sub-Random performance. This adds to the existing reports of the robustness of BADGE for higher label regimes [4, 58].
P4 Classifier configuration. Our results show that method configuration on a properly sized validation set is essential for realistic evaluation in AL. For instance, our method configuration has the same effect on the classifier performance as increasing the number of labeled training samples by a factor of  $\sim 5$  (e.g. our ST model reach approximately the same accuracy of  $\sim 44\%$  trained on 200 samples compared to models in the AL study by Bengar et al. [6] trained on 1k samples). The effectiveness of our proposed lightweight HP selection on the starting budget including only three parameters (Sec. 3) is demonstrated by the fact that all our ST models substantially outperform respective models found in relevant literature [6, 20, 30, 35, 43, 54, 55] where HP optimization is generally neglected. Details for this comparison are provided in Appendix H. This raises the question of to which extent reported AL advantages could have been achieved by simple classifier configurations. Further, our models
also generally outperform expensively configured models by Munjal et al. [44]. Thus, we conclude that manually constraining the search space renders HP optimization feasible in practice without decreasing performance and ensures performance gains by Active Learning are not overstated. The importance of the proposed strategy to optimize HPs on the starting budget for each new dataset is supported by the fact that the resulting configurations change across datasets.
P5 Alternative training paradigms. Based on our study benchmarking AL in the context of both Self-SL and Semi-SL, we see that while Self-SL generally leads to improvements across all experiments, Semi-SL only leads to considerably improved performance on the simpler datasets CIFAR-10/100, on which Semi-SL methods are typically developed. Generally, models trained with either of the two training paradigms receive a lower performance gain from AL (over random querying) compared to ST. Crucially, Self-SL models converge around 2.5 times faster than ST models, while the training time of Semi-SL models is around 80 times longer than ST and often yields only small benefits over Self-SL or ST models. The fact that AL entails multiple training iterations amplifies the computational burden of Semi-SL, rendering their combination prohibitively expensive in most practical scenarios. Further, the fact that our Semi-SL models based on Fixmatch do not seem to generalize to more complex datasets in our setting stands in stark contrast to conclusions drawn by [20, 43] as to which the emergence of Semi-SL renders AL redundant. Interestingly, the exact settings where Semi-SL does not provide benefits in our study are the ones where AL proved advantageous. The described contradiction with the literature underlines the importance of our proposed protocol testing for a method's generalizability to unseen datasets. This is especially critical for Semi-SL, which is known for unstable performance on noisy and class imbalanced datasets as noted by studies focusing on Semi-SL [7, 22, 23, 29, 45, 56].
Limitations. We propose a light-weight and practically feasible strategy for HP optimization and made other design choices (e.g. ResNet-18 classifier), thus we can not guarantee that our configurations are optimal for all compared training paradigms and would like to provide a critical discussion: 1) The ResNet-18 in combination with our shortened training times, might hinder Semi-SL performance more than other training paradigms. This setting is necessary to be able to cope with the computational cost of Semi-SL (factor  $\sim 400$  training time compared with ST). 2) The validation set size of  $5\times$  starting budget size (i.e. training set) could be considered as larger than practically desirable, where most data would be used for training. This design decision follows the study of Oliver et al. [45], showing that an adequately sized validation set is necessary for proper HP selection (especially for Semi-SL). 3) We observe a performance dip of ST models on MIO-TCD and ISIC-2019 at  $\sim 7\mathrm{k}$  samples, which we attribute to our HP selection scheme. This indicates that HPs might need to be re-selected occasionally at certain training iterations. However, such cases are immediately detected in practice allowing for correction where necessary by re-optimizing the HPs (see Appendix I.1). A more extensive discussion of limitations can be found in Appendix I.
5 Conclusion & Take-Aways
Our experiments provide strong empirical evidence that the current evaluation protocols do not sufficiently answer the key question every potential AL practitioner faces: Should I employ AL on my dataset? Answering this question entails estimating whether an AL algorithm will provide performance gains over random queries and thus whether the expected reduction in labeling cost outweighs both the additional computational and engineering cost attributed to AL. We argue that our proposed protocol for realistic evaluation represents a cornerstone towards enabling informed decisions in this context. This is made possible by focusing on evaluating the generalizability of AL to new settings under real-world conditions. This perspective manifests itself in the form of five key pitfalls in the current AL literature regarding data distribution, starting budget, query size, classifier configuration, and alternative training paradigms (see Sec. 2.3). Even though the thorough evaluation increases the computational cost once during method development, it will lead to a net cost reduction in the long-term by enabling an informed selection of AL methods in practice, thereby effectively reducing annotation cost upon application.
We hope that the proposed evaluation protocol in combination with the publicly-available benchmark will help to push active learning towards robust and wide-spread real-world application.
Main empirical insights revealed by our study:
- Assessment of AL methods in the literature is substantially hampered by subpar classifier configurations, but meaningful assessment can be restored by our protocol for lightweight hyperparameter tuning on the starting budget.
- AL generally provides substantial gains in class-imbalanced settings.
- BADGE is the best-performing QM across a realistic range of datasets, starting budgets, and query sizes and exhibits no failure modes (i.e. sub-Random performance).
- Combining AL with Self-SL considerably improves performance, shortens the training time, and stabilizes optimization, especially in low-label settings.
FixMatch results indicate that Semi-SL methods perform well on datasets where they have been developed, but may struggle to generalize to more realistic scenarios such as class imbalanced data. Combining the paradigm with AL additionally suffers from extensive training times.
- BALD with Self-SL pre-trained models benefits from smaller query sizes on small starting budgets possibly circumventing the "cold start problem".
Take-aways for developing and proposing new Active Learning algorithms:
AL methods should be tested for generalizability on roll-out datasets and come with a clear recipe for real-world application including how to adapt all design choices to new settings. Since the expected benefit of AL on a new setting increases with lower application costs, we believe there is a high potential for wide-spread real-world use of AL by reducing the two prevalent cost factors: 1) Engineering costs, which are reduced by building easy-to-use AL tools. 2) Computational costs, which are reduced by explicitly including methods that shorten the training time in AL.
Take-aways for the cost-benefit analysis of deploying Active Learning:
1. Since we identify BADGE as a robust query method exhibiting no sub-random-sampling performance across all tested settings, the potential harm of AL is minimized allowing to base decisions around deploying AL on the described cost-benefit analysis.
2. The expected benefit is high in settings, where there is a mismatch between the task-specific importance of individual classes and their frequency in the dataset (e.g. class-imbalanced datasets in combination with tasks requiring a balanced classifier).
3. The expected benefit further increases with the labeling cost that will be reduced by AL. AL is thus likely to yield a net benefit in settings with high label cost and low computational and engineering costs.
Future research:
Future research may investigate to what extend AL benefits from foundation models such as CLIP [47], which have been shown to reach strong performance in low-label settings through finetuning or knowledge extraction. Additionally, their rapid re-finetuning during AL iterations may enable real-time interactive labelling. A further important aspect might be to study the effect of AL training on the classifiers' ability to detect outliers [18] or catch failures under distributions shifts [28].
This work was funded by Helmholtz Imaging (HI), a platform of the Helmholtz Incubator on Information and Data Science. This work is supported by the Helmholtz Association Initiative and Networking Fund under the Helmholtz AI platform grant (ALEGRA (ZT-I-PF-5-121)). We thank Maximilian Zenk, Sebastian Ziegler and Fabian Isensee for insightful discussions and feedback.
[50] Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.
[51] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[52] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 596-608. Curran Associates, Inc., 2020.
[53] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1): 1-9, 2018.
[54] John Seon Keun Yi, Minseok Seo, Jongchan Park, and Dong-Geol Choi. Using Self-Supervised Pretext Tasks for Active Learning. arXiv:2201.07459 [cs], January 2022.
[55] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[56] Maximilian Zenk, David Zimmerer, Fabian Isensee, Paul F Jäger, Jakob Wasserthal, and Klaus Maier-Hein. Realistic evaluation of fixmatch on imbalanced medical image classification tasks. In Bildverarbeitung für die Medizin 2022, pages 291–296. Springer, 2022.
[57] Xueying Zhan, Huan Liu, Qing Li, and Antoni B Chan. A comparative survey: Benchmarking for pool-based active learning. In *IJCAI*, pages 4679-4686, 2021.
[58] Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. arXiv preprint arXiv:2203.13450, 2022.
A Active learning, in more detail
First, we will give an additional task description of Active Learning (AL) in Appendix A.1 introducing necessary concepts and mathematical notation for our compared query methods which are discussed in Appendix A.2. Finally, we give intuitions where the connection of AL to Self-Supervised Learning (Self-SL) and Semi-Supervised Learning (Semi-SL) lies in Appendix A.3 and Appendix F.4.
A.1 From supervised to active learning
In supervised learning, we are given a labeled dataset of sample-target pairs  $(x,y)\in \mathcal{L}$  sampled from an unknown joint distribution  $p(x,y)$ . Our goal is to produce a prediction function  $p(Y|x,\theta)$  parametrized by  $\theta$ , which outputs a target value distribution for previously unseen samples from  $p(x)$ . Choosing  $\theta$  might amount, for example, to optimizing a loss function which reflects the extent to which  $\operatorname{argmax}_c p(Y = c|x,\theta) = y$  for  $(x,y)\in \mathcal{L}$ . However, in pool-based AL we are additionally given a collection of unlabeled samples  $\mathcal{U}$ , sampled from  $p(x,y)$ . By using a query method (QM), we hope to leverage this data efficiently through querying and successively labeling the most informative samples  $(x_{1},\dots,x_{B})$  from the pool. This should lead to a prediction function better reflecting  $p(Y|x)$  than if random samples were queried.
From a more abstract perspective, the goal of AL is to use the information of the labeled dataset  $\mathcal{L}$  and the prediction function  $p(Y|x,\theta)$ , to find the samples giving the most information where  $p(Y|x)$  deviates from  $p(Y|x,\theta)$ . This is also reflected by the way performance in AL is measured - which is the relative performance gain of the prediction function with queries from a QM compared to random queries. Making the prediction function implicitly the gauge for measuring the "success" of an AL strategy.
At the heart of AL is an optimization problem: AL is a game of reducing cost - one trades in computation cost with the expectation of lowering labeling cost which is deemed to be the bottleneck.
A.2 Query methods
A comprehensive overview of AL and its methods is out of the scope of this paper, we refer interested readers to [50] as a basis and [40] for an overview of current research. Most QMs fall into two categories following either: explorative strategies, which enforce queried samples to explore the distribution of  $p(x)$ ; and uncertainty-based strategies, which make direct use of the prediction function  $p(Y|x,\theta)$ . The principled combination of both strategies, especially to allow larger query sizes for uncertainty-based QM, is an open research question.
In our work, we focus exclusively on QMs which induce no changes to the prediction function and add no additional HPs except for Bayesian QMs modeled with dropout. This immediately rules out QMs like Learning-Loss [55] or TA-VAAL [30], changing the prediction function, and QMs like VAAL [51], introducing new HPs. The QMs we use for our comparisons are currently state-of-the-art in AL on classification tasks. Further, they require no additional hyperparameters (HPs) to be set for the query function which is hard to evaluate in practice due to the validation paradox.
For this chapter we follow the notation introduced in [31], where e.g.  $X$  represents a random variable and  $x$  represents a concrete sample of variable  $X$ .
Random Draws samples from the pool  $\mathcal{U}$  randomly which follows  $p(x,y)$ . Therefore it can be interpreted as an exploratory QM.
Core-Set This method is based on the finding that the decision boundaries of convolutional neural networks are based on a small set of samples. To find these samples, the Core-Set QM queries samples that minimize the maximal shortest path from unlabeled to labeled sample in the representation space of the classifier [49]. This is also known as the K-Center problem, for which we use the K-Center greedy approximation. It draws queries especially from tails of the data distribution,
2This is conceptually similar to the exploration and exploitation paradigm seen in Reinforcement Learning and there actually exist strong parallels between Reinforcement Learning and Active Learning - so much so that Reinforcement Learning has been proposed to use in AL and AL-based strategies have been proposed to be used in Reinforcement Learning.
to cover the whole dataset as well as possible. Therefore we classify it as an explorative strategy.
Entropy The Entropy QM greedily queries the samples  $x$  with the highest uncertainty of the model as shown in Equation (1) with  $C$  being the number of classes.
$$
H (Y | x, \theta) = \sum_ {c = 1} ^ {C} p (Y = c | x, \theta) \cdot \log (p (Y = c | x, \theta)) \tag {1}
$$
BALD Uses a bayesian model and selects greedily a query of samples with the highest mutual information between the predicted labels  $Y$  and weights  $\Theta$  for a sample  $x$  following [19]. From the weight variable  $\Theta$  the concrete values  $\theta \sim p(\theta | \mathcal{L})$  are then obtained by MC sampling of a bayesian dropout model [19].
$$
\begin{array}{l} \operatorname {M I} (Y; \Theta | x, \mathcal {L}) = \sum_ {c = 1} ^ {C} (p (Y = c | x, \mathcal {L}) \cdot \log (p (Y = c | x, \mathcal {L})) \tag {2} \\ - \mathbb {E} _ {p (\theta | \mathcal {L})} [ H (Y | x, \theta) ]) \\ \end{array}
$$
Where  $p(Y|x,\mathcal{L}) = \mathbb{E}_{p(\theta |\mathcal{L})}[p(Y|x,\theta)]$