Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment
Authors: Carsten Tim Lüth
Till J. Bungert
Lukas Klein
Paul F Jaeger
Venue: NeurIPS 2023 poster (2023)
Keywords: Active Learning
Evaluation
Study
TL;DR: We propose an evaluation framework and conduct a large-scale study on Active Learning (AL), including semi-supervised and self-supervised training, to address literature inconsistencies and provide practical recommendations for AL adoption.

Abstract: Active Learning (AL) aims to reduce the labeling burden by interactively selecting the most informative samples from a pool of unlabeled data. While there has been extensive research on improving AL query methods in recent years, some studies have questioned the effectiveness of AL compared to emerging paradigms such as semi-supervised (Semi-SL) and self-supervised learning (Self-SL), or a simple optimization of classifier configurations. Thus, today’s AL literature presents an inconsistent and contradictory landscape, leaving practitioners uncertain about whether and how to use AL in their tasks. In this work, we make the case that this inconsistency arises from a lack of systematic and realistic evaluation of AL methods. Specifically, we identify five key pitfalls in the current literature that reflect the delicate considerations required for AL evaluation. Further, we present an evaluation framework that overcomes these pitfalls and thus enables meaningful statements about the performance of AL methods. To demonstrate the relevance of our protocol, we present a large-scale empirical study and benchmark for image classification spanning various data sets, query methods, AL settings, and training paradigms. Our findings clarify the inconsistent picture in the literature and enable us to give hands-on recommendations for practitioners. The benchmark is hosted at https://github.com/IML-DKFZ/realistic-al.

Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment
Carsten T. Lüth
Till J. Bungert Lukas Klein 1,2,3
Paul F. Jaeger
German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Germany Helmholtz Imaging, German Cancer Research Center (DKFZ) Heidelberg, Heidelberg, Germany "Institute for Machine Learning, ETH Zürich, Switzerland
carsten. rrrstemm_hstti@kfff-hhoideeeeee
Abstract
Active Learning (AL) aims reduce the labeling burden by interactively selecting the most informative samples from a pool of unlabeled data. While there has been extensive research on improving AL query methods in recent years, some studies have questioned the effectiveness of AL compared to emerging paradigms such as semi-supervised (Semi-SL) and self-supervieed learning (Self-SL), or simple optimization of classifier configurations. Thus, today's s AL literature presents an inconsistent and contradictory landscape, leaving practitioners uncertain about whether and how to AL in their tasks. In this work, make the case that this inconsistency arises from a lack of systematic and realistic evaluation AL identify five key pitfalls in the current literature that
methods. Specifically, reflect the delicate considerations required for AL evaluation. Further, we present evaluation framework that overcomes these pitfalls and thus enables meaningful statements about the performance of AL methods. To demonstrate the relevance our protocol, we present large-scale empirical study and benchmark for image classification spanning various data sets, query methods, AL settings, and training paradigms. Our findings clarify the inconsistent picture in the literature and enable us to give hands-on recommendations for practitioners. The benchmark is hosted at http:///eetaoooooo///M///////tee
Introduction
Active Learning (AL) is popular approach to efficiently label large pools of data by interactively querying the most informative samples for training classification system. While some parts of the
AL community actively propose new query methods (QMs) advance the field [14] 30). others report AL to be generally outperformed by alternative training paradigms to standard training (ST) such as Semi-Supervised Learning (Semi-SI) (20 43 and Self-Supervised Learning (Self-SI.) [6], or
even by well-configured standard baselines 44 To add to the confusion, further studies have shown
that AL can even decrease classification performance in certain settings, a phenomenon referred as
the "cold start problem" [6] 20 43
This heterogeneous state of research in AL poses significant challenge for anyone seeking to efficiently annotate their dataset and facing the questions: On which tasks and datasets is AL beneficial? And how to best employ AL on my dataset?
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Model V13a119
Trahtdd Model
Phiddl
P1
P3
PJ
Pd
Contigration
Cumpaed Aquas
Statinng
Quary
Prrfiem
Uptm
TC
Training Paradigm (P3)
Rdued Week
Baclacc
Ostoact
Musid al
+4
Busdge
Dale (istr.PP)
Upcating
Crucid
P5
Gusty slage
Boga et
Query Sizs (P3)
Yi
paramelero (4) moadd the sarssee
Krideon al
(
1
Beck
v
(
(
Annotsted Quariies
Queris
Zua ci
S
Training Valldation
2 Labeled
2 Unlateded
Human abbling
Panurater altraahval caradd
the valccalaun
Dups
(a)
(b)
Figure 1: (a): The five pitfalls (P1-P5) for meaningful evaluation the context the Active Learning loop. Detailed information is provided in Sec. 2 (b): The five pitfalls are highly prevalent in the current literature (green ticks denote successful avoidance of the respective pitfall). A detailed correspondance between individual studies and pitfalls is provided in Appendix B Our study is the first to avoid all pitfalls and enable trustworthy performance assessment of AL. methods.
We argue that the inconsistency arises from lack of systematic and realistic evaluation of AL methods. AL inherently comes with specific requirements on how methods will be applied in practice. First and foremost, the stated purpose of "reducing the labeling effort task" implies that AL methods need to be rolled out to unseen datasets different from the labeled development data. This inherent requirement of cross-task generalization needs be reflected method evaluation, posing a need to test methods under diverse settings to identify robust and trustworthy configurations for the subsequent "blind" application on real-life tasks (see "validation paradox". Sec. 2.2 However, such considerations are generally neglected in AL research, as identified in our work by means of five key pitfalls in the current literature, spanning from a lack of tested AL settings and tasks to lack of appropriate baselines (see Figure and P1-P5 in Sec. 2.3
To this end, present evaluation framework for deep active classification that overcomes the five pitfalls and demonstrate the relevance of this contribution by means of a large-scale empirical concrete
study spanning various datasets, QMs, AL settings, and training paradigms. To give example for the disruptive impact of our work: We address the widespread pitfall of neglecting classifier configuration (see P4 in Figure 1b) by introducing light-weight protocol for high-quality configuration. The results demonstrate how this simple protocol lets even our random-query baseline
exceed the originally reported performance of recent AL methods [ 30 35 on the widely-uses
CIFAR- (IFAR-10/10 10/10000/// datasets, while drastically reducing computational effort compared to the configuration protocol of Munjal al. 48
This example showcases, that the novelty of our work does not lie in presenting entirely novel results and insights, but in the fact that comprehensive and systematic approach is the first to address all five key pitfalls and thus to provide trustworthy insights into the real-life capabilities of current AL methods. By relating these insights recent studies that have been subject flawed evaluation, we are able to resolve existing inconsistencies in the field and provide robust guidelines for when and how to apply AL on given task.
2 Realistic Evaluation in Active Learning
2.1 Active Learning Task Formulation
As depicted in Figure La AL describes a classification task, where dataset D is given that is divided into a labeled set L and an unlabeled pool 4. Initially, only fraction of the data is labelled ("starting budget"), which ideally split further into a training set and validation set for hyperparameter (HP) tuning and performance monitoring. After initial training, the QM is used to generate queries Qu of a certain amount ("query size") that represent the most informative samples from 24 based on the current classifier predictions. Subsequently, queried samples are labeled (Qc), moved from 2/ to L, and the classifier re-trained on L. This process is repeated until classifier performance is satisfying, monitored on the validation set.

---

2.2 Overview over critical concepts in Active Learning evaluation
Evaluating an AL algorithm typically means testing how much classification performance is gained by data samples queried over several training iterations. The QM selecting those samples is con- sidered useful if the performance gains exceed the gains of randomly queried samples. While this process is well-established, it is prone to neglecting critical concepts for evaluation and thus to over-simplification of how AL algorithms are applied in practice.
AL validation paradox. When applying AL on a new, mostly unlabelled dataset, one can not directly validate which QM type or configuration performs best, this would require labeling individual AL trajectories through the dataset for each setting. This excessive labeling would directly contradict the goal of AL to reduce the labeling effort, a predicament which we refer as the AL validation paradox. Notably, this is in contrast to standard ML, where fixed labeled training and validation splits generally suffices for model selection and configuration.
Special requirements on evaluation. As the described paradox impedes on-the-spot validation, forces one to, instead, estimate how well certain QMs will perform on the given task based solely on prior knowledge. The quality of this prior knowledge depends on how extensively the respective QM has been validated prior to application, i.c. on the development data. This implies several critical requirements on evaluation:
A generalization gap between development and application arises, because the latter typically comes with practical constraints such as on the size of the starting budget, the query size, or a given class imbalance. To avoid AL failure, such constraints need to be anticipated by evaluating QMs on a realistic range of the corresponding design choices.
Given these inherent generalization requirements in AL, it is crucial to simulate the generalization process by evaluating QMs on roll-out datasets, i.e. tasks that were not part of the configuration process on the development datasets.
Evaluation needs to acknowledge that AL is an elaborate and costly training paradigm, and thus ensure that methods are tested against simpler and cheaper alternatives. For instance, if the gain of a QM over random sampling vanishes by simply tuning the learning rate of the classifier, or by single sulf-super pretraining, the QM, and AL in general, provides no practical value.
To effectively describe the current oversight of these requirements the field, we identified five concrete evaluation pitfalls (P1-P5), which are detailed in Section 2.3.
Cold start problem. Neglecting these pitfalls could render AL ineffective, or even counterproductive, for specific tasks, as it might result in QMs underperforming relative to a random-sampling bascline. Such failures occur predominantly in few-label settings and are recognized challenge the research community, often termed as the cold start problem 200)
2.3 Current pitfalls of Active Learning evaluation
The current AL literature features an inconsistent landscape of evaluation protocols, but, as Figure 1b shows, none of them adhere to all requirements for evaluation described above. To study the current oversight of these requirements, we identify five key pitfalls (P1-P5) that need to be overcome for meaningful evaluation of QMs. For visual overview of the pitfalls and how they integrate into the AL setting see Figure
Pl: Lack of evaluated data distribution settings. To ensure that QMs work out-of-the-bos in broad data distribution. Relevant aspects of
real-life settings, they need to be evaluated distribution in the AL-context go beyond the data domain and include class distribution, the relative difficulty of separation across classes, well potential mismatch between the frequency and importance of classes. All of these aspects directly affect the functionality of QM and may lead to real-life failure of AL when not considered in the evaluation.
Current practice: Most current work is limited to evaluating QMs on balanced datasets from one specific domain (e.g. CIFAR-10/100) and under the assumption of equal class importance. To our knowlede knowledge, testing the generalizability of a fixed QM setting to new datasets ("roll-out") has not been performed before. There some experiments conducted on artificially imbalanced dataset
3

---

suggesting good AL performance. Further, Gal et al. [19] study AL on the 1SIC-2016 dataset, but obtain volatile results due to the small dataset size. Atighehchian et al. D] study AL on the MIO-TCD dataset and reported performance improvements for underrepresented classes. In the large study Beck et al. 4 perform experiments on five mostly balanced datasets datasets and [58] use 13 datasets for mostly standard AL experiments.
(CIFAR-LT) B3 44
+ Proposed solution: We argue that the underrepresentation of dass-imbalanced datasets in the field is one reason for current doubts regarding the general functionality of AL. Real-life settings will most likely not be class balanced providing a natural advantage of AL over random sampling. We propose to consider diverse datasets with real class imbalances as an essential part of AL evaluation and advocate for the inclusion of "roll-out" datasets, as a real-lifé life test of selected and fixed AL settings.
P2: Lack of evaluated starting budgets. There are two reasons for why this parameter is essential aspect of AL evaluation: 1) Upon application, the budget might be fixed and the QM is required to generalize out-of-the-box works since a too large budget implies inefficient labeling (equivalent to random queries) and a too small budget is likely cause AL failure (cold start problem). This search needs to be performed prior to an AL application due to the validation paradox.
this setting. 2) We are interested the minimal budget at which the QM
Current practice: Most recent studies evaluate AL single starting budget made of thousands of samples on datasets such as CIFAR-10 10 and CIFAR- 1FFAR-10023 30,
4 [4]] [55]. information-theoree publi- cations commonly use smaller starting budget [9 32,. but typically on even simpler datasets such as MNIST [38]. Beck et al. 4 compare two starting budgets on MNIST reporting no performance drop. On the other hand, some studies benchmarking AL against Semi-SI. and Self-SL compare two [6] or three [20 43) starting budgets often with the conclusion that smaller starting budgets lead to AL failure. Bengar et al. [6] report that there exists a relationship between the number of classes in a task and the optimal starting budget (the intuition being that class number is a proxy for task complexity).
+ Proposed solution: To overcome this pitfall and resolve the current contradictions, we evaluate all QMs for three different starting budgets on all datasets. We refer to these settings as the low-, mid- and high-label regime. Extending on the findings of Bengar et al. [6. adequate budget sizes are determined using heuristics based the number of classes per task.
P3: Lack of evaluated query sizes. The number of samples queried for labelling in each AL iteration is an essential aspect of QM evaluation. This is because, upon application, this parameter might be predefined by the to higher computational efforts but might enable more informed queries and thus less labeling). Since query size cannot be validated on the task at hand due the validation paradox. the generalizability of QMs to various settings of this parameter needs to be evaluated beforehand.
cost ratio of the respective task (a smaller query size amounts
Current practice: In current literature, there is concerning disconnect between theoretical and practical papers regarding what constitutes reasonable query size. Information-theoretie papers typically select the smallest query size possible and QMs such as BatchBALD [32) are specifically designed to simulate reduced query sizes 19 46). In contrast, practically-oriente
papers usually but only in combination with large starting budgets (P2), where cold start problems generally do not occur. Only a few studies perform limited evaluations of varying query sizes. Beck et al. 4 and Zhan et al. (58) report negligible effect of varying query sizes, but only evaluate in combination with large starting budgets (1000 samples). In line with this, Munjal et al. [44] conclude that the choice of query size does not matter, but only compared two large values (2500 versus 5000 samples) on a fixed large starting budget (5000 samples). Atighehchian et al. 3 come to a similar conclusion, but also only considered relatively large starting budget (500) for
employ larger query sizes [30
43
44
51
55
prretrrande models on CIFAR-10, where, again, no cold start problem occurs. Bengar et al. [6] employ varying query sizes without further analysis of the parameter.
+ Proposed solution: To overcome this pitfall and reliably study the effect of query sizes also in low-label, i.e. high-risk, settings, we evaluate all QMs for three different query sizes in combination with varying starting budgets on all datasets (i.e. as part of the low-, mid-, and high-label regimes). For specific focus the effect of query size in the lowelabel settings, we perform an additional

---

ablation with varying query sizes on small fixed starting budget.
P4: Neglection of the classifier configuration. As stated in Sec. 2.2 when aiming to draw conclusions about the performance usefulness of QM, it is critical that this evaluation be based on well-configured classifiers. Otherwise, performance gains might be attributed to AL that could have been achieved by simple hyperparameter (HP) modifications. Separating a validation split from the training data is a crucial requirement for sound HP tuning.
Current practice: Most studies in AL literature do not report how HPs obtained and do not mention the use of validation splits 30 35 43 51 [5]. Typically, reported settings are copied from fully labeled data scenarios. In some cases, even the proposed QMs feature delicate HPs without reporting how they were optimized raising the question of whether these settings generalize to new [ demonstrate how adequate HP tuning on validation set allows a random query baseline to outperform current QMs under their originally proposed HP settings. However, they run full grid search for every QM and AL training iteration, which might not be feasible in practice.
data 30 55 [5). Munjal et al.
+ Proposed solution: To overcome this pitfall and enable meaningful performance assessment of AL methods, we define a validation dataset of a size deducted heuristically from the starting budget. Based on this data, small selection of HPs (learning rate, weight decay and data augmentations [7] is tuned only once per AL experiment while training on the starting budget. The limited search space and discarding of multiple tuning iterations result in a lightweight and practically feasible protocol for classifier configuration.
P5: Neglection of alternative training paradigms. Analogously to arguments made in P4, mean- ingful evaluation of AL requires comparison against alternative approaches that address the same problem. Specifically, the training paradigms Self-SL 13 25] and Semi-SL [3 [2] have shown strong potential to make efficient use of an unlabeled data pool in classification task thus alleviating the labeling burden. Additionally to benchmarking AL against Self-SL and Semi-SL. the question arises of whether AL can yield performance gains when combined with these paradigms.
Current practice: While most AL studies do not consider Self-SI. and Semi-SI., there are a few recent exceptions: Bengar et al. (6 benchmark AL in combination with Self-SL. and conclude that AL only yields gains under sufficiently high starting budgets. However, these results suffer from inadequate classifier configuration (P4). Yi et al. (54] propose QM in combination with Self-SL, but the employed Self-SL strategy is limited by compatibility with the proposed QM. Further, Gao et al. 20 combine Semi-SL with AL and report superior performance compared to ST for CIFAR- and ImageNet [8), 1 i.c. the datasets on which Semi-SL methods have been developed. Similarly, Mittal et al. [43] evaluate the combination of Semi-SL and AL on CIFAR- 10/100, 100, reporting strong improvements compared to ST and find that AL decreases performance for small starting budgets.
+ Proposed solution: To overcome this pitfall and resolve current inconsistencies, we benchmark all QMs against both Self-SL and Semi-SI, and evaluate the combination of AL with these paradigms. Crucially, we are the first to study these relations as part of reliable evaluation, i.e. while avoiding all other key pitfalls (PI-P4).
3 Experimental Setup
This section describes the design of our empirical study in light of the proposed improvements for AL evaluation (detailed experimental settings can be found in Appendix Dj. We first address P1 by extending our evaluation to 5 different datasets, containing different label distributions. Specifically, these datasets include CIFAR- 10, CIFAR- 100, CIFAR-10 LT, ISIC-2019 and MIO-TCD, where the first three are developmental datasets and the latter two are used exclusively for the proposed roll-out evaluation. Further, we address P2 and P3 by defining three different label regimes which we refer to as "low-lahel", "mid-label" and "high-label" regimes. Starting budgets and query sizes are both set to 5 x C, 25 x C' and 100 x C' for the three label regimes, where C. denotes the number of classes. []] To address P4, we configure our classifiers for all three label regimes based on a validation set five
'These deviate for CIFAR- 100 15 C' (low-), 10 x C' (mid-), 50 x C' (high-label) due to smaller ratio of dataset size number of classes.
5

---

times the size of the starting budget. Further, addressing P4 and P5 we use ResNet-18 [24] as the backbone all experiments and optimize the essential HPs for each respective training paradigm. At last, we address P5 by comparing randomly initialized models (standard training) against Self-SI. pre-trained models and Semi-SI. models.
Compared query methods. In this section, we describe the applied QM in more general terms and refer to Appendix Aor details. We focus exclusively on QMs which do not alter the classifier and require no configuration of additional HPs except for bayesian QMs which are based on dropout. Generally, QMs can be divided into two categories as they are either based on uncertainty estimation or enforcing exploration. Random: The baseline all QMs are compared against which randomly draws samples from the pool U. Core-Set: This explorative QM aims to find the core-set of convolutional neural network [49 by means of a K-Center Greedy approximation on intermediate QM selects the samples with the highest entropy across predicted class scores [50]. BALD: This uncertainty-based QM uses the mutual information between the class label and the model parameters with regard to each sample for greedy selection (27], it was introduced with dropout for deep bayesian active learning [9]] BADGE: This QM performs clustering based on per-sample gradient vectors obtained via proxy labels. This enables a selection that is both diverse and guided by uncertainty 2
representations. Entropy: This uncertainty-
Datasets. The initial datasets for our experiments are CIFAR-10/100 [6) For further analysis we added CIFAR-10 LT [10], an artificially created dataset built upon CIFAR-10 with a long-tail distribution of classes following an exponential decay for the training split. The imbalance factor
was selected to be 50, following [35]. On all three of these datasets, we use accuracy on the class-balanced test set as the primary performance metric. Finally, we selected ISIC-2019 [5] 16 53) and MIO-TCD as roll-out datasets to verify the generalizability of AL methods to new settings. Both of these datasets feature inherently imbalanced class distributions, and are more likely to be subject to label noise compared to the CIFAR datasets. As such, we deem the two roll-out datasets an essential step toward a realistic evaluation of AL methods. For both MIO-TCD and ISIC-2019, we use balanced accuracy as the primary performance measure (Appendix D)
Active learning setup. We report performance measures for each dataset on identical test splits based on three experiments using different seeded models and different train and validation splits to reduce the possible influence of these parameters on our results. Further, we train the models from scratch on every training step to avoid correlated queries [32]
Training paradigms. Randomly initialized and supervised-trained models are referred to ST models. Further, we use the popular contrastive SimCLR [3] training as a basis for Self-SL pre- training. These models are fine-tuned and are referred to Self-SL models. Self-SL models have a two-layer MLP as a classification head to make better use of the representations (ablation in Appendix E For ST and Self-SL models, we obtain bayesian models by adding dropout to the final representations before the classification head following [19]. As Semi-SL method, we use FixMatch [52] which combines the principles of consistency and uncertainty reduction. Due the long training times (factor 80) compared to ST training, we only ran experiments in the low- and mid-label regime while increasing the query size by a factor of three to reduce training costs.
Hyperparameter selection. The HPs for our models are selected for each label regime before the AL loop is started using the corresponding validation set. For Self-SL and ST models we use fixed training recipe (Scheduler, Optimizer, Batch Size, Epochs) and only optimize learning rate, weight decay and data augmentations. The data augmentations used are standard augmentations and Randaugment which uses stronger augmentations acting as a regularization [17]. For Semi-SL SL methods we fix all HPs with the exception of learning rate and weight decay following Sohn et al. [52]. Model selection for ST and Self-SL models is based on the best validation set epoch, while for Semi-SI. models the final checkpoint is used. For imbalanced datasets, we use oversampling for ST and Self-SL. pre-trained models Buda et al. g and use weighted cross-entropy-loss and distribution alignment for FixMatch.
Low-Label query size ablation. To investigate the effect of query size in the low-label regime, we conduct an ablation with Self-SL pre-trained models on CIFAR-I00 and ISIC-2019. For 100 the query sizes are 50, 500 and 2000, while for ISIC-2019 they are 10, 40 and 160.

---

031
CIFAR-100
Labeled
Labeled
Query Methods:
Training Strategies:
EEnropy
Standard Training
Core-Sec
Self-SL Pro- Pro-Trained
BALD
Semi-SL
BADGE
100% Labeled Baseline
Random
Development dataset
+
Roll-out dataset
CIFAR-10
HL
Low- mid- high-label regime
1500
6200
Lubeled Bampies
Bulbrro Acouroy
1.1
MIO-TCD ML
HL
L1
ISIC-2019
ML
HLL
Labnleed Sumpie
Labrled Sumples
Figure 2: Results obtained with our proposed evaluation protocol over five different datasets and
the three label regimes. These experiments AL and reveal insights along the lines of the five key parameters as discussed in Sec. The strong performance dip on MIO-TCD and ISIC-2019 is discussed in Sec.
our knowledge the largest conducted study for
4 Results & Discussion
The results of our empirical study are shown in Figure An in-depth analysis of results for individual datasets and analysis based on the pairwise penalty matrix (2) and area under the budget curve [57] be found in Appendix F Here, will discuss the main findings along the lines of our five identified pitfalls of evaluation (P1-P5). Our findings demonstrate the relevance of the proposed protocol for realistic evaluation and its potential to generate trustworthy insights on when and how AL works. If not otherwise mentioned, all references in this chapter refer to AL studies.
P1 Data distribution. The proposed evaluation over diverse selection of dataset distributions including specific roll-out datasets proved essential for realistic evaluation of QMs as well as the different training strategies. One main insight is the fact that class distribution is crucial predictor for the potential performance gains of AL on dataset: Performance gains of are generally higher on imbalanced datasets and occur consistently even for ST models with small starting budget, which are typically prone to experience cold start problems. This observation is consistent with a few previous studics [30 35 54]. Further, results underpin the importance of the roll-out datasets c.g. when looking at the sub-random performance of BALD (with Self-SL) and Entropy (with ST) on MIO-TCD, Such worst-case failures of AL application (increased compute and labeling effort due to AL) could not have been predicted based on development data where all AL-paarameter are optimized. Another example is the lack of generalizability of Semi-SL, where performance in relation to other Self-SL and ST decreases gradually with data complexity (going from CIFAR- 10 to CIFAR-100/-11 LT to the roll-out datasets MIO-TCD and ISIC-2019.)
P2 Starting budget. The comprehensive study of various starting budgets on datasets reveals that AL methods are more robust with regard small starting budgets than previously reported
7

---

Query Size 50
Query Size 500
Query Size 2000
Quary Size
Quary Size
Quary Size 16D
Staancdd AUc REcuaa
4.53
CIFAR-100
ISIC-2019 19
130
200
Labokoc Barrples
2000
2000
20 Labele Bample
Query Methods:
Entropy
Core-Set
BALD
-ADGE
-Random
Training Strategy:
Self-SL
Figure 3: Low-label query size ablation on CIFAR- 100 and ISIC-2019. On CIFAR- 100, reducing the query size resolves the observed failure mode of BALD. However, no improvement is observed on 2019- presumably because BALD already shows the best performance. Further, BADGE performs consistently well across all query sizes without failure modes, revealing its robustness also for low-label settings.
With the exception of Entropy we did not observe cold start problems even for any QM even in combination with notoriously prone ST models. The described robustness is presumably enabled by our thorough classifier configuration (P4) and heuristically adapted query sizes (P3). This finding has great impact potential suggesting that AL can be applied at carlier points in the annotation process thereby further reducing the labeling cost, especially in combination with BADGE which performed consistently better or on par with Random. Similarly for Self-SL models, AL performs well small starting budgets with the exceptions of CIFAR- 100 (BALD, Entropy) and MIO-TCD (BALD, Core-Set, Entropy).
6 20
P3 Query size. Based on our evaluation of the query size we can empirically confirm its importance with regard to 1) general AL performance and 2) counteracting the cold start problem. The are, however, surprising findings indicating that the exact interaction between query size and performance remains an open research question. For instance, we observe the cold start problem for BALD with Self-SL on CIFAR-100 (~ 50% accuracy at 2k labeled samples for query sizes of 500 (low-label regime) and 1k (mid-label regime). On the other hand, in the high label regime (budget of 5k and query size of 5k) ST and Self-SL models with similar accuracies of 50% and 60%, respectively, benefit from BALD. Since cold start problems are commonly associated with large query sizes, this finding seems counter- intuitre, but has been reported before although without further investigations 6 20 43 To gain a better understanding of this phenomenon and how QMs interact with query size in the low-label regime, we performed a dedicated experiment series for Self-SL training on CIFAR-100 and ISIC-2019 (see Figure 3 and Appendix (.2).. Due to significant improvement of BALD on CIFAR- 100 for even smaller query sizes from aub-Random to Random performance and no worsening on ISI9-2019, we can conclude that small query sizes represent an effective countermeasure against the cold-start problem for BALD contrary to the findings of [44, (88 and currently not considered as solution 66 20 43 This potentially explains the gap between these findings and more theoretical works which advertise using the smallest query size possible [99 33
Importantly, while smaller query sizes appear as a potential solution for the observed instabilities of BALD, they seem to have no considerable effect on the other QMs. Moreover, our ablation reveals that even in the low-label settings, BADGE is the overall most reliable of all compared QMs and exhibits no nab-Random performance. This adds to the existing reports of the robustness of BADGE for higher label regimes 4 588
P4 Classifer configuration. Our results show that method configuration on a properly sized validation set is essential for realistic evaluation in AL. For instance, our method configuration has the same effect on the classifier performance as increasing the number of labeled training samples by factor
of ~ 5 (e.g. our ST model reach approximately the same accuracy of ~ 44% trained on 200 samples compared to models in the AL study by Bengar et al. [6] trained on 1k samples). The effectiveness of our proposed lightweight HP selection on the starting budget including only three parameters (Sec.) 3 demonstrated by the fact that all our ST models substantially outperform respective models found
in relevant literature 6 20 30 35
43
54
55 where HP optimization is generally neglected. Details
for this comparison are provided in Appendix H This raises the question of to which extent reported
AL advantages could have been achieved by simple classifier configurations. Further, our models
8

---

also generally outperform expensively configured models by Munjal et al. 44] Thus, we conclude that manually constraining the search space renders HP optimization feasible in practice without decreasing performance and ensures performance gains by Active Learning are not overstated. The importance of the proposed strategy to optimize HPs on the starting budget for each new dataset supported by the fact that the resulting configurations change across datasets.
P5 Alternative training paradigms. Based on our study benchmarking AL in the context of both Self-SL and Semi-SL, we see that while Self-SL generally leads to improvements across all experiments, Semi-SL only leads to considerably improved performance on the simpler datasets CIFAR- 10/100, on which Semi-SL methods are typically developed. Generally, models trained with either of the two training paradigms receive lower performance gain from AL (over random querying) compared to ST. Crucially, Self-S) SL models converge around 2.5 times faster than ST models, while the training time of Semi-SI. models around 80 times longer than ST and often yields only small benefits over Self-SL. or ST models. The fact that AL entails multiple training iterations amplifies the computational burden of Semi-SI., rendering their combination prohibitively expensive in most practical scenarios. Further, the fact that our Semi-SL models based on Fixmatch do not seem to generalize to more complex datasets in our setting stands in stark contrast to conclusions drawn by 20 43 to which the emergence of Semi-SL renders AL redundant. Interestingly, the exact settings where Semi-SL does not provide benefits in our study are the ones where AL proved advantageous. The described contradiction with the literature underlines the importance of our proposed protocol testing for a method's ''s generalizability to unseen datasets. This is especially critical for Semi-SL, which is known for unstable performance noisy and class imbalanced datasets noted by studies focusing on Semi-SI.
22
45 56
Limitations. We propose light-weight and practically feasible strategy for HP optimization and made other design choices (e.g. ResNet-18 classifier), thus we can not guarantee that our configura- tions are optimal for all compared training paradigms and would like to provide a critical discussion: 1) The ResNet-18 18 in combination with our shortened training times, might hinder Semi-SL perfor- mance more than other training paradigms. This setting is necessary to be able to cope with the computational cost of Semi-SL (factor ~ 400 training time compared with ST). 2) The validation set size of 5x starting budget size (i.e. training set) could be considered larger than practically desirable, where most data would be used for training. This design decision follows the study of Oliver et al. (5). showing that an adequately sized validation set is necessary for proper HP selection (especially for Semi-SL). 3) We observe a performance dip of ST models on MIO-TCD and ISIC-2019 at 7k samples, which we attribute our HP selection scheme. This indicates that HPs might need to be re-selected occasionally at certain training iterations. However, such cases are immediately detected in practice allowing for correction where necessary by re-optimizing the HPs (see Appendix L.T. A more extensive discussion of limitations can be found in Appendix
5 Conclusion & Take-Aways
Our experiments provide strong empirical evidence that the current evaluation protocols do not sufficiently answer the key question every potential AL practitioner faces: Should I employ AL on my dataset? Answering this question entails estimating whether an AL algorithm will provide performance gains over random queries and thus whether the expected reduction in labeling cost outweighs both the additional computational and engineering cost attributed to AL. We argue that our proposed protocol for realistic evaluation represents cornerstone towards enabling informed decisions in this context. This is made possible by focusing on evaluating the generalizability of AL to new settings under real-world conditions. This perspective manifests itself in the form of five key pitfalls in the current AL literature regarding data distribution, starting budget, query size, classifier configuration, and alternative training paradigms (see Sec. 2.3p. Even though the thorough evaluation increases the computational cost once during method development, it will lead to net cost reduction in the long-term by enabling an informed selection of AL methods in practice, thereby effectively reducing annotation cost upon application.
We hope that the proposed evaluation protocol in combination with the aublicl--vvailabl benchmark will help to push active learning towards robust and wide-spread real-world application.

---

Main empirical insights revealed by our study:
Assessment AL methods literature substantially hampered by subpar classifier configu-
rations, but meaningful assessment can be restored by our protocol for lightweight hyperparameter tuning on the starting budget.
AL generally provides substantial gains in
settings.
BADGE is the best- best-performing QM across a realistic range of datasets, starting budgets, and query sizes and exhibits no failure modes (i.e. sub-Random performance).
Combining AL with Self-SL considerably improves performance, shortens the training time, and stabilizes optimization, especially in low-label settings.
FixMatch results indicate that Semi-SL methods perform well on datasets where they have been developed, but may struggle to generalize to more realistic scenarios such as class imbalanced data. Combining the paradigm with AL additionally suffers from extensive training times.
BALD with Self-SL pre-trained models benefits from smaller query sizes on small starting budgets possibly circumventing the "cold start problem".
Take-aways for developing and proposing new Active Learning algorithms:
AL methods should be tested for generalizability roll-out datasets and come with clear recipe for real-world application including how to adapt all design choices to new settings. Since the expected benefit of AL high potential for wide-spread real-world use of AL by reducing the two prevalent cost factors: 1) Engineering costs, which are reduced by building easy-to-use AL tools. 2) Computational costs, which are reduced by explicitly including methods that shorten the training time AL.
setting increases with lower application costs, believe there is
Take-aways for the cost-benefit analysis of deploying Active Learning:
Since identify BADGE as a robust query method exhibiting no sub-random random-sampliin perfor- mance across all tested settings, the potential harm of AL is minimized allowing to base decisions around deploying AL on the deseribed cost-benefit analysis.
The expected benefit is high in settings, where there mismatch between the task-specific importance of individual classes and their frequency in the dataset (e.g. class-imbalanced datasets in combination with tasks requiring balanced classifier).
3. The expected benefit further increases with the labeling cost that will be reduced by AL. AL is thus likely to yield net benefit in settings with high label cost and low computational and engincering costs.
Future research:
Future research may investigate to what extend AI benefits from foundation models such as CLIP [7], which have been shown to reach strong performance in low-label settings through finetuning or knowledge extraction. Additionally, their rapid re- c-ffintuning during AL iterations may enable real-time interactive labelling. A further important aspect might be to study the effect of AL training on the classifiers' ability to detect outliers [18] or catch failures under distributions shifts (28
10

---

Acknowledgements
This work was funded by Helmholtz Imaging (HT), a platform of the Helmholtz Incubator on Information and Data Science. This work is supported by the Helmholtz Association Initiative and Networking Fund under the Helmholtz Al platform grant (ALEGRA (ZT-I-PF-5-12I)). We thank Maximilian Zenk, Sebastian Ziegler and Fabian Isensee for insightful discussions and feedback.
References
[1] Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. European Conference on Computer Vision, pages 137-153. Springer, 2020.
[2] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. arXiv: 03667/ [cs, stat), February 2020.
[3] Parmida Atighehchian, Frédéric Branchaud-Charce and Alexandre Lacoste. Bayesian active learning for production, systematic study and a reusable library. 1111 [cs, stat], June 2020.
[4] Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh Iyer. Effective evaluation of deep active learning on image classification tasks. arXiv preprint arXiv:2106. 15324, 2021.
[5] William H. Beluch, Tim Genewein, Andreas Nurnberger, and Jan M. Kohler. The Power of Ensembles for Active Learning in Image Classification. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9368-9377, Salt Lake City, UT, June 2018.
IEEE. ISBN 978-1-5386-6420-
doi: 11111///VV...218..09999999
[6] Javad Zolfaghari Bengar, Joost van de Weijer, Bartlomiej Twardowski. and Bogdan Raducanu. Reducing label effort: Self- Self-supervised meets active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1631-1639, October 2021.
[7] Sayedali Shetab Boushehri, Ahmad Bin Qasim, Dominik Waibel, Fabian Schmich, and Carsten Marr. Systematic comparison incomplete-supervisin approaches for biomedical imaging classification. Preprint, Bioinformatics, December 2020.
[8] Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 201 8. ISSN 0893-6080. doi:
URL. htttt:://////
[9] Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pages 108-1 122, 2013.
[10] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aubel-distribution-awaaaa margin loss. Advances in neural information processing systems, 32, 2019.
[11] Yao-Chun Chan, Mingchen Li, and Samet Oymak. On the Marginal Benefit of Active Learning: Does Supervisrnv Eat its Cake? In ICASSP 2021 202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3455-3459, Toronto, ON, Canada, June 2021. IEEE. ISBN 978- 978-1-72817-605-5 doi: 10. 101100//ASSSS33925
0221.9444665. 14665.
[12] Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative Pretraining from Pixels. ICML, page 13, 2020.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, July 2020.

---

[14] Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34:1 1933-1 1194,, 2021.
[15] Noel CF Codella. David Guiman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo. Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168-172. IEEE, 2018.
[16] Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:7908. Stti://90.022500. 02288, 2019.
[17] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition workshops, pages 702-703, 2020.
[18] Arna Das, Gantavya Bhatt, Megh Bhalerao, Vianne Gao, Rui Yang, and Jeff Bilmes. Accelerat- ing batch active learning using continual learning techniques. arXiv preprint arXiv:2305.06400 2023.
[19] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1 1192 PMLR, 2017.
[20] Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ö. Ank,, Larry S. Davis, and Tomas Pfister. Consistency- Semi-supervised Active Learning: Towards Minimizing Labeling Cost. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020, volume 12355, pages 510-526. Springer International Publishing, Cham, 2020. ISBN 978-3-030-58606-5 778-3-030-58607-2. doi: (0.1007/978-3-000 1--0003050----
[21] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. DXXiv:/803.07722 [cs], March 2018.
[22] Lan-Zhe Guo, Zhen-Yo Yu Zhang, Yuan Jiang. Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In International Conference on Machine Learning, pages 3897-3906. PMLR, 2020.
[23] Lan-Zhe Guo, Zhi Zhou, and Yu-Feng Li. Robust deep semi-supervised learning: A brief introduction. arXiv preprint arXiv:2202.0597. 2022.
[24] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv:1911.088 [cx], March 2020.
[26] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. In The collected works of Wassily Hoeffding. pages 409-426. Springer, 1994.
[27] Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning. arXiv: 121.57445 [cs, stat], December 2011.
[28] Paul F Jaeger, Carsten Tim Lüth, Lukas Klein, and Till J Bungert. A call to reflect on evaluation practices for failure detection in image classification. In The Eleventh International Conference on Learning Representations, 2023.
[29] Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang. Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-laber for imbalanced remi-superviedd learning. Advances in neural information processing systems, 33: 33:14567-14579, 2020.
[30] Kwanyoung Kim, Dongwon Park, K Kwang In Kim, and Se Young Chun. Task-Aware Variational Adversarial Active Learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8162-8171. 11..2887..NNsi..... Nashville, TN, USA, June 2021. IEEE. ISBN 978-1- 66544-509-2. doi: 10.10999CPR4444..222.0000
[31] Andreas Kirsch and Yarin Gal. A Practical & Unified Notation for information-Thherrtiic Quantities in ML. arXiv:2106 12062 [cs, stat], December 2021.
12

---

[32] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. BatchBALD: Efficient and diverse batch acquisition for deep bayesian active learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[33] Andreas Kirsch, Sebastian Farquhar, Parmida Atighehchian. Andrew Jesson, Frederic Baanchaud-Charronn and Yarin Gal. Stochastic Batch Acquisition for Deep Active Learn- ing. arXiv:2/06. /0059 [cs, stat], January 2022.
[34] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active Testing: Sample- Efficient Model Evaluation. WrXiv:2103.05333 [cs. stat]. June 2021.
[35] Ranganath Krishnan, Nilesh Ahuja, Alok Sinha, Mahesh Subedar, Omesh Tickoo, and Ravi Iyer. Improving Robustness and Efficiency in Active Learning with Contrastive Loss. 27X6v:2109.06873 [es]. September 2021.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[37] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-Tuming can Distort Pretrained Features and Underperform Out-of-Distributte arXiv:2202.1005- [cs], February 2022.
[38] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmistl, 1998.
[39] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning. ICML, volume 3, page 896, 2013.
[40] Peng Liu, Lizhe Wang, Guojin He, and Lei Zhao. A Survey on Active Deep Learning: From Model-driven to Data-driven, February 2022.
[41] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang. Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2021. doi: 10. 10.1199/TDEE202 1.390866..
[42] Zhiming Luo, Frédéric Tranchaud-Charron Carl Lemaire, Janusz Konrad, Shaozi Li, Akshaya Mishra, Andrew Achkar, Justin Eichel, and Pierre-Mare Jodoin. Mio-ted: A new benchmark dataset for vehicle classification and localization. IEEE Transactions on Image Processing, 27 (10):5129-514) 2018. 8. doi: 10. 10.1 109/119.2001.10000 8.2448705.
[43] Sudhanshu Mittal, Maxim Tatarchenko, Özgün Çiçek, and Thomas Brox. Parting with Illusions about Deep Active Learning. arXiv: 1912. 9..66666.11 [cs], December 2019.
[44] Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 223-232. 232, 2022.
[45] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic Evaluation of Deep Semi- Semi-Supervised Learning Algorithms. arXiv:1804.09177 [cs, stat), June 2019.
[46] Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hemándezz-Lobtto. Bayesian Batch Active Learning as Sparse Subset Approximation. arXiv:1908.022 [cs, stat], February 2021.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 18-24 Jul 2021. URL
mlr, wwasppressvvlls pptes//v13//araorr222..
html
[48] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), (TCV),,15(3):2))) 115():2111222 2015. doi: 10. 10011077/ 12077//1126--00--00166-
[49] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. In International Conference Learning Representations, 2018.
13

---

[50] Burr Settles, Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Matee
2009.
[51] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[52] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi- supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol- ume 33, pages 596-608. Curran Associates, Inc., 2020.
[53] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham1000) dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1): 1-9, 2018.
[54] John Scon Keun Yi, Minseok Seo, Jongchan Park, and Dong-Geol Choi. Using Self-Supervised Pretext Tasks for Active Learning. arXiv:2201.01 07459 [cs], January 2022.
[55] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[56] Maximilian Zenk, David Zimmerer, Fabian Isensee, Paul F Jäger, Jakob Wasserthal, and Klaus Maier-Hein. Realistic evaluation of fixmatch on imbalanced medical image classification tasks. In Bildverarbeitung für die Medizin 2022, pages 291-296. Springer, 2022.
[57] Xueying Zhan, Huan Liu, Qing Li, and Antoni B Chan. comparative survey: Benchmarking for pool-based active learning. In ICCAI, pages 4679 2021.
[58] Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan, A comparative survey of deep active learning. arXiv preprint arXiv:2203.13450, 2022.
14

---

A Active learning, in more detail
First, we will give an additional task description of Active Learning (AL) in Appendix A.1 introducing necessary concepts and mathematical notation for our compared query methods which are discussed in Appendix A.2 Finally, we give intuitions where the connection of AL to Self-Supervised Learning (Self-SL) and Semi-Supervised Learning (Semi-SL) lies in Appendix A.3 and Appendix F.4
A.1 From supervised to active learning
In supervised learning, we are given labeled dataset of sample-target pairs (v. y) E L sampled from an unknown joint distribution p(r, y). Our goal is to produce a prediction function (Y|ar, 0) parametrized by 6, which outputs a target value distribution for previously unseen samples from p(z). Choosing 0 might amount, for example, to optimizing loss function which reflects the extent to which argmax,
c|z, 0) y for (z, y) E C. However, in pool-based AL we are additionally given a collection of unlabeled samples 24, sampled from p(æ, v). By using query method (QM), we hope to leverage this data efficiently through querying and successively labeling the most informative samples (Bg) from the pool. This should lead to prediction function better reflecting p({(z)) than if random samples were queried.
From a more abstract perspective, the goal of AL is to use the information of the labeled dataset L and the prediction function p(Y|a,, 0), to find the samples giving the most information where p({(ar) deviates from p(Y [zz 0). This is also reflected by the way performance in AL is measured - which is the relative performance gain of the prediction function with queries from a QM compared to random queries. Making the prediction function implicitly the gauge for measuring the "success" of an AL strategy.
At the heart of AL is an optimization problem: AL is a game of reducing cost - one trades in computation cost with the expectation of lowering labeling cost which is deemed to be the bottleneck.
A.2 Query methods
comprehensive overview of AL and its methods is out of the scope of this paper, we refer interested readers to [50] as basis and (40) for an overview of current research. Most QMs fall into two categories following either: explorative strategies, which enforce queried samples explore the distribution of p(r); and uncertainty-base -bsed strategies, which make direct use of the prediction function p(Y|z: [z, 0). F The principled combination of both strategies, especially allow larger query sizes for uncertainty- based QM, is an open research question.
In our work, we focus exclusively on QMs which induce no changes to the prediction function and add no additional HPs except for Bayesian QMs modeled with dropout. This immediately rules out QMs like Learning-Lose (55) or TA-VAAL EQ. changing the prediction function, and QMs like VAAL [51]. introducing new HPs. The QMs we use for our comparisons are currently state-of-the-ar in AL on classification tasks. Further, they require no additional hyperparameters (HPs) to be set for the query function which is hard to evaluate in practice due to the validation paradox.
For this chapter we follow the notation introduced and x represents a concrete sample of variable X.
B1 where e.g. X represents random variable
Random Draws samples from the pool U randomly which follows p(æ, v). Therefore it can be interpreted an exploratory QM.
Core-Set This method is based on the finding that the decision boundaries of convolutional neural networks are based on a small set of samples. To find these samples, the Core-Set QM queries samples that minimize the maximal shortest path from unlabeled labeled sample in the representation space of the classifier [49]. This is also known as the K-Center problem, for which we use the K-Center greedy approximation. It draws queries especially from tails of the data distribution,
TTii conceptually similar to the exploration and exploitation paradigm seen in Reinforcement Learning and there actually exist strong parallels between Reinforcement Learning and Active Learning- so much so that Reinforcement Learning has been proposed to use in AL and AL-based strategies have been proposed to be used in Reinforcement Leamingg
15

---

to cover the whole dataset as well as possible. Therefore we classify it as an explorative strategy.
Entropy The Entropy QM greedily queries the samples x with the highest uncertainty of the model with C being the number of classes.
as shown in Equation
H(Y 0) Σ p(Y = c/a,, 0) log(p(( = =/2,,)
(I)
BALD Uses bayesian model and selects greedily query of samples with the highest mutual
information between the predicted labels Y and weights & for a sample following [19]. From the weight variable θ the concrete values 0 p(0[C) then obtained by MC sampling of bayesian dropout model [19].
0I(y = (((( c/2, C) log(p(Y = c/r, L))
c=1
- -Ep(112 [H({|z,
Where pr)(2, L) = Ep(oic) (p(Y|x,, ()]
BADGE
Uses the K-MEANS+ initialization algorithm on the last layer gradient embeddings G Woo-,C(ff p(Y x, (which capture diversity due diverse parameter updates) and large loss gradients (capturing uncertainty due to large loss changes) (2]
x, 0r E U} to obtain B centers. These are likely to have diverse directions
A.3 Connection to self-supervised learning
The high-level concept of Self-SL pre-training is obtain model by training it with proxy task that is not dependent on annotations, leading to representations that generalize well to a specific task. This allows the induction of information from unlabeled data into the model the form of an initialization. which can be interpreted as a form of bias. Usually, these representations are supposed to be clustered based on some form of similarity, which often induced directly by the proxy task and also the reason why different proxy tasks are useful for different downstream tasks. Several different Self-SL pre-training strategies were developed based on different tasks s.a. generative models or clustering [2 21 25), with contrastive training being currently the de facto standard in image classification. For more thorough overview over Self-SL we refer interested reader to [41] Based on this, we use the popular contrastive SimCLR 13
training strategy as a basis for our Self-SL pre-training.
A.4 Connection to semi-supvervised learning
In Semi-SL the core idea to regularize (x, 9) by inducing information about the structure of p(x) using the unlabeled pool additionally to the labeled dataset. Usually, this leads to the representations of unlabeled samples with the clustering being more in line with the structure of the supervised task (39.. Several different Semi-SL methods were developed based on regularizations on unlabeled samples, which often fall into the category of enforcing consistency of predictions against perturbations and/or reducing the uncertainty of predictions (for more information, we refer to (55). For more thorough overview of Semi-SI. we refer interested readers to 7 45) In our experiments, use FixMatch [52] as Semi-SL method, which combines both aforementioned principles of consistency and uncertainty reduction in a simple manner.
16

---

B Active learning literature, in more detail
We will discuss the current literature landscape of deep active classification with a focus on our proposed key-pitfalls shown in Figure Tb
The rules for evaluation of each the five pitfalls (PI-P5) are:
PI Data distribution
Use of multiple datasets for evaluation featuring class-imbalanced datasets.
P2 Starting budget
Evaluation or ablating the influence of the starting budget on multi- ple datasets explicitly.
P3 Query size
Evaluation or ablating the influence of the query size on multiple datasets explicitly.
P4 Performant baselines
Performance is close to ours or Munjal et al. 44 for ST models on CIFAR- 10/ (seee Appendix for details).
P4 HP Optim. & Val. Split
The use of a dedicated validation set to configure the classifier.
P5 Self-SL
Benchmarking AL with a performant Self-SL training paradigms. Benchmarking AL with performant Semi-SL training paradigm
P5 Semi-SL
Munjal et al. 44 Evaluate the performance of AL methods and compare against and with well finetuned baseline models using AutoML.
PI Data Distribution: Perform experiments on CIFAR-10, CIFAR-100 and limited experiments on ImageNet. They perform an ablation on CIFAR-100 with artificial imbalanced dataset.
due limited imbalanced datasets.
P2 Starting Budget: Perform no experiments at all regarding the starting budget.
X
P3 Query Size: Perform ablations on CIFAR-10/100 comparing query sizes of 5% (2500) 10% (5000). v
P4 Performant Baselines: They achieve performance CIFAR-10/100 on par with ours (see Appendix
P4 HP Optim. & Val. Split They explicitly use a validation set and finetune their hyperparameters based on the validation set performance using AutoML.
P5 Self-SL: They do not consider using models pre-trained with Self-SL. +X X
P5 Semi-SL: They do not consider using models trained with semi-supervised training paradigms. + X
Mittal et al. [43 Evaluate the performance of AL methods and set them into context with semi- supervised training paradigms.
PI Data Distribution: Perform experiments on CIFAR- 10, CIFAR-100.
Perform experiments both on the standard setting with starting budget of 5000 (10%) CIFAR-10/100 well 250 (CIFAR-10) and 500 (CIFAR-100). due limited settings.
P2 Starting Budget:
P3 Query Size: They do not consider speicifally ablating the query size.
'We only base this on ST models, as getting good performance with Self-SL and Semi-SL. for low data settings can be achieved without taking HP configuration into account, as they can simply be taken from paper focusing them which often very large validation

---

P4 Performant Baselines:
Their random baseline is more performant on CIFARI0/100 than most of the literature. However, not as good as Munjal et 44 or ours (see Appendix H
due to performance being good but no on par with ours.
P4 HP Optim. & Val. Split They do not state optimizing their hyperparameters based on validation set.
+ X
P5 Self-SL: They do not consider using models pre-trained with Self-SL. + X
P5 Semi-SL: They evaluate AL with and against the semi-supervised training paradigm 'Unsupervised Data Augmentation for Consistency Training'.
Bengar et al. 66 Evaluate the performance of AL methods and set them into context with self-
supervised training paradigms.
PI Data Distribution: They perform experiments on CIFAR-10/100 and TinylmageNet. All of which class balanced datasets.
+ X due to only evaluating balanced datasets.
P2 Starting Budget: They 3 different starting budgets each of their three datasets. CIFAR-IK 10:
0.1%. 1%, 10%; CIFAR-100 1%, 2%, 10%; Tiny ImageNet: 1%, 2%, 10% (% the whole dataset).
P3 Query Size: Each of the three different starting budgets has different query size resulting in overlapping experiments. Therefore it would be possible to draw some conclusions about the influence of the query size.
-()) due missing selective evaluation of query size.
P4 Performant Baselines: Their supervised random baseline is performing worse on CIFAR-10/100 than most models in the literature (see Appendix
X
P4 HP Optim. & Val. Split: They do not state optimizing their hyperparameters based on validation set.
P5 Self-St. They evaluate AL methods with and against one self-supervised training paradigm (SimSiam).
P5 Semi-Supervise Learning: They do not consider using models trained with semi-supervised training paradigms.
X
Gao et al. [20] Evaluate the performance of AL methods against and in the context of semi- supervised training paradigms. Further, they propose new query method designed for AL with models that are trained with Semi-SL training paradigm.
PI Data Distribution: They perform experiments on CIFAR-10/100 and ImageNet.
+ X due to only evaluating balanced datasets.
They perform a specific ablation about the importance of the starting budget CIFAR-10 10 with multiple settings and discuss it.
P2 Starting Budget:
P3 Query Size: In addition the standard experiments, they perform experiments with query sizes of 50 and 250. However, they do not specifically discuss its importance.
(V) due missing selective evaluation of query size.
18

---

P4 Performant Baselines: The performance of their supervised random baseline models in the main comparison is not close to our performance on CIFAR-10/100.
-
P4 HP Optim. Vol. Split: They do state optimizing their hyperparameters based validation set. +X
P5 Self-SL: They do not consider using models pre-trained with Self-SL.
+ X
P5 Semi-SL: They evaluate AL with and against the semi-supervised training paradigm (MixMatch).
Yi et al. [54) They propose to use self-super pre-text as basis for query functions.
PI Data Distribution: Perform experiments on CIFAR-10, an imbalanced version of CIFAR-10, Caltech- 101 and ImageNet.
P2 Starting Budget: One experiment is performed where they select the starting budget with their proposed Active Learning method on CIFAR-10. Otherwise, they do not evaluate the performance under different starting budgets.
P3 Query Size: They do not evaluate the performance with regard to different query sizes.
+ X
P4 Performant Baselines: Their supervised random baseline models are not close to the performance of our random baseline models on CIFAR-10 (see Appendix H)
+
P4 HP Optim, Val. Split: They do not state optimizing their hyperparameters based on a validation set.
+
P5 Self-St.: They consider several different Self-SI. paradigms ( ('Rotation Prediction', 'Colorization' "Solving jigsaw puzzles' and 'SimSiam') based on which they select rotation prediction for their experiments.
due to selection of non state-of-the-art Self-SL paradigm.
P5 Semi- Semi-Supervise Learning: They do not consider using models trained with semi-supervised training paradigms.
X
Krishnan et al. 35 two AL. methods.
They propose to use supervised contrastive training paradigm as a basis for
PI Data Distribution: Perform experiments on Fashion-MNIST, SVHN and CIFAR-10 and an imbalanced version of CIFAR-IO. 10.
due to imbalanced CIFAR- 10 being simulated.
P2 Starting Budget: They do not evaluate the performance under different starting budgets.
X
P3 Query Size: They do not evaluate the performance with regard to different query sizes.
+ X
19

---

P4 Performant Baselines: Their supervised random baseline models are not close to the performance of our random baseline models on CIFAR-10 (see Appendix HD.
X
P4 HP Optim. Vol. Split: They do state optimizing their hyperparameters based validation set.
P5 Self-SL: They do not consider using models pre-trained with Self-SL. + X
P5 Suni-Supervise Learning: They do not consider using models trained with semi-supervised training paradigms.
Kim et al. (30)
They propose task-aware active learning which is combination of learning loss active learning and variational adversarial active learning.
PI Data Distribution: Perform experiments on CIFAR-10 10. CIFAR- 100, CALTECH 101 and imbal- anced CIFARS.
P2 Starting Budget: They do not evaluate the performance under different starting budgets.
P3 Query Size: They do not evaluate the performance with regard to different query sizes.
P4 Performant Baselines: Their supervised random baseline models perform good but not on par with ours on CIFAR-10 and 100.
P4 HP Optim. Val. Split: They do not state optimizing their hyperparameters based on a validation set.
-X
P5 Self-SL: They do not consider using models pre-trained with Self-SL.
P5 Semi-SL: They do not consider using models trained with semi-supervised training paradigms.
X
Beck et al. 4 Evaluate several AL methods in different settings to gain an understanding which AL methods outperform random queries. Further, they provide the Al toolkit DISTIL.
Perform experiments on CIFAR-10, CIFAR-100, Fashion- Fashion-MNIST, SVHN and MNIST. +X due class imbalance.
PI Data Distribution:
P2 Starting Budget: They perform one experiment, where they evaluate a lower starting budget for MNIST.
due limited dataset.
P3 Query Size: They evaluate three different query sizes on CIFAR-10, but do so only for Random, Entropy and BADGE.
due to limited scope.
P4 Performant Baselines: Their supervised random baseline models perform good but no par with our random baseline models on CIFAR-10/100 (see Appendix H).
due to limited scope.
P4 HP Optim. & Val. Split: They do not state optimizing their hyperparameters based on validation set.
+ X
20

---

P5 Self-SL: They do not consider using models pre-trained with Self-SL.
X
P5 Semi-SI. They do not consider using models trained with semi-supervised training paradigms.
X
Zhan et al. (58)
Evaluate multitude of different AL methods and provide the AL toolkit DeepAL*.
PI Data Distribution: Perform experiments on Tiny ImageNet, CIFAR-10 (and CIFAR- 10 imbal- anced), CIFAR-10N 100,
EMNIST and SVHN. Further Experiments are performed on an Histopathological image Classification Task (BreakHis) and Chest X-Ray Pneumonia classification Iheemmnnia-NNIST well as the Waterbird dataset adopted from object recognition with correlated backgrounds.
P2 Starting Budget: They do not evaluate the performance under different starting budgets.
P3 Query Size: They evaluate multiple different query sizes on CIFAR-10 and analyze the difference.
P4 Performant Baselines: Their supervised random baseline models are close the performance of our random baseline models on CIFAR-10/100.
X
P4 HP Optim. Val. Split: They do not state optimizing their hyperparameters based on a validation set. +X
P5 Self-S:: They do not consider using models pre-trained with Self-SL.
P5 Semi-SL: They do not consider using models trained with semi-supervise training paradigms.
X
Chan et al. [11 Evaluate how AL methods interact with self- and semi-supervised training paradigms and how and whether they yield a benefit. The experiments in this paper differ from standard AL experiments by using only one query cycle, making hard compare systematically.
PI Data Distribution: Perform experiments on CIFAR-10 and CIFAR-100.
+X
P2 Starting Budget: Experiments are performed with a fixed starting budget of 3 samples per class or 2 samples per class in one case. This does not allow for evaluate of the influence of the starting budget on AL methods. +
P3 Query Size: They query all samples for their final performance in query step. This does allow for evaluation of the influence of the query size on AL methods.
XX
P4 Performant Baselines: Their supervised random baseline models perform on par with our random baseline models CIFAR-10/100.
P4 HP Optim. & Val. Split: They do not state optimizing their hyperparameters based on validation set. +X X
P5 Self-SL: They Debiased Contrastive Learning self-supervised pretext task.
P5 Semi-SL: They use Pseudo-labeling and FixMatch as semi-supervised training paradigms.
21

---

Table 1: Number Samples for each class in CIFAR- 10 LT dataset, Validation and sets are balanced.
Class
Train Split
airplane
4500
automobile (but not truck or pickup truck)
2913
bird
1886
cat
1221
deer
790
dog
512
frog
331
horse
214
ship truck (but no pickup truck)
139
90
C
Dataset details
Each dataset split into a training, a validation and a test split.
For CIFAR-IO/ 10/100 (LT) datasets the test split of size 10000 observations is already given and for MIO-TCD and 1SIC-2019 we use a custom test split of 25% random observations of the entire dataset size. For MIO-TCD and ISIC-2019 the train, validation and test splits are imbalanced. The validation split for all CIFAR- 10 and CIFAR- 100 datasets are 5000 randomly drawn observations corresponding 10% of the entire dataset. For CIFAR- 10 LT the validation split also consists of 5000 samples obtained from the dataset before the long-tail distribution is applied onto the training split. The CIFAR- 10 LT validation split is therefore balanced. For MIO-TCD and ISIC-2019 the validation splits consist of 15% of the entire dataset.
The shared training & pool dataset for CIFAR- 10/100 consists of 45000 observations. For CIFAR- 10 LT the training & pool datasets consist of 12,600 observations. For MIO-TCD and ISIC-2019 the training pool datasets consist of 60% the dataset.
C.1 Dataset descriptions
1. CIFAR-10: natural images containing 10 classes, label distribution is uniform Splits: (Train:45000; Val: 5000; Test; 10000)
Whole Dataset: 60000
2. CIFAR-100; natural images containing 100 classes, label distribution is uniform
Splits:
(Train:45000; Val: 5000; Test; 10000)
Whole Dataset: 60000
3. CIFAR-10 natural images containing 10 classes, label distribution of test and validation split is uniform, label distribution of train split is artifically altered with imbalance factor 50 according to [IO]. The resulting label distribution shown in Tab. T Splits:
p=
(Train:~12,60)) Val: 5000; Test; 10000)
Whole Dataset: 27600
4. ISIC-2019: dermoscopic images containing classes, label distribution of the dataset is imbalanced and shown in Tab,
Splits:
Srrain1520000 Val: 3799; Test; 6332)
Whole Dataset: 25331
5. MIO-TCD: natural images of traffic participants containing 11 classes, label distribution of the dataset is imbalanced and shown in Tab. 3 Splits:
(Train:31 1498; Val: 77875; Test; 129791)
Whole Dataset: 519164
22

---

Table 2: Number of Samples for each class in ISIC-2019
Class
Whole Dataset
Melanoma
4522
Melanocytic nevus
12875
Basal cell carcinoma
3323
Benign keratosis
867
Dermatofibroma
197
Vascular lesion
63
Squamos cell carcinoma
64
Table 3: Number of samples for each class in MIO-TCD
Class
Whole Dataset
Articulated Truck
10346
Background
16000
Bicycle
2284
Bus
10316
Car
260518 18
Motorcycle
1982
Non-motorized vehicle
1751
Pedestrian
6262
Pickup truck
50906
Single unit truck
5120
Work van
9679
D Experimental setup, in more detail
Here we detail the most crucial information for reprocubility, re-implementation and checking our implementation. When in doubt, trust the information documented here with regard to what wanted to do in our code.
D.1 Initial dataset setup
Before we do anything else the datasets are split according to Figure resulting in tain split, validation split and a test split. Each dataset has 3 different validation splits while always using the same test split. This is to ensure comparability across these splits without relying cross-vale tross-vaiddation The exact splits for each dataset are detailed in Appendix C After that the final datasets use for training and validation F) For all balanced datasets, we use class balanced label strategies since the label strategy only leads to different outcomes for imbalanced datasets. For CIFAR-10 LTT the train split only, whereas for MIO-TCD and ISIC-2019 we use the label strategy on both train and validation split. The amount of data which labeled for the final datasets of each split is then dependent upon the label-regime (described in more detail in Appendix D.2).
then labeled according to the 'Tabel strategy', which is described in
the label strategy
D.2 Label regimes
The exact label regimes are obtained by first taking the corresponding splits and then using the proper label strategy (see Figure 5) in combination with the starting budget and validation set size according
Tab. 4
D.3 Model architecture and training
each training step the model is trained from its initialization to avoid a 'mode collapse' Further we select the checkpoint with the best validation set performance in the spirit of [19 A ResNet-18 24 the backbone for all of our experiments with weight decay disabled on bias parameters. If not otherwise noted, nesterov momentum optimizer with momentum of 0.9 is used.
23

---

Dataset
Split
off
60% or Rest
15% or Val Split
25% or Test Split
Train Split
Validation Split
Test Split
Label Strategy
Label Strategy
Pool
Train
Val. Pool
Validation
Test
Training & Active Learning
Hyperparameters: & Performance
Performance Oracle 4
Labeled
Unlabeled
(Un)labeled
Can only be assessed in practice with great labelling effort
Figure Description of the three different data splits and their use-cases. The complete separation of a validation split allows to compare across label regimes and incorporate techniques for performance evaluation s.a. Active Testing
(34]. For evaluation and development the split should be as big
as possible since QM recommendations are based on the test set performance making it form of "oracle". An estimate of the size a dataset required have to measure specific performance differences can be derived using Hoeffding's 's inequality 2 45
24

---

Label Strategy
(Un)labeled Set
Create Labeled Set with N samples.
Consisting of 2 sets:
1. Class Balanced
2. Random
3. Rest
1. Draw B samples of each class
2. Draw randomly N-BC samples
Motivation:
Task is known.
All Classes are present
Labeled Set
Merge
Class
Follows p(x.y)
Unlabeled
Balanced
Labeled
(Unjlabeled
Unlabeled Set
Labeled Set
The Label Strategy used on the two roll-out datasets MIO-TCD and ISIC-2019 and for train and pool set CIFAR- 1- LT. For class balanced datasets this strategy does not induce meaningful changes to balanced starting budgets.
Figure 5:
Table 4: The exact values for all label regimes. Final Budget denotes the amount of labeled training samples at the end of the AL pipeline.
Dataset
CIFAR-10
CIFAR-100
CIFAR-IP L.T
MIO-TDD
ISIC-2019
Label Regime
Low
Mid
High
Low
High
Low
High
High
Low
Mid
High
Starting Budget
50
250
1000
500
1000
5000
50
250
1000
275
1100
40
200
800
Query Size
1100
Final Budget
500
2500
10000
5000
10000
25000
500
2500
10000
550
2750
11000
400
2000
8000
Validation Set Size
250
1250
5000
2500
250
5000
5500
200
800
3799
For Self-SL. models we use a two layer ML. as classification head to make better use of the Self-SL. representations with further details in Appendix D.5) To obtain bayesian models add dropout the final representations before the classification head with probability (p = 0.5) following [19]. For all experiments on imbalanced datasets, we use the weighted CE-Loss following 444 based on the implementation in SK-Learn [ if not otherwise noted. Models trained purely on the labeled dataset
upsample size of 5500 following
32 if the labeled train set smaller.
Bayesian Models All steps that require bayesian properties of the models including the prediction obtained by drawing 50 MC samples following 19 32
ST
ST models are trained for 200 epochs with Cosine Annealing and 10 epochs warmup.
Self-SI. pre-trained models are trained for 80 epochs with reduction of the learning rate with a factor of 10 every 20 epochs (MultiStepLR) and using a Mulit-Layer-Perpaah classification head (detailed description in Appendix D.5). The complete setup of the training for SimCLR is described in Appendix D.4
Self-SL
(MLP)
Semi-SL Semi-SL training is identical to the one proposed with the FixMatch method 122 except that we not use exponentially moving average models and restrict the training step from 1e6 to 2e5. The FixMatch implementation in experiments is based the open-source implementation
25

---

Table 5: HPs of the SimCLR pre-text training each dataset. HP for CIFAR datasets directly
taken 13
whereas MIO-TCD and ISIC-2019 HP are adapted from ImageNet experiments.
Dataset
MAR--OOOCFARROOCCFEEE LT
MIO-TCD
ISIC-2019
Epochs
1000
200
1000
Optimizer
LARS
LARS
Scheduler
Cosine Annealing
Cosine Annealing
Warmup Epochs
10
10
Temperature
0.5
0.1
Batch Size
512
256
Learning Rate
0.3
Weight Decay
IE-4
IE-6
Transform. Gauss Blur
False
True
Transform. Color Jitter
Strength=0.5
Strength=1.0
Table 6: MLP Head Ablation for Self-SL models on CIFAR-10, over all labeled training set small improvement for Multi-Layer-Pereee Peree---ror is measurable compared to Linear classification head models. Reported as mean (std).
Labeled Train Set
Classification Head
Accuracy (Val) %
Accuracy (Test) %
50
Linear
69.87(1.62)
69.90(2.18)
50
2 Layer MLP
71.47(3.06)
71.54(0.56)
500
Linear
84.67
83.51(0.45)
500
2 Layer MLP Linear
85.37(0.16)
84.60(0.37)
1000
87.13(0.69)
85.97(0.64)
1000
2 Layer MLP Linear
87.69(0.55)
86.57
5000
90.77(0.44)
90.20(0.21)
5000
2 Layer MLP
91.12(0.322
90.25
of and MixMatch for distribution alignment We always select the final Semi-SI. model of the training for testing and querying. On imbalanced datasets we change the supervised term to the weighted CE-Loss and use distribution alignment on every dataset except for CIFAR-IO (where does not improve performance (52). The HP sweep for our Semi-SL. models includes weight decay and learning rate.
Hyperparameters All information with regard to the final HPs and our proposed methodology of finding them is detailed in Appendix E
D.4 Self-supervised SimCLR pre-text training
Our implementation wraps the Choorch-Lightning-BBor implementation of SimCLR: https://iihtningg- swwpeee//eefsssperiiieeeeeee pnperrised.htmlffiimee. The training of our SimCLR models performed by excluding the validation splits. Therefore three models are trained on each dataset, one for each different validation split. In Tab. 5e give a list of the HPs used each of our five different datasets. All other HPs are taken from [3). Further, we not optimize the HPs for SimCLR at all, meaning that on MIO-TCD and ISIC-2019 Self-SL models could perform even better than reported here.
D.5 MLP head for self-supervised pretrained models
The MLP Head used for the Self-SL models has hidden layer of size 512 uses ReLU nonlinearities and BatchNorm. The results on CIFAR-10 based on which this design decision is based on is shown in Tab. 6
26

---

D.6 List of data transformations
Standard
The standard augmentations we use are based on the different datasets.
For CIFAR datasets these are in order of execution: Random/Horzzotallliin. RandomCrop to 32x32 with padding size
For MIO-TCD we use the standard ImageNet transformations: RandomRessizedCrpp to 224x224, Random Horizontal Flip.
For ISIC-2019 we use ISIC transformations which are: Resize to 300x300, Random/HorizontllFiii. Random Firp ColorJitter(0.02, 0.02, 0.02, 0.01) Random/oaatioonrruttioo---- 180), translate=(0.1, 0.1), scale=(0.7, 1.3)), RandomAffine(-180) 180), RandomCrop to 224x224.
These are based on the ISIC-2018 challenge best single model submission: Beatgg///SSSnssssiieeeeeiotttoo Pytorch
We use the same set of image transformations used in Randen 17 with the parameters N=2 and M=10. detailed list of image transformations alongside the corresponding values be seen in [52 (Table 12).
RandAAugmentMCC
The RandAuumentMCC transformations were used additionally after the corresponding standard trans- formations for each dataset. RandAugmemtMC(CIIAN also adds cutout as a final transformation.
AC weak Works identical RandAugmentMC and uses the same set of image transformations as for RandAugmentMCC but changed its parameters to N=1 and M-2. Therefore the maximal range of values is divided by a factor of
RandAugmentMC weak does not use cutout in difference to RandAugmentMC on CIFAR datasets.
D.7 Performance measure
As measure of performance CIFAR-10, CIFAR- 100 and CIFAR- 10 LT the accuracy while on MIO-TCD and ISIC-2019 we use balanced accuracy which is identical to mean recall shown in Equation
TP. CTP. +NN
Mean Recall
Where C' denotes the number of classes TP. is the number of true positives for class and FN, being the number of samples belonging to class being wrongly misclassified as another class.
D.8 Computational effort
Experiments were executed on a Cluster with access to multiple NVIDIA graphics cards. All ST and Self-SI. pre-trained experiments used a single Nvidia RTX 2080 (10.7GH video-ram) graphic cards except for the BADGE experiments on CIFAR-100 and MIO-TCD which required more video-ram using Nvidia Titan RTX (23.6GB video-ram). The Semi-SI. models on the CIFAR- 10/100 (LT) datasets used also single Nvidia RTX 2080 while on MIO-TCD and ISIC-2019 the Nvidia Titan RTX was utilized. For the results in our main table (excluding the HP optimization), the overall runtime was:
All ST experiments: 1800 GPU hours
All Self-SL pre-trained experiments: 1350 GPU hours
All Semi-SL. experiments 11200 GPU hours
E
Proposed hyperparameter optimization
Our proposed HP optimization for AL is based on the notion of minimizing HP selection effort by simplifying and reducing the search space. We use SGD Optimizer with Nesterov momentum of 0.9
6 Exclluding the pre-training
For only 2 label regimes and exluding MIO-TCD and ISIC-2019
27

---

Table 7: Final HPs for each dataset and label regime for our ST models based on HP tuning. HPs denoted with a are fixed across datasets and HP denoted with a + are pre-selected for each dataset while all other HP are obtained via sweeping.
Aegicn
Lave
Hi SeHtioomee O
Hil
rag.
Nearr a5 Colbe
feveio Cocar
Colar
CEGAN
CECA
Seriling
503
403
20
SEL
2E-1
Recdkograatie ICENAS
(CEAA)
ZaiAk VCEARO
Table 8: Final HPs for each dataset and label regime for our Self-SL models based on our HP tuning. Overall Performance remarkably stable with regard to HPss and stronger augmentations did not necessarily improve performance in the same way as for ST models. This is presumably due to the mre-trained representations. HP denoted with * are fixed across datasets and HP denoted with + are pre-selected for each dataset while all other HP are obtained via sweeping.
CRAA
Labet
Les
BACE
and select a number of epochs that always allow a complete fit of the model. The scheduler is also fixed across experiments as are the warmup epochs if used, Secondly, we pre-select the batchsize for each dataset since it is usually not critical HP as long it is big enough for BatchNorm to work properly.
ST For our ST models the final HP for each dataset and label regime are shown in Tab. 7 HP sweep: weight decay: (5E-3, 5E-4); learning rate: (0.1, 0.01); data transformation: (RandAug- mentMC, Standard)
Self-SL For our Self-SL pre-trained models the final HP for each dataset and label regime are shown in Tab. g
HP sweep: weight decay: (SB-3, 5E-4); learning rate: (0.01, 0.001); data transformation: (RandAug-
mentMC weak, Standard)
Semi-SL For our Semi-SL models follow (52) with regard to HP selection as closely possible. The final HP for each datasct and label regime are shown in Tab. g
HP sweep: weight decay and learning rate.
Table 9: Final HPs for each dataset and label regime for our Semi-SL models based on our HP tuning. HP denoted with * are fixed across datasets and HP denoted with + are pre- seleced for each dataset while all other HP are obtained via sweeping. denotes not performed experiments.
CIAR-IU
CBNMB-C0
MDO-CCD
BS0022119
Optesiaat en Skype"
Schokdy"
Coose Annaiing
Cuetec Ansaling
Costa, Aceiding
Counee Atacciigg
Losati Anacclle
Losp"
Cb-Loe
veighha CisLoo stadac
vecighal Clb viaadart
Sunping'
viadudd
Dirzbaaae Allpanas"
Pibs
Thec
The
The
Thes
Leartiu Ras
tut
D00
0.00
Worgh
Tura Aegneratis"
Sandud (CEFAR)
Standadd CCEAAS
Sexdadd/CTAAR Raddagprashic (CIPAR)
Saadud drummee RendirgreeMM lovaeNoo
Standied (1SC))
Uhindde Augressiee
RastovgronrMM (CEAM)
Rasdsapraetti (CTEAA)
28

---

F Detailed results
F.1
Main results
General observations: For all datasets, the overall performance of models was primarily deter- mined by the training strategy and the HP selection, with the benefits of AL being generally smaller compared to the proper selection of both. For the three toy datasets, Semi-SL generally performed best, followed by Self-SL and ST last, whereas, for the two real-world datasets, Semi-SL. showed no substantial improvement over ST in the first training stage and, therefore, further runs were omitted. Also, the absolute performance gains for Self-SL models with AL are generally smaller compared to ST models. For Semi-SL, there were generally only very small performance gains or substantially worse performance with AL observed. Concerning the effect of AL, the high-label regime proved to work for ST models on all datasets and Self-SI. models. On the two real-world datasets, MIO-TCD and ISIC-2019, a dip in performance at 7k samples for all ST models could be observed. This behavior is ablated in Appendix L.
Evaluation using the pair-wise penalty matrix: We use the pair-wise penalty matrix (PPM) to compare whether the performance of one query method significantly outperforms the others. It is essentially a measure of how often one method significantly outperforms another method based on a t-test with 0.05 (more info in [2] 4). This allows to aggregate results over different datasets and label regimes, with the disadvantage being that the absolute performance is not taken into consideration. When reading PPM, each row indicates the number of settings in which method i beats other methods, while column j indicates the number of settings in which method j is beaten by another method.
We show the PPMs aggregated over all datasets and label regimes for each training paradigm in F
For all methods, BADGE is the QM that is least often outperformed by other QMs. Further, for Self-SL models. is never significantly outperformed by Random, whereas is seldomly significantly outperformed for ST models. Based on this, we deem BADGE to be the best of our compared QMs for both ST and Self-SL. models. Since BADGE more often outperformed by Random (0.5) on the Semi-SL datasets and the additional high training cost for each iteration, we believe that Random is the better choice in many cases.
Evaluation using the area under the budget curve: For each of the following subsections, we added the area under the budget eurve (AUBC) for each dataset and label regime allow assessing the absolute performance each QM brings. Generally, higher values are better. For more information, we refer to [57 58).
The results the dataset for AUBC also show that BADGE is always one of the best performing AL methods. This is in line with the findings based on the PPM.
29

---

Pro Pre-Trained(15)
Standard Training(25)
INBG Seffirmay Rondon
40
1.75
B4L
85
00
93
30
BALD
05
00
24
10
20
25
Cor-Sae
00
35
00
12
Gan S4t
60
02
027
06
1.25
Gdnway
04
1.3
Intrapy
Extrope
1.00
-15
15
685
0.50
0.
0.5
(a)
(b)
Figure 6: PPMs aggregated over all experiments for Standard Models (a), Self-SI Pre-Trained Models (b) and Semi-SL models(c).
The value in the title (X) gives the highest possible value in cell and the lowest row is the mean value across a column j without row = j signaling how often on average on QM is outperformed by another.
F111 CIFAR-10
The AUBC values are shown in Tab. 10
Table 10: Area Under Budget Curve values for CIFAR-10.
Label Regime
Low-Label
Medium-Label
High-Label
Mean
STD
Mean
STD
Mean
STD
Training
Query Method
ST
BADGE
0.4730
0.0105
0.7289
0.0025
0.8599
0.0016
BALD
0.4744
0.0106
0.7253
0.0051
0.8578
0.0017
Entropy
0.4307
0.0018
0.6859
0.0042
0.8498
0.0025
Core-Set
0.4681
0.0038
0.7282
0.0043
0.8629
0.0017
Random
0.4720
0.0144
0.7309
0.0068
0.8526
0.0030
Self-SI.
BADGE
0.8282
0.0016
0.8728
0.0018
0.9086
0.0012
BALD
0.8005
0.0056
0.8692
0.0011
0.9093
0.0010
Entropy
0.8002
0.0090
0.8663
0.0017
0.9071
0.0010
Core-Set
0,8224
0.0026
0.8670
0.0009
0.9015
0.0009
Random
0.8117
0.0040
0.8669
0.0009
0.8989
0.0007
Semi-SL
BADGE
0.9349
0.0010
0.9488
0.0022
Entropy
0.9193
0.0082
0.9497
0.0007
Core-Set
0.9343
0.0018
0.9442
0.0007
Random
0.9326
0.0050
0.9478
0.0002
ST
Results are shown in Figure 7
Self-SL
Results are shown in Figure
Semi-SL
Results are shown in Figure
30

---

0.60
0.80
0.90
0.55
0.75
0.50
0.70
Accuracy
0.45
0.65
0.80
0.60
0.35
0.75
0.30
0.50
0.25
Labeled Samples
Labeled Samples
Labeled Samples
Query Methed
Core-Set
Random
Training
Standard Training
BALD
Entropy
Figure 7: CIFAR-10 ST
0.36
0.84
0.88
0.92
0.91
0.80
0.86
0.90
0.78
0.89
0.84 0.76
0.88
0.82
0.72
0.86
0.70
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
Entropy
Training
Self-SL Pre-Trained
Figure 8: CIFAR-10 Self-SI.
31

---

0.94
0.955
0.8
0.92
0.950
0.90
0.6
Accuracy
Accuracy
No Experiments Performed
0.88
0.4
0.86
0.84
0.935
0.0
2500
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
BADGE
Random
Training
Semi-5L
Core-Set
Entropy
Figure 9: CIFAR-10 Semi-SL
32

---

0.50
0.75
0.6
0.45
0.70
0.40
0.5
Accuracy
0.35
Accuracy
Accuracy
0.60
0.55
0.15
0.2
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
Random
Training
Standard Training
Entropy
Figure 10: CIFAR-100 ST
F11.2 CIFAR-100
The AUBC values are shown in Tab.
Table 11: Area Under Budget Curve values for CIFAR-100.
Label Regime
Low-Label
Medium-
High-Label
Mean
STD
Mean
STD
Mean
STD
Training
Query Method
ST
BADGE
0.3525
0.0042
0.4855
0.0044
0.6627
0.0017
BALD
0.3586
0.0007
0.4865
0.0019
0.6658
0,0002
Entropy
0.3036
0.0095
0.4440
0.0007
0.6569
0.0021
Core-Set
0.3458
0.0010
0.4767
0.0031
0.6560
0.0004
Random
0.3599
0.0027
0.4791
0.0044
0.6474
0.0015
Self-SL
BADGE
0.5397
0.0030
0,6020
0.0019
0.6858
0.0021
BALD
0,5028
0.0043
0.5754
0.0027
0.6784
0.0009
Entropy
0.5111
0.0066
0.5857
0.0028
0.6857
0.0032
Core-Set
0,5337
0.0044
0.5917
0.0032
0.6804
0.0013
Random
0.5365
0.0017
0.5970
0.0017
0.6757
0.0013
Semi-SL
BADGE
0.5562
0.0033
0.6222
0.0041
Entropy
0.5328
0.0158
0.6152
0.0038
Core-Set
0.5220
0.0101
0.6083
0.0061
Random
0.5713
0.0066
0.6307
0.0016
ST
Results are shown in Figure I0
Self-SL
Results are shown in Figure
Semi-SL Results are shown in Figure 12
33

---

0.60
0.65
0.72
0.55
0.60
0.70
Accuracy
0.50
Accuracy
Accuracy
0.68
0.55
0.64
0.40
0.62
0.60
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
Entropy
Training
Self-L Pre-trained
Figure 11: CIFAR- 100R-100RA Self-SL
0.65
0.70
0.60
0.55
0.6
Accuracy
Accuracy
Accuracy
No Experiments Performed
0.50
0.4
0.45
0.50
0.0
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
BADGE
Training
Semi-SL
Entropy
Figure 12: CIFAR-100 Semi-SP SLL
34

---

0.85
0.80
0.70
0.65
0.40
0.60
0.70
Accuracy
0.65
0.35
0.50
0.55
0.40
0.25
0.50
0.35
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
Random Entropy
Training
Standard Training
Figure 13: CIFAR- 10 LT ST
F1..3 CIFAR-10 LT
The AUBC values are shown in Tab. 12
Table 12: Area Under Budget Curve values for CIFAR-10 LT.
Label Regime
Low-Label
Medium-Label
High-Label
Mean
STD
Mean
STD
Mean
STD
Training
Query Method
ST
BADGE
0.3788
0.0152
0.5887
0.0098
0.7577
0.0004
BALD
0.3919
0.0111
0.5935
0.0141
0.7590
0.0034
Entropy
0.3740
0.0064
0.5793
0.0114
0.7454
0.0023
Core-Set
0.3939
0.0249
0.6162
0.0200
0.7667
0.0058
Random
0.3615
0.0118
0.5446
0.0214
0.7263
0.0044
Self-SL
BADGE
0.5373
0.0233
0.6501
0.0026
0.7704
0.0093
BALD
0,5431
0.0202
0.6549
0.0097
0.7742
0.0036
Entropy
0.5282
0.0225
0.6450
0.0090
0.7707
0.0061
Core-Set
0.5298
0.0182
0.6171
0.0154
0.7555
0.0032
Random
0.5397
0.0208
0.6173
0.0097
0.7554
0.0069
Semi-SL
BADGE
0.7233
0.0166
0.7616
0.0087
Entropy
0.6934
0.0289
0.7590
0.0101
Core-Set
0.6825
0.0103
0.7608
0.0108
Random
0.6965
0.0264
0.7363
0.0077
ST
Results are shown in Figure 13
Self-SL
Results are shown in Figure 14
Semi-SL
Results are shown in Figure 15
35

---

0.82
0.60
0.80
0.70
0.58
0.78
0.56
0.54
0.74
0.60
0.72
0.70
0.55
0.46
2000
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Random
BADGE
Training
Self-SL Pre-Trained
Entropy
Figure 14: CIFAR- 10 LT Self-SL
0.75
0.78
0.70
0.6
Accuracy
0.65
Accuracy
Accuracy
No Experiments Performed
0.74
0.4
0.60
0.72
0.55
0.70
0.50
0.0
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
BADGE
Training
Semi-SL
Entropy
Figure 15: CIFAR- 10 LT Semi-SL
36

---

0.425
0.55
0.50
0.70
Balanced Accuracy
Balanced Accuracy
Balanced Accuracy
0.375
0.65
0.350
0.45
0.60
0.325
0.55
0.40
0.275
0.35
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Core-Set
Random
Training
Standard Training
Entropy
Figure 16: MIO-TCD ST
F1.4 MIO-TCD
The AUBC values are shown in Tab. 13
Table 13: Area Under Budget Curve values for MIO-TCD.
Label Regime
Low-Label
Medium-Label
Higb-Label
Mean
STD
Mean
STD
Mean
STD
Training
Query Method
ST
BADGE
0.3539
0.0041
0.4688
0.0153
0.6614
0.0080
BALD
0.3254
0.0155
0.4830
0.0092
0.6884
0.0104
Entropy
0.3201
0.0097
0.4134
0.0230
0.6078
0.0176
Core-Set
0.3514
0.0134
0.4678
0.0181
0.7098
0.0056
Random
0.3510
0.0151
0.4564
0.0140
0.6065
0.0120
Self-SL
BADGE
0.5446
0,0122
0.6365
0.0054
0.7174
0.0040
BALD
0.4494
0,0102
0,5741
0.0138
0.6041
0.0092
Entropy
0.5105
0.0092
0.6416
0.0075
0.6972
0.0029
Core-Set
0.5060
0.0082
0.5900
0.0190
0.6699
0.0166
Random
0.5298
0.0109
0.6124
0.0032
0.6975
0.0054
ST
Results are shown in Figure 16
Self-SL
Results are shown in Figure 17
Semi-SL
We performed AL experiments due to the bad performance of Semi-SL on the starting
budgets. More information can be found in Appendix F.4
37

---

0.750
0.600
0.675
0.575
0.725
Balanced Accuracy
0.550
Balanced Accuracy
Balanced Accuracy
0.700
0.625
0.675
0.500
0.600
0.650
0.575
0.625
0.450
0.550
0.600
0.525
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Random
Entropy
Training
Self-SL Pre-Trained
BADGE
Figure 17: MIO-TCD Self-SL
F1.55
ISIC-2019
The AUBC values are shown Tab. 14
Table 14: Area Under Budget Curve values for ISIC-2019.
Label Regime
Low-Label
Medium-Label
Higb-Label
Mean
STD
Mean
STD
Mean
STD
Training
Query Method
ST
BADGE
0.3204
0.0099
0.4331
0.0146
0.5628
0.0101
BALD
0.3190
0.0133
0.4521
0.0211
0.5534
0.0052
Entropy
0.3241
0.0067
0.4207
0.0335
0.5631
0.0061
Core-Set
0.3426
0.0099
0.4501
0.0139
0.5708
0.0096
Random
0.3376
0.0243
0.4116
0.0201
0.5273
0.0048
Self-SL
BADGE
0.3809
0,0168
0.4679
0.0174
0.5761
0.0063
BALD
0,3949
0,0209
0.4847
0.0018
0.5914
0.0080
Entropy
0.3666
0.0165
0.4659
0.0104
0.5872
0.0096
Core-Set
0.3752
0.0205
0.4472
0.0071
0.5556
0.0069
Random
0.3736
0.0092
0.4555
0.0053
0.5547
0.0066
ST
Results are shown in Figure 18
Self-SL
Results are shown in Figure 19
Semi-SL We performed no AL experiments due to the bad performance of Semi-SL on the starting budgets. More information can be found in Appendix F.4
38

---

0.55
0.400
0.65
0.50
0.375
0.60
Accuracy
Galanced Accuracy
Balanced Accuracy
0.45
0.325
0.40
0.50
0.300
0.45
0.35
0.40
0.30
0.225
300
1500
2000
6000
8000
Labeled Samples
Labeled Samples
Labeled Semples
Query Method
BALD BADGE
Entropy Core-Set
Training
Standard Training
Random
Figure 18: ISIC-2019 ST
0.70
0.44
0.65
0.42
0.500
Belanced Accuracy
Accurscy
Balanced Accuracy
0.40
0.475
0.55
0.450
0.36
0.50
0.425
0.34
0.45
0.32
0.400
0.40
0.375
Labeled Samples
Labeled Samples
Labeled Samples
Query Method
Entropy
Random
Training
Self-SL Pre-Trained
Core-Se
BADGE
Figure 19: ISIC-2019 Self-SI. Pre-Trained Models
39

---

F.2 Low-Label Query Size
To investigate the effect query size in the low-label regime, we conduct an ablation with Self-SL pre-trained models on CIFAR-100 and ISIC-2019. For CIFAR-100 the query sizes are 50, 500 and 2000, while for ISIC-2019 they are 10, 40 and 160.
Here the accuracies for the overlapping labeled samples are shown which are analyzed using a t-test.
The results are shown in Tab. 15 and Tab. 16 When comparing the performance of the same QM using different query sizes only BALD and Core-Set lead to statistically significant difference in performance. While is consistent across both comparisons for BALD with the the performance difference widening for larger labeled sets and more iterations the trend for Core-Set is not clear.
CIFAR100
ISIC-2019 The results are shown in Tab. 17 and Tab. 18 Entropy is the only QM showing a significant difference, indicating that a query size of 40 outperforms query size of 10 for a training set size of 160. However, this behavior does not extend to the other training set sizes. Whereas, for BALD, there is consistent trend that smaller query sizes lead to increased performance.
Table 15: Accuracies % for the low-label comparison for CIFAR 100 with query sizes 50 and 500 at overlapping training set sizes. Reported as mean (std). Values with significant difference (I-test) across query sizes are denoted with *.
Labeled Samples
1000
1500
Query Size
50
500
50
500
BADGE
45.75 (0.75)
46.32 (0.20)
50.02 (0.92)
50.24 (0.61)
BALD
45.54 (0.20)
42.31 (1.71)
49.41 (0.39)*
46.00 (0.44)"
Core-Set
46.53 (0.76)
46.02 (0.74)
49.34 (0.72)
49.70 (0.60)
Entropy
43.82 (0.40)
42.95 (1.32)
47.05 (0.91)
46.75 (0.76)
Random
46.10 (0.41)
45.22 (0.34)
49.88 (0.26)
49.79 (0.27)
Table 16: Accuracies % for the low-label comparison for CIFARI00 with query sizes 500 and 2000 overlapping training set sizes. Reported mean (std). Values with significant difference (t-test) across query sizes are denoted with *.
Labeled Samples
2500
4500
Query Size
500
2000
500
2000
BADGE
54.62 (0.60)
55.03 (0.38)
50.92 (0.12)
59.98 (0.60)
BALD
50.92 (0.30)
49.96 (1,43)
55.86 (0.21)*
52.14 (1.36)*
Core-Set
54.72 (0.16)*
53.52 (0.33)*
58.71 (0.61)
58.28 (0.43)
Entropy
51.41 (0.53)
51.79 (0.76)
57.12 (0.53)
57.53 (0.43)
Random
54.71 (0.38)
54.38 (0.21)
59.98 (0.60)
59.48 (0.48)
Table 17: Accuracies % for the low-label comparison for ISIC-2019 with query sizes 10 and 40 at overlapping training set sizes. Reported as mean (std). Values with a significant difference (t-test) across query sizes are denoted with *.
Sumple
Query
(1.76)
(2.24)
Emnopy
(2.15)
(1.46)
40

---

Table 18: Accuracies % for the low-label comparison for ISIC-2019 with query sizes and 160 overlapping training set sizes. Reported as mean (std). Values with significant difference (1-test) across query sizes are denoted with *.
Labeled Samples
200
360
Query Size
40
160
40
160
BADGE
37.52 (1.16)
38.11 (1.07)
42.90 (3.29)
42.86 (1.44)
BALD
40.09 (2.59)
39.67 (2.54)
44.49 (1.78)
42.83 (1.05)
Core-Set
37.79 (2.93)
37.71 (3.18)
39.56 (1.48)
39.47 (1.04)
Entropy
39.93 (2.38)
37.80 (1.46)
42.03 (1.16)
40.82 (1.46)
Random
40.29 (1.78)
40.80 (2.34)
42.10 (2.29)
42.10 (2.29)
41

---

02
ISIC-2019
MIO-TCD
1000
2000
6EX
Libeted
Libale
Liband A0V330
Query Entropy
Training Strategy:
Training.
Figure 20: Maero-averaged Fl-scores for ISIC-2019 and MIO-TCD datasets. Across the board, BADGE is still the best- best-performing QM which can also be seen in Table [9 Please interpret the results with care, the model configurations optimized for balanced accuracy.
Table 19: Area Under Budget Curve values based on macro-averaged F1 P1-scores for ISIC-2019 and MIO-TCD (corresponding plots Figure 20
(a) ISIC-2019
(b) MIO-TCD
Lakel
Lowlale
MilLuee
HighsLalel
Regine
Low
Mil
High-Lale
Moun
STD
Mias
STD
Moin
STD
Moas
STD
Moen
STD
Mean
STD
Trann
Quary Methed
Traning
Vunry Merhod
Sundied
BAIKE:
0:3191
0.931
0.0067
111299
0J931
Standard
BADGI
0.25015
0.3111
5.3804
0113
01579
B,005
BAI
0.7987
0.0:45
(3415))
0.116
D-156
C.0.00
BALE
0-23775
0.3137
03335
01907
01574
B0116
Panops
017261
Frrmpy
0,009
Rendien
028117
0.3634
0.009
04774
6/035
Bazsken
0.2445
0.5091
0.3455
0/132
0.505
03065
SEL:SL Pic Traned
BAIICE
03300
00001
03395
0.003
133300
000ST
SSIT SL. Tranno)
BXDGE
014205
002000
23337
00020
0391
03055
60039
0.0111
0.008
Circ-Set
0.3010
000.25
0.383
0.004
0.545
600133
0.3525
0.2066
04970
00060
0.5575
03138
Randivn
Rondom
044755
F33
Macro averaged Fl-scores
Additionally, we provide the macro-averaged F1-scores for 1SIC-2019 and MIO-TCD dataset. A plot showing the performance of both ST and Self-SL models is shown Figure 20 and the resulting AUBC values are shown in Tab. 19
F4 Semi-Supervised Learning
Results of FixMatch for all HPs on the whole validation splits are shown separately for MIO-TCD in Tab. 20 and ISIC-2019 in Tab. 21 Based on the performance which did not improve substantially over even ST models we decided to omit all further AL experiments.
Table 20: MIO-TCD FixMatch results reported on the test sets (balanced accuracy in %). Reported as mean (std).
FixMatch Sweep MIO-TCD
Labeled Train Samples
Learning Rate
Weight Decay
Balanced Accuracy (Test)
5E-3
18. (8..))
0.3
5E-4
28.0(0.4)
55
5E-3
26.8(0.5)
0.03
5E-4
29.7(2.4)
5E-3
20.2(6.2)
0.3
5E-4
28.2(1.5)
275
5E-3
31.4(0.7)
0.03
5E-4
36.0(1.)
42

---

Table 21: ISIC-2019 FixMatch results reported on the test sets (balanced accuracy in %). Reported as
mean (std).
FixMatch Sweep ISIC-2019
Labeled Train Samples
Learning Rate
Weight Decay
Balanced Accuracy (Test)
5E-3
14.0(2.5)
0.3
5E-4
26.9(1.5)
40
5E-3
24.5(4.4)
0.03
5E-4
31.3(1.7)
5E-3
15.2(3.)
0.3
5E4
19.1(1.5)
200
5E-3
26.3(0.8)
0.03
5E-4
24.3(3.6)
G Discussion and further observations
The results interpreted based the assumption that QM performing similar level as Random is not drawback as long as it brings other settings performance improvements over random queries. This mostly follows in line with the PPM a performance metric but mostly focuses on the row that compares each QM with random queries. However, if a QM shows behavior leading much worse behavior than random as Entropy does shows signs the cold start problem, deem this as highly problematic. In these settings, one loses significant performance whilst paying cost computing and setup corresponding AL. Therefore, we use random queries as baseline for all QMs.
Based on this our recommendation for BADGE given for Self-SL and ST trainings. The main disadvantage of this approach is that absolute performance difference are not captured in this aggregated format.
43

---

H
Comparing random-sampling baselines across studies
Here we compare the performance of random-samee sampling baselines on the most commonly utilized dataset CIFAR-10 and CIFAR-100 across different studies for ST, Self-SL and Semi-SL models along strategic point where overlap in between papers occurs. For CIFAR-I0 the results of this comparison are shown for the high-label regime in Tab. 22 and the low- and mid-label regime in Tab. 23 Similarly CIFAR-I00 the results are shown in Tab. 24 for high-label regime and Tab. 25 for the low- and mid-label regimes. Overall our ST random baselines outperform all other random baselines. Our Self-SL. models also outperform the only other relevant literature (6] on CIFAR-10 10, Further, our Semi-SL models also outperform the relevant literature 220 43
on CIFAR-10 10 and CIFAR-100.
Table 22: Comparison of random baseline model accuracy in % the test set for the high label- regime for CIFAR-I0 across different papers. Best performing models for each training strategy are highlighted. Values denoted with represent performed experiments. Values with denoted with a are reprinted from 44]. error.
Values which are sourced from graph are subject to human read-out
Information
Number Labeled Training Samples
Paper
Training
Model
Source
1k
2k
5k
10k
15k
20k
QBC
ST
DenseNet121
Graph
74
82.5
VAAL
ST
VGG16
Graph
61.35*
68.17*
72.96*
75.99*
CoreSet
ST
VGG16
Graph
60
68*
71
74
Agarwal et al.
ST
VGG16
Graph
61.5
68
72
76
Munjal-SR
ST
VGG16
Table
82,16
85.07
89.43
91.16
Mittal et al.
ST
WRN28-2
Graph
82.5
90.7
LLAL
ST
ResNet18
Graph
51
81
87
CoreCGN
ST
ResNet18
Graph
50
64
80
85.5
TA-VAAL
ST
ResNet18
Graph
50
65
81
87.5
Krishnan al.
ST
ResNet18
Graph
47
60
78
86
Yi al.
ST
ResNet18
Graph
47.5
56
78
86
Bengar al.
ST
ResNet18
Graph
85
Beck et al.
ST
ResNet18
Graph
55
85
90.5
Zhan et al.
ST
ResNet18
Graph
45
76
Munjal-SR
ST
ResNet18
Table
84.69
88.45
89.98
92.29
Ours
ST
ResNet18
Table
72.4
79.8
85.5
90.5
Bengar al.
Self-LL
ResNet18
Graph
87
88
89.5
90.5.
91
91.5
Ours
Self-SL
ResNet18
Table
86.2
88.3
90.1
91.4
Mittal et al.
Semi-SL
WRN28-2
Graph
88
91
92.5
93.8
94
94.5
Gao et al.
Semi-SL
WRN28-2
Graph
91.5
91
Ours
Semi-SI.
ResNet18
Table
94.7
95.0
44

---

Table 23: Comparison of random baseline model accuracy in % on the test set for the low- and mid-label regime for CIFAR-10 10 across different papers. Best performing models for each training strategy are highlighted. Values denoted with represent not performed experiments. Values which sourced from graph subject human read-out emror.
Information
Number Labeled Training Samples
Paper
Training
Model
Source 50
100
200
250
500
Chan et al.
ST
WRN28-2
Table
40.9
Mittal al.
ST
WRN28-2
Graph
36
48
Bengar et al.
ST
ResNet18
Graph
38
Ours
ST
ResNet18
Table
25.1
32.3
44.4
47.0
61.2
Chan et al.
Self-SL
WRN28-2
Table
76.7
Bengar et al.
Self-SL
ResNet18
Graph
62
77
81
83
85
Ours
Self-SL
ResNet18
Table
71.3
76.8
81.2
81.4
84.1
Chan et al.
Semi-SL
WRN28-2
Table
93.1
Mittal al.
Semi-SL
WRN28-2
Graph
82
85
Gao et al.
Semi-SL
WRN28-2
Table
47.9
89.2
90.2
Ours
Semi-SL
ResNet18
Graph
90
91
93
93
94
Table 24: Comparison of random baseline model accuracy in % the for the high-label regime for CIFAR-100 across different papers. Best performing models for each training strategy are highlighted. Values denoted with represent performed experiments. Values which sourced from a graph are subject to human read-out error.
Information Paper
Number Labeled Training Samples
Training
Model
Source
5k
10k
15k
20k
Agarwal et al.
ST
VGG16
Graph
35
41.5
46
Agarwal et al.
ST
ResNet18
Graph
29.5
38
45
49
Core-Set
ST
VGG16
Graph
27
37
42
49
VAAL
ST
VGG16
Graph 28
35
42
46
Munjal et al.
ST
VGG16
Graph
39.44
49
55
59
VAAL
ST
ResNet18
Graph 28
38
45
49
TA-VAAL
ST
ResNet18
Graph 43
52
60
63.5
Bengar et al.
ST
ResNet18
Graph
27
45
52
58
Beck et al.
ST
ResNet18
Graph 40
53
60
64
Zhan al.
ST
ResNet18
Graph
39
Munjal et al.
ST
ResNet18
Table
?
61.1
66.9
69.8
Mittal al.
ST
WRN28-2
Graph
44.9
58
64
68
Ours
ST
ResNet18
Table
49.2
61.3
66.7
70.2
Bengar et
Self-SI.
ResNet18
Table
60
63
63.5
64
Ours
Self-SI.
ResNet18
Table
60.4
64.8
68.4
70.7
Mittal et al.
Semi-SL
WRN28-2
Graph
59
65
70
71
Gao et al.
Semi-SL
WRN28-2
Table
63.4
67
68
70
Ours
Semi-SL
ResNet18
Graph
63.5
68.5
45

---

Table 25: Comparison of random baseline model accuracy in % on the test set for the low- and mid- label regime for CIFAR-I00 across different papers. Best performing models for each training strategy are highlighted. Values denoted with represent not performed experiments. Values which sourced from graph subject human read-out error.
Information
Number Labeled Training Samples
Paper
Training
Model
Source
500
1000
2000
2500
Chan et al.
ST
WRN28-2
Table
33.2
Mittal et al.
ST
WRN28-2
Graph
9
27
TA-VAAL
ST
ResNet18
Graph
20
Bengar et al.
ST
ResNet18
Graph
9
12
17
Ours
ST
ResNet18
Table
14.0
22.4
32.0
36.3
Chan et al.
Self-SL
WRN28-2
Table
49.1
Bengar et al.
Self-SL
ResNet18
Table
47
50
56
Ours
Self-SL
ResNet18
Table
37.3
45.2
52.2
54.7
Chan al.
Semi-SL.
WRN28-2
Table
67.6
Mittal et al.
Semi-SI.
WRN28-2
Graph
26
35.5
44.5
49
Ours
Semi-SL
ResNet18
Graph
41
56.5
46

---

I Detailed limitations
Additionally to the limitations already discussed Sec. we would like to critically reflect on the following points:
Query methods We only evaluate four different QMs which is only a small sub-sclection of all the QMs proposed in the literature. We argue that this may not be optimal, however, deem it justified due to the variety of other factors which we evaluated. Further, we excluded all QMs which induce changes in the classifier (s.a. LLAL (55)) or add a substantial additional computational cost by training new components (s.a. VAAL (1I). These QMs might induce changes in the HPs for every dataset and were therefore deemed too costly to properly optimize.
We leave a combination of P4 with these QMs for future research.
Validation set size The potential shortcomings of our validation set were already discussed. However, we would like to point out that a principled inclusion of K-Fold Cross- Validation into AL might alleviate this problem. This would also give direct access to ensembles which have been shown numerous times to be beneficial with regard to final performance (also in AL) [ How this would allow us to assess performance gains in practice and also make use of improved techniques for performance evaluation s.a. Active Testing [34] in the same way as our proposed solution shown in Figure i is not clear to us. Therefore leave this point up for future research.
Performance of ST models On the imbalanced datasets, the performance of our models is not steadily increasing for more samples which can be traced back to sub-optimal HP selection according to 14 We believe that our approach of simplified HP tuning improves over the state-of-the- thetthr in AL showcased by the superior performance of our models on CIFAR-10 and CIFAR-100. However, regularly re-optimizing HPs might be an alternative solution.
Performance of Self-SL models Our Self-SL models are outperformed on the low-label regime on CIFAR-100 by the Self-SL models by (6, whereas on the medium- and high-label regime our Self-SL models outperform them. We believe that this might be due to our fine-tuning schedule and the possibility that Sim-Siam improves over SimCLR on CIFAR-100. Since our Self-SI models still outperform most Semi-SL models in the literature we believe that drawing conclusions from our results is still feasible. An interesting research direction would be to make better use of the Self-SI. representations s.a. improved fine-tuning regimes (37.
No Bayesian Query Methods for Semi-SL The Semi-SL models were neither combined with BALD nor BatchBALD as query functions, even though we showed that small query sizes and BatchBALD can counteract the cold-start problem. Further our Semi-SL models had bigger query sizes by factor of three, possibly additionally hindering performance gains obtainable with AL. However, in previous experiments with FixMatch, we were not able to combine with Dropout whilst keeping the performance of models without dropout. This clearly might have been an oversight by us, but we would like to point out that in the works focusing on AL, using Semi-SL without bayesian QMs is common practice 20
Changing both starting budget and query size We correlated the two parameters (smaller query size for smaller starting budget etc.) since 1) in practice, we deem the size of the starting budget to be dependent on labeling cost (therefore, large query sizes for small starting budgets are unrealistic and vice versa) and 2) In this work, we are especially interested in smaller starting budgets ("cold-start" territory) compared to the ones in the literature, since AL typically shows robust performance for larger starting budgets. Theory shows that our adapted smaller query size for this case can only positively affect the result 19 32). The only possible confounder could be that we interpret the performance of small starting budget too positively due to a hidden effect of the smaller query size. However, we performed the labelabel query size ablation, showcasing that varying the query size for small starting budgets did not have considerable effects on performance for all QMs, except BALD, where, clear performance increase for smaller query sizes was observed.
L.1 Instability of hyperparameters for class imbalanced datasets
The substantial dip in performance on MIO-TCD and ISIC-2019 for approx 7k samples shown in Figure[6]andd Figure 18 is ablated in Tab. 26 where we show that simply changing the learning rate leads to stabilizing the performance on both datasets for these cases.
47

---

Table 26: Ablation study on the performance-dip MIO-TCD and ISIC-2019 for ST models with regard to HP. Reported as mean (std).
Datasel
Labeled Train
Data Augmentation
Learning Rate
Weight Decay
Balanced Accurscy (Vil)
Balinced Accuracy (Test
ISIC-2019
7200
0.01
0.1
SE-4
5E:3
SE-3
58.001.7)
57.2(1.7)
54.411.2
55.4(0.9)
54.5(2.1)
55.6(1.0)
$2.611.9)
MIO-TCD
7700
RandAngmenMIC (ImageNee))
0.01
0.1
SE.3
SE-4
56-4
(3.7(18)
64. ((1.))
63 63.800.7)
65.922.4)
62.2(1.1)
(3.62277
62.9(1.))
11.11
Balanced Sampling
Weighted CE-Loss
CIFAR-1
1IFAR-10
180
309
40
LOe 1500
2009
2500
2000 ANCO 1000
000 1000
300
200
200
1509
200
2000 HOOR
3000
0.2
2019
19IC-2019 ML
200
2000
2000 A000
4000
0.8
HL
MIO-TCD ML
HL
200
400
2000
2100 5000
1500
10000
2008
Query
Entropy
Training Strategy:
Training.
Figure 21: Comparison between balanced sampling and weighted cross-entropy-loss (weighted CE-Loss). Whereas the ST models overall seem to benefit more from balanced sampling, the Self-SL models perform slightly better for weighted CE-Loss. Generally the observed performance gains in the imbalanced settings are still present.
However, this dip in performance also arises using weighted Cross-Entropy (weighted CE-Loss) as loss function as shown in the following ablation Figure 21
48


Reviews:
[{"review_id": "z3lJQITq5P", "reviewer": "Reviewer_QpNG", "summary": "The paper identifies the pitfalls in current evaluation frameworks for AL, and proposed solutions that lead to a new evaluation framework (codebase) to tackle these problems. Empirical results reveal insights on the current state of AL methods.", "questions": "See above.", "limitations": "The paper discusses its limitation in its experimental setup.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper aims to tackle a very timely and important topic on how to more systematically and realistically evaluate AL methods.\n- The paper identifies several pitfalls in current AL evaluation and proposed solutions to overcome them.\n- The paper presents quite thorough experimental results accompanied by code releasing for future work to build on.\n- The key takeaways provide insights on the current state of AL literature.", "weaknesses": "- Line 71 AL validation paradox: It is not very clear to me why AL parameters cannot be adapted to the dataset at hand. Why can't one adapt or even change query method in an online fashion like in [1]? I agree that one should not \"use the performance\" in a hindsight to choose the best AL method but that doesn't imply that we are only allowed to use fixed \"pre-configured AL settings\" for all new dataset?\n- While AL validation paradox appears to be the most important pitfall in AL evaluation, it's not immediately clear to me as how P1-5 addresses the validation paradox. Can the authors discuss the connection between AL paradox to P1-5?\n- While the paper has emphasized the importance of testing AL methods on new datasets and on realistic settings, the datasets included in the current experiments are limited in number, modality (only images) and relatively less realistic (e.g., CIFAR). Results/takeaways can be more convincing if more realistic datasets are used, and even include modalities other than image.\n\n[1] Active Learning by Learning. Hsu et al. 2015. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to carefully review our paper as well as for your insightful comments and questions. Note that the major changes made in our manuscript are detailed in the general reply above. \n\n-> “Line 71 AL validation paradox: It is not very clear to me why AL parameters cannot be adapted to the dataset at hand. Why can't one adapt or even change query method in an online fashion like in [a]? I agree that one should not \"use the performance\" in a hindsight to choose the best AL method but that doesn't imply that we are only allowed to use fixed \"pre-configured AL settings\" for all new dataset?”\n- We hope the updated description of the validation paradox, the pitfalls and the roll-out datasets (see general reply above) answer this question.\nThe bottom line is: The validation paradox does indeed force us to, on a new unlabelled dataset, choose the best query method (QM) based on prior knowledge (note that this is in stark contrast to standard ML, where the optimal configuration on a new dataset can be found using a validation split). This is why the avoidance of P1-P5, and thus the thorough validation proposed in this work, is crucial to enable an informed choice in a new setting. \nAs for your suggestion to use a live-adapting QM [a], this is ofcourse possible, but the reliability of this live-adaptation needs to be validated equally beforehand under different settings. As such, the adaptation rules itself can be seen as a “preconfigured AL setting” whose reliability needs to be checked before applied on arbitrary tasks.\n\n-> “While AL validation paradox appears to be the most important pitfall in AL evaluation, it's not immediately clear to me as how P1-5 addresses the validation paradox. Can the authors discuss the connection between AL paradox to P1-5?”\n- Thank you for pointing out the lack of clarity regarding these main concepts in our manuscript. We thoroughly revised the description (see general reply) and hope the concepts are clearly conveyed in the updated text.\n\n-> “While the paper has emphasized the importance of testing AL methods on new datasets and on realistic settings, the datasets included in the current experiments are limited in number, modality (only images) and relatively less realistic (e.g., CIFAR). Results/takeaways can be more convincing if more realistic datasets are used, and even include modalities other than image.”\n- We argue that the number and diversity of datasets and settings in this study do indeed provide sufficient empirical evidence for the insights we provide. For instance, both ISIC-2019 and MIO-TCD are datasets that are commonly argued to represent real-world use-cases and should not be lumped together with simple datasets like CIFAR. To our knowledge, our work is one of the most extensive AL studies for deep learning based image classification. Other reviewers have commented on our experiments as “extensive” and “thorough”. The number, complexity, and diversity of our utilized datasets and settings (label-regimes, training strategies) exceeds by far the standard in the field. Note that adding more datasets is computationally more complex for AL compared to standard ML tasks.\nRegarding the suggestion to add more modalities, we clarified in the manuscript that the scope of our work is deep active classification of images.\n\n\nThank you again for your valuable feedback. As we believe in having resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending  a stronger acceptance of our work.\n\n*Additional references:*\n\n[a] Active Learning by Learning. Hsu et al. 2015."}, {"review_id": "OjdmuymEjH", "reviewer": "Reviewer_GZfx", "summary": "The authors in this work try to address the common issues as to why active learning has not been in use compared to other methods like semi-supervised or self-supervised learning.  The authors argue that the active learning methods are not evaluated in robust manner and in a cross-task generalisation setting which leads to lower performance compared to semi-supervised and supervised learning. Hence, they also provide an evaluation framework to overcome these issues. The code for all their experiments is also made available.\n\nThis work is useful to anyone who is new to the field of active learning and are deciding whether or not active learning would be useful for their task and dataset.\n\nThe authors list five common pitfalls in the evaluation of active learning settings and suggest means to address these issues. Some of these issues are lack of evaluated data distribution settings, lack of evaluated starting budget, lack of evaluated query sizes, neglection of the classifier configuration and neglection of alternative training paradigms.\n\nSome conclusions of this work, like active learning could provide substantial gains in class-imbalanced settings, reiterating that BADGE is the most robust QM etc can help the readers intending to use AL in their works.", "questions": "- Can you clarify the relation between the starting budget and the labelling regimes(high-label,low-label etc)\n- Model selection for ST and Self-SL models is based on the best validation set epoch, while for Semi-SL models the final checkpoint is used. - Why?\n- Do the cost for the checks of the generalizability of AL in real-world settings itself negate the purpose of AL?", "limitations": "The authors have clearly mentioned the limitations of their studies. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors conduct a thorough ablation study to evaluate the active learning gains by changing the parameters of only one of these issues and keeping the rest constant. This has helped in filling the research gap between theoretical and practical works to use smaller query sizes as a potential solution for the instabilities of certain QMs like BALD. It has also helped in recalibrating the potential gains from active learning measured by previous works which could be attributed to hyperparameter settings. By introducing a practical way to obtain the best hyperparamter settings, this work shows the actual gains of active learning without overstating it. ", "weaknesses": "Since it was mentioned in the limitation that the training may not have been optimal for semi-SL method, how can the conclusion that \"semi-SL performs best on toy datasets but does not seem to generalize to more complex settings\" be made?\n\nPresentation - \n- Some lines contain too many pieces of information in a single line which can be organised in a more concise and clearer manner\n- A breif description of the citation would help in following the work better. For ex, writing a few words on what [6] is in line 313 would make the presentation better\n- The abbreviation ST was used in line 186 before clarifying the meaning in line 235 which can be confusing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to carefully review our paper as well as for your insightful comments and questions. Note that the major changes made in our manuscript are detailed in the general reply above. \n\n-> “Since it was mentioned in the limitation that the training may not have been optimal for semi-SL method, how can the conclusion that \"semi-SL performs best on toy datasets but does not seem to generalize to more complex settings\" be made?””\n- Thank you for pointing this out. We agree that based on our setting, this original conclusion about Semi-SL methods was too broad. As detailed in the general reply, we toned down our statement and put it into the context of similar reports in related work.\n\n-> Issues with Presentation:\n- We thank you very much for pointing out these flaws and revised our work thoroughly to resolve them. Most notably:\n    - We decreased the information density in our manuscript by pushing less-relevant information to the Appendix. For instance, we removed section 2.3 to a new Section in the Appendix and instead provide a more intuitive and crisp explanation of the importance of our five pitfalls and other important concepts as described in the general reply.\n    - All references are given adequate descriptions to improve readability.\n    - As suggested, we introduce the abbreviation Standard Training (ST) in line 186 now.\n\n-> “Can you clarify the relation between the starting budget and the labelling regimes(high-label,low-label etc)”\n- We define label regimes as specific settings for the sizes of starting budget and query size. The starting budgets for the low-, mid- and high-label regime vary based on the number of classes (5x, 25x, 100x #Classes) with the exception of CIFAR-100 (5x, 10x, 50x #Classes) due to the different ratio of #Classes to the dataset size. The query size is for each label regime identical to the starting budget (assuming both sizes derive from the labeling cost of a given problem). By running AL experiments on these different settings (label regimes) we wanted to test how well QMs generalize given different sizes of starting budgets. We were especially interested in the low- and mid-label regimes to see how well QMs perform with starting budgets which are usually attributed to be prone to the cold-start problem but one might unknowingly run into unto application. On the entanglement of starting budget and query size: The theory shows that our adapted smaller query size for this case can only positively affect the result compared to a larger query size [27]. Therefore, we could interpret the performance of a small starting budget too positively due to a hidden effect of the smaller query size. However, we performed an ablation study in Figure 3, showcasing that varying the query size for small starting budgets did not have considerable effects on performance for all QMs, except BALD, where expectedly, a clear performance increase for smaller query sizes was observed. We thank the reviewer for pointing out the vagueness and clarified the corresponding Section 3 in the manuscript.\n\n-> “Model selection for ST and Self-SL models is based on the best validation set epoch, while for Semi-SL models the final checkpoint is used. - Why?”\n- For ST it is considered good practice to select the best checkpoint [39] and it is intuitive to follow the same approach for Self-SL (finetuning). However, for Semi-SL, the current practice is to take the last checkpoint since it is assumed that the performance generally improves over the span of the long training with little risk of overfitting thanks to including the unlabeled data pool in the training [46], whereas both ST and Self-SL (finetuning) are prone to overfitting in this setting.\nWe argue it is important to follow the standard practice in these choices to ensure comparability of our results as well as adequate settings for all training strategies.\n\n-> “Do the cost for the checks of the generalizability of AL in real-world settings itself negate the purpose of AL?”\n- Thank you for pointing out this trade-off. \nOur message is: if done right, users can save substantial costs with AL. While our protocol increases computational efforts for the initial validation of a newly proposed QM, this effort has to be made only once and does not come with additional labeling cost. However, if validation of QMs is done with our protocol, it enables reliable performance estimates for QMs in diverse new settings and applications, such that users can make high-quality decisions on when and how to apply AL on their customized problems. Thus, our protocol enables users to benefit from AL upon application reliably. We added a respective sentence to the discussion in our manuscript.\n\nThank you again for your valuable feedback. As we believe to have resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending acceptance of our work.\n\nWe use the same citation keys as in our manuscript."}, {"review_id": "6EdcniJmeV", "reviewer": "Reviewer_WUPX", "summary": "Active Learning is a crucial problem that focuses on selecting a subset of examples from an unlabeled dataset to be labeled. Primarily community focus on two pillars for defining acquisition functions -- uncertainty and diversity based acquisition methods. With emergence of the highly powerful pretrained models and alternative paradigm such as self SL and semi SL, the usecase of when to use AL is getting questionable. That being said, this work begins with listing down pillars at which AL scheme should be evaluated, starting from the hyperparameters to proper comparison against SSL baselines. Paper list down total 5 \"pitfalls\" and show how existing community has not been evaluating the proposed AL methods correctly. Paper then extensively evaluate the common AL QMs on different settings, and then giving concluding remarks on some of the existing methods, and guidance on when is AL beneficial. ", "questions": "Please refer to weakness section.", "limitations": "Please refer to weakness section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well written, easy to follow and targets a well motivated problem. Previous AL works often become cynical, and when not carefully studied the mentioned pitfall, it can give a false sense of label efficacy. Conducted experiments are solid, and the community would benefit with it.  ", "weaknesses": "Weakness are more of some additional experimental suggestions. Given the abundance of pretrained CLIP models, I think adding them would also have been a great idea. Secondly, in terms of difficulty, paper could've added some large scale datasets such as Imagenet variants. Lastly, one additional pillar could also be studying the OOD robustness. This was recently studied in the AL paper -- \"Continual Active Learning\" which aimed to speedup the active learning, and proposed additional evaluation based on robustness. \n\nLastly, there is a typo on Fig 2a, HL, where 10000 is written as 1000. \n\n\n-  Das et al. 2022 Continual Active Learning. https://openreview.net/forum?id=GC5MsCxrU- / https://arxiv.org/abs/2305.06408", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to carefully review our paper as well as for your insightful comments and questions. Note that the major changes made in our manuscript are detailed in the general reply above.  \n\n-> “Weakness are more of some additional experimental suggestions. Given the abundance of pretrained CLIP models, I think adding them would also have been a great idea. “\n- We agree that foundation models such as CLIP pretrained models hold great promise for Active Learning. However, given the already extensive scope of our work, we leave the necessary experiments to future work. As the focus of our work also lies in establishing proper evaluation protocols, these methods can be evaluated following our protocol. Based on your suggestion, we extended our conclusion and take-aways section with reference to CLIP and foundation models.\n\n-> “Secondly, in terms of difficulty, paper could've added some large scale datasets such as Imagenet variants.“\n- MIO-TCD Dataset is a large scale dataset with approx 500,000 images. Further, both ISIC-2019 and MIO-TCD are datasets that are commonly argued to represent real-world use-cases and should not be lumped together with simple datasets like CIFAR. To our knowledge, our work is one of the most extensive AL studies for deep learning based image classification. Other reviewers have commented on our experiments as “extensive” and “thorough” and the number, complexity, and diversity of our utilized datasets and settings (label-regimes, training strategies) already exceeds the standard in the field. \n\n-> “Lastly, one additional pillar could also be studying the OOD robustness. This was recently studied in the AL paper -- \"Continual Active Learning\" [a] which aimed to speedup the active learning, and proposed additional evaluation based on robustness.”\n- We agree that studying other characteristics of the model, such as OOD robustness, would allow for further characterization of AL methods. However, in our study, we aimed to improve the evaluation of the standard classification metric by which all AL methods are evaluated. To our understanding, [a] mainly evaluated OOD robustness as they suspect that continually trained models exhibit worse generalization performance against shifts similar to models pre-trained on a different source dataset [b]. Since our trainings do not employ continual learning, this particular problem does not directly apply to our benchmark. However, as the general notion of evaluating robustness (such as in [a]) is very valuable for the community, we added a sentence in the conclusion & take-aways section stating that testing the robustness of AL, especially in the light of novel training schemes like continual learning is a potential future extension of our protocol.\n\n-> Lastly, there is a typo on Fig 2a, HL, where 10000 is written as 1000. \n- We thank the reviewer for pointing this out. We changed the last tick in the x-axis from 1000 to 10000 in Figure 2 (CIFAR-10) high-label regime.\n\n\nThank you again for your valuable feedback. As we believe to have resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending stronger acceptance of our work.\n\n*Additional References*:\n\n[a] Das et al. 2022 Continual Active Learning. \nhttps://openreview.net/forum?id=GC5MsCxrU- / https://arxiv.org/abs/2305.06408\n\n[b] Ash, Jordan, and Ryan P. Adams. \"On warm-starting neural network training.\" Advances in neural information processing systems 33 (2020): 3884-3894.\n"}, {"review_id": "5We7G1sHpe", "reviewer": "Reviewer_DMVM", "summary": "The submission presents a large collection of experimental results for four active learning methods (entropy, core-set, BALD, BATCH) applied to image classification using ResNet-18 models, investigating the effect of differences in the data distribution (e.g., \"roll-out data\" from a different domain), size of initial labeled dataset, size of query batch, and integration of self- and semi-supervised learning. Special care is taken to optimize the supervised baseline and the active learning methods (on the development datasets).", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The submission presents an extensive collection of results and appears to be the only study so far that jointly considers all the five \"pitfalls\" referred to in the manuscript.\n\nThe poor performance of semi-supervised learning on the roll-out datasets is remarkable, and the results bolster the case for the usefulness of active learning.\n\nIt is very useful to see an independent evaluation of BADGE that indicates its performance is robust.\n", "weaknesses": "Two of the main findings do not appear to be novel: the usefulness of careful parameter tuning and the beneficial effect of self-supervised learning (although this new work perhaps provides more extensive empirical support).\n\nThe results are based on a single neural network architecture (ResNet18) and a single performance metric on each of the two groups of datasets (plain accuracy on the development datasets and balanced accuracy on the roll-out datasets).\n\nOnly a single semi-supervised learning algorithm is considered (a version of FixMatch). Considering the statement that \"Semi-SL performs best on toy datasets but does not seem to generalize to more complex settings.\", this seems problematic.\n\nThe experiments seem to jointly vary the batch size and the size of the initial labeled dataset. It seems difficult to disentangle the effect of these two parameters based on the presentation of the results in the paper, and it is unclear whether all the claims regarding the effect of these two parameters are supported.\n\nThe submission implies that active learning is particularly useful when dealing with imbalanced data. However, it does not appear to consider techniques applied in standard supervised learning to deal with imbalanced data.\n\nFull results for semi-supervised learning on MIO-TCD and ISIC-2019 should be included.\n\n-----\nThe rebuttal has satisfactorily addressed these comments.\n-----", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to carefully review our paper as well as for your insightful comments and questions. Note that the major changes made in our manuscript are detailed in the general reply above. \n\n-> “Two of the main findings do not appear to be novel: the usefulness of careful parameter tuning and the beneficial effect of self-supervised learning [...].”\n- We believe there is a misconception about the main concepts of our work. We have revised the description of these concepts (see general reply): AL validation paradox, P1-P5, and roll-out data sets. We hope this update resolves the misconception. Essentially our argument is that all five pitfalls need to be avoided in order to be able to make meaningful statements about the practical value of QMs. As shown in Table 1, *none of the related work currently checks all boxes*. Thus, even though some of our findings are not entirely novel, our holistic approach enables statements about the practical value of AL methods such that they transfer to a wide range of application scenarios. Further, the approach implied in your comment about hyperparameter tuning using AutoML on every AL training step [39] is substantially more expensive (factor of 10) compared to our light-weight protocol, where hyperparameters are tuned once on the starting budget without compromising classifier performance (see Appendix H). Our contribution therefore includes a practically feasible solution that might enable widespread adoption in the field.\nSimilarly, regarding the efficiency of self-supervised learning, we are the first to verify the advantage over well configured supervised baselines on multiple and imbalanced datasets.\n\n-> “The results are based on a single neural network architecture [...] and a single performance metric on each of the two groups of datasets [...].”\n- The most widely used architectures in the deep AL classification community are VGG and ResNet18 [2,4,6,25,30,39,43,45,49,50,53]. We focused on the ResNet-18 as it performs better and allows for simple integration of Self-SL and Semi-SL paradigms. Please also note that other reviewers commented on our experiments as “thorough” and “extensive”. \n- Deriving insights based on a single classification metric per dataset is the standard in AL literature [4,6,11,18,19,25,38,39,49,53]. Exceeding the usual analysis, we added two metrics specific to AL evaluation. The Area Under Budget Curve[52] and the Pairwise Penalty Matrix [2]. Nevertheless, to resolve your concern regarding the small number of employed metrics, we added results to Appendix F and the PDF alongside the general reply showing macro-averaged F1-scores for ISIC-2019 and MIO-TCD.\n\n\n-> Problematic statement about Semi-SL:\n- Thank you for pointing this out. We agree that our original statement about Semi-SL methods was too broad. As detailed in the general reply, we toned down our statement.\n\n-> “The experiments seem to jointly vary the batch size and the size of the initial labeled dataset. It seems difficult to disentangle the effect of these two parameters based on the presentation of the results in the paper, and it is unclear whether all the claims regarding the effect of these two parameters are supported.”\n- We correlated the two parameters (smaller query size for smaller starting budget etc.) since 1) in practice, we deem the size of the starting budget to be dependent on labeling cost (therefore, large query sizes for small starting budgets are unrealistic and vice versa) and 2) In this work, we are especially interested in smaller starting budgets (“cold-start” territory) compared to the ones in the literature, since AL typically shows robust performance for larger starting budgets. Theory shows that our adapted smaller query size for this case can only positively affect the result [27]. The only possible confounder could be that we interpret the performance of a small starting budget too positively due to a hidden effect of the smaller query size. However, we performed an ablation study in Figure 3, showcasing that varying the query size for small starting budgets did not have considerable effects on performance for all QMs, except BALD, where expectedly, a clear performance increase for smaller query sizes was observed.\nThank you for pointing out a lack of clarity in this design choice. We thoroughly revised the respective parts in our manuscript.\n\n-> “The submission implies that active learning is particularly useful when dealing with imbalanced data. However, it does not appear to consider techniques applied in standard supervised learning to deal with imbalanced data.”\n- As stated in Section 3 (lines 250-254), we apply balanced sampling of classes for standard training and Self-SL (finetuning) and weighted CE-Loss for Semi-SL. These are standard techniques to deal with imbalanced data. To further resolve your concern, we now provide an additional ablation for all experiments using weighted-CE-Lossi for ST and Self-SL models. For results, we kindly refer to the general reply and the provided PDF.\n\n-> “Full results for semi-supervised learning on MIO-TCD and ISIC-2019 should be included.”\n- Expectably, on CIFAR-X datasets we observed the gain of Semi-SL over ST to be decreasing with more annotated data (i.e. largest gain on the starting budget). Since on MIO-TCD and ISIC-2019, we observe no Semi-SL gain already on the starting budgets of the low- and mid-label regime, it is reasonable to assume that no gain (over ST and Self-SL) will occur with more annotated data. Given this reasoning and that adding these experiments requires approximately 18k GPU hours, we think it is fair to omit the experiments. We clarified this reasoning in the manuscript.\n\nThank you again for your valuable feedback. As we believe to have resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending acceptance of our work.\n\nWe use the citation keys from our manuscript."}, {"review_id": "PuX1qzJg4x", "reviewer": "Reviewer_D3At", "summary": "The authors identify five main pitfalls that have impacted previous Active Learning (AL) empirical evaluations. They propose a novel evaluation framework that avoids these pitfalls, and they study both Active Learning and its potential synergies with Semi-Supervised and Self-supervised learning. The paper provides a comprehensive empirical evaluation, together with copious appendices.", "questions": "1) the four \"motivating papers (lines 23-24)\" [6, 19, 38, 39] use, besides CIFAR-10/100, ImageNet and Tiny ImageNet; why didn't you consider these datasets, given that some of those papers' mistaken claims are made on them?\n\n2) on line 198-199, you claim that the three CIFARs are \"developmental datasets,\" while the other two are used for roll-out. While this is a great idea, an even better one would be to (i) drop the CIFARs, which are too \"nice & easy\" to begin with, (ii) add another 2-3 non-CIFAR, real-world datasets, and (iiI) perform either Cross-Validation or Leave-One-Out on the 5 datasets - say, by taking turns in using 1-4 of them as dev datasets, and the remaining one(s) as test dataset. Did you consider such an approach?\n\n3) I am not sure if this is just a flaw of the empirical evaluation, or is should be added as a 6th pitfall, but it is always bothersome to see that the CIFAR-10/100 evaluations of an active learner ends way earlier than the SOTA performance of 99% and 96%, respectively (in this paper, the learning curves end around 94% and 76%, respectively). After all, AL is NOT supposed to be for \"quickly reaching mediocre performance,\" but rather \"creating the best model with less data than required by random sampling.\"  It could be that this is due to the uniform distribution of the CIFARs (thus making them unrealistic & unsuitable for this evaluation), but this issue should be addressed head-on rather omitted.  \n  \n", "limitations": "See Weaknesses & Questions above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's main contribution is to carefully catalogue and map the various critical aspects of a proper AL evaluation, aka The Five Pitfalls; in contrast, prior work has properly addressed at most 3 of the 5 pitfalls. The thorough empirical evaluation emphasizes the proper way to create an evaluation framework for active learning, and how to also take into account Semi- & Self- Supervised methods. The paper is fairly easy to read and follow.   ", "weaknesses": "The paper can be further improved along the readability dimension.\n\nFirst of all, in its current form, the paper has \"too much content in too little space\" Even though, at first glance, the current split between \"the main paper\" and \"the appendices\" seems reasonable, the paper would greatly benefit by compressing Section 2.3 (which could be then added \"as is\" in an APPENDIX). On the other hand, Section 4 is hard to follow because the graphs in Fig 2 are too small to read, so the prose is not really supported by Fig 2. One possible way to address this issue would be to move most of the 3-page Section 4 to the APPENDICES and replace it with (A) a brief, 1-page summary of the results - no figures, and (B) a new section with an INTUITIVE, PRESCRIPTIVE section that crisply summarizes the main findings: \"the ideal evaluation framework for active learning for image classification is the following ...\" \n\nSecond, the idea of \"rolling-out Active Learning\" to \"unseen data sets different from the labeled development data\" (first introduced at lines 32-33) is mentioned 13 times without being properly explained. In this reviewer's opinion, the idea is worth being presented in the main part of the paper, but the authors should invest the necessary 2-3 paragraphs to crisply & intuitively explain it.\n\nOTHER COMMENTS:\n- active learning does not begin/end with DL and image classification. As such, the authors could easily take the 1-2 sentences required to explicitly define the scope of their investigation.   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to carefully review our paper as well as for your insightful comments and questions. Note that the major changes made in our manuscript are detailed in the general reply above. \n\n-> Presentational issues: “[...] the paper has \"too much content in too little space\". [...]the paper would greatly benefit by compressing Section 2.3 .[...] the graphs in Fig 2 are too small to read[...]. ”\n- We thank you for these detailed suggestions. Based on these, we restructured our manuscript to improve readability and clarity. Details on this restructuring are provided in the general reply. Further, we would like to highlight that all results in Fig. 2 are also shown in larger individual figures in Appendix F, which we reference in Section 4 (line260). We made this reference more prominent in the manuscript.\n\n-> “Second, the idea of \"rolling-out Active Learning\" to \"unseen data sets different from the labeled development data\" [...] is [...] not properly explained. [...] the idea is worth being presented in the main part of the paper [...].”\n- Thank you for pointing this out. We now provide an intuitive and crips explanation for the concept of roll-out datasets. The updated descriptive part of our core concepts is detailed in the general reply. \n\n-> “active learning does not begin/end with DL and image classification. As such, the authors could easily take the 1-2 sentences required to explicitly define the scope of their investigation.”\n- We thank the author for this suggestion and added the following description of the scope in:\n    - Abstract (Line 3):  “...research on improving AL query methods for deep active classification of image data…”\n    - Section 1 (Line 40): “To this end, we present an evaluation framework for deep active classification of image data that overcomes the five pitfalls and demonstrate the relevance of this contribution by means of a large-scale empirical study spanning various datasets, query methods, AL settings, and training paradigms”.\n    - Section 2.1 as the first sentence: “The scope of our study includes deep-learning based active learning methods used for image classification.”\n\n\n-> “the four \"motivating papers (lines 23-24)\" [6, 19, 38, 39] use, besides CIFAR-10/100, ImageNet and Tiny ImageNet; why didn't you consider these datasets, given that some of those papers' mistaken claims are made on them?”\n- We used the CIFAR-X datasets as they were used by practically all related work so that we can compare performance across studies (shown in Appendix H) and decided to focus on real-world data on ISIC-2019 and MIO-TCD, which span a broader diversity of datasets compared to adding the suggested ones (ImageNet, CIFAR-X, STL-10). Further, Self-SL and Semi-SL methods are usually developed on these datasets; thus, using these might have biased the evaluation in their favor.\n\n\n-> “on line 198-199, you claim that the three CIFARs are \"developmental datasets,\" while the other two are used for roll-out. While this is a great idea, an even better one would be to (i) drop the CIFARs, which are too \"nice & easy\" to begin with, (ii) add another 2-3 non-CIFAR, real-world datasets, and (iiI) perform either Cross-Validation or Leave-One-Out on the 5 datasets - say, by taking turns in using 1-4 of them as dev datasets, and the remaining one(s) as test dataset. Did you consider such an approach?”\n- We agree that the evaluation could be improved with additional datasets aside from the CIFARs (reason for inclusion see one paragraph above). We did not consider Cross-Validation (CV) for the evaluation and opted for a static split instead, as we (A) wanted to minimize leakage of knowledge into the test datasets and (B)  CV would increase the required compute for the experiments by a factor of at least #folds.\nThe general idea, however, is very interesting and with proper execution might further improve the AL evaluation protocol. \nWe added this comment to the conclusion & take-aways section in the manuscript.\n\n-> “I am not sure if this is just a flaw of the empirical evaluation, or is should be added as a 6th pitfall, but it is always bothersome to see that the CIFAR-10/100 evaluations of an active learner ends way earlier than the SOTA performance of 99% and 96%, respectively (in this paper, the learning curves end around 94% and 76%, respectively). After all, AL is NOT supposed to be for \"quickly reaching mediocre performance,\" but rather \"creating the best model with less data than required by random sampling.\" [...]”\n- We agree with the notion that AL is about \"creating the best model with less data than required by random sampling”. Also, we agree that it is not about quickly achieving mediocre performance. \nHowever, we would like to point out that our performances on CIFAR-X of 94% (of ~96%) and 76% (of ~79%) are very close to SOTA performance with a ResNet-18 on the whole dataset annotated (in brackets). The accuracies you mention are essentially always achieved with very large models pretrained on ImageNet21k/1k [a]. Using methods like this would greatly improve the computational expense for running Active Learning experiments and, in the case of CIFAR-X lead to results that are also favorably biased as the classes are a subset of ImageNet that do not generalize to datasets/tasks where this is not the case.\nHowever, we agree that finetuning of foundation models has the potential to improve the baseline performance of all models greatly and added this to our conclusion & take-aways section.\n \n\nThank you again for your valuable feedback. As we believe to have resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending stronger acceptance of our work.\n\n*Additional references*:\n\n[a]Foret, Pierre, et al. \"Sharpness-aware minimization for efficiently improving generalization.\" arXiv preprint arXiv:2010.01412 (2020).\n\n\n"}]

Author rebuttal:
We sincerely thank all reviewers for their valuable comments. While the reviewers generally agreed on the added value of our work (“timely problem”,“extensive evaluation”, “useful insights”),  concerns were raised regarding the clarity of the manuscript and the toning regarding Semi-SL. We have thoroughly revised our manuscript to clarify the respective parts and resolve these concerns. 

Next to the point-by-point responses below, we address the main recurring comments here: 

**Presentation of the core concepts**: The original presentation of the core concepts of our work seems to be the driver behind most of the raised questions and comments. Thus, we  updated the explanation of the “AL validation paradox”, and how it interconnects with the five pitfalls P1-P5 as well as the concept of “roll-out” datasets:  

“When applying AL on a new, mostly unlabelled data set, one can not directly validate whether QM1 or QM2 performs better, as this would require labeling individual AL trajectories through the data set for each QM. This excessive labeling would directly contradict the goal of AL to reduce labeling (“validation paradox”). Notably, this is in contrast to standard ML, where a fixed labeled validation split suffices to optimize the model configuration. Thus, the validation paradox forces one to instead estimate how well certain QMs will perform on the given task based only on prior knowledge. The quality of this estimate directly depends on the prior knowledge available, i.e. on how extensively the respective QM has been validated in the first place. The application at hand typically comes with a given class imbalance and constraints on the size of the starting budget and the query size. To ensure that the prior knowledge about the performance of a QM covers such specific application constraints, the QM needs to be validated on a realistic range of all of these parameters. Not doing so causes P1-P3 and limits the reliability of the required performance estimates of a QM in practice. Further, in P4 (“sub-optimal classifier configuration”) and P5 (“lack of alternative training paradigms SSL/Semi-SL”) we argue that AL is costly and thus assessing the practical value of a QM must include comparison against cheaper alternatives. For instance, if the gain of a QM over random sampling can also be achieved by simply tuning the learning rate of the classifier, the QM provides no practical value (P4), and similarly, if the gain can be achieved by a single SSL pretraining, the QM provides no practical value either (P5). Finally, several hyperparameters have to be configured in AL application like the classifier configuration (P4), but also possible hyperparameters of the QM itself. Since it is not feasible to validate a QM for all settings in this high-dimensional parameter space, as required by the validation paradox, we instead propose to test the robustness of a few optimized settings on additional datasets (“roll-out datasets”) that were not part of the optimization process (“development datasets”).
Taken together, statements about the practical value of a QM require addressing all five pitfalls P1-P5 and testing on “roll-out datasets”. As Table 1 shows impressively, the proposed protocol is needed to close the gap between current AL literature and a meaningful assessment of QMs.” 
Semi-SL verdict: We toned down our original verdict about SemiSL (“Semi-SL performs best on toy datasets but does not seem to generalize to more complex settings”) and added more supporting literature for this observation: ”Our results based on FixMatch indicate that Semi-SL methods perform well on datasets where they have been developed, but may struggle to generalize to more realistic scenarios such as class imbalanced data. This observation is in line with various findings in literature including Semi-SL methods other than FixMatch [a,b,c,4,7,20]”.

**Organization of the manuscript**: To improve the readability and clarity of the manuscript, we implemented structural changes as suggested by reviewer D3At: 
The current section 2.3 has been moved to the appendix and has been replaced with a more concise and clear description of the core concepts (see statement above), including an intuitive motivation for the roll-out datasets.
Instead, we increase the size of Figure 2 making the main results more readable. We further point to Appendix F now, containing detailed and zoomed-in graphs of all experiments.

**Additional experiments**: Some reviewers suggested adding new datasets, modalities or network architectures.While we agree with the scientific interest in these experiments we argue that the empirical support for our insights already exceeds the standard in the field.  To our knowledge this work is one of the most extensive AL studies out there with most reviewers acknowledging our “thorough” and “extensive” experiments. Nevertheless, we tried to include as many suggestions as possible and now provide:
An ablation of our over-sampling strategy on imbalance data, using weighted CE loss (as suggested by reviewer DMVM). Showing 1) a performance decrease of weighted CE compared to over-sampling for ST, and 2) even though the strategy for handling imbalanced data influences the performance of AL methods, the observed gain of AL methods over random sampling seems to be consistent.
Macro-averaged F1-score as an additional performance metric for the imbalance datasets MIO-TCD and ISIC-2019 (as suggested by reviewer DMVM).  Showing that the general trend observed with balanced accuracy holds for the macro-averaged F1-score.


*Additional references*:

[a] Guo et al. “Robust Deep Semi-Supervised Learning A Brief Introduction”. 2022

[b] Guo et al. “Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data”. 2020

[c] Kim et al. “Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning”. 2020


Meta-review: This paper tackles the challenges in evaluating Active Learning (AL) methods, proposing an evaluation framework and conducting extensive experiments. The reviewers agree that this is a timely topic and the pitfalls identified can be useful to the community. Some reviewers point out the room for improvement in terms of dataset diversity, deeper study on the hyperparameter effects, and more careful study of advanced imbalanced learning techniques that are being coupled with AL. The authors' rebuttal addresses concerns and clarifies key issues. The authors should take the reviewers' comments into account in the camera-ready version.
Decision: Accept (poster)