SlowGuess/NeurIPS_2023_OCR huggingface dataset:
BADGE Uses the K-MEANS++ initialization algorithm on the last layer gradient embeddings  $\mathcal{G} = \{\nabla_{\theta_{L-1}}\mathcal{L}(\hat{y}(x), p(Y|x, \theta)|x \in \mathcal{U}\}$  to obtain B centers. These are likely to have diverse directions (which capture diversity due to diverse parameter updates) and large loss gradients (capturing uncertainty due to large loss changes) [2].
A.3 Connection to self-supervised learning
The high-level concept of Self-SL pre-training is to obtain a model by training it with a proxy task that is not dependent on annotations, leading to representations that generalize well to a specific task. This allows the induction of information from unlabeled data into the model in the form of an initialization, which can be interpreted as a form of bias. Usually, these representations are supposed to be clustered based on some form of similarity, which is often induced directly by the proxy task and also the reason why different proxy tasks are useful for different downstream tasks. Several different Self-SL pre-training strategies were developed based on different tasks s.a. generative models or clustering [12, 13, 21, 25], with contrastive training being currently the de facto standard in image classification. For a more thorough overview over Self-SL we refer the interested reader to [41]. Based on this, we use the popular contrastive SimCLR [13] training strategy as a basis for our Self-SL pre-training.
A.4 Connection to semi-supervised learning
In Semi-SL the core idea is to regularize  $p(Y|x,\theta)$  by inducing information about the structure of  $p(x)$  using the unlabeled pool additionally to the labeled dataset. Usually, this leads to the representations of unlabeled samples with the clustering being more in line with the structure of the supervised task [39]. Several different Semi-SL methods were developed based on regularizations on unlabeled samples, which often fall into the category of enforcing consistency of predictions against perturbations and/or reducing the uncertainty of predictions (for more information, we refer to [45]). For a more thorough overview of Semi-SL we refer interested readers to [7, 45] In our experiments, we use FixMatch [52] as a Semi-SL method, which combines both aforementioned principles of consistency and uncertainty reduction in a simple manner.
B Active learning literature, in more detail
We will discuss the current literature landscape of deep active classification with a focus on our proposed key-pitfalls as shown in Figure 1b.
The rules for evaluation of each of the five pitfalls (P1-P5) are:
P1 Data distribution Use of multiple datasets for evaluation featuring class-imbalanced datasets.
P2 Starting budget Evaluation or abating the influence of the starting budget on multiple datasets explicitly.
P3 Query size Evaluation or ablating the influence of the query size on multiple datasets explicitly.
P4 Performant baselines Performance is close to ours or Munjal et al. [44] for ST models on CIFAR-10/100(see Appendix H for details).<sup>3</sup>
P4 HP Optim. & Val. Split The use of a dedicated validation set to configure the classifier.
P5 Self-SL Benchmarking AL with a performant Self-SL training paradigms.
P5 Semi-SL Benchmarking AL with a performant Semi-SL training paradigm
Munjal et al. [44] Evaluate the performance of AL methods and compare against and with well finetuned baseline models using AutoML.
P1 Data Distribution: Perform experiments on CIFAR-10, CIFAR-100 and limited experiments on ImageNet. They perform an ablation on CIFAR-100 with an artificial imbalanced dataset.
$\rightarrow (\checkmark)$  due to limited imbalanced datasets.
P2 Starting Budget: Perform no experiments at all regarding the starting budget.
$\rightarrow \mathbf{X}$
P3 Query Size: Perform ablations on CIFAR-10/100 comparing query sizes of  $5\%$  (2500) to  $10\%$  (5000).  $\rightarrow$
P4 Performant Baselines: They achieve performance on CIFAR-10/100 on par with ours (see Appendix H).  $\rightarrow$
P4 HP Optim. & Val. Split They explicitly use a validation set and finetune their hyperparameters based on the validation set performance using AutoML.
P5 Self-SL: They do not consider using models pre-trained with Self-SL.
P5 Semi-SL: They do not consider using models trained with semi-supervised training paradigms.
Mittal et al. [43] Evaluate the performance of AL methods and set them into context with semi-supervised training paradigms.
P1 Data Distribution: Perform experiments on CIFAR-10, CIFAR-100.
→X
P2 Starting Budget: Perform experiments both on the standard setting with starting budget of 5000 (10%) on CIFAR-10/100 as well as 250 (CIFAR-10) and 500 (CIFAR-100).
$\rightarrow (\checkmark)$  due to limited settings.
P3 Query Size: They do not consider specifally ablating the query size.
<sup>3</sup>We only base this on ST models, as getting good performance with Self-SL and Semi-SL for low data settings can be achieved without taking HP configuration into account, as they can simply be taken from a paper focusing on them which often use very large validation sets.
P4 Performant Baselines: Their random baseline is more performant on CIFAR10/100 than most of the literature. However, not as good as Munjal et al. [44] or ours (see Appendix H).
$\rightarrow (\vee)$  due to performance being good but no on par with ours.
P4 HP Optim. & Val. Split They do not state optimizing their hyperparameters based on a validation set.
P5 Semi-SL: They evaluate AL with and against the semi-supervised training paradigm 'Unsupervised Data Augmentation for Consistency Training'.
Bengar et al. [6] Evaluate the performance of AL methods and set them into context with self-supervised training paradigms.
P1 Data Distribution: They perform experiments on CIFAR-10/100 and TinyImageNet. All of which are class balanced datasets.
$\rightarrow \mathbf{X}$  due to only evaluating balanced datasets.
P2 Starting Budget: They use 3 different starting budgets on each of their three datasets. CIFAR-10:  $0.1\%$ ,  $1\%$ ,  $10\%$ ; CIFAR-100:  $1\%$ ,  $2\%$ ,  $10\%$ ; Tiny ImageNet:  $1\%$ ,  $2\%$ ,  $10\%$  (\% of the whole dataset).
P3 Query Size: Each of the three different starting budgets has a different query size resulting in overlapping experiments. Therefore it would be possible to draw some conclusions about the influence of the query size.
$\rightarrow (\checkmark)$  due missing selective evaluation of query size.
P4 Performant Baselines: Their supervised random baseline is performing worse on CIFAR-10/100 than most models in the literature (see Appendix H).
P4 HP Optim. & Val. Split: They do not state optimizing their hyperparameters based on a validation set.
P5 Self-SL: They evaluate AL methods with and against one self-supervised training paradigm (SimSiam).
P5 Semi-Supervise Learning: They do not consider using models trained with semi-supervised training paradigms.
Gao et al. [20] Evaluate the performance of AL methods against and in the context of semi-supervised training paradigms. Further, they propose a new query method designed for AL with models that are trained with a Semi-SL training paradigm.
P1 Data Distribution: They perform experiments on CIFAR-10/100 and ImageNet.
P2 Starting Budget: They perform a specific ablation about the importance of the starting budget on CIFAR-10 with multiple settings and discuss it.
P3 Query Size: In addition to the standard experiments, they perform experiments with query sizes of 50 and 250. However, they do not specifically discuss its importance.
$\rightarrow (\mathbb{V})$  due missing selective evaluation of query size.
P4 Performant Baselines: The performance of their supervised random baseline models in the main comparison is not close to our performance on CIFAR-10/100.
$$
\rightarrow \mathbf {X}
$$
P5 Semi-SL: They evaluate AL with and against the semi-supervised training paradigm (MixMatch).
$$
\rightarrow \checkmark
$$
Yi et al. [54] They propose to use self-supervised pre-text as a basis for query functions.
P1 Data Distribution: Perform experiments on CIFAR-10, an imbalanced version of CIFAR-10, Caltech-101 and ImageNet.
P2 Starting Budget: One experiment is performed where they select the starting budget with their proposed Active Learning method on CIFAR-10. Otherwise, they do not evaluate the performance under different starting budgets.
P3 Query Size: They do not evaluate the performance with regard to different query sizes.
P4 Performant Baselines: Their supervised random baseline models are not close to the performance of our random baseline models on CIFAR-10 (see Appendix H).
P5 Self-SL: They consider several different Self-SL paradigms ('Rotation Prediction', 'Colorization', 'Solving jigsaw puzzles' and 'SimSiam') based on which they select rotation prediction for their experiments.
$\rightarrow (\vee)$  due to selection of non state-of-the-art Self-SL paradigm.
Krishnan et al. [35] They propose to use a supervised contrastive training paradigm as a basis for two AL methods.
P1 Data Distribution: Perform experiments on Fashion-MNIST, SVHN and CIFAR-10 and an imbalanced version of CIFAR-10.
$\rightarrow (\mathbf{V})$  due to imbalanced CIFAR-10 being simulated.
P2 Starting Budget: They do not evaluate the performance under different starting budgets.
Kim et al. [30] They propose task-aware active learning which is a combination of learning loss, active learning and variational adversarial active learning.
P1 Data Distribution: Perform experiments on CIFAR-10, CIFAR-100, CALTECH 101 and imbalanced CIFARS.
$$
\rightarrow (\checkmark)
$$
P4 Performant Baselines: Their supervised random baseline models perform good but not on par with ours on CIFAR-10 and 100.
Beck et al. [4] Evaluate several AL methods in different settings to gain an understanding which AL methods outperform random queries. Further, they provide the Al toolkit DISTIL.
P1 Data Distribution: Perform experiments on CIFAR-10, CIFAR-100, Fashion-MNIST, SVHN and MNIST.  $\rightarrow$  X due to no class imbalance.
P2 Starting Budget: They perform one experiment, where they evaluate a lower starting budget for MNIST.
$\rightarrow (\vee)$  due limited dataset.
P3 Query Size: They evaluate three different query sizes on CIFAR-10, but do so only for Random, Entropy and BADGE.
$\rightarrow (\checkmark)$  due to limited scope.
P4 Performant Baselines: Their supervised random baseline models perform good but no par with our random baseline models on CIFAR-10/100 (see Appendix H).
$\rightarrow (\vee)$  due to limited scope.
Zhan et al. [58] Evaluate a multitude of different AL methods and provide the AL toolkit  $DeepAL^{+}$ .
P1 Data Distribution: Perform experiments on Tiny ImageNet, CIFAR-10 (and CIFAR-10 imbalanced), CIFAR-100, Fashion-MNIST, EMNIST and SVHN. Further Experiments are performed on an Histopathological image Classification Task (BreakHis) and Chest X-Ray Pneumonia classification (Pneumonia-MNIST) as well as the Waterbird dataset adopted from object recognition with correlated backgrounds.
P3 Query Size: They evaluate multiple different query sizes on CIFAR-10 and analyze the difference.
$$
\rightarrow \swarrow
$$
P4 Performant Baselines: Their supervised random baseline models are not close to the performance of our random baseline models on CIFAR-10/100.
Chan et al. [11] Evaluate how AL methods interact with self- and semi-supervised training paradigms and how and whether they yield a benefit. The experiments in this paper differ from standard AL experiments by using only one query cycle, making it hard to compare systematically.
P1 Data Distribution: Perform experiments on CIFAR-10 and CIFAR-100.
P2 Starting Budget: Experiments are performed with a fixed starting budget of 3 samples per class or 2 samples per class in one case. This does not allow for evaluate of the influence of the starting budget on AL methods.
P3 Query Size: They query all samples for their final performance in one query step. This does not allow for evaluation of the influence of the query size on AL methods.
P4 Performant Baselines: Their supervised random baseline models perform on par with our random baseline models on CIFAR-10/100.
P5 Self-SL: They use Debiased Contrastive Learning as self-supervised pretext task.
P5 Semi-SL: They use Pseudo-labeling and FixMatch as semi-supervised training paradigms.
Table 1: Number of Samples for each class in CIFAR-10 LT dataset. Validation and test sets are balanced.
Class	Train Split
airplane	4500
automobile (but not truck or pickup truck)	2913
bird	1886
cat	1221
deer	790
dog	512
frog	331
horse	214
ship	139
truck (but no pickup truck)	90
Each dataset is split into a training, a validation and a test split.
For CIFAR-10/100 (LT) datasets the test split of size 10000 observations is already given and for MIO-TCD and ISIC-2019 we use a custom test split of  $25\%$  random observations of the entire dataset size. For MIO-TCD and ISIC-2019 the train, validation and test splits are imbalanced.
The validation split for all CIFAR-10 and CIFAR-100 datasets are 5000 randomly drawn observations corresponding to  $10\%$  of the entire dataset. For CIFAR-10 LT the validation split also consists of 5000 samples obtained from the dataset before the long-tail distribution is applied onto the training split. The CIFAR-10 LT validation split is therefore balanced. For MIO-TCD and ISIC-2019 the validation splits consist of  $15\%$  of the entire dataset.
The shared training & pool dataset for CIFAR-10/100 consists of 45000 observations. For CIFAR-10 LT the training & pool datasets consist of 12,600 observations. For MIO-TCD and ISIC-2019 the training & pool datasets consist of  $60\%$  the dataset.
C.1 Dataset descriptions
1. CIFAR-10: natural images containing 10 classes, label distribution is uniform  
Splits: (Train:45000; Val: 5000; Test: 10000)  
Whole Dataset: 60000
2. CIFAR-100: natural images containing 100 classes, label distribution is uniform  
Splits: (Train:45000; Val: 5000; Test: 10000)  
Whole Dataset: 60000
3. CIFAR-10 LT: natural images containing 10 classes, label distribution of test and validation split is uniform, label distribution of train split is artificially altered with imbalance factor  $\rho = 50$  according to [10]. The resulting label distribution is shown in Tab. 1. Splits: (Train:  $\sim 12,600$ ; Val: 5000; Test: 10000) Whole Dataset: 27600
4. ISIC-2019: dermoscopic images containing 8 classes, label distribution of the dataset is imbalanced and shown in Tab. 2  
Splits: (Train:15200; Val: 3799; Test: 6332)  
Whole Dataset: 25331
5. MIO-TCD: natural images of traffic participants containing 11 classes, label distribution of the dataset is imbalanced and shown in Tab. 3 Splits: (Train:311498; Val: 77875; Test; 129791) Whole Dataset: 519164
Table 2: Number of Samples for each class in ISIC-2019
Class	Whole Dataset
Melanoma	4522
Melanocytic nevus	12875
Basal cell carcinoma	3323
Benign keratosis	867
Dermatofibroma	197
Vascular lesion	63
Squamos cell carcinoma	64
Table 3: Number of samples for each class in MIO-TCD
Class	Whole Dataset
Articulated Truck	10346
Background	16000
Bicycle	2284
Bus	10316
Car	260518
Motorcycle	1982
Non-motorized vehicle	1751
Pedestrian	6262
Pickup truck	50906
Single unit truck	5120
Work van	9679
D Experimental setup, in more detail
Here we detail the most crucial information for reproducibility, re-implementation and checking our implementation. When in doubt, trust the information documented here with regard to what we wanted to do in our code.
D.1 Initial dataset setup
Before we do anything else the datasets are split according to Figure 4 resulting in a tain split, a validation split and a test split. Each dataset has 3 different validation splits while always using the same test split. This is to ensure comparability across these splits without relying on cross-validation. The exact splits for each dataset are detailed in Appendix C. After that the final datasets use for training and validation are then labeled according to the 'label strategy', which is described in Figure 5. For all balanced datasets, we use class balanced label strategies since the label strategy only leads to different outcomes for imbalanced datasets. For CIFAR-10 LT we use the label strategy on the train split only, whereas for MIO-TCD and ISIC-2019 we use the label strategy on both train and validation split. The amount of data which is labeled for the final datasets of each split is then dependent upon the label-regime (described in more detail in Appendix D.2).
D.2 Label regimes
The exact label regimes are obtained by first taking the corresponding splits and then using the proper label strategy (see Figure 5) in combination with the starting budget and validation set size according to Tab. 4.
D.3 Model architecture and training
On each training step the model is trained from its initialization to avoid a 'mode collapse' [32]. Further we select the checkpoint with the best validation set performance in the spirit of [19]. A ResNet-18 [24] is the backbone for all of our experiments with weight decay disabled on bias parameters. If not otherwise noted, a nesterov momentum optimizer with momentum of 0.9 is used.
Figure 4: Description of the three different data splits and their use-cases. The complete separation of a validation split allows to compare across label regimes and incorporate techniques for performance evaluation s.a. Active Testing [34]. For evaluation and development the test split should be as big as possible since QM recommendations are based on the test set performance making it a form of "oracle". An estimate of the size a dataset is required to have to measure specific performance differences can be derived using Hoeffding's inequality [26, 45].
Figure 5: The Label Strategy used on the two roll-out datasets MIO-TCD and ISIC-2019 and for train and pool set on CIFAR-10 LT. For class balanced datasets this strategy does not induce meaningful changes to balanced starting budgets.
Table 4: The exact values for all label regimes. Final Budget denotes the amount of labeled training samples at the end of the AL pipeline.
Dataset Label Regime	CIFAR-10	CIFAR-100	CIFAR-10 LT	MIO-TCD	ISIC-2019
Low	Mid	High	Low	Mid	High	Low	Mid	High	Low	Mid	High	Low	Mid	High
Starting Budget	50	250	1000	500	1000	5000	50	250	1000	55	275	1100	40	200	800
Query Size	50	250	1000	500	1000	5000	50	250	1000	55	275	1100	40	200	800
Final Budget	500	2500	10000	5000	10000	25000	500	2500	10000	550	2750	11000	400	2000	8000
Validation Set Size	250	1250	5000	2500	5000	5000	250	1250	5000	275	1375	5500	200	800	3799
For Self-SL models we use a two layer MLP as a classification head to make better use of the Self-SL representations with further details in Appendix D.5. To obtain bayesian models we add dropout on the final representations before the classification head with probability  $(p = 0.5)$  following [19]. For all experiments on imbalanced datasets, we use the weighted CE-Loss following [44] based on the implementation in SK-Learn [9] if not otherwise noted. Models trained purely on the labeled dataset upsample it to a size of 5500 following [32] if the labeled train set is smaller.
Bayesian Models All steps that require bayesian properties of the models including the prediction are obtained by drawing 50 MC samples following [19, 32].
ST ST models are trained for 200 epochs with Cosine Annealing and 10 epochs warmup.
Self-SL Self-SL pre-trained models are trained for 80 epochs with a reduction of the learning rate with a factor of 10 every 20 epochs (MultiStepLR) and using a Multit-Layer-Percpeptron (MLP) classification head (detailed description in Appendix D.5). The complete setup of the training for SimCLR is described in Appendix D.4.
Semi-SL Semi-SL training is identical to the one proposed with the FixMatch method [52], except that we do not use exponentially moving average models and restrict the training step from 1e6 to 2e5. The FixMatch implementation in our experiments is based on the open-source implementation
Table 5: HPs of the SimCLR pre-text training on each dataset. HP for CIFAR datasets are directly taken [13] whereas MIO-TCD and ISIC-2019 HP are adapted from ImageNet experiments.
Dataset	CIFAR-10/CIFAR100/CIFAR-10 LT	MIO-TCD	ISIC-2019
Epochs	1000	200	1000
Optimizer	LARS	LARS
Scheduler	Cosine Annealing	Cosine Annealing
Warmup Epochs	10	10
Temperature	0.5	0.1
Batch Size	512	256
Learning Rate	1	0.3
Weight Decay	1E-4	1E-6
Transform. Gauss Blur	False	True
Transform. Color Jitter	Strength=0.5	Strength=1.0
Table 6: MLP Head Ablation for Self-SL models on CIFAR-10, over all labeled training set a small improvement for Multi-Layer-Perceptron is measurable compared to Linear classification head models. Reported as mean (std).
Labeled Train Set	Classification Head	Accuracy (Val) %	Accuracy (Test) %
50	Linear	69.87(1.62)	69.90(2.18)
50	2 Layer MLP	71.47(3.06)	71.54(0.56)
500	Linear	84.67(0.36)	83.51(0.45)
500	2 Layer MLP	85.37(0.16)	84.60(0.37)
1000	Linear	87.13(0.69)	85.97(0.64)
1000	2 Layer MLP	87.69(0.55)	86.57(0.42)
5000	Linear	90.77(0.44)	90.20(0.21)
5000	2 Layer MLP	91.12(0.32)	90.25(0.24)
of  $^4$  and MixMatch for distribution alignment  $^5$ . We always select the final Semi-SL model of the training for testing and querying. On imbalanced datasets we change the supervised term to the weighted CE-Loss and use distribution alignment on every dataset except for CIFAR-10 (where it does not improve performance [52]). The HP sweep for our Semi-SL models includes weight decay and learning rate.
Hyperparameters All information with regard to the final HPs and our proposed methodology of finding them is detailed in Appendix E
D.4 Self-supervised SimCLR pre-text training
Our implementation wraps the Pytorch-Lighting-Bolts implementation of SimCLR: https://lightningbolts.readthedocs.io/en/latest/models/self_supervised.html#simclr. The training of our SimCLR models is performed by excluding the validation splits. Therefore three models are trained on each dataset, one for each different validation split. In Tab. 5 we give a list of the HPs used on each of our five different datasets. All other HPs are taken from [13]. Further, we did not optimize the HPs for SimCLR at all, meaning that on MIO-TCD and ISIC-2019 Self-SL models could perform even better than reported here.
D.5 MLP head for self-supervised pretrained models
The MLP Head used for the Self-SL models has 1 hidden layer of size 512 uses ReLU nonlinearities and BatchNorm. The results on CIFAR-10 based on which this design decision is based on is shown in Tab. 6.
4https://github.com/kekmodel/FixMatch-pytorch
<sup>5</sup>https://github.com/google-research/mixmatch
D.6 List of data transformations
Standard The standard augmentations we use are based on the different datasets.
For CIFAR datasets these are in order of execution: RandomHorizontalFlip, RandomCrop to 32x32 with padding of size 4.
For MIO-TCD we use the standard ImageNet transformations: RandomResizedCrop to 224x224, Random Horizontal Flip.
For ISIC-2019 we use ISIC transformations which are: Resize to 300x300, RandomHorizontalFlip, RandomVerticalFlip, ColorJitter(0.02, 0.02, 0.02, 0.01), RandomRotation(rotation=(-180, 180)), translate=(0.1, 0.1), scale=(0.7, 1.3)), RandomAffine(-180, 180), RandomCrop to 224x224.
These are based on the ISIC-2018 challenge best single model submission:
https://github.com/JiaxinZhuang/Skin-Lesion-Recognition.Pytorch
RandAugmentMC We use the same set of image transformations used in RandAugment [17] with the parameters  $N = 2$  and  $M = 10$ . A detailed list of image transformations alongside the corresponding values can be seen in [52] (Table 12).
The RandAugmentMC transformations were used additionally after the corresponding standard transformations for each dataset. RandAugmentMC(CIFAR) also adds cutout as a final transformation.
RandAugmentMC weak Works identical as RandAugmentMC and uses the same set of image transformations as for RandAugmentMC but changed its parameters to  $N = 1$  and  $M = 2$ . Therefore the maximal range of values is divided by a factor of 5.
RandAugmentMC weak does not use cutout in difference to RandAugmentMC on CIFAR datasets.
D.7 Performance measure