SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As a measure of performance on CIFAR-10, CIFAR-100 and CIFAR-10 LT we use the accuracy while on MIO-TCD and ISIC-2019 we use balanced accuracy which is identical to mean recall shown in Equation (3).
$$
\text {M e a n R e c a l l} = \sum_ {c = 1} ^ {C} \frac {1}{C} \frac {\mathrm {T P} _ {c}}{\mathrm {T P} _ {c} + \mathrm {F N} _ {c}} \tag {3}
$$
Where  $C$  denotes the number of classes  $\mathrm{TP}_c$  is the number of true positives for class  $c$  and  $\mathrm{FN}_c$  being the number of samples belonging to class  $c$  being wrongly misclassified as another class.
D.8 Computational effort
Experiments were executed on a Cluster with access to multiple NVIDIA graphics cards. All ST and Self-SL pre-trained experiments used a single Nvidia RTX 2080 (10.7GB video-ram) graphic cards except for the BADGE experiments on CIFAR-100 and MIO-TCD which required more video-ram using Nvidia Titan RTX (23.6GB video-ram). The Semi-SL models on the CIFAR-10/100 (LT) datasets used also a single Nvidia RTX 2080 while on MIO-TCD and ISIC-2019 the Nvidia Titan RTX was utilized. For the results in our main table (excluding the HP optimization), the overall runtime was:
- All ST experiments: 1800 GPU hours
- All Self-SL pre-trained experiments: 1350 GPU hours<sup>6</sup>
- All Semi-SL experiments: 11200 GPU hours
E Proposed hyperparameter optimization
Our proposed HP optimization for AL is based on the notion of minimizing HP selection effort by simplifying and reducing the search space. We use SGD Optimizer with Nesterov momentum of 0.9
6Excluding the pre-training
For only 2 label regimes and excluding MIO-TCD and ISIC-2019
Table 7: Final HPs for each dataset and label regime for our ST models based on our HP tuning. HPs denoted with a * are fixed across datasets and HP denoted with a + are pre-selected for each dataset while all other HP are obtained via sweeping.
Dataset Label Regime	CIFAR-10	CIFAR-100	CIFAR-10 LT	MIO-TCD	ISIC-2019
Low	Mid	Low	Mid	Low	Mid	Low	Mid	Low	Mid
Epochs*	200	200	200	200	200
Optimizer*	SGD Nesterov 0.9	SGD Nesterov 0.9	SGD Nesterov 0.9	SGD Nesterov 0.9	SGD Nesterov 0.9
Scheduler*	Cosine Annealing	Cosine Annealing	Cosine Annealing	Cosine Annealing	Cosine Annealing
Warmup Epoches*	10	10	10	10	10
Loss*	CE-Loss	CE-Loss	CE-Loss	CE-Loss	CE-Loss
Sampling+	standard	standard	oversampling	oversampling	oversampling
Batch Size+	1024	1024	1024	512	512
Learning Rate	0.1	0.1	0.1	0.01	0.1	0.1
Weight Decay	SE-3	SE-3	SE-3	SE-3	SE-3	SE-3
Data Augmentation	RandAugmentMC (CIFAR)	RandAugmentMC (CIFAR)	RandAugmentMC (CIFAR)	RandAugmentMC (ImageNet)	RandAugmentMC (ISIC)
Table 8: Final HPs for each dataset and label regime for our Self-SL models based on our HP tuning. Overall Performance was remarkably stable with regard to HPs and stronger augmentations did not necessarily improve performance in the same way as for ST models. This is presumably due to the pre-trained representations. HP denoted with a * are fixed across datasets and HP denoted with a + are pre-selected for each dataset while all other HP are obtained via sweeping.
Dataset Label Regime	CIFAR-10	CIFAR-100	CIFAR-10 LT	MIO-TCD	ISIC-2019
Low	Mid	High	Low	Mid	High	Low	Mid	High	Low	Mid	High	Low	Mid	High
Epochs*	80	80	80	80	80
Optimizer*	SGD Nesterov 0.9	SGD Nesterov 0.9	SGD Nesterov 0.9	SGD Nesterov 0.9	SGD Nesterov 0.9
Scheduler*	MultiStepLR	MultiStepLR	MultiStepLR	MultiStepLR	MultiStepLR
Warmup Epoches*	0	0	0	0	0
Loss*	CE-Loss	CE-Loss	CE-Loss	CE-Loss	CE-Loss
Sampling*	standard	standard	oversampling	oversampling	oversampling
Batch Size*	64	64	64	256	128
Learning Rate	0.001	0.001	0.01	0.01	0.001	0.001	0.01	0.001	0.01
Weight Decay	5E-3	5E-3	5E-4	5E-4	5E-3	5E-3	5e-3	5E-4	5e-3
Data Augmentation	RandAugmentMC weak (CIFAR)	RandAugmentMC weak (CIFAR)	Standard (CFAR)	RandAugmentMC weak (ImageNet)	Standard (ImageNet)
and select a number of epochs that always allow a complete fit of the model. The scheduler is also fixed across experiments as are the warmup epochs if used. Secondly, we pre-select the batchsize for each dataset since it is usually not a critical HP as long as it is big enough for BatchNorm to work properly.
ST For our ST models the final HP for each dataset and label regime are shown in Tab. 7.
HP sweep: weight decay: (5E-3, 5E-4); learning rate: (0.1, 0.01); data transformation: (RandAugmentMC, Standard)
Self-SL For our Self-SL pre-trained models the final HP for each dataset and label regime are shown in Tab. 8.
HP sweep: weight decay: (5E-3, 5E-4); learning rate: (0.01, 0.001); data transformation: (RandAugmentMC weak, Standard)
Semi-SL For our Semi-SL models we follow [52] with regard to HP selection as closely as possible. The final HP for each dataset and label regime are shown in Tab. 9.
HP sweep: weight decay and learning rate.
Table 9: Final HPs for each dataset and label regime for our Semi-SL models based on our HP tuning. HP denoted with a * are fixed across datasets and HP denoted with a + are pre-selected for each dataset while all other HP are obtained via sweeping. - denotes not performed experiments.
Dataset Label Regime	CIFAR-10	CIFAR-100	CIFAR-10 LT	MIO-TCD	ISIC-2019
Low	Mid	High	Low	Mid	High	Low	Mid	High	Low	Mid	High	Low	Mid	High
Optimization Steps*	2E5	-	2E5	-	2E5	-	2E5	-	2E5	-
Optimizer*	SGD Nesterov 0.9	-	SGD Nesterov 0.9	-	SGD Nesterov 0.9	-	SGD Nesterov 0.9	-	SGD Nesterov 0.9	-
Scheduler*	Cosine Annealing	-	Cosine Annealing	-	Cosine Annealing	-	Cosine Annealing	-	Cosine Annealing	-
Warmup Steps+	0	-	0	-	0	-	3000	-	3000	-
Loss+	CE-Loss	-	CE-Loss	-	weighted CE-Loss	-	weighted CE-Loss	-	weighted CE-Loss	-
Sampling*	standard	-	standard	-	standard	-	standard	-	standard	-
λu*	1	-	1	-	1	-	1	-	1	-
μ*	7	-	7	-	7	-	7	-	7	-
τ*	0.95	-	0.95	-	0.95	-	0.95	-	0.95	-
Distribution Alignment+	False	-	True	-	True	-	True	-	True	-
Batch Size*	64	-	64	-	64	-	64	-	64	-
Learning Rate	0.03	-	0.03	-	0.03	-	-	-	-	-
Weight Decay	5E-4	-	5E-4	-	1E-3	5E-4	-	-	-	-
Data Augmentation+	Standard (CIFAR)	-	Standard (CIFAR)	-	Standard (CIFAR)	-	Standard (ImageNet)	-	Standard (ISIC)	-
Unlabeled Augmentation+	RandAugmentMC (CIFAR)	-	RandAugmentMC (CIFAR)	-	RandAugmentMC (CIFAR)	-	RandAugmentMC (ImageNet)	-	RandAugmentMC (ISIC)	-
F Detailed results
F.1 Main results
General observations: For all datasets, the overall performance of models was primarily determined by the training strategy and the HP selection, with the benefits of AL being generally smaller compared to the proper selection of both. For the three toy datasets, Semi-SL generally performed best, followed by Self-SL and ST last, whereas, for the two real-world datasets, Semi-SL showed no substantial improvement over ST in the first training stage and, therefore, further runs were omitted. Also, the absolute performance gains for Self-SL models with AL are generally smaller compared to ST models. For Semi-SL, there were generally only very small performance gains or substantially worse performance with AL observed. Concerning the effect of AL, the high-label regime proved to work for ST models on all datasets and Self-SL models. On the two real-world datasets, MIO-TCD and ISIC-2019, a dip in performance at 7k samples for all ST models could be observed. This behavior is ablated in Appendix I.1.
Evaluation using the pair-wise penalty matrix: We use the pair-wise penalty matrix (PPM) to compare whether the performance of one query method significantly outperforms the others. It is essentially a measure of how often one method significantly outperforms another method based on a t-test with  $\alpha = 0.05$  (more info in [2, 4]). This allows to aggregate results over different datasets and label regimes, with the disadvantage being that the absolute performance is not taken into consideration. When reading a PPM, each row  $i$  indicates the number of settings in which method  $i$  beats other methods, while column  $j$  indicates the number of settings in which method  $j$  is beaten by another method.
We show the PPMs aggregated over all datasets and label regimes for each training paradigm in Figure 6.
For all methods, BADGE is the QM that is least often outperformed by other QMs. Further, for Self-SL models, it is never significantly outperformed by Random, whereas it is seldomly significantly outperformed for ST models. Based on this, we deem BADGE to be the best of our compared QMs for both ST and Self-SL models. Since BADGE is more often outperformed by Random (0.5) on the Semi-SL datasets and the additional high training cost for each iteration, we believe that Random is the better choice in many cases.
Evaluation using the area under the budget curve: For each of the following subsections, we added the area under the budget curve (AUBC) for each dataset and label regime to allow assessing the absolute performance each QM brings. Generally, higher values are better. For more information, we refer to [57, 58].
The results on the dataset for AUBC also show that BADGE is always one of the best performing AL methods. This is in line with the findings based on the PPM.
Figure 6: PPMs aggregated over all experiments for Standard Models (a), Self-S1 Pre-Trained Models (b) and Semi-SL models(c).
F.1.1 CIFAR-10
The AUBC values are shown in Tab. 10.
Table 10: Area Under Budget Curve values for CIFAR-10.
Training	Label Regime	Low-Label	Medium-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
ST	BADGE	0.4730	0.0105	0.7289	0.0025	0.8599	0.0016
BALD	0.4744	0.0106	0.7253	0.0051	0.8578	0.0017
Entropy	0.4307	0.0018	0.6859	0.0042	0.8498	0.0025
Core-Set	0.4681	0.0038	0.7282	0.0043	0.8629	0.0017
Random	0.4720	0.0144	0.7309	0.0068	0.8526	0.0030
Self-SL	BADGE	0.8282	0.0016	0.8728	0.0018	0.9086	0.0012
BALD	0.8005	0.0056	0.8692	0.0011	0.9093	0.0010
Entropy	0.8002	0.0090	0.8663	0.0017	0.9071	0.0010
Core-Set	0.8224	0.0026	0.8670	0.0009	0.9015	0.0009
Random	0.8117	0.0040	0.8669	0.0009	0.8989	0.0007
Semi-SL	BADGE	0.9349	0.0010	0.9488	0.0022	-	-
Entropy	0.9193	0.0082	0.9497	0.0007	-	-
Core-Set	0.9343	0.0018	0.9442	0.0007	-	-
Random	0.9326	0.0050	0.9478	0.0002	-	-
ST Results are shown in Figure 7.
Self-SL Results are shown in Figure 8.
Semi-SL Results are shown in Figure 9.
Figure 7: CIFAR-10 ST
Figure 8: CIFAR-10 Self-SL
Figure 9: CIFAR-10 Semi-SL
Figure 10: CIFAR-100 ST
F.1.2 CIFAR-100
The AUBC values are shown in Tab. 11.
Table 11: Area Under Budget Curve values for CIFAR-100.
Training	Label Regime	Low-Label	Medium-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
ST	BADGE	0.3525	0.0042	0.4855	0.0044	0.6627	0.0017
BALD	0.3586	0.0007	0.4865	0.0019	0.6658	0.0002
Entropy	0.3036	0.0095	0.4440	0.0007	0.6569	0.0021
Core-Set	0.3458	0.0010	0.4767	0.0031	0.6560	0.0004
Random	0.3599	0.0027	0.4791	0.0044	0.6474	0.0015
Self-SL	BADGE	0.5397	0.0030	0.6020	0.0019	0.6858	0.0021
BALD	0.5028	0.0043	0.5754	0.0027	0.6784	0.0009
Entropy	0.5111	0.0066	0.5857	0.0028	0.6857	0.0032
Core-Set	0.5337	0.0044	0.5917	0.0032	0.6804	0.0013
Random	0.5365	0.0017	0.5970	0.0017	0.6757	0.0013
Semi-SL	BADGE	0.5562	0.0033	0.6222	0.0041	-	-
Entropy	0.5328	0.0158	0.6152	0.0038	-	-
Core-Set	0.5220	0.0101	0.6083	0.0061	-	-
Random	0.5713	0.0066	0.6307	0.0016	-	-
ST Results are shown in Figure 10.
Self-SL Results are shown in Figure 11.
Semi-SL Results are shown in Figure 12.
Figure 11: CIFAR-100 Self-SL
Figure 12: CIFAR-100 Semi-SL
Figure 13: CIFAR-10 LT ST
F.1.3 CIFAR-10 LT
The AUBC values are shown in Tab. 12.
Table 12: Area Under Budget Curve values for CIFAR-10 LT.
Training	Label Regime	Low-Label	Medium-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
ST	BADGE	0.3788	0.0152	0.5887	0.0098	0.7577	0.0004
BALD	0.3919	0.0111	0.5935	0.0141	0.7590	0.0034
Entropy	0.3740	0.0064	0.5793	0.0114	0.7454	0.0023
Core-Set	0.3939	0.0249	0.6162	0.0200	0.7667	0.0058
Random	0.3615	0.0118	0.5446	0.0214	0.7263	0.0044
Self-SL	BADGE	0.5373	0.0233	0.6501	0.0026	0.7704	0.0093
BALD	0.5431	0.0202	0.6549	0.0097	0.7742	0.0036
Entropy	0.5282	0.0225	0.6450	0.0090	0.7707	0.0061
Core-Set	0.5298	0.0182	0.6171	0.0154	0.7555	0.0032
Random	0.5397	0.0208	0.6173	0.0097	0.7554	0.0069
Semi-SL	BADGE	0.7233	0.0166	0.7616	0.0087	-	-
Entropy	0.6934	0.0289	0.7590	0.0101	-	-
Core-Set	0.6825	0.0103	0.7608	0.0108	-	-
Random	0.6965	0.0264	0.7363	0.0077	-	-
ST Results are shown in Figure 13.
Self-SL Results are shown in Figure 14.
Semi-SL Results are shown in Figure 15.
Figure 14: CIFAR-10 LT Self-SL
Figure 15: CIFAR-10 LT Semi-SL
Figure 16: MIO-TCD ST
F.1.4 MIO-TCD
The AUBC values are shown in Tab. 13.
Table 13: Area Under Budget Curve values for MIO-TCD.
Training	Label Regime	Low-Label	Medium-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
ST	BADGE	0.3539	0.0041	0.4688	0.0153	0.6614	0.0080
BALD	0.3254	0.0155	0.4830	0.0092	0.6884	0.0104
Entropy	0.3201	0.0097	0.4134	0.0230	0.6078	0.0176
Core-Set	0.3514	0.0134	0.4678	0.0181	0.7098	0.0056
Random	0.3510	0.0151	0.4564	0.0140	0.6065	0.0120
Self-SL	BADGE	0.5446	0.0122	0.6365	0.0054	0.7174	0.0040
BALD	0.4494	0.0102	0.5741	0.0138	0.6041	0.0092
Entropy	0.5105	0.0092	0.6416	0.0075	0.6972	0.0029
Core-Set	0.5060	0.0082	0.5900	0.0190	0.6699	0.0166
Random	0.5298	0.0109	0.6124	0.0032	0.6975	0.0054
ST Results are shown in Figure 16.
Self-SL Results are shown in Figure 17.
Semi-SL We performed no AL experiments due to the bad performance of Semi-SL on the starting budgets. More information can be found in Appendix F.4.
Figure 17: MIO-TCD Self-SL
F.1.5 ISIC-2019
The AUBC values are shown in Tab. 14.
Table 14: Area Under Budget Curve values for ISIC-2019.
Training	Label Regime	Low-Label	Medium-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
ST	BADGE	0.3204	0.0099	0.4331	0.0146	0.5628	0.0101
BALD	0.3190	0.0133	0.4521	0.0211	0.5534	0.0052
Entropy	0.3241	0.0067	0.4207	0.0335	0.5631	0.0061
Core-Set	0.3426	0.0099	0.4501	0.0139	0.5708	0.0096
Random	0.3376	0.0243	0.4116	0.0201	0.5273	0.0048
Self-SL	BADGE	0.3809	0.0168	0.4679	0.0174	0.5761	0.0063
BALD	0.3949	0.0209	0.4847	0.0018	0.5914	0.0080
Entropy	0.3666	0.0165	0.4659	0.0104	0.5872	0.0096
Core-Set	0.3752	0.0205	0.4472	0.0071	0.5556	0.0069
Random	0.3736	0.0092	0.4555	0.0053	0.5547	0.0066
ST Results are shown in Figure 18.
Self-SL Results are shown in Figure 19.
Figure 18: ISIC-2019 ST
Figure 19: ISIC-2019 Self-SL Pre-Trained Models
F.2 Low-Label Query Size
To investigate the effect of query size in the low-label regime, we conduct an ablation with Self-SL pre-trained models on CIFAR-100 and ISIC-2019. For CIFAR-100 the query sizes are 50, 500 and 2000, while for ISIC-2019 they are 10, 40 and 160.
Here the accuracies for the overlapping labeled samples are shown which are analyzed using a t-test.
CIFAR100 The results are shown in Tab. 15 and Tab. 16. When comparing the performance of the same QM using different query sizes only BALD and Core-Set lead to statistically significant difference in performance. While it is consistent across both comparisons for BALD with the the performance difference widening for larger labeled sets and more iterations the trend for Core-Set is not as clear.
ISIC-2019 The results are shown in Tab. 17 and Tab. 18. Entropy is the only QM showing a significant difference, indicating that a query size of 40 outperforms a query size of 10 for a training set size of 160. However, this behavior does not extend to the other training set sizes. Whereas, for BALD, there is a consistent trend that smaller query sizes lead to increased performance.
Table 15: Accuracies % for the low-label comparison for CIFAR100 with query sizes 50 and 500 at overlapping training set sizes. Reported as mean (std). Values with a significant difference (t-test) across query sizes are denoted with  $*$ .
Labeled Samples	1000	1500
Query Size	50	500	50	500
BADGE	45.75 (0.75)	46.32 (0.20)	50.02 (0.92)	50.24 (0.61)
BALD	45.54 (0.20)	42.31 (1.71)	49.41 (0.39)*	46.00 (0.44)*
Core-Set	46.53 (0.76)	46.02 (0.74)	49.34 (0.72)	49.70 (0.60)
Entropy	43.82 (0.40)	42.95 (1.32)	47.05 (0.91)	46.75 (0.76)
Random	46.10 (0.41)	45.22 (0.34)	49.88 (0.26)	49.79 (0.27)
Table 16: Accuracies % for the low-label comparison for CIFAR100 with query sizes 500 and 2000 at overlapping training set sizes. Reported as mean (std). Values with a significant difference (t-test) across query sizes are denoted with  $*$ .
Labeled Samples	2500	4500
Query Size	500	2000	500	2000
BADGE	54.62 (0.60)	55.03 (0.38)	50.92 (0.12)	59.98 (0.60)
BALD	50.92 (0.30)	49.96 (1.43)	55.86 (0.21)*	52.14 (1.36)*
Core-Set	54.72 (0.16)*	53.52 (0.33)*	58.71 (0.61)	58.28 (0.43)
Entropy	51.41 (0.53)	51.79 (0.76)	57.12 (0.53)	57.53 (0.43)
Random	54.71 (0.38)	54.38 (0.21)	59.98 (0.60)	59.48 (0.48)
Table 17: Accuracies % for the low-label comparison for ISIC-2019 with query sizes 10 and 40 at overlapping training set sizes. Reported as mean (std). Values with a significant difference (t-test) across query sizes are denoted with  $*$ .
Labeled Sample	80	120	160	200	240
Query Size	10	40	10	40	10	40	10	40	10	40
BADGE	34.39 (0.86)	36.03 (0.39)	37.58 (1.93)	36.40 (1.10)	37.58 (2.51)	37.24 (2.15)	38.32 (1.59)	37.52 (1.16)	38.78 (1.64)	38.94 (2.97)
BALD	38.11 (1.42)	36.51 (1.26)	38.80 (1.19)	37.26 (4.90)	40.40 (1.76)	39.23(1.89)	42.15 (1.49)	40.09 (2.59)	42.26 (0.97)	40.18 (2.70)
Core-Set	35.57 (1.49)	35.38 (0.50)	36.87 (1.73)	36.52 (2.24)	38.55 (2.37)	38.19 (1.69)	38.41 (1.80)	37.19 (2.93)	39.04 (2.94)	38.30 (1.66)
Entropy	35.19 (0.63)	34.87 (0.91)	37.06 (1.63)	38.59 (2.06)	36.95 (1.13)*	39.67 (0.55)*	38.25 (3.11)	39.93 (2.38)	39.38 (2.62)	40.26 (2.49)
Random	36.10 (0.70)	35.69 (0.68)	37.57 (1.79)	37.65 (2.15)	37.43 (1.77)	40.62 (0.62)	38.41 (1.46)	40.29 (1.78)	3835 (2.43)	41.14 (0.37)
Table 18: Accuracies % for the low-label comparison for ISIC-2019 with query sizes 40 and 160 at overlapping training set sizes. Reported as mean (std). Values with a significant difference (t-test) across query sizes are denoted with * .
Labeled Samples	200	360
Query Size	40	160	40	160
BADGE	37.52 (1.16)	38.11 (1.07)	42.90 (3.29)	42.86 (1.44)
BALD	40.09 (2.59)	39.67 (2.54)	44.49 (1.78)	42.83 (1.05)
Core-Set	37.79 (2.93)	37.71 (3.18)	39.56 (1.48)	39.47 (1.04)
Entropy	39.93 (2.38)	37.80 (1.46)	42.03 (1.16)	40.82 (1.46)
Random	40.29 (1.78)	40.80 (2.34)	42.10 (2.29)	42.10 (2.29)
Figure 20: Macro-averaged F1-scores for ISIC-2019 and MIO-TCD datasets. Across the board, BADGE is still the best-performing QM which can also be seen in Table 19. Please interpret the results with care, as the model configurations are optimized for balanced accuracy.
Table 19: Area Under Budget Curve values based on macro-averaged F1-scores for ISIC-2019 and MIO-TCD (corresponding plots in Figure 20).
(a) ISIC-2019
Training	Label Regime	Low-Label	Mid-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
Standard Training	BADGE	0.2791	0.0081	0.3810	0.0167	0.5059	0.0054
BALD	0.2587	0.0196	0.4050	0.0216	0.4866	0.0100
Entropy	0.2652	0.0147	0.3763	0.0453	0.4988	0.0056
Core-Set	0.2800	0.0105	0.3937	0.0082	0.5139	0.0141
Random	0.2817	0.0153	0.3638	0.0292	0.4774	0.0035
Self-SL Pre-Trained	BADGE	0.3390	0.0107	0.3978	0.0053	0.5380	0.0067
BALD	0.3273	0.0054	0.4065	0.0047	0.5173	0.0008
Entropy	0.3232	0.0039	0.4081	0.0061	0.5586	0.0143
Core-Set	0.3010	0.0129	0.3833	0.0094	0.5045	0.0123
Random	0.3333	0.0033	0.3852	0.0153	0.5100	0.0059
(b) MIO-TCD
Training	Label Regime	Low-Label	Mid-Label	High-Label
Query Method	Mean	STD	Mean	STD	Mean	STD
Standard Training	BADGE	0.2503	0.0111	0.3604	0.0249	0.5759	0.0075
BALD	0.2023	0.0127	0.3325	0.0082	0.5758	0.0116
Entropy	0.1957	0.0208	0.2929	0.0218	0.5066	0.0290
Core-Set	0.2301	0.0155	0.3450	0.0203	0.5945	0.0099
Random	0.2446	0.0091	0.3438	0.0132	0.5085	0.0093
Self-SL Pre-Trained	BADGE	0.4295	0.0205	0.5487	0.0129	0.5994	0.0035
BALD	0.2560	0.0132	0.4488	0.0236	0.4335	0.0088
Entropy	0.3398	0.0104	0.4977	0.0095	0.5481	0.0078
Core-Set	0.3525	0.0066	0.4970	0.0160	0.5375	0.0138
Random	0.4175	0.0158	0.5194	0.0101	0.5681	0.0036
F.3 Macro averaged F1-scores
Additionally, we provide the macro-averaged F1-scores for ISIC-2019 and MIO-TCD dataset. A plot showing the performance of both ST and Self-SL models is shown in Figure 20 and the resulting AUBC values are shown in Tab. 19.
F.4 Semi-Supervised Learning
Results of FixMatch for all HPs on the whole validation splits are shown separately for MIO-TCD in Tab. 20 and ISIC-2019 in Tab. 21. Based on the performance which did not improve substantially over even ST models we decided to omit all further AL experiments.
Table 20: MIO-TCD FixMatch results reported on the test sets (balanced accuracy in%). Reported as mean (std).
FixMatch Sweep MIO-TCD
Labeled Train Samples	Learning Rate	Weight Decay	Balanced Accuracy (Test)
55	0.3	5E-3	18.1(1.1)
5E-4	28.0(0.4)
0.03	5E-3	26.8(0.5)
5E-4	29.7(2.4)
275	0.3	5E-3	20.2(6.2)
5E-4	28.2(1.5)
0.03	5E-3	31.4(0.7)
5E-4	36.0(1.8)
Table 21: ISIC-2019 FixMatch results reported on the test sets (balanced accuracy in %). Reported as mean (std).
FixMatch Sweep ISIC-2019
Labeled Train Samples	Learning Rate	Weight Decay	Balanced Accuracy (Test)
40	0.3	5E-3	14.0(2.5)
5E-4	26.9(1.5)
0.03	5E-3	24.5(4.4)
5E-4	31.3(1.7)
200	0.3	5E-3	15.2(3.6)
5E-4	19.1(1.5)
0.03	5E-3	26.3(0.8)
5E-4	24.3(3.6)
G Discussion and further observations
The results are interpreted based on the assumption that a QM performing on a similar level as Random is not a drawback as long as it brings in other settings performance improvements over random queries. This mostly follows in line with the PPM as a performance metric but mostly focuses on the row that compares each QM with random queries. However, if a QM shows behavior leading to much worse behavior than random as Entropy does or shows signs of the cold start problem, we deem this as highly problematic. In these settings, one loses significant performance whilst paying a cost in computing and setup corresponding to AL. Therefore, we use random queries as a baseline for all QMs.
Based on this our recommendation for BADGE is given for Self-SL and ST trainings.
The main disadvantage of this approach is that absolute performance difference are not captured in this aggregated format.
H Comparing random-sampling baselines across studies
Here we compare the performance of random-sampling baselines on the most commonly utilized dataset CIFAR-10 and CIFAR-100 across different studies for ST, Self-SL and Semi-SL models along strategic point where overlap in between papers occurs. For CIFAR-10 the results of this comparison are shown for the high-label regime in Tab. 22 and the low- and mid-label regime in Tab. 23. Similarly for CIFAR-100 the results are shown in Tab. 24 for the high-label regime and Tab. 25 for the low- and mid-label regimes. Overall our ST random baselines outperform all other random baselines. Our Self-SL models also outperform the only other relevant literature [6] on CIFAR-10. Further, our Semi-SL models also outperform the relevant literature [20, 43] on CIFAR-10 and CIFAR-100.
Table 22: Comparison of random baseline model accuracy in % on the test set for the high label-regime for CIFAR-10 across different papers. Best performing models for each training strategy are highlighted. Values denoted with - represent not performed experiments. Values with a denoted with a * are reprinted from [44]. Values which are sourced from a graph are subject to human read-out error.
Information
Paper	Training	Model	Source	Number Labeled Training Samples
1k	2k	5k	10k	15k	20k
QBC	ST	DenseNet121	Graph	74*	82.5*	-	-
VAAL	ST	VGG16	Graph	-	-	61.35*	68.17*	72.96*	75.99*
CoreSet	ST	VGG16	Graph	-	-	60*	68*	71*	74*
Agarwal et al.	ST	VGG16	Graph	-	-	61.5	68	72	76
Munjal-SR	ST	VGG16	Table	-	-	82.16	85.07	89.43	91.16
Mittal et al.	ST	WRN28-2	Graph	57	73	82.5	86	90.7	92
LLAL	ST	ResNet18	Graph	51	63	81*	87*	-	-
CoreCGN	ST	ResNet18	Graph	50	64	80*	85.5*	-	-
TA-VAAL	ST	ResNet18	Graph	50	65	81*	87.5*	-	-
Krishnan et al.	ST	ResNet18	Graph	47	60	78	86	-	-
Yi et al.	ST	ResNet18	Graph	47.5	56	78	86	-	-
Bengar et al.	ST	ResNet18	Graph	45	55	73	81	85	88
Beck et al.	ST	ResNet18	Graph	55	-	-	84	85	90.5
Zhan et al.	ST	ResNet18	Graph	45	-	-	76	-	-
Munjal-SR	ST	ResNet18	Table	-	-	84.69	88.45	89.98	92.29
Ours	ST	ResNet18	Table	72.4	79.8	85.5	90.5	-	-
Bengar et al.	Self-SL	ResNet18	Graph	87	88	89.5	90.5.	91	91.5
Ours	Self-SL	ResNet18	Table	86.2	88.3	90.1	91.4	-	-
Mittal et al.	Semi-SL	WRN28-2	Graph	88	91	92.5	93.8	94	94.5
Gao et al.	Semi-SL	WRN28-2	Graph	91.5	91	-	-	-	-
Ours	Semi-SL	ResNet18	Table	94.7	95.0	-	-	-	-