Best Practices in Pool-based Active Learning for Image Classification
Authors: Adrian Lang
Christoph Mayer
Radu Timofte
Venue: ICLR 2022 Submitted (2022)
Keywords: Active Learning
Deep Learning
Image classification

Abstract: The recent popularity of active learning (AL) methods for image classification using deep-learning has led to a large number of publications that lead to significant progress in the field. Benchmarking the latest works in an exhaustive and unified way and evaluating the improvements made by the novel methods is of key importance to advance the research in AL. Reproducing state-of-the-art AL methods is often cumbersome, since the results and the ranking order of different strategies are highly dependent on several factors, such as training settings, used data type, network architectures, loss function and more. With our work we highlight the main factors that should be considered when proposing new AL strategies. In addition, we provide solid benchmarks to compare new with existing methods. We therefore conduct a comprehensive study on the influence of these key aspects, providing best practices in pool-based AL for image classification. We emphasize aspects such as the importance of using data augmentation, the need of separating the contribution of a classification network and the acquisition strategy to the overall performance, the advantages that a proper initialization of the network can bring to AL. Moreover, we make a new codebase available, that enables state-of-the-art performance for the investigated methods, which we hope will serve the AL community as a new starting point when proposing new AL strategies.

Under review as a conference paper at ICLR 2022
BEST PRACTICES IN POOL-BASED ACTIVE LEARNING FOR IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
The recent popularity of active learning (AL) methods for image classification using deep-learning has led to a large number of publications that contributed to significant progress in the field. Benchmarking the latest works in an exhaustive and unified way and evaluating the improvements made by the novel methods is of key importance to advance the research in AL. Reproducing state-of-the-arr AL methods is often cumbersome, since the results and the ranking order of differ- ent strategies are highly dependent on several factors, such as training settings, used data type, network architectures, loss function and more. With our work we highlight the main factors that should be considered when proposing new AL strategies. In addition, provide solid benchmarks compare new with existing methods. We therefore conduct a comprehensive study on the influence of these key aspects, providing best practices in pool-based AL for image classification. We emphasize aspects such as the importance of using data augmentation, the need of separating the contribution of a classification network and the acquisition strategy to the overall performance, the advantages that a proper initialization of the network can bring to AL. Moreover, we make a new codebase available, that enables state-of-the-ar performance for the investigated methods, which we hope will serve the AL community as a new starting point when proposing new AL strategies.
INTRODUCTION
Deep learning methods require large amounts of labeled data samples to train. Unfortunately. anno- tating new datasets consisting of thousands or millions of images very costly. A research topic that focuses on maximizing the performance of deep learning models with given annotation budget is active learning (AL). AL is machine learning's sub-field in which the algorithm is allowed to query an information source for the label of new data samples (Settles 2009). Recently, abundance of pool-based AL methods for image classification have been proposed (Caramalau et al., 202 b) Kim et al. 2021 Liu et al. 2021 Choi et al., 2021). In pool-based AL (Lewis & Gale 1994) there exists large unlabeled and a small labeled sample set L. The labeled set is used to train the classifier and the large unlabeled set is used query new samples that then annotated and added to the labeled set in order to include them for training. The goal of pool-based AL is to iteratively sample and annotate unlabeled training samples from the pool and add them to the training set in order to achieve the best performance given a certain annotation budget. Pool based AL is often particularly useful because large amounts of unlabeled data is often available but annotating it is cumbersome and costly.
In AL there exist multiple baselines that either form lower or upper bounds for AL methods. Typi- cally, using all the unlabeled samples, annotate them and include them for training forms an upper bound that AL methods aim to reach with as few samples as possible. In contrast, selecting new samples that should be labeled and used for training randomly from the set of unlabeled samples establishes a lower bound that any AL method should exceed in every AL cycle. AL methods can be split into uncertainty diversity based samples. The former select samples for which the classifier
uncertain (Gal et al. 2017; Beluch et al., 2018 Yoo & Kweon, 2019 Mayer & Timofte 2020).
while the latter selects samples in a diverse way (Sener & Savarese 2018 [inha al. 2019 Gissin & Shalev- Shwartz 2019 Caramalau et al.] 2021a), ensuring to cover the whole data distribution of the unlabeled set. Recently proposed strategies, are often still based these concepts, combine

---

Under review as a conference paper at ICLR 2022
them (Ash et al.
2019 Zhang al., 2020 Kim et al. 2021) or propose new ideas such as using
unlabeled data to train addition modules (Shui et al. 2020; Caramalau et al. (021b) used for sam-
ple selection combine AL with semi-supervised learning (Sim' coni et al. 2021 Huang et al. 2021 Gao et al. 2020) Mittal et al., 2019 Liao et al. 2021). The recent popularity of AL methods complicates comparing these works in a complete and standardized manner. However, assessing the improvements made by new methods is crucial to boost the progress in AL. Hence, this work provides best practices to keep in mind, when proposing new pool-based AL methods for image classification using deep learning and comparing them to existing methods form the literature.
Contributions: In summary, our contributions are as follows: (i) We perform an extensive analysis of popular AL methods and study the effect of different settings typically used in the literature. (ii) In addition, we discuss the merit of recent trends in AL such as using unlabeled data, pretraining or initial set construction. (iii) We incorporate our findings into a collection of best practices to keep in mind when evaluating pool based AL methods for image classification. (iv) We make a new Pytorch code base available I For AL that contains many popular AL methods and datasets and allows to evaluate and compare different strategies in a fair way. Code will be released upon publication.
RELATED WORK
AL evaluations: Recently the attention on benchmarking and unifying results of new AL strategies increased. Beck et al. (202T) conducted a rich set of experiments to evaluate the performance of AL methods, including (Wei et al., 2015) Sener & Savarese 2018 Ash et al. [219 Killamsetty jtt al. 2021). They assessed the robustness of the methods by adding redundancy to the datasets, concluding that diversity based strategies are more stable in this scenario. In addition, they studied the effect of using data augmentation, using different optimizers, updating the models instead of re- initializing them in every iteration, and using differrent batch sizes. In contrast, perform similar experiments but include more AL methods on multiple datasets, evaluated additional settings, and study the effect of using unlabeled samples for AL. Munjal et al. (2020) focused benchmarking the strategies proposed in (Gal et al.] [2017] [Blluch et al. [018]] [Snner& & Savarese (2018) [inha et al,, 2019) using a uniform experimental setting. They vary the AL batch size, the validatioo set size, and the amount of class imbalance. They find that under changing experimental condition no strategy offers gain over Random sampling. Conversely, our experiments show that AL methods clearly outperform Random sampling when training the classifier carefully. Chong et al. (202I) evaluates traditional AL methods and Coreset on an imbalanced medical dataset (Rahman et al. 202T] and on CIFARIO. They highlight the importance of respecting class imbalance in AL and conclude that Coreset performs better than Uncertainty sampling- In contrast to these last two works, we omit analyzing the robustness of AL strategies on imbalanced datasets but rather focus balanced datasets such as CIFARIO.
Investigated AL strategies: The investigated AL strategies are a relevant selection of recent deep pool based AL methods used in deep learning. In particular, study Coreset, BADGE, LL4AL, JLS and WAAL, and Uncertainty sampling. We briefly summarize the most recent ones. Coreset (Sener& & Savarese 2018) selects new unlabeled samples by solving the k-Center problem (Woli 2011) on the feature space of the last fully-connected layer. [Ash et al. (2019) propose an acquisition strategy (BADGE) based on k-MEANS++ (Arthur & Vassilvitskii 2007), which is applied on so called hallucinated gradients with respect to the parameters of the final layer of the deep network. Hallucinated gradients are used because the true labels are not available for unlabeled samples. Therefore, the most likely labels according to the classifier are used to compute the gradient. Yoo & Kweon 2019) propose LL4AL, based on the assumption that data points for which the classification network is likely to produce wrong prediction are most informative and should be used for training. To this end, they jointly train a target module and a loss prediction module that aims at predicting the loss of a given unlabeled sample. The samples achieving the highest losses are then added to the training set. JLS (Caramalau et al. (02 employs a discriminator network besides the classifier. The task of the discriminator is differentiating whether a given sample belongs to the set of labeled or unlabeled samples. The samples that the discriminator predicts to be unlabeled with high con- fidence are selected and annotated. JLS shares intermediate backbone layers for classification and discrimination such that training the discriminator with unlabeled data turns out to be beneficial for
The code will be made available after publication.
2

---

Under review as a conference paper at ICLR 2022
the classifier as well. Shui et al. (2020) propose an acquisition strategy (WAAL) that combines the uncertainty score of the task model with a diversity term based on the Wasserstein metric. To com- discriminate between
pute the diversity metrie, adversarial network is used that is optimized labeled and unlabeled data. Both networks share the same feature extractor and are jointly trained through a min-max optimization problem that uses labeled and unlabeled data.
3 EXPERIMENTAL SETUP
Since the accuracy range of AL methods extensively changes across the literature and their pro- vided codebases, we provide our own that aims at maximizing AL. accuracy for all the implemented methods. Our codebase builds upon repository which only implements the LL4AL strategy for CIFARIO and achieves the result presented (Yoo & Kweon, 2019).. We re-implement multiple other methods such as Random sampling. Uncertainty sampling, Coreset (greedy K-center variant), BADGE, JLS, VAAL and include them into the codebase. Furthermore, we adapt the codebase in order to support additional datasets such as SVHN and FashionMNIST.. The different methods can be tested on any dataset and our codebase allows to examine the effect of various different settings such warm cold start, different classifiers, unsupervised pretraining, changing the acquisition strategy to sample the initial dataset, different training settings, such as jointly training a discrimi- nator or a loss prediction module for all methods.
We use the settings in (Yoo & Kweon 2019) because they achieve results with high accuracy on CIFARIO for all the reported methods. Note that the default settings serves mainly as comparison benchmark, and does not lead to the highest absolute accuracy for each method across all datasets. Unless stated otherwise, our default training setting is as follows: We use ResNet-18 (He et al.] 2016) as backbone and train with mini-batch size of 128, for 200 epochs, with SGD optimizer, an initial learning rate of 0.1, momentum of 0.9 and weight decay of 0.0005. After training for 160 epochs, we decrease the learning rate to 0.01. Instead of initializing the network weights in every AL cycle we use the weights of the network obtained in the previous AL cycle (warm start). We use data augmentation (random horizontal flips and random crop) and normalize the input images. The initial training set consists of 1000 labeled samples. We add 1000 samples in each AL step, except for the experiments focusing the of unlabeled data where use the JLS and WAAL settings.
Our codebase allows to reproduce state-of-te ofate-of-the-ar results for the aforementioned methods for all three datasets, for different training setting and annotation budgets. All results reported in the paper are averaged over three runs with different random seeds.
For the experiments use three common datasets used in the AL literature: CIFAR10 (Krizhessky) 2009), consisting of 60k 32 x 32 RGB images, divided into 50k training and 10k testing images split into 10 classes. The datasets are balanced, i.e. each class has the same number of images. SVHN (Netzer et al. 2011) contains images showing house numbers obtained from Google Street View. It consists of 73,257 training, 26,032 testing, and 53,1131 extra training images. The RGB images have a resolution of 32 x 32, are centered around single digit and are split into 10 classes. The extra training images are not used in this work. FashionMNIST (Xiao et al., 2017) contains 28 x 28 gray-scale images showing articles from an online shop. The dataset consists of 60k training and 10k testing images.
shows the results when using our codebase and the default settings three datasets. In the paper we often report only the accuracy difference plots but omit the figures showing the absolute single dataset, or omit other AL strategies such as
Figure S
accuracy, or we only show the result VAAL or JLS to keep the figures clean, or we only report the mean but omit the standard deviation. For additional plots containing all these results refer the reader to the Appendix.
4 DIFFERENT TRAINING SETTINGS AND THEIR EFFECT ON AL EVALUATION
Newly proposed AL strategies typically come with their own hyper parameters, training strategies and model architectures in order to outperform existing methods on the same dataset. However, changing these aspects complicates comparing different AL methods with each other. For exam- ple, when using another classification network (backbone) than existing methods fair comparison
hhans://gittub.co////ppsssss/// www.aoonoonoo...o---aaaeaao Laaming-Loss-foo-Actteee
3

---

Under review as a conference paper at ICLR 2022
Table I: Comparison of existing AL methods in terms of model architecture, optimizer, learning rate, whether data augmentation is used or not and whether cold or warm start is used to initialize the network parameters. " means that the information was not mentioned.
Concect
VAAL
BATC
LL4AT
JES.
WAAL
SRAAI
TA-VAAI
ISAL
FKSSAL
OCN
VaB-Al
Scnor
Suvaroed
Sinte at
Ashrrtal
You
Excon
Camaualan
Stares
Plangeta at
Kiures
Gunkorsity
Cles
[
Backhone
VGG 16
VGG 16
ReiNet 15
KeeNer 15
VGG 16
VGG 16
ResNer 18
Recher 18
ReiNer 18
ReiNNe 18
ReiNer 18
ResNerl
Optmmize
SGD
Adam
SGD
SGD
SGD
SGD
SGD
SGU
SGD
SGD
LK
0.001
0.01
0.01
0.1
001
001
0.1
0.1
0.1
0.1
0.1
Data Ang
X
Start
Cold
Cold
Cold
Wamm
Cold
Cold
Wtrm
Cold
Cold
Cold
Cold
96
94-
8.0
AccurHt (16)
92
7.0
ba dae
90
badge
bo dge random
60-
candom
88
randam
uncertainty
itt
188
uncertainty coreset
50
coreset
coreset
LL4AL
84
LL4AL
86
LL4AL
40
supervised
82
supervised
supervised
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
2000
4000
600D
8000
10000
Number of labiled data
Number labeled data
Number of labeled data
CIFARIO
(b) SVHN
(c) FaSHi
Figure 1: Results for the default setting of the AL strategies using our codebase.
requires to re-implement these methods or change the provided source code. However, the default hyper parameters and training settings may not be optimal for the new backbone. In Table sum- marize settings for recent AL methods. We observe multiple choices for all the reported settings. In order to enable fair comparison among AL methods unified evaluation strategy is required. Hence, we examine in this section the effect on AL when using the different settings by changing the default values in our proposed code base. In particular, we study whether changing the settings affect only the overall accuracy, leads to change in the ranking of AL methods or even results into vanishing benefit of AL. We compare four existing AL methods (LL4AL, BADGE, Uncertainty sampling and Coreset) with each other and the baseline of Random sampling. Finally, we compare our code base with the code base used for BADGE that not only provides the implementation of BADGE but also of few existing methods.
Backhone Architecture: First, we investigate the impact on AL performance when changing the backbone. In particular, we compare five methods on CIFARTO using either VGG-16, ResNet-18 or De1. 2 the achieved accuracy for the different backbones. We conclude, that different backbones achieve different classification accuracy, with ResNet-18 achieving the highest. Hence, it is crucial to ensure that only methods using the same backbone are compared with each other. In addition, we observe that the ranking order of AL methods changes for Densenet-201 compared ResNet-18 and 16.-16. This probably due to overall low accuracy leading to poor sample selection. Therefore, we propose to validate new methods on ResNet- 18 since it achieves the highest accuracy and significant improvements over Random sampling for all investigated methods. For methods that benefit from using other backbones we recommend to validate for this case on both the new backbone and R8 to enable a fair and insightful comparison.
Initializing Backbone Weights: In AL the classifier is re-trained in every active learning cycle to leverage the newly sampled and annotated data. According to Table T two different strategies are common in the literature: cold and warm start. Cold start describes the approach to re-initialize the network weights in every cycle from scratch. In contrast, warm start uses the network weights obtained in the former AL. cycle by training with the corresponding annotated samples. Warm start resembles fine-tuning but we use the same learning rate and number of training epochs for cold and warm start. Figure 3 shows the accuracy achieved with warm start minus the accuracy of cold start three datasets. We observe that depending the dataset either cold or warm start performs better in the first few AL cycles but the difference diminishes on all datasets for almost all methods. Except LL4AL that performs better with warm start across all datasets. Figure 4c compares the performance of LL4AL with other AL methods when all use cold start and we observe that LL4AL no longer outperforms other AL methods (see figure Tab but achieves comparable accuracy.
4

---

Under review as a conference paper at ICLR 2022
90
90
90
80
80
Accuraty D60
Ac20080011 (%0
Accurso
88
random
50
uncertaiity
50
uncortainty
50
uncerfartty
cornset
40
40
2000
4000
8:000
8000
10000
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Number of labeled data
Number labe led data
Number of labeled data
(a) ResNet-18
(b) VGG-16
(c) Densenct-201
Figure 2: Results when employing different backbones as classifiers.
uncoriancy acoracy
corcsst aecutacy o'
Ascurc gy fiforenee
LLAAL accuracy d!!
1:29
2.00
2e0p scy
15
badge accurasy dil
randem dif
usoetta sccurey
1.44L
0.5
-2.9
accurecy dif
2000
40c0
2000
0902
30300
2000
4000
0002
8002
10020
Numbst cflabeted coto
hunbar
lutedol Jute
Number of labeled data
(a) CIFARIO
(b) SVHN
(c) FashionMNIST
Figure 3: Accuracy difference between Warm-start and Cold-start.
Hence, we conclude that cold or warm start influences AL performance in particular at the beginning of AL but the difference tends to diminish later on. To ensure that the improvement of AL meth- ods originate from a superior sample selection strategy instead from improved settings method, advise
the approach which works best for the investigated method. For LL4AL advise to always use warm start, while for the other methods it depends on the annotation budget and on the dataset.
Optimizer and Learning Rates: Different optimizers such as SGD, Adam and RMSProp with varying learning rates from 0.001 to 0.1 are commonly used in the literature (see Table T. First, investigate the effect of varying learning rates when using the most popular optimizer SGD. We perform experiments when using cold and warm start to initialize the network weights, motivated by the assumption that different initialization approaches may require different learning rates. Fig- ures 4a and 4b show the accuracy achieved with learning rate of 0.1 minus the accuracy with 0.01 for CIFARIO, IO. We observe that using larger learning rate beneficial for both, cold and warm start. In addition, smaller learning rate leads to higher classification accuracy when training only on the initial dataset.
Furthermore, we perform experiments with two different optimizers: SGD and Adam. We tested different learning rates used the literature (see Table [ and use the learning rate 0.1 that leads to the best performance for either optimizer (see A.13 for more details). Figure shows the accuracy achieved with SGD minus the accuracy achieve when using Adam. We observe SGD leads in particular CIFARIO 10 to superior performance by achieving roughly 6% points higher accuracy than training with Adam. On SVHN and FashionMNIST SGD outperforms Adam as well but the difference is smaller with 0.5 1.0% 0.5 1.5% points.
Hence, we conclude that using the same optimizer and learning rate to train the backbone when com- paring different AL methods is crucial. Otherwise it is unclear whether the performance difference stems from the AL method or the training setup.
With and Without Data Augmentation: Whether to train the backbone with or without data aug- mentation is another setting that often differs between different AL works (see Table T.. Disabling lata--uugmnntttion typically decreases the overall accuracy of the classifier (see figure 7 and can feet the ranking order of AL. methods. Figure Ta shows the result code base with and without
5

---

Under review as a conference paper at ICLR 2022
basge acursiy dff random accurasy dif
Accomry difereere
uncertatity sccureyy of? coense sccurery
80
LLAK acnurary
badge
50
uncertaint, nty
uncedatity
coreset
LIAAL
2030
4.000
6020
12003
2000
4500
6262
0003
20000
2000
4000
6000
8000
10000
Number oflabeled data
Number of labeled data
Number labeled dalta
(a) Difference for cold start
(b) Difference for warm start
(c) Accuracy for Cold start
Figure 4: Accuracy difference between Ir = 0.1 and Ir = 0.01 with Warm-start, and Cold-start.
Results on CIFARIO when using Cold-start approach and the default learning rate of 0.1.
bedge netem seourey dTf uncertarry scoursy di corost recuray alf Li4AL accuracy di!
Accurery difeerecee 1961
3.0
2.5
0.5
Secge Uccaiety def
-3.0
1rnden Securnty
-15
LLAKL
2030
-2.
Number atlabeled data
Numb or of lnbeled cafa
Number labeled dats
CIFARIO
(b) SVHN
(c) MNSTI
Figure 5: Accuracy difference when using either SGD or Adam as optimizer.
90
80
70-
Accuracy
60
Accuracy
60
concat
50
50
1e000 (RADGE))
conse (WASE))
badge
uncerakiity 3ug careset
40
badge aug
40
randomm
connost no aug
30
rendom
L1AM
30
unceriainty
LLA.
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Number
Number
(a) With or without data augmentation
(b) Two different code bases
Figure 6: Results on CIFARIO for different code bases or with and without data augmentation.
data augmentation on CIFARIO. We observe, that the accuracy is least 10% points lower when not using data augmentation and can differ up to 20% points and more. In addition, all AL meth- ods outperform random sampling when using data augmentation. Conversely, the baseline random sampling outperforms all AL methods at least once on different operating points when data augmen- tation is not used. Hence, disabling data augmentation not only decrease the overall performance but also leads to failure of multiple AL methods.
Supported by the above observations we therefore recommend data augmentation when comparing AL for image classification.
Having examined all the aforementioned settings for AL it becomes clear that these settings are crucial when comparing AL methods. While it is important to enable a fair environment when comparing different methods, aims at maximizing the overall accuracy. In this part, we want to highlight the importance of such codebase by comparing to the codehase used for BADGE. In contrast to codebase they
Comparing Codebases:
argue that it is useful use codebase that

---

Under review as a conference paper at ICLR 2022
275
20-
accurs offter dree
coresst acuratt dif
ULALL scorrcy dcy dif
badge accursy
20-
budge occurcc dil
randam acoursyy
d!f
LLGAL 41f
2002
4200
2002
8920
10200
2002
4200
5000
8020
10002
0.
2000
4000
5000
3000
10002
Number of dilabbed daia
Number of lubeled data
Number
labelee data
(a) CIFARIO
(b) SVHN
(c) FaSHI
Figure 7: Accuracy difference with data augmentation enabled or disabled.
95
E5-
Acouncy
Accuracy
70
80-
70-
badge
randor
uncortannty WAAL
uncertainty
75
coeesst
arch
coresel
J5
60
40
70
5000
75.00
10000
12520 19000 17500
20500
2000
4000
60m0
B0od
10000
12000
2000
4000
6:000
8000
10000
Number labeled data
Number labeled data
Number of labeled data
(a) With and without JLS Arch. (b) With and without WAAL Arch. (c) With the loss prediction module
Figure 8: Results when jointly training the discriminators for different methods and their perfor- mance without training it or when jointly training the loss prediction module for all methods.
use Adam instead of SGD, disable data augmentation and train for fewer epochs in each AL cycle and employ early stopping. Figure 6b shows the results of four AL methods using our codebase in solid lines and the results of BADGE's codebase in dashed lines. We observe that our codebase leads to much higher accuracy. In particular we achieve between 20% to almost 50% points higher classification accuracy CIFARIO using the same annotation budget. Furthermore, we observe that every AL method outperforms Random sampling when using our codebase. In contrast, the baseline often outperforms AL. methods when using the codebase of BADGE and therefore results in AL failure. However, this not problem the AL methods but caused by the codebase. We conclude that poorly trained backbones perform poorly for sample selection and therefore result low performance of the AL strategy. Furthermore, it is clear that codebase achieving too low classification accuracy is unsuitable for fair comparison of different AL methods.
5 USING UNLABELED DATA
In this section we focus on the use of unlabeled data. Recent AL methods such as VAAL, LL4AL or JLS employ additional module that is trained in each active learning cycle and used to query new samples. While most AL methods do not use unlabeled data to train the classification network (e.g.. VAAL), or only use labeled data (e.g., LL4AL), there exist methods (e.g., JLS, WAAL) that unlabeled data to train the backbone. Hence, comparing methods that use unlabeled samples with methods that only use labeled samples is unfair. In order to compare such methods with existing ones, we propose to train the backbone of all methods with unlabeled samples. We perform this experiments using the training strategies of JL.S and WAAL. In particular, we use exactly the same network for each AL strategy including the discriminators but we use the original sample selection strategy when querying new samples each AL cycle. Figure Sal shows a comparison between the performance of AL methods when training the classifier and discriminator with unlabeled data (dashed lines) compared to their performance when only the backbone trained (solid lines). We observe that traditional methods such Coreset and Uncertainty sampling outperform JLS when using the same networks and train with unlabeled samples. Figure 85 shows similar results for WAAL using their codebase instead of ours.

---

Under review as a conference paper at ICLR 2022
6 EFFECT OF ADDING MODULES FOR COMPLEMENTARY TASKS
In this section, we demonstrate, that it is important to separate the achieved improvement caused by the acquisition strategy and better trained classifier when learning additional tasks beside clas- sification. LL4AL trains the backbone combined with the loss prediction module, optimizing thus a different loss function. To fairly asses the benefit of their acquisition strategy, we train each AL method using exactly the same architecture in LLLAI,, but use the original sampling strategy when querying new images. In figure 8c all methods train jointly also the loss prediction mod- ule. Comparing the results to the default case, depicted in figure La] we observe that LL4AL is not achieving the top performance anymore. The other methods achieve comparable accuracy. This ex- periment shows that it not the novel acquisition strategy proposed in LL4AL that leads to superior performance, but it is in fact the classifier that benefits of learning an additional task and results in better generalization.
7 BEFORE ACTIVE LEARNING STARTS
In this section we discuss two approaches that can influence the performance of AL methods, by either building the initial labeled dataset following a certain strategy instead of random sampling of the data or by pretraining the backbone in an unsupervised setting.
Construction of the Initial Training Set: Instead of randomly selecting the samples for the initial training set, it is possible to use more sophisticate approach to construct the dataset. In this part, investigate whether it is worth focusing on selecting such a set and if potential benefit is preserved over multiple AL cycles or diminishes quickly. In particular, we use a pretrained VGG-16 on ImageNet (Deng et al., 2009) and compute the features of its last layer for every image in the datasets. We then use K-means++ and greedy K-center on the extracted features to select the set of initial training samples. We evaluate different AL methods and report figure 9 the accuracy of randomly created initial sets minus the accuracy achieved when using Bemeanstt/--ennteeee to create the initial training set.
We observe in figure Da that in the first iteration the accuracy improves for all methods beside LL4AL using K-means++ based initialization on CIFARIO (more than 2% points comparing with the absolute numbers) and that the benefit rapidly vanishes within the next few AL iterations. Im- agenet and CIFAR1O both contain natural images in contrast to SVHN that contains images less similar Imagenet. Therefore, it is of interest verify if there is still benefit of another method to create the initial set than random sampling on SVHN. The results depicted in figure Dhwsow that the samples selected with Kmeans++ result in lower performance in the first iteration. In figure 9c we observe that using K-center for constructing the initial labeled set, deteriorates the accuracy by more than 10% points in the first AL cycle. We assume that a diversity based method probably mainly selects outliers present in the dataset and results therefore in such a poor performance. In contrast, Zhang al. (2020) experienced 2% points accuracy improvement on CIFARIO with 5000 training samples. However, they used the deep latent features of trained VAE, which encourages dataset specific features that are spread over the latent space according to a defined distribution. One key observation is that the benefit or the drawback of using another method than Random sampling to construct the initial set disappears rapidly within the next few AL cycles.
badge accuracy dff random acourscy diff
badge accuracy
badge acouracy random accuracy dff uncertainty accuracy dtr accuraty dir accuracy
Accunary differnce 1841
10
Acourscyy eiffornnce 196]
30
random acouracy dif
conwent wenuracy diff
uncertainty acouracy dift
8
uncertainty atturscy dtf
accuracy dif
Accurscy cifferenc
eiferrecee
8
eccuraty
accuracy ciff
2
10000
10300
8000 Number labeled
Number labeled
Number labeled
(a) K-means++ on CIFAR10
(b) K-means++ on SVHN
(c) K-center on CIFARIO
Figure 9: Accuracy difference between random initialization and two initialization methods.
8

---

Under review as a conference paper at ICLR 2022
92
88
Accuracy differencc
Accuracy
-10
84
badge
-15
badga accurscy drf random racy sif unceriainty acturacy diff
randorn
uncertainty
-20
80
coreset accuracy diff
LLAAL
-25
LI4A) accurscy JLS accuracy dit!
76
2000
.0000
5000
2000
10000
2000
4000
6000
8000 10000
Number of labeled data
Number of labeled data
(a) Absolute accuracy
(b) Accuracy difference.
Figure 10: Results on CIFARIO when using unsupervised pretrained weight at each AL cycle. The accuracy difference corresponds to randomly initialized vs pretrained weights.
Unsupervised pretraining of the network: Typically, the parameters of the classifier randomly initialized before AL starts. Conversely, in a warm start scenario, the weights obtained in the pre- vious AL cycle are used in the current instead of re-initializing the weights from seratch. In such setting, AL performance could be increased by using unsupervised pretraining of the classifier and use these dataset specific features as starting point for AL. G...riis et al. (2018) showed massive improvement in accuracy for the task of classification when using their proposed unsupervised pre- training method. The idea to train the network on predicting which 2d rotation has been applied to the input image. This initialization method has already been used in AL context by Gudovskiy jet al. (2020). Preliminary experiments show that re- ce-initializing the model at every AL cycle with the unsupervised weights compared to using them only for the first iteration is beneficial, thus followed the former approach.
Figure 10a shows the absolute result when using unsupervised pretraining for different AL methods. In contrast, Figure 10b] shows the accuracy of the classifier using randomly initialized weights minus achieve accuracy when using the unsupervised weights. The results show that with the unsupervised pretrained network the overall accuracy improves by 25% in the first AL cycle, and that the accuracy difference diminishes the number of labeled samples increases. Note, that LL4AL does not outperform the other benchmark in this case, probably because learning the additional task of the loss module does not help as much when the network is already properly trained (compare with the findings in section (). Furthermore, JLS that in addition to the unsupervised pretrained weights trains with unlabelled data each training cycle achieves even lower performance than other AL methods. The benefit of jointly training the backbone and the discriminator by leveraging unlabeled data is only visible the first iteration, but standard acquisition strategies, such as Uncertainty sampling outperform JLS afterwards.
8 CONCLUSION
In this paper we provide comprehensive evaluation of recent deep AL strategies, setting a solid benchmark for different datasets. We analyze different training settings and observe that they can cause overall accuracy differences and ranking shifts between AL methods. Data augmentation and SGD combined with learning rate of 0.1 provide higher accuracy for all AL methods when using a ResNet-18. We study the effect of different initialization methods in the AL context. In particular, advanced initial set construction methods rapidly reach the performance of randomly selected one. Furthermore, we observe that using unsupervised pretraining of the network greatly benefits the classification performance especially when the number of labeled samples is small. Future AL methods should try to clearly disentangle the performance improvement obtained by changes in the classification network and by the proposed acquisition strategy. When evaluating a new method, it should ensured that the existing state-of-the-art AL strategies used benchmarks achieve comparable results with the original accuracy reported on a specific dataset and AL interval. Deviating from the standard settings for novel method complicates fair comparison. Nonetheless, in such cases a new method should only compare to the baselines that use the novel settings instead of the standard ones if they achieve better results.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Arthur and Sergei Vassilvitskii. K-means++: The advantages of careful seeding. volume 8, pp. 1027-1035, 01 2007. doi: 10.145512833331233444,
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. CoRR, abs/1906.03671, 2019,
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh K. Iyer. Effective evaluation of deep active learning on image classification tasks. ArXiv, abs/2106.15324, 2021.
William H. Beluch, Tim Genewein, Andreas Nurnberger, and Jan M. Kohler. The power of en- sembles for active learning in image classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9368-93771 2018. doi: 10.1 101.000/...0...0000000
Razvan Caramalau, Binod Bhattarai, and Fau-Kyun Kim. Sequential graph convolutional network for active learning. In CVPR, 2021a.
Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Visual transformer for task-aware active learning. ArXiv, abs/2106.038800 2021b.
Jongwon Choi, K. M. Yi, Jihoon Kim, Jincho Choo, Byoungjip Kim, Jin- Yeop Chang, Youngjune Gwon, and H. Chang. Vab-al: Incorporating class imbalance and difficulty with variational bayes for active learning. In CVPR, 2021.
Quok Zong Chong, William J. Knottenbelt, and Kanwal K. Bhatia. Evaluation of active learn- ing techniques on medical image classification with unbalanced data distributions. In Sandy Engelhardt, Ilkay Oksuz, Dajiang Zhu, Yixuan Yuan, Anirban Mukhopadhyay, Nicholas Heller, Sharon Xiaolei Huang, Hien Nguyen, Raphael Sznitman, and Yuan Xue (eds.), Deep Genera- rive Models, and Data Augmentation. Labelling, and Imperfections, pp. Springer International Publishing. ISBN 378-3-030-88210-
Cham, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. ArXiv, hts/1703.02900 2017.
Mingfei Gao, Zizhao Zhang, Guo-Ding Yu, Sercan Ö. Arik, Larry S. Davis, and Tomas Pfister. Consistency-based supervise active learningg Towards minimizing labeling cost. ArXiv, abs/1910.07153, 2020.
Spyros Gidaris, Praveer Singh, and N. Komodakis. Unsupervised representation learning by pre- dicting image rotations. ArXiv, fbbs/183..07788, 2018.
Daniel Gissin and Shalev-Shwartz. Diseriminative active learning. ArXiv, abs/1907.066447 2019.
D. Gudovskiy, A. Hodgkinson, T. Yamaguchi, and Tsukizawa. Deep active learning for biased datasets via fisher kernel self-supervision. In 2020 IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 9038-90488 Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10. 10.11099CYPR42200220000
URL https://dol. 11099 201109//VVR2200000 www.://ww.............2222 00906
keeecomputersocleyy.oroo htt.://e....................///.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2017.
Siyu Huang, Tianyang Wang, Haoyi Xiong, Jun Huan, and Dejing Dou. Semi-supervised active learning with temporal output discrepancy. ArXiv, Abs/2107.14153, 2021.
10

---

Under review as a conference paper at ICLR 2022
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Rishabh Iyer University of Texas at Dallas, Indian Institute of Technology Bombay Institution One, and IN Two. Glister: Generalization based data subset selection for efficient and robust learning. In AAAI, 2021.
Kwan-Young Kim, Dongwon Park, K. Kim, and S. Chun. Task-aware variational adversarial active learning. In CVPR, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
D. Lewis and W. Gale. A sequential algorithm for training text classifiers. In SIGIR '94, 1994.
Yuan-Hong Liao, Amlan Kar, and Sanja Fidler. Towards good practices for efficiently annotating large-scale image classification datasets. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4348 4358-437, 2021.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. ArXiv, abs/2108.00331 2021.
Christoph Mayer and R. Timofte. Adversarial sampling for active learning. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3060-3068, 2020.
Sudhanshu Mittal, Maxim Tatarchenko, Özgün Çiçek, and Thomas Brox. Parting with illusions about deep active learning. ArXiv, abs/1912.05361 2019.
Prateek Munjal, Nasir Hayat, Munawar Hayat, J. Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. ArXiv, abs/2002.09564, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS, 01 2011.
Tawsifur Rahman, Amith Khandakar, Yazan Qiblawey, Anas M. Tahir, Serkan Kiranyaz, Saad Bin Abul Kashem, Mohammad Tariqul Islam, Somaya Al Maadeed, Susu M. Zughaier, Muham- mad Salman Khan, and Muhammad Enamul Hoque Chowdhury. Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Computers in Biology and Medicine, 132: 104319- - 104319, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018. URL https:// openreview net foremt/foorur/// iidii Tuk-RV
Burr Settles. Active learning literature survey. 2009.
Changjian Shui, Fan Zhou, Christian Gagn'e, and B. Wang. Deep active learning: Unified and principled method for query and training. In AISTATS, 2020.
Oriane Sim' 'oni,, Mateusz Budnik, Yannis Avrithis, and G. Gravier. Rethinking deep active learn- ing: Using unlabeled data at model training. 2020 25th International Conference on Pattern Recognition (ICPR), pp. 1220-1227, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs//409, 1556, 2015.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. 2019 JEEF/CVF International Conference on Computer Vision (ICCV), pp. 5971-5980, 2019.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In Sanjoy Dasgupta and David McAllester (eds.), Pro- ceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pp. 1139-1147, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1954-1963, Lille, France, 07-09 Jul 2015. PMLR.
11

---

Under review as a conference paper at ICLR 2022
Gent W Wolf. Facility location: concepts, models, algorithms and case studies., 2011.
Han Xiao, Kashif Rasul, and Roland Vollgraf. massi:n-mnist: a novel image dataset for benchmark- ing machine learning algorithms. ArXiv, alss//00.007077444 2017.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Beichen Zhang, Liandeng Li, Shijie Yang. Shuhui Wang, Zhengjun Zha, and Qingming Huang. State-relabeling adversarial active learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8753-8762, 2020.
A APPENDIX
A.1 DATASETS
The AL strategy must be robust to changes of the dataset.
Figure I shows the accuracy that the investigated AL methods have on CIFARIO, CIFARIO0, SVHN, FashionMNIST and TinylmageNet.
Overall the AL strategies show improvement over random sampling across the datasets, beside for TinylmageNet. The overall small accuracy on indicate that the network poorly trained and there still room for further improvements for new AL strategies and training settings.
we have seen, often AL fails when dealing with poorly trained networks. and the big variance across the different random seeds
Table shows performance of the default configuration when trained on the full dataset.
Table Results on the full datasets
Dataset
Accuracy %
CIFAR10
92.88 0.23
SVHN
95.22 + 0.46
FashionMNIST
94.20 0.17
CIFAR100
73.31 + 0.28
TinyImageNet
24.56 0.98
A.2 BACKBONE
The influence of the backbone used for the classification task is another key aspect that must be taken into account. Since for LL4AL the loss module is trained with the backbone by sharing the features of intermediate layers and no guideline was provided for which to use different architecture, we chose use for VGG networks (Simonyan & Zisserman, (015) the features of
the last four max pooling layers and for DenseNet architectures (Huang et al. 2017) the last four transition layers. The influence of different backbones is depicted in figure
Badge, Uncertainty and Coreset performer similarly to Random sampling with Densener201. ResNet-18 achieves the overall best performance.
A.3 CODERASE
The results for different codebases with their standard deviation are depicted in figure 13
A.4
DATA AUGMENTATION VS NO AUGMENTATION
For all datasets LL4AL has a better relative ranking compared to the other methods when no data augmentation is used. This probably due the fact that without augmentation the networks
12

---

Under review as a conference paper at ICLR 2022
Model: resnet, Botch size: 1000, Datare:: CIFAR10
Model: rasnet, Batch sice: 1000, Dotaset: SUNN
ACCUnC D663
baage MOouaay
Tedge accerney
uncorianty
mcoo uncomblery acturtty -15
Thaa corneet accuENCy
reean coneast accuracy
neat cornect accurccy +-
reeen comnset acourccy 5M
50
meen LEAN anrarayy
rico LUAA DocuRay
2000
1000
1000
0000
33000
2002
8928
9200
5000
10002
Model: resnet. Batch 1000, Dataset: fochionMNIST
Model: resnet. Botch 2003, Dataset: CIMR110
ACCUFTIO 1h41
704
hadge acounrry
near badgr securscy 10d
rander accurayy
rECR Eandom ZOOARCY
accurny
01
LEMAL ZCOPPSY 20
rrcon LEAAL ROUEEAY
12500 Winseral barin 1963
kimteer 1f
Model: resnet, Batch size: 10000, Dotaset: Tnyimagsiett
25.0
ACCUIOO Th6
20.0
uncelannly
15.9
meon UNONTSITES SCCurDYy
(TEAT CONEDN aCCSTRCY
(TANY Coreset aCCORIEy 51
mhan STOEERY
10.9
10000
20033
30003
#0000: 50000 BORED 70000 50003 20020
Figure 11: Results for different datasets
mept baac my
4l
Henbrr andlod
Figure 12: Results for different backbones: ResNet-18 (left), VGG-11 (center), DenseNet-201 (right)
trained worse and the additional task of learning the losses increases more the ability of the classifier to generalize. Overall data augmentation improves significantly the accuracy of AL strategies as can be seen from figure 15
13

---

Under review as a conference paper at ICLR 2022
Model: resnet, Botch size: 1000, Datare:: CIFAR10
Model: resnet, Batch suc: 1000, Dataset: CIFAR10
FEAA Tadge ACCurNEy
budgge OEE STREE
SCOFECE
50
ACCUnC D66
rrei ACOUTACY
PIAAN rommoed an acournyy
badoe ACOuraTyy
nander
1o5n (S1SEr accurcy 51
THeaT uncetarty acouracy
Teas unoettartyy acournTy
500
CONTRET ACTLENCY
OCCLERY
2000
6000
1000
0000
33000
2003
8920
9220
5000
10002
Figure 13: Results on our codehase and on BADGE's codebase
A.5 OPTIMIZER: SGD VS ADAM
Badge performs similarly to better than LL4AL with the Adam optimizer. The overall accuracy is worse. Coreset performs poorer on the FashionMNIST dataset with Adam. Figure [7 shows the accuracy difference between the default settings using SGD optimizer and the results using Adam optimizer for different datasets.
A.6 LEARNING RATES
The results of using different learning rates (LR) are shown in figure 18
Using a smaller LR seems beneficial in the first iterations but it worsens the performance as the iterations progress. Since a smaller LR combined with a Warm-start approach may lead to a model that doesn't 't learn enough from the new added samples, learning rates also with a cold-start approach as can be seen in figure 19
investigated the influence of different
Again using smaller learning rate seems beneficial in the first iterations but it worsens the perfor- mance as the iterations progress.
A.7 WARM START VS COLD START
We corroborate the findings of Beck et al. (021) that especially in the first iterations retraining the model every cycle improves the overall performance for most strategiie and that the difference between cold and warm start diminishes in the later AL rounds. Surprisingly LLL4AL benefits from updating the backbone and shows worse performance when compared the other AL strategies.
Note that however this findings do not generalize over different datasets. In fact in figure 21 we see that a warm-start approach is to prefer when using the SVHN dataset for all the methods under investigation and that no significant difference between the two approaches can be observed on MaShionMNISTT, beside LL4AL that performs slightly worse compared the other methods with the cold-start approach. Figure 22 shows the performance of the strategies when selecting for LLLAI. a Warm-start approach while for the others Cold-start. It emphasises the importance of choosing the best possible configuration for specific AL strategy, since here neither LL4AL outperforms the other benchmark strategies nor the contrary is true.
A.8 BATCH SIZE (NUMBER OF QUERIED SAMPLES IN EACH AL CYCLE)
In this section we want to analyze the effect of the AL batch size (the number of queried samples in each AL cycle). For this matter, we the same experiment with three different batch sizes: 100, 1000 and 2000. In all of them, we started with 1000 labeled data points and stopped when more than 10000 were labeled. The results are reported in figure 23
Using a warm-start approach combined with a small batch size improves the overall accuracy of all AL strategies, especially in the first iterations, can be seen from figure 24
14

---

Under review as a conference paper at ICLR 2022
Model: resnet, Botch size: 1000, Datare:: CIFAR10
Model: roaner, Satch abc: 1000, Datosat: CIRAR1 naaug
ACOU
Secge SECUrRT
3COMDKY
ACCUnC D663
TrANE conest NODUANy
mize LAA occursty
11221 JANN occancy 2d
uncotianty
1057 UnoNTsiney SESUPN/
Thaar corneet accuEnCy
neat cornect accurccy 1-
50
2000
1000
1000
0000
13000
2002
1920
9200
5000
10002
Nodel: resnet, Batch size: 1000. Dotaret: SWNN
Model: resnet, Batch 1000,
31
APORIN
ACCUTLOC Ph1
-0
TAST baRge MOUuaryy
reen tenge arriney
18
mear badge securscy 10d
TOSST FanderT Sccurayy
renen budge accimess snd
rEcon Izndom ZcCaaay
ONSSERT accuENY
Thean rendom accuracy
88
Lncmstiont accurety
46
50-
82
LEAM
11021 LEMAL 2COPPCY 20
mccn LUAN aOcuRaY
kimterr is
Ninneera baniee dots
Model: resnet, Batch size: 1000, Dataset: ToshionMNIST
Model: reshet Eatch $2o: 1000, Dataset: FachionNNSST neaug
ACCURICC T661
Fedge
TOST UNONTSTRR SEOrENY
riCO UECEVTsieyy accurity 51
06
CONNEET aCCLENCY
TEEAT coneAnt aCCUTACY
Cornect accurcy 9-
34
rrean COMESET aCCuTRCy
mean LEAN anvaray
rEcon LUAAL DOCMROY
2000
4000
5000
2000
33000
2002
8020
9200
5000
2000
Figure 14: Results when using data augmentation (left) vs no data augmentation (right) on different datasets
PAs Uoein pay devreee
uo dif
Figure 15: Accuracy difference between using data augmentation vs not using it for different datasets
A.9
USE OF LABELED AND UNLABELED DATA
Figure 25 shows a comparison between the performance of the AL strategies when all methods train jointly also the discriminator compared to their performance when only the backbone is trained. Similarly figure shows the same for the WAAL method.
15

---

Under review as a conference paper at ICLR 2022
Model: resnet, Botch size: 1000, Datare:: CIFAR1O
Model: resnet, Batch suc: 1000, Dataset: CIFAR1O
50
ACCUnC D663
baage MOouaay
75
Telge
15
bedge
T057 UnosTsine SEOUrN/ 500
micO uncomblityyy..o..m....c.... accurity 515
Thear cornest accuEncy
meee coneant accuracy
neat cornect accurccy 1-
reeen comnset acourccy
50
meen LLAN anramayy
ricon LUAN DOUMRay
2000
1000
1000
0000
33000
2002
1908
5000
10002
Nodel: resnet, Batch size: 1000. Dataret: S
Modell: resnet, Batch size: 1000, Dotsoet: SWNN
91
12
ACCUTIIC Ph4
18
-0
mHar badge securscy 1-d
TAST barge MoUraryy
randerT Sccurayy
80
rennn bedge serimes snd
rEcoft (2ndom zocaasy
reen Fedge arriney
88
uncoriandy aCCurEE
mean rendom accuraty
Lncmstonnty accurecy
accuriyy
46
82
L1AM
11021 LLAAL 2COPPCY 20
mccn LUAA DOCURAY'
kimterr AsS
Winserrt binie dots
Model: resnot, Batch size: 1000, Dataset: TashionMNSST
Kode: resnet, Batch sire: 1000, Datarat FachioMMNST aud
ACCURICC T663
Fwige
TOST UNONTSTRY SCOUENY
micO UnceVTsliyy accurtty 51
06
CONNEET aCCLENCY
Cornect accurcy 9-
36
Teeam coneast acCurACy
rrean CONESEC acCuriCy
mean LEAN anvaray
01
rEcon LUAAL DOCUROY
2000
4000
5000
2000
33000
2002
8020
9200
5000
10003
Figure 16: Results when using SGD optimizer (left) vs Adam optimizer (right) on different datasets
DELAA ALLAND
Figure 17: accuracy difference between the default settings using SGD optimizer and the results using Adam optimizer for different datasets
A. NETWORK ARCHITECTURE/ LOSS FUNCTION
Figure 27 shows a comparison between the performance of the AL strategies when all methods train jointly also the loss module compared to their performance when only the backbone is trained.
16

---

Under review as a conference paper at ICLR 2022
Jula anato
Di Bccay?
Figure 18: LR 0.1 (left), LR 0.01 (center), accuracy difference between the two (right)
Indel LAC3 the 1201 DIGEN LIVALE
Hodel mome such 1200 tirane
Tear Dense (CURS
Figure 19: Cold-start: LR = 0.1 (left), LR = 0.01 (center), accuracy difference between the two (right)
A.11
INITIALIZATION
A.11.1
INITIAL TRAINING SET
we see that in the first iteration we have 2% points accuracy improvement for all the methods beside LL4AL, and that the benefit rapidly vanishes with the following AL iterations. The results depicted in figure 29 show us that on SVHN the samples selected with Kmeans++ with the Imagenet features worsen the performance in the first iteration.
From figure 28
Figure 30 shows accuracy deterioration of more than 10% points in the first AL cycle using the initial labeled pool selected with K-center.
wee see that K-center initialization performs worse than random also with SVHN. In this case the accuracy drop smaller compared to the one observed with CIFARIO.
From figure 31
A.11.2 UNSUPERVISED PRETRAINING OF THE NETWORK
Results when using unsupervised pretraining of the network on CIFARIO are shown in figure 32 and on CIFAR100 figure
A.11.3 SUPERVISED PRETRAINING OF THE NET BETK
Preliminary results, showed that sing downloaded pre-trained (on ImageNet) ResNet 18 slightly worsen the overall performance of the AL methods. This probably because of the higher resolution of ImageNet. Since pre-training ResNet18 on a low resolution ImageNet is very time consuming we opted to pre-train R8sNet-18 on the TinylmageNet dataset and used it as our initial model every AL cycle. As figure 34 shows, better performance is achieved using the supervised rreitrined network, compared randomly initialized network.
A. LABELING EFFICIENCY
Usually, when new AL methods are developed, the results are shown by comparing the test accuracy with respect to the number of labeled points for the methods and for existing benchmarks. This allows to highlight the benefit of a newly proposed strategy compared to the basclines, but it does not give explicitly the reduction of annotated samples that sampling strategy needs achieve
17

---

Under review as a conference paper at ICLR 2022
Figure 20: From left to right: warm start, cold start and their accuracy difference
ciy amart
tary
Loreet
Lobder
saden tat
coaeienn pncoary All
Figure 21: From left to right: warm start, cold start and their accuracy difference on SVHN (top) and FNSHi (bottom)
the same test accuracy random sampling, which is the main goal of every AL strategy. Hence, we show in figure 33| the labelling efficiency of different AL methods using the default setting. The labeling efficiency for given test accuracy is defined as the ratio of the minimum number of labeled images needed by random sampling divided by the minimum number of images needed by the investigated AL strategy, reach given accuracy. We conclude that all AL methods label efficient across most datasets.
A. 13 CHOICE OF THE LEARNING RATE FOR THE OPTIMIZER COMPARISON
To support our choice of comparing the optimizers using LR of 0. we present in figure 36 the results of the different AL strategies when using Adam or SGD as optimizers with a learning rate of 0.1 or 0.01. Comparing the results, we can say that overall the methods reach a lower accuracy when using Adam optimizer combined with a LR of 0.01 compared to the methods when they use SGD and a LR of 0.01. In addition using Adam combined with LR of 0.01 reaches slightly lower accuracy for all the methods compared to Adam and LR 0. making the latter the more appropriate choice when comparing different optimizers. Concerning ranking shifts, only the top performer when using SGD combined with a LR of 0. while the ranking between the other methods stays the same.
that LL4AL is
18

---

Under review as a conference paper at ICLR 2022
Model: resnet, Batch size: 1300. Dataset: CTRR1
80
mown hadge antuiary
maan badge accurccy St3
ETeWe rardem actu wny
rardom ---
luiilyy
LL-/ accuracy
mann LLAAL warms acsuracy
2000
4000
5000
8000
30000
Number lapenee data
Figure 22: LL4AL Warm-star others Cold-start
modt
ALLAND
cay
Met Jki coueey
LULL CONTTY
Figure 23: Results for different batch sizes on CIFAR10. From left to right: 100, 1000, 2000 samples queried respectively.
90
90
80
80
Accuracy (%)
70
70
badge
badge
random
random
uncertainty
uncertainty
coresat
coreset
40
LL4AL
40
LL4AL
10000
10000
Number of labeled data
Number labeled data
(a) Adam LR
(b) Adam LR 0.01
80
Accuracy
70-
70-
60
badge
60
badge
random
random
50
uncertainty
50
uncertainty
coresel
cor esee
LL4AL
LL4AL
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Number
Number data
(c) SGD LR 0.1
(d) SGD LR 0.01
Figure 36: Comparison between different optimizers and learning rates for several AL methods on CIEAR10.
19

---

Under review as a conference paper at ICLR 2022
Model: reonet, Botch size: 1000, Datace:: CIFAR10
Model: resnet. Batch suc: 1000, Dctaset: CIFAR10
meen badge accumty diter encw
2.-
meen rensimn eccurery difierno
mese ancenalitty accurcyy
Acammoy orfieseen (N)
2100000 oPhorece
2
mean hodge acorory aifenence
menn random accurec di ofference
mean uxcertarty accuracy dfference
mon conart acrarar flrrnee
monn L. 2008 diference
2000
6000
5000
8000
33000
3000
2000
3000
1000
5030
6000
7000
2000
5000
Figure 24: Accuracy difference between a batch size of 1000 with one of 100 (left) and a batch size of 1000 with one of 2000 (right)
resnet,
recnet, Eatch 2500, Dataset: ciar30
12
ACUnEO
aCCARGSY
mian AS acourayy
TAKT aNTURNC w-
menn H-S acnurety --thd
pandor accumey
meem rendom ccouraty
TOSST DXTSER SCCurDy 55 RCOuENT
52
mcon rrndom scCrary
mean uecattaenty acturacy
06
12003
2000
1000
32000 WIIIN
Figure 25: Results when jointly training the discriminator network for all methods only for JLS
A. STARTING POINT
There is no consensus across the literature on how many labeled data-points should be used for the first AL cycle when reporting the results. Figure 37 shows how methods that rely on properly trained backbone, such as Coreset which uses its resulting feature representations of the images for the acquisition, the Uncertainty method which relies on its output probabilities, perform similarly to Random sampling when the amount of labeled data is extremely small, while methods such as
LL4AL which learns jointly the additional task of the loss module performs significantly better than Random sampling also for the backbone trained on 200 images. On the contrary when increasing by factor of 5 the starting amount of labeled data significant difference in the ranking of the methods is seen compared to the default configuration.
A. REDUNDANCY
Redundancy is achieved by modifying the training set, such that there is chosen amount of unique data-points and the remaining ones rest are repeated given number of time. In our experiment we chose to have 6000 unique data-points and duplicate the remaining images 2, respectively 20, times. Figure 38 shows the respective results. We corroborate the results of Beck et al.] (202I) that the diversity based approaches (BADGE and Coreset) are more robust to redundant dataset compared uncertainty based approaches (Uncertainty and LL4AL). Since in our framework the Uncertainty random subset of the unlabeled
method is performed also in the case of a 20x redundant dataset. For LL4AL and Coreset no experiment were presented and therefore comparison between our and their results is not possible.
we still outperform Random sampling
20

---

Under review as a conference paper at ICLR 2022
Model: vgg. Batch size: 2000, Dataset: CIFAR10
Model: vgp. Batch sizc: 2000, Dotaset: CIFAR1O
8910
12.5
rander accureyy
mean urndm
72.5
To5r UnoNtsiee SESURN/ 500
cornont aCCLENCY
mcon uncomblityy accumdyy 515
mean coreast aCCuTACy
cornect accurDCy 9-
rreen COMESC aCOuTICy
RETURNES
mren RAA SECurRY
WALL accUrAYy
TECOI NYW. JECuracy
2000
3003
8300
5000
10003
12000
2000
<000
5033
5000
30000
12002
Figure 26: Results when jointly training the discriminator network for all methods vs only for WAAL
Model:
1000. Dabace:: CIFAR1O
Model: Botch size: CIFAR10
30-
badge accuracy
mean Eudge
acrumey
meen rindon zecuary
maan rindom scounacy -dd
mase Lnsataant accoraty
00
ReAA Lecerhiity Artumrr +
CEEDE HELLEWY
coteal accuraTy
50
2000
1000
4025
1000
Figure 27: Results when training for all methods the loss module vs only for LLAAL
A.16 SUBSET SELECTION
Uncertainty and LL4AL are acquisition strategies that are applied on random subset of the un- labeled data as it is proposed in (Beluch et al. 2018) and (Yoo & Kweon 2019). This subset has been chosen to be 10x the number of queried data, In this subsection we investigate the influence of selecting a subset not randomly, but using diversity based query strategies, such BADGE and Coreset, to see if the uncertainty based methods benefit from a diverse subset. The results are shown in figure 39
The proposed methods perform almost en-par with their benchmarks. The most relevant results the accuracy difference between the modified Uncertainty, respectively LL4AL, and the default version (green and blue lines in the accuracy difference plots).
The benefit of this subset selection can be seen best when the diversity of the subset is critical, for example in the case of redundant dataset. Since LL4AL and Uncertainty performed worse than Coreset and BADGE on the 20x redundant dataset, we rerun the same experiment with the improved subset selections. The results are shown in figure 40) where we see how LL4AL for both subset selection methods outperform all benchmarks, in particular BADGE and Coreset. There is significant improvement also for Uncertainty, which now performs en-par with BADGE and Coreset.
A.
RANDOM SUBSET IS BENEFICIAL?
A natural question that arises is whether the random subset selection is really beneficial for the Un- certainty method. Figure 42 shows a comparison of Uncertainty applied on the complete unlabeled dataset and only on a random subset of Figure [ shows what happens when we change the starting point and the AL batch size for Uncertainty on the full unlabeled set. For the case where
21

---

Under review as a conference paper at ICLR 2022
Figure 28: From left to right: reference result, result with Kmeans++ initialization, accuracy differ- ence between the two strategies on CIFARIO.
Nadel we, sacieiia L003, DHMMA WAM
Misht Woset LUCC 2e 3800, ciraaed 5405s
Model relaer BIASS CTA SODA CIAMNC VAN
Figure 29: From left to right: reference result, result with Kmeans++ initialization, accuracy differ- ence between the two strategies SVHN.
start with 5000 labeled data and add 2500 at each iteration, we see that Uncertainty without subset selection outperforms by large margin all benchmark methods and sets strong benchmark that new proposed AL strategies should try to beat when considering this range of labeled data.
A.17
VAAL
The results for VAAL have not been consistent across the AL literature. Some papers, such as (Sinha
al. 2019 Zhang al.) 2020 Kim al. 2021), , find that VAAL outperforms Random sampling by significant margin, while others, such as (Gudovskiy et al. 2020, Caramalau et al., 2021a) find that
VAAL achieves comparable performance as Random sampling. We have reproduced the results of VAAL using their codebase finding a comparable accuracy curve as presented in the paper. However, the results for Random sampling using their codebase were significantly better than the reported ones and comparable to the results of VAAL. We also adapted their code to our framework, showing a significant improvement in the overall accuracy (z 15%). However, comparing the best result for VAAL with the best result for Random sampling in our codebase we found again that the two methods perform en-par. In detail our implementation differs in the following: We separated the training of the backbone from the training of the sampler; we trained the backbone with the same settings as for the other method; we got rid of the validation set, accounting for 10% of the whole dataset, to select the best model, since it is not common to have one in the AL literature (see table) and the trade-off between having additional data to train on compared to use them just for validation goes clearly in the direction of our adopted version.
A.18 JLS
What is the influence of the main difference hyperparameter settings that our default configuration has compared to the ones of the origial JI.S implementation? Figure shows the results thereof. To clarify the legend we give small explanation for every curve: JLS standard is the original paper implementaiton; Random standard is Random sampling using their default configuration; Random train sampler is Random sampling when the backbone is jointly trained with the discriminator; LR_1 refers to a LR = 0.1 and LR.2 to a LR = 0.01; the order of the appearance of the learning rates is corresponding the LR of the backbone (if it appears first) or to the LR of the discriminator (if it appears second);; JLS NoNesterov means we do not use Nesterov momentum (Sutskever et al.) 2013) in the SGD optimizer; JLS Model update refers to the original implementation but with
22

---

Under review as a conference paper at ICLR 2022
Figure 30: From left to right: reference result, result with K-center initialization, accuracy difference between the two strategies on CIFARIO.
Nadel SEAC dits L00A, Dibaan VAM
Midet WHAML LUCC 2e 380 USEaaed 5005
Model relaer, BIASS CTA SODA CMAar VIN
sige esaro
4001
Figure 31: From left to right: reference result, result with K-center initialization, accuracy difference between the two strategies on SVHN.
Warm-start approach. Since the best performing variant is the JLS Model_update we adopted that configuration in our codebase.
The results of JLS in our codebase for different datasets can be seen in figure 46 JLS on SVHN (not reported in (Caramalau et al. (021b)) after the first AL cycle performs worse than the other benchmark methods. The authors reported better accuracy also for FashionMNIST, but chose for this dataset to use an extreme AL case with an initial labeled set consisting of only 100 images and a final one of 700 images.
A. EXTREME AL
Which are the observations we can make when only a handful of samples are used? Figure 47 shows the results for this extreme AL scenario different datasets. We can see that more challenging datasets (CIFARIO 10 and CIFARI00) where the accuracy is relatively poor, there is no consistent gain of the AL strategies over Random sampling. On CIF LL4AL, Uncertainty and Coreset even underperform Random sampling. On easier datasets, such as FashionMNIST the methods perform slightly better than Random sampling. In general from the dashed lines we see a big variance in the results especially in the first iterations. If the CNN is better trained (unsupervised pretraining), are we going to see better performance for the benchmark methods compared Random sampling? Figure 48 shows the corresponding experiment.
Even though the performance much better with unsupervised pretraining (more than 100% accu- racy improve in the first cycles and still more than 40% in the last iteration) the benchmark methods perform comparably to Random sampling. Note however that a real improvement over Random sampling is seen only after 75% accuracy in the case where we start with 1000 samples and add 1000 at each iteration. This is also seen in this experiment: when the accuracy is 2 75% the methods tend to outperform Random sampling. Still the variance stays high leading to unexpected behaviours like the huge accuracy drops for all methods after subsequent AL. cycles.
23

---

Under review as a conference paper at ICLR 2022
Model: resnet, Botch size: 1000, Datare:: CIFARID
Model: resnet. Batch suc: 1000, Dataset: CIFA110
TesT bagge HOUrCY
485
bedge accuracy
diferenne
mesn rondom secursty differnnce
Trear LEEAL ECOurACy
mean uncertarity accurncy difterace
Thaa LLAAL ECOPICY
meen coeset recuacy diffeence
meen accumry offerenee
70
diferox
2000
6000
1000
0000
33000
2002
9920
9200
5000
10002
Figure 32: Results for different algorithms when starting with an unsupervised pretrained network at each AL round (left) and the accuracy difference with the default configuration (right)
Modsl: resnet, Barch size: 1000, Datase:: CMAR100
uncetartty
coreset occurncy
moan corpent seruncy and
moean LLAAL sccuracy
mean annuacy
muan accurery
155 accurccy
2004
1200
5000
8003
20030
Figure 33: Results for different algorithms on CIFARI00 with an unsupervised pretrained network
B TABLES
In this section we provide two tables: Table summarizee the recent proposed AL strategies, defines the investigated AL setup (AL budget, number of queried samples, etc) and gives the reported accu- racy range of the proposed method and of Random sampling. Table provides the implementation details of the methods. Unless otherwise stated, if for different datasets the setup differed too, then the corresponding setups where ordered in the same way as the datasets and separated by slash. A
*_" means that the information was not mentioned. The following abbreviation are used for the AL
methods: Coreset for (Sener & Savarese 2018). VAAL for (Sinha et al. 2019), BADGE for (Ash
al.
2019), LLLAI for (Yoo Kweon, 2019), JL.S for (Caramalau et al. 202Tb), WAAL for (hhui
et al.
2020), SRAAL for (Zhang et al.
2020), TA-VAAL for (Kim et al. 2021). ). ISAL for (Liu et al.
2021
FKSSAL for (Gudovskiy et al. 2020). CoreGCN and UncertainCGN for (Caramalau et al,
202
VaB-AL for (Choi al.) 2021
24

---

Under review as a conference paper at ICLR 2022
Medlod
Diversly
Datraet
Acquiition stralegy
Cycle
range
range
Fill
Cude
[5o]start cnd)
random
Cormet
Diversay
CIE CIARROCIIA
R. cester greedy on feaure space last
10%
10
10%
00-81-9027-57-65
00.77-90/27-52-6555 55:---5----------
Acquiation strang only
X 10005V HIN
68-99 93-99
96
VAAL
Diverstty
CTF AF
Data For which
10%
5%
61-77-8028.55 5-41.
31.89-74/2855994444488
Calted--
NeragyNet
confidert cornnn unlabeled
5-53
RADGE
Divenity- Unentainty
CTF ARIG-SVHN
K-Mrans 4 on
100
500/50/5
10011000/1000
10-50-58 (ReeNet-
All machede almost st-gan
yes
AINSS
"hallucinated"
18, CIFARIO,
final
1000)-10788-8
(coreguted using Slady label)
(VOG-
(1,CIFAR10,1000)
11.4AL
Uncertaicty
CIF AR30
Data with highest gredicted loss
1000
1000
52-84-90.5
51-50-87
0.302
yes (bot
offica)
JL.5
Diverityy
CIF CIF
Data for which
10% 100 (fe
59s/100 (fee
81-87-90.54433
62-70-76/28 57-44-8-777
yes
R1 FalliosM
disciminatoo most
Fashion3MNNST))
66:58-81-84
91
NIST
confi dent comning
WAAL
Divensay Uncettantty
CIE CI TastsotMNNST
Combisation of
2000 2000-2000/000
2000/1000/0000
55-68- 55-68-75/755
7751-72202-42-8777--77
90/73-83-6 866
coming divesity term coming from adver sertal setwock, wttch discmiinates labeled
Uncetaiity
CIFAA R1 Caltech-
Disctiminatrr 10%
57%
62.565
unlabeted
67:65-85-44
(sot 0/1 aedigns [0.1]
natue), and the tog
selected
TA VAAL
Diversit Uncestaiity
CII II
Discriminator pueducts unlabeled data's state value (based on She makking the sosses)
1000-2000
1000/2000
53 9021 68
50 97/20
2e5 bot not
R.100
complete
ISALL
Nether nDC
CIF AF
Select points woth the most aelvvve performance, compused starting ty their espected gradieats
1000-5000/2%
10
1000)5000/2%
45 45-82-90/57-55-
45 1--8-87777-55311886655
00
R 100-5V HEN
63:38-63-75
70.5
0.25%0.25%%%%
583-85
98.5%6995%/00
ImageNee
Fister Kenzel qminimiize distribation
duit betwees
unlabeled and weakdy-labeled
validation eet)
ConaCCN
Diveraty: Uncertintt
CIFARIGCIA
Coceset on first layer
1000 20000000000000 10
1000/2000/
51-85-99.5/20-57. 51.82 31.82.8600-5666333888
CacemalcCC
R100 FadronM
CCN Uncettintty output OCN
000
68/5-01-94853-0055 011-9944--90--2115555
VaB-AL
Nerber BDE
Darapoints
2000
5/
2000
27 011511222--856
1245--87/2-48-59
D0
R.100
lighee probablity of leading daná Brcatin
Table 3: Description and setup of different AL methods
25

---

Under review as a conference paper at ICLR 2022
Mathod
Dataset
BaciDone
Archifecture
Data augmentation
Optin beer
Length training [spochs]
Learning rate
Iaittal labeled
Use of validation
Cold Start
Unmpertfsed pro-trainig
Use ualabeled
CIFARIO CIFAR VGG-16
Badkbone only
Random
Cold Start
CIFARIOCCI VOG-16
Badkbone (traised separaelly Disainisator (minimas training)
Random Harizoutal-
for Adam for other
100
0.01
Random
Cold Start
but backhose
Flip
255 ImageNet
BADGE
CIFARIO
Re VVGG VGG.
Badibene only
Adam
until training 0.01
Random
00
Cold Start
SVIST
L14AL
CIFAR10
RenNet-18
Backbene Toss jointly
Randimm Horizontal-
SGD
200
Rander
Warm-Start
Plip. Random Crop
trained
ILS
CIFARIO/CIF ARI VGG-1&VGG- 061 70TTiSSoNNNSS
Bacebone disatminat or jointly trained
Random Horizontal- Plip Random Crop
SGD
200
0.01
Random
10
Cold-Sra StarSttrt
10
ye5
16 ResNeet11
Random
Cold
aationMNIST
16 Leeeeee
dicrrminnter jointly
uses itandom Hedzonnal Flip.
stopping
deature estractors
trained with
MLP
ganee Random- Crop)
damiñerr
SRAAL
CIFAR 10/ CIFAR ResNet-18 Calteds
Backbene (trined
Random and
Warm-Sun Sta--Statt
yms bet not
separaeeyy)
(on VAE)
Repecsentation Generator
(mirimas traning)
TA-VAAL
CIFAR 10 ResNet: 18
Dadibone loss modile jointly
Random Haritontal rlip, Random Crop
SGD Baddhone for other nets
200
Random
D0
Cold Start
Yes, bot not for
100
bockheee
Disciminator (minimas traine))
ISAL.
AR ResNet: 18
Dadebone only
Random Hoarizonral
SGD
200
0.1
Random
Cold Start
Flip. Random Crop
FKSSSAL
MNIST:SVEN 2m
LeNEL ResNe-10
Badhone only
SGD
50/
35/
0.05/0.10..1
Random
yes (wealdy, Cold-Start dabddd)
yes
RedNes--1
ConeGCN
CIFARIOICIF ARI ResNet-18
Bacchone + Graph
Random Harizontal-
SGD
200
0.1
Randem
no
Cold-Start
yes, but not
Uucertaia
Conn-lutional
Rasdam (only
Network
distingiih betrees
CIFARI00)
labd and unbeld
VaB-AL
CIFARIO/CIFFAS ResNet-18
yes, backbere
00
Table Implementation details for different AL methods
26

---

Under review as a conference paper at ICLR 2022
87.
80
85.0
Accuragy
70
badge
80.0
60
uncertainty
77.5
random
bodge
coresct
uncerainty
50
72.5
LUALL
corneet
4000
6000
4000
Number of labeled data
Number of labeled data
(a) random initialization
(b) pretraining on TinylmageNet
Figure 34: Comparison between randomly initialized network weights (left) and pre-trained weights on TinylmageNet (right).
Euddge uncerfality
Labaling efficency
1.4
efticency
1.3
1.00
0.98
0.96
0.94
1.0
60
70
[%)
Accuracy
(a) CIFARIO
(b) CIFAR100
2.4
badge
uncorta Iny
LLGAL
1.8
Labeling
Labeling
1.4
1.4
1.2
1.0
82
88
Accuracy [96]
Accuracy [%)
(c) SVHN
(d) FashionMNIST
Figure 35: Labelling efficiency on different datasets
LELL LIMANC LINGLE
uart 10US ChiBU
model mase surch 2An urrane
Una alker toi de
Figure 37: Results for different starting points and number of queried points
27

---

Under review as a conference paper at ICLR 2022
Padi: morer. und INC, UNDUE, MACO
Hasi HOGE LUTT 12C. DISHN CRIAA
MIGeL MEaE barh CErae CHAAES
lalp MAMI
Mith de WETO MOTHO
Datch
scrett Each
Model Bach
vita WOarTO
LLAE HANDYY
Figure 38: Results with standard datasets (left), 2x redundant datasets (center), 20x redundant datasets (right)
Morbel: pesnee, Batch size 10n0, Datawer: FIFRR1
Meviel: rpsret Bitch spe 1800, Dataset CIFAR1
badge- badge uncerta my mesn accuracy ofference
ccreset coroict ptcertartyy metn zc0uaay ofference
uacortenity bodge unouta
actuney dffirence
urcolantty -o -croed, uncorlanit moan ecceuacy diflaunne
0.25
0.50
-0.5
unsatal
-0.00
meen comoet act
meun cortad, jeocstantt ucureey
meon bodgt uncertaiiyy accurscy
6300
ICU
19M0
3022
10000
5000
kumber of data
baaile
00
Aconney
mean hadge actuaay moan ceroad assitasy scousey
badge bodge scouracy dfference
30
Kimber ofisbeled
banite
Figure 39: Overall performance of the investigated methods and their benchmarks (left) accuracy difference between the benchmarks and the proposed methods (right)
28

---

Under review as a conference paper at ICLR 2022
Model: size: Dalasets CIFAR10
AccaPnyy [N
62
meenn comore accu
redin LURAL accuricy
uncemairy accuracy
bedig accurncy
bedgge accumcy
2585
1220
2080
29930
Figure 40: Performance of proposed subset selection methods vs their benchmarks on a 20 redun- dant CIFAR1O dataset
Medal: resnet, Barch sibe: 1000, Datase:: CIFAR10
Model: resnet Batch sze: 1000, Dataset: CIRARIO
Accumey Affrrncc END
Accueess detrrone DNU
budge badge L1AA
difse
badge budge uncatarl eccuret offerenss rineet moseee jeram meen sinpary retwoeee
rveve rnveeet Ii nesr arrrayy antrenee LLARL zedie 1844, reen actumry offenrce LLAAL wnoee LoN mosn uccarase diloreoo.
uncatarity badge unort myy mesn acrunes affernee uncelanty wnedd pnseaamattaii untulaiiyy most suconsyy diflounoc
2000
4000
1000
0000
13000
4000
6002
5000
10002
Number of libeicc-ddle
Numbor of lubo tato
Figure 41: Accuracy difference between proposed subset selection methods and their benchmarks on a 20x redundant CIFARIO dataset
Figure 42: Results for the Uncertainty method with and without random subset selection for different datasets.
29

---

Under review as a conference paper at ICLR 2022
Model: resnoe, Batch size: 1C0, Dataset: CIFAR10
Model: resnet Batch 5:z0: 2500, Datasat: CIRAR10
meeu torsom uccuress
meen mind accerncy
mTen uxattarly uscuracy
20
Ieea uncartunty occarsdy
frean uncatiiliyy rencbatt accuracy
mren uncartanty rosibutt acCurNCy
rrcen LUSAL SCOUTOY
maan LIAA acCERWC
200
403
500
800
1000
6003 1000
30000
33000 14000 16003 18000 20300 kitoledd
Number
Figure 43: Results for Uncertainty with and without random subset selection for different starting points and AL batch sizes
Model: 16 Butchsioe 2500, Duravel: CIFAR1O
Model: Bulch CIFR10
855
Thear tunden accuracy
stnummy
accuracy
rieen LLAAL JUAEER
mean LLAM actaracy
moan "aando SCCoriCy
ricon J. acOurDT
moan Trandam
mean 3.S acournty
65
most FUnSER
accorusy
accurry
accerccy
6000 8073
12003
14000 29920
1000
200000
60003
2000
J0000
19009 18000 28000 20300 32000
Figure 44: VAAL results: the left the original implementation implementation, the right VAAL compared to other methods in our framework
AUCUaaY 156
Standard hancor lranttertte
KaTser szndy
JN [N IM.2
35 LAL
12500
Figure 45: Hyperparameter study JLS
30

---

Under review as a conference paper at ICLR 2022
Modelt resned, Barth sio 10m0, Datasen (IFRR10
Madiel: rocnet, Satch sow 1800, TatAset
SUHN
30
00
THEer badgg sccuracy
Trean Tudge IECETRES
noar budge ascrrsy std
rastort actuney
FLAA moddm acnurny
10
nandom accuney
cocurnty
LLAA,
mann LEAAL
naxt 1104) ecouncy
mAAn LLAN Accuracy etd
Thdar actutncy
12een13 etouracy
Thaa" accurccy 11d
ricn j5 acouracy 500
6000
6GU
10M0
3030
9220
100
UM3
kumber oftsbele
lasied
1800 MAUT
C1AR100
STANE LESNTRRRY AOOUNCy
F1221 OPESEE HCOESCY
rander
THEEE
nander
Thaa uniortaiiyy scouracy
Thear uncetarty acouracy a-a
heruney
Kimber of
Nimberdd labnlee
Figure 46: Results of the JLS method on different datasets
Model: recnst Batch sbee 160, Dataset: CIFAR10
Modal: racnet, BaIti dize 130, Datasat SUHI
meen Tundo accurecy
meo rUNdO accurncy
accumey
50
LLMA)
42
LL44L
mean badge accurncy
bodge accurtt
mean Lenwenerry
coroid,
mean coneset accurncy
ricon LLAM, accuracy
20
mean L1AM accarayy
200
400
$10
800
1000
700
400
600
5033
1000
Number dats
latsled
Model: Batch 100, IonANIST
Model: Batch 100, CIRAR100
05.0
$2.5
11226 uncertancy scCusay
SCOUENY
TIEE coNnEt
nost bagge accuracy
23
THES candor acuuney
nandor WALIAA
72.5
humber ofisbele dats
Number fatrited
Figure 47: Results of extreme AL for different datasets
31

---

Under review as a conference paper at ICLR 2022
Mudklt resnd, Batch 100,
Model rec/el Dalizel: CIRAR10
caage ecourit
meat crent scoraty 1-1
niee JAAL
60
THEST UNONTARIYy ScCurnTy
Tr03" CORSSET aCCLERCY
CORRART ACCLERY
200
100
SEU
500
M000
200
420
600
1003
3000
Figure 48: Results for extreme AL with and without unsupervised pretraining
32


Reviews:
[{"review_id": "XX_NW8tGane", "reviewer": "Reviewer_k1Y4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) The paper can be used as a guidance to utilize AL methods. \n2) The conclusions drawn in the submission can be used to design and improve the performance of AL methods. \n\nCons:\n1) The paper is more like a technical report where different experiments are conducted and corresponding results are reported. Minimal analysis or insights are given to discuss why the results are obtained. For example, the authors give a conclusion where LL4AL should use a warm start. But what exactly affects the warm start to be effective for different AL methods and datasets? What results could be expected on other datasets that are not included in this paper? \n2) Can the conclusions be generalized to other AL methods? If so, it would be interesting to see the authors using the conclusions to construct an AL system to achieve better performance.\n\nMinor:\n1) The font size is too small for the figures to be read. Specifically, please enlarge the font size in Fig. 1,2,3,4,5,7,8.\n2) Please consider to put different shapes or use different line types for the methods in comparison for better presentation.", "summary_of_the_paper": "In this paper, the authors provided a study on how to effectively utilize the existing active learning (AL) methods. Specifically, the authors conducted experiments on the performance of different AL approaches with different training strategies and data manipulations. \n", "summary_of_the_review": "To sum up, the paper can be employed as a practical guidance for researchers to conduct AL for image classification. However, minimal analysis or insights are given for the results to be generalized to other datasets or methods. Thus, my initial rating on this submission is borderline reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rf-nGVD2NP8", "reviewer": "Reviewer_Zkvo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This article is focused on an extensive analysis of popular AL methods and studies the effect of different training settings typically used in the literature. In addition, it discusses the merit of recent trends in AL such as using unlabeled data, initial set construction, and unsupervised pre-training of the network. The work is topical and on a topic that should be of interest to readers. There are a few parts of the article that might benefit from some attention in revision:      \nThe greatest concern is with the assessment of classification accuracy. The article is lacking detail on this important issue. Most AL works (Wei et al., 2015; Sener & Savarese, 2018; Ash et al., 2019; Killamsetty et al., 2021) present their results comparing the test accuracy with respect to the number of labeled points, to highlight the benefit of the selection algorithm compared to their baselines. In contrast, this paper does not explicitly show the labeling efficiency of selection algorithms with respect to random sampling. In addition, it is not clear how many fewer labels a selection algorithm needs in order to achieve that test accuracy with respect to random sampling. I doubt this problem can be removed by the use of labeling efficiency plots and motivate the need for studying labeling efficiency in future works as a means to more holistically understand AL selection algorithms.  \nI suspect that the codebase of this article would be a useful resource for others to build on this work and help give this article impact on the community.\n", "summary_of_the_paper": "This paper conducts an extensive analysis of state-of-the-art Active Learning (AL) methods (Coreset, BADGE, LL4AL, JLS, and WAAL) and studies the effect of different training settings (Backbone architecture, Initializing backbone weights, Optimizer, and learning rates, with and without data augmentation) and their effect on AL evaluation for image classification. It also highlights the main factors that can influence the performance of AL methods: the construction of an initial training set following a certain strategy instead of random sampling, and pretraining the backbone of the network in an unsupervised manner. In addition, it provides solid benchmarks to compare new with existing methods in sections 5 and 6.", "summary_of_the_review": "This paper reproduces state-of-the-art results of popular AL methods with different datasets and settings typically used in the literature. It also highlights some approaches (sections 5, 6, and 7) that can influence the performance of AL methods by comparing only test accuracy with respect to a number of labelled points. It is not clear how many fewer labels a selection algorithm needs in order to achieve that test accuracy with respect to random sampling. They also provide a new Pytorch codebase that will allow future researchers to evaluate and compare different strategies in a fair way. This article lacks details on the labeling efficiency of different methods and their quantitative evaluation as done in Beck et al., 2021.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C4rkpC7nT-T", "reviewer": "Reviewer_ow4N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:  \n\n- Recent active learning methods show results using different settings, and this paper tackles the important job of evaluating these methods in a consistent manner. \n\n- The analysis w.r.t using additional modules for complementary tasks, and unsupervised pre-training is interesting and useful. \n\n- The paper is clear to understand and the most results are presented effectively.  \n\n\nWeaknesses: \n\n- When comparing different optimizers, the optimal learning rate for different optimizers can be different. Hence, the argument of using the learning rate which is better for one optimizer (SGD), also for the other optimizers (Adam and RMSProp), is not precise. This is especially concerning because the relative ordering of methods might change, and the conclusion w.r.t optimizer might not hold. \n- Similar concern as above for experiments with different backbones, as different backbones might need different learning rates to obtain the best performance.   \n\n- Other popular methods to use unlabeled data in terms of semi-supervised learning of the classifier should be compared ([1], [2]).\n\n- It would be valuable to see how the performance of these methods would change with datasets having different number of classes, which is another important factor. The three datasets studied here all have 10 classes. \n\n- It would be good to explicitly mention the active learning settings (i.e. initial pool size, budget size) in the experimental setup. \n\n- The effect of supervised pre-training of the network on other larger datasets (e.g imagenet) in section 7 would be useful and make the analysis more complete.  \n\n\n[1]: Liao et. al. Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets  \n[2]: Mittal et. al. Parting with Illusions about Deep Active Learning\n\n------------------------------------------------------------------------------------------\nSome typos:\n\n* Introduction, paragraph 1: it exists a large unlabeled U → there exists a large unlabeled set U \n* Introduction, paragraph 2: include the for training → include them for training \n* Introduction, paragraph 2: divers way → diverse way \n* Section 7, unsupervised pretraining of the network: In such as setting → In such a setting \n* Section 7, unsupervised pretraining of the network: Figure 10a show → Figure 10a shows  \n", "summary_of_the_paper": "\nThe paper provides benchmarking of some of the popular active learning methods on CIFAR10, SVHN and FashionMNIST datasets. Effects of factors such as choice of backbone, data augmentation, optimizers, learning rate, cold vs warm starting are studied and the conclusions are provided as best practices. Analysis is also performed w.r.t using unlabeled data, choosing the initial labeled pool (random vs K-Means++/K-Center), and unsupervised pre-training of the backbone.\n", "summary_of_the_review": "I believe more thorough experimentation is needed for backing up conclusions in some parts like the effect of optimizers and backbone. It would be very valuable to compare popular semi-supervised methods in the section on using unlabeled data.  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7TuL1FMIEkb", "reviewer": "Reviewer_3Uts", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The experiments are extensive and valuable for the better understanding the existing strategies of training Active Learning models. Compared to previous codebase, this work seems to show some different observations and insights. Meanwhile, the conclusions about  various factors in AL methods are insightful and useful for guiding the development of AL strategies.\n\nSome concerns:\n1. Is it possible to extend these experiments into more real and complex image datasets, like ImageNet?\n2. The conclusions or observations are straightforward, but scattered. Is it possible to provide an overview of the importance of various setting, like training setting, dataset configuration?\n3. It would be better to discuss the potential research direction, considering the properties of existing AL methods and datasets.\n\n", "summary_of_the_paper": "This paper provids a codebase for fair comprisons of existing Active Learning methods, and performs a lot of comparative studies on the various influence factors for Active Learning. Some new observations are yielded and help figure out the optimal practices of evaluating AL methods.", "summary_of_the_review": "Overall, the experiments are extensive and produce many findings that could help understanding of existing pool-based AL methods. The new codebase could provide solid benchmarks for making fair comparisons. Although some adjustments may make a better contribution, this paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study the training settings that may affect active deep learning performance, including code/warm start, leveraging unlabeled data, and initial set selection, for each active learning strategy. The findings on several data sets help understand AL more, with some pieces of insights to inspire future research.

The reviewers were at best lukewarm about the work prior to the rebuttal. Some turned more positive but none were willing to strongly champion for the paper's acceptance, even after the authors provided a decent rebuttal. This leaves the paper to be a borderline case, and the recommendation comes from carefully checking the latest revision and calibrating its score with other submissions.

The reviewers are generally positive about the breadth of the study, the potential impact of the codebase and the systematic study that can inspire future works. Some clarified issues include comments on future research directions and the labeling efficiency plot (which is, however, not analyzed deeper in the main text), and results on additional settings like transfer learning (somewhat preliminary). In the end, two remaining concerns surround whether the technical contribution and the conclusions are sufficiently solid, including

* limited insights: Some reviewers comment that the insights are on the lighter side. The authors identify several issues that may affect the performance of the underlying tasks of active learning, and find that the best setting differs across different active learning strategies. But given that the paper offers at best "best practices of training models on actively-queried labels", it is not clear whether the authors achieve their claimed goal of "compare different strategies in a fair way"---in particular, the conclusion for this particular comparison seems to be missing (e.g. which is recommended in practice, BADGE or LL4AL or others?). Also, given that only three data sets (5 after rebuttal) have been studied in this work (see item below), the "generalization ability" of the conclusions in this paper cannot be clearly established. While the authors provided some additional pieces in the rebuttal, the pieces can use more study to be fully conclusive. Some reviewers are also concerned that the conclusions are rather scattered.

From a practical perspective, it appears to be a chicken-egg problem on whether to fix the active strategy first (and then train the model with the best setting/practice), or fix the training setting first (and then select the best strategy). The authors may want to add more arguments on why they focus on the former rather than the latter.

* limited experiments: several reviewers point out that the few data sets used could not fully justify the "best practice", and demand data sets like ImageNet. The authors offered some new results on TinyImageNet and CIFAR100, but those are not studied as deeply as other data sets at the current point. A more careful study on the two (and other) data sets are thus strongly recommended.
Decision: Reject