Best Practices in Pool-based Active Learning for Image Classification

1 INTRODUCTION . Deep learning methods require large amounts of labeled data samples to train . Unfortunately , annotating new datasets consisting of thousands or millions of images is very costly . A research topic that focuses on maximizing the performance of deep learning models with a given annotation budget is active learning ( AL ) . AL is a machine learning ’ s sub-field in which the algorithm is allowed to query an information source for the label of new data samples ( Settles , 2009 ) . Recently , an abundance of pool-based AL methods for image classification have been proposed ( Caramalau et al. , 2021a ; b ; Kim et al. , 2021 ; Liu et al. , 2021 ; Choi et al. , 2021 ) . In pool-based AL ( Lewis & Gale , 1994 ) there exists a large unlabeled U and a small labeled sample set L. The labeled set is used to train the classifier and the large unlabeled set is used to query new samples that then are annotated and added to the labeled set in order to include them for training . The goal of pool-based AL is to iteratively sample and annotate unlabeled training samples from the pool and add them to the training set in order to achieve the best performance given a certain annotation budget . Pool based AL is often particularly useful because large amounts of unlabeled data is often available but annotating it is cumbersome and costly . In AL there exist multiple baselines that either form lower or upper bounds for AL methods . Typically , using all the unlabeled samples , annotate them and include them for training forms an upper bound that AL methods aim to reach with as few samples as possible . In contrast , selecting new samples that should be labeled and used for training randomly from the set of unlabeled samples establishes a lower bound that any AL method should exceed in every AL cycle . AL methods can be split into uncertainty or diversity based samples . The former select samples for which the classifier is uncertain ( Gal et al. , 2017 ; Beluch et al. , 2018 ; Yoo & Kweon , 2019 ; Mayer & Timofte , 2020 ) , while the latter selects samples in a diverse way ( Sener & Savarese , 2018 ; Sinha et al. , 2019 ; Gissin & Shalev-Shwartz , 2019 ; Caramalau et al. , 2021a ) , ensuring to cover the whole data distribution of the unlabeled set . Recently proposed strategies , are often still based on these concepts , combine them ( Ash et al. , 2019 ; Zhang et al. , 2020 ; Kim et al. , 2021 ) or propose new ideas such as using unlabeled data to train addition modules ( Shui et al. , 2020 ; Caramalau et al. , 2021b ) used for sample selection or combine AL with semi-supervised learning ( Sim ’ eoni et al. , 2021 ; Huang et al. , 2021 ; Gao et al. , 2020 ; Mittal et al. , 2019 ; Liao et al. , 2021 ) . The recent popularity of AL methods complicates comparing these works in a complete and standardized manner . However , assessing the improvements made by new methods is crucial to boost the progress in AL . Hence , this work provides best practices to keep in mind , when proposing new pool-based AL methods for image classification using deep learning and comparing them to existing methods form the literature . Contributions : In summary , our contributions are as follows : ( i ) We perform an extensive analysis of popular AL methods and study the effect of different settings typically used in the literature . ( ii ) In addition , we discuss the merit of recent trends in AL such as using unlabeled data , pretraining or initial set construction . ( iii ) We incorporate our findings into a collection of best practices to keep in mind when evaluating pool based AL methods for image classification . ( iv ) We make a new Pytorch code base available 1for AL that contains many popular AL methods and datasets and allows to evaluate and compare different strategies in a fair way . Code will be released upon publication . 2 RELATED WORK . AL evaluations : Recently the attention on benchmarking and unifying results of new AL strategies increased . Beck et al . ( 2021 ) conducted a rich set of experiments to evaluate the performance of AL methods , including ( Wei et al. , 2015 ; Sener & Savarese , 2018 ; Ash et al. , 2019 ; Killamsetty et al. , 2021 ) . They assessed the robustness of the methods by adding redundancy to the datasets , concluding that diversity based strategies are more stable in this scenario . In addition , they studied the effect of using data augmentation , using different optimizers , updating the models instead of reinitializing them in every iteration , and using different batch sizes . In contrast , we perform similar experiments but include more AL methods on multiple datasets , evaluated additional settings , and study the effect of using unlabeled samples for AL . Munjal et al . ( 2020 ) focused on benchmarking the strategies proposed in ( Gal et al. , 2017 ; Beluch et al. , 2018 ; Sener & Savarese , 2018 ; Sinha et al. , 2019 ) using a uniform experimental setting . They vary the AL batch size , the validation set size , and the amount of class imbalance . They find that under changing experimental condition no strategy offers gain over Random sampling . Conversely , our experiments show that AL methods clearly outperform Random sampling when training the classifier carefully . Chong et al . ( 2021 ) evaluates traditional AL methods and Coreset on an imbalanced medical dataset ( Rahman et al. , 2021 ) and on CIFAR10 . They highlight the importance of respecting class imbalance in AL and conclude that Coreset performs better than Uncertainty sampling . In contrast to these last two works , we omit analyzing the robustness of AL strategies on imbalanced datasets but rather focus on balanced datasets such as CIFAR10 . Investigated AL strategies : The investigated AL strategies are a relevant selection of recent deep pool based AL methods used in deep learning . In particular , we study Coreset , BADGE , LL4AL , JLS and WAAL , and Uncertainty sampling . We briefly summarize the most recent ones . Coreset ( Sener & Savarese , 2018 ) selects new unlabeled samples by solving the k-Center problem ( Wolf , 2011 ) on the feature space of the last fully-connected layer . Ash et al . ( 2019 ) propose an acquisition strategy ( BADGE ) based on k-MEANS++ ( Arthur & Vassilvitskii , 2007 ) , which is applied on so called hallucinated gradients with respect to the parameters of the final layer of the deep network . Hallucinated gradients are used because the true labels are not available for unlabeled samples . Therefore , the most likely labels according to the classifier are used to compute the gradient . Yoo & Kweon ( 2019 ) propose LL4AL , based on the assumption that data points for which the classification network is likely to produce a wrong prediction are most informative and should be used for training . To this end , they jointly train a target module and a loss prediction module that aims at predicting the loss of a given unlabeled sample . The samples achieving the highest losses are then added to the training set . JLS ( Caramalau et al. , 2021b ) employs a discriminator network besides the classifier . The task of the discriminator is differentiating whether a given sample belongs to the set of labeled or unlabeled samples . The samples that the discriminator predicts to be unlabeled with a high confidence are selected and annotated . JLS shares intermediate backbone layers for classification and discrimination such that training the discriminator with unlabeled data turns out to be beneficial for 1The code will be made available after publication . the classifier as well . Shui et al . ( 2020 ) propose an acquisition strategy ( WAAL ) that combines the uncertainty score of the task model with a diversity term based on the Wasserstein metric . To compute the diversity metric , an adversarial network is used that is optimized to discriminate between labeled and unlabeled data . Both networks share the same feature extractor and are jointly trained through a min-max optimization problem that uses labeled and unlabeled data . 3 EXPERIMENTAL SETUP . Since the accuracy range of AL methods extensively changes across the literature and their provided codebases , we provide our own that aims at maximizing AL accuracy for all the implemented methods . Our codebase builds upon a repository2 which only implements the LL4AL strategy for CIFAR10 and achieves the result presented in ( Yoo & Kweon , 2019 ) . We re-implement multiple other methods such as Random sampling , Uncertainty sampling , Coreset ( greedy K-center variant ) , BADGE , JLS , VAAL and include them into the codebase . Furthermore , we adapt the codebase in order to support additional datasets such as SVHN and FashionMNIST . The different methods can be tested on any dataset and our codebase allows to examine the effect of various different settings such as warm or cold start , different classifiers , unsupervised pretraining , changing the acquisition strategy to sample the initial dataset , different training settings , such as jointly training a discriminator or a loss prediction module for all methods . We use the settings in ( Yoo & Kweon , 2019 ) because they achieve results with high accuracy on CIFAR10 for all the reported methods . Note that the default settings serves mainly as comparison benchmark , and does not lead to the highest absolute accuracy for each method across all datasets . Unless stated otherwise , our default training setting is as follows : We use ResNet-18 ( He et al. , 2016 ) as backbone and train with a mini-batch size of 128 , for 200 epochs , with SGD optimizer , an initial learning rate of 0.1 , momentum of 0.9 and weight decay of 0.0005 . After training for 160 epochs , we decrease the learning rate to 0.01 . Instead of initializing the network weights in every AL cycle we use the weights of the network obtained in the previous AL cycle ( warm start ) . We use data augmentation ( random horizontal flips and random crop ) and normalize the input images . The initial training set consists of 1000 labeled samples . We add 1000 samples in each AL step , except for the experiments focusing on the use of unlabeled data where we use the JLS and WAAL settings . Our codebase allows to reproduce state-of-the-art results for the aforementioned methods for all three datasets , for different training setting and annotation budgets . All results reported in the paper are averaged over three runs with different random seeds . For the experiments we use three common datasets used in the AL literature : CIFAR10 ( Krizhevsky , 2009 ) , consisting of 60k 32 × 32 RGB images , divided into 50k training and 10k testing images split into 10 classes . The datasets are balanced , i.e . each class has the same number of images . SVHN ( Netzer et al. , 2011 ) contains images showing house numbers obtained from Google Street View . It consists of 73,257 training , 26,032 testing , and 53,1131 extra training images . The RGB images have a resolution of 32× 32 , are centered around a single digit and are split into 10 classes . The extra training images are not used in this work . FashionMNIST ( Xiao et al. , 2017 ) contains 28×28 gray-scale images showing articles from an online shop . The dataset consists of 60k training and 10k testing images . Figure 1 shows the results when using our codebase and the default settings on three datasets . In the paper we often report only the accuracy difference plots but omit the figures showing the absolute accuracy , or we only show the result on a single dataset , or we omit other AL strategies such as VAAL or JLS to keep the figures clean , or we only report the mean but omit the standard deviation . For additional plots containing all these results we refer the reader to the Appendix .

Summary:  The paper provides benchmarking of some of the popular active learning methods on CIFAR10, SVHN and FashionMNIST datasets. Effects of factors such as choice of backbone, data augmentation, optimizers, learning rate, cold vs warm starting are studied and the conclusions are provided as best practices. Analysis is also performed w.r.t using unlabeled data, choosing the initial labeled pool (random vs K-Means++/K-Center), and unsupervised pre-training of the backbone.