An Empirical Study on the Efficacy of Deep Active Learning Techniques
Authors: YU LI
Muxi Chen
Yannan Liu
Daojing He
Qiang Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep neural networks
active learning
semi-supervised learning

Abstract: Deep Active Learning (DAL) has been advocated as a promising method to reduce labeling costs in supervised learning. However, existing evaluations of DAL methods are based on different settings, and their results are controversial. To tackle this issue, this paper comprehensively evaluates 19 existing DAL methods in a uniform setting, including traditional fully-\underline{s}upervised \underline{a}ctive \underline{l}earning (SAL) strategies and emerging \underline{s}emi-\underline{s}upervised \underline{a}ctive \underline{l}earning (SSAL) techniques. We have several non-trivial findings. First, most SAL methods cannot achieve higher accuracy than random selection. Second, semi-supervised training brings significant performance improvement compared to pure SAL methods. Third, performing data selection in the SSAL setting can achieve a significant and consistent performance improvement, especially with abundant unlabeled data. Our findings produce the following guidance for practitioners: one should (i) apply SSAL as early as possible and (ii) collect more unlabeled data whenever possible, for better model performance. We will release our code upon acceptance.

Under review as a conference paper at ICLR 2023
AN EMPIRICAL STUDY ON THE EFFICACY OF DEEP ACTIVE LEARNING TECHNIQUES
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Active Learning (DAL) has been advocated as promising method to reduce labeling costs supervised learning. However, existing evaluations of DAL meth- ods are based different setrings, and their results are controversial. To tackle this issue, this paper comprehensively evaluates 19 existing DAL methods a uniform setting, including traditional fully-supervised active learning (SAL) strategies and emerging semi-supervised active learning (SSAL) techniques. We have several non-trivial findings. First, most SAL methods cannot achieve higher accuracy than random selection. Second, semi-supervised training brings significant performance improvement compared to pure SAL methods. Third, performing data selection the SSAL setting can achieve significant and consistent performance improve- ment, especially with abundant unlabeled data. Our findings produce the following guidance for practitioners: one should (i) apply SSAL early possible and (ii) collect more unlabeled data whenever possible, for better model performance. We will release code upon acceptance.
INTRODUCTION
Training a well-performed Deep Neural Network (DNN) generally requires substantial amount of labeled data. However, data collection and labeling can be quite costly, especially for those tasks that require expert knowledge (e.g., medical image analysis (Hoi et al.) and malware detection (Nissim jet al. 2014). Deep Active Learning (DAL) has thus long been advocated to mitigate this issue, wherein we proactively select and label the most informative training samples. That is, given a pool of unlabeled data, DAL iteratively performs data selection and training until the given labeling budget reached, as shown in Figure
Various DAL techniques are proposed in the literature. Most of them are fully-supervised ones (SAL) and aim for a better data selection strategy"] The SAL strategies can be roughly grouped into three categories: 1) model-based selection; 2) data distribution-bar based selection; and 3) hybrid selection. Model-based selection prefers annotating data that are most uncertain under the task model (Gal et al.) 2017 Bellch et al. 2018). Data distribution-bar based methods select data according to their density or diversity (Sener & Savarese 2018 Sinha et al., 2019). Hybrid methods consider task model information and data distribution when selecting (Ash et al. 2020).
By applying pseudo-labels (Arazo et al. 2019) to the unlabeled data or consistency regulariza- tion (Berthelot et al., 2019), semi-supervised learning (SSL) can improve the model performance substantially. Consequently, is attractive to apply active learning on top of SSL techniques, referred to semi-supervised active learning (SSAL). (Song al. (2019) incorporate the well-known SSL method MixMatch (Berthelot et al. 2019) during training. Gao et al.] augment the unlabeled samples and enforce the model to have consistent predictions on the unlabeled samples and their corresponding augmentations. Also, Gao et al.) develop a data selection strategy for the SSL-bbasd method, i.e., selecting samples with inconsistent predictions. Besides, WAAL formulates the DAL distribution matching problem and trains the task model with an additional loss evaluated from the distributional difference between labeled and unlabeled data (Shui al. 2020).
Despite the effectiveness of existing methods, the results from previous works often contradict each other. For example, CoreSet (Sener & Savarese 2018) and the DBAL method (Gal al. 2017) ) are
Among the 19 investigated methods, 14 of them are fully-supervised ones

---

Under review as a conference paper at ICLR 2023
Semi- Suppevieee
Model Training
Human Labeling
Uncertainty
Hybrid
Diversity
Unlabeled Data
Data Selection
The pool-based deep active learning process. In each iteration, we first train the task model. Based the trained model and the available unlabeled data, select subset of the unlaheled data for labeling (marked as red circle or triangle). The process is iterated until certain model accuracy achieved or the labeling budget is used up. Existing methods mainly focus on the data selection strategy or the model training strategy.
Figure I:
shown to perform worse than the random selection (RS) in (Beluch et al., 2018) and (Sinha et al. 2019), respectively. Such result inconsistency is often due the different experimental settings in evaluation (see Appendix 5). Several empirical studies are therefore proposed to address this problem (Beck et al. 2021 Munjal
2020p. However, again, they have controversial observations. Beck et al. 202 claim that there is little no benefits for different DAL methods over RS, while Munjal al. (2020) show that DAL methods are much better than RS.
These inconsistencies motivate to unify the experimental settings and conduct a thorough empirical study on the effectiveness of DAL techniques. Our contributions are summarized as follows:
We re-implement and perform extensive evaluations of 19 deep active learning methods on several popular image classification tasks, including MNIST, CIFAR-10, 10, and GTSRB. To the best of our knowledge, our evaluation is the most comprehensive one, which not only includes most state-of-the-art SAL solutions but also incorporates various SSAL methods.
Through extensive experiments, we conclude that SSAL techniques are preferred. Traditional SAL methods can hardly beat random selection, and there is SAL method that can consistently outperform others. In contrast, SSAL methods not only easily outperform all the SAL methods by a large margin. More importantly, active sample selection plays an important role in SSAL methods, which can achieve significant and consistent performance improvements compared to random selection.
We conduct in-depth analysis of SSAL methods and provide two guidance to the practi- tioners. First, one should conduct SSAL as early as possible. Second, one should seek more unlabeled data whenever possible achieve better performance.
The rest of the paper is organized as follows. Section pressnts the related works. Section illustrates the experimental setup. We present our empirical study on the performance in Section Further studies SSAL methods presented in Section 5 Section concludes this paper.
2 RELATED WORKS
In this section, we introduce existing DAL methods and empirical studies on these DAL methods. As shown in Figure existing DAL works can be roughly grouped into fully-supervised active learning (SAL. and semi-supervised active learning (SSAL), depending on whether they use unlabeled data to train the task model.
2.1 FULLY-SUPERVISED ACTIVE LEARNING (SAL)
We categorize the SAL strategies into three classes: uncertainty-baase selection, diversity/ representativeness- based selection, and hybrid selection, which is the combination of the above two methods.
2

---

Under review as a conference paper at ICLR 2023
Uncertainty-basee selection. The uncertainty- approaches estimate the uncertainty for each unlabeled sample based on the task model's output and select uncertain data points for labeling. Various methods have been developed to estimate uncertainties. Least Confidence, Entropy Sampling, and Margin Sampling directly use the model output for an input image for sample uncertainty estimation (Wang& & Shang 2014). Specifically, Least Confidence calculates the uncertainty of sample as 1 largest output probability, Entropy Sampling uses the output entropy to represent the uncertainty, and Margin Sampling uses the difference between the largest and the second largest output probability as the uncertainty of sample. BALD Dropout runs the task model multiple times for an unlabeled sample with certain dropout rate. Data uncertainty is defined by the difference between the mean entropy of multiple probability predictions and the entropy of the mean probability prediction (Gal et al.). Learning Loss trains an additional module to predict the loss of the original model and regards the loss as sample uncertainty (Yoo & Kweon 2019). Uncertain GCN uses graph convolution network (GCN) to select samples that are unlike the labeled samples (Caramalau et al. 202Ip. VAAL trains variational auto-encoder (VAE) and a discriminator to distinguish labeled data and unlabeled data through adversarial learning (Sinha et al. (2019). They regard the probability of belonging to the unlabeled dataset the uncertainty of a sample.
based methods aim at finding diverse representative data set from the unlabeled data pool such that model learned over the subset is competitive over the whole dataset. CoreSet maps the high-dimensional data to low-dimensional feature space and based on which they select the k-center points (Sener & Savarese 2018). KMeans clusters data points into multiple clusters based on their features and selects samples that are closest to the center of each cluster [Zhdanov 2019). CoreGCN is variant of uncertainGCN which applies the CoreSet algorithm on the feature learned by the GCN (Caramalau et al. 2021).
selection.
Hybrid selection. In practice, DAL methods typically acquire a batch of data at once. However, on the one hand, samples selected by the uncertainty-basee method may contain redundant information. On the other hand, selecting samples without considering the task model's feedback can select samples that the model already accurately infers, thus providing little information to the model. To address this challenge, some methods propose to combine the uncertainty and diversity methods. LctieeLcaminnByLLammee
(ALBL) proposes to blend different AL strategies and select the best one by estimating their contribution at each iteration (Mayer & Timofte 2020). BadgeSampling leverages the estimated gradient of the parameters in the last layer for selection. The selected samples have high gradient magnitude and diverse gradient directions (Ash et al. 2020). ClusterMargin clusters the unlabeled data and uses round-robin strategy to select each cluster's most uncertain data (Citovsky ett al. 202T). MCADL considers multiple criteria simultaneously and adaptively fuses the result of uncertainty-basee selection and density and similarity- -based selection (Yuan et al. 2019).
2.2 SEMI-SUPERVISED ACTIVE LEARNING (SSAL)
Another group of works that address the annotation-effccient learning problem is semi-supervised learning (SSL.. The main idea of SSL. is to mine the useful information the unlabeled data for model training. One way is to assign each unlabeled sample a pseudo-label and then use these pseudo-labeled samples together with labeled samples for training. Another way is to use consistency regularization evaluated on the unlabeled data for better model training.
Recently, there have been few attempts to combine SAL and SSL to form the semi-supervised active learning methods (SSAL). [Song et al. (2019) incorporates one of the popular SSL techniques MixMatch, into the SAL training pipeline, and the model accuracy is greatly improved. They use two sample selection algorithms, and we denote them SSLDiff2AuDDiree and SSLDiff2AugKmean
The first step of this method is calculate the average prediction for each sample and its augmentation. Based on the averaged prediction, the authors use the difference between the largest and the second largest probability as the uncertainty for sample selection.
SSI.Dif22AuDDrrect
SSLDiffZAuKKmean They use the k-means algorithm to cluster all unlabeled samples. Then, they select the top uncertain samples from each cluster. The uncertainty is calculated as the SSi.Dif22uggDires method.
3

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of empirical studies of DAL methods.
Paper
Publication Year
Methods
SSL
Performance
Robustness
Cost
Ours
2022
19
Yes
Yes
Yes
Yes
Beck et al. (2021
2021
8
No
Yes
Yes
Yes
Hu et al.
202T
2021
12
No
Yes
Yes
No
Munjal al.
2220
2020
6
No
Yes
Yes
No
Milial 2019
2019
6
Yes
Yes
No
Another SSAL method (we denote it as SSI.Consisteney) uses consistency regularization to improve the training process. Specifically, for each unlabeled sample, the authors enforce the model to produce consistent predictions on the sample and its augmented version. SSLConsistency also proposes a sample selection mechanism based on this consistency and selects the least consistent ones to label (Gao et a..). Please note that we also include the WAAL method as an SSAL because uses unlabeled data to train the task model. Specifically, WAAL enhances the task model's 's feature extractor by letting it to differentiate the labeled and unlabeled empirical distributions based on the Wasserstein distance. It selects unlabeled samples that are different from the current labeled data.
2.3 EMPIRICAL STUDY OF ACTIVE LEARNING TECHNIQUES
Several empirical studies on the effectiveness of DAL methods have been proposed in recent years (Beck et al.] 202T: Hu et al., 202I] Munjal et al., 2020; Mittal et al., 2019). We summarize their characteristics in Table 1 Recent empirical studies either fail to study SAL methods under variety of experiment settings or fail to cover state-of-the ofrtt---rttttttt methods. Specifically, most of them ignore a growing line of work that combines supervipe learning with active learning. While (Mittal ct al. 2019) compares the performances of SSAL and SAL, it does not study the characteristics of SSAL methods further.
In contrast, our work unifies the SAL and SSAL into a single evaluation framework. Also, we provide an in-depth analysis of the SSAL methods. This analysis provides practical guidance to the users.
3 EXPERIMENT SETUP
DAL techniques under study. We re-implement and reproduce the results for 19 DAL methods, including the fully-supervised active learning (SAL) and the semi-supervised active learning (SSAL). For SAL methods, we cover RandomSampling. BALDDropout, BadgeSampling, ClusterVarginSampling, CoreSet, K.MeanSSampling, Learn- ingLoss. LeastConfidence, MCADL, CoreGCN, UncertainGCN, and VAAL. For SSAL methods, we cover SSL Random, CSs.Consist,ncy SSL.Dif2AugDireed JSS.Dif22ugKKmaam and WAAL. For detailed description of each method, please refer to Section
AdversarialBIM,
Datasets. In this paper, we use three widely used image classification datasets for evaluation: MNIST (LeCun et al. 1998). CIFAR10 (Krizhevsky et al.) 2009), and GTSRB (Houben et al. 2013). MNIST contains 70,000 examples of handwritten digits. The size of each image is 28 x 28 x I. CIFAR1O is composed of ten classes of natural images and has 60,000 images in total. The size of each image is 32 x 32 x 3. GTSRB contains 43 classes of traffic sign images. There are 39,209 labeled images in total. Each image has 32 32 x pixels. Please note that the GTSRB dataset has imbalanced classes.
Training settings. We evaluate all methods with a unified setting, which is adopted from (Ash et al., 2020). Specifically, for MNIST, we set the start budget (denoted as nStart) as 2%. the total budget (denoted as nEnd) as 20%, the step size (denoted as nuQery) as 2%, and the training epochs as 50, That is to say, we will randomly select 2% samples from the dataset to initialize the task model. Then, we query 2% extra unlabeled samples in each iteration for labeling. The process is iterated until the labeled set size reaches 20%. For the rest of the datasets, we set the start point nStart as 10%, the total budget nEnd as 70%, the step size nQuery as 5%, and the training epochs as 150. We use SGD as the optimizer since it performs better than Adam (Beck et al., 2021). The learning rate is set to 0.1 for all datasets. According to (Beck et al. (0221). resetting the model after each iteration achieve better performance than fine-tume fine-tuning the model from previous rounds. Hence, we reset the model for each DAL iteration and retrain the model from scratch.
4

---

Under review as a conference paper at ICLR 2023
Table 2: The dataset and the fully-supervised model accuracy.
Dataset
# Classes
Train/Test
Acc (%)
MNIST
10
1..000/10,000
97.68
CIFAR-10
10
50,000/10,000 100,00
92.15
GTSRB
43
39.209/12.630
99.33
Please note that we evaluate each method for five runs, and we try to fix the five random seeds so that every method has the same initial accuracy. However, there are some exceptions: WAAL and LearningLoss. This is because WAAL uses unlabeled data for training, and the model learned by learning loss will be affected by the loss model.
Platform and Hardware. We use the Pytorch framework and two Titian V GPUs for evaluation.
Evaluation metries. In this paper, we use three metrics to evaluate different DAL methods: Accuracy, Average Performance Improvement (API), and Cost. The accuracy of trained model is defined as the ratio between the correctly classified test inputs out of total test inputs. Due to the large number of methods we study, plotting the accuracy for all methods at each iteration in single figure is hard for visualization. Hence, we use API, i.c., the average performance of a DAL method over all iterations. API is calculated follows:
APIal
(ace)
acc;
random
where N represents the total iterations, i.e., N = + int( a.nd nStart
Also, the acc,' stands for the accuracy achieved by a deep active learning method (e.g.. CoreSet) at iteration i. The accrandom stands for the accuracy achieved by RandomSampling at iteration
nStep
Besides, we also report the cost of each deep active learning method as the running time in seconds. The cost mainly contains two parts: the data selection part and the training part. The calculation of the cost is as follows:
N Costal Σ
(selection_cosl] + traininggcos@f
(2)
N
where N represents the total iterations. The selection_cose and training_ cost represents the selection cost and the training cost for certain active learning method at iteration i.
EXPERIMENTAL RESULTS AND ANALYSIS
4.1 PERFORMANCE COMPARISON
We report the performance of each method in Table We break the results into the SAL part and the SSAL part. For ease of comparison, we report the average accuracy improvement over RandomSSlection. Besides, we show the detailed accuracy vs. budget curve of these techniques in the Appendix (see Figure Appendix). We have several observations from the results.
Comparison among different SAL methods. We observe that RandomSampling a strong baseline for SAL methods. For each dataset, many SAL methods perform worse than RandomSampling according to Table 3 To be specific, there are 6/14, 3/14, and 6/14 methods perform worse than RandomSampling for MNIST, CIFAR-10, and GTSRB, respectively. Also, most methods perform worse than RandomSampling in at least one dataset. Only BadgeSampling, ClusterMarginSampling and LeastConfidence constantly perform better than RandomSampling all studied datasets.
Another important observation for SAL methods is that: there no SAL method that can outperform others every dataset. For instance, out of the three methods that are better than RandomSelection. BadgeSampling outperforms the other two methods on MNIST, but it performs worse than LeastCon- fidence GTSRB and CIFAR-10. More studies SAL methods can be found in Appendix.
5

---

Under review as a conference paper at ICLR 2023
Table 3: The performance and cost comparison of different DAL methods. The performance is calculated as the average accuracy improvement (API) over RandomSampling, We bold the best method from both SAL and SSAL methods. Also, highlight the rows that consistently outperform the RandomSampling. The cost is normalized using the cost of RandomSampling.
Performance
Cost
Strategy
MNIST
CIFAR-10
GTSRB
MNIST
CIFAR-I0
GTSRB
The SAL methods
RandomSampling
0.00
0.00
ActiveLearningByL... Learning (Maver & Timofte 2020
-0.15
1.58
0.34
1.51
0.86
1.04
AdversarialBIM (Ducoffe & Precioso 2018
1.44
0.25
-0.24
6.39
1.49
2.27
BAIDDrrpont al.
BadgeSampling (Asi
2020
1.84
1.89
0.27
2.74
1.52
2.08
Cluster CaugterarginSammiinn (Citvvkyy < et
2021
1.76
0.28
0.23
1.44
0.87
1.08
CoreSet (Sener
-1.15
KMMeanSumpling (Zhdanov
-0.22
0.17
0.20
5.62
1.82
1.40
I. Learning Loss Yoo & Kweon
2019
-2.32
-2.38
0..4
0.66
0.62
0.75
LeaftConfeeence (Wang Shang
2014
1.40
1,95
0.34
1.00
MCADL (Yuan
0.57
-0.50
0.03
7.21
1.41
1.32
CoreGCN (Caramalau e ni.
2021
1.21
1.16
-8.52
2.35
1.25
1.12
UncertainGC ( dramataa e
2021
-0.85
-8.77
VAAL Sinna
-0.34
-0.49
-24.79
3.03
6.62
6.57
The SSAL methods
WAAL Shui
2020
3.00
2.05
0.21
3.90
ASL.onsistencyy (Gao 2019
3,39
5.83
1.19
12.11
4.82
3.03
SSLDiff2 Direct (Song al.
3.49
5.78
1.19
11.90
5.05
2.99
SSLDiff2Aug (Song
3.50
5.74
1.17
4.82
5.26
SSL Random
1.68
4.57
1.11
11.70
5.76
4.93
1.00
50 1.0 .100
Uncortainty
Inconsistency
48
46
10
0.92
44
0.6
10-2
42
40
10-3
38
0.2
36
10
S5LRandom
SSLConsistency
SSLDII2ZugDDieet
SSLDiTZAuggMMeas
Figure 2: Comparison between different SSL-based active learning methods. These four methods use the same SSL method to train the model. This figure is produced using MNIST dataset.
Comparison between SAL and SSAL methods. When comparing SSAL with SAL, we have the following observations. First, in general, SSAL methods outperforms SAL methods for a large margin. On the one hand, the SSLRandom method has a 1.68, 4.57, and 1.11 average accuracy improvement over the RandomSampling on MNIST, CIFAR-10, 10, and GTSRB, respectively. On the other hand, the SSAL improves the best method SAL by 1.5-1.8441.66.. 5.83-1.95=3. 1.19-0.34=0.85 w.r.t the API on MNIST, CIFAR-I0, and GTSRB respectively. Second, SSAL methods more stable than SAL methods. Each SSAL method outperforms the RandomSampling consistently on datasets.
This indicates that using unlabeled data to train the task model is useful. The unlabeled data provides extra information for the model to learn so that the ability of the model can be improved.
Comparison among different SSAL methods. Within the SSAL methods, observe that SSLCon- sistency, SSLDiffZAggDiee and SSLDif22AgKKmens are much better than SSLRandom, while WAAL sometimes performs than SSLRandom. We also observe that the performance dif- ference between SSLConsistency, SSLDif22AuDirret and SSLDiff2 AugDiianAuKKKmans are negligibly small.
We investigate the reason behind the above observations by studying the property of the selected data in Figure and Figure To generate these figures, randomly label 2% samples and

---

Under review as a conference paper at ICLR 2023
SSLRandom
Consistency
SSLDIffZAuDDiret
SSLDifZAuigKMeans
Figure 3: Visualization of selected samples (red points) out of total samples (gray) for different SSAL methods. This figure is produced using the MNIST dataset.
train a model using SSL. methods. Then, we select and label 1 % extra samples to retrain the model and report the resulting accuracy. To simplify the process, we only experiment on SSLRandom, SSLConsistency, AugDiit2AugDiree,, and SSLDiffZAgKKmean This is because they use similar semi-supervised training methods, and their difference largely comes from the selection strategies. Figure shows the diversity, uncertainty, and consistency of the selected samples at certain iteration. The diversity of the selected samples is calculated as the average distance between selected samples. prob, where prob stands for the highest confidence for one
The uncertainty is calculated sample. The inconsistency is calculated as the prediction variations between one sample and its augmented version. Figure visualizes the selected samples among all samples.
First, we find that choosing uncertain or inconsistent samples in the SSAL setting can improve the model performance consistently and significantly. This is why the performance difference between SSLRandom and the other three methods (e.g., SSLCoonsisenccy. SSLDif22ugDireec, and SSLDDif2AugKmmnn)) large. illustrate, Figure 2 and Figure 3 shows that samples chosen by SSLRandom are much more certain and consistent. In contrast, the rest three methods focus more on uncertain or inconsistent samples. Remember that in the SAI. setting, most selection methods perform worse than RandomSampling. This may be because semi-supervised learning provides the model with better feature extraction ability. Hence, they can estimate the uncertain samples more accurately than in the SAL setting.
Second, sample uncertainty more important than sample diversity during sample selection in SSAL. Since SSLConsistency, SSLDiff2AugDires and SSLDiffAugKKmann all focus selecting uncertain samples, the performance differences between them are small. To illustrate, although SSL.Diff2AukKmeen
considers diversity during sample selection, the performance gain is tiny MNIST, and there is no performance gain on CIFAR- 10 and GTSRB. This result indicates that sample diversity is not useful as sample uncertainty in the SSAL setting. The underlying reason may be that the learning procedure has already utilized all the unlabeled data and learned the diverse information in these data. Hence, focusing on uncertain samples is more urgent.
Hence, we conclude that active sample selection can significantly improve performance under the SSL setting. That to say, SSAL is helpful for model performance improvement. Second, we prefer uncertain samples to diverse samples when applying SSAL methods since uncertain samples can bring better results.
4.2 COST COMPARISON
The cost of each method is shown on the right-hand side of Table 3 We normalize the cost of each method w.r.t. the random selection method. The original time cost for RandomSanpling on MNIST, CIFAR-10, 10, and GTSRB are 131, 1047, and 658 seconds, respectively. We have the following observations. First, SSAL methods introduce much more time cost than SAL methods. For example, SSAL methods consume around 4x to 6x energy than SAL methods MNIST. This is because the supi-supervised training will evaluate every unlabeled and labeled sample to calculate the loss. This also means that when the number of unlabeled data increases. the training time will increase. For example, the increase of cost on MNIST is more than that on GTSRB because the number of unlabeled samples in MNIST is more than that in GTSRB.
7

---

Under review as a conference paper at ICLR 2023
98.0
90.0
94.0
Start at
89.0
Start at
88.0
Start at Without
Start 20
Without Al.
8
10
15 20
25
30
(a) MNIST
(b) CIFARIO
Figure 4: The accuracy achieved by SSLDietZAugDirec when using different initial budgets (%). The x-axis represents the start budget measured in percentage, and the y-axis stands for the accuracy. We observe that the performance is negatively correlated with the initial budget. This indicates that should apply the SSAL method early possible to achieve better performance.
IN-DEPTH INVESTIGATION OF SSAL ALGORITHMS
According Section SSAL is the most promising DAL method as it achieves the best performance across multiple datasets. In this section, we aim to conduct an in-depth analysis of the SSAL method and seek provide practical guidance
5.1 START SSAL AS EARLY AS POSSIBLE
The previous session shows the effectiveness of the SSAL methods. In this part, we investigate when to start the SSAL process would be the best, i.e., we want to decide a proper nStart for a given nEnd. To this end, fix the total labeling budget and start the SSAL process different points. Specifically, we set the total budget as 10% and 30% for MNIST and CIFAR-10. respectively. Since the SSAL methods have similar performance results, randomly choose one, e.g., SSLDiff2AuDDireed to generate the result. The result is shown in Figure 4
On the one hand, intuitively, starting the SSAL too late would result in poor performance. The later the SSAL process starts, the more samples are randomly sampled, and the accuracy is closer to SSLRandom. This can be validated by Figure 4 For example, we get around percent accuracy loss if we start from 10% instead of 2% on MNIST. This accuracy loss is enormous. On the other hand, some may argue that starting the SSAL process too early will also cause problems. Because when the labeled data is highly sparse, SSAL can perform worse than SSLRandom (Mittal et al., 2019). To consider this, we start the process from 0.5% for MNIST, and the result shows that the final accuracy achieved by this starting point is the highest. Hence, we suggest starting the SSAL process as early as possible to achieve the best accuracy.
5.2 THE MORE UNLABELED DATA, THE BETTER
From Table we observe that the effectiveness of SSAL methods is different for different datasets. We use the performance difference between specific SSAL method and SSLRandom to reflect the effectiveness of that SSAL method. For example, the performance of SSLDiff2AugDirec over SSLRandom are 3 3.49/1.68 2.07, 5.78/4.57=1.26. 1.19/1.11=1.09 and 6.33/5.97=1.00 for MNIST, CIFAR-I0, GTSRB, and ImageNet, respectively. We hypothesize that the performance improvement of SSAL methods is correlated to the number of unlabeled samples available for selection. As shown in Table 5.2 the average number of samples are 6000, 5000, 911, and 500 for MNIST, CIFAR 10, GTSRB, and Tiny-ImageNet, respectively. More average samples per class mean more data is available for selection, which case we have a higher probability obtaining valuable samples.
To validate this hypothesis, study the effectiveness of SSAL methods by fixing the dataset and only varying the amount unlabeled data. Specifically, we initialize the model with 5000 labeled images randomly drawn from the CIFAR10 dataset. Then, amounts of unlabeled data and use SSLDif22AugDreet for model training and sample selection. The result is shown in Figure 5.2 The x-axis of Figure 5.2 stands for the percentage of data selected
provide the model with different
8

---

Under review as a conference paper at ICLR 2023
2.0
Performance
Average Training Sample Per Class
Performance Improvement
1.4
Dataset
MNIST
6000
2.07
CIFAR-10
5000
1.26
GISRB
911
1.07
1.0
Tiny-ImageNet
1.06
40
50
60
70
80
90
100
Avallable Data
Table 4: The average training sample per class and the performance improvement for differ- ent datasets.
Figure 5: The performance gap between SSLD- iff2AugDirect and SSLRandom w.r.t the number of unlabeled data on CIFAR10 dataset.
from the entire CIFARIO dataset to train the model. For example, if the selected portion is 40%, then the amount of unlabeled data equals to 40% x 50000 5000 = 15000 images. We observe that the performance gap between AigDiiett and SSLRandom increases with the increase of unlabeled data. This result indicates that active data selection performs better if more unlabeled data available. Hence, we recommend the user collect more unlabeled data before performing SSAL.
5.3 THE QUERY SIZE HAS LITTLE INFLUENCE ON SSAL PERFORMANCE
We show the influence of the query size on SSAL methods in Figure 6 Intuitively, the query size can influence the performance of the SSAL meth- ods. If the query size is too small, the selected samples are too less be statistically significant. If the query size is too big, the selection would be coarse-grained.
93.0
92.5
SSLRandom
this experiment, we fix the start- ing budget 10% and the total budget as 50%. Then, we train the model by varying the query size in each iteration. Despite the performance of SLDiff2AuDireed and SSL.Consisteccy decreasing slightly, their performance is still much higher than SSLRandom. Hence, the SSAL methods can per- form well in various query sizes.
92.0
SSL SSLDif2AuDDirect
SSL AiLLDi/AAAKKmaass
S5LConsistency
91.5
91.0
10.0
15.0
20.0
Figure 6: The influence of query size the performance of SSAL methods.
CONCLUSION
This paper studies the effectiveness of existing deep active learning methods. Our study is the most comprehensive one, which covers 19 DAL methods, including the state-of-ther art SAL and SSAL ones. Through extensive experiments, we observe that the emerging SSAL techniques provide promising results. Specifically, the traditional SAL methods can hardly beat random selection, and no SAL method can consistently outperform others. However, SSLAL methods can easily surpass SAL methods, and we find that active sample selection in SSAL can bring huge performance improvement. In this paper, also conduct in-depth analysis of the SSAL methods and give two guidances to the practitioners. First, one should perform SSAL as early as possible and seek more unlabeled data whenever possible achieve better performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. Mcguinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. 2019.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttps://openrevieett wwt. wtt.:///ooorommerrryyygyz.
/
Nathan Beck, Durga Sivasubramaniaan Apurva Dani, Ganesh Ramakrishnan, and Rishabh K. Iyer. Effective evaluation of deep active learning on image classification tasks. CoRR, abs/2106. 15324, 2021. URL https: ptttpp::/arxiv..oo org. abs / 2106..16. 15324
William H. Beluch, Tim Genewein, A. Nürnberger, and Jan M. Köhler. The power of ensembles for active learning in image classification. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9368-9377, 2018.
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.
5050-5066 2019. URL https: / http::://cccceeeeeeeeeeeee neurips cc hccppapper/2019//aaee 90000/9/hahh lcd1 fcd138d04930 68 4bbb wteeeffwwbbbbbbbeeooo www.e0 www.:eeeeee0eeeedd2222
html
Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Sequential graph convolutional network for active learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 9583-9592, Computer Vision Foundation IEFE, 2021. URL
2021 paper.hte.
Gui Citovsky,
Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http:///eeeeeeweeeeeeeeeeeeeaaaa
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint N0X1:1/802.09841,.22 2018.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1199 183--199 192. PMLR, 2017.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ömer Arik, Larry S. Davis, and Tomas Pfister. Consistency-basee semi-supervised active learning: Towards minimizing labeling cost. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X.
Steven C. H. Hoi, Rong Jin, Jianke Zhu, and Michael R. Lyu. Batch mode active learning and its application to medical image classification. In Machine Learning, Proceedings of the Twenty-Tiie International Conference (ICML 2006), Pinssurgh,, Pennsylvania, USA, June 25-29, 2006.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference Neural Networks, number 1288, 2013.
10

---

Under review as a conference paper at ICLR 2023
Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Wei Ma, Mike Papadakis, and Yves Le Traon. Towards exploring the limitations of active learning: An empirical study. In 36th IEEE/ACM International Conference
Automated Software Engineering. ASE 2021, Melbourne, Australia, URL.
November 15-19, 2021, pp. 917-929. IEFE, 2021. doi: 10. 10994AE55522..0011..1.. https://dos htttss://dl.orgg 101..11111//155224452 20....2621166666600 9678672
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2224, 1998.
Christoph Mayer and Radu Timofte. Adversarial sampling for active learning. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pp. 3060-3068. IEEE, 2020. doi: 10.1 /1/////CC55.222000000035355 URL https: /doi. /trg/1.... 1000/.1199/10000 TTT.0////AV5552222 2020. 9093556
Sudhanshu Mittal, Maxim Tatarchenko, Özgün Çiçek, and Thomas Brox. Parting with illusions about abs/1912
deep active learning. CoRR, abs/1912.05580 2019. http:://///axv.ccoo
05361
Prateck Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. CoRR, abs/2002.09564, 2020. URL
https://arxiv. org. abs 2002 2200209564 09564
Nir Nissim, Aviad Cohen, Robert Moskovitch, Asaf Shabtai, Mattan Edry, Oren Bar-Ad, and Yuval Elovici. ALPD: active learning framework for enhancing the detection of malicious PDF files. In IEEE Joint Intelligence and Security Informatics Conference, JISIC 2014, The Hague, The Netherlands, 24-26 September, 2014, pp. 91-98. IEEE, 2014. doi: 10. 10111 URL 110.1119//13000
https: /odii /ttp://6....ggg/111.1/////..... 011.1.01111111111111...... .23
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httts:://ppenreviet net /tt//fouummt dide:idddHHaaia
Rw
Changjian Shui, Fan Zhou, Christian Gagné. and Boyu Wang. Deep active learning: Unified and principled method for query and training. In International Conference on Artificial Intelligence and Statistics, pp. 2308-1316 PMLR, 2020.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 5971-5998 IEEE, 2019, doi: 10.1 10.110///CV.2219000660 URL https: /ttp::://oooo
10. 11 09/ http://ccv.c2....... 201...001110000
Shuang Song, David Berthelot, and Afshin Rostamizadeh. Combining mixmatch and active learning for better accuracy with fewer labels. CoRR, abs/1912.00594, 2019. URL httt:://aaaa
orq/abs/191s 00594
Dan Wang and Shang. new active labeling method for deep learning. 2014 International Joint Conference Neural Networks, IJCNN 2014, Beijing, China, July 6-11, 2014, pp. 112- 119. IEEE, 2014. doi: N0.1109ICCNN22044.6888800
URL
IJCNN 6889477
Donggeun Yoo and In So Kweon. Learning loss for active learning. In IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 93-102. Computer Vision Foundation IEEE, 2019. doi: 10. 10.1109/CVPR.201 00018. URL httpllttp:p:/ppppcccessss thecff. com/ content CVPR 2009/html//// Learning Loss for Active Learning_CVai CVPR 2019 Balil/_pppeeeeeee_e hhml
Jin Yuan, Xingxing Hou, Yaoqiang Xiao, Da Cao, Weili Guan, and Liqiang Nie. Multi-criteria active deep learning for image classification. Knowl. Based Syst. 172:86-94, 2019. doi: 10. knosys.2019.02.113 URL htt.h:ttt///dttc http::://dooooo////// 100 10w. wtw.6//..kooys..220 2019. 02. 02.013
Fedor Zhdanov. Diverse mini-batch active learning. ArXiv, abss1191..00 000954991..0..010 2019.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 EXPERIMENTAL SETTINGS OF DIFFERENT DAL PAPER
We summarize the detailed settings of some representative DAL papers in Table As we can observe, these hyperparameters differ lot in different papers, including the start budget size (Start), step size, learning rate (LR), training epoch, accuracy calculation criteria, and optimizer. This is the reason why cannot compare their performance only by looking for the reported numbers in their paper. Our empirical study unifies these settings and provides a fair comparison of these methods.
Table 5: Hyperparameters used by different active learning methods.
Method
Start
Step Size
LR
Training Epoch
Accuracy Calculation
Optimizer
Terminate when 99% accuracy archived
BadgeSampling
0.2%
0.2%//2%/20%
1e-3
After train
adam
ChuserrMagiinaamlling
10%
10%
1c-3
batch SGD
CoreSet
10%
10%
le-3
RMSProp
KMeunsSampling
10%
10%
1c-3
RMSProp
LearningLoss
2%
2%
lo-1, le-2
200
After train
SGD
MCADL
4%
128
WAAL
2%
2%
lo-2
80
After train
SGD
256
2%
le-2
le-2
200
SGD
Ensemble
0.4%
0.8%
150
RMSprop
VAAT.
10%
5%
le.2
100
Use hest model validate
SGD
A.2 DETAILED PERFORMANCE OF DAL METHOSS
We show the detailed accuracy against the budget curve in Figure 7 Note that we also include the performance of
to privide more information. Tiny-ImageNet is subset of the ILSVRC- 2012 classification dataset. It consists of 200 object classes. All images were down-sampled to 64 x 64 3 pixels. Tiny-ImageNer is more practical and complex dataset. As we can observe, the SSAL methods consistently outperform SAL methods in all datasets.
Artictt eorannpyy Forming Adveranteda0
Aaciyniamsing Chaterturastemshtgg
VCAPI
WhM GINGONN
Randor
Sandonn pg
uncetar-GO
0.0
35
71.
33-
2.5
150
2.
(8) MNIST
(b) CIFAR-10
(b) GTSRB
(b) Tmage-e
Figure 7: The performance of the deep active learning techniques. Note that for better visualization, we only draw the lines for methods that is better than random selection. The x-axis denotes the budget in percentage, and the y-axis denotes the accuracy.
A.3 ROBUSTNESS STUDY OF SAL METHODS
We investigate two factors that can have large influence on the performance of the SAL method: the initial budget and the query size. If the initial budget is too small, the initial model can poorly trained. Selecting new data using this model can cause less optimal results. If the initial budget is too large, most of the labeled samples are randomly selected, and hence the model accuracy would be similar to that of random selection. If the query size is too small, the selected samples are too less to be statistically significant. If the query size is too big, the selection would be coarse-grain -raaseera To study these two factors, we choose five representative methods: BadgeSampling, LeastConfidence, CoreGCN, ClusterMargin, and BALDDropout. The experiment in this section is performed on CIFAR-10.
12

---

Under review as a conference paper at ICLR 2023
00.50
91.0
06.25
90.0
89.0
87.50
88.0
BadgeSampling
CensfCcneieence
87.25
87.0
CoroGEN
Closterrarginneempiigg
86.75
86.0
RAI DDmppatt
10
10
15
30
35
40
(a) Different Initial Budget (%)
(b) Different Query Budget (%)
Figure 8: The performance of five SAL methods under different initial budget and query budget. (a) the impact of the initial budget. The nQuery and nEnd 5% and 40%, respectively. (b) the impact of the query budget. The nStart and nEnd are 10% and 50%, respectively.
Start budget. The influence of the initial budget the performance is shown in Figure 8 (a). In this experiment, we fix the total budget as 30% and the query size as 2% for each method, and train the model with different initial budgets. From Figure (a), observe that the performance of each method does not change much with different initial budgets.
Query size. The influence of the query size in each DAL iteration is shown in Figure 8 (b). In this experiment, we fix the initial budget as 10% and the end budget as 50%. Then, we train the model by varying the query size in each iteration. We observe that the performance decreases when increase the query size. This is because the redundancy can increase when we select more samples at once. Note that our observation is in contrast with Beck et al. (202I), where the authors claims that the query size has little influence on the performance. We suspect the difference is that the query size in Beck et (021) too small reflect the trend.
13


Reviews:
[{"review_id": "c56-MAoEt0v", "reviewer": "Reviewer_9BF2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is an empirical study of the efficacy of DAL on image classification tasks. It benchmarked 19 DAL algorithms, grouped into supervised and semi-supervised and tested on three simple different datasets (MNIST, CIFAR, GTSRB). After benchmarking, it concluded that the semi-supervised active learning techniques does a much better job than SAL and traditional SAL is not always better than even random sampling.", "summary_of_the_review": "I think the paper definitely has its contribution to the field, pointing out the problem in the AL field that they might not work as advertised when benchmarked carefully together and that SSAL performs great. However, as I mentioned in the weakness part, there are several important misses in the current form of the argument that needs to be addressed until this gets published. Therefore I would like to recommend reject/marginally reject of its current form, with a open mindset to change my ratings once those concerns are addressed.", "strength_and_weaknesses": "Strength:\n\nThe drawing of figure 1 is really cute. The related work is well done with enough details and references. The clarity of the paper is strong and evaluation metrics are nicely introduced.\n\nThe scope of the paper is a strength with a focus on benchmarking DALs together in a similar setting. Directly exposing the weakness of current DAL approaches that they sometimes don't even beat random sampling (the majority of them from Table 3).\n\n\nWeaknesses:\n\nThe title is broader than the study. Wil active learning is a more and more broad term used by more communities, since only image classification tasks are studied in this work, I suggest adding such limitation in the title.\n\nThe comparison feels unfair frankly speaking. As this work aims to evaluate the efficacy of \"ACTIVE LEARNING\", comparing semi-supervised AL with random sampling is not fair. Instead, authors should consider comparing SSAL with semi-supervised learning to show the effect of active learning piece.\n\nThe input dimension of the datasets are too small (28 x 28) for image classification tasks from my perspective. As majority of real-life applications would be carried out in larger image scales, whether the same conclusion holds for larger image sizes is a gap for me and impacts the significance of this work.\n\nClaim 5.1 \"START SSAL AS EARLY AS POSSIBLE\": The smallest starting point in the experimental result is 0.5% instead of 0%. With minimal starting labels, the models would not even be appropriately trained, and therefore I highly doubt the efficacy of such a model providing good proposals for unlabeled points to be labeled. I understand the authors' suggestion, but the claim needs to be modified as this is not true (at least not proved if you start from 0.5%, not 0%).\n\nWhat model architecture was used across the different tasks? This seems like a crucial detail that should be included in the experimental setting.\n\nFor the cost in the unit of seconds, I am wondering why some active learning methods can be faster than random sampling? \n\n\n\nQuestions:\n1. For such a comprehensive analysis, the exclusion of ImageNet seems odd to me, if the computational resource is an issue, smaller version of ImageNet can be considered?\n2. Why is the step size of MNIST different from the other two methods?\n3. May I request the proof of \"SGD optimizer performs better than Adam\" in the supplementary? This seems to be controversial that a single citation is not enough.\n4. Did the learning rate have any decay scheduler? Active learning should mimic the real process of the machine learning cycle and therefore, the training process should be mimicking the same effort of a practitioner training for the best performance.\n5. What does Figure 3 convey? I didn't see the contribution of supporting any claims from Figure 3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QbdxlskoOii", "reviewer": "Reviewer_5Vo7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors review 19 methods for supervised and semi-supervised active learning and benchmark their performance on three small CV datasets (MNIST, CIFAR, GTSRB).   The paper shows that compared to supervised active learning, a semi-supervised active learning (SSAL) approach is consistently better (with RandomSampling being a baseline which many of the SAL methods underperform) - Further, they show that SSAL with uncertainty objective improves over SSLRandom and that sample uncertainty is more important than diversity in the SSAL setting. Further, the query size has limited role and higher unlabelled data helps.", "summary_of_the_review": "Overall it is a nicely written survey which makes its point clearly. This is good since there is considerable value to this kind of evaluation work since it gleans insight from existing works each focussed on individual approaches.  However, in terms of contribution, I would have liked to see (1) real-world datasets if focussing only on vision; and ideally, (2) similar analysis in another application domain to highlight wider applicability of the lessons gleaned. \n\n", "strength_and_weaknesses": "+ The paper is clearly written. \n+ Several relevant methods have been compared and key insights clearly highlighted.  \n\n- The datasets are toy datasets \n- I would have been very interested in seeing this evaluation in the context of NLP (where data augmentation has limited benefits compared to vision) and AL/HITL is a key component of modern ML systems. \n\n- Page 7 paragraph 3, \"Second, sample uncertainty is more important than sample diversity during sample selection in\nSSAL.\" I would have liked to see some empirical evaluation highlighting this - the explanation provided in the text is a bit unclear to me (or likely i misunderstood). \n\nMinor corrections: \n\n- Page 5, \"Platform and Hardware\". \"Titian\" -> \"Titan\"  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MZqKb8FD5R", "reviewer": "Reviewer_aA7n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts an expirical study of deep active learning, considering 19 approaches on 4 datasets. For the choice of deep AL approaches, they consider both supervised learning and semi-supervised learning based approaches.\n", "summary_of_the_review": "This paper provides an empirical study of how SSL boosts AL performance. This paper still needs to be improved by enriching the comparisons with related work, especially (Mittal et al., 2019) and [r3].", "strength_and_weaknesses": "Strengths: this paper focus on the difference beween SAL and SSAL and conduct comparative experiments, it's quite interesting and meaningful. We can see from the comparative experiments that SSL does promote the AL performance.\n\nWeakness: the evaluation of a empirical study is to see how many new things it brings to us and whether the experiments are reliable. However, this paper is somewhat limited:\n\n1. About the comparison with existing empirical studies, the author should at least add: 1) [r1], conduct experiments on 19 DAL approaches across 10 datasets, the author said \"Our study is the most comprehensive one, which covers 19 DAL methods,\nincluding the state-of-the-art SAL and SSAL ones\". But the experimental scale of this paper is smaller than [r1], [r1] also contain SSAL ones like WAAL; 2) [r2] DAL methods for NLP tasks; 3) [r3] empricial study of how AL affected by semi-supervised and self-supervised learning. Additionally, the author should provide a detailed analysis of comparison between the findings of their paper and related work like (Mittal et al., 2019) and [r3].\n\n2. The author said \"Another important observation for SAL methods is that: there is no SAL method that can outperform\nothers on every dataset\". But in this empirical study, only 3 datasets are included, furthermore, MNIST is a very simple task and it can achieve more than 95% accuracy with a few data under strong basic classifiers, it's not enough representative. These datasets are just image classification tasks. For other tasks, e.g., in (Mittal et al., 2019), they found that random selection with SSL performs best under semantic segmentation tasks. The basic tasks should be enriched.\n\n3. In Table 3, the performance of learningloss method is very strange. At least in [r1], the performance of LearningLoss is fairly good on CIFAR10. I cannot check the code since the author does not provide the code. Maybe there is because the learning rate is too high, as analysed in [r4] appendix E, high learning rate will lead poor performance.\n\n4. In Table 3, why UncertainyGCN and CoreGCN differs so much?\n\n5. The author said they run each method 5 runs, can author provide variance/standard deviation/error bars of the 5 runs?\n\n6. GTSRB is a imbalanced dataset, but lack of analysis of how different AL methods perform on imbalance data. Does author have new findings?\n\n[r1] Zhan X, Wang Q, Huang K, et al. A comparative survey of deep active learning[J]. arXiv preprint arXiv:2203.13450, 2022.\n\n[r2] Dor L E, Halfon A, Gera A, et al. Active learning for BERT: an empirical study[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 7949-7962.\n\n[r3] Chan Y C, Li M, Oymak S. On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021: 3455-3459.\n\n[r4] Mohamadi M A, Bae W, Sutherland D J. Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels[J]. arXiv preprint arXiv:2206.12569, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-ZL47hkSGC8", "reviewer": "Reviewer_ysmb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs a benchmark of active learning for deep nets (aka DAL). \nThe authors make the distinction between _supervised_ methods and _semi-supervised_ methods (respectively SAL and SSAL) by the ability to harness unlabeled data.\nThe proposed benchmark is comprehensive (as in the amount of methods tested) and puts an emphasis on both on usage of unlabeled data as well as measuring the cost of the *selection* process.\n\nThe supervised flavor of DAL is further divided into three types of data selection:\n- Uncertainty based selection - uses the model's output to estimate the uncertainty a data sample, and select the ones with closest to the decision boundary. \n- Diversity/Representative-ness based selection - selection that maximizes the diversity of the data points, usually over a set of clusters or a metric space.\n- Hybrid selection - a combination of the two former approaches\n\nSemi-supervised approaches perform some of the learning process without the labels.\nThree of the methods here use an approach termed in the literature as *self-supervised*, which are based on comparison of two forward runs through the net, one of which is of an augmented/corrupted version of the data. A fourth approach, WAAL, is base on a adversarial approach of training a discriminator-net to differentiate between the labeled an unlabeled data.\n\n---\n\nOne of the main findings of this work is that unlabeled data should be mapped (as soon a possible), by and unsupervised approach. \nMethods that do not harness such an un/semi- supervised approach might not achieve performance better that random selection", "summary_of_the_review": "This work provides an empirical evidence of how DAL should be addressed in practical sense. The notion that unlabeled data should *preferably* be consumed *earlier* via unsupervised methods is not surprising, but important - and was not addressed in most prior benchmarks. \n\n---\nThe main limitation of this work is that it is based on academic/toy datasets like MNIST and CIFAR, as I detail above.\n\n\n---\n\nNevertheless, this work passes the bar IMHO, and lay a basis for future works to complete the gap(s) I mentioned", "strength_and_weaknesses": "## Strength \nThis paper is well written and easy to follow. \nMost recent DAL methods are tested here, with the important emphasis both on (pre-) processing of unlabeled data, as well as the cost of such an approach.\n\nThe main insight of this work seems important - DAL struggles to beat the ransom baseline without usage of such unsupervised approaches. \n\n## Weakness \n- The one notion that weakens this work is the specific datasets on which this benchmark is based, in two aspects:\n   1) There are substantial evidence that both MNIST and CIFAR are saturated (= contain non-negligible label noise)y, but mainly do not pose challenges like class-imbalance or long-tail distribution, which is the one of the main goals of DAL, IMHO.\n   2) Why did the authors not tackle larger datasets - while there are some initial works using the (old but very large) ImageNet, I would argue that there are quite a few examples between the latter and (the too academic academic) MNIST/CIFAR \n- An important question here is regarding the **re-implementation** of all methods - some of the methods have made the code publically available, and such re-implementation might be unfair (both in performance and cost). I would expect, at the very least, to compare to the original implementation when available - or to state formally that there is little difference in performance or cost.\n- The authors state that **5 runs were made** for each approach, why is the STD not reported?\n- I understand that 2%/5% increments are from the previous benchmarks, but why not test smaller increments?\n- SSLRandom is not properly defined in the text.\n- I would strongly recommend using the term *self-supervised* whenever appropriate - that is whenever an augmented/corrupted version of the data is compared to the original (or another augmented version) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject