Rethinking deep active learning: Using unlabeled data at model training
Authors: Oriane Siméoni
Mateusz Budnik
Yannis Avrithis
Guillaume Gravier
Venue:  (2020)
Keywords: active learning
deep learning
semi-supervised learning
unsupervised feature learning

Abstract: Active learning typically focuses on training a model on few labeled examples alone, while unlabeled ones are only used for acquisition. In this work we depart from this setting by using both labeled and unlabeled data during model training across active learning cycles. We do so by using unsupervised feature learning at the beginning of the active learning pipeline and semi-supervised learning at every active learning cycle, on all available data. The former has not been investigated before in active learning, while the study of latter in the context of deep learning is scarce and recent findings are not conclusive with respect to its benefit. Our idea is orthogonal to acquisition strategies by using more data, much like ensemble methods use more models. By systematically evaluating on a number of popular acquisition strategies and datasets, we find that the use of unlabeled data during model training brings a spectacular accuracy improvement in image classification, compared to the differences between acquisition strategies. We thus explore smaller label budgets, even one label per class. 

Under review as a conference paper at ICLR 2020
RETHINKING DEEP ACTIVE LEARNING: USING UNLABELED DATA AT MODEL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Active learning typically focuses on training a model on few labeled examples alone, while unlabeled ones are only used for acquisition. In this work we depart from this setting by using both labeled and unlabeled data during model training across active learning cycles. We do so by using unsupervised feature learning the beginning of the active learning pipeline and semi-supervised learning at every active learning cycle, on all available data. The former has not been investigated before in active learning, while the study of latter in the context of deep learning is scarce and recent findings are not conclusive with respect to its benefit. Our idea is orthogonal to acquisition strategies by using more data, much like ensemble methods use more models. By systematically evaluating on a number of popular acquisition strategies and datasets, we find that the use of unlabeled data during model training brings a spectacular accuracy improvement in image classification, compared to the differences between acquisition strategies. We thus explore smaller label budgets, even one label per class.
INTRODUCTION
Active learning (Settles, 2009) is an important pillar of machine learning but it has not been explored much in the context of deep learning until recently (Gal et al., 2017; Beluch et al., 2018; Wang al., 2017: Geifman & El-ivn.v. 2017; Sener & Savarese, 2018). The standard active learning scenario focuses on training model on few labeled examples alone, while unlabeled data are only used for acquisition, ie., performing inference and selecting subset for annotation. This is the opposite of what would normally work well when learning deep model from scratch, i.e., training on a lot data with some loss function that may need labels or not. At the same time, evidence is being accumulated that, when training powerful deep models, the difference in performance between acquisition strategies is small (Gissin & Shalev-Shwartz, 2018; Chitta et al., 2019; Beluch et al., 2018).
In this work, focusing on image classification, we revisit active deep learning with the seminal idea of using all data, whether labeled or not, during model training at each active learning cycle. This departs from the standard scenario in that unlabeled data are now directly contributing to the cost function being minimized and to subsequent parameter updates, rather than just being used to perform inference for acquisition, whereby parameters are fixed. We implement our idea using two principles: unsupervised feature learning and semi-supervised learning. While both are well recognized in deep learning in general, we argue that their value has been unexplored or underestimated in the context of deep active learning.
Unsupervised feature learning or self- learning is a very active area of research in deep learning, often taking the form of pre-training on artificial tasks with no human supervision for representation learning. followed by supervised fine-tuning different target tasks like classification or object detection (Doersch et al., 2015; Wang & Gupta, 2015; Gidaris et al., 2018; Caron et al., 2018). To our knowledge, all deep active learning research so far considers training deep models from scratch. In this work, we perform unsupervised feature learning on all data once at the beginning of the active learning pipeline and use the resulting parameters to initialize the model at each active learning cycle. Relying on Caron et al. (2018), we show that such unsupervised pre-training improves accuracy in many cases little additional cost.

---

Under review as a conference paper at ICLR 2020
Semi-supervisedd learning (Chapelle et al., 2006) and active learning can be seen as two facets of the same problem: the former focuses on most certain model predictions on unlabeled examples, while the latter least certain ones. Combined approaches appeared quite early (McCallum & Nigam, 1998; Zhu et al., 2003). In the context of deep learning however, such combinations are scarce (Wang et at., 2017) and have even been found harmful in cases (Ducoffe & Precioso, 2018). It has also been argued that the two individual approaches have similar performance, while active learning has lower cost (Gal al., 2017). In the meantime, research on deep semi-supervised learning is very active, bringing significant progress (Tarvainen & Valpola, 2017; Laine & Aila, 2017; Iscen et al., 2019; Verma et al., 2019). In this work, we use semi-supervised learning on all data at every active learning cycle, replacing supervised learning on labeled examples alone. Relying on Iscen et al. (2019), and contrary to previous findings Wang et al. (2017); Gal et al. (2017), we show that this consistently brings a dramatic accuracy improvement.
Since Iscen et al. (2019) uses label propagation (Zhou et al., 2003a) to explore the manifold structure of the feature space, an important question is whether it is the manifold similarity or the use of unlabeled data during model training that actually helps. We address this question by introducing a new acquisition strategy that is based on label propagation.
In summary, we make the following contributions:
We systematically benchmark number of existing acquisition strategies, as well as new one, on number of datasets, evaluating the benefit of unsupervised pre-training and supervuper learning in all cases.
Contrary to previous findings, we show that using unlabeled data during model training can yield a dramatic gain compared to differences between acquisition strategies.
Armed with this finding, we explore smaller budget (fewer labeled examples) than prior work, and we find that the random baseline may actually outperform all other acquisition strategies by a large margin in cases.
2 RELATED WORK
We focus on deep active and semi-supervised learning as well as their combination.
Active learning. Geometric methods like core sets (Geifman & Elniv,,v 2017; Sener & Savarese, 2018) select examples based on distances in the feature space. The goal is to select a subset of examples that best approximate the whole unlabeled set. We introduce a similar approach where Euclidean distances are replaced by manifold ranking. There are methods inspired by adversarial learning. For instance, a binary classifier can be trained to discriminate whether an example belongs to the labeled or unlabeled set (Gissin & Shalev-Shwartz, 2018; Sinha et al., 2019). Adversarial examples have been used, being matched to the nearest unlabeled example (Mayer Timofte, 2018) or added to the labeled pool (Ducoffe & Precioso, 2018).
It has been observed however that deep networks can perform similarly regardless of the acquisition function (Gissin & Shalev-Shwartz, 2018; Chitta et al., 2019), which we further investigate here. Ensemble and Bayesian methods (Gal et al., 2017; Beluch et al., 2018; Chitta et al., 2019) target representing model uncertainty, which than can be used by different acquisition functions. This idea is orthogonal to acquisition strategies. In fact, Beluch et al. (2018); Chitta et al. (2019) show that the gain of ensemble models is more pronounced than the gain of any acquisition strategy. Of course, ensemble and Bayesian methods are more expensive than single models. Approximations include for instance a single model producing different outputs by dropout (Gal et al., 2017). Our idea of using all data during model training is also orthogonal to acquisition strategies. It is also more expensive than using labeled data alone, but the gain spectacular in this case. This allows the use of much smaller label budget for the same accuracy, which is the essence of active learning.
Semi-supervised active learning has a long history (McCallum & Nigam, 1998; Muslea et al., 2002; Zhu et al., 2003; Zhou et al., 2004; Long et al., 2008). A recent deep learning approach acquires the least certain unlabeled examples for labeling and at the same time assigns predicted pseudo- labels to most certain examples (Wang al., 2017). This does not always help (Ducoffe & Precioso, 2018). In some cases, semi-supervised algorithms are incorporated as part of an active learning evaluation (Li et al., 2019; Sener& & Savarese, 2018). A comparative study suggests that semi-supervised learning
2

---

Under review as a conference paper at ICLR 2020
does not significantly improve over active learning, despite its additional cost due training more data (Gal et al., 2017). We show that this is clearly not the case, using a state of the art semi-supervised method (Iscen et al., 2019) that is an inductive version of label propagation (Zhou et al., 2003a). This is related to Zhu & Ghahramani (2002); Zhu et al. (2003); Long et al. (2008), which however are limited transductive learning.
Unsupervised feature learning. A number of unsupervised feature learning approaches pair match- ing images to learn the representation using siamese architecture. These pairs can come as fragments of the same image (Doersch et al., 2015; Noroozi & Favaro, 2016) or as result of tracking in video (Wang & Gupla, 2015). Alternatively, the network is trained on an artificial task like image rotation prediction (Gidaris et al., 2018) or even matching images to a noisy target (Bojanowski & Joulin, 2017). The latter is conceptually related to deep clustering (Caron al., 2018), the approach we use in this work, where the network learns targets resulting from unsupervised clustering. It interesting that the context of semi-supervised learning, unsupervised pre-training has been recently investigated by Rebuffi et al. (2019), with results are consistent with ours. However, the use of unsupervised pre-training in deep active learning remains unexplored.
3 PROBLEM FORMULATION AND BACKGROUND
Problem. We are given a set X (xihez X. of n examples where I : [n] {1, .....,.. n) and, initially, a collection yo == (Ui)ie of b labels yi E C for i E Lo, where C == [c] is a set of e classes and T.g C I a set of indices with |'ol = < n. The goal of active learning (AL.) (Settles, 2009) is to train a classifier in cycles, where in cycle j = 0, 1.... we use collection y, of labels for training, and then we acquire (or sample) a new batch Sj of indices with |S,| = b to label the corresponding examples for the next cycle j +1. Let L, == - US C I be the set of indices of labeled examples in cycle j 2 1 and U, == I' L, the indices of the unlabeled examples for j 2 O. Then yj =- (yi)ie b, are the labels in cycle and S C Ug is selected from the unlabeled examples. To keep notation simple, we will refer to a single cycle in the following, dropping subscripts j.
Classifier learning. The classifier fa : X + R with parameters 8 maps new examples to vector of probabilities per class. Given x E X, its prediction is the class of maximum probability
t (p) = arg max Pk: kEC
where pk is the k-th element of vector p = fx(x). As by-product of learning parameters 0, we have access to an embedding function th : X + Rdd mapping an example x E X' to a feature vector de (x).. For instance, fe may be a linear classifier
top of features obtained by Φe.
In a typical AL scenario, given a set of indices L of labeled examples and labels y. the parameters of the classifier are learned by minimizing the cost function
J(X, L. = Σ ())))
(2)
RL
on labeled examples x; for E L, where cross-entropy E(p, y) = - log for p E RGG y E C.
Acquisition. Given the set of indices U of unlabeled examples and the parameters resulting from training, one typically acquires a new batch by initializing S + ( and then greedily updating by
t S (a(X, L U, U 0))
(3)
until [S] 2 Here is an acquisition (or sampling) function, each time selecting one example from U U.S S. For each i E S, the corresponding example x, is then given as query to an oracle (often a human expert), who returns label yi to be used the next cycle.
Geometry. Given parameters 6, simple acquisition strategy is to use the geometry of examples in the feature space Fg (2),, without considering the classifier. Each example x, represented by the feature vector de (x,) for i E I. One particular example is the function (Geifman & El- Elniv,,
2017; Sener& & Savarese, 2018)
a X, L, 0: == arg max min [x)o) (xx))
(4)
EE kEL
3

---

Under review as a conference paper at ICLR 2020
each time selecting the unlabeled example in U that is the most distant to its nearest labeled or previously acquired example in L. Such geometric approaches are inherently related to clustering. For instance, + (Arthur & Vassilvitskii, 2007) is probabilistic version of (4).
Uncertainty. A common acquisition strategy that considers the classifier is some measure of uncertainty in its prediction. Given a vector of probabilitiees p. such measure is the entropy
(p) = Σ Pa log
(5)
taking values in [0, log Given parameters 0, each example x, represented by the vector of probabilities fa (x,) for i E I. Then, acquisition is defined by
L,X,L, == arg max H (Jo( iED
(6)
effectively selecting the most uncertain unlabeled examples for labeling.
Pseudo-labels. It is possible to more data than the labeled examples while learning. In Wang ct al. (2017) for example, given indices L, U of labeled and unlabeled examples respectively and parameters 0, one represents example x; by P: fx(x;), selects the most certain unlabeled examples
i = EiE (p.)) c).
:= n (P:) by (1) for i E L. The same cost function J defined by (2) can and assigns pseudo-label h : now be used by augmenting to I. U L. and y to (y, where y == (it)xe This augmentation occurs once per cycle in Wang et al. (2017). This an example of active semi-supervised learning.
Transductive label propagation (Zhou et al., 2003a) refers to graph-based, semi-supervised learning. A nearest neighbor graph of the dataset X is used, represented by a symmetric non-negative adjacency matrix W with zero diagonal. This matrix is symmetrically normalized as W == D- 1-1/2WDDll where D == diag(W1) is the degree matrix and 1 is the all-ones vector. The given labels y == (gr) are represented by n x e zero-one matrix Y x(L,y) where row is c-vector that is a one-hot encoding of label yi if example x; is labeled and zero otherwise,
y). = {
1, i LA ys = k, otherwise
for E I and &e E C. Zhou et al. (2003a) define the n x è matrix := nl((Y)]', where
A(Y) = (1- - a) -WW)
I is the n identity matrix, and 00, is parameter. The i-th row P. of P represents vector of class probabilities of unlabeled example x,, and a prediction can be made by x (P:) (1) for i E U. This method is transductive because cannot make predictions previously unseen data without access to the original data X.
Inductive label propagation. Although the previous methods do not apply to unseen data by themselves, the predictions made on X can again be used as pseudo-labels to train a classifier. This is done in Iscen et al. (2019), applied to semi-supervised learning. Like Wang et al. (2017), a pseudo-label is generated for unlabeled example x; as ÿs := T(p.) by (1), only now p; is the i-th row of the result P of label propagation according
(9) rather than the classifier output
fo (x)). Unlike Wang et al. (2017), all unlabeled examples are pseudo-labeles and an additional cost term Ju (X, U. := Exer (x))) ûs) applies to those examples, where == (ūx)eu and w; : (pp)) is weight reflecting the certainty in the prediction of y::
H(p)
8(p) = log
(10)
Unlike Wang et al. (2017), the graph and the pseudo-labele are updated once per epoch during leamingg in Iscen et al. (2019), where there are no cycles.
We denote by o/ A) = dAag(AA))1 -I A and r//a] := (a 1a-a the (row-wise) L Eynormaaization of nonnega-
tive matrix A and vector respectively.
4

---

Under review as a conference paper at ICLR 2020
Unsupervised feature learning. Finally, it is possible to train embedding function in unsuper- vised fashion. A simple method that does not make any assumption on the nature or structure of the data is Caron et al. (2018). Simply put, starting by randomly initialized parameters 8, the data Φy (X) are clustered by k-means, each example is assigned to the nearest centroid, clusters and assignments treated classes C' and pseudo-labels y respectively, and learning takes place according to (X, 0) (2). By updating the parameters 0, (x(X) is updated too. The method therefore alternates between clusteringgpseuudollbbeiin and feature learning, typically once per epoch.
4 TRAINING THE MODEL ON UNLABELED DATA
We argue that acquiring examples for labeling is not making the best use of unlabeled data: unlabeled data should be used during model training, appearing in the cost function that is being minimized. We choose two ways of doing so: unsupervised feature learning and semi-supervised learning. As outlined in Algorithm 1, we follow the standard active learning setup, adding unsupervised pre- training at the beginning and replacing supervised learning on L by semi- semi-supervised learning on LUU U at each cycle. The individual components are discussed in more detail below.
Algorithm 1: Semi-supervised active learning
Data: data X, indices of labeled examples L, labels y, batch size b U I 0g t PRE(X)
>> indices of unlabeled examples > unsupervised pre-training > active learning cycles > supervised learning on L only epochs > pseudo-labels ý and labels w semi-supervised learning all data
for (0.... do
0 SUP(XX1A y;
for E (L, do
(y, w) + LP(X, L.
0 SUM((X,LUU w:
S+0
while SS < do
acquire a batch S C U for labeling
L S < LUS,X,LUS \S.y:)
11
y + (A,LABE((S)) L LUSUU-U USS
D- obtain true labels on S by oracle b update indices
12
Unsupervised pre-training (PRF) takes place at the beginning of the algorithm. We follow Caron et al. (2018), randomly initializing 0 and then alternating between clustering the features (x(X) by k-means and learning on cluster assignment pseudo-labels ý of X according to IX 0) (2). The result is a set of parameters 6u used to initialize the classifier at every cycle.
Learning per eycle follows inductive label propagation (Iscen et al., 2019). This consists of supervised learning followed by alternating label propagation and supervipe learning on all examples U at every epoch. The supervised learning (SUP) is performed on the labeled examples L only using labels y, according to J(X, y, (2), where the parameters 0 are initialized by tg.
Label propagation (L.P involves a reciprocal k-nearest neighbor graph on features (x( (Iscen et al., 2019). As in Zhou et al. (2003a), the resulting affinity matrix W is normalized as W := D D-1/2W1 D D-1// 1/2 Label propagation is then performed according to P ry (Y)) (9), by solving the corresponding linear system using the conjugate gradient (CG) method (Iscen et al., 2019). The label matrix Y == y) (8) is defined on the true labeled examples I. that remain fixed over epochs but grow over cycles. With p. being the i-th row of P, a pseudo-label iJ = x (P:) (1) and a weight wi = (p.) (10) are defined for every i E U (Iscen et al., 2019).
Semi-supervised learning (SEMI) takes place on all examples LUU = I, where examples L have true labels and examples in U pseudo-labels == (Wi)iet. Different than Iscen et al. (2019), we minimize the standard cost function J(X, LUU, V) (2), but we do take weights w := (w, (ee into account in mini-batch sampling, f noromallze drawn uniformly at random from L, while the other part drawn with replacement from the discrete distribution v/ v/ww on U: an example may be drawn more than once per epoch never.
w/ In particular, part of each mini- batc- is
5

---

Under review as a conference paper at ICLR 2020
Data size
Image size
Mini-barch size
Budget
Total labols
train test
w/o SEMU SEMI
MNIST
600000 / 10000
28
x 28
10/64 64
10
50
32// 128
CIFAR-10
50000/ / 10000
32 32
32/ 128
100
500
CIFAR-10
50000/ 10000
32 32
128
1000
5000
CIFAR- 100
50000/ 10000
32 32
32/ 128
1000
5000
Table 1: Datasets used in this paper. including the mini-batch sizes used in training with and without SEMI, acquisition size at each active learning step and the total number of labeled images.
Discussion.
The above probabilistic weighting decouples the size of the epoch from and indeed we experiment with epochs smaller than n, accelerating learning compared to Iscen et al. (2019). It is similar importance sampling, which is typically based on loss values (Katharopoulos & Fleuret, 2017; Cheng et al., 2018) or predicted class probabilities (Yang et al., 2015). Acceleration is important as training on all examples more expensive than just the labeled ones, and repeated at every cycle. On the contrary, unsupervised pre-trained only occurs once at the beginning.
The particular choice of components is not important: any unsupervised representation learning could replace Caron et al. (2018) in line 2 and any semi-supervised learning could replace Iscen et al. (2019) in lines 4-7 of Algorithm 1. We keep the pipeline as simple as possible, facilitating comparisons with more effective choices in the future.
5 INVESTIGATING MANIFOLD SIMILARITY IN THE ACQUISITION FUNCTION
Label propagation (Zhou et al., 2003a; Iscen et at., 2019) is based on the manifold structure of the feature space, captured by the normalized affinity matrix W. Rather than just using this information for propagating labels to unlabeled examples, can we use it in the acquisition function as well? This important in interpreting the effect of semi-supervised learning in Algorithm I: is any gain due to the use of manifold similarity, or to training the model on more data?
Joint label propagation (jLP), introduced here, is attempt answer these questions. It is acquisition function similar in nature to the geometric approach (4), with Euclidean distance replaced by manifold similarity. particular, the n-vector Y 1e, the row-wise sum of Y x(L.y) (8), can be expressed as Y1, S(L) E IR",, where
{ 1, iEL,
6(L); : 0. otherwise
(11)
for E I. Hence, in the terminology of manifold ranking (Zhou al., 2003b), vector Y1, represents a set of queries, one for each example x; for i E L, and the i-th element of the n-vector h(Y) I in (9) expresses the manifold similarity of x, to the queries for I. Similar to (4), we acquire the example in U that is the least similar to examples in I. that are labeled or previously acquired:
a(X, L, U, 0) == arg min ((())))))))))) iEU
(12)
This strategy is only geometric and bears similarities to discriminative active learning (Gissin & Shalev-Shwartz, 2018), which learns binary classifier to discriminate labeled from unlabeled examples and acquires examples least confidence in the "labeled" class.
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments on four datasets that are most often used in deep active learning: MNIST (LeCun et al., 1998). SVHN (Netzer al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky, 2009). Table presents statistics of the datasets. Following Tarvainen & Valpola (2017); Iscen et al. (2019), we augment input images by 4 x 4 random translations and random horizontal flips.
Networks and training. For all experiments we use 13-layer convolutional network used previously in I Laine& & Aila (2016). We train the model from scratch at each active learning cycle, using

---

Under review as a conference paper at ICLR 2020
Random JLP
CoreSet CFAT.
Uncertainty
Rundom
CoroSet
Unvertaintt
CEAL
PRE
average accuricy
80
50
800
60
45
60
50
40
40
average
40
35
20
30
20
30
3
3
2
3
2
3
(a) SVHN (b 100)
(b) CIFAR-IO (b 100)
(a) SVHN (b 100)
(b) CIFAR- 10 (b 100)
avenuge socuracy
50
Erunce 201u000
50
80
40
75
20
60
20
20
0
2
3
0
2
3
eyele
eyele
Cycle
(c) CIFAR-10 (b = 1000X()) CIFAR-100 (b- = 1000)
(c) CIFAR-10 (b = 1000) CIFAR-100 (b = 1000)
Average accuracy cycle on different setups and acquisition strategies.
Figure 1:
Figure 2: Average accuracy vs. cycle different setups and acquisition strategies with PRE.
SGD with momentum of 0.9 for 200 epochs. An initial learning rate of 0.2 is decayed by cosine annealing (Loshchilov & Hutter, 2017), scheduled to reach zero 210 epochs. The mini-batch size is 32 for standard training and 128 when SEMI is used, except for MNIST where the size of the mini-batch 10 and 64 with SEMI. All other parameters follow Tarvainen & Valpola (2017).
Unsupervised pre-training. We use k-means as the clustering algorithm and follow the settings of Caron et al. (2018). The model is trained for 250 epochs on the respective datasets.
Semi-supervised learning. Following Iscen et al. (2019), we construct a reciprocal k-nearest neighbor graph on features (X), with k 50 neighbors and similarity function s(u, v) [á v}} for u, v E Rdd where ú is the Ez-normalized counterpart of u, while a 0.99 in (9). We follow Iseen et al. (2019) in splitting mini-batche into two parts: 50 examples (10 for MNIST) are labeled and the remaining pseudo-labeled labeled. For the latter, we draw examples using normalized weights as a discrete distribution. The epoch ends when 1 |0] pseudo-labels have been drawn, that is the epoch is 2 50% compared to Iscen et al. (2019). Given that [L] << [U] in most cases, the labeled examples are typically repeated more than once.
Acquisition strategies. We evaluate our new acquisition strategy jLP along with the following baselines: (a) Random; (b) Uncertainty based on entropy (5); (c) CEAL (Wang al,, 2017), combining entropy with pseudo-labels (7); (d) the greedy version of CoreSet (4) (Sener & Savarese, 2018; Geifman & El-Yaniv, 2017).
Baselines. For all acquisition strategies, we show results of the complete Algorithm well as the the standard baseline, that is without pre- pre-training and only fully supervised on labeled examples L, and unsupervised pre- pre-training (PRE) alone without supersuper In some cases, we show semi-supervised (SEMI) alone. For instance, in the scenario of 100 labels per class, the effect of pre-training is small, especially in the presence of semi-supervised. CEAL (Wang et al., 2017) is a baseline with its own pseudo-labels, so we do not combine it with semi-supervised. The length of the epoch is fixed for Algorithm and increases with each cycle.
Label budget and cycles. We consider three different scenarios, as shown in Table 1. the first, we use an initial balanced label set Lo of 10 labels per class, translating into a total of 100 for CIFAR-10 and SVHN and 1000 for CIFAR-IO0 We use the same values as label budget b for all cycles. In the second, we use initially 100 labels per class in CIFAR-10 10 with b 1000 per cycle; this is not interesting for CIFAR- 100 as it results in complete labeling of the training set after 4 cycles. Finally, investigate the use of one label per class both as the initial set and the label budget, MNIST, translating to 10 labels per cycle. All experiments are carried out for 5 cycles and repeated times using different initial label sets Lo. We report average accuracy and standard deviation.
7

---

Under review as a conference paper at ICLR 2020
METROD
CIFAR-IO
CIFAR- 100
Random
CoreSet
Uncertainty SEXI
BUDGRT
100
1000
JIP
CFAT.
CYCLE
jLP
29.17
19.63-1-0.9
PRE
35.20+-2.26
23.71 +:0.86
SEMI
36.734-5.00
25.06
average socumicy
80
PRE SEMI
38.05:22.92
27.04:10.78
CYCLE I
60
JLP
38.8611.36
42.07
33.48:10.55
40
SEMI
46.766:3.27
37.99+2247
48.661:2.44
40.30:11.55
CYCLE
20
0
2
13
jLP
42.3011.511
1..65:1:.21
PRE
47.99-1117
40.81 +:0.00
cycle
SEMI
51,53:3002
46.39::149
MNIST (b 10)
PRE SEMI
51.1811.80
47.03-10447
Figure 3: Average accuracy vs. cycle on different setups and acquisition strategies.
Table 2: Ablation study. Evalua- tion of results obtained with Ran- dom while adding PRE and/or SEMI.
6.2 STANDARD BASELINE RESULTS
We first evaluate acquisition functions without using any unlabeled data. Figure presents results SVHN, CIFAR-10 10 and CIFAR-100. The differences between acquisition functions are not significant, except when compared Random. On SVHN Random appears to be considerably better than the other acquisition functions and worse CIFAR-10 10 with b = 1000. All the other acquisition functions give near identical results; in particular, there is no clear winner in the case of 10 labels per class on CIFAR-10 and CIFAR-100 (Figure (bb) and (d), respectively).
This confirms similar observations made in Gissin Shalev-Shwartz (2018) and Chitta et al. (2019). Our jLP is no exception, giving similar results to the other acquisition functions. We study this phenomenon in Appendix B. In summary, we find that while the ranks examples according to different strategies may be uncorrelated, the resulting predictions of label propagation mostly agree, Even in cases of disagreement, the corresponding examples have small weights, hence their contribution to the cost function is small. Since those predictions used pseudo- in Iscen et al. (2019), this can explain why the performance of the learned model also similar in the presence of semi-supervised learning.
6.3 THE EFFECT OF UNSUPERVISED GRE-TIAINING
As shown in Figure 2, pre-training can be beneficial. PRE by itself brings substantial gain on SVHN and CIFAR-10 with 6 100, up 6%, while the improvements on CIFAR- 100 are moderate. In addition, numerical results in Table for our acquisition strategy jLP show that PRE is beneficial with or without SEMI in most cases. Pre-training provides a relatively easy and cost-effective improvement. It is performed only once at the beginning of the active learning process. While Caron et al. (2018) was originally tested on large datasets like ImageNet YFCC100M, we show that it be beneficial even on smaller datasets like CIFAR-10 or SVHN.
6.4 THE EFFECT OF SEMI-SUPERVISED LEARNING
Figure shows results on different datasets and acquisition strategies like Figure 2, but including both PRE and PRE + SEML For the purpose of reproducibility, numeric results, including average and standard deviation measurements, are given in Appendix for all cycles and datasets.
The combination PRE SEMI yields further significant improvement over PRE and the standard baseline, on all acquisition functions and datasets. For instance, on CIFAR- 10 with a budget of 100, the most noticeable improvement comes from Random, where the improvement of PRE SEMI is around 15% over the standard baseline at all cycles. The improvement is around 10% in most other cases, which is by far greater than any potential difference between the acquisition methods. Also,
8

---

Under review as a conference paper at ICLR 2020
Rundom
ConsSet
Uncertainty
JLP
CEAL
90
60
50
60
50
40
40
30
30
29
0
eycle
cycle
cycle
cycle
(a) SVHN (6 = 100)
(b) CIFAR-10 (b= 100) (c) CIFAR-IO (6 1000) (d) CIFAR-100 (6 = 1000)
Figure 4: Average accuracy vs. cycle on different setups and acquisition strategies: Baseline, PRE and PRE +SEMI. PRE and PRE + SEMI scenarios are represented using different dashed lines as presented in the legend. For reference, the full training accuracy is 96.97% for SVHN, 94.84 % for CIFAR-10 and 76.43 % for CIFAR-100.
noticeably, in the case of SVHN, Random with PRE + SEMI reaches nearly the fully supervised accuracy after just 2 cycles (300 labeled examples in total).
The gain from semi-supervised learning is dramatic in the few-labels regime of CIFAR-10 with b = 100. A single cycle with PRE + SEMI achieves the accuracy of cycles of the standard baseline in this case, which translates to a significant reduction of cost for human annotation.
In Table 2 we present the effect of all four combinations: with/without PRE and with/without SEMI. We focus on our jLP acquisition strategy, which has similar performance as all other strategies and uses manifold similarity just like SEMI. In most cases, PRE improves over SEMI alone by around 2%. The use of PRE appears to be particularly beneficial in the first cycles, while impact decreases as the model performance improves.
is worth noting that CEAL, which makes use of pseudo-labels, has a low performance. This has been observed before (Ducoffe & Precioso, 2018) and can be attributed to the fact that it is using the same set of pseudo-labels every epoch. By contrast, pseudo-labels are updated every epoch in case.
6.5 LABEL PROPAGATION WITH ONE LABEL PER CLASS
Since PRE and SEMI have significant gain in classification accuracy, it is reasonable attempt fewer labeled examples than in previous work on active learning. We investigate the extreme case of label per class using MNIST benchmark, that is, label budget each cycle is equal the number of classes. Figure 3 shows results on all acquisition strategies with and without SEMI. As in the previous experiments, there is consistent winner among the selection strategies alone, and accuracy remain below 80% after 5 cycles (50 labels in total) without SEMI. By contrast, Random with SEMI arrives 90.89% accuracy after two cycles (20 labeled examples), which is 40% better than without SEMI.
DISCUSSION
In this work, we have shown the benefit of using both labeled and unlabeled data during model training in deep active learning for image classification. This leads to more accurate model while requiring less labeled data, which is in itself one of the main objectives of active learning. We have used two particular choices for unsupervised feature learning and semi-supervised learning as components in our pipeline. There are several state of the art methods that could be used for the same purpose, for instance Tarvainen & Valpola (2017); Verma et al. (2019); Berthelot al. (2019); Rebuffi et al. (2019) for semi- suni-supervised learning. Our pipeline is as simple as possible, facilitating comparisons with more effective choices, which can only strengthen the improvement coming from recent acquisition strategies is marginal in many scenarios, an active learning approach that uses unlabeled data for training and not just acquisition appears be very
results. While

---

Under review as a conference paper at ICLR 2020
good option for deep network models. Our findings have impact on how deep active learning is evaluated in the future. For instance, the relative performance the random baseline to all other acquisition strategies depends strongly on the label budget, the cycle and the presence of pre-training and semi-supervised learning.
REFERENCES
D. Arthur and S. Vassilvitskii. K-means++:: the advantages of careful seeding. In SODA. Society for Industrial and Applied Mathematics, 2007.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In CVPR, 2018.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249, 2019.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noisc. In ICML, 2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised
learning of visual features. arXiv preprint arXiv: 1807.05520 2018.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Superviie Learning. MIT Press, 2006.
Bowen Cheng, Yunchao Wei, Honghui Shi, Shiyu Chang, Jinjun Xiong, and Thomas S. Huang. Revisiting pre-training: An efficient training method for image classification. arXiv preprint arXiv: 1811.81111107.. 2018.
Kashyap Chitta,
Alvarez, Adam Lesnikowski. Large-scale visual active learning with deep prohabilistic ensembles. arXiv preprint 1rXiv:181 2019.
Carl Doersch, Abhinav Gupta, and Alexci Efros. Unsupervised visual representation leaming by context prediction. ICCV, 2015.
Melanie Ducoffe and Frederie Precioso. Adversarial active learning for deep networks: margin based approach. arXiv preprint arXiv: /00.:/802.0984 2018.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data, arXiv preprint TrXiiv1703.022 10, 2017.
Yonatan Geifman and Ran Yaniv. Deep active learning over the long tail. arXiv preprimt arXiv:1711.009: 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. 2018.
A. Iscen. G. Tolias, Y. Avrithis, and O. Chum. Label propagation for deep semi- emiissueeeiiee learning. In CVPR, 2019.
Angelos Katharopoulos and François Fleuret. Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.0000 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for sumi-super learning. arXiv: 121.02261. 2016.
arXiv preprint
Samuli Laine and Timo Aila. Temporal ensembling for semi- comi-supervised Icarning. In ICLR, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applicd document recognition. Proceedings of the IEEE, 86(1 1) (122)122222222 1998.
Yanchao Li. Yong li Wang, Dong-Jun Yu. Ye Ning, Peng Hu, and Ruxin Zhao. Ascent: Active supervision for semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, 2019.
Jun Long, Jianping Yin, Wentao Zhao, and En Zhu. Graph-based active learning based on label propagation. In International Conference on Modeling Decisions for Artificial Intelligence, pp. 179-190. Springer, 2008.
Ilya Loshchilov and Frank Hutter, Sgdr: Stochastic gradient descent with restarts. In ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2020
Christoph Mayer and Radu Timofte. Adversarial sampling for active learning. arXiv preprint arXiv:1808.06671. 2018.
M McCallum and K Nigam. Employing em in pool-based active learning for text classification, 1998. In ICML, 1998.
Ion Muslea, Steven Minton, and Craig A Knoblock. Active+ semi-supervised learning robust multi-view learning. In ICML, 2002.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 01 2011.
Mehdi Noroozi Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Sylvestre-Alviss Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with searce annotations. arXiv preprint 10Xx::9905008455 2019.
Ozan Sener Silvio Savarese. Active learning convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708 00489, 2018.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. arXiv preprint arXiv:1904000370, 2019.
Antti Tarvainen Harri Valpola. Mean teachers are hetter role models: Weight-averaged consistency targets improve suni-super deep learning results. In NIPS, 2017.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv preprint 1oXXv:19933.08880 2019.
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification, IEEE Truns. CSVT, 27(12):2511-2600 2017.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. ICCV, 2015.
Weixin Yang, Lianwen Jin, Dacheng Tao, Zecheng Xie, and Ziyong Feng. Dropsample: A new training method to enhance deep convolutional neural networks for large-scale unconstrained handwritten chinese character recognition. arXiv preprint 1xxiv:1505.00 05354, 2015.
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. learning with local and global consistency. In NIPS, 2003a.
Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard SchÖIkopf. Ranking on data manifolds. In NIPS. 2003b.
Zui-Hua Zhou, Ke-Jia Chen, and Yuan Jiang. Exploiting unlabeled data in content-based image retrieval. In ECML, 2004.
Xiaojin Zhu and Zoubin Ghahramani, Learning from laheled and unlabeled data with label propagation. Technical report, 2002.
Xiaojin Zhu, John Lafferty, and Zoubin Ghahramani. Combining active learning and semi semi-supervised learning using gaussian fields and harmonic functions. In ICML 2003 workshop the continuum from labeled to unlabeled data in machine learning and data mining, 2003.
11

---

Under review as a conference paper at ICLR 2020
A ALL DETAILED RESULTS
In order to facilitate reproducibility. in this section we present all the detailed results in Table and Table 4. We describe results obtained with the five methods presented before, namely Random, Uncertainty, CEAL, CoreSet and jLP. We evaluate them on CIFAR-10 with 10 and 100 labels per class (budget 100 and b 1000 respectively), CIFAR- 1-00 with b 1000 in Table 3. We present results obtained on MNIST with only label per class (b = 10) and SVHN with b = 100 in Table 4.
B STUDYING THE AGREEMENT OF ACQUISITION STRATEGIES
It has been observed that most acquisition strategies do not provide a significant improvement over standard uncertainty when using deep neural networks; for instance, all strategies perform similarly on CIFAR-10 and CIFAR-10 100 according to Gissin & Shalev-Shwartz (2018) 1) and Chitta et al. (2019). To better understand the differences, the ranks of examples acquired by different strategies are compared pairwise by Gissin & Shalev-Shwartz (2018). We make step further in this direction, using label propagation as a tool.
B.1 MEASURING AGREEMENT
After the classifier is trained at any cycle using any reference acquisition function a, we apply two different acquisition functions, say (!)) and a²)), followed by labeling of acquired examples and label propagation, obtaining two different sets of predicted pseudo-labels f011) and ((2) and weights w(1)
and (2 on the unlabeled examples U. We define the weighted accuracy
Au. 2')
Σ n[w].d... iEU
(13)
for z, z' RI where is the Kronecker delta function. Using the average weights w : (w(()
w(2)), we then measure the weighted accuracy ( ( expressing the agreement of the two strategies, as well as the weighted accuracy ( (y(k), t) of (*() relative to the true labels t on U for k 1, More measurements include weighted accuracies relative to true labels subsets of U where the two strategies agree or disagree. This way, assuming knowledge of the true labels on the entire set X, we evaluate the quality of pseudo-labels used in semi-supervised learning in each cycle, casting label propagation as an efficient surrogate of the learning process.
B.2 RESULTS
We show results on CIFAR-I0 with b = 1000 in this study. Following the experiments of Gissin & Shalev-Shwarzz (2018), we first investigate the correlation of the ranks of unlabeled examples obtained by two acquisition functions. As shown Figure 5(a), Uncertainty and jLP are not as heavily correlated compared to, for example, CoreSel and Uncertainty in Figure 5(b). The correlation between jLP and CoreSet is also quite low as shown in Figure 5(c).
It may of course be possible that strategies with uncorrelated ranks still yield models of similar accuracy. To investigate this, we measure agreement as described above. Results are shown in Table 5, Uncertainty is used Uncertainty and then measure agreement and disagreement of another strategy to Uncertainty. After cycle 1, any two methods agree on around 80% of the pseudo-labels, while the remaining 20% have on average smaller weights compared to when the methods agree.
reference strategy, i.e. we train the model for number of cycles using
We reach the same conclusions from similar experiment where we actually train the model rather than perform label propagation. Hence, although examples are ranked differently by different strategies, their effect on prediction, either by training or label propagation, is small.
12

---

Under review as a conference paper at ICLR 2020
METHOD
CIFAR 10, 100
CIFAR-IO, 6 1000
CIFAR-100. 1000
SEMI
CYCLE D
100 LABELS
IK LABELS
1x LABELS
Random
29.1741622
39.44268
78.85+0.86
19.63+0.99
27.46-0.52
CYCLE
200 LABELS
26 LABELS
2x LABELS
Random
16.66:1:
1.37
20.69:1.95
75.09:10:51
83.49:0..1
32.44=169
34.88-
40.65=061
Uncertainty
37.59:11.93
40.56:1221
46.04:12.78
76.22.10..8
84.94:1:.35
32.09
0.754=0.70
3.18
CoreSet
39.23-H1.117
43.04::092
48.084-1664
76.44+10.34
84.98
32.05=11400
33.95=0.57
39.63=0.70
38.92:42.00
31.59-0.95
JLP (ours)
38.86
42.07:0074
48.66222.64
75.74:20.9
84.6220047
12.16.199
33.48=0.52
40.30=1.51
CYCLE 2
300 LABBLS
3K LABELS
3x LABELS
Random
42.124183
40
58.72-14.04
79.4
85.134:042
0.53
Uncertainty
13.66:11.57
44.02:11.73
52,04::246
81.26:100.0
87.65100.29
40.43:00.3
41.04 41.0440.27
46.30=1112
CoreSet
43.01:12.144
47.00:12.57
50.85:1-423
81.11 1:0.61
87.21
41.32200700
0.33
46.74
CEAL
41.74:11.15
44.9244200
81.37-t0.54
41.19=041
41.55=0045
JLP (ours)
42.30:11611
1.1
51.18:1.50 so
80.97
87.166:0444
40.65
40.81 =0.40
47.03 47.03=2047
CYCLE 3
400 LABELS
4k LABELS
4x LABELS
Random
45.91:11.63
50.63:10.59
1..37
82.33
86.66:100.21
47.85
47.54
0..3
Uncertainty
47.89-+1.78
50.03-+1.38
$5.47++2.10
84.47 0-449
89.32-t0.24
47.26::0.79
46.39
50.42-00244
CoreSet
46.75:2241
11.40:1999
26.9122000
84.27-20.36
88.75:0045
46.2220.39
46 46.34=00:2
50.85=0.32
CEAL
45.55+42.39
198823+-182
84.05 +-0.44
46.34=0044
46.67
(ours)
45.49
56.67:2.588
88.85
46.52-0.99
45.94=0:44
CYCLE 4
500 LABELS
5k LABELS
5x LABELS
Random
50.94:11.75
$5.31 1.28
64.35:11.37
84. 10:10010010
87.23:1:.21
01.43:0.56
51.40
53.58:10:44
Uncertainty
90,42:+0.28
CoreSet
50.11=11.00
54.17+040
62.94
86.39-t0.6
90.33-t0.13
50.48=0.84
49 49.54=0:95
53.67=1.29
48.1441.24
50.62-0.28
1866.6
jLP (ours)
48.93:12.22
13.8911142
59.83114002
85.94.10.38
89.91
50.24:0.93
50.20=0.44
53.37:00.44
Table 3: Average accuracy and standard deviation for different label budget b and cycle on CIFAR-10 and CIFAR-100. Following Algorithm 1, we show the effect of unsupervised pre-training (PRE) and semi-supervised learning (SEMI) compared to the standard baseline.
-10
10"
CoroSet
CoreSet
3
3
Uncertainty
4
Uncertainty
-10
1044
(b)
(c)
Figure 5: Ranks of examples obtained by one acquisition strategy vs. the ranks of another on CIFAR-10 with b 1000 after cycle A random 5% subset of all examples is shown.
13

---

Under review as a conference paper at ICLR 2020
METHOD
MNIST, 10
SVHN, 100
SEMI
CYCLE
10 LADELS
100 LABELS
Random
26.334415
70.0621287
18.0022447
23.832443
19.11561
Random
51.68-422772
90.8914484
45.95-411.7
53.87-5544
81.25:4-822
Uncertainty
53.18.4-5888
16.12
31.63:88755
2..52+2..6
37 37.84=21.00
CoreSet
57.94:1.116
86.59210.00 10.98
76.39:7.16
52.4925.76
11.80
CEAL
51.57:13.18
38.21:12.70
44.04:14.56
jLP (ours)
48.6004:.15
89.161:5.53
34.04:1:4.75
46.78
24.98
CYCLE
30 LABBLS
300 LABELS
Random
67.31::5.19
91.861:3.9
62.05::3.25
64.882499
29.05
Uncertainty
63:55
13.05
44.09:11349
63.8 33.55
64.14:6.36
CoreSet
63.6613.84
15.28
$2.59499.0
67.23-3001
13.88=13.44
CEAL
56.62:17.05
51,53355.05
63.58:22.80
jLP (ours)
62.71:12.82
80.23-14.11
44.74:11..50
58.43:1..82
66.68
CYCLE
40 LABBLS
400 LABELS
Random
71.05::166
93.388::.99
706288:1.67
72.50=2205
90.69::0.73
Uncertainty
67.87:13.26
93.03:14.88
66.21:3368
70.90:2248
56.6015.60
CoreSet
69.79:13.36
86.93:1762
63.53:1:6.4
71.79:3.558
75.8811695
CEAL
65.244+-743
66.48-4-2.80
68.95+#2005
jLP (ours)
65.55::401
90.75:5576
63.33::9.9
71.20
11.08
CYCLE
Random
76.81:22.19
95.20::1.1
75.78::1.90
7..93=1155
91.44:0.80
Uncertainty
72.8815.82
83.42:550
68.0416.58
76. 1.11
15.42=10:49
CoreSet
75.76:13.93
87.04:16.
66 17:16.16111
75. 13.40
72.51
CEAL
72.02-47.96
66.14:1:1:42
74.48
jLP (ours)
73.36::4.45
92.37::538
60.12:::006
75.33
72.98=12.01
Table Average accuracy and standard deviation for different label budget and cycle on MNIST and SVHN. Following Algorithm 1, we show the effect of unsupervised pre-training (PRE) and semi-supervised learning (SEMI) compared to the standard baseline.
CYCLE
2
MEASURE
Kagree
accuracy (13)
weights
Kagree
accuracy (13)
avg weights
AGREE?
*
#
Random
79,98
79.97
38.39
0.32
0.17
86.98
88.07
39.77
0.46
0.28
CoreSet
80.58
79.52
44.57
0.27
0.16
87.32
87.94
43.80
0.45
0.29
jL.P (ours) 80.24
80.03
48.79
0.27
0.15
86.96
88.12
45.55
0.43
0.27
Table 5: Agreement results between acquisition strategies CIFAR-10 with b 1000 after cycles 1
and 2. All strategies are compared to Uncertainty as reference, which is also employed in the previous cycles. Kagree is percentage of pseudo-labels agreeing the reference. Accuracy is weighted according to (13) and weights are according to (10). Measurements denoted by = (*) refer to the set of labudo-labels that agree (disagree) with the reference.
14


Reviews:
[{"review_id": "SJlVOqRh9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem of incorporating unsupervised (representation pre-training) learning and semi-supervised learning into active learning for image classification; specifically, performing pre-training before active learning starts [Caron, et al., 2018] and then applying inductive label propagation [Issen, et al., 2019] (slightly modification in the cost function to look more like importance sampling) before active learning querying occurs for each round (Algorithm 1).  The most novel technical innovation of this submission is the joint label propagation (jLP) querying function (which is a method of ‘spanning’ the learned manifold space).  Experiments are conducted on four (multi-class) image classification datasets (MNIST, SVHN, CIFAR-10, CIFAR-100), showing that unsupervised learning and semi-supervised learning can improve active learning on these datasets — although random selection often works better (as best as I can tell) implying that negative results are also a contribution of this paper. Finally, some active learning experiments are conducted using a per-round label budget of one example per class — also demonstrating mixed results with random sampling performing better in general. \n\nIn my mind, this paper has two primary components: (1) taking the position that semi-supervised and unsupervised learning can improve overall performance and, in principle, help with active learning and (2) propose jLP, which is a learning algorithm agnostic approach to spanning the manifold space. However, jLP doesn’t really seem to work in general. Thus, the main result is the first point — updating previous (pre-deep learning) results on SS/US AL to deep learning. Honestly, I think the primary conclusion is that semi-supervised and unsupervised learning has improved over the past decade (especially semi-supervised learning for image classification). The second result is that active learning in deep learning (at least for this application) hasn’t kept up. Wrt to (1), as the authors have pointed out, many others have applied semi-supervised learning to AL (including more that the authors didn’t include). Additionally, many have used unsupervised learning for AL (which the authors seem less aware of) from pre-clustering (e.g., [Nguyen & Smeulders, Active Learning using Pre-clustering; ICML04]) to one/few-shot learning (e.g., [Woodward & Finn, Active One-Shot Learning; NeurIPS16 workshop]) to using pre-trained embeddings for many ‘real-world tasks’ (e.g., NER [Shen, et al., Deep Active Learning for Named Entity Recognition; ICLR18] using word2vec). Thus, the interesting question would be to compare multiple pre-training techniques and ideally the relative effect on the active learning component (assuming this is the focus of the paper). With respect to semi-supervised learning, they have validated that inductive label propagation [Issen, et al., 2019] works for this task, but haven’t shown that this helps with active learning. Since this is a negative results without a theoretical contribution, I would again expect trying several semi-supervised algorithm and evaluating their relative performance in general and wrt the active learning querying strategy. Accordingly, I don’t think the contribution of this work in its current state is sufficiently well-developed — and would lean toward rejecting in its current form.\n\nBelow are some additional detailed comments (some also covered above): \n— Given that this points toward a negative result, a more convincing direction to take would be to consider more combinations of unsupervised and semi-supervised approaches — specifically emphasizing how they affect the active learning component. This might point to more general findings and maybe toward a theory (maybe even consider a second application).\n— The empirical emphasis is more around overall performance rather than the interaction between unsupervised representation learning and active learning, which is more toward the stated goal of the paper.\n— Wouldn’t the right way to do (deep) representation learning in multiple rounds be to fine-tune at least some fraction of the time?  If the only claim is pre-training or pre-clustering, people certainly do this — just often not as a point of emphasis.\n— The ‘first semi-supervised’ claim really only holds in the context of deep learning; however, scope is really more like semi-supervised applied to image classification, which would be a pretty narrow contribution in scope.\n— Overall, there is a general overstatement of contributions and results: this is certainly not the first SSAL or USAL and the statement relative to deep learning is subtle; some of the empirical results are interesting, but I am not sure about ‘spectacular gains’ (and these gains aren’t seemingly due to the contribution of the paper).\n— I don’t understand the ensemble model analogy in the abstract; is it because it is a ‘meta-algorithm’?\n\nSome more positive notes: \n+ It is interesting that there is some contradictory evidence relative to [Wang, et al., 2017; Gal, et al., 2017]; this is probably worth digging into a bit deeper.\n+ The experimental details well-described given space constraints.\n\nIn summary, there are some interesting observations that are probably worth pursuing. However, the current contribution is basically that: (1) active learning doesn’t seem to really help, (2) semi-supervised learning and unsupervised learning improve performance for this task. Since (1) was really the point of the paper (as stated) in the title, I don’t think there is enough here to accept in its current form.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJg_1902tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the setting where unsupervised/semi-supervised learning is combined with active learning. The results are that active learning doesn't really help. This paper is interesting in that it provides additional experiments for the intersection of active learning and unsupervised/semi-supervised learning. However, I don't really see the point of this paper. Active learning and unsupervised/semi-supervised learning have been combined before and there are other papers submitted to ICLR this year that combine these. The paper does not claim to provide anything new algorithmically (other than jLP which appears to work no better than random and isn't really advertised as the point of this paper). The only conclusion that I can draw is that sometimes unsupervised/semi-supervised learning works better than active learning, but no understanding of when and why this is the case (from other papers, it is not always the case).\n\n\nComments:\n\n - Although the paper claims to yield a general framework, it only does so partially. For instance, the framework in this work is restricted to semi-supervised methods that use pseudo-labels. \n\n - It may be the case that active learning doesn't help or even hurts because the batch size is too large and/or the initial seed set size is too small. Although this paper varies the acquisition strategies, these other hyper-parameters are equally, if not more, important.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJe-PntOtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that active learning (AL) methods shold combine unsupervised and semi-supervised learning during the iterative training process. Combining these complementary is indeed sensible, and this work is therefore a welcome effort. However, the results are quite mixed, and in fact seem to suggest that AL is rather ineffective. Therefore, what one might take from these results is that unsupervised and semi-supervised learning methods can boost predictive performance; but I think this is widely appreciated already. Perhaps a better framing for this work is: AL using standard metrics seems to be comparatively ineffective, especially when one uses pre-training/semi-supervised learning. \n\nSome specific comments and questions:\n\n- The authors have decided to frame this paper in terms of improving AL using un/semi-supervised learning. But given that, by the authors' own admission, the \"random baseline may actually outperform all other acquisition strategies by a large margin\", what is the motivation for adopting \"AL\" at all? I mean, if we are performing random (iid) sampling, this just reduces to vanilla learning with pre-training and semi-supervision; the 'active' component becomes irrelevant.\n\n- I think the characterization of AL is not quite right on page 2. The authors write that AL is focuses on the \"least certain\" instances. This is often true -- namely under the popular uncertainty sampling regime -- but not all acquisition strategies use this heuristic. Indeed, even the geometry method the authors use explicitly ignores classifier confidence. \n\n- The use of sampling in the SSL component is interesting, although an ablation here investigating this specific choice (as opposed to, say, naive sampling with uniform probability over unlabeled instances).\n\n- I would not characterize the gains brought by unlabeled data here as \"spectacular\".\n\n- As is often the case in work on AL, there is no real notion of a 'test set' here; instead the authors repeat experiments using different seed label sets. It is not entirely clear how much hyperparameter/architecture fine tuning was performed informally, but there is a lot going on here, so I would assume at least some. Therefore there is a risk that all results reported are in some sense optimistic, potentially being \"overfit\" to these datasets. It would be best to provide additional comparisons of approaches on completely unseen datasets. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper argues that incorporating unsupervised/semi-supervised learning into the training process can dramatically increase the performance of models. In particular, its incorporation can result in performance gains that dwarf the gains obtained by collecting data actively alone. The experiments effectively demonstrate this phenomenon. 

The paper is written with a tone that implicitly assumes that "active learning for deep learning is effective" and therefore it is a surprise and a challenge to the status quo that using unlabelled data in intelligent ways alone gets such a boost. On the contrary, reviewers found that active learning not working very well for deep learning is a well-known state of affairs. This is not surprising because the most effective theoretically justifiable active learning algorithms rely on finite capacity assumptions about the model class, which deep learning disobeys. 

Thus, the reviewers found the conclusions to lack novelty as the power of semi-supervised and unsupervised learning is well known. Reject. 
Decision: Reject