Consistency-Based Semi-Supervised Active Learning: Towards Minimizing Labeling Budget
Authors: Mingfei Gao
Zizhao Zhang
Guo Yu
Sercan O. Arik
Larry S. Davis
Tomas Pfister
Venue:  (2020)
Keywords: Active learning
semi-supervised learning

Abstract: Active learning (AL) aims to integrate data labeling and model training in a unified way, and to minimize the labeling budget by prioritizing the selection of high value data that can best improve model performance. Readily-available unlabeled data are used to evaluate selection mechanisms, but are not used for model training in conventional pool-based AL. To minimize the labeling budget, we unify unlabeled sample selection and model training based on two principles. First, we exploit both labeled and unlabeled data using semi-supervised learning (SSL) to distill information from unlabeled data that improves representation learning and sample selection. Second, we propose a simple yet effective selection metric that is coherent with the training objective such that the selected samples are effective at improving model performance. Our experimental results demonstrate superior performance with our proposed principles for limited labeled data compared to alternative AL and SSL combinations.  In addition, we study the AL phenomena of `cold start', which is becoming an increasingly more important factor to enable optimal unification of data labeling, model training and labeling budget minimization.  We propose a measure that is found to be empirically correlated with the AL target loss. This measure can be used to assist in determining the proper start size.

Under review as a conference paper at ICLR 2020
CONSISTENCY-BASED SEMI-SUPERVISED ACTIVE LEARNING: TOWARDS MINIMIZING LABELING BUD- GET
Anonymous authors Paper under double-blind review
ABSTRACT
Active learning (AL) integrates data labeling and model training minimize the labeling cost by prioritizing the selection of high value data that can best im- prove model performance. Readily-availabe unlabeled data used for selection mechanisms, but are not used for model training in most conventional pool- based AL methods. To minimize the labeling cost, we unify unlabeled sample selection and model training based on two principles. First, we exploit both labeled and unlabeled data using semi-supervised learning (SSL) to distill information from unlabeled data that improves representation learning and sample selection. Sec- ond, we propose a simple yet effective selection metric that is coherent with the training objective such that the selected samples are effective at improving model performance. Experimental results demonstrate superior performance of our pro- posed principles for limited labeled data compared alternative AL and SSL combinations. In addition, we study an important problem "When can we start AL?". We propose measure that is empirically correlated with the AL target loss and can be used to assist in determining the proper start point.
INTRODUCTION
Deep learning models are improved when trained with more labeled data (Goodfellow et al., 2016). standard deep learning development procedure involves constructing a large-scale labeled dataset and optimizing model with it. Yet, in many real-world scenarios, large-scale labeled datasets can be very costly to acquire, especially when expert annotators are required, as in medical diagnosis and loan prediction. An ideal framework would integrate data labeling and model training improve model performance with minimal amount of labeled data.
Active learning (AL) (Balcan et al., 2009) assists the learning procedure by judicious selection of unlabeled samples for labeling, with the goal maximizing the ultimate model performance with minimal labeling cost. We focus pool-based AL where unlabeled data pool is given initially and the AL. mechanism iteratively selects batches to label in conjunction with training. As the name "Jearning- based AL selection" suggests, each batch is selected with guidance from the previously- trained model, is labeled, and then added into the labeled dataset on which the model is trained.
Maximization of performance with minimal labeled data requires properly leveraging model learn- ing and AL sample selection, especially in early AL cycles. While using the unlabeled pool as the candidates in the AL sample selection phase, natural for pool-based AL methods to inte- grate SSL objectives to improve performance by learning meaningful data representations from the unlabeled pool (Zhu et al., 2003; Tomanek Hahn, 2009). However, fewer existing AL methods consider SSL during training (Drugman et al., 2019a; Rhee et al., 2017: Zbu et al., 2003; Sener & Savarese, 2018) compared to those utilizing only labeled samples. Moreover, we believe that AL selection criterion should be in coherence with the SSL objectives to select the most valuable samples, since (1) unsupervised losses could alter the learned representation and decision manifolds significantly (Oliver al., 2018), and the AL sample selection should reflect that; (2) SSL already results in the embodiment knowledge from unlabeled data in a meaningful way; thus AL selection should reflect the extra value of the labeled data top of it. Motivated by these observations,

---

Under review as a conference paper at ICLR 2020
propose an AL framework that combines SSL with AL and also selection metric that is strongly related to the training objective.
In the absence of labeled data, a common practice to initiate AL is to uniformly select small start- ing subset of data for labeling. Learning-bart based AL selection is then used in subsequent cycles. The size of the starting subset affects AL performance when the start size is not sufficiently large, the models learned in subsequent AL cycles are highly-skewed and result in biased selection, a phe- nomenon commonly known the cold start problem (Konyushkova et al., 2017: Houlsby et al., 2014). When cold start issues arise, learning-based selection yields samples that lead to lower per- formance improvement than using naive uniform sampling (Konyushkova et al., 2017). Increasing the start size alleviates the cold start problem, but consumes a larger portion of the labeling bud- get before learning-basee AL. selection utilized. With better understanding of data, our method relieves this problem by allowing learning-base sample selection to be initialized from much smaller start size. However, an ideal solution is determining a proper start size that is large enough in avoiding cold start problems, yet sufficiently small to minimize the labeling cost. To this end, we propose a measure that is empirically shown to be helpful in estimating the proper start size.
Contributions: We propose simple yet effective selection metric that is in coherent with training objectives in SSL. The proposed AL method is based on an insight that has driven recent advances in SSL (Berthelot et al., 2019; Verma et al., 2019; Xie et al., 2019): a model should be consistent in its decisions between a sample and its meaningfully-disttore
versions. This motivates us to introduce an AL selection principle: sample along with its distorted variants that yields low consistency in predictions indicates that the SSL model is incapable of distilling useful information from that un- labeled sample, thus human labeling is needed. Experiments demonstrate that our proposed metric outperforms previous methods integrated with SSL. With various quantitative and qualitative anal- yses, we demonstrate the rationale behind why such a selection criteria is highly effective in AL. In addition, in an exploratory analysis we propose measure that can be used to assist in determining the proper start size to mitigate cold start problems.
2 RELATED WORK
Extensive research has been done in AL (Dasgupta et al., 2008; Dasgupta & Hsu, 2008; Balcan et al., 2009; Cortes et al., 2019a). Traditional AL methods can be roughly classified into three categories:
uncertainty-basee based methods, diversity-basee methods and expected model change- change-based methods. Among mncrraiitty-aseer ones, methods based on max entropy (Lewis & Catlett, 1994; Lewis & Gale, 1994) and max margin (Roth& & Small, 2006; Balcan et al., 2007: Joshi et al., 2009) are popular for their simplicity. Some other uncertainty-ban based methods measure distances between samples and the decision boundary (Tong & Koller, 2001; Brinker, 2003). Most uncertainty-based methods use heuristics, while recent work (Yoo & Kweon, 2019a) directly learns the target loss of inputs jointly with the training phase and shows promising results. Diversity-based methods select diverse samples that span the input space the most (Nguyen & Smeulders, 2004; Mac Aodha et al., 2014; Hasan & Roy-Chowdhury 2015; Sener & Savarese, 2018). There are also methods that consider both uncertainty and diversity (Guo, 2010; Elhamifar et al., 2013; Yang et al., 2015). The third category estimates the future model status and selects samples that encourage optimal model improvement (Roy & McCallum, 2001; Settles et al., 2008; Freytag et al., 2014).
Both AL and SSL aim to improve learning with limited labeled data, thus they are naturally related. Only a few works have considered combining AL and SSL in different tasks. In Drugman et al. (2019b), joint application of SSL and AL is considered for speech understanding, and significant error reduction is demonstrated with limited labeled speech data. For AL, their selection criteria based on confidence score that quantifies the observed probabilities of words being correct. Rhee et al. (2017) propose an active semi-supervised learning system which demonstrates superior performance in the pedestrian detection task. Zhu et al. (2003) combine AL. and SSL using Gaussian fields and validate their method on synthetic datasets. Sener & Savarese (2018) also consider SSL during AL cycles. However, in their setting, the performance improvement is marginal when adding SSL in comparison to their supervised counterpart, potentially due to the suboptimal SSL method.
Agreement- based methods, also referred as "querr-by-coommiiii committee", base the selection decisions on the opinions of a committee which consist of independent AL metrics models (Seung et al., 1992; Cohn et al., 1994; McCallumzy & Nigamy, 1998; Iglesias et al., 2011; Beluch et al., 2018; Cortes et al., 2019b). Our method is related agreement-hasee AL where samples are determined based
2

---

Under review as a conference paper at ICLR 2020
Algorithm A semi-supervised learning based AL framework
Require:
Unlabeled data pool D, the total number of steps T, AL batch size K, start size Ko < D
Bn < uniformly sampling from D with |Ba] Ko
Uo Bo
Lo - [(r] x Bo), where (x) stands for the assigned label of x.
for t O,..., do
(training) Mt + arg mİnM / [Tel E(Z,DEEL, Li(e,, Vr M) + पo. ExEU, Lu( (selection) Bi+1 + arg max BCU, [C( M1), ; 8.l. | -B=Kk K}
(labeling) L f- L, U (1,
IE B411
(pool update) Uc+1 f Up Bi+1 end for
MT +- arg minM
[TT E(e)eE Li(,, + गें Eneuy Mu(
return Mr
on the conformity of different metrics or models. Specifically, our method selects data that mostly disagrees with the predictions of its augmentations.
CONSISTENCY-BASED SEMI-SUPERVISED AL
3.1 PROPOSED METHOD
We consider the setting of pool-based AL, where an unlabeled data pool is available for selection of samples label. To minimize the labeling cost, propose a method that unifies selection and model updates, overviewed in Algorithm 1. The proposed method has two key aspects.
Most conventional AL methods base model learning only on the available labeled data, which ig- nores the useful information in the unlabeled data. Our first contribution is incorporating semi- supervised learning (SSL) objective in the training phases of AL. Specifically, as shown in Algorithm I, each model M, is learned by minimizing objective loss function of the form Li + L.- The model should both fit the labeled data well and obtain a good representations of the unlabeled data.
The design of the selection criterion plays crucial role in integrating SSL and AL. To this end, our second contribution is selection criterion C to better integrate AL selection mechanism in the SSL. training framework.
has been observed that predictions of deep neural networks are sensitive small perturba- tions the input data (Zheng et al., 2016; Azulay & Weiss, 2018). Recent successes in SSL (Athiwaratkun et al., 2019; Berthelot et al., 2019; Verma et al., 2019) are based on minimiz- ing the notion of sensitivity to perturbations with the idea of inducing "consistency", i.e., im- posing similarity in predictions when the input is perturbed perceptual content. For consistency-based semi-supervised training, common choice of loss is (x, M) D( f|x, M), P(Ý EJā M)), where D is a distance function such as KL divergence (Xie et al., 2019), or L2 norm (Laine & Aila, 2017; Berthelot et al., 2019) and क de- notes perturbation (augmentation) of the input x. Our proposal is motivated by the following intuition. First, the unsupervised objective exploits unlabeled data by encouraging consistent pre- dictions across slightly distorted version of each unlabeled sample. Labeling samples with highly inconsistent predictions is valuable, since these samples are hard to be minimized using L..- Thus, they need human annotations provide further useful supervision for model training. Second, the data that yields large model performance gain is not necessarily the data with the highest uncertainty, since neural network prefers learning with a particular curriculum (Bengio et al., 2009). The most uncertain data could be too hard learn, and including them in training would be misleading. Thus, we argue that labeling samples that can be recognized to some extent but not consistently should benefit learning more compared the most uncertain ones.
a way that would not change its
Specifically, we propose a simple metric C measures the inconsistency across perturbations. There various ways to quantify consistency. Due its empirically--ossreeed superior performance,
3

---

Under review as a conference paper at ICLR 2020
Methods
labeled samples in total
Setting
100
150
200
250
Uniform
06.13=#0.38
51.10::0.60
53.45=0.71
Supervised
Entropy
41.85
46.05.10.34
50.15:10.79
52.83++0.82
k-center
48.33-1-0.49
50.96-+0.45
53.77=+1.01
Semi-supervised
Our method
83.81
87.57+0.311
89.20
90.23-10.49
Table Comparison with AL methods trained in supervised and semi-supervised setting on CIFAR- 10. All methods start from 100 labeled samples (the third column). The following columns are results of different methods with the same selection batch size.
choose C)) = =xEB (xx, where
E(x, M) = Σ Var p( M), P( M), ... P(- f(xx, 1=1
(I)
is the number of response classes and N is the number of perturbed samples of the original input data v, {ž1,..,, J. which can be obtained by standard augmentation operations I Our method selects data samples with high C values for labeling, which may possess varying level of difficulty for the model to classify.
3.2 COMPARISONS WITH BASELINES
The practical performance of our method is demonstrated on two commonly used datasets: CIFAR- 10 and CIFAR- 100 (Krizhevsky et al., 2009) on the image classification task. Both datasets have 60K images in total, of which 10K images for testing. CIFAR-10 10 consists of 10 classes and CIFAR- 100 has 100 fine-graine preegrained classes. Different variants of SSL methods encourage consistency loss in different ways. In implementation,
adopt the Mixmatch (Berthelot et al., 2019), which proposes a specific loss term to encourage consistency. Following (Berthelot et al., al, 2019), base model and keep the default hyper-parameters for different settings from (Berthelot et al., 2019). In each cycle, M, is initialized with M, We select K 0.5 LLa samples for labeling by default. 50 augmentations of each image are obtained by horizontally flipping and random cropping, but we observe that augmentations can produce comparable results. For fair comparison, different selection methods start from the same initial model (Mo) and the reported results are over 5 trials.
state-of-thh-arr method,
Wide ResNet-28 (Oliver et al., 2018) with 32 filters the
We consider three representative selection methods for comparison:
Uniform indicates random selection (no AL).
Entropy is widely considered as an uncertainty-bast -baed baseline in previous methods (Sener & Savarese, 2018; Yoo & Kweon, 2019a). It selects uncertain samples that have maximum entropy of its predicted class probabilities.
k-center (Sener Savarese, 2018) selects representative samples by maximizing the distance between a selected sample and its nearest neighbor in the labeled pool. The feature from the last fully connected layer of the target model is used to calculate distances between samples.
As shown in Table 1, our method significantly outperforms the baseline methods which only learn from labeled data at each cycle. When 200 samples in total are labeled, our method outperforms kcenter by 39. 39.24% accuracy. Next, focus comparing different methods in SSL framework. Figure 1 shows the effectiveness of our consistency-based selection in SSL setting by comparing with the baselines, when they are integrated into SSL. Our method outperforms baselines by clear margin: on CIFAR-10, with 250 labeled images, our method outperforms uniform (passive selection) by 2.5% and outperforms k-center, the state-of-the -ht-r-rrttttttt method, by 1.5%. As the number of labels increases, it is harder to improve model performance, but our method outperforms the uniform selection with 4K labels using only 2K labels, halving the labeled data requirements for the similar performance. Given access to all the labels (50K) for the entire training set, fully-supervised model
We follow https//gituu..cmm// wwwpgooggggg.errr///miiiiaa randomly flipped and then randomly cropped.
to perform data augmentation: the input images
4

---

Under review as a conference paper at ICLR 2020
Comparison cifar10
Comparison cifa/100
58. tyrn 55, exree
53 2mmm
92-
86
Accuracy
90
Accuracy
60
84
20
0
500
1,00
3500
2000
2500
3000
3500
4000
2000
4000
G000
5000
10000
12000
Number labeled training samples
Number of labeled training samples
Figure 1: Model performance comparison with different sample selection methods on CIFAR-10
and CIFAR-100. Solid lines are average results over 5 trials. Shadows represent standard deviation.
# of labeled samples in total
Methods
250
500
1000
2000
4000
Uniform
87.78:10.23
90.50++0221
01.15=0..15
92.81=0.17
93.45=t0.16
Entropy
88.24
89.95=0.58
91.53:+0.35
92.42::0.53
93.28=00.61
k-center
88.75:10.42
90.94110.53
92.24.10.24
93.30:10.21
94.03-10.25
Our method
90.23:10.39
91.84:10.29
9.22
93.78:10.38
94.57+10.06
Table Comparison of different sampling methods in the SSL setting on CIFAR-10 10. Note that all the methods are under the SSI. setting and start from 100 labeled samples. When the number of labeled samples reaches 250, AL batch size K is set be Tz2]-
achieves an accuracy of 95.83% (Berthelot et al., 2019). Our method with 4K examples has 30% more error compared to the fully supervised method. CIFAR- 100 is more challenging dataset as it has 10x more categories. On CIFAR-100, we observe a consistent outperformance our method at all AL. cycles.
3.3 ANALYSES CONSISTENCY-BASHR SELECTION
To build insights on its superior performance, we analyze the samples selected by our method from several attributes, which are known to important for AL..
Uncertainty- -bsed AL methods query the data samples close to the decision boundary. However, deep neural networks yield poorly-calibrated un- certainty estimates when the raw outputs are considered they tend to be overconfident even when they are wrong (Guo et al., 2017; Lakshminarayanan et al., 2017). entropy-based AL metrics would not distinguished such overconfident mis nis-classificationns, thus result in suboptimal selection. Fig- ure 2 (left) demonstrates that our consistency- based selection is superior in detecting high-confident than entropy. We entropy measure the uncertainty of the selected samples by different methods in Figure 2 (middle). It compares different approaches and shows that uniform and k-center methods do not base selection on uncertainty at all, whereas consistency tends to select highly-uncertaie samples but not necessarily the top ones. Such samples should contribute the performance gap with entropy. Figure 2 (right) illustrates some selected samples that mis-classified with high confidence.
Uncertainty and overconfident mis-classffication:
mis-classification
Diversity: Diversity has been proposed as key factor for AL (Yang al., 2015). k-center is -tate-of-the-art AL method based on diversity (it prefers to select data points that span the whole input space). Towards this end, Figure (right) visualizes the diversity of samples selected by dif- ferent methods. We use principal components analysis to reduce the dimensionality of embedded samples to a two-dimensional space. uniform chooses samples equally-likely from the unlabeled pool. Samples selected by entropy are clustered in certain regions. On the other hand, consistency selects data samples as diverse as those selected by k-center. The average distances between top 1% samples selected by different methods shown in Figure 3 (top-left). We can see that tropy chooses samples having small average distances, while consistency has a much larger average distance which is comparable uniform and k-center.
5

---

Under review as a conference paper at ICLR 2020
25L randrnddo methed
Unfomm Enerspy
Boca led Numbar 3 7
Entery
18 18 Are
High-confidence threshold
group
Figure 2: Left: Number of overconfident mis-classified samples in top 1% samples ranked by differ- ent methods. Overconfident samples are defined as those having the highest class probability larger than threshold. Middle: the average entropy of unlabeled samples ranked by different selection metrics. The ranked samples are divided into 100 groups for computing average entropy. Right: Examples of overconfident mis-classifen samples selected by consistency, which yield low entropy yet high inconsistency. Bird to horse indicates bird is mis-classified as horse.
Consistency
Entropy
point distance
03-
02-
Consistency
Entropy
Kcenter
Uniform
Uniform
Kcenter
0.1-
4:2
-02
-0.4-
Error rate
Class distribution
Figure 3: Average distance between samples (top-left): the average pair-wise L2 distance of top 1% unlabeled samples ranked by different selection metrics. Per-class error rate vs. sample class distribution (bottom-left) are shown. Diversity visualization (right): Dots and crosses indicate unla- beled uus-seected) samples and the selected samples (top 100), respectively. Each color represent ground truth class.
Class distribution complies with classification error: Figure 3 (bottom-left) shows the per-class classification error and the class distribution of samples selected by different metrics. Samples selected by entropy and consistency correlated with per class classification error, unlike the samples selected by uniform and k-center.
4 WHEN CAN WE START LEARNING-BASED AL SELECTION?
4.1 COLD-START FAILURE
When the size of the initial labeled dataset is too small, the learned decision boundaries could be skewed and AL selection based on the model outputs could be biased. To illustrate the problem, Figure 4 shows the toy two-moons dataset using simple support vector machine (in supervised setting with the RBF kernel) to learn the decision boundary (Oliver et al., 2018). As can be seen, the naive uniform sampling approach achieves better predictive accuracy by exploring the whole space. On the other hand, the samples selected by max entropy concentrate around a poorly learned boundary. another example, we study the effects of cold start using deep neural networks on CIFAR- 10. shown in Figure 5, Using uniform sampling to select different starting sizes, AL methods achieve different predictive accuracy. For example, the model starting with Ko = 50 data points clearly under-performs the model starting with K, 100 samples, when both models reach 150

---

Under review as a conference paper at ICLR 2020
(m, ny: mn labeled data and n accuracy
Labeled data Unlabeled
Expected boundary
Uniform
Current boundary
-05
-14
14. 0.731
0.95
0.971
(256. 1.00)
(1500. 1.00)
Max entropy
1.
[4. 0.711
(16, 0.86)
164. 0.61)
(128. 0.861
(256. 1.00)
(1500. 1.00)
Figure 4: Illustration of cold-start problems for uncertainty-based AL. The start size and end size has the same labeled data. When the learned decision boundary far away from the expected boundary (the boundary when all labels are available for the entire training set), e.g. the second and third columns, the selected samples by uncertainty-baat based AL is biased.
Iconter
45
Accursy
90
60-
25
starss 50
70
100
70
strrts
150
02
200
02
200
100
155
250
303
350
250 Number lublledd training sumples
150
150
200
300 350
100
L50 200 250
350
Number labeled training sumples
Number traiiing sample.
Figure 5: Performance comparison of different sampling methods trained with SSL framework CIFAR-10 when AL starts from different number of labeled samples.
labeled samples. It may due to the cold start problem encountered when Ko = 50. While, given limited labeling budget, naively choosing large start size is also not practically desirable, because selection. For example, our method starting from
it may lead to under-utilization of learning-basee Ko 100 labeled samples has better performance than starting from 150 200, since have more AI. cycles in the former case given the same label budget.
The semi-supervised nature of our learning proposal encourages the practice of initiating learning- based sample selection from a much smaller start size. However, the learned model can still be skewed extreme early AL stages. These observations motivate propose systematic way of inferring a proper starting size. We analyze this problem and propose an approach to assist in determining the start size in practice.
4.2 AN EXPLORATORY ANALYSIS IN START SIZE SELECTION
Recall from the last step of Algorithm 1, if T is set such that Uz 0. i.e., if the entire dataset has been labeled, then the final model Mr is trained to minimize the purely supervised loss Lı on the total labeled dataset LT. Consider the cross-entropy loss function for any classifier p(Ý) X which call the AL target loss:
Ci [[T,P((YX)]
Σ (x.))ELr
log = y/Xx = x).
(2)
LT
Note that the goal of an AL method can be viewed as minimizing the AL target loss with a small subset of the entire training set Lr (Zhu al., 2003). In any intermediate AL step, expect the loss on the current labeled subset to mimic the target loss. If cold start problems occur, the model does poor job in approximating and minimizing equation The quality of the samples selected

---

Under review as a conference paper at ICLR 2020
Sem--upeeward trainin (CIFAR(0))
Supervised trining (CIFART
3.0
Sent sop---ppdd triming (CIFARTOU)
S Superwied traning (CIFAR100)
2A255
Mar (01.8)
2.
2.375
1.5
4.54
2,350
5,0
23325
475
① 86 Namber of dlabeeddon trning samples
2,300
10500
6000
750
rag
10%80
20000
Figure 6: Empirical risk (i.e. the target loss) on the entire training data (in blue) and cross-entropy between p(Y) and p(Y) show strong correlations in both semi- semi-supervised and supervised settings.
in the subsequent AL cycles would be consequently poor. Therefore, it is crucial understand the performance of the currently-laared model in minimizing the criterion in equation 2. However, since the labeled data set L, at cycle is strict subset of the total training set LT, it impossible to simply plug the most recently learned model Y in equation 2 for direct calculation.
Our approximation to the target loss is based on the following proposition, which gives upper and lower bounds on the expected loss, which the target loss approximates:
Proposition 1. For any given distribution of Y, and any learned model Ý, we have
& Rh [x(11)] = -xx { (p(r(x), [p()))((>)] - H[[(X)]]
- H ((())(()) H[p(x)] log Z
(3)
where H[p, q) is the cross-entropy between two distributions p and q. H [p([)] is the entropy of the random variable X, and 2 ming, p(X = x Y = y)
Proposition 1 indicates that the expected cross-entropy loss can be both upper and lower bounded. In particular, both bounds involve the quantity (pp()), (Y)), which suggests that H(p(Y), p()) could potentially be tracked to analyze R& [p())X)) for different numbers of samples. Unlike the unavailable target loss the entire training set, H (PY)), p(Ý) does not need all data to be labeled. In fact, to compute [I((Y), p(Y),, we just need specify a distribution for Y, which could be assumed from prior knowledge or estimated using all of the labels in the starting cycle.
In Figure 6, we observe a strong correlation between the target loss and H[p(Y),p(()) where Y is assumed to be uniform. We see how H HIp(Y), can be used to identify the trend when the actual target minimized. Particularly, in SSL. setting, a practitioner may set the starting set size to 100 or 150 labeled samples on CIFAR-IO, as the value of H (p(()), essentially ceases decreasing, which coincide with the oracle stopping points if we were given access to the target loss. In contrast, start size of 50 has much higher H (YY),, p(Y)), which leads to less favorable performance. A similar pattern in the supervised learning setting is shown in Figure 6.
CONCLUSION AND FUTURE WORK
We presented a simple pool-based AL selection metric to select data for labeling by leveraging unsupervised information of unlabeled data during training. Experiments show that our method outperforms previous state-of-the-an ahe-the-rt AL methods under the SSL setting. Our proposed metric im- plicitly balances uncertainty and diversity when making selection. The design of our method focuses on the principles of consistency in SSL. For alternative SSL methods based on other principles, it is necessary to revisit AL selection with respect to their training objectives, which will be considered in future work. In addition, we study and address very practically valuable yet challenging question
"When can we start learning- -based AL selection?". We present a measure to assist in determin- ing proper start size. Experimental analysis demonstrates that the proposed measure correlates well with the AL target loss (i.e. the ultimate the supervised loss on all labeled data). In practice, it can be tracked to examine the model without requesting large validation set.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many con- sistent explanations of unlabeled data: Why you should average. International Conference on Learning Representations, 2019.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv: 10..805..117... /2177, 22177,0118.
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. Interna- tional Conference on Computational Learning Theory, pp. 35-50. Springer, 2007.
Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Journal of Computer and System Sciences, 75(1):78 0177788-9277 2009.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning image classification. Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pp. 9368-9377, 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv: 1.K5v09905.022000 02249, 2019.
Klaus Brinker. Incorporating diversity in active learning with support vector machines. In Proceed- ings of the 20th international conference on machine learning (ICML-003, pp. 59 59-66, 2003.
David Cohn, Les Atlas, and Richard Ladner. Improving generaiizaionn with active learning. Machine learning, 15(2):201-221, 1994.
Corinna Cortes, Giulia DeSalvo, Mehryar Mohri, and Ningshan Zhang, Agnostic active learning without constraints. International Conference on Machine Learning, 2019a.
Corinna Cortes, Giulia DeSalvo, Mehryar Mohri, Ningshan Zhang, and Claudio Gentile. Active learning with disagreement graphs. In International Conference on Machine Learning, pp. 1379- 1387, 2019b.
Sanjoy Dasgupta and Daniel Hsu. Hierarchical sampling for active learning. In Proceedings of the 25th international conference on Machine learning, pp. 208
208-215. 215. ACM, 2008
Sanjoy Dasgupta, Daniel J Hsu, and Claire Monteleoni. A general agnostic active learning algo- rithm. In Advances neural information processing systems, pp. 353-360, 2008
Thomas Drugman, Janne Pylkkonen, and Reinhard Kneser. Active and semi-supervised learning in asr: Benefits on the acoustic and language models. arXiv preprint 1903:0193330282. 2019a.
Thomas Drugman, Janne Pylkkönen, and Reinhard Kneser. Active and semi-supervised learning in ASR: benefits on the acoustic and language models. arXiv:1903.0283 2019b.
Ehsan Elhamifar, Guillermo Sapiro, Allen Yang, and S Shankar Sasrty. convex optimization framework for active learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 209-216, 2013.
Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learn- ing with expected model output changes. In European Conference on Computer Vision, pp. 562- 577. Springer, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning. MIT Press, 2016. http: wwwwwwww..o dewpl ea ea www.nnnnnnonnnonm......ooooooooe org.

---

Under review as a conference paper at ICLR 2020
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1321-1330. JMLR. org, 2017.
Yuhong Guo. Active instance sampling via matrix partition. In Advances in Neural Information Processing Systems, pp. 802-810, 2010.
Mahmudul Hasan and Amit K Soy-Chowdhury. Context aware active learning of activity recog- nition models. In Proceedings of the IEEE International Conference on Computer Vision, pp- 4543 4551, 2015.
Neil Houlsby, José Miguel Hernández-Lobato, and Zoubin Ghahramani. Cold-start active learning with robust ordinal matrix factorization. International Conference on Machine Learning, pp- 766-774, 2014.
Juan Fugenio Iglesias, Ender Konukoglu, Albert Montillo, Zhuowen Tu, and Antonio Criminisi. Combining generative and discriminative models for semantic segmentation of ct scans via active learning. In Biennial International Conference on Information Processing in Medical Imaging, pp. 25-36. Springer, 2011.
Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2372- 2379. IEEE, 2009.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Advances in Neural Information Processing Systems, pp. 4225 10.55-423,20000
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. International Conference on Learning Representations, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--aalibatee classifiers for detecting out-tf-distributiinn samples. Neural Information Processing Systems, 2018.
David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pp. 148-156. Elsevier, 1994.
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR94, pp. 3-12. Springer, 1994.
Oisin Mac Aodha, Neill DF Campbell, Jan Kautz, and Gabriel J Brostow. Hierarchical subquery evaluation for active learning on a graph. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 564-571, 2014.
Andrew Kachites McCallumzy and Kamal Nigamy. Employing em and pool-based active learning for text classification. In Proc. International Conference on Machine Learning (ICML), pp. 359- 367. Citeseer, 1998.
Hieu Nguyen and Amold Smeulders. Active learning using pre-clustering, In Proceedings of the twenty-first international conference on Machine learning, pp. 79, ACM, 2004.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018.
Phill Kyu Rhee, Enkhbayar Erdenee, Shin Dong Kyun, Minhaz Uddin Ahmed, and Songguo Jin. Active and semi semi-supervised learning for object detection with imperfect data. Cognitive Systems Research, 45:109-123, 2017.
10

---

Under review as a conference paper at ICLR 2020
Dan Roth and Kevin Small. Margin-based active learning for structured output spaces. In European Conference on Machine Learning, pp. Springer, 2006.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estima- tion of error reduction. ICML, Williamstown, pp. 441-448, 2001.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning Representations, 2018.
Burr Settles, Mark Craven, and Soumya Ray. Multiple-instanee active learning. In Advances in neural information processing systems, pp. 1289-1296, 2008.
Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop Computational learning theory, pp. 287-294. ACM, 1992.
Katrin Tomanek and Udo Hahn. Semi- supi-vuper active learning for sequence labeling. In Proceed- ings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pp. 1039-1047. Association for Computational Linguistics, 2009.
Simon Tong and Daphne Koller. Support vector machine active learning with applications text classification. Journal of machine learning research, 2(Nov):45 66, 2001.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation con- sistency training for semi-supervised learning. International Joint Conferences Artifical Intel- ligence, 2019.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 190.:9944.1848, 2019.
Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 1 13- 2015.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 93-102, 2019a.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In 2019 IEEE Conference on Computer Vision and Pullern Recognition. IEEE, 2019b.
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480 4488, 2016.
Xiaojin Zhu, John Lafferty, and Zoubin Ghahramani. Combining active learning and semi- supervised learning using gaussian fields and harmonic functions. In ICML 2003 workshop on the continuum from labeled unlabeled data machine learning and data mining, volume 3, 2003.

---

Under review as a conference paper at ICLR 2020
APPENDIX
A PROOF OF PROPOSITION
An illustration of Proposition 1
log Ze
Risk
AL cycle!
Figure A1: An illustration of Proposition 1: the blue curve represents the (expected) cross-entropy, and the two red curves are the lower and upper bounds. The value log 2, characterizes the range of the bounds.
Proof. Denote X' as the feature space and JJ as the label space. Note that by Baye's 's formula and the law of total probability, we have
Ra Rapp(pX)) = -Ex { H ((X1))
Σ Σ p(Y y/ 1)) logr(' 1l y/x x))(X x))
xE y=1
p(X = x, =) y) p(ÿ = =v)p(x = x1/8== = 8)
p(X = x²)
p(X x|1 y)
ΣΣ Ex(x= p(X = 8) = - Σ Σ
p(x = x, = y) log
p(X xx)
yELZEX
y=1xex
Σ p(Y = y) logp(p y) ΣΣ p(x x, Y = = log p(x x//²= 5)}
xEX'=11
+ p(x- = Y= log pp(x = a:)l
=
TEX y=1
-H p()))
+ Σ p(X = x)) logip(X = xx]] Σ Σ
p(X = ==,Y= yy [p(X= = 2/ = w)
=
xEX
xEX y=1
-H [()) - H [x(x)] -b((x)
ΣΣ p(x x, = log p(X 2/8² = w)]
(4)
v=
We first give a lower bound. Note that p(X = ==/²= =)) S for any (x. v) E XXx x (J], so equation implies that
Ex { H [[([((xx),x(xx)))) 2 H [p(Y) - H [x(x)
12

---

Under review as a conference paper at ICLR 2020
of labeled samples total
Methods
1000
1500
2000
2500
3000
Uniform
75.38:10.17
77.46=10.3
78.79:0038
80.81=0.28
Entropy
76.31+00118
09.50:+..29
81.3011.31
82.67=10.55
72.93
k-center
77.56++0.30 76.63+10.17
74.25-10.29
79.50++0.20 79.39
81.70=t0.32
Our method
80.9910.39
82.75=0.266
Table 3: Comparison of different sampling methods in the supervised setting CIFAR-10. All methods start from 1000 labeled samples.
To prove the upper bound, denote min(e.))exx(x]](XX x/6 = = (0, 1) where (z,y) X' [J]. Then from equation 4
Ex {H (0011x))x(xx00 SH |0X)) [x(x)] - log 2 ΣΣ
p(X = x, =
2EX =EX'
HH (p()),000) - H [x(X) - log Z.
B MORE DISCUSSION
B.1 CONSISTENCY-BASSE AL IN SUPERVISED LEARNING
We also curious about how well our method performs under supervised learning using only labeled samples. Following Yoo & Kweon (2019b), start with 1000 labeled samples CIFAR-10. As shown in Table 3, after 4 AL cycles (B 500, totaling 3000 labels), uniform, k-center, entropy and our method (consistency) achieve accuracy of 80.81%, 81. 82.67% and 82. 82.75%. respec- tively. It shows that consistency still works even if our model is trained using only labeled samples. However, the improvement of consistency compared to other baseline methods (especially entropy) is marginal.
B.2 OUT-OF-DISTRIBUTION AND CHALLENGING SAMPLES
In real-world scenarios, it very likely that not unlabeled data are irrelevant to the task. There- fore, if sample remains high uncertainty given arbitrary perturbations, it is probably out-of- distribution example (Lee et al., 2018). In addition, selecting the hardest samples are not preferred, because it could be "over-challenging" for current model suggested by the study of curriculum learning (Bengio et al., 2009). It can be easily inferred that our proposed selection can avoid such cases (see equation 1). More exploration of active learning with for future work.
samples is left
13


Reviews:
[{"review_id": "rkghWIQitS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a semi-supervised active learning method to reduce the labeling cost. In the proposed method, a selection criterion to better integrate AL selection mechanism in SSL training framework is designed. The simple metric that aims to measure the inconsistency across a certain number of meaningful perturbations. It considers N perturbed samples of the original input data x, which can be obtained by standard augmentation operations (e.g. random crops and horizontal flips for image data). Then the variance is adopted to quantify consistency.  In this way, the proposed method prefers data samples with high values, which may possess varying level of difficulty for the model to classify. To verify the effectiveness of the proposed method, several baseline methods are compared on several benchmark data sets, and the proposed method has achieved better performance. Meanwhile, to deal with the “cold start” problem, a measure that is found to be empirically correlated with the AL target loss is proposed, and this measure can be used to assist in determining the proper start size. However, there are some minor concerns:\n[1] The consistency of a sample is measured based on the perturbed samples. How to generate these perturbed samples may have a great influence on the query results. In the paper, it said that these samples are generated by standard augmentation operations (e.g. random crops and horizontal flips for image data). This representation is hard to follow in the experiments. If possible, it is better to show in details.\n[2] In the uncertainty of active learning, the samples are selected from different distributions in the unlabeled data, for example, the marginal sampling selects the samples around the classification hyperlanes (Settles, Burr. \"Active learning.\" Synthesis Lectures on Artificial Intelligence and Machine Learning 6.1 (2012): 1-114.). Can you show which samples may be selected in the unlabeled data. In this way, the proposed criterion can be followed more easily.\n[3] In the experiments, whether the proposed method can select a batch of samples at each iteration. How about the influence of the batch size. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyeG10GntB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new combination method for active learning and semi-supervised learning, where the objective is to make predictions that are robust to perturbations (for SSL) and select points for labeling with labels that differ under perturbations. This technique achieves 2x label efficiency over SSL with uniform-random sampling. Additionally, the authors assess (at least for CIFAR-10 with batch size 50) the best starting random seed set as 100 labels, known as K_0 in this work. This work yields pretty good empirical results and has a conceptually unified approach to SSL and active learning building off of recent works. \n\n\nComments:\n\n - This paper compares in Table 1 the difference between just active learning vs active learning + SSL. I'm not sure this is a fair comparison. I think the better comparison is shown in Table 2. \n\n - The authors write that \"when only 100 samples are labeled, our method outperforms kcenter by 39.24% accuracy\". Do the authors mean after 100 additional labels are acquired (so 200 labels) or is this number off?\n\n - Can the authors clarify what is meant by \"or some labels correspond to rare cases, as in self-driving cars\"? Why are such datasets more costly to acquire? Is it because of the size of the self-driving car datasets?\n\n - Although the method is more unified than some other AL + SSL approaches, I wonder if the L_u(x,M) can be made to look more like the C(B,M) = \\sum E(x,M). In particular, L_u(x,M) uses just a single perturbation and a different \"distance\" function than E(x,M) which uses N perturbations.\n\n - The authors state that they lose 1.26% accuracy to the fully supervised model. However, this is very much not within the margin of measurement error and 1.26% accuracy is rather significant for accuracies around 95%. Another way of putting it is that the method in the paper with 4K examples has 30% more error compared to the fully supervised method. Can the authors either change this claim or provide a number of labels where their method achieves the fully-supervised accuracy?\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors leverage advances in semi-supervised learning and data augmentation to propose a method for active learning. The AL method is based on the principle that a model should consistently label across perturbation/augmentations of examples, and thus propose to choose samples for active learning based on how much the estimated label distribution changes based on different perturbations of a given example. The method is intuitive and the experiments provide some evidence of efficacy. However, during discussion there was a lingering question of novelty that eventually swayed the group to reject this paper. 
Decision: Reject