Combining MixMatch and Active Learning for Better Accuracy with Fewer Labels
Authors: Shuang Song
David Berthelot
Afshin Rostamizadeh
Venue:  (2020)
Keywords: active learning
semi-supervised learning

Abstract: We propose using active learning based techniques to further improve the state-of-the-art semi-supervised learning MixMatch algorithm. We provide a thorough empirical evaluation of several active-learning and baseline methods, which successfully demonstrate a significant improvement on the benchmark CIFAR-10, CIFAR-100, and SVHN datasets (as much as 1.5% in absolute accuracy). 
We also provide an empirical analysis of the cost trade-off between incrementally gathering more labeled versus unlabeled data. This analysis can be used to measure the relative value of labeled/unlabeled data at different points of the learning curve, where we find that although the incremental value of labeled data can be as much as 20x that of unlabeled, it quickly diminishes to less than 3x once more than 2,000 labeled example are observed.

Under review as a conference paper at ICLR 2020
COMBINING MIXMATCH AND ACTIVE LEARNING FOR BETTER ACCURACY WITH FEWER LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose using active learning based techniques to further improve the state- of-the-art semi-supervised learning Mix.Match algorithm. We provide thorough empirical evaluation of several active-learning and baseline methods, which suc- cessfully demonstrate significant improvement on the benchmark CIFAR-10 10, CIFAR-100, and SVHN datasets (as much as 1.5% in absolute accuracy). We also provide an empirical analysis of the cost trade-off between incrementally gather- ing more labeled versus unlabeled data. This analysis can be used to measure the relative value of labeled/unlabeled data different points of the learning curve, where we find that although the incremental value of labeled data can be as much as 20x that of unlabeled, quickly diminishes less than 3x once more than 2,000 labeled example are observed.
INTRODUCTION
Sophisticated machine learning models have demonstrated state-of-the-art performance across many different domains, such as vision, audio, and text. However, to train these models one often needs access very large amounts of labeled data, which can be costly to produce. Consider, for example, laborious tasks such as image annotation, audio transcription, or natural language part-of of of-sseech tagging. Several lines of work in machine learning take this cost into account and attempt to reduce the dependence on large quantities of labeled data. In semi-supervised learning (SSL.), both labeled and unlabeled data (which is often much cheaper to obtain) are leveraged to train a model. Unla- beled data can be used to learn properties of the distribution of features, which then allow for more sophisticated and effective regularization schemes. For example, enforcing that examples close in feature space are labeled similarly. Another, different, approach for address costly labeled data is that of active learning (AL). Here, a model is still trained using only labeled data, but extra care is taken when deciding which unlabeled data examples are to be labeled. Often, the data will be labeled iteratively in batches, where at each iteration an update is made to a current view of the distribution over labels and the batch of points is selected from regions where the distribution is least certain. As discussed in depth in the following section, the approaches of semi-supervised and active learning can be complementary and used in conjunction to help solve the problem of costly in labels.
Our Contributions: In this paper, we take MixMatch, the leading remi-supervised learning tech- nique, and thoroughly evaluate its performance when combined with several active learning meth- ods. We find very encouraging results, which show that ofate-of-the-art performance is achieved in the limited label setting on CIFAR- 10, CIFAR-100, and SVHN datasets, demonstrating that combin- ing active learning techniques with MixMatch provides significant improvement. Furthermore, we perform an analysis, exploring the incremental benefits of labeled versus unlabeled data at different points in the learning curve. Given the relative costs of labeled and unlabeled data, such an analysis aids us in deciding how to best spend a given budget in acquiring labeled versus unlabeled data.
The remainder of the paper is organized as follows: We first give a high-level review of active learning related work and the MixMatch algorithm in Sections 2 and B] respectively. The evalu- ated methods and experimental setup are described in Section 4 while the experimental results are presented and analyzed in Section 5 Finally, we conclude in Section

---

Under review as a conference paper at ICLR 2020
2 ACTIVE LEARNING AND RELATED WORK
Active learning (or active sampling) methods are designed to answer the question: Given a limited labeling budget, which examples should I expend my budget on? Like semi-supervised learning, active learning is particularly is useful when labeled data costly or scarce for any host of reasons. Generally, an active learning algorithm iteratively selects samples to label, based on the data labeled thus far as well as the model family being trained. Once the labels are received, they are added to the labeled training set and the process continues to the next iteration, until the budget is finally exhausted. Thus, active learning can be considered to be part of the model training process, where classical model training is interleaved with data labeling. Here give very brief introduction to a few classes of active learning algorithms, and their use with semi-supervised learning.
There are several active learning algorithms with strong theoretical guarantees, which either explic- itly or implicitly define a version space that maintains the set of "good" candidate classifiers based on the data labeled thus far. The algorithms then suggest labeling points that would most quickly shrink this version space (Cohn et al. 1994] Dasgupta et al.] 2008 [eygelzimer et al.) 2009). In practice, tracking this version space can be computationally inefficient for all but the simplest (e.g. linear) model families. Another, related approach, is that of query-by-comiiter where points are selected for labeling based on the level of disagreement between a committee of classifiers, which have been selected using the currently labeled pool. bhery-by-committe methods benefit from theoretical guarantees (Abe & Mamitsukal 1998), as well as practical implementations based on bagging (Seung et al.) 1992). These methods, along with uncertainty sampling that we introduce next, are only a few prototypical examples of active learning algorithms. For a broader introductory survey of algorithms and techniques, please see Settless (2009) and the references therein.
Arguably, the most popular and practically effective active learning technique is that of uncertainty or margin sampling (Lewis& & Gale 1994; Lewis & Catlett 1994). At each iteration, this approach first trains a model using the currently labeled subset, then makes predictions on all unlabeled points under consideration and, finally, it queries labels for those points where the confidence in the models' prediction is the smallest. The notion of confidence can be defined several different ways, as will be detailed in Section 4.2 In this light, we can see that active learning methods and semi- supervised learning methods can work in a complementary fashion. At high level, active learning methods seek to find training examples for which we have least confidence in the underlying labels in order to query for those labels, while semi-supervised learning algorithms can focus on training examples where there strong confidence in the label distribution and reasonable assumption con- clusions can be made regarding unlabeled examples. It is no surprise that combining active learning and semi-supervised learning has been investigated previously (Hoi et al. 22009; Muslea et al. 2002) We highlight few of these examples below (also see Section 7.1 of (Setlles (2009))
One of the earliest lines of work combining the two techniques that of McCallum & Nigam (1998), where the meery-by-commitee framework is combined with expectation maximization (EM) approach that is used to provide pscudo-labels to those examples that have not been queried for true underlying labels by the active learning method. They show that, a text classification task, the improvement in label complexity of the combined method is better than what either of semi-supervised or active learning method alone can provide. In the work of Zhu al. (2003), the two approaches are combined using Gaussian random field model. Given a model trained on the union of currently labeled and unlabeled datasets, the expected reduction in the estimated risk due to receiving the label of a particular example can be computed and greedily optimized. Tur al. (2005) and Tomanck & Hahn (2009) both combine uncertainty sampling to label uncertain points with machine labeling of confident examples and show their effectiveness in applications to a spoken language understanding and sequence labeling tasks, respectively. In this work, we combine uncertainty-samplis based active learning method and demonstrate that the already ofate-of-the-art semi-supervised performance the MixMatch algorithm can be significantly further improved.
MIXMATCH
We only focus deep learning based semi-supervised learning (SSL) in this work. We use the framework from Oliver et al. (2018) to perform a realistic evaluation of the results (same splits of the labeled/unlabeled initial data, same network architecture). Recent popular techniques are built
2

---

Under review as a conference paper at ICLR 2020
around two concepts: Consistency Regularization (Mean Teacher Tarvainen & Valpola 2017
II-model (Laine & Aila 2017). Virtual Adversarial Training (VAT) (Miyato et al., 2018)) which enforces that two augmentations of the same image must have the same label and Entropy Mini- mization (Pseudo-Laabell Lee 2013), Entropy Minimization (EntMin) (Grandvaler & Bengio 2005))
which states that prediction should be confident. These two concepts are sometimes combined, for example the technique VAT EntMin is a combination of VAT and Entropy Minimization and MixMatch is also using both of these concepts.
MixMatch, a recent state-of-the-art semi-supervised learning (SSL) technique, is designed around the idea of guessing labels for the unlabeled data followed by using standard fully supervised training (Berthelot et al. 2019). Consider a classification task with classes C. The input of MixMatch are a batch of (images, labels) pairs X = ((T,, Pb) p5)) where each label ps an one-hot vector over the class C, and a batch of unlabeled examples (images) U = {ms, h5 Note, in this section, a "batch" in reference to the subset of points used in the iterative optimization procedure that is used to train the model, while elsewhere in the paper "batch" refers to set of newly labeled points that are added to the training set during the iterative active learning process.
Label Guessing. Label guessing is done by averaging the training model's own prediction on several augmentation us,s of a same image up. This average prediction is then sharpened to produce a low-entropy soft label qb for each image us- Formally, the average is defined as To _े Di=1 (modal (=,;;; 0) where Pmodel (y xz; 0) is the model's output distribution over class labels y on input x with parameters 6. A sharpening is applied to the average prediction: 96 = Sharpen( In practice, MixMatch uses a standard softmax temperature reduction computed as Sharpen(p)) (p); : P: 1/T /E 1/T where T 1/2 is a fixed
Data Augmentation. MixMatch only uses standard (weak) augmentations. For the SVHN dataset, we use only random pixel shifts, while for CIFAR-10 and CIFAR-100 we also use random mirroring.
Fully supervised. Finally, MixMatch uses fully supervised techniques. In practice, it use weight decay and MixUp across the labeled and unlabeled data (Zhang al. 2017). In essence, MixUp gen- erates new training examples by computing the convex combination of two existing ones. Specifi- cally, it does a pixel level interpolation between images and pairwise interpolation between probabil- ity distribution. The resulting interpolated label is a soft label. Such examples encourage the model to make smooth transitions between classes. Let X = ((rs. Ps)] B and Ú {(us, (b) h be the results of data augmentation and label guessing. MixMatch shuffles the union of the two batches X O into a batch W of size 2B and performs MixUp to produce the output:
X' = MixUp(A, (I) (mixing up X with the first half of W) and U MixUp(Ú, W(n+1,.....1) (mixing up Ú with the second half of W).
Given two examples (I1, and (I2, P2) where 21, are feature vectors and p1. P2 are one-hot encoding or soft label, depending on whether the corresponding feature vector is labeled or not, MixMach performs MixUp as follows:
(1) Sample Beta(a, a:) from a Beta distribution parameterized by the hyper-parameter o: (2) For X' = max X, A), compute x' = Xxi + (1 - X)zz2 and p XPp + -X)p2.
Loss function. Similar to other SSL paradigms,,, the loss function of MixMatch consists of a sum of two terms: (1) a cross-entropy loss between predicted label distribution with the ground-truth label and (2) a Brier score (L2 loss) for the unlabeled data which is less sensitive to incorrectly predicted labels. On a MixMatch batch (X', UU),, the loss function is L = Lx + Xu Ly where
Cx
1
Σ
CrossEntropy (p, Prmotee (2) 0)),
|X'l
(xxp)eX)
Lo
1
Σ (u.O)Et)
11 Pmodel (1/ (y/1 0)122
(2)
CCII""
Here A is the hyper-parameter controlling the importance of the unlabeled data to the training process. We set dy to be 75 for CIFAR-10, 150 for CIFAR-100 and 250 for SVHN. We also fix the MixUp hyper-parameter 0.75.
3

---

Under review as a conference paper at ICLR 2020
4 PROPOSED METHOD AND ANALYSIS
In this section, we describe our proposed method for introducing active learning into MixMatch, we then review several specific active learning strategies which we will be evaluated, and finally we also describe our methodology for analyzing the relative value of labeled versus unlabeled data.
4.1 PROPOSED METHOD
In the standard supervise learning setting, classifier is trained with n samples where only m of them are labeled (m & n.). The m labeled samples are usually considered to be fixed and uniformly sampled subset from the n samples. This also the setting that the original MixMatch algorithm considers (Berthelot et al., 2019). In this section, we propose MMA, combination of MixMatch and active learning as well as deñin method for comparing the relative value of labeled and unlabeled samples.
Direct gains from active learning A natural extension is to consider whether using active learning in place of uniform sampling can improve MixMatch results. In practice, instead of randomly sam- pling m samples to be labeled all at once, we incrementally grow the labeled set as the training pro- cess goes on. Starting with fixed pool of n. unlabeled sample, first randomly sample small la- beled set of Lo of size mg- This set is grown times as the training progresses To C L1 C... C Lx with respective sizes my < mi ... my- The selection process determine what labels 14-1 L, to add at step is done using active learning. For example, from a fixed set of n samples, we start with a set Lo formed by labeling 250 randomly selected samples. After training for some time, we can grow by 50 unlabeled examples that the current model considers "hardest" (i.e., the most uncertain ones).
Determining the worth of labels Both labeled data and unlabeled data have cost. It is commonly the case that labeled data much more expensive than unlabeled data. From a joint active learning and superviper learning perspective, two datasets can be grown: the labeled set I. and the unlabeled set U. In this context, the question to be answered is this: with the goal to reach target accuracy accuracytargeet at time step t is it better to grow the labeled set the unlabeled set? As a corollary question, how do various accuracy targets relate to each other? In other words, can the model response to data be predicted from lower accuracy targets?
4.2 COMPARISON OF POPULAR AL METHODS
is a common strategy in active learning to select the samples that the current model, trained on the data that is available thus far, is most uncertain about. However, directly selecting the most uncertain samples can result a large number of similar samples. Existing methods avoid this issue by picking uncertain yet diverse samples. See [Settles] (2009) for survey of these and several other active learning techniques. The design of MMA's 's active learning strategies considers two components: (I) uncertainty measure and (2) diversification.
Measuring uncertainty. Following previous active learning frameworks, MMA gathers labels of the samples that the current model is most uncertain about. Intuitively, the uncertain samples are often the most helpful for improving the model. The first question we ask is how should we measure uncertainty? We consider two approaches.
For sample the current model predicts its label as probability vector Pmode æz; 0) E [o, 1| where the c-th dimension Pmodet iz; 0). the probability assigning to class c. Let s(x) be the uncertainty of sample Common ways of measuring uncertainty include
max:
Measuring the maximum confidence that the model has in any one label s(x) := 1 maxe (Pmodel (y/ 0)c}-
di diff2: Measuring the gap margin between the two most likely classes, s(x) 1 (Pmodeay)22; e),, Pmodel (g (z)), where ei and C2 the classes with the 1st and 2nd highest probabilities.
Additionally, we reuse MixMatch technique make the uncertainty measurement more robust. Instead of using the model prediction on the original sample, we can average its predictions from K different augmented versions of the example. We call this method aug and combine with
4

---

Under review as a conference paper at ICLR 2020
and diff2. It results in two more measurement methods: max.aug and di diff2.aug. We choose K = 2 to stay consistent with the setting of MixMatch.
Diversification, Always simply sampling the most uncertain samples (which we will call the direct method) can cause the problems that the sampling targets specific subsets of the data (typically those larger classes). In such cases, it is helpful to add form of regularization to ensure diversity within our sampled batch. MMA considers two commonly used diversification methods:
kmeans: We first cluster all unlabeled samples using the k-means clustering algorithm (Lloyd 1982). At each sampling step, we select the top-n uncertain samples from each cluster where the sample size is proportional the cluster size. We fix the number of clusters be 20.
infoD: the information density framework adds an additional term that measures how represen-
tative the sample is, i.e., for sample I, we calculate s'(x) = s(x) x [ Eree sim(ar, x²)
for some uncertainty measure s(.), similarity measure sim(:.-) and some user- ueereeefined value 3. Note, if [ is prohibitively large, s'(.) can be computed using a uniform random subsample. The i f FDD method picks the samples with highest s' values. In our experiment, we set 3 and set sim(:.) )) to be the cosine similarity.
In both methods, we need some representation of samples to measure the distance or similarity of samples. Instead of using the original feature vector, in our experiments, we use the output the second from the last layer of the neural network as the embedding representation of the input.
Other active learning methods. Although other families of active learning algorithms exist (such as those found in Settles (2009)), we find that they not easily adapted to this setting complex and costly to train neural networks. Ensemble-base methods, such as query-by-commitee let al. 1992), require training many copies of model resulting in resource and potentially time neck. Version-space based approaches, such as IWAL (Beygelzimer et al.] 2009) or DHM (Dasgupta et al.) 2008) are not readily applicable in a setting with complex hypothesis space, where either explicitly or implicitly tracking the version space becomes a complex optimization problem itself. Perhaps expected model change (Setiles al. 2008) or expected error reduction (Roy & McCallum 2001) are the baseline methods that are the next most amenable to the setting, however, even in these cases one would need compute gradients/peedictions for each individual example and each possible labeling. In cases where the number of labels is large, e.g. CIFAR-100, the direct use of such methods become impractical.
(Seung
4.3 COST ANALYSIS MODEL FOR LABELED VS UNLABELED DATA
In this section, we formalize how we propose to perform the cost analysis of adding labeled vs unlabeled data. Let e and c, be the costs of obtaining new labeled sample and a new unlabeled sample respectively. For desired accuracy accuracy target- we can obtain several groups of (labeled, unlabeled) samples that allow the trained model to reach this accuracy. In realistic setting, long as accuracy target is less than the model best accuracy, such groups can be created by removing data from U and L until the model reaches the desired accuracy. Formally, let's consider an ordered series of training sets Ti, = (L,, U2.)) where Vi < ki L, C La and Vjj < h, Ut, C U2 Given two contiguous training sets Tr, and Ti+1,k reaching accuracy target we can estimate under what condition better to add labeled data by solving: aq L.] + cu (u]l > ei Li+1l + cu (U In doing so, we obtain the cost ratio Cratso(s) of unlabeled data over labeled data:
Cratio(i)
((Ua.] |U++,,))/ (|L++1)- [..))
If Cratio(i) > ct/Cu, we can conclude is better to collect labeled data and otherwise it is better to collected unlabeled data.
EXPERIMENTS
In this section, we evaluate the effectiveness of MMA by applying it to image classification tasks under the standard sumi-supe and active learning setting. We compare the performance of different active learning variants of MMA with the non-active MixMatch. As an additional contribution, we conduct an extensive cost analysis of active data labeling under varied dataset sizes.
5

---

Under review as a conference paper at ICLR 2020
5.1 DATASETS, EXPERIMENT SETTINGS, AND IMPLEMENTATION DETAILS
We evaluate the effectiveness of MMA on three semi-supervised learning benchmarks: CIFAR-10, CIFAR-100 (Krizhevsky al. (2009), and SVHN (Netzer al. (201T)) CIFAR-IO and CIFAR- 100 each consists of 50000 images classified into 10 and 100 classes. SVHN consists of 73257 images of street view house number, classified into 10 classes. An extra set of 531131 samples exists for SVHN; we call the combination of the two, which is a dataset with 604388 examples, SVHN+Extra.
Our implementation is primarily based on that of MixMatch (see Berthelot et al. 2019) and GitHub
repo referenced therein). The implementation uses the wide ResNet-28 model Oliver al. (2018).
We also utilize weight decay and exponential moving averaging, again following the settings in MixMatch. All hyper-parameters related to MixMatch are the same as (Berthelot et al.) 2019).
Details can be found in Appendix A We report the median of the last 20 checkpoints' accuracy where a checkpoint is computed every 1024 training iterations. Each experiment repeated 5 times with different random initial sets, and we report the mean and standard deviation.
Each run of the experiment is given a fixed labeling budget. The training process proceeds as follows. MMA first starts training the model with the initial labeled examples. After fixed number of training steps, MMA grows the labeled set by querying the labels of some unlabeled examples; this is repeated until the labeling budget is used up. Finally, the model is trained futher until convergence.
To save computational resources, when we vary the budget size for the same active learning algo- rithm, we store the model checkpoint at the end of each labeling interval and resume from it when we increase the budget size.
We compare MMA with two baselines: Mi xMatch and random. Mi xMatch essen- tially considers the passive setting: the set of labeled data is sampled uniformly at random in the beginning and is fixed throughout the entire training process. The random baseline is under the MMA framework, and selects randomly as set of samples to query each time.
Baselines:
5.2 EFFECTIVENESS OF MMA
We test the effectiveness of the different active learning strategies used in MMA. Recall that MMA considers two uncertainty measurements: diff2 and max. Each has its data augmentation vari- ant which we denote as di ff2.aug and max.aug. Moreover, we consider using the uncertainty measurement directly (direct) and with diversification strategics infoD and kmeans. Due to space constraints, we present the accuracy of a subset of representative AL methods (diff2.aug- direct and wi.ff2.aug-kmeea with MMA (best) in Tables 2 and with full tables available in Appendix A.
which are consistently the best performers (see comparison
Several of the MMA variants improve significantly upon MixMatch, which is the state-of-the-ee semi-supervised learning algorithm. With the same labeling budget MMA outperforms the original MixMatch by up to 1.47% for CIFAR-10, by up 1.16% for CIFAR-100, and by 0.43% for SVHN.
CIFAR-10. We evaluate Mi xMatch and all variants of MMA on CIFAR-10 with 500, 1000, 2000 and 4000 labeling budget. We first train the model for 262144 steps with 250 randomly selected labeled samples. Then we grow the labeled set by 50 examples at each active learning iteration. Each time after obtaining more labeled samples, we continue training the model for 32768 steps.
Table 1: Performance of MMA on 5 repeated runs 10.
Label Budgets
500
1000
2000
4000
16000
Mi cixMatch
90.58 + 0.83
91.61 + 0.54
93.20 + 0.11
93.70 + 0.16
94.99 + 0.15
random
90.54 0.82
91.85 + 0.67
92.97
+ 0.29
93.73 0.18
di Ff2.aug--d rect
91.69 + 0.52
92.79
+ 0.41
94.11 0.14
95.17 + 0.13
di auiffe.augkkmeans
91.46 + 0.38
92.62
1 0.32
93.98 1 0.14
95.06 1 0.19
Table I shows the results for CIFAR-10. 10. All the active learning methods outperforms the random or the MixMatch ch baselines (see details from Table Appendix(A.I]. The improvement is consis- tently about 1% across different label budgets. It is clear that acquiring labels actively is beneficial

---

Under review as a conference paper at ICLR 2020
in the semi-supervised setting. MMA is also significantly more label-efficient. Even with only 2000 labels, MMA outperforms MixMatch ch with 4000 (2x) labels 194.11% vs. 93.7%). With 4000 labels, MMA outperforms Mi xMatch with 16000 (4x) labels (95.17% vs. 94.99%). This is only 0.66% lower than the same model trained on all the 50000 (12.5x) examples in the supervised setting (95.17% vs. 95.83%, number is from Berthelot et al. (019))
Comparing the different active learning strategies, we found that di di.ffaaug-drrect consistently ranks top-2 among all active learning variants for CIFAR-IO, with other active learning strategies performing roughly equally well. Additionally, we observe that di diff2 outperforms max on average and it is usually beneficial to use aug (see Tabel | in Appendix A.2 We therefor consider only dif diff2 as the uncertainty meaurement for the other datasets.
CIFAR-100. We evaluate Mi xMat ch and variants of MMA that uses di ff2 as uncertainty mea- surement on CIFAR- 100 with 4000, 5000, 8000 and 10000 labeling budget. We first train the model for 262144 44 steps with 2500 randomly selected labeled samples. Then we grow the labeled set by 500 examples at each active learning iteration and train the model for 32768 additional steps. Note that since CIFAR- 100 has 10x more classes than the other 2 datasets, we increase proportionally both the # initial examples and the # examples per query so that the number of samples per class remains unchanged. We also increase the label budgets to a range of 4000 to 10000. Following Berthelot et al. (2019), we increase the size of the ResNet-28 model to 128 filters per layer for more reasonable comparison.
Table 2 shows the results for CIFAR-100. Again, found that active learning helps. The perfor- mance gain compared to the passive Mi xMat ch ranges from 0.1 % (4000 labels) to 1.16% (10000 labels). We found that methods with diversification (kmeans infoD) consistently outperfomm their variants without diversification reirect. We conjecture that this is because given a large number of classes, diversifying the queried examples helps keep the sampled data class distribution more balanced. Besides, as in CIFAR- 10, this is no significant difference between different active learning strategies, so we do not dismiss their effectiveness well.
Table 2: Performance of MMA on 5 repeated runs CIFAR-100.
Label Budgets
4000
5000
8000
10000
Mi MixMat
67.72 = 0.79
69.41
0.19
72.87
0.32
74.00 + 0.10
random
67.56 = 0.50
69.12
0.33
72.27
0.28
73.62 + 0.24
dif aigfi.aug-direct
67.72 = 0.50
69.45 = 0.50
72.80 + 0.28
74.66 = 0.19
di diff2.aug-kmeans
67.82 + 0.48
69.60 = 0.30
73.16 + 0.29
75.10 + 0.12
SVHN. We evaluate Mi xMatch and variants of MMA with di ff2 as uncertainty measurement on SVHN with 500, 1000, 2000 and 4000 labeling budget. We first train the model for 131072 steps with 250 randomly selected labeled samples. Then grow the labeled set by 50 examples each time and continue training the model for 16384 steps. An adjustment made to MMA here is that we select the initial set of samples such that the class distribution is the same that of the whole training set. This is because unlike CIFAR- 10 and CIFAR- SVHN has unbalanced classes, where the largest class has 2.8 times more samples than the smallest. We note that MixMatch (Berthelot al. 2019) selects the labeled samples in the same way.
Table shows the results for SVHN. MMA outperforms Mi xMa th in all labeling budgets. MMA with 4000 examples is only 0.02% less accurate (97.39% vs. 97.41 1% according to Berthelot et al.) (2019) than the same model trained on all the 73257 examples (18.31x x more) in the training set. We also found that diversification is useful, with kmeans method performing the best. We believe that diversification helps prevent same--ng less popular classes, which can be beneficial when the classes are unbalanced.
'The discrepancy between the value for MixMat t ch and the values in Berthelot et al. (2019) for 10000 labels (which is 74.12 + 0.30) can be explained by the fact that Mi xMatch selects samples uniformly random while Berthelot al. 2019) samples the same number from each class.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Performance of MMA on 5 repeated runs on SVHN (no extra data).
Label Budgets
500
1000
2000
4000
Mi xMatch
96.36 0.46
96.73 + 0.31
96.96
0.13 97.11 + 0.06
random
96.44 = 0.18
96.73 = 0.11
0.00
0.11
diff2.aug-dirret
96.59 = 0.09
96.82 = 0.08
77.03
diff2.aug-kmeans
96.69 = 0.07
96.90 = 0.08
97.14
97.39 0.08
5.3 COMPARATIVE ADVANTAGE OF ADDING LABELED Vs. UNLABELED DATA
In this section, we compare the accuracy improvements offered by adding labeled data and unlabeled data. As proposed Section 4.3 we measured the accuracy obtained by our model for various amounts of labeled and unlabeled data (see tables in Appendix A.2). To perform the analysis, we need to estimate the number of unlabeled samples u, required to achieve a laryel accuracy for fixed amount of labeled data. For practical computational constraints, we approximate this process by using a linear interpolation between consecutive accuracy measurements when growing the amount of unlabeled data for fixed amount of labeled data (see Appendix A.33 for details).
We plot the cost ratio Cratio(s) as a function of the number of labeled examples for CIFAR- and SVHN+Extra in Figure I For example, is cost efficient to label an image when the cost of labeling an image is less than 20 times the cost of obtaining new unlabeled image. We make the following observations:
that for CIFAR-10 10 with 500 labeled samples, it
The more labeled samples are present, the more the value of labeling additional samples drops. Intriguingly, for SVHN, the ratio drops below 0 in some cases. This means that it is always better to add unlabeled data regardless of the labeling cost. This usually occurs when the assumption |L| < (U] is violated, while MixMatch is tuned to operate under this assumption. Re-tuning of the parameters may necessary to counter this effect.
Generally, the highest cost ratio we observed was 20 and frequently less than 3x which makes labeling data a costly alternative with sample efficient techniques such as MixMatch.
Likely, there exists critical mass below which labeled data is always needed, but we did not observe it in our experiments. We conjecture that this number is below 50 samples per class.
20
targat acc=88%
20
targel acc=04%
gel aco-89%
fargel 90c-95%
15
target aco-90%
16
targel 90c-
forr gel acc-81%
fargel aoc-97%
10
1000
vs.
500 vs.
1300 2000 Number of Labeled Datepoints
2000 4000
Number of Labeled Datapoints
(a) CI
(b) SVHN+Extra
Figure 1: Plot of Cratio measure of relative cost of labeling vs. obtaining unlabeled data.
CONCLUSION
In this work we have demonstrated that the state-of-the-art SSL MixMatch algorithm can be signif- icantly improved when combined with various active learning methods, pushing the tfte-of-thhear even further. We found that generally, uncertainty sampling (via the diff2 definition) performs ro- bustly across several baseline datasets. We furthermore, provided an analysis to measure the relative value of labeled an unlabeled data and, interestingly, see that in certain regimes labeled data provide only small constant factor (i.e. < 3x) additional value of unlabeled data. Future directions for this line work include evaluating a trade-off aware algorithm, which dynamically adjusts the number of labeled and unlabeled examples that are gathered given the costs of each.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Naoki Abe and Hiroshi Mamitsuka. Query learning strategies using boosting and bagging. In Machine learning: proceedings of the fifteenth international conference (ICML98), volume 1998.
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Paperno, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. CoRR, abs/1905 1905.0224,, 2019. LtL.ttpp://xxx..otrr/aaaas/ org. 0ateaa/1905..220
Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th International Conference on Machine Learning, pp. 49-56. ACM, 2009.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15(2):001-221, 1994.
Sanjoy Dasgupta, Daniel J Hsu, and Claire Monteleoni. A general agmostic active learning algo- rithm. In Advances neural information processing systems, pp. 353-360, 2008
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances in Neural Information Processing Systems, 2005.
Steven CH Hoi, Rong Jin, Jianke Zhu, and Michael Lyu. Semisupervised svm batch mode active learning with applications to image retrieval. ACM Transactions on Information Systems (TOIS), 27(3):16, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for supi-supe learning. Fifth Interna- tional Conference on Learning Representations, 2017.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep
neural networks. In ICML Workshop on Challenges in Representation Learning, 2013.
David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pp. 148-156. Elsevier, 1994.
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR94, pp. 3-12. Springer, 1994.
Stuart Lloyd. Least squares quantization pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Andrew Kachites McCallum and Kamal Nigam. Employing em and pool-based active learning for text classification. In Proc. International Conference on Machine Learning (ICML), pp. 359-367, 1998.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions pattern analysis and machine intelligence, 2018.
Ion Muslca, Steven Minton, and Craig A Knoblock. Active + semi-supervised learning robust multi-view learning. In ICML, volume 2, pp. 435 442, 2002.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Avital Oliver, Augustus Odena, Colin A Raffel. Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estima- tion of emor reduction. ICML, Williamstown, pp. 441-448. 2001.

---

Under review as a conference paper at ICLR 2020
Burr Settles. Active learning literature survey. Technical report, University of Besconsin-MMadise Department of Computer Sciences, 2009.
Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning. In Advances in neural information processing systems, pp. 1289-1296, 2008.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 287-294. ACM, 1992.
Antti Tarvainen and Harri Valpola. Mcan teachers better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 2017.
Katrin Tomanek and Udo Hahn. Semi-supervised active learning for sequence labeling. In Proceed- ings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2- 2-Volume 2, pp. 1039-1047. Association for Computational Linguisties, 2009.
Gokhan Tur, Dilek Hakkani-Tüir, and Robert Schapire. Combining active and semi-supervised learning for spoken language understanding. Speech Communication, 45(2):171-186, 2005.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint veXiv:1110.09411 2017.
Xiaojin Zhu, John Lafferty, and Zoubin Ghahramani. Combining active learning and semi- supervised learning using gaussian fields and harmonic functions. In ICML 2003 workshop on the continuum from labeled to unlabeled data in machine learning and data mining, volume 3, 2003.
A APPENDIX
A.1 HYPER-PARE PARAMETERS
Table shows the parameeassss used throughout the paper.
Table 4: A summary of byper-paramest Au is the hyper-parameter controlling the importance of the unlabeled data to the training process.
CIFAR-10 CIFAR-100 SVHN
SVHN+Extra
filters Àu
128
75
150
250
250
MixUp
0.75
0.75
0.75
0.25
weight decay
0.02
0.04
0.02
0.0001
A.2
FULL RESULTS
A.2.1 EFFECTIVENESS OF MMA
T 6 are the full versions of Table | 2 respectively, presenting the results of different MMA
variants for CIFAR-10, CIFAR-100 and SVHN.
Figure 2 show the plots of accuracy achieved versus the number of labeled samples for CIFAR-10, CIFAR- 100 and SVHN.
10

---

Under review as a conference paper at ICLR 2020
Table 5: This is the full version of Tabel Performance of MMA on CIFAR-IO. Each entry is the average of 5 repeated runs with standard deviation. The highlighted entries are the two best performing methods within each column. The best methods for 500 is max-kmeans and for the rest is di wifff.auu-direct.
Label Budgets
500
1000
2000
4000
16000
Supervised
60.72 + 1.04
71.64 + 0.80
80.26
0.76
86.37 0.19
MeanTeacherr
61.95 + 6.80
81.57 2.49
87.28 + 0.84
89.37 + 0.14
VAI
73.69 + 2.34
$1.12
85.88 +148+000 0.47
88.56 0.24
PiModel
56.67 + 1.63
68.77 0.82
77.29 0.39
84.23 = 0.80
PseudeLabel
57.56 +
1.66 68.96
77.94 0.55
0.28 83.84 1: 0.28
MixMatch
90.58 1 0.83
91.61 1 0.54
93.20 1 0.11
93.70 - 0.16
94.99 1 0.15
random
90.54 + 0.82
91.85 0.67
92.97 0.29
93.73 0.18
diff2.aug-direct
91.69 1 0.52
92.7910.44
94.11 + 0.14
95.17 1 0.13
diff2-direct
91.32 +: 1.24
92.89 0.24
93.97 + 0.15
95.04 1 0.23
diff2.aug-kmeanns
91.46 + 0.38
9.32
95.06 = 0.19
diff2-kmeans
0.71
9.66
93.99 0.09
95.09 = 0.11
diff2.aug-infoD
91.52 + 0.67
9..7
93.97
95.12 0.00
difff-infoD
90.18 + 1.90
0.276+0.25
94.07 0.18
95.10 0.11
wex.aug-direct.
90.87 1.65
9.41
93.72+
94.98 1.98+000 0.10
max-direct
91.02 0.55
0.51
9
max.aug-kreans
91.05 + 0.42
92.06
93.61 0.21
94.90 0.21
max-kmeans
91.78 + 0.42
9.445
0.11
95.00 - 0.09
max.aug-infoD
90.94 + 0.59
12.00
93.55 + 0.36
95.06 0.09
max-infoD
90.60 + 0.82
1.25+0..4
93.75 + 0.22
95.07 0.07
Table 6: This is the full version of Tabel Performance of MMA on CIFAR-100. Each entry is the average of 5 repeated runs with standard deviation. The highlighted entries the two best perform- ing methods within each column. Methods with diversification (kmeans or infoD) consistently outperform their variants without diversification (direct).
Label Budgets
4000
5000
8000
10000
Mi MixMatch
67.72 0.79
69.41 = 0.19
72.87 + 0.32
74.00 + 0.10
random
67.56 + 0.50
69.12 = 0.33
72.27 + 0.28
73.62 + 0.24
diff2-direct
67.37 0.26
68.95 = 0.35
72.61 + 0.18
74.54 0.18
diff2.aug-direct
67.72 = 0.50
69.45 = 0.50
72.80 = 0.28
74.66 0.19
difff-kmeans
67.83 0.32
69.99 + 0.18
73.47 0.15
75.16
di.ff2aug-kkeans
67.82 + 0.48
69.60 = 0.30
73.16 0.29
75.10
diff2-infoD
67.76 0.57
69.84 + 0.16
73.32
0.23
75.03 0.22
diff2.agginfoo
67.78 0.61
69.56 = 0.57
72.95 + 0.33
74.94 + 0.30
A.2.2
COMPARATIVE ADVANTAGE OF ADDING LABELED VS.UNLABELED DATA
A.3 LINEAR INTERPOLATION COST RATIO MEASUREMENT
To efficiently find the number of unlabeled examples needed in order to reach target accuracy at a fixed number of labeled training examples, consider the measurements in a table such as Table For each column, we bisect the row values to find row such that accuracy,. accurocytarget
"The discrepancy between the value for MixMat ch and the values in Berthelot et al. (219) for 10000 labels (which is 74.12 0.30) due the fact that MixMatch selects samples uniformly random while Berthelot 2019) samples the number from each class. The difference hecomes significant when the number classes grows. 2The 5 runs reaches accuracy 97.26, 97. 97.39, 97.26, 93.42, 97.40 and the variance is due to the oscillations the training process.

---

Under review as a conference paper at ICLR 2020
Table 7: CIFAR-100. Starting from 2500 random samples trained for 262144 steps, querying 100 and training for 16384 steps each time. Each entry is the average of 5 repeated runs with standard deviation. The highlighted entries are the two best performing methods within each column. The accuracy is not significant different from that when query 500 each time (T
Label Budgets
4000
5000
8000
10000
Mi xMatch
67.72 + 0.79
69.41 = 0.19
72.87 + 0.32
74.00 + 0.10
random
66.84 0.43
68.72 = 0.34
72.18 + 0.31
73.70 = 0.41
diff2-direct
67.47 + 0.29
69.25 = 0.27
72.79 + 0.26
74.68 + 0.33
di difff2.ag-diiress
67.46 0.43
69.35 0.37
72.91 0.25
74.50 + 0.35
diff2-kmeans
67.89 = 0.27
69.83 0.45
73.10 0.33
74.80 = 0.13
di difff2.aug-kmeans
67.64 0.61
67.78 3.79
73.07
0.20
74.82 0.26
diff2-infoD
68.06 = 0.58
70.18 = 0.52
73.90 = 0.21
75.26
= 0.23
di ooo
68.00 0.76
70.04 + 0.41 73.70 0.30
75.45 0.27
Table 8: This is the full version of Tabel Performance MMA on SVHN (no extra data). Each entry is the average of repeated runs with standard deviation. The highlighted entries the two best performing methods within each column. Methods with kmeans diversification generally outperforms the other methods.
Label Budgets
500
1000
2000
4000
MixMatch
96.36 0.46
96.73
0.31
96.96 + 0.13
97.11 + 0.06
random di diffe.aug-direct
96.44 0.18
96.73
0.11
96.83 + 0.07
97.01 0.11
96.59 + 0.09
96.82 = 0.08
97.03 + 0.05
97.32 : 0.03
diff2-direct
96.56 0.15
96.85
0.03
97.01
97.33 1
di diff2-kmeans
96.59 = 0.06
96.91 0.07
9..0
97.36
difff.aug-kmeans
96.69 = 0.07
96.90
9.014
07.39
diff2-infoD
96.51 0.13
96.76 0.12
97.02 + 0.03
97.29 0.05
di dif
96.51 0.15
96.73 0.10
96.28 + 1.60
97.37 0.04
accuracyi,rt1 We then compute the interpolation factor
accuracy target - accuracy
accuracys, i,r accuracy
Finally the number of unlabeled samples can be linearly approximated U, X accuracy,, (1 A,) accurucy
MixMatch
random
diff2.aug-direes
aug-kann
MMA (best)
97.5%
Test Accuracy
Test Accuracy
Test Accuracy
97.0%
96.5%
2000
10000
98.0%
000
1000
2000
4000
Number Labeled Datapoints
Number of Labeled Datapoints
Number Labeled Datapoints
(a) CIFAR-10
(b) CIFAR-100
SVHN
Figure 2: Test accuracy (y-axis) under different number of labeled samples (x-axis, log scale). MMA (best) refers to the highest accuracy that any MMA variants achieves (with full results listed in tables found in Appendix A.2). Note, the AL methods essentially overlapping.
12

---

Under review as a conference paper at ICLR 2020
Table 9: CIFAR-10. Accuracy of MMA with di ff2.uuggd rectusing different amounts of labeled and unlabeled data. The first column indicates the total number of datapoints (labeled + unlabeled).
Labeled
Total
500
1000
2000
4000
5000
64.45 0.98
66.69 + 1.15
71.71 + 1.27
80.33 + 0.71
10000
72.79 + 2.02
74.81
1.57
78.78 + 0.79
84.65 0.36
15000
77.65 2.13
19.57=1.0
83.26 + 1.09
87.87 = 0.62
20000
80.80 1.48
83.16
87.20 1.31
90.98 0.52
25000
85.29 + 1.24
87.96
90.40
93.12 - 0.22
30000
87.61 = 0.59
#.799
92.31
0.32
93.95 0.19
35000
89.32 0.60
93.10 9.67
94.55 0.12
40000
89.97 + 0.99
0.13 0.70
94.59 0.15
45000
91.14 0.46
9.360
95.08
0.04
50000
91.69 = 0.52
94.11 0.41
95.17
Table 10: CIFAR-100. MMA with
Values at (5000, 5000) and (10000, 10000)
computed with fully supervised learning the same model.
Labeled
Total
4000
5000
8000
10000
5000
48.40
0.50
55.42 + 0.30
10000
53.12
0.55
55.39
+
60.21 + 0.27
65.93 + 0.34
20000
59.78 = 0.21
61.67 0.227
65.11 + 0.11
67.01 0.21
50000
67.82 + 0.48
09.60
73.16
75.10 - 0.12
Table 11: SVHN+Extra. MMA with di difff.aug-kmeans..
Labeled
Total
500
1000
2000
4000
5000
91.04 + 0.27
90.93 + 0.30
90.50 + 0.42
89.66 + 0.25
10000
03.71
93170
93.54
93.35 0.14
20000
95.28 0.23
95.32
96.27
95.37
50000
96.48
06.61+0..0
96.75
96.83 0.07
100000
97.07
9..17
9.07
0..56
200000
96.55 1.55+:155
0.08
97.66
9..07
604388
97.38 0.19
97.53
97.80
97.98
13


Reviews:
[{"review_id": "HyeIXNKTYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to combine active learning techniques with MixMatch for semi-supervised learning. First, they review active learning and semi-supervised learning, especially MixMatch. Instead of traditional semi-supervised learning with a fixed set of labeled examples, they incrementally grow the labeled set as the training process goes on. They consider several different choices in active learning strategies: uncertainty measure and diversification. Diversification methods are used to balance the samples in different classes and ensure diversity. The cost analysis of adding labeled vs unlabeled data looks interesting. They perform an empirical evaluation on image benchmarks and improve over MixMatch.\n\nOverall, the paper is clearly written and easy to follow. However,  I cannot recommend acceptance because\n\n1. Novelty concern. The combination of two existing techniques seems not novel enough.\n\n2. Missing important baselines in related work and experiments. In the related work on semi-supervised learning (Section 3), the authors only review MixMatch but neglect other literature, e.g.[1,2,3,4]. And semi-supervised learning has a long history and it is not restricted to recent deep learning-based approaches. A thorough review can make the approach well-placed in the literature. In experiments, the authors only compare with MixMatch. I suggest that the authors include the missing literature in the next version.\n\n3. The cost analysis is the most interesting to me. However, Figure 2(b) in Section 5.3 is weird. How can the ratio less than 0? According to the definition in Section 4.3, $L_i \\subset L_{i+1}$ and $|L_{i+1}| > |L_i|$ and similar case for $U_{i,j}$, the cost ratio should not be less than 0. I'm also confused by the explanation in Section 5.3.\n\n4. From Figure 1(b) and Table 2, we can see that on CIFAR-100, the improvement of the proposed MMA is not statistically significant, especially when the label budget is low. But on a simpler dataset CIFAR-10, MMA performs better. How does MMA perform on a more challenging task with more classes, e.g. ImageNet?\n\n5. Training time comparison. At the expense of spending more time on selecting uncertain examples and techniques like k-means clustering, MMA is slightly better than MixMatch. A comparison of training time and complexity would be better to convince me.\n\n\n\n***\nMinor:\npage 4 “Starting with from a fixed pool of n unlabeled sample”\npage 4 “A corollary question is how do various accuracy targets relate to each other?” \npage 5 “While there are there additional active learning”\npage 5 “ Let’s define cl and cu as the cost of respectively obtaining a new labeled sample and a new unlabeled sample.” --> costs\n\n\n\n***\nReferences\n[1] Temporal Ensembling for Semi-Supervised Learning, ICLR 2017.\n[2] Smooth Neighbors on Teacher Graphs for Semi-supervised Learning, CVPR 2018.\n[3] Realistic Evaluation of Semi-supervised Learning Algorithms, NeurIPS 2018.\n[4] There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average, ICLR 2019.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkeRfPx2Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize the paper:\n\nThis paper proposes a method that can deal with an active-learning scenario for the recently proposed semi-supervised learning method: MixMatch.  More specifically, the proposed method considers uncertainty measures to choose samples and a diversification step to ensure diversity within the sampled batch.  For uncertainty measures, the paper considers the simple maximum confidence and the gap between two most likely classes.  Additional augmentation techniques inspired from MixMatch are used.  For diversification, a clustering method and an information density method are considered.  Furthermore, the paper proposes a cost analysis model to compare labeled and unlabeled samples.  Experiments demonstrate the behavior of the proposed method.\n\n\nPros of the paper:\n\n- The experimental results seem to be strong and encouraging.\n- The discussions on the cost of labeled and unlabeled samples seems to be an important contribution for semi-supervised active learning.\n- The motivation and direction of the paper is simple and easy to follow: Take the state-of-the-art semi-supervised learning algorithm and propose an active-learning version of it.\n- It is not a straightforward combination of MixMatch and active learning, and there are some specialized techniques such as “aug” used in the design of the proposed algorithm.\n\n\nCons of the paper:\n\n- Only uncertainty based sampling methods are considered, but is this enough?  There seems to be no other papers that deal with active semi-supervised learning for a deep learning context, so it might be important to really explore the many sampling methods (e.g., from survey of Settles 2009).\n\n- A more minor comment: The same issue goes for the semi-supervised learning side.  MixMatch is the state of the art in terms of accuracy for image domains, but it is an ensemble of several semi-supervised learning methods, and have strong assumptions, e.g., smoothness assumption, small distribution overlap, etc.  This will mean the proposed method will also have those strong assumptions and limits the method’s applicability.\n\n- In experiments, it would be better to have figures that are usually used in active learning experiments, where the x-axis is the remaining budget and y-axis is the performance measure.\n\n\nAdditional comments:\n\nActive learning methods gives labels to unlabeled samples in different epochs until the budget is used up, but it would be interesting to give the final labeled and unlabeled dataset after budget is used up as a fixed dataset, and then train the traditional passive MixMatch with this.  Then we can really compare the original MixMatch and active MixMatch.  If the proposed method still works better,  then the proposed method might be meaningful not only as an active learning method but also as a curriculum learning method.\n\n********************\nI would like to thank the authors for answering my questions and updating the paper, but would like to keep my score due to the 2nd point of the cons.  A minor comment on the second point of the author response:  The sharpening step in MixMatch can be regarded as an entropy minimization procedure, which I think is based on the assumption of low distribution overlap.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJedOIhjKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes a look at using active learning techniques instead of random sampling for the \n\"state-of-the-art\" semi-supervised learning (SSL) method MixMatch. At least for this one SSL algorithm, the authors give a strong argument that active learning helps MixMatch (4x label efficiency in some cases) and highlight the active learning algorithms that work best. An additionally interesting point is the value of labeled vs unlabeled data in this setting. For these reasons, I argue for acceptance of this paper. However, I have some reservations which are given below, that perhaps the authors can clarify.  \n\n\nThings that would have improved my score:\n\n - This paper relies on MixMatch very heavily as the sole semi-supervised technique. It would be nice to see more of an argument for this choice of a relatively recent paper that hasn't stood the test of time.\n\n - I am confused why the authors \"report the median of the last 20 checkpoints' accuracy where a checkpoint is computed every 65,536 training iterations\". As we see later, the authors train after each batch of selected examples for 32,768 iterations which is half of the time between checkpoints. Can the authors comment on this choice?\n\n\nMinor comments:\n\n - The end of the abstract makes it sound like like the conclusions are universal (\"quickly diminishes to less than 3x once more than 2000 labeled example are observed\"). I would be surprised if the authors meant this as a universal statement since no theory is provided and the experiments are on similar datasets.\n\n - I don't understand why section 3 is not simply titled \"MixMatch\" since the paper doesn't really touch any other \"modern SSL\" methods.\n\n - In the experiments section, 262144 and 32768 iterations seem to come from nowhere. Only later did I realize that these were powers of 2. Can this be clarified?\n\n - What's the difference between MixMatch and MMA with random selection? Shouldn't these perform the same (which they seem to anyways)?\n\n - I really like that this paper assesses the label efficiency of their algorithm, rather than merely reporting raw accuracy numbers which aren't as meaningful.\n\n - I wonder if MMA seems to not give as big gains on CIFAR-100 because the batch size is 10x larger. Generally, I've found that active learning (especially uncertainty sampling methods) work best with smaller batch sizes and I'm not sure I agree with the reasoning that more classes mean one should select larger batch sizes.\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends state of the art semi-supervised learning techniques (i.e., MixMatch) to collect new data adaptively and studies the benefit of getting new labels versus adding more unlabeled data. Active learning is incorporated in a natural and simple (albeit, unsurprising) way and the experiments are convincing that this approach has merit.

While the approach works, reviewers were concerned about the novelty of the combination given that its somewhat obvious and straightforward to accomplish. Reviewers were also concerned that the space of both semi-supervised learning algorithms and active learning algorithms was not sufficiently exhaustively studied. As one reviewer points out: neither of these ideas are new or particular to deep learning.

Due to lack of novelty, this paper is not suited for a top tier conference. 
Decision: Reject