Combining MixMatch and Active Learning for Better Accuracy with Fewer Labels

We propose using active learning based techniques to further improve the stateof-the-art semi-supervised learning MixMatch algorithm . We provide a thorough empirical evaluation of several active-learning and baseline methods , which successfully demonstrate a significant improvement on the benchmark CIFAR-10 , CIFAR-100 , and SVHN datasets ( as much as 1.5 % in absolute accuracy ) . We also provide an empirical analysis of the cost trade-off between incrementally gathering more labeled versus unlabeled data . This analysis can be used to measure the relative value of labeled/unlabeled data at different points of the learning curve , where we find that although the incremental value of labeled data can be as much as 20x that of unlabeled , it quickly diminishes to less than 3x once more than 2,000 labeled example are observed . 1 INTRODUCTION . Sophisticated machine learning models have demonstrated state-of-the-art performance across many different domains , such as vision , audio , and text . However , to train these models one often needs access to very large amounts of labeled data , which can be costly to produce . Consider , for example , laborious tasks such as image annotation , audio transcription , or natural language part-of-speech tagging . Several lines of work in machine learning take this cost into account and attempt to reduce the dependence on large quantities of labeled data . In semi-supervised learning ( SSL ) , both labeled and unlabeled data ( which is often much cheaper to obtain ) are leveraged to train a model . Unlabeled data can be used to learn properties of the distribution of features , which then allow for more sophisticated and effective regularization schemes . For example , enforcing that examples close in feature space are labeled similarly . Another , different , approach for address costly labeled data is that of active learning ( AL ) . Here , a model is still trained using only labeled data , but extra care is taken when deciding which unlabeled data examples are to be labeled . Often , the data will be labeled iteratively in batches , where at each iteration an update is made to a current view of the distribution over labels and the next batch of points is selected from regions where the distribution is least certain . As discussed in depth in the following section , the approaches of semi-supervised and active learning can be complementary and used in conjunction to help solve the problem of costly in labels . Our Contributions : In this paper , we take MixMatch , the leading semi-supervised learning technique , and thoroughly evaluate its performance when combined with several active learning methods . We find very encouraging results , which show that state-of-the-art performance is achieved in the limited label setting on CIFAR-10 , CIFAR-100 , and SVHN datasets , demonstrating that combining active learning techniques with MixMatch provides a significant improvement . Furthermore , we perform an analysis , exploring the incremental benefits of labeled versus unlabeled data at different points in the learning curve . Given the relative costs of labeled and unlabeled data , such an analysis aids us in deciding how to best spend a given budget in acquiring labeled versus unlabeled data . The remainder of the paper is organized as follows : We first give a high-level review of active learning related work and the MixMatch algorithm in Sections 2 and 3 , respectively . The evaluated methods and experimental setup are described in Section 4 , while the experimental results are presented and analyzed in Section 5 . Finally , we conclude in Section 6 . 2 ACTIVE LEARNING AND RELATED WORK . Active learning ( or active sampling ) methods are designed to answer the question : Given a limited labeling budget , which examples should I expend my budget on ? Like semi-supervised learning , active learning is particularly is useful when labeled data is costly or scarce for any host of reasons . Generally , an active learning algorithm iteratively selects samples to label , based on the data labeled thus far as well as the model family being trained . Once the labels are received , they are added to the labeled training set and the process continues to the next iteration , until the budget is finally exhausted . Thus , active learning can be considered to be part of the model training process , where classical model training is interleaved with data labeling . Here we give a very brief introduction to a few classes of active learning algorithms , and their use with semi-supervised learning . There are several active learning algorithms with strong theoretical guarantees , which either explicitly or implicitly define a version space that maintains the set of “ good ” candidate classifiers based on the data labeled thus far . The algorithms then suggest labeling points that would most quickly shrink this version space ( Cohn et al. , 1994 ; Dasgupta et al. , 2008 ; Beygelzimer et al. , 2009 ) . In practice , tracking this version space can be computationally inefficient for all but the simplest ( e.g . linear ) model families . Another , related approach , is that of query-by-committee , where points are selected for labeling based on the level of disagreement between a committee of classifiers , which have been selected using the currently labeled pool . Query-by-committee methods benefit from theoretical guarantees ( Abe & Mamitsuka , 1998 ) , as well as practical implementations based on bagging ( Seung et al. , 1992 ) . These methods , along with uncertainty sampling that we introduce next , are only a few prototypical examples of active learning algorithms . For a broader introductory survey of algorithms and techniques , please see Settles ( 2009 ) and the references therein . Arguably , the most popular and practically effective active learning technique is that of uncertainty or margin sampling ( Lewis & Gale , 1994 ; Lewis & Catlett , 1994 ) . At each iteration , this approach first trains a model using the currently labeled subset , then it makes predictions on all unlabeled points under consideration and , finally , it queries labels for those points where the confidence in the models ’ prediction is the smallest . The notion of confidence can be defined several different ways , as will be detailed in Section 4.2 . In this light , we can see that active learning methods and semisupervised learning methods can work in a complementary fashion . At a high level , active learning methods seek to find training examples for which we have least confidence in the underlying labels in order to query for those labels , while semi-supervised learning algorithms can focus on training examples where there is strong confidence in the label distribution and reasonable assumption conclusions can be made regarding unlabeled examples . It is no surprise that combining active learning and semi-supervised learning has been investigated previously ( Hoi et al. , 2009 ; Muslea et al. , 2002 ) . We highlight a few of these examples below ( also see Section 7.1 of Settles ( 2009 ) ) . One of the earliest lines of work combining the two techniques is that of McCallum & Nigam ( 1998 ) , where the query-by-committee framework is combined with an expectation maximization ( EM ) approach that is used to provide pseudo-labels to those examples that have not been queried for true underlying labels by the active learning method . They show that , on a text classification task , the improvement in label complexity of the combined method is better than what either of semi-supervised or active learning method alone can provide . In the work of Zhu et al . ( 2003 ) , the two approaches are combined using a Gaussian random field model . Given a model trained on the union of currently labeled and unlabeled datasets , the expected reduction in the estimated risk due to receiving the label of a particular example can be computed and greedily optimized . Tur et al . ( 2005 ) and Tomanek & Hahn ( 2009 ) both combine uncertainty sampling to label uncertain points with machine labeling of confident examples and show their effectiveness in applications to a spoken language understanding and sequence labeling tasks , respectively . In this work , we combine uncertainty-sampling based active learning method and demonstrate that the already state-of-the-art semi-supervised performance of the MixMatch algorithm can be significantly further improved . 3 MIXMATCH . We only focus on deep learning based semi-supervised learning ( SSL ) in this work . We use the framework from Oliver et al . ( 2018 ) to perform a realistic evaluation of the results ( same splits of the labeled/unlabeled initial data , same network architecture ) . Recent popular techniques are built around two concepts : Consistency Regularization ( Mean Teacher ( Tarvainen & Valpola , 2017 ) , Π-model ( Laine & Aila , 2017 ) , Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2018 ) ) which enforces that two augmentations of the same image must have the same label and Entropy Minimization ( Pseudo-Label Lee ( 2013 ) , Entropy Minimization ( EntMin ) ( Grandvalet & Bengio , 2005 ) ) which states that a prediction should be confident . These two concepts are sometimes combined , for example the technique VAT EntMin is a combination of VAT and Entropy Minimization and MixMatch is also using both of these concepts . MixMatch , a recent state-of-the-art semi-supervised learning ( SSL ) technique , is designed around the idea of guessing labels for the unlabeled data followed by using standard fully supervised training ( Berthelot et al. , 2019 ) . Consider a classification task with classes C. The input of MixMatch are a batch of B ( images , labels ) pairs X = { ( xb , pb ) } 1≤b≤B , where each label pb is an one-hot vector over the class C , and a batch of unlabeled examples ( images ) U = { ub } 1≤b≤B . Note , in this section , a “ batch ” is in reference to the subset of points used in the iterative optimization procedure that is used to train the model , while elsewhere in the paper “ batch ” refers to a set of newly labeled points that are added to the training set during the iterative active learning process . Label Guessing . Label guessing is done by averaging the training model ’ s own prediction on several augmentation ûb , i of a same image ub . This average prediction is then sharpened to produce a low-entropy soft label qb for each image ub . Formally , the average is defined as q̄b = 1 K ∑K i=1 pmodel ( y|ûb , k ; θ ) where pmodel ( y|x ; θ ) is the model ’ s output distribution over class labels y on input x with parameters θ . A sharpening is applied to the average prediction : qb = Sharpen ( q̄b ) . In practice , MixMatch uses a standard softmax temperature reduction computed as Sharpen ( p ) i : = p 1/T i / ∑|C| j=1 p 1/T j where T = 1/2 is a fixed hyper-parameter . Data Augmentation . MixMatch only uses standard ( weak ) augmentations . For the SVHN dataset , we use only random pixel shifts , while for CIFAR-10 and CIFAR-100 we also use random mirroring . Fully supervised . Finally , MixMatch uses fully supervised techniques . In practice , it use weight decay and MixUp across the labeled and unlabeled data ( Zhang et al. , 2017 ) . In essence , MixUp generates new training examples by computing the convex combination of two existing ones . Specifically , it does a pixel level interpolation between images and pairwise interpolation between probability distribution . The resulting interpolated label is a soft label . Such examples encourage the model to make smooth transitions between classes . Let X̂ = { ( x̂b , pb ) } 1≤b≤B and Û = { ( ub , qb ) } 1≤b≤B be the results of data augmentation and label guessing . MixMatch shuffles the union of the two batches X̂ ∪ Û into a batch W of size 2B and performs MixUp to produce the output : • X ′ = MixUp ( X̂ , W [ 1 , ... , B ] ) ( mixing up X̂ with the first half of W ) and • U ′ = MixUp ( Û , W [ B+1 , ... ,2B ] ) ( mixing up Û with the second half of W ) . Given two examples ( x1 , p1 ) and ( x2 , p2 ) where x1 , x2 are feature vectors and p1 , p2 are one-hot encoding or soft label , depending on whether the corresponding feature vector is labeled or not , MixMach performs MixUp as follows : ( 1 ) Sample λ ∼ Beta ( α , α ) from a Beta distribution parameterized by the hyper-parameter α ; ( 2 ) For λ′ = max ( 1− λ , λ ) , compute x′ = λ′x1 + ( 1− λ′ ) x2 and p′ = λ′p1 + ( 1− λ′ ) p2 . Loss function . Similar to other SSL paradigms , the loss function of MixMatch consists of a sum of two terms : ( 1 ) a cross-entropy loss between a predicted label distribution with the ground-truth label and ( 2 ) a Brier score ( L2 loss ) for the unlabeled data which is less sensitive to incorrectly predicted labels . On a MixMatch batch ( X ′ , U ′ ) , the loss function is L = LX + λULU where LX = 1 |X ′| ∑ ( x , p ) ∈X′ CrossEntropy ( p , pmodel ( y|x ; θ ) ) , ( 1 ) LU = 1 |C||U ′| ∑ ( u , q ) ∈U ′ ‖q − pmodel ( y|u ; θ ) ‖2 . ( 2 ) Here λU is the hyper-parameter controlling the importance of the unlabeled data to the training process . We set λU to be 75 for CIFAR-10 , 150 for CIFAR-100 , and 250 for SVHN . We also fix the MixUp hyper-parameter α = 0.75 .

Summary: This paper proposes a method that can deal with an active-learning scenario for the recently proposed semi-supervised learning method: MixMatch.  More specifically, the proposed method considers uncertainty measures to choose samples and a diversification step to ensure diversity within the sampled batch.  For uncertainty measures, the paper considers the simple maximum confidence and the gap between two most likely classes.  Additional augmentation techniques inspired from MixMatch are used.  For diversification, a clustering method and an information density method are considered.  Furthermore, the paper proposes a cost analysis model to compare labeled and unlabeled samples.  Experiments demonstrate the behavior of the proposed method.