On Pseudo-Labeling for Class-Mismatch Semi-Supervised Learning

1 INTRODUCTION labeled data cat dog unlabeled data cat dog plane car ID data OOD data Figure 1 : Realistic Semi-Supervised Learning may simultaneously contain unlabeled ID and OOD data . ID data come from the same classes as labeled data while OOD data come from classes that are not seen in labeled data . Deep Semi-Supervised Learning ( SSL ) methods are proposed to reduce dependency on massive labeled data by utilizing a number of cheap , accessible unlabeled data . Pseudo-Labeling ( Lee et al. , 2013 ) is a simple but effective and widely used method that creates pseudo-labels according to predictions of the training model itself . Then SSL can be transformed to standard supervised learning . Other representative SSL methods are consistency regularization ( Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2019 ) , holistic methods ( Berthelot et al. , 2019 ; Sohn et al. , 2020 ) and generative methods ( Kingma et al. , 2014 ) . The recent development of SSL shows that these methods have achieved competitive performance to supervised learning methods . However , all of these SSL methods achieve their good results based on an assumption that the unlabeled data are drawn from the same distribution as the labeled data . This assumption can be easily violated in real-world applications . One of the common cases is that some unlabeled data come from unseen classes . For example , as is illustrated in Figure 1 , in image classification , we can collect a lot of unlabeled images from the internet but usually they cover broader category concepts than labeled data . Oliver et al . ( 2018 ) have shown that on such class-mismatched conditions , performance of traditional SSL methods is damaged . To deal with this problem , several methods have been proposed . These methods include filtering out OOD data ( Yu et al. , 2020 ; Chen et al. , 2020 ) , down weighting OOD data ( Chen et al. , 2020 ) and re-use OOD data by neural style transfer ( Luo et al. , 2021 ) or self-supervised learning ( Huang et al. , 2021 ) . Although these methods achieve good results , why do OOD data damage performance and how will OOD data help remain unclear . Here , we focus on analyzing Pseudo-Labeling ( PL ) in class-mismatched SSL and give some answers to these two questions . In this paper , we empirically analyze PL in class-mismatched SSL setting . These experiments aim to answer the following questions : ( 1 ) How do OOD data influence PL ? ( 2 ) What are the better pseudo-labels for OOD data ? For question ( 1 ) , we investigate pseudo-labels created by PL . The main finding is that pseudo-labels on OOD data tend to be imbalanced while on ID data , it remains balanced . We further show that PL ’ s performance is damaged due to such imbalance on OOD data . For question ( 2 ) , several strategies for labeling OOD data are investigated . We conclude that it is beneficial when labeling OOD data as a class different from ID data , and the performance can be further improved when the pseudo-labels partition unlabeled OOD data into their semantic clusters . Based on the experimental analysis , we propose a two-branched model called Υ-Model , which processes unlabeled data according to their confidence score on ID classes . The first branch performs re-balanced pseudo-labeling on high-confidence data . It utilizes the property of imbalanced pseudolabels on OOD data , truncating the number of pseudo-labeled data for each class to their minimum . This procedure filters out many OOD data and also prevents the negative effect of imbalanced pseudo-labels . For the other branch , semantic exploration clustering is performed on low-confidence data . They are considered as OOD data and their semantics will be mined by clustering into different partitions on extra classes . The clustering result provides better pseudo-labels for these OOD data than vanilla PL . Experiments on different SSL benchmarks show that our model can achieve steady improvement in comparison to supervised baseline . We summarize our contributions as follows : • We analyze the Pseudo-Labeling model for ID and OOD data . The findings lead to two primary conclusions : ( 1 ) Imbalance of pseudo-labels on OOD data damages PL ’ s performance . ( 2 ) Best pseudo-labels for unlabeled OOD data are those different from ID classes and partitioning them into their semantic clusters . • We propose our two-branched Υ-Model . One branch re-balances pseudo-labels on ID classes and filter out OOD data . The other branch explores semantics of OOD data by clustering on extra classes . • Experiments on different SSL benchmarks empirically validate effectiveness of our model . 2 PRELIMINARY . 2.1 CLASS-MISMATCHED SSL . Similar to the SSL problem , the training dataset of the class-mismatched SSL problem contrains n ID labeled samples Dl = { ( xli , yli ) } ni=1 and m unlabeled samples Du = { xui } mi=1 , ( usually , m ≫ n , ) yli ∈ YID = { 1 , . . . , KID } , while different from SSL , the underlying ground truth yu of unlabeled data may be different from labeled data . i.e. , yuj ∈ YID ∪ YOOD , YOOD = { KID + 1 , . . . , KID + KOOD } . The goal of class-mismatched SSL is to correctly classify ID samples into YID using labeled set with ID samples and unlabeled set possibly with OOD samples . 2.2 PSEUDO-LABELING . Pseudo-Labeling ( PL ) leverages the idea that we can use the model itself to obtain artificial labels for unlabeled data ( Lee et al. , 2013 ) . PL first perform supervised learning on labeled data to get a pre-trained model f , which outputs the probability of belonging to each ID class . It then creates the pseudo-labels for each unlabeled sample : y′ = { argmaxy∈YID f ( y|x ) , c ( x ) > τ reject , otherwise , ( 1 ) c ( x ) = max y∈YID f ( y|x ) , ( 2 ) where c ( x ) is the confidence score for x . All the pseudo-labeled unlabel data will be treated as labeled data for the next supervised learning generation . PL iteratively performs supervised learning and pseudo-label creation until stop condition . 3 ANALYSIS OF PSEUDO-LABELING IN CLASS-MISMATCHED SSL . In class-mismatched SSL , vanilla PL can only create pseudo-labels on ID classes even for OOD data . We will analyze how these OOD data influence vanilla PL and what are the better pseudo-labels for them in this section . 3.1 SETUP . We use CIFAR-10 ( Krizhevsky et al. , 2009 ) as our experimental dataset . The data set contains 10 categories – 6 animal classes and 4 vehicle classes . Following Guo et al . ( 2020 ) , we perform a classification task on animal classes ( denoted as class 0-5 ) and select 400 images per class to construct the labeled data set , i.e. , 2,400 labeled examples . The other 4 vehicle classes are taken as OOD classes ( denoted as classes 6-9 ) . 20,000 images are randomly selected from all the 10 classes as the unlabeled data set . We vary the ratio of unlabeled images to modulate class distribution mismatch . For example , the extent is 50 % means half of the unlabeled data comes from animal classes and the others come from vehicle classes . We use Wide-ResNet-28-2 ( Zagoruyko & Komodakis , 2016 ) as our backbone . We also adopt data augmentation techniques including random resized crop , random color distortion and random horizontal flip . We train our network for 400 epochs . For each epoch , we iterate over the unlabeled set and random sample labeled data , each unlabeled and labeled minibatch contains 128 samples . We adopt Adam as the optimization algorithm with the initial learning rate 3× 10−3 . We report the averaged accuracy of the last 20 epochs , pretending there is no reliable ( too small ) validation set to perform early stop ( Oliver et al. , 2018 ) . 3.2 IMBALANCE OF PSEUDO-LABELS ON OOD DATA . In this section , we analyze the pre-trained model that creates the first set of pseudo-labels , and the final model trained by Pseudo-Labeling . Pretrained model . First , we draw the distribution of confidence score on OOD data and ID data . Figure 2 ( a ) tells that , like what is concluded in OOD detection ( Hendrycks & Gimpel , 2017 ) , proportion of high-confidence data in ID data is larger than OOD data . However , in class-mismatched SSL , the unlabeled data are in much larger quantities . When the class mismatch ratio is large , there are quite a few OOD data with high confidence scores . We will show in the final model experiments that these high-confidence OOD data damages performance . Secondly , we study pseudo-labels on both ID data and OOD data . Figure 2 ( b ) shows that pseudo-labels ID data is balanced . However , they are rather imbalanced on OOD data ( Figure 2 ( c ) ) . This is attributed to the different distribution they are drawn from . Samples with certain pattern bias to certain classes . ID data bias to ID classes uniformly because they are sampled by the same distribution . However , with little probability , OOD data will also bias to ID classes uniformly since they have little relevance to ID data . Final Pseudo-Labeling model . As an old saying goes , a good beginning is half done . However , such imbalance of the first set of pseudo-labels starts PL model badly when there is a large portion of OOD data , putting the model in danger of imbalanced learning . We run vanilla PL and show that the imbalance of pseudo-labels harms the performance . Figure 3 ( a ) shows the performance of PL model with different OOD ratios . In accord with Oliver et al . ( 2018 ) , PL model degrades as the portion of OOD data gets larger . Figure 3 ( b ) displays the confusion matrix of the PL model on the whole test set containing both ID and OOD data . Since only 6 classes are known to us , the confusion matrix is a rectangle . We can see almost all the OOD samples ( class 6-9 ) are classified as class 0 , which means the imbalance effect on OOD data gets even worse as the PL training goes on . The possible reason is that , unlike Pseudo-labeling on ID data , supervision of labeled data can not help correct pseudo-labels on OOD data . Thus the imbalance continuously deteriorates . The imbalance on OOD data also influences classification performance on ID data . Samples of major classes ( class 0 ) overwhelm the loss and gradient , leading to a degenerate model ( Lin et al. , 2017 ) . We can see the PL model mistakenly classifies many of data with class 1-5 into class 0 . 3.3 PSEUDO-LABELING STRATEGY FOR OOD DATA . The previous section shows OOD data hurt performance of vanilla PL . Then here comes the question : Assuming that we already know which data are OOD , how do we use these OOD data ? Is omitting them the best way ? If not , what are the better pseudo-labels for them ? To answer these questions , we investigate four strategies to create pseudo-labels for OOD data : • Baseline . This baseline omits all the OOD data and only trains on the labeled ID data . • Re-Assigned Labeling . This strategy assigns data of each OOD class to an ID class . It ensures that different OOD class is assigned to different ID class , keeping the semantics unchanged between OOD classes . For example , ( ship , trunk , airline , automobile ) can be assigned to ( bird , cat , deer , dog ) . This strategy can be seen as training a classifier of “ super-classes ” . • Open-Set Labeling . This strategy is named after the related setting – Open-Set Recognition ( Scheirer et al. , 2013 ; Bendale & Boult , 2016 ) . This strategy treats all OOD data as one unified class KID + 1 . Thus this model outputs probability over KID + 1 classes . • Oracle Labeling . This strategy uses the ground truth of OOD data . Thus this model outputs probability over KID +KOOD classes . Note that Open-Set Labeling and Oracle Labeling can classify samples into more than KID classes . However , during evaluation , we only classify samples into KID ID classes . For these model , the predicted label ŷ of a test sample x is calculated as : ŷ ( x ) = argmax y∈YID f ( y|x ) ( 3 ) The overall comparison of the four strategies is illustrated in Figure 4 . We also report test accuracy when class mismatch ratio is 100 % . From Figure 4 , we can get several important conclusions . ( 1 ) Re-Assigned Labeling underperforms baseline a little1 . This indicates that assign samples with OOD classes to ID classes does not help the model distinguish between ID classes even if we somehow know which OOD data are semantically different . It also reveals that performing vanilla PL on OOD data may never help even if we do it perfectly . ( 2 ) Open-Set Labeling outperforms baseline , which indicates it improves the performance if we label the OOD data as a class other than ID classes . ( 3 ) We can see Oracle Labeling improves the performance and achieves the best result among the four strategies . It means that in addition to label OOD data as extra classes , if we can further assign OOD data with different semantics to different classes , the model will achieve better results .

Summary: This paper works on the class-mismatch semi-supervised learning problem, where the assumption is that the unlabeled samples include class labels that do not appear in the labeled data, i.e., out-of-distribution (OOD) data, in addition to the class labels that appear in the labeled data, i.e., in-distribution (ID) data. It is known that if there are OOD data included in the unlabeled data, it can degrade performance of semi-supervised learning algorithms. This paper focuses on one of the semi-supervised learning methods, which is the pseudo-labeling method. The paper first investigates how using pseudo-labels in the class-mismatch semi-supervised learning setup can be problematic. The experiments show that proportions of high-confidence data in ID data is larger than OOD data. Experiments also show that pseudo-labels (w.r.t. class predictions) are more balanced in ID data while it is unbalanced in OOD data at the beginning of training (with a pretrained model). This phenomenon becomes exaggerated at the end of training, where almost all of the OOD data is predicted into a single ID class. Assuming we know the labels of the unlabeled data, the paper performs further experiments that compare a few methods, and finds out that it is ideal if we have access to the underlying single class labels of OOD data, and solve the problem as a multi-class classification problem with K_id + K_ood (number of classes of ID and OOD) classes. Based on these observations, the paper proposes an Upsilon model, that consists of a re-balanced pseudo-labelling (RPL) and semantic exploration clustering (SEC). RPL aims to balance the distributions of pseudo-labels based on the observation that pseudo-labels become balanced/imbalanced with ID/OOD data, respectively. SEC is a workaround for not having ground truth labels of OOD data.  Experiments show comparison with both traditional SSL methods and class-mismatch SSL methods, and show how the proposed Upsilon model works better than these baselines. Ablation study shows why the components within the Upsilon model is necessary.