OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization

1 Introduction . Semi-supervised learning ( SSL ) leverages unlabeled data to improve a model ’ s performance [ 27 , 38 , 2 , 1 , 35 , 22 , 37 ] . An SSL model can propagate the class information of a small set of labeled data to a large set of unlabeled data , which significantly improves the recognition accuracy without any additional annotation cost . A common assumption of SSL is that the label spaces of labeled and unlabeled data are identical , but , in practice , the assumption is easily violated . Depending on how it was collected , the unlabeled data may contain novel categories unseen in the labeled training data , i.e. , outliers . Since these outliers can significantly harm the performance of SSL algorithms [ 14 ] , detecting them is necessary to make SSL more practical . Ideally , a model should classify samples of known categories i.e. , inliers , into correct classes while identifying samples of novel categories as outliers . This task is called Open-set Semi-supervised Learning ( OSSL ) [ 44 ] . While OSSL is a more realistic and practical scenario than standard SSL , it has not been as widely explored . Existing strong SSL methods [ 35 , 41 ] do not work well for OSSL . For example , FixMatch [ 35 ] generates pseudo-labels using the model ’ s predictions on weakly augmented unlabeled images and trains the model to match its predictions on strongly augmented images with the pseudo-labels . This method exploits the advantage of pseudo-labeling as well as regularizing a model with the consistency between differently augmented images . But , in OSSL , this risks assigning pseudo-labels of known categories to outliers , which degrades the recognition accuracy . A possible solution is to compute the SSL objective only for unlabeled samples considered to be inliers , where confidence thresholding is used to pick inliers . For instance , MTC [ 44 ] regards some proportion of samples as outliers by using Otsu thresholding [ 25 ] . However , this is not robust to varying proportions of outliers as discussed in 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 44 ] . DS3L [ 14 ] proposes meta-optimization that attempts to pick unlabeled data useful to improve generalization performance . But , this method does not have an objective to separate inliers from outliers . Given the limitations of the existing methods , we aim to learn representations that separate outliers from inliers in a feature space and a threshold effective for detecting outliers . The challenge is learning such representations with a small number of labeled inliers and no supervision to find outliers . Moreover , choosing an accurate threshold value is not a trivial problem . We propose a new framework , OpenMatch , to address the above drawbacks of OSSL . First , we propose to utilize a One-Vs-All ( OVA ) network [ 32 ] that can learn a threshold to distinguish outliers from inliers . A separate OVA-classifier is trained for each class , and a sample is labeled an outlier if all of the classifiers determine it to be one . Thus , this technique allows us to identify outliers in an unsupervised way . We call this an outlier detector . Second , we propose a novel open-set soft-consistency loss to learn more effective representations for detecting outliers . We first transform an unlabeled input in two ways and obtain two logits from the outlier detector . Then , we minimize the distance between the two logits to encourage consistency ( See Fig . 1 ) . The main difference between SSL and OSSL is that unlabeled outliers do not have any neighboring labeled samples , which makes it risky to perform hard-labeling such as pseudo-labeling . The outlier detector outputs a distance from inliers given an input , and enhancing the smoothness of this function allows us to improve its ability to find outliers . Empirically , this objective provides significant improvements in detecting outliers . Finally , to correctly classify inliers , we propose to apply FixMatch [ 35 ] to unlabeled samples that are considered to be inliers by the outlier detector . The resulting framework shows consistent gains over baselines on various datasets and settings of OSSL . For example , OpenMatch achieves a 10.4 % error rate on CIFAR10 with 300 labeled examples compared to the previous state-of-the-art of 20.3 % . Surprisingly , OpenMatch demonstrates good performance in detecting outliers unseen in unlabeled training data . For instance , in the experiments on CIFAR10 with 100 labels per class , OpenMatch achieves a 3.4 % higher AUROC in detecting outliers than a supervised model trained with all training samples . To summarize , our contributions are as follows : • A soft open-set consistency regularization ( SOCR ) , to improve outlier detection in OSSL . • A new framework , OpenMatch , which combines a OVA-classifier , SOCR , and FixMatch . • A new state-of-the-art in both correctly classifying inliers and detecting outliers , even when the outliers are unseen in unlabeled training data . 2 Related Work . Semi-supervised Learning . Pseudo-labeling ( PL ) is a strong baseline in semi-supervised learning [ 23 ] , and methods like FixMatch [ 35 ] or UDA [ 41 ] combining data augmentation and PL show the highest performance on many benchmark datasets . However , even with these strong methods , performance drops when a model is exposed to noisy unlabeled data that includes novel categories . On the other hand , methods based on soft consistency regularization enforce smoothness of the decision boundary with respect to stochastic transformations [ 34 ] or models [ 22 , 37 ] . By `` soft '' , we mean that no sharpening or pseudo-labeling is applied to logits to propagate training signals through soft targets . The soft consistency regularization can be understood as approximated Jacobian regularization method [ 12 , 31 ] . We propose to apply soft consistency regularization to our outlier detector , which we call soft open-set consistency regularization . Training a model with self-supervised learning and SSL achieves label-efficient training [ 5 , 46 ] . Self-supervised learning and SSL seem to be complementary to each other to improve the recognition accuracy . But , unlabeled data with outliers will hurt the performance of these approaches since they rely on existing SSL objectives . Since our work focuses on the aspect of SSL , we hope it is complementary to self-supervised learning . Open-set SSL methods include MTC [ 44 ] , D3SL [ 14 ] , and UASD [ 6 ] . MTC updates the network parameters and the anomaly score of unlabeled data alternately . In addition , it minimizes SSL loss ( MixMatch [ 2 ] for a part of unlabeled data that is considered to be inliers . D3SL selectively employs unlabeled data to optimize SSL loss as well as optimizing a function that selects unlabeled data . UASD generates soft targets of the classifier for inliers ( closed-set classifier ) by averaging over many temporally ensembled networks ’ predictions . By contrast , OpenMatch does not have to store the temporal ensembles and applies soft consistency loss to the outlier detector . Our empirical results show that the consistency loss for the outlier detector outperforms that for the closed-set classifier by a lot ( See Sec . 4.4 ) . Open-set Domain Adaptation ( ODA ) . OSSL is similar to open-set domain adaptation [ 3 , 33 ] in that the unlabeled data contains novel categories . A key difference is that the unlabeled and labeled data follow different data distributions . Another key difference is that we aim to train a model from scratch while the domain adaptation task assumes access to models pre-trained on ImageNet [ 9 ] . Since the ImageNet contains 1,000 diverse categories , the pre-trained models have discriminative representations useful to detect outliers . In this point , our task is more challenging than ODA . To detect outliers in an unsupervised way , we employ a OVA-classifier model [ 32 ] proposed for this task . Novelty Detection . Novelty detection or open-set classification aims to identify outliers that are completely unseen during training [ 17 , 24 , 10 , 11 ] . Self-supervised learning such as rotation prediction and contrastive learning is shown to be useful to separate outliers from inliers [ 19 , 36 , 40 ] . The task assumes plenty of labeled inliers in training data . Padhy et al . employ OVA-classifiers for this task [ 30 ] . Hendrycks et al . reveal that exposing a model to outlier data allows it to effectively detect anomalies [ 18 ] and train it on out-of-distribution training data in a supervised way . But , collecting such useful out-of-distribution data is not always possible . In OSSL , a model can not be trained on many labeled samples and needs to detect which samples are inlier or outliers in unlabeled data . In this sense , OSSL is more challenging and more realistic . 3 OpenMatch . Problem Setting . Our task is to learn a classifier using open-set semi-supervised learning . For a K-way classification problem , let X = { ( xb , yb ) : b ∈ ( 1 , ... B ) } be a batch of B labeled examples xb randomly sampled from the labeled training set , S , where yb ∈ ( 1 , ... K ) is the corresponding label . Let U = { ( ub ) : b ∈ ( 1 , ... µB ) } be a batch of µB data randomly sampled from the unlabeled training set , Su , where µ is a hyper-parameter that determines the relative sizes of X and U . Unlike in SSL , which assumes all unlabeled data come from the K known classes , in our OSSL setting they may come from unknown outlier classes . The goal of OSSL is to classify inliers into the correct classes while learning to detect the outliers . Approach Overview . Our model has three components : ( 1 ) a shared feature extractor F ( · ) , ( 2 ) an outlier detector consisting of K one-vs-all ( OVA ) sub-classifiers Dj ( · ) , j ∈ { 1 , ... , K } , and ( 3 ) a closed-set classifier C ( · ) which outputs a probability vector ∈ RK for K-class classification . At test time , the closed-set classifier is first applied to predict the K-way label , ŷ . If Dŷ predicts an inlier , then the output class is ŷ , otherwise , the output is `` outlier '' . The main technical novelty comes from the choice of OVA-classifiers for outlier detection as well as training them with soft open-set consistency regularization . We first describe the training of the OVA-classifiers [ 32 ] before describing the remaining details of our framework . 3.1 One-vs-All Outlier Detector . For open-set [ 33 ] or universal domain adaptation [ 42 ] , Saito et al . [ 32 ] propose to train OVA-classifiers to detect outlier samples of an unlabeled target domain . They aim to learn a boundary between inliers and outliers for each class . Each sub-classifier is trained to distinguish if the sample is an inlier for the corresponding class . For example , the sub-classifier for class j , Dj , outputs a 2-dimensional vector zjb = D j ( F ( xb ) ) ∈ R2 , where each dimension indicates the score of a sample being an inlier and outlier respectively . We denote pj ( t = 0|xb ) and pj ( t = 1|xb ) as the probability of xb being the inlier and outlier for the class j , computed by Softmax ( zjb ) ( Note that p j ( t = 0|xb ) + pj ( t = 1|xb ) = 1 ) . Then , the sub-classifier Dj is trained such that the data from class j is treated as positives , and the data from all the other classes is treated as negatives . To achieve this , we minimize the following loss for the outlier detector for a given batch X : = { xb , yb } Bb=1 : Lova ( X ) : = 1 B B∑ b=1 − log ( pyb ( t = 0|xb ) ) −min i 6=yb log ( pi ( t = 1|xb ) ) . ( 1 ) Here we use the hard-negative sub-classifier sampling technique proposed by [ 32 ] to effectively learn the threshold . We refer the reader to [ 32 ] for more details . The key is that each sub-classifier outputs a distance representing how far the input is from the corresponding class . Therefore , the classifiers are effective to identify unlabeled outliers . The output of Dj is used to detect outliers as follows . If the probability of a sample being an outlier for the class highest-scored by C ( ) is larger than that of a sample being an inlier , we consider the input as an outlier . To be specific , we identify the unlabeled sample ub as an outlier if pŷ ( t = 0|ub ) < 0.5 , where ŷ = argmaxj C ( F ( ub ) ) . This detection is employed in selecting pseudo-inliers from unlabeled data as explained in Sec . 3.3 . Open-set Entropy Minimization . For unlabeled samples , [ 32 ] apply entropy minimization with respect to the OVA-classifiers , denoted by Lem ( U ) . The separation between inliers and outliers is enhanced through the minimization of this loss for a given batch U : = { ub } µBb=1 : Lem ( U ) : = − 1 µB µB∑ b=1 K∑ j=1 pj ( t = 0|ub ) log pj ( t = 0|ub ) + pj ( t = 1|ub ) log pj ( t = 1|ub ) . ( 2 ) The difference between our setup and [ 32 ] is that they employ a model pre-trained on ImageNet [ 9 ] , which extracts highly discriminative features , thus , inliers and outliers are well-separated even before fine-tuning . However , we aim to train a model to learn representations separating them from scratch . Thus the inliers and outliers are confused before training , and we need an objective to ensure the separation in addition to Eq . 2 .

Summary: The authors propose an idea to improve model's performance under the open-set semi-supervised learning setting. This work proposed to leverage OOD detector, combined with FixMatch, and soft open-set consistency regularization (SOCR) to improve the model training during open-set semi-supervised learning. I think the usage of OOD on ignore outliers is not new and an unsupervised method on learning the consistency between augmentations for unlabeled data is also not new. Thus, I am not sure if I can find too much of novelty from this work itself, but I do appreciate the author's effort in putting them together to be a complete model, which in fact make solid progress on the open-set semi-supervised learning setting.