OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization

1 Introduction . Semi-supervised learning ( SSL ) leverages unlabeled data to improve a model ’ s performance [ 27 , 38 , 2 , 1 , 35 , 22 , 37 ] . An SSL model can propagate the class information of a small set of labeled data to a large set of unlabeled data , which significantly improves the recognition accuracy without any additional annotation cost . A common assumption of SSL is that the label spaces of labeled and unlabeled data are identical , but , in practice , the assumption is easily violated . Depending on how it was collected , the unlabeled data may contain novel categories unseen in the labeled training data , i.e. , outliers . Since these outliers can significantly harm the performance of SSL algorithms [ 14 ] , detecting them is necessary to make SSL more practical . Ideally , a model should classify samples of known categories i.e. , inliers , into correct classes while identifying samples of novel categories as outliers . This task is called Open-set Semi-supervised Learning ( OSSL ) [ 44 ] . While OSSL is a more realistic and practical scenario than standard SSL , it has not been as widely explored . Existing strong SSL methods [ 35 , 41 ] do not work well for OSSL . For example , FixMatch [ 35 ] generates pseudo-labels using the model ’ s predictions on weakly augmented unlabeled images and trains the model to match its predictions on strongly augmented images with the pseudo-labels . This method exploits the advantage of pseudo-labeling as well as regularizing a model with the consistency between differently augmented images . But , in OSSL , this risks assigning pseudo-labels of known categories to outliers , which degrades the recognition accuracy . A possible solution is to compute the SSL objective only for unlabeled samples considered to be inliers , where confidence thresholding is used to pick inliers . For instance , MTC [ 44 ] regards some proportion of samples as outliers by using Otsu thresholding [ 25 ] . However , this is not robust to varying proportions of outliers as discussed in 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 44 ] . DS3L [ 14 ] proposes meta-optimization that attempts to pick unlabeled data useful to improve generalization performance . But , this method does not have an objective to separate inliers from outliers . Given the limitations of the existing methods , we aim to learn representations that separate outliers from inliers in a feature space and a threshold effective for detecting outliers . The challenge is learning such representations with a small number of labeled inliers and no supervision to find outliers . Moreover , choosing an accurate threshold value is not a trivial problem . We propose a new framework , OpenMatch , to address the above drawbacks of OSSL . First , we propose to utilize a One-Vs-All ( OVA ) network [ 32 ] that can learn a threshold to distinguish outliers from inliers . A separate OVA-classifier is trained for each class , and a sample is labeled an outlier if all of the classifiers determine it to be one . Thus , this technique allows us to identify outliers in an unsupervised way . We call this an outlier detector . Second , we propose a novel open-set soft-consistency loss to learn more effective representations for detecting outliers . We first transform an unlabeled input in two ways and obtain two logits from the outlier detector . Then , we minimize the distance between the two logits to encourage consistency ( See Fig . 1 ) . The main difference between SSL and OSSL is that unlabeled outliers do not have any neighboring labeled samples , which makes it risky to perform hard-labeling such as pseudo-labeling . The outlier detector outputs a distance from inliers given an input , and enhancing the smoothness of this function allows us to improve its ability to find outliers . Empirically , this objective provides significant improvements in detecting outliers . Finally , to correctly classify inliers , we propose to apply FixMatch [ 35 ] to unlabeled samples that are considered to be inliers by the outlier detector . The resulting framework shows consistent gains over baselines on various datasets and settings of OSSL . For example , OpenMatch achieves a 10.4 % error rate on CIFAR10 with 300 labeled examples compared to the previous state-of-the-art of 20.3 % . Surprisingly , OpenMatch demonstrates good performance in detecting outliers unseen in unlabeled training data . For instance , in the experiments on CIFAR10 with 100 labels per class , OpenMatch achieves a 3.4 % higher AUROC in detecting outliers than a supervised model trained with all training samples . To summarize , our contributions are as follows : • A soft open-set consistency regularization ( SOCR ) , to improve outlier detection in OSSL . • A new framework , OpenMatch , which combines a OVA-classifier , SOCR , and FixMatch . • A new state-of-the-art in both correctly classifying inliers and detecting outliers , even when the outliers are unseen in unlabeled training data . 2 Related Work . Semi-supervised Learning . Pseudo-labeling ( PL ) is a strong baseline in semi-supervised learning [ 23 ] , and methods like FixMatch [ 35 ] or UDA [ 41 ] combining data augmentation and PL show the highest performance on many benchmark datasets . However , even with these strong methods , performance drops when a model is exposed to noisy unlabeled data that includes novel categories . On the other hand , methods based on soft consistency regularization enforce smoothness of the decision boundary with respect to stochastic transformations [ 34 ] or models [ 22 , 37 ] . By `` soft '' , we mean that no sharpening or pseudo-labeling is applied to logits to propagate training signals through soft targets . The soft consistency regularization can be understood as approximated Jacobian regularization method [ 12 , 31 ] . We propose to apply soft consistency regularization to our outlier detector , which we call soft open-set consistency regularization . Training a model with self-supervised learning and SSL achieves label-efficient training [ 5 , 46 ] . Self-supervised learning and SSL seem to be complementary to each other to improve the recognition accuracy . But , unlabeled data with outliers will hurt the performance of these approaches since they rely on existing SSL objectives . Since our work focuses on the aspect of SSL , we hope it is complementary to self-supervised learning . Open-set SSL methods include MTC [ 44 ] , D3SL [ 14 ] , and UASD [ 6 ] . MTC updates the network parameters and the anomaly score of unlabeled data alternately . In addition , it minimizes SSL loss ( MixMatch [ 2 ] for a part of unlabeled data that is considered to be inliers . D3SL selectively employs unlabeled data to optimize SSL loss as well as optimizing a function that selects unlabeled data . UASD generates soft targets of the classifier for inliers ( closed-set classifier ) by averaging over many temporally ensembled networks ’ predictions . By contrast , OpenMatch does not have to store the temporal ensembles and applies soft consistency loss to the outlier detector . Our empirical results show that the consistency loss for the outlier detector outperforms that for the closed-set classifier by a lot ( See Sec . 4.4 ) . Open-set Domain Adaptation ( ODA ) . OSSL is similar to open-set domain adaptation [ 3 , 33 ] in that the unlabeled data contains novel categories . A key difference is that the unlabeled and labeled data follow different data distributions . Another key difference is that we aim to train a model from scratch while the domain adaptation task assumes access to models pre-trained on ImageNet [ 9 ] . Since the ImageNet contains 1,000 diverse categories , the pre-trained models have discriminative representations useful to detect outliers . In this point , our task is more challenging than ODA . To detect outliers in an unsupervised way , we employ a OVA-classifier model [ 32 ] proposed for this task . Novelty Detection . Novelty detection or open-set classification aims to identify outliers that are completely unseen during training [ 17 , 24 , 10 , 11 ] . Self-supervised learning such as rotation prediction and contrastive learning is shown to be useful to separate outliers from inliers [ 19 , 36 , 40 ] . The task assumes plenty of labeled inliers in training data . Padhy et al . employ OVA-classifiers for this task [ 30 ] . Hendrycks et al . reveal that exposing a model to outlier data allows it to effectively detect anomalies [ 18 ] and train it on out-of-distribution training data in a supervised way . But , collecting such useful out-of-distribution data is not always possible . In OSSL , a model can not be trained on many labeled samples and needs to detect which samples are inlier or outliers in unlabeled data . In this sense , OSSL is more challenging and more realistic . 3 OpenMatch . Problem Setting . Our task is to learn a classifier using open-set semi-supervised learning . For a K-way classification problem , let X = { ( xb , yb ) : b ∈ ( 1 , ... B ) } be a batch of B labeled examples xb randomly sampled from the labeled training set , S , where yb ∈ ( 1 , ... K ) is the corresponding label . Let U = { ( ub ) : b ∈ ( 1 , ... µB ) } be a batch of µB data randomly sampled from the unlabeled training set , Su , where µ is a hyper-parameter that determines the relative sizes of X and U . Unlike in SSL , which assumes all unlabeled data come from the K known classes , in our OSSL setting they may come from unknown outlier classes . The goal of OSSL is to classify inliers into the correct classes while learning to detect the outliers . Approach Overview . Our model has three components : ( 1 ) a shared feature extractor F ( · ) , ( 2 ) an outlier detector consisting of K one-vs-all ( OVA ) sub-classifiers Dj ( · ) , j ∈ { 1 , ... , K } , and ( 3 ) a closed-set classifier C ( · ) which outputs a probability vector ∈ RK for K-class classification . At test time , the closed-set classifier is first applied to predict the K-way label , ŷ . If Dŷ predicts an inlier , then the output class is ŷ , otherwise , the output is `` outlier '' . The main technical novelty comes from the choice of OVA-classifiers for outlier detection as well as training them with soft open-set consistency regularization . We first describe the training of the OVA-classifiers [ 32 ] before describing the remaining details of our framework . 3.1 One-vs-All Outlier Detector . For open-set [ 33 ] or universal domain adaptation [ 42 ] , Saito et al . [ 32 ] propose to train OVA-classifiers to detect outlier samples of an unlabeled target domain . They aim to learn a boundary between inliers and outliers for each class . Each sub-classifier is trained to distinguish if the sample is an inlier for the corresponding class . For example , the sub-classifier for class j , Dj , outputs a 2-dimensional vector zjb = D j ( F ( xb ) ) ∈ R2 , where each dimension indicates the score of a sample being an inlier and outlier respectively . We denote pj ( t = 0|xb ) and pj ( t = 1|xb ) as the probability of xb being the inlier and outlier for the class j , computed by Softmax ( zjb ) ( Note that p j ( t = 0|xb ) + pj ( t = 1|xb ) = 1 ) . Then , the sub-classifier Dj is trained such that the data from class j is treated as positives , and the data from all the other classes is treated as negatives . To achieve this , we minimize the following loss for the outlier detector for a given batch X : = { xb , yb } Bb=1 : Lova ( X ) : = 1 B B∑ b=1 − log ( pyb ( t = 0|xb ) ) −min i 6=yb log ( pi ( t = 1|xb ) ) . ( 1 ) Here we use the hard-negative sub-classifier sampling technique proposed by [ 32 ] to effectively learn the threshold . We refer the reader to [ 32 ] for more details . The key is that each sub-classifier outputs a distance representing how far the input is from the corresponding class . Therefore , the classifiers are effective to identify unlabeled outliers . The output of Dj is used to detect outliers as follows . If the probability of a sample being an outlier for the class highest-scored by C ( ) is larger than that of a sample being an inlier , we consider the input as an outlier . To be specific , we identify the unlabeled sample ub as an outlier if pŷ ( t = 0|ub ) < 0.5 , where ŷ = argmaxj C ( F ( ub ) ) . This detection is employed in selecting pseudo-inliers from unlabeled data as explained in Sec . 3.3 . Open-set Entropy Minimization . For unlabeled samples , [ 32 ] apply entropy minimization with respect to the OVA-classifiers , denoted by Lem ( U ) . The separation between inliers and outliers is enhanced through the minimization of this loss for a given batch U : = { ub } µBb=1 : Lem ( U ) : = − 1 µB µB∑ b=1 K∑ j=1 pj ( t = 0|ub ) log pj ( t = 0|ub ) + pj ( t = 1|ub ) log pj ( t = 1|ub ) . ( 2 ) The difference between our setup and [ 32 ] is that they employ a model pre-trained on ImageNet [ 9 ] , which extracts highly discriminative features , thus , inliers and outliers are well-separated even before fine-tuning . However , we aim to train a model to learn representations separating them from scratch . Thus the inliers and outliers are confused before training , and we need an objective to ensure the separation in addition to Eq . 2 .

Summary: The paper proposes a model for open-set semi-supervised learning. As a semi-supervised learning model, it tackles classification where only a portion of the training data is labeled. "Open set" refers to the fact that the unlabeled data are noisy and can contain out of distribution examples, for which the class is not among the known classes. Out of distribution samples are present also during test, and the model has to recognize them and avoid classification on them. The proposed model builds on top of existing works, combining several loss terms and ideas such as one-versus-all classifiers and FixMatch as the main semi-supervised learning engine. The main technical contribution is a loss term enforcing the out-of-distribution score for two augmented versions of the same unlabeled example to be similar. Extensive experiments validate the proposal against recent state-of-the-art models.