OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data

1 INTRODUCTION . Despite the recent success of deep neural networks with large-scale labeled data , many real-world scenarios suffer from expensive data acquisition and labeling costs . This has motivated the community to develop semi-supervised learning ( SSL ; Grandvalet & Bengio 2004 ; Chapelle et al . 2009 ) , i.e. , by further incorporating unlabeled data for training . Indeed , recent SSL works ( Berthelot et al. , 2019 ; 2020 ; Sohn et al. , 2020 ) demonstrate promising results on several benchmark datasets , as they could even approach the performance of fully supervised learning using only a small number of labels , e.g. , 93.73 % accuracy on CIFAR-10 with 250 labeled data ( Berthelot et al. , 2020 ) . However , SSL methods often fail to generalize when there is a mismatch between the classdistributions of labeled and unlabeled data ( Oliver et al. , 2018 ; Chen et al. , 2020c ; Guo et al. , 2020 ) , i.e. , when the unlabeled data contains out-of-class samples , whose ground-truth labels are not contained in the labeled dataset ( as illustrated in Figure 1 ( a ) ) . In this scenario , various label-guessing techniques used in the existing SSL methods may label those out-of-class samples incorrectly , which in turn significantly harms the overall training through their inner-process of entropy minimization ( Grandvalet & Bengio , 2004 ; Lee , 2013 ) or consistency regularization ( Xie et al. , 2019 ; Sohn et al. , 2020 ) . This problem may largely hinder the existing SSL methods from being used in practice , considering the open-set nature of unlabeled data collected in the wild ( Bendale & Boult , 2016 ) . Contribution . In this paper , we focus on a realistic SSL scenario , where unlabeled data may contain some unknown out-of-class samples , i.e. , there is a class distribution mismatch between labeled and unlabeled data ( Oliver et al. , 2018 ) . Compared to prior approaches that have bypassed this problem by simply filtering out them with some heuristic detection scores ( Nair et al. , 2019 ; Chen et al. , 2020c ) , the unique characteristic in our approach is to further leverage the information in out-of-class samples by assigning soft-labels to them : they may still contain some useful features for the in-classes . Somewhat surprisingly , we found that a recent technique of contrastive unsupervised learning ( Wu et al. , 2018 ; He et al. , 2020 ; Chen et al. , 2020a ) can play a key role for our goal . More specifically , we show that a pre-trained representation via contrastive learning , namely SimCLR ( Chen et al. , 2020a ) , on both labeled and unlabeled data enables us to design ( a ) an effective score for detecting out-of-class samples in unlabeled data , and ( b ) a systematic way to assign soft-labels to the detected out-of-class samples , by modeling class-conditional likelihoods from labeled data . Finally , we found ( c ) auxiliary batch normalization layers ( Xie et al. , 2020 ) could further help to mitigate the class-distribution mismatch via decoupling batch normalization layers . We propose a generic SSL framework , coined OpenCoS , based on the aforementioned techniques for handling open-set unlabeled data , which can be integrated with any existing SSL methods . We verify the effectiveness of the proposed method on a wide range of SSL benchmarks based on CIFAR-10 , CIFAR-100 ( Krizhevsky et al. , 2009 ) , and ImageNet ( Deng et al. , 2009 ) datasets , assuming the presence of various out-of-class data , e.g. , SVHN ( Netzer et al. , 2011 ) and TinyImageNet datasets . Our experimental results demonstrate that OpenCoS greatly improves existing state-of-the-art SSL methods ( Berthelot et al. , 2019 ; 2020 ; Sohn et al. , 2020 ) , not only by discarding out-of-class samples , but also by further leveraging them into training . We also compare our method to other recent works ( Nair et al. , 2019 ; Chen et al. , 2020c ; Guo et al. , 2020 ) addressing the same class distribution mismatch problem in SSL , and again confirms the effectiveness of our framework , e.g. , we achieve an accuracy of 68.37 % with 40 labels ( just 4 labels per class ) on CIFAR-10 with TinyImageNet as out-of-class , compared to DS3L ( Guo et al. , 2020 ) of 56.32 % . Overall , our work highlights the benefit of unsupervised representations in ( semi- ) supervised learning : such a label-free representation turns out to enhance model generalization due to its robustness on the novel , out-of-class samples . 2 PRELIMINARIES . 2.1 SEMI-SUPERVISED LEARNING . The goal of semi-supervised learning for classification is to train a classifier f : X → Y from a labeled dataset Dl = { x ( i ) l , y ( i ) l } Nl i=1 where each label yl is from a set of classes Y : = { 1 , · · · , C } , and an unlabeled dataset Du = { x ( i ) u } Nui=1 where each yu exists but is assumed to be unknown . In an attempt to leverage the extra information in Du , a number of techniques have been proposed , e.g. , entropy minimization ( Grandvalet & Bengio , 2004 ; Lee , 2013 ) and consistency regularization ( Sajjadi et al. , 2016 ) . In general , recent approaches in semi-supervised learning can be distinguished by the prior they adopt for the representation of unlabeled data : for example , the consistency regularization technique ( Sajjadi et al. , 2016 ) attempt to minimize the cross-entropy loss between any two predictions of different augmentations t1 ( xu ) and t2 ( xu ) from a given unlabeled sample xu , jointly with the standard training for a labeled sample ( xl , yl ) : LSSL ( xl , xu ) : = H ( yl , f ( xl ) ) + β ·H ( f ( t1 ( xu ) ) , f ( t2 ( xu ) ) ) , ( 1 ) where H is a standard cross-entropy loss for labeled data , and β is a hyperparameter . Recently , several “ holistic ” approaches of various techniques ( Zhang et al. , 2018 ; Cubuk et al. , 2019 ) have shown remarkable performance in practice , e.g. , MixMatch ( Berthelot et al. , 2019 ) , ReMixMatch ( Berthelot et al. , 2020 ) , and FixMatch ( Sohn et al. , 2020 ) , which we mainly consider in this paper . We note that our scheme can be integrated with any recent semi-supervised learning methods . 2.2 CONTRASTIVE REPRESENTATION LEARNING . Contrastive learning ( Oord et al. , 2018 ; Hénaff et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020a ) defines an unsupervised task for an encoder fe : X → Rde from a set of samples { xi } : assume that a “ query ” sample xq is given and there is a positive “ key ” x+ ∈ { xi } that xq matches . Then the contrastive loss is defined to let f to extract the necessary information to identify x+ from xq as follows : Lcon ( fe , xq , x+ ; { xi } ) : = − log exp ( h ( fe ( xq ) , fe ( x+ ) ) /τ ) ∑ i exp ( h ( fe ( xq ) , fe ( xi ) ) /τ ) , ( 2 ) where h ( · , · ) is a pre-defined similarity score , and τ is a temperature hyperparameter . In this paper , we primarily focus on SimCLR ( Chen et al. , 2020a ) , a particular form of contrastive learning : for a given { xi } Ni=1 , SimCLR first samples two separate data augmentation operations from a pre-defined family T , namely t1 , t2 ∼ T , and matches ( x̃i , x̃i+N ) : = ( t1 ( xi ) , t2 ( xi ) ) as a query-key pair interchangeably . The actual loss is then defined as follows : LSimCLR ( fe ; { xi } Ni=1 ) : = 1 2N 2N∑ q=1 Lcon ( fe , x̃q , x̃ ( q+N ) mod 2N ; { x̃i } 2Ni=1 \ { x̃q } ) , ( 3 ) hSimCLR ( v1 , v2 ) : = CosineSimilarity ( g ( v1 ) , g ( v2 ) ) = g ( v1 ) · g ( v2 ) ||g ( v1 ) ||2||g ( v2 ) ||2 , ( 4 ) where g : Rde → Rdp is a 2-layer neural network called projection header . In other words , the SimCLR loss defines a task to identify a “ semantically equivalent ” sample to xq up to the set of data augmentations T . 3 OPENCOS : A FRAMEWORK FOR OPEN-SET SEMI-SUPERVISED LEARNING . We consider semi-supervised classification problems involving C classes . In addition to the standard assumption of semi-supervised learning ( SSL ) , we assume that the unlabeled dataset Du is open-set , i.e. , the hidden labels yu of xu may not be in Y : = { 1 , · · · , C } . In this scenario , existing semisupervised learning techniques may degrade the classification performance , possibly due to incorrect label-guessing procedure for those out-of-class samples . In this respect , we introduce OpenCoS , a generic method for detecting and labeling out-of-class unlabeled samples in semi-supervised learning . Overall , our key intuition is to utilize the unsupervised representation from contrastive learning ( Wu et al. , 2018 ; He et al. , 2020 ; Chen et al. , 2020a ) to leverage such out-of-class samples in an appropriate manner . We present a brief overview of our method in Section 3.1 , and describe how our approach , OpenCoS , can handle out-of-class samples in Section 3.2 and 3.3 . 3.1 OVERVIEW OF OPENCOS . Recall that our goal is to train a classifier f : X → Y from a labeled dataset Dl and an open-set unlabeled dataset Du . Overall , OpenCoS aims to overcome the presence of out-of-class samples in Du through the following procedure : 1 . Pre-training via contrastive learning . OpenCoS first learns an unsupervised representation of f via SimCLR1 ( Chen et al. , 2020a ) , using both Dl and Du without labels . More specifically , we learn the penultimate features of f , denoted by fe , by minimizing the contrastive loss defined in ( 3 ) . We also introduce a projection header g ( 4 ) , which is a 2-layer MLP as per ( Chen et al. , 2020a ) . 2 . Detecting out-of-class samples . From a learned representation of fe and g , OpenCoS identifies an out-of-class unlabeled data Doutu from the given data Du = Dinu ∪ Doutu . This detection process is based on the similarity score between Dl and Du in the representation space of fe and g ( see Section 3.2 ) . 3 . Semi-supervised learning with auxiliary loss and batch normalization . Now , one can use any semi-supervised learning scheme to train f using Dl and Dinu , e.g. , ReMixMatch ( Berthelot et al. , 2020 ) . In addition , OpenCoS minimizes an auxiliary loss that assigns a soft-label to each sample in Doutu , which is also based on the representation of fe and g ( see Section 3.3 ) . Furthermore , we found maintaining auxiliary batch normalization layers ( Xie et al. , 2020 ) for Doutu is beneficial to our loss as they mitigate the distribution mismatch arisen from Doutu . Putting it all together , OpenCoS provides an effective and systematic way to detect and utilize outof-class data for semi-supervised learning . Due to its simplicity , our framework can incorporate the most recently proposed semi-supervised learning methods ( Berthelot et al. , 2019 ; 2020 ; Sohn et al. , 2020 ) and improve their performance in the presence of out-of-class samples . Figure 2 illustrates the overall training scheme of OpenCoS .

Summary: This paper considers the problem of semi-supervised learning, where the unlabeled data may include out-of-class samples. To address this task, the paper proposes a method consisting of three steps: (1) detecting out-of-class samples in the unlabeled set, (2) assigning soft-labels to the detected out-of-class samples using class-conditional likelihoods from labeled data, and (3) using auxiliary batch normalization layers  to help mitigate the class distribution mismatch problem. Experiments are conducted on CIFAR-10, CIFAR-100, ImageNet datasets. Results show improvements over competing methods.