Open-world Semi-supervised Learning

1 INTRODUCTION . With the advent of deep learning , remarkable breakthroughs have been achieved and current machine learning systems excel on tasks with large quantities of labeled data ( Hinton et al. , 2012 ; LeCun et al. , 2015 ; Silver et al. , 2016 ; Esteva et al. , 2017 ) . Despite the strengths , the vast majority of models are designed for the closed-world setting rooted in the assumption that training and test data come from the same set of predefined classes ( Bendale & Boult , 2015 ; Boult et al. , 2019 ) . This assumption , however , rarely holds for data in-the-wild , as labeling data depends on having the complete knowledge of a given domain , which is rarely the case in practice . For example , biologists may prelabel some of the known cell types ( seen classes ) , and then want to train and apply the model to a new tissue to identify known cell types but also to discover novel previously unknown cell types ( unseen classes ) . Similarly , in social networks one may want to classify users into predefined interest groups while also discovering new unknown/unlabeled interests of users . Thus , in contrast to the commonly assumed closed world , many real-world problems are inherently open-world—new classes can emerge in the test data that may have never been seen ( and labeled ) during training . Here we introduce open-world semi-supervised learning ( open-world SSL ) setting that generalizes semi-supervised learning and novel class discovery . Under open-world SSL , we are given a labeled training dataset as well as an unlabeled dataset . The labeled dataset contains instances that belong to a set of seen classes , while instances in the unlabeled/test dataset belong to both the seen classes as well as to an unknown number of unseen classes ( Figure 1 ) . Under this setting , the model then needs to simultaneously classify instances into previously seen classes and then also discover new classes and assign instances to them . In other words , open-world SSL is a transductive learning setting under class distribution mismatch in which unlabeled test set may contain classes that have never been labeled during training , i.e. , are not part of the labeled training set . Given unlabeled test set , the model needs to either assign instances to one of the classes previously seen in the labeled set , or form a novel class and assign instances to it . Open-world SSL is fundamentally different but closely related to two recent lines of work : robust semi-supervised learning ( SSL ) and novel class discovery . Robust SSL ( Oliver et al. , 2018 ; Guo et al. , 2020 ; Chen et al. , 2020b ; Guo et al. , 2020 ; Yu et al. , 2020 ) assumes class distribution mismatch between labeled and unlabeled data , but in this setting the model only needs to be able to recognize ( reject ) instances belonging to novel classes in the unlabeled data as out-of-distribution instances . In contrast , instead of rejecting instances belonging to novel classes , open-world SSL aims at discovering individual novel classes and then assigning instances to them . Novel class discovery ( Hsu et al. , 2018 ; 2019 ; Han et al. , 2019 ; 2020 ; Zhong et al. , 2021 ) is a clustering problem where one assumes unlabeled data is composed only of novel classes . In contrast , open-world SSL is more general as instances in the unlabeled datan can come from seen as well as from novel classes . To apply robust SSL and novel class discovery methods to open-world SSL , one could in principle adopt a multi-step approach , where one could first use robust SSL to first reject examples from novel classes and then apply a novel class discovery method on rejected instances to discover novel classes . An alternative would be that one could treat all classes as “ novel ” , apply novel class discovery methods and then match some of the classes back to the seen classes in the labeled dataset . However , our experiments show that such ad hoc approaches do not perform well in practice . Therefore , it is necessary to design a method that can solve this practical problem in an end-to-end framework . In this paper we propose ORCA ( Open-woRld with unCertainty based Adaptive margin ) that operates under the novel open-world SSL setting . ORCA effectively assigns examples from the unlabeled data to either previously seen classes , or forms a novel classes by grouping similar instances . ORCA is an end-to-end deep learning framework , where the key to our approach is a novel uncertainty adaptive margin mechanism that gradually decreases plasticity and increases discriminability of the model during training . This mechanism effectively reduces an undesired gap between intra-class variance of seen with respect to the novel classes caused by learning seen classes faster than the novel , which we show is a critical difficulty in this setting . We then develop a special model training procedure that learns to classify data points into a set of previously seen classes while also learning to use an additional classification head for each newly discovered class . Classification heads for seen classes are used to assign the unlabeled examples to classes from the labeled set , while activating additional classification heads allows ORCA to form a novel class . ORCA does not need to know the number of novel classes ahead of time and can automatically discover them at the deployment time . We evaluate ORCA on three benchmark image classification datasets adapted for open-world SSL and a single-cell dataset from biology domain . Since no existing methods can operate under the open-world SSL setting we first extend existing state-of-the-art SSL , open-set recognition and novel class discovery methods to the open-world SSL setting and then compare them to ORCA . Experimental results demonstrate that ORCA effectively addresses the challenges of open-world SSL and consistently outperforms all baselines by a large margin . Specifically , ORCA achieves 25 % and 96 % improvements on seen and novel classes of the ImageNet dataset . Moreover , we show that ORCA is robust to unknown number of novel classes , different distributions of seen and novel classes , unbalanced data distributions , pretraining strategies and a small number of labeled examples . 2 RELATED WORK . We summarize similarities and differences between open-world SSL and related settings . Additional related work is given in Appendix A . Novel class discovery . In novel class discovery ( Hsu et al. , 2018 ; Han et al. , 2020 ; Brbic et al. , 2020 ; Zhong et al. , 2021 ) , the task is to cluster unlabeled dataset consisting of similar , but completely disjoint , classes than those present in the labeled dataset which is utilized to learn better representation for clustering . These methods assume that at the test time all the classes are novel . While these methods are able to discover novel classes , they do not recognize the seen/known classes . On the contrary , our open-world SSL is more general because unlabeled test set consists of novel classes but also classes previously seen in the labeled data that need to be identified . In principle , one could extend novel class discovery methods by treating all classes as “ novel ” at test time and then matching some of them to the known classes from the labeled dataset . We adopt such approaches as our baselines , but our experiments show that they do not perform well in practice . Semi-supervised learning ( SSL ) . SSL methods ( Chapelle et al. , 2009 ; Kingma et al. , 2014 ; Laine & Aila , 2017 ; Zhai et al. , 2019 ; Lee , 2013 ; Xie et al. , 2020 ; Berthelot et al. , 2019 ; 2020 ; Sohn et al. , 2020 ) assume closed-world setting in which labeled and unlabeled data come from the same set of classes . Robust SSL methods ( Oliver et al. , 2018 ; Chen et al. , 2020b ; Guo et al. , 2020 ; Yu et al. , 2020 ) relax the SSL assumption by assuming that instances from novel classes may appear in the unlabeled test set . The goal in robust SSL is to reject instances from novel classes which are treated as out-of-distribution instances . Instead of rejecting instances from novel classes , in open-world SSL the goal is to discover individual novel classes and then assign datapoints to them . To extend robust SSL to open-world SSL , one could take discarded points and then apply clustering/novel class discovery . Early work ( Miller & Browning , 2003 ) considered solving the problem in such a way using extension of the EM algorithm . However , our experiments show that by discarding the instances the embedding learned by these methods does not allow for accurate discovery of novel classes . Open-set and open-world recognition . Open-set recognition ( Scheirer et al. , 2012 ; Geng et al. , 2020 ; Bendale & Boult , 2016 ; Ge et al. , 2017 ; Sun et al. , 2020a ) considers the inductive setting in which novel classes can appear during testing , and the model needs to reject instances from novel classes . To extend these methods to open-world setting , we include a baseline that discovers classes on rejected instances . However , results show that such approaches can not effectively address the challenges of open-world SSL . Similarly , open-world recognition approaches ( Bendale & Boult , 2015 ; Rudd et al. , 2017 ; Boult et al. , 2019 ) require the system to incrementally learn and extend the set of known classes with novel classes . These methods incrementally label novel classes by human-in-the-loop . In contrast , open-world SSL leverages unlabeled data in the learning stage and does not require human-in-the-loop . Generalized zero-shot learning ( GZSL ) . Like open-world SSL , GZSL ( Xian et al. , 2017 ; Liu et al. , 2018 ; Chao et al. , 2016 ) assumes that classes seen in the labeled set and novel classes are present at the test time . However , GZSL imposes additional assumption about availability of prior knowledge given as auxiliary attributes that uniquely describe each individual class including the novel classes . This restrictive assumption severely limits the application of GZSL methods in practice . In contrast , open-world SSL is more general as it does not assume any prior information about classes . 3 PROPOSED APPROACH . In this section , we first define the open-world SSL setting . We follow by an overview of ORCA framework and then introduce each of the components of our framework in details . 3.1 OPEN-WORLD SEMI-SUPERVISED LEARNING SETTING . In the open-world SSL , we assume transductive learning setting in which a labeled part of the dataset Dl = { ( xi , yi ) } mi=1 and an unlabeled part of the dataset Du = { ( xi ) } ni=1 are given at the input . We denote the set of classes seen in the labeled data as Cl and set of classes in the unlabeled test data as Cu , respectively . In the open-world SSL , we assume category/class shift , i.e. , Cl ∩ Cu 6= ∅ and Cl 6= Cu . We consider Cs = Cl ∩ Cu as a set of seen classes , and Cn = Cu\Cl as a set of novel classes . Definition 1 ( Open-world SSL ) . In the open-world SSL , the model needs to assign instances from Du either to previously seen classes Cs , or form a novel class c ∈ Cn and assign datapoints to it . Note that open-world SSL generalizes novel class discovery and traditional ( closed-world ) SSL . Novel class discovery assumes that classes in labeled and unlabeled data are disjoint , i.e. , Cl∩Cu = ∅ , while ( closed-world ) SSL assumes the same classes in labeled and unlabeled data , i.e. , Cl = Cu .

Summary: The paper considers open-world SSL settings  where the model recognizes previously seen classes, and detects novel classes which are not present in the labeled dataset. The method contains three losses to train a model in this setting: a) supervised loss on labeled data, b) unsupervised loss on unlabeled data from pseudo-labels obtained  from confident pairwise similarities, and c)  regularization term that avoids assigning all the unlabeled samples to the same class. Then the paper evaluates the effectiveness of the  method on CIFAR-10/100 and ImageNet-100 datasets.