Open-world Semi-supervised Learning

1 INTRODUCTION . With the advent of deep learning , remarkable breakthroughs have been achieved and current machine learning systems excel on tasks with large quantities of labeled data ( Hinton et al. , 2012 ; LeCun et al. , 2015 ; Silver et al. , 2016 ; Esteva et al. , 2017 ) . Despite the strengths , the vast majority of models are designed for the closed-world setting rooted in the assumption that training and test data come from the same set of predefined classes ( Bendale & Boult , 2015 ; Boult et al. , 2019 ) . This assumption , however , rarely holds for data in-the-wild , as labeling data depends on having the complete knowledge of a given domain , which is rarely the case in practice . For example , biologists may prelabel some of the known cell types ( seen classes ) , and then want to train and apply the model to a new tissue to identify known cell types but also to discover novel previously unknown cell types ( unseen classes ) . Similarly , in social networks one may want to classify users into predefined interest groups while also discovering new unknown/unlabeled interests of users . Thus , in contrast to the commonly assumed closed world , many real-world problems are inherently open-world—new classes can emerge in the test data that may have never been seen ( and labeled ) during training . Here we introduce open-world semi-supervised learning ( open-world SSL ) setting that generalizes semi-supervised learning and novel class discovery . Under open-world SSL , we are given a labeled training dataset as well as an unlabeled dataset . The labeled dataset contains instances that belong to a set of seen classes , while instances in the unlabeled/test dataset belong to both the seen classes as well as to an unknown number of unseen classes ( Figure 1 ) . Under this setting , the model then needs to simultaneously classify instances into previously seen classes and then also discover new classes and assign instances to them . In other words , open-world SSL is a transductive learning setting under class distribution mismatch in which unlabeled test set may contain classes that have never been labeled during training , i.e. , are not part of the labeled training set . Given unlabeled test set , the model needs to either assign instances to one of the classes previously seen in the labeled set , or form a novel class and assign instances to it . Open-world SSL is fundamentally different but closely related to two recent lines of work : robust semi-supervised learning ( SSL ) and novel class discovery . Robust SSL ( Oliver et al. , 2018 ; Guo et al. , 2020 ; Chen et al. , 2020b ; Guo et al. , 2020 ; Yu et al. , 2020 ) assumes class distribution mismatch between labeled and unlabeled data , but in this setting the model only needs to be able to recognize ( reject ) instances belonging to novel classes in the unlabeled data as out-of-distribution instances . In contrast , instead of rejecting instances belonging to novel classes , open-world SSL aims at discovering individual novel classes and then assigning instances to them . Novel class discovery ( Hsu et al. , 2018 ; 2019 ; Han et al. , 2019 ; 2020 ; Zhong et al. , 2021 ) is a clustering problem where one assumes unlabeled data is composed only of novel classes . In contrast , open-world SSL is more general as instances in the unlabeled datan can come from seen as well as from novel classes . To apply robust SSL and novel class discovery methods to open-world SSL , one could in principle adopt a multi-step approach , where one could first use robust SSL to first reject examples from novel classes and then apply a novel class discovery method on rejected instances to discover novel classes . An alternative would be that one could treat all classes as “ novel ” , apply novel class discovery methods and then match some of the classes back to the seen classes in the labeled dataset . However , our experiments show that such ad hoc approaches do not perform well in practice . Therefore , it is necessary to design a method that can solve this practical problem in an end-to-end framework . In this paper we propose ORCA ( Open-woRld with unCertainty based Adaptive margin ) that operates under the novel open-world SSL setting . ORCA effectively assigns examples from the unlabeled data to either previously seen classes , or forms a novel classes by grouping similar instances . ORCA is an end-to-end deep learning framework , where the key to our approach is a novel uncertainty adaptive margin mechanism that gradually decreases plasticity and increases discriminability of the model during training . This mechanism effectively reduces an undesired gap between intra-class variance of seen with respect to the novel classes caused by learning seen classes faster than the novel , which we show is a critical difficulty in this setting . We then develop a special model training procedure that learns to classify data points into a set of previously seen classes while also learning to use an additional classification head for each newly discovered class . Classification heads for seen classes are used to assign the unlabeled examples to classes from the labeled set , while activating additional classification heads allows ORCA to form a novel class . ORCA does not need to know the number of novel classes ahead of time and can automatically discover them at the deployment time . We evaluate ORCA on three benchmark image classification datasets adapted for open-world SSL and a single-cell dataset from biology domain . Since no existing methods can operate under the open-world SSL setting we first extend existing state-of-the-art SSL , open-set recognition and novel class discovery methods to the open-world SSL setting and then compare them to ORCA . Experimental results demonstrate that ORCA effectively addresses the challenges of open-world SSL and consistently outperforms all baselines by a large margin . Specifically , ORCA achieves 25 % and 96 % improvements on seen and novel classes of the ImageNet dataset . Moreover , we show that ORCA is robust to unknown number of novel classes , different distributions of seen and novel classes , unbalanced data distributions , pretraining strategies and a small number of labeled examples . 2 RELATED WORK . We summarize similarities and differences between open-world SSL and related settings . Additional related work is given in Appendix A . Novel class discovery . In novel class discovery ( Hsu et al. , 2018 ; Han et al. , 2020 ; Brbic et al. , 2020 ; Zhong et al. , 2021 ) , the task is to cluster unlabeled dataset consisting of similar , but completely disjoint , classes than those present in the labeled dataset which is utilized to learn better representation for clustering . These methods assume that at the test time all the classes are novel . While these methods are able to discover novel classes , they do not recognize the seen/known classes . On the contrary , our open-world SSL is more general because unlabeled test set consists of novel classes but also classes previously seen in the labeled data that need to be identified . In principle , one could extend novel class discovery methods by treating all classes as “ novel ” at test time and then matching some of them to the known classes from the labeled dataset . We adopt such approaches as our baselines , but our experiments show that they do not perform well in practice . Semi-supervised learning ( SSL ) . SSL methods ( Chapelle et al. , 2009 ; Kingma et al. , 2014 ; Laine & Aila , 2017 ; Zhai et al. , 2019 ; Lee , 2013 ; Xie et al. , 2020 ; Berthelot et al. , 2019 ; 2020 ; Sohn et al. , 2020 ) assume closed-world setting in which labeled and unlabeled data come from the same set of classes . Robust SSL methods ( Oliver et al. , 2018 ; Chen et al. , 2020b ; Guo et al. , 2020 ; Yu et al. , 2020 ) relax the SSL assumption by assuming that instances from novel classes may appear in the unlabeled test set . The goal in robust SSL is to reject instances from novel classes which are treated as out-of-distribution instances . Instead of rejecting instances from novel classes , in open-world SSL the goal is to discover individual novel classes and then assign datapoints to them . To extend robust SSL to open-world SSL , one could take discarded points and then apply clustering/novel class discovery . Early work ( Miller & Browning , 2003 ) considered solving the problem in such a way using extension of the EM algorithm . However , our experiments show that by discarding the instances the embedding learned by these methods does not allow for accurate discovery of novel classes . Open-set and open-world recognition . Open-set recognition ( Scheirer et al. , 2012 ; Geng et al. , 2020 ; Bendale & Boult , 2016 ; Ge et al. , 2017 ; Sun et al. , 2020a ) considers the inductive setting in which novel classes can appear during testing , and the model needs to reject instances from novel classes . To extend these methods to open-world setting , we include a baseline that discovers classes on rejected instances . However , results show that such approaches can not effectively address the challenges of open-world SSL . Similarly , open-world recognition approaches ( Bendale & Boult , 2015 ; Rudd et al. , 2017 ; Boult et al. , 2019 ) require the system to incrementally learn and extend the set of known classes with novel classes . These methods incrementally label novel classes by human-in-the-loop . In contrast , open-world SSL leverages unlabeled data in the learning stage and does not require human-in-the-loop . Generalized zero-shot learning ( GZSL ) . Like open-world SSL , GZSL ( Xian et al. , 2017 ; Liu et al. , 2018 ; Chao et al. , 2016 ) assumes that classes seen in the labeled set and novel classes are present at the test time . However , GZSL imposes additional assumption about availability of prior knowledge given as auxiliary attributes that uniquely describe each individual class including the novel classes . This restrictive assumption severely limits the application of GZSL methods in practice . In contrast , open-world SSL is more general as it does not assume any prior information about classes . 3 PROPOSED APPROACH . In this section , we first define the open-world SSL setting . We follow by an overview of ORCA framework and then introduce each of the components of our framework in details . 3.1 OPEN-WORLD SEMI-SUPERVISED LEARNING SETTING . In the open-world SSL , we assume transductive learning setting in which a labeled part of the dataset Dl = { ( xi , yi ) } mi=1 and an unlabeled part of the dataset Du = { ( xi ) } ni=1 are given at the input . We denote the set of classes seen in the labeled data as Cl and set of classes in the unlabeled test data as Cu , respectively . In the open-world SSL , we assume category/class shift , i.e. , Cl ∩ Cu 6= ∅ and Cl 6= Cu . We consider Cs = Cl ∩ Cu as a set of seen classes , and Cn = Cu\Cl as a set of novel classes . Definition 1 ( Open-world SSL ) . In the open-world SSL , the model needs to assign instances from Du either to previously seen classes Cs , or form a novel class c ∈ Cn and assign datapoints to it . Note that open-world SSL generalizes novel class discovery and traditional ( closed-world ) SSL . Novel class discovery assumes that classes in labeled and unlabeled data are disjoint , i.e. , Cl∩Cu = ∅ , while ( closed-world ) SSL assumes the same classes in labeled and unlabeled data , i.e. , Cl = Cu .

Summary: The authors propose a method to tackle a new problem setting of semi-supervised learning, called an open-world semi-supervised learning, where the model is required to accurately discriminate known-class data as well as to appropriately discover unknown classes contained in an unlabeled dataset. The objective function to be minimized in the proposed method comprises three terms: unsupervised loss, supervised loss with uncertainty based adaptive margin, and entropy regularization. The experimental results with several datasets have validated the advantage of the proposed method.