Meta Discovery: Learning to Discover Novel Classes given Very Limited Data

In learning to discover novel classes ( L2DNC ) , we are given labeled data from seen classes and unlabeled data from unseen classes , and we train clustering models for the unseen classes . However , the implicit assumptions behind L2DNC are still unclear . In this paper , we demystify assumptions behind L2DNC and find that high-level semantic features should be shared among the seen and unseen classes . Based on this finding , L2DNC is theoretically solvable under certain assumptions and can be naturally linked to meta-learning that has exactly the same assumption as L2DNC . Thus , we can empirically solve the L2DNC problem by meta-learning algorithms after slight modifications . This meta-learning-based methodology significantly reduces the amount of unlabeled data needed for training and makes it more practical , as demonstrated in experiments . The use of very limited data is also justified by the application scenario of L2DNC : since it is unnatural to label only seen-class data , L2DNC is sampling instead of labeling in causality . Therefore , unseen-class data should be collected on the way of collecting seen-class data , which is why they are novel and first need to be clustered . 1 INTRODUCTION . With the development of high-performance computing , we can train deep networks to achieve various tasks well ( Deng et al. , 2009 ; Song et al. , 2019 ; Jing et al. , 2020 ; Song et al. , 2020 ; Liu et al. , 2020b ) . However , the trained networks can only recognize the classes seen in the training set ( i.e. , known/seen classes ) , and can not identify and cluster novel classes ( i.e. , unseen classes ) like human beings . A prime example is that human can easily tell a novel animal category ( e.g. , okapi ) after learning a few seen animal categories ( e.g. , horse and dog ) . Namely , human can effortlessly discover ( cluster ) novel categories of animals . Inspired by this fact , previous works formulated a novel problem called learning to discover novel classes ( L2DNC ) ( Hsu et al. , 2018 ; Han et al. , 2019 ) , where we train a clustering model using plenty of unlabeled novel-class and labeled known-class data . However , if L2DNC is labeling in causality ( X → Y ) , there exists two issues : L2DNC might not be a theoretically solvable problem . For example , if novel classes are completely different from known classes , then it is unrealistic to use the known classes ( like animals ) to help precisely cluster novel classes ( like cars , Figure 1 ( a ) ) . Moreover , L2DNC might not be a realistic problem in some scenarios where novel classes might only be seen once or twice . This does not satisfy the assumptions considered in existing L2DNC methods . These issues naturally motivate us to find out when L2DNC can be theoretically solved and what assumptions are considered behind L2DNC . In this paper , we revisit L2DNC and find that L2DNC will be a well-defined problem if L2DNC is sampling in causality ( Y → X ) , i.e. , novel and known classes are sampled together ( Figure 1 ( b ) ) . In this sampling process , data are often obtained because of a given purpose , and novel classes and known classes are obtained in the same scenario . For instance , botanists sample plant specimens for research purposes in the forests . Except for the plants they are interested in ( i.e. , known classes ) , they also find scarce plants never seen before ( i.e. , finding novel classes ( Smith , 1874 ) ) . Since a trip to forests is relatively costly and toilsome , botanists had better sampled these scarce plants passingly for future research . From this example , it can be seen that botanists will have plenty of labeled data with known classes , and few unlabeled data with novel classes . Since both data are sampled together from the same scenario ( e.g. , plants in the forests ) , it is reasonable to leverage knowledge of known classes to help cluster novel classes , which is like “ discovering novel categories ” happened in our daily life . Therefore , we argue that the key assumption behind L2DNC is that , known classes and novel classes share high-level semantic features . For example , known classes and novel classes are different plants but all of them have the leaf , the stem , and the roots ( Walters , 1977 ) . Then , we reformulate the L2DNC problem and show that L2DNC is theoretically solvable under mild assumptions ( including the key assumption above ) . We also show an impossibility theorem for previous L2DNC setting . This theorem shows that it might not be necessary to introduce known-class data to help cluster novel-class data if known classes and novel classes do not share high-level semantic features . Namely , L2DNC might be an ill-defined problem if known and novel class do not share high-level semantic features . Although L2DNC is theoretically solvable under mild assumptions , we still need abundant data from known and novel classes to ensure L2DNC can be empirically solved . However , as we mentioned previously , the novel classes might only be seen once or twice in some scenarios . In such scenarios , we find that previous L2DNC methods do not work well ( Figure 2 ) . To address the L2DNC given very limited data ( L2DNCL ) , we link L2DNC to meta-learning that also assumes that known and unknown ( novel ) classes share the high-level semantic features ( Chen et al. , 2020a ; Maurer , 2005 ) . The key difference between meta-learning and L2DNCL lies in their inner-tasks . In meta-learning , the inner-task is a classification task , while in L2DNCL , it is a clustering task . Thus , we can modify the training strategies of the inner-tasks of meta-learning methods such that they can discover novel classes , i.e. , meta discovery . Specifically , we first propose a novel method to sample training tasks for meta-learning methods . In sampled tasks , labeled and unlabeled data share high-level semantic features and the same clustering rule ( Figure 3 ) . Then , based on this novel sampling method , we realize meta discovery using two representative meta-learning methods : model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) and prototypical network ( ProtoNet ) ( Snell et al. , 2017 ) . Figure 2 demonstrates two realizations of meta discovery ( i.e. , meta discovery with MAML ( MM ) and meta discovery with ProtoNet ( MP ) ) can perform much better than existing methods in L2DNCL . We conduct experiments on four benchmarks and compare our method with five competitive baselines ( MacQueen et al. , 1967 ; Hsu et al. , 2018 ; 2019 ; Han et al. , 2019 ; 2020 ) . Empirical results show that our method outperforms these baselines significantly when novel-class data are very limited . Moreover , we provide a new practical application to the prosperous meta-learning community ( Chen et al. , 2020a ) , which lights up a novel road for L2DNCL . 2 RELATED WORK . Our proposal is mainly related to L2DNC and meta-learning that are briefly reviewed here . More detailed reviews can be found in Appendix A . 0 20 40 60 80 100 Epoch 25 30 35 40 45 50 55 60 A cc ( % ) 0 20 40 60 80 100 Epoch 25 30 35 40 45 50 55 60 A cc ( % ) 0 20 40 60 80 100 Epoch 20 30 40 50 60 A cc ( % ) 0 20 40 60 80 100 Epoch 20 30 40 50 60 A cc ( % ) MP MM RS KCL MCL DTC 0 50 100 Epoch 30 40 50 60 A cc ( % ) CIFAR-10 ( a ) 5-way 5-observation 0 50 100 Epoch 30 40 50 60 A cc ( % ) CIFAR-10 ( b ) 5-way 1-observation 0 50 100 Epoch 20 30 40 50 60 A cc ( % ) SVHN ( c ) 5-way 5-observation 0 50 100 Epoch 20 30 40 50 60 A cc ( % ) SVHN ( d ) 5-way 1-observation Figure 2 : We conducted experiments on CIFAR-10 and SVHN and reported the average clustering accuracy ( ACC ( % ) , Section 5 ) when using existing and our methods to address the L2DNCL . We can observe that existing methods can not address the L2DNCL , while our methods ( MM and MP ) can address the L2DNCL well . Learning to discover novel classes . L2DNC was proposed in recent years , aiming to cluster unlabeled novel-class data according to underlying categories . Compared to unsupervised learning ( Barlow , 1989 ) , L2DNC also requires labeled known-class data to help cluster novel-class data . The pioneering methods include the KL-Divergence-based contrastive loss ( KCL ) ( Hsu et al. , 2018 ) , the meta classification likelihood ( MCL ) ( Hsu et al. , 2019 ) , deep transfer clustering ( DTC ) ( Han et al. , 2019 ) , and the rank statistics ( RS ) ( Han et al. , 2020 ) . Compared to existing works ( Hsu et al. , 2018 ; Zhong et al. , 2021b ; a ) , we focus on clustering unlabeled data when their quantity is very limited . Meta-learning . Meta-learning is also known as learning-to-learn , which trains a meta-model over a wide variety of learning tasks ( Yao et al. , 2021 ; Wei et al. , 2021 ; Ravi & Larochelle , 2017 ; Chen et al. , 2020b ) . In meta-learning , we often assume that data share the same high-level features , which ensures that meta-learning can be theoretically addressed ( Maurer , 2005 ) . According to Hospedales et al . ( 2020 ) , there are three common approaches to meta-learning : optimization-based ( Finn et al. , 2017 ) , model-based ( Santoro et al. , 2016 ) , and metric-based ( Snell et al. , 2017 ) . In ( Jiang & Verma , 2019 ) , researchers trained a recurrent model that learns how to cluster given multiple types of training datasets . Compared to our meta discovery , Jiang & Verma ( 2019 ) learn a clustering model with multiple unlabeled datasets , while meta discovery aims to learn a clustering model with labeled known data and ( abundant or few ) unlabeled novel data from the same dataset . 3 ASSUMPTIONS BEHIND L2DNC AND ANALYSIS OF SOLVABILITY . Since L2DNC focuses on clustering novel-class data , we first show definitions regarding the separation of a random variable ( r.v . ) X ∼ PX defined on X ⊂ Rd . Then , we give a formal definition of L2DNC and introduce assumptions behind L2DNC . Finally , we present one theorem to show L2DNC is solvable in theory and one theorem to show a failure situation where previous setting encounters . The proofs of both theorems can be seen in Appendix B . Definition 1 ( K- -Separable r.v. ) . Given the r.v . X ∼ PX , X is K- -separable with a non-empty function set F = { f : X → I } if ∀f ∈ F τ ( X , f ( X ) ) : = max i , j∈I , i6=j PX ( RX|f ( X ) =i ∩RX|f ( X ) =j ) = , ( 1 ) where I = { i1 , . . . , iK } is an index set , f ( X ) is an induced r.v . whose source of randomness is X exclusively , and RX|f ( X ) =i = supp ( PX|f ( X ) =i ) is the support set of PX|f ( X ) =i . In Eq . ( 1 ) , τ ( · , · ) represents the largest overlap between any different clusters in the sense of the probability measure PX . If = 0 , then we can perfectly partition observations of X into K clusters using some distance-based clustering algorithm , e.g. , K-means ( MacQueen et al. , 1967 ) . However , when X is a complex space and the dimension d is much larger than the intrinsic dimension of X ( e.g. , images ) , it is not reliable to measure the distance between observations from X using original features ( Liu et al. , 2020b ) , which will result in poor clustering performance . Non-linear transformation . To overcome issues caused by the complex space , researchers suggest apply a non-linear transformation to extract high-level features of X ( Fang et al. , 2020 ; Liu et al. , 2020b ) . Based on these features , we can measure the distance between two observations well ( Liu et al. , 2020b ) . Let π : X → Rdr be a transformation where dr is the reduced dimension and dr d , and we expect that the transformed r.v . π ( X ) can be K- -separable as well . Namely , we hope the following function set exists . Definition 2 ( Consistent K- -separable Transformation Set ) . Given r.v . X that is K- -separable with a function set F , a transformed r.v . π ( X ) is K- -separable with F if ∀f ∈ F , τ ( π ( X ) , f ( X ) ) : = max i , j∈I , i6=j Pπ ( X ) ( Rπ ( X ) |f ( X ) =i ∩Rπ ( X ) |f ( X ) =j ) = , ( 2 ) where π : X → Rdr is a transformation . Then , a consistent K- -separable transformation set is a non-empty set Π satisfying that ∀f ∈ F , ∀π ∈ Π , τ ( π ( X ) , f ( X ) ) = . Remark 1 . If dr d and = 0 , we will need much less observations to estimate the density of π ( X ) compared to X , thus it will be much easier to perfectly separate π ( X ) than X . For example , there probably exists a linear function g : Rdr → I such that f ( X ) = g ◦ π ( X ) . It is clear that such linear function g is easier to find than directly finding f using K-means . Problem Setup of L2DNC . Based on the above definitions , we will formally define the L2DNC problem below . In L2DNC , we have two r.v.s X l , Xu defined on X , the ground-truth labeling function f l : X → Y for X l and a function set F = { f : X → I } , where Y = { il1 , . . . , ilK l } and I = { iu1 , . . . , iuKu } . Based on Definitions 1 and 2 , we have the following assumptions in L2DNC . ( A ) The support set of X l and the support set of Xu are disjoint , and underlying classes of X l are different from those of Xu ( i.e. , I ∩ Y = ∅ ) ; ( B ) X l is K l- l-separable with F l = { f l } and Xu is Ku- u-separable with Fu , where l = τ ( X l , f l ( X l ) ) < 1 and u = minf∈F τ ( Xu , f ( Xu ) ) < 1 ; ( C ) There exist a consistent K l- l-separable transformation set Πl for X l and a consistent Ku- u-separable transformation set Πu for Xu ; ( D ) Πl ∩Πu 6= ∅ . ( A ) ensures that known and novel classes are disjoint . ( B ) implies that it is meaningful to separate observations from X l and Xu . ( C ) means that we can find good high-level features for X l or Xu . Based on these features , it is much easier to separate X l or Xu . ( D ) says that the high-level features of X l and Xu are shared , as demonstrated in the introduction . Then , we can define L2DNC formally . Problem 1 ( L2DNC ) . Given X l , Xu and f l defined above and assume ( A ) - ( D ) hold , in L2DNC , we aim to learn a function π̂ : X → Rdr via minimizingJ ( π̂ ) = τ ( π̂ ( X l ) , f l ( X l ) ) +τ ( π̂ ( Xu ) , f u ( Xu ) ) , where f u ∈ F and dr d. We expect that π̂ ( Xu ) is Ku- u-separable . Theorem 1 ( L2DNC is Theoretically Solvable ) . Given X l , Xu and f l defined above and assume ( A ) - ( D ) hold , then π̂ ( Xu ) is Ku- u-separable . If u = 0 , then L2DNC is theoretically solvable . Theorem 1 means that it is possible to find a good transformation π̂ such that π̂ ( Xu ) is separable although we introduce X l1 . Then we show that L2DNC might be ill-defined if ( D ) does not hold . Theorem 2 ( Impossibility Theorem ) . Given X l , Xu and f l defined above and assume ( A ) - ( C ) hold , if maxπ∈Πl τ ( π ( Xu ) , f u ( Xu ) ) < minπ∈Π−Πl τ ( π ( X l ) , f l ( X l ) ) , ( D ) does not hold and l ≤ u , then τ ( π̂ ( Xu ) , f u ( Xu ) ) > u , where Π = { π : X → Rdr } and f u ∈ F . Remark 2 . Condition maxπ∈Πl τ ( π ( Xu ) , f u ( Xu ) ) < minπ∈Π−Πl τ ( π ( X l ) , f l ( X l ) ) indicates that the worst case of clustering novel classes with transformations that are suitable for known classes is better than the best case of clustering known classes with transformations that are not suitable for known classes . Besides , l ≤ u indicates that the transformations in X l is more separable than Xu . In previous setting , data acquiring process is unclear . As we discussed , if L2DNC is labelling in causality , then novel and known classes might not share high-level semantic features ( Figure 1 ( a ) ) . Namely , ( D ) might not hold , resulting in that X l might bring negative effects under some conditions ( Theorem 2 ) . Thus , based on Theorems 1 and 2 , it is clear that ( D ) plays a key role in L2DNC , which also justifies that L2DNC will be well-defined if it is sampling in causality . Noting that sampling in causality ( Y → X ) is the sufficient unnecessary condition of ( D ) . For example , if there are some constraints to make the novel-class data ( to be annotated ) are obtained in the same scenario with known-class data , then data generated by labeling ( X → Y ) also satisfy ( D ) . 1Note that , the objective of clustering problem is different from that of L2DNC . In clustering problem , we aim to find π̂ via minimizing τ ( π̂ ( Xu ) , f u ( Xu ) ) rather than τ ( π̂ ( X l ) , f l ( X l ) ) + τ ( π̂ ( Xu ) , f u ( Xu ) ) . Based on Theorem 1 , if there are abundant observations to estimate J ( π̂ ) , then we can find the optimal transformation π̂ to help obtain a good partition for observations from Xu . However , as discussed before , the novel classes might only be seen once or twice in some scenarios . In such scenarios , we find that previous L2DNC methods do not work well empirically ( Figure 2 ) . To address the L2DNC given very limited data ( L2DNCL ) , we link L2DNC to meta-learning that also assumes that known and unknown ( novel ) classes share the high-level semantic features ( Chen et al. , 2020a ) , which is exactly the same as ( D ) in L2DNC . Thus , it is natural to address L2DNCL by meta-learning .

Summary: Learning to discover novel class is a very challenging task and a new research topic in recent years. In this task, a known-class dataset can be used to help cluster novel classes. “Novel” means that there are no overlaps between novel classes and known classes. This setting looks ill-defined since it is not clear why we need known classes to help cluster novel classes. However, this paper answers this question based on a novel concept: K-epsilon separation and points out when this setting is ill-defined/well-defined. Based on this contribution, this paper is above the acceptance borderline.   Nevertheless, the presentation should be polished. Although I understand the setting and the contribution in the end, the presentation flow is not smooth. Some typos make me struggle when I read this paper. Besides, the connection between sampling process and Assumption (D) should be discussed deeply. I find that they might not the same thing.