Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation

1 Introduction . Superior performance on many problems has been achieved by recent machine learning models , especially the ones based on deep learning . While the success comes at the cost of large-scale human annotation , which is prohibitively expensive in practice . For example , modern convolutional neural networks ( CNNs ) can surpass human-level recognition performance on ImageNet after training with over one million labelled images [ 22 ] . On the one hand , it is not possible to annotate all possible classes in the real world , as there are way more classes than the 1 , 000 classes in ImageNet and new classes keep growing over time . On the other hand , annotating specific data , such as the medical data , may require specific expertise , which renders the large-scale annotation extremely difficult , if not impossible . Therefore , it is desired to enable the machine learning systems to deal with unlabelled data automatically . Recently , the problem of novel category discovery was formalized in [ 20 , 18 ] , which aims at discovering new visual categories on unlabelled data by transferring knowledge from labelled data . The labelled data is assumed to contain similar but different categories to those in the unlabelled data . This problem is similar to semi-supervised learning in the sense that both labelled and unlabelled data are used to learn the model . While novel category discovery is much harder because semi-supervised learning assumes that every class contains labelled instances , while for novel category discovery there are no labels available for the new classes in the unlabelled data . This setting is also relevant to ∗ Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) unsupervised clustering . But differently , novel category discovery makes use of the labelled data to extract a specific class prior ( i.e. , the properties that delineate a class ) for partitioning the unlabelled data , while the unsupervised clustering may produce multiple different but equally valid clustering results by adopting different properties ( e.g. , color , shape , pose , lighting , etc ) . In this paper , we introduce a simple and effective two-branch framework for novel category discovery , with one branch focusing on local part-level information and the other branch focusing on overall characteristics . Our contributions are as follows . First , we propose to apply dual ranking statistics for transferring knowledge from the known classes in the labelled data to the unlabelled data , resulting in more robust pseudo label generation for learning on the unlabelled data . We maintain a dynamic object part dictionary and apply part-level ranking statistics on the similarity distribution of each instance over the dictionary to obtain pseudo labels , which are complementary to the pseudo labels obtained by simply examining the ranking statistics of global descriptors . Second , we introduce a mutual knowledge distillation method to allow information exchange and encourage agreement between the local and global branches . The dual ranking statistics provide global and part-level information for learning in two branches separately . The mutual knowledge distillation further allows information exchange between the two branches and makes them benefit from each other . Unlike conventional methods that distill between teacher and student models with known labels , our method distills between two branches of the same model without any manual annotations . Third , we comprehensively evaluate our method on public benchmarks for generic object classification , including CIFAR10 , CIFAR100 , and ImageNet , obtaining state-of-the-art results . Furthermore , we also validate our approach on the more challenging fine-grained datasets CUB-200 , Stanford-Cars , and FGVC-Aircraft , in which the local details are more important to distinguish different classes . Our method outperforms existing methods by a substantial margin , thanks to the ability of our model to subtly exploit both local and global information . Our code can be found at https : //github.com/DTennant/dual-rank-ncd . 2 Related work . Our work is relevant to novel category discovery , knowledge distillation , and part-level feature learning . We briefly review the most relevant work below . Novel category discovery is a relatively new problem setting recently formalized by [ 20 , 18 ] with the task being automatically discovering new object categories in the unlabelled data by making use of a labeled dataset containing different but relevant object classes . The purpose of using the extra labelled data is to learn a category prior to reduce the ambiguity of class definition . This task is closely related to unsupervised clustering and semi-supervised learning , but also significantly different from them . Unsupervised clustering has been studied for decades with many classical approaches [ 35 , 3 , 9 ] and deep learning based solutions [ 50 , 16 , 40 ] being proposed . Due to the lack of a proper class prior , multiple equally valid clustering results can be achieved by different criteria . Therefore , it can not be directly applied to discovery new classes as we expect the model to follow a unique class definition . In semi-supervised learning [ 43 , 39 , 6 , 38 , 46 ] , unlabelled data are used together with labelled data to train a more robust model , with the assumption that all classes in the unlabelled data have labelled instances . However , this is unlikely to be true for real applications where unlabelled data may come from new classes . Thus , novel category discovery is a more realistic setting . The DTC method introduced in [ 20 ] approaches this problem in two steps . The model is firstly trained with supervised learning on the labelled data to capture high-level semantic class information and then trained on the unlabelled data with a clustering loss to discover new categories . In [ 18 , 19 ] , Han et al . proposed to use self-supervision to bootstrap feature learning and introduced ranking statistics on the feature embedding to provide pseudo labels for training on unlabelled data . The KCL [ 25 ] and MCL [ 26 ] methods were designed for general transfer learning across domains and tasks , which can also be applied for novel category discovery . These two methods maintain two models for training and testing , a pretrained binary classifier for pseudo label generation and a clustering model . The binary classifier pretrained on the labelled data is used to provide pseudo labels to train the clustering model on unlabelled data . Concurrent to our work , several methods [ 28 , 58 , 57 , 14 ] are proposed to improve the performance of novel category discovery from different perspectives , showing promising results . The existing methods only consider the global feature descriptors while ignoring the local object parts which are essential to distinguish classes that look similar . In our approach , we jointly consider both and allow information exchange between them for more reliable new category discovery . Knowledge distillation is often employed to learn a compact student model using the knowledge distilled from a larger teacher model by enforcing the agreement of outputs or representations between the two models ( e.g. , [ 24 , 41 , 54 , 52 , 27 , 29 , 1 , 48 ] ) . Apart from distilling a larger teacher model to a smaller student model , self-distillation methods show that distilling between two identical models can also improve the performance [ 53 , 15 , 4 ] . Mutual learning [ 56 ] is a similar technique that trains two models of the same architecture simultaneously but with different initialization and encourages them to learn collaboratively from each other . It has been shown that different initialization leads the model to focus on different regions of the input and thus the trained model can better capture the holistic structure of the data , resulting in better performance . Though effective , the above methods require the labels for training . Thus they can not be applied for novel category discovery . Recently , [ 13 ] introduced a self-supervised distillation method called SEED to improve the representation learning on a small model ( student ) by distilling from a pretrained large model ( teacher ) . The student model is trained to predict the same similarity score distribution inferred by the frozen teacher model over a queue of instances . We draw inspiration from [ 13 ] to design the mutual knowledge distillation module in our method for novel category discovery between the local and global branches . Part-level features have been shown to be effective for tasks involving image verification such as few-shot learning [ 10 , 55 , 12 ] and image retrieval [ 7 , 34 , 5 , 2 , 37 , 42 , 45 ] . In [ 10 ] , the part-level visual concepts are generated by clustering feature vectors from the feature maps obtained using a trained neural network . These visual concepts are used as matching primitives at test time for few-shot learning . Zhang et al . [ 55 ] used the Earth-Mover ’ s Distance ( EMD ) between local part features to measure the distance between two images for matching . Doersch et al . [ 12 ] proposed CrossTransformers to model the cross-attention between each local part of a query image and a set of support images for few-shot learning . Part-level features have been more widely used in the domain of image retrieval with both classical [ 34 , 5 ] and deep learning based [ 2 , 37 , 42 , 45 , 7 ] methods . Intuitively , the part-level features provide better precision for the retrieved results because the local matching is more restricted , and global features yield better recall [ 7 ] . In our work , we leverage both local and global features to enjoy the benefits of both through the dual ranking statistics and the mutual knowledge distillation , for more robust novel category discovery . 3 Method . The goal of novel category discovery is to automatically partition unlabelled instances xui ∈ Du into Cu semantic clusters by transferring the knowledge from the labelled instances ( xli , y l i ) ∈ Dl , where yli ∈ { 1 , . . . , Cl } . The key assumption is that the classes in Dl and Du are relevant though different , so that the properties that constitute a class learned from Dl can be transferred to Du . Following [ 18 ] , we assume Cu is known a priori . When it is unknown , we can employ off-the-shelf methods such as [ 20 ] to get an estimate . To tackle this challenge , we introduce a framework with mutual knowledge distillation between a global branch and a local branch ( see fig . 1 ) . The global branch is designed to capture the overall feature , while the local branch is designed to focus on each individual spatial local part . They have a shared feature extractor fθ . Each branch has a feature projection layer and two linear heads . We denote the feature projection layer as ψ and the two linear heads as ηl and ηu respectively . We also denote ψ followed by the average pooling operation as ψ̄ . Unless stated otherwise , we use subscripts g and p to differentiate feature projection layers and linear heads of global and local branches respectively in the rest of the paper . The two linear heads are responsible for classifying Cl labelled categories and clustering Cu unlabelled categories respectively . To transfer knowledge from the labelled data to the unlabelled data , we adopt global image level and local object part-level ranking statistics on the feature representations . In this way , the model will have reliable pseudo labels for training on the unlabelled data . Moreover , we also incorporate mutual knowledge distillation between the two branches to enforce global and local agreement , leading to more reliable novel category discovery . Next , we will introduce each component of our framework in more detail . In section 3.1 , we first introduce the our knowledge transfer method using dual ranking statistics . In section 3.2 , we describe our mutual knowledge distillation method for novel category discovery . Lastly , we summarize the overall training loss in section 3.3 .

Summary: The paper tackles novel class discovery of unlabeled data via transfer learning-based two-brach learning with pairwise pseudo-labels by two different criteria: similarity on global descriptor and local features. The distinct criteria generate more robust pseudo labels than the single ones and alleviate the negative impact from false pseudo-labels in a complementary manner. The proposed mutual knowledge distillation further facilitates effective mutual learning between two branches. Extensive experiments on standard classification datasets and more challenging fine-grained datasets show that their method has significant advantages over SOTA owing to robust pseudo-labels by dual rank statistics.