Automatically Discovering and Learning New Visual Categories with Ranking Statistics

1 INTRODUCTION . Modern machine learning systems can match or surpass human-level performance in tasks such as image classification ( Deng et al. , 2009 ) , but at the cost of collecting large quantities of annotated training data . Semi-supervised learning ( SSL ) ( Oliver et al. , 2018 ) can alleviate this issue by mixing labelled with unlabelled data , which is usually much cheaper to obtain . However , these methods still require some annotations for each of the classes that one wishes to learn . We argue this is not always possible in real applications . For instance , consider the task of recognizing products in supermarkets . Thousands of new products are introduced in stores every week , and it would be very expensive to annotate them all . However , new products do not differ drastically from the existing ones , so it should be possible to discover them automatically as they arise in the data . Unfortunately , machines are still unable to effectively learn new classes without manual annotations . In this paper , we thus consider the problem of discovering new visual classes automatically , assuming that a certain number of classes are already known by the model ( Hsu et al. , 2018 ; 2019 ; Han et al. , 2019 ) . This knowledge comes in the form of a labelled dataset of images for a certain set of classes . Given that this data is labelled , off-the-shelf supervised learning techniques can be used to train a very effective classifier for the known classes , particularly if Convolutional Neural Networks ( CNNs ) are employed . However , this does not mean that the learned features are useful as a representation of the new classes . Furthermore , even if the representation transfers well , one still has the problem of identifying the new classes in an unlabelled dataset , which is a clustering problem . We tackle these problems by introducing a novel approach that combines three key ideas ( section 2 and fig . 1 ) . The first idea is to pre-train the image representation ( a CNN ) using all available images , both labelled and unlabelled , using a self-supervised learning objective . Crucially , this objective does not leverage the known labels , resulting in features that are much less biased towards the labelled ∗indicates equal contribution classes . Labels are used only after pre-training to learn a classifier specific to the labelled data as well as to fine-tune the deepest layers of the CNN , for which self-supervision is not as effective . The second idea is a new approach to transfer the information contained in the labelled images to the problem of clustering the unlabelled ones . Information is transferred by sharing the same representation between labelled and unlabelled images , motivated by the fact that the new classes are often similar to the known ones . In more detail , pairs of unlabelled images are compared via their representation vectors . The comparison is done using robust rank statistics , by testing if two images share the same subset of k maximally activated representation components . This test is used to decide if two unlabelled images belong to the same ( new ) class or not , generating a set of noisy pairwise pseudo-labels . The pseudo-labels are then used to learn a similarity function for the unlabelled images . The third and final idea is , after bootstrapping the representation , to optimise the model by minimizing a joint objective function , containing terms for both the labelled and unlabelled subsets , using respectively the given labels and the generated pseudo-labels , thus avoiding the forgetting issue that may arise with a sequential approach . A further boost is obtained by incorporating incremental learning of the discovered classes in the classification task , which allows information to flow between the labelled and unlabelled images . We evaluate our method on several public benchmarks ( section 3 ) , outperforming by a large margin all existing techniques ( section 4 ) that can be applied to this problem , demonstrating the effectiveness of our approach . We conclude the paper by summarizing our findings ( section 5 ) . Our code can be found at http : //www.robots.ox.ac.uk/˜vgg/research/auto_novel . 2 METHOD . Given an unlabelled dataset Du = { xui , i = 1 , . . . , M } of images xui ∈ R3×H×W , our goal is to automatically cluster the images into a number of classes Cu , which we assume to be known a priori . We also assume to have a second labelled image dataset Dl = { ( xli , yli ) , i = 1 , . . . , N } where yli ∈ { 1 , . . . , Cl } is the class label for image xli . We also assume that the set of Cl labelled classes is disjoint from the set of Cu unlabelled ones . While the statistics of Dl and Du thus differ , we hypothesize that a general notion of what constitutes a “ good class ” can be extracted from Dl and that the latter can be used to better cluster Du . We approach the problem by learning an image representation Φ : x 7→ Φ ( x ) ∈ Rd in the form of a CNN . The goal of the representation is to help to recognize the known classes and to discover the new ones . In order to learn this representation , we combine three ideas , detailed in the next three sections . 2.1 SELF-SUPERVISED LEARNING . Given that we have a certain number of labelled images Dl at our disposal , the obvious idea is to use these labels to bootstrap the representation Φ by minimizing a standard supervised objective such as the cross-entropy loss . However , experiments show that this causes the representation to overly-specialize for the classes in Dl , providing a poor representation of the new classes in Du . Thus we resist the temptation of using the labels right away and use instead a self-supervised learning method to bootstrap the representation Φ. Self-supervised learning has been shown ( Kolesnikov et al. , 2019 ; Gidaris et al. , 2018 ) to produce robust low-level features , especially for the first few layers of typical CNNs . It has the benefit that no data annotations are needed , and thus it can be applied to both labelled and unlabelled images during training . In this way , we achieve the key benefit of ensuring that the representation is initialized without being biased towards the labelled data . In detail , we first pre-train our model Φ with self-supervision on the union of Dl and Du ( ignoring all labels ) . We use the RotNet ( Gidaris et al. , 2018 ) approach1 due to its simplicity and efficacy , but any self-supervised method could be used instead . We then extend the pre-trained network Φ with a classification head ηl : Rd → RCl implemented as a single linear layer followed by a softmax layer . The function ηl ◦ Φ is fine-tuned on the labelled dataset Dl in order to learn a classifier for the Cl known classes , this time using the labels yi and optimizing the standard cross-entropy ( CE ) loss : LCE = − 1 N N∑ i=1 log ηlyi ( z l i ) ( 1 ) where zli = Φ ( x l i ) ∈ Rd is the representation of image xli . Only ηl and the last macro-block of Φ ( section 3 ) are updated in order to avoid overfitting the representation to the labelled data . 2.2 TRANSFER LEARNING VIA RANK STATISTICS . Once the representation Φ and the classifier ηl have been trained , we are ready to look for the new classes in Du . Since the classes in Du are unknown , we represent them by defining a relation among pairs of unlabelled images ( xui , x u j ) . The idea is that similar images should belong to the same ( new ) class , which we denote by the symbol sij = 1 , while dissimilar ones should not , which we denote by sij = 0 . The problem is then to obtain the labels sij . Our assumption is that the new classes will have some degree of visual similarity with the known ones . Hence , the learned representation should be applicable to old and new classes equally well . As a consequence , we expect the descriptors zui = Φ ( x u i ) and z u j = Φ ( x u j ) of two images x u i , x u j from the new classes to be close if they are from the same ( new ) class , and to be distinct otherwise . Rather than comparing vectors zui , z u j directly ( e.g. , by a scalar product ) , however , we use a more robust rank statistics . Specifically , we rank the values in vector zui by magnitude . Then , if the rankings obtained for two unlabelled images xui and x u j are the same , they are very likely to belong to the same ( new ) class , so we set sij = 1 . Otherwise , we set sij = 0 . In practice , it is too strict to require the two rankings to be identical if the dimension of zui is high ( otherwise we may end up with sij = 0 for all pairs ( i , j ) , i 6= j ) . Therefore , we relax this requirement by only testing if the sets of the top-k ranked dimensions are the same ( we use k = 5 in our experiments ) , i.e . : sij = 1 { topk ( Φ ( x u i ) ) = topk ( Φ ( x u j ) ) } , ( 2 ) where topk : Rd → P ( { 1 , . . . , d } ) associates to a vector z the subset of indices { 1 , . . . , d } of its top-k elements . Once the labels sij have been obtained , we use them as pseudo-labels to train a comparison function for the unlabelled data . In order to do this , we apply a new head ηu : Rd → RCu to the image representation zui = Φ ( x u i ) to extract a new descriptor vector η u ( zui ) optimized for the unlabelled data . As in section 2.1 , the head is composed of a linear layer followed by a softmax . Then , the inner product ηu ( zui ) > ηu ( zuj ) is used as a score for whether images x u i and x u j belong to the same class or not . Note that ηu ( zui ) is a normalized vector due to the softmax layer in η u . This descriptor is trained by optimizing the binary cross-entropy ( BCE ) loss : LBCE = − 1 M2 M∑ i=1 M∑ j=1 [ sij log η u ( zui ) > ηu ( zuj ) + ( 1− sij ) log ( 1− ηu ( zui ) > ηu ( zuj ) ) ] . ( 3 ) 1 We present to the network Φ randomly-rotated versions Rx of each image and task it with predicting R. The problem is formulated as a 4-way classification of the rotation angle , with angle in { 0◦ , 90◦ , 180◦ , 270◦ } . The model η ◦ Φ ( Rx ) is terminated by a single linear layer η with 4 outputs each scoring an hypothesis . The parameters of η and Φ are optimized by minimizing the cross-entropy loss on the rotation prediction . Furthermore , we structure ηu in a particular manner : We set its output dimension to be equal to the number of new classes Cu . In this manner , we can use the index of the maximum element of each vector ŷui = argmaxy [ η u ◦ Φ ( xui ) ] y as prediction ŷui for the class of image xui ( as opposed to assigning labels via a clustering method such as k-means ) .

Summary: This paper addresses the problem of clustering unseen classes. To learn a robust feature extractor, this paper proposes a multi-stage training framework, which leverages different supervised manners in each stage. Specifically, they initialize the network using the self-supervised learning on the union of all available data and then further finetune it using labelled data. Based on this, they propose the rank statistics which leverages the activation knowledge on labelled classes and rank the activated dimensions. Unseen data having similar rank results are clustered to obtain the initial pseudo labels. Finally, the network is jointly optimized with the ground-truth and generated pseudo labels (the pseudo ones will be updated during training). Extensive experiments on 5 datasets show that their method has significant advantages over SOTA owing to the learned robust feature extractor.