Automatically Discovering and Learning New Visual Categories with Ranking Statistics

1 INTRODUCTION . Modern machine learning systems can match or surpass human-level performance in tasks such as image classification ( Deng et al. , 2009 ) , but at the cost of collecting large quantities of annotated training data . Semi-supervised learning ( SSL ) ( Oliver et al. , 2018 ) can alleviate this issue by mixing labelled with unlabelled data , which is usually much cheaper to obtain . However , these methods still require some annotations for each of the classes that one wishes to learn . We argue this is not always possible in real applications . For instance , consider the task of recognizing products in supermarkets . Thousands of new products are introduced in stores every week , and it would be very expensive to annotate them all . However , new products do not differ drastically from the existing ones , so it should be possible to discover them automatically as they arise in the data . Unfortunately , machines are still unable to effectively learn new classes without manual annotations . In this paper , we thus consider the problem of discovering new visual classes automatically , assuming that a certain number of classes are already known by the model ( Hsu et al. , 2018 ; 2019 ; Han et al. , 2019 ) . This knowledge comes in the form of a labelled dataset of images for a certain set of classes . Given that this data is labelled , off-the-shelf supervised learning techniques can be used to train a very effective classifier for the known classes , particularly if Convolutional Neural Networks ( CNNs ) are employed . However , this does not mean that the learned features are useful as a representation of the new classes . Furthermore , even if the representation transfers well , one still has the problem of identifying the new classes in an unlabelled dataset , which is a clustering problem . We tackle these problems by introducing a novel approach that combines three key ideas ( section 2 and fig . 1 ) . The first idea is to pre-train the image representation ( a CNN ) using all available images , both labelled and unlabelled , using a self-supervised learning objective . Crucially , this objective does not leverage the known labels , resulting in features that are much less biased towards the labelled ∗indicates equal contribution classes . Labels are used only after pre-training to learn a classifier specific to the labelled data as well as to fine-tune the deepest layers of the CNN , for which self-supervision is not as effective . The second idea is a new approach to transfer the information contained in the labelled images to the problem of clustering the unlabelled ones . Information is transferred by sharing the same representation between labelled and unlabelled images , motivated by the fact that the new classes are often similar to the known ones . In more detail , pairs of unlabelled images are compared via their representation vectors . The comparison is done using robust rank statistics , by testing if two images share the same subset of k maximally activated representation components . This test is used to decide if two unlabelled images belong to the same ( new ) class or not , generating a set of noisy pairwise pseudo-labels . The pseudo-labels are then used to learn a similarity function for the unlabelled images . The third and final idea is , after bootstrapping the representation , to optimise the model by minimizing a joint objective function , containing terms for both the labelled and unlabelled subsets , using respectively the given labels and the generated pseudo-labels , thus avoiding the forgetting issue that may arise with a sequential approach . A further boost is obtained by incorporating incremental learning of the discovered classes in the classification task , which allows information to flow between the labelled and unlabelled images . We evaluate our method on several public benchmarks ( section 3 ) , outperforming by a large margin all existing techniques ( section 4 ) that can be applied to this problem , demonstrating the effectiveness of our approach . We conclude the paper by summarizing our findings ( section 5 ) . Our code can be found at http : //www.robots.ox.ac.uk/˜vgg/research/auto_novel . 2 METHOD . Given an unlabelled dataset Du = { xui , i = 1 , . . . , M } of images xui ∈ R3×H×W , our goal is to automatically cluster the images into a number of classes Cu , which we assume to be known a priori . We also assume to have a second labelled image dataset Dl = { ( xli , yli ) , i = 1 , . . . , N } where yli ∈ { 1 , . . . , Cl } is the class label for image xli . We also assume that the set of Cl labelled classes is disjoint from the set of Cu unlabelled ones . While the statistics of Dl and Du thus differ , we hypothesize that a general notion of what constitutes a “ good class ” can be extracted from Dl and that the latter can be used to better cluster Du . We approach the problem by learning an image representation Φ : x 7→ Φ ( x ) ∈ Rd in the form of a CNN . The goal of the representation is to help to recognize the known classes and to discover the new ones . In order to learn this representation , we combine three ideas , detailed in the next three sections . 2.1 SELF-SUPERVISED LEARNING . Given that we have a certain number of labelled images Dl at our disposal , the obvious idea is to use these labels to bootstrap the representation Φ by minimizing a standard supervised objective such as the cross-entropy loss . However , experiments show that this causes the representation to overly-specialize for the classes in Dl , providing a poor representation of the new classes in Du . Thus we resist the temptation of using the labels right away and use instead a self-supervised learning method to bootstrap the representation Φ. Self-supervised learning has been shown ( Kolesnikov et al. , 2019 ; Gidaris et al. , 2018 ) to produce robust low-level features , especially for the first few layers of typical CNNs . It has the benefit that no data annotations are needed , and thus it can be applied to both labelled and unlabelled images during training . In this way , we achieve the key benefit of ensuring that the representation is initialized without being biased towards the labelled data . In detail , we first pre-train our model Φ with self-supervision on the union of Dl and Du ( ignoring all labels ) . We use the RotNet ( Gidaris et al. , 2018 ) approach1 due to its simplicity and efficacy , but any self-supervised method could be used instead . We then extend the pre-trained network Φ with a classification head ηl : Rd → RCl implemented as a single linear layer followed by a softmax layer . The function ηl ◦ Φ is fine-tuned on the labelled dataset Dl in order to learn a classifier for the Cl known classes , this time using the labels yi and optimizing the standard cross-entropy ( CE ) loss : LCE = − 1 N N∑ i=1 log ηlyi ( z l i ) ( 1 ) where zli = Φ ( x l i ) ∈ Rd is the representation of image xli . Only ηl and the last macro-block of Φ ( section 3 ) are updated in order to avoid overfitting the representation to the labelled data . 2.2 TRANSFER LEARNING VIA RANK STATISTICS . Once the representation Φ and the classifier ηl have been trained , we are ready to look for the new classes in Du . Since the classes in Du are unknown , we represent them by defining a relation among pairs of unlabelled images ( xui , x u j ) . The idea is that similar images should belong to the same ( new ) class , which we denote by the symbol sij = 1 , while dissimilar ones should not , which we denote by sij = 0 . The problem is then to obtain the labels sij . Our assumption is that the new classes will have some degree of visual similarity with the known ones . Hence , the learned representation should be applicable to old and new classes equally well . As a consequence , we expect the descriptors zui = Φ ( x u i ) and z u j = Φ ( x u j ) of two images x u i , x u j from the new classes to be close if they are from the same ( new ) class , and to be distinct otherwise . Rather than comparing vectors zui , z u j directly ( e.g. , by a scalar product ) , however , we use a more robust rank statistics . Specifically , we rank the values in vector zui by magnitude . Then , if the rankings obtained for two unlabelled images xui and x u j are the same , they are very likely to belong to the same ( new ) class , so we set sij = 1 . Otherwise , we set sij = 0 . In practice , it is too strict to require the two rankings to be identical if the dimension of zui is high ( otherwise we may end up with sij = 0 for all pairs ( i , j ) , i 6= j ) . Therefore , we relax this requirement by only testing if the sets of the top-k ranked dimensions are the same ( we use k = 5 in our experiments ) , i.e . : sij = 1 { topk ( Φ ( x u i ) ) = topk ( Φ ( x u j ) ) } , ( 2 ) where topk : Rd → P ( { 1 , . . . , d } ) associates to a vector z the subset of indices { 1 , . . . , d } of its top-k elements . Once the labels sij have been obtained , we use them as pseudo-labels to train a comparison function for the unlabelled data . In order to do this , we apply a new head ηu : Rd → RCu to the image representation zui = Φ ( x u i ) to extract a new descriptor vector η u ( zui ) optimized for the unlabelled data . As in section 2.1 , the head is composed of a linear layer followed by a softmax . Then , the inner product ηu ( zui ) > ηu ( zuj ) is used as a score for whether images x u i and x u j belong to the same class or not . Note that ηu ( zui ) is a normalized vector due to the softmax layer in η u . This descriptor is trained by optimizing the binary cross-entropy ( BCE ) loss : LBCE = − 1 M2 M∑ i=1 M∑ j=1 [ sij log η u ( zui ) > ηu ( zuj ) + ( 1− sij ) log ( 1− ηu ( zui ) > ηu ( zuj ) ) ] . ( 3 ) 1 We present to the network Φ randomly-rotated versions Rx of each image and task it with predicting R. The problem is formulated as a 4-way classification of the rotation angle , with angle in { 0◦ , 90◦ , 180◦ , 270◦ } . The model η ◦ Φ ( Rx ) is terminated by a single linear layer η with 4 outputs each scoring an hypothesis . The parameters of η and Φ are optimized by minimizing the cross-entropy loss on the rotation prediction . Furthermore , we structure ηu in a particular manner : We set its output dimension to be equal to the number of new classes Cu . In this manner , we can use the index of the maximum element of each vector ŷui = argmaxy [ η u ◦ Φ ( xui ) ] y as prediction ŷui for the class of image xui ( as opposed to assigning labels via a clustering method such as k-means ) .

Summary:   This paper tackles the problem of unsupervised object discovery, whereby a labeled dataset must be leveraged in order to then cluster an unlabeled dataset with a set of unknown categories. The paper contributes three main ideas to succeed at this task, namely 1) use of self-supervised learning to initialize the representations in a way that doesn't bias them to the labeled data, 2) a robust rank-based metric to generate estimates of similarity/dissimilarity along with consistency-based regularization to improve optimization, and 3) Joint optimization/refinement using a combination of labeled/unlabeled losses, as well as ability to learn incrementally without forgeting the original labeled classes. Results are shown on a range of datasets including OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN. The results demonstrate improvement over the current state of art for this task.