Unsupervised Few Shot Learning via Self-supervised Training

1 INTRODUCTION . Few-shot learning , which aims to accomplish a learning task by using very few training examples , is receiving increasing attention in the machine learning community . The challenge of few-shot learning lies on that traditional techniques such as fine-tuning would normally incur overfitting ( Wang et al. , 2018 ) . To overcome this difficulty , a set-to-set meta-learning ( episodic learning ) paradigm was proposed ( Vinyals et al. , 2016 ) . In such a paradigm , the conventional mini-batch training is replaced by the episodic training , in term of that a batch of episodic tasks , each of which having the same setting as the testing environment , are presented to the learning model ; and in each episodic task , the model learns to predict the classes of unlabeled points ( the query set ) using very few labeled examples ( the support set ) . By this , the learning model acquires the transferable knowledge ( optimized feature representations ) across tasks , and due to the consistency between the training and testing environments , the model is able to generalize to novel but related tasks . Although this set-to-set few-shot learning paradigm has made great progress , in its current supervised form , it requires a large number of labeled examples for constructing episodic tasks , which is often infeasible or too expensive in practice . So , can we build up a few-shot learner in the paradigm of episodic training using only unlabeled data ? It is well-known that humans have the remarkable ability to learn a concept when given only several exposures to its instances , for example , young children can effortlessly learn and generalize the concept of “ giraffe ” after seeing a few pictures of giraffes . While the specifics of the human learning process are complex ( trial-based , perpetual , multi-sourced , and simultaneous for multiple tasks ) and yet to be solved , previous works agree that its nature is progressive and unsupervised in many cases ( Dupoux , 2018 ) . Given a set of unlabeled items , humans are able to organize them into different clusters by comparing one with another . The comparing or associating process follows a coarse-to-fine manner . At the beginning of learning , humans tend to group items based on fuzzyrough knowledge such as color , shape or size . Subsequently , humans build up associations between items using more fine-grained knowledge , i.e. , stripes of images , functions of items or other domain knowledge . Furthermore , humans can extract representative representations across categories and apply this capability to learn new concepts ( Wang et al. , 2014b ) . In the present study , inspired by the unsupervised and progressive characteristics of human learning , we propose an unsupervised model for few-shot learning via a self-supervised training procedure ( UFLST ) . Different from previous unsupervised learning methods , our model integrates unsupervised learning and episodic training into a unified framework , which facilitates feature extraction and model training iteratively . Basically , we adopt the episodic training paradigm , taking advantage of its capability of extracting transferable knowledge across tasks , but we use an unsupervised strategy to construct episodic tasks . Specifically , we apply progressive clustering to generate pseudo labels for unlabeled data , and this is done alternatively with feature optimization via few-shot learning in an iterative manner ( Fig . 1 ) . Initially , unlabeled data points are assigned into several clusters , and we sample a few training examples from each cluster together with their pseudo labels ( the identities of clusters ) to construct a set of episodic tasks having the same setting as the testing environment . We then train the few-shot learner using the constructed episodic tasks and obtain improved feature representations for the data . In the next round , we use the improved features to recluster data points , generating new pseudo labels and constructing new episodic tasks , and train the few-shot learner again . The above two steps are repeated till a stopping criterion is reached . After training , we expect that the few-shot learner has acquired the transferable knowledge ( the optimized feature representations ) suitable for a novel task of the same setting as in the episodic training . Using benchmark datasets , we demonstrate that our model outperforms other unsupervised few-shot learning methods and approaches to the performances of fully supervised models . 2 RELATED WORKS . In the paradigm of episodic training , few-shot learning algorithms can be divided into two main categories : “ learning to optimize ” and “ learning to compare ” . The former aims to develop a learning algorithm which can adapt to a new task efficiently using only few labeled examples or with few steps of parameter updating ( Finn et al. , 2017 ; Ravi & Larochelle , 2016 ; Mishra et al. , 2017 ; Rusu et al. , 2018 ; Nichol & Schulman , 2018 ; Andrychowicz et al. , 2016 ) , and the latter aims to learn a proper embedding function , so that prediction is based on the distance ( metric ) of a novel example to the labeled instances ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Ren et al. , 2018 ; Sung et al. , 2018 ; Liu et al. , 2018 ) . In the present study , we focus on the “ learning to compare ” framework , although the other framework can also be integrated into our model . Only very recently , people have tried to develop unsupervised few-shot learning models . Hsu et al . ( 2018 ) proposed a method called CACTUs , which uses progressive clustering to leverage feature representations before carrying out episodic training . This is different from our model , in term of that we carry out progressive clustering and episodic training concurrently , and the two processes facilitate with each other . Khodadadeh et al . ( 2018 ) proposed a method called UMTRA , which uti- lizes the statistical diversity properties and domain-specific augmentations to generate training and validation data . Antoniou & Storkey ( 2019 ) proposed a similar model called AAL , which uses data augmentations of the unlabeled support set to generate the query data . Both methods are different from our model , in term of that we use a progressive clustering strategy to generate pseudo labels for constructing episodic tasks . The idea of self-supervised training is to artificially generate pseudo labels for unlabeled data , which is useful when supervisory signals are not available or too expensive ( de Sa , 1994 ) . Progressive ( deep ) clustering is a promising method for self-supervised training , which aims to optimize feature representations and pseudo labels ( cluster assignments ) in an iterative manner . This idea was first applied in NLP tasks , which tries to self-train a two-phase parser-reranker system using unlabeled data ( McClosky et al. , 2006 ) . Xie et al . ( 2016 ) proposed a Deep Embedded Clustering network to jointly learn cluster centers and network parameters . Caron et al . ( 2018 ) further proposed strategies to solve the degenerated solution problem in progressive clustering . Fan et al . ( 2018 ) and Song et al . ( 2018 ) applied the progressive clustering idea to the person re-identification task , both of which aim to transfer the extracted feature representations to an unseen domain . None of these studies have integrated progressive clustering and episodic training in few-shot learning as we do in this work . 3 METHOD . In this section , we describe the model UFLST in detail . Let us first introduce some notations . Denote the model at the training round t as M t , { xi } the unlabeled dataset with the number of examples N , and { zi } t is the corresponded feature vector with dimentionalityD , which is given by fθt : x Mt−−→ z , where fθt representing the feature extracter and θt the training parameters of M t. { z̃i } t and { x̃i } t represent , respectively , the selected features and unlabeled data after removing outliers from the clustering results , and { ỹi } t the corresponding pseudo labels . 3.1 PROGRESSIVE CLUSTERING . 3.1.1 K-RECIPROCAL JACCARD DISTANCE . To cluster unlabeled data points , we adopt the k-reciprocal Jaccard distance ( KRJD ) metric to measure the distance between data points ( Qin et al. , 2011 ; Zhong et al. , 2017 ) , and it is done in the feature space rather than the raw pixel space . First , we calculate the k-reciprocal nearest neighbours of each feature point , which are given by , R ( z , k ) = { zj | ( zj ∈ N ( z , k ) ) ∩ ( z ∈ N ( zj , k ) ) } , ( 1 ) where N ( z , k ) denotes the k nearest neighbours of z. R ( z , k ) imposes that z and each element of R ( z , k ) are mutually k nearest neighbours of each other . Second , we compute KRJD between two feature points , which is given by Jij = 1− |R ( zi , k ) ∩R ( zj , k ) | |R ( zi , k ) ∪R ( zj , k ) | . ( 2 ) Compared to the Euclidean distance , KRJD takes into account the reciprocal relationship between data points , and hence is a stricter rule measuring whether two feature points matches or not . We find that KRJD is crucial to our model , which outperforms the Euclidean metric as demonstrated in Fig . 2 ( more comparisons of two metrics are given in Appendix A ) . 3.1.2 DENSITY-BASED SPATIAL CLUSTERING ALGORITHM . For the clustering strategy , we choose the density-based spatial clustering algorithm ( DBSCAN ) ( Ester et al. , 1996 ) , which performs better than other methods in our model . The reasons are : 1 ) other clustering methods such as k-means and hierarchical clustering are useful to find spherical shaped or convex clusters in the embedding space , while DBSCAN works well for arbitrarily shaped clusters ; 2 ) DBSCAN can detect outliers as noise points , which is very useful at the beginning of training when the distribution of data points in the embedding space is highly noisy ; 3 ) DBSCAN does not need to specify the number of clusters to be generated , which is appealing for unsupervised learning . After removing noisy points ( outliers ) as done in DBSCAN , pseudo labels ( i.e. , cluster identities ) of data points { x̃i } t can be expressed as , { ỹi } t = DBSCAN ( , ms , J ) , ( 3 ) where denotes the maximum distance for two points to be considered as in the same neighborhood , ms the minimum number of points huddled together for a region to be considered as dense , and J the KRJD matrix . The value of relies on the cluster density of feature points , which is set to be the mean distance of top P minimum distances in KRJD in this study ( see Appendix B ) . 3.2 EPISODIC TRAINING . 3.2.1 CONSTRUCTING EPISODIC TASKS . After each round of clustering , we construct a number of episodic tasks , denoted as T = { T1 , T2 , ... , TS } with S the number of tasks , from the pseudo-labeled data set { x̃i , ỹi } t. For each episodic task , we randomly sample NC classes and NE examples per class . Notably , the setting of each episodic task follows that of the test environment to be performed after training . We will apply two different ways to implement few-shot learning ( see below ) . One uses the prototype loss , which aims to learn the prototypes of each class and discriminate a novel example based on its distances to the prototypes . In this case , we further split NE into a support set S and a query set Q , i.e. , NE = NS +NQ . Following Snell et al . ( 2017 ) , we choose a larger value of NC in training than that in testing , but keeps NS the same . The other way to implement few-shot learning is to use the triplet loss or the hardtriplet loss , which separates examples from different classes with a positive marginm . In this case , no splitting support and query data is needed . To mine hard negative examples in triplets , we also use a larger NC in training than that in testing .

Summary: This paper considers the problem of learning an image representation for few-shot learning without using image labels during training. This is a well-motivated problem since (as the paper points out) learning such a representation using "episodes" of low-shot learning problems as examples may require a large amount of annotated data. The paper proposes an iterative algorithm which alternates between clustering the images using the current model and updating the model using the clusters as "pseudo-labels". This approach is not particularly elegant as there is no clear objective being optimized, but it may nevertheless be effective. One main claim of the paper is that the iterative nature of this process is key to the success of the algorithm. The choice of model is a multi-layer conv-net which is trained using SGD (Adam). The paper investigates both triplet (with and without hard negatives) and "prototype" losses for learning the model parameters. To find clusters, the paper adopts the DBSCAN algorithm using the Jaccard similarity of the k-reciprocal neighbour sets.