Unsupervised Few Shot Learning via Self-supervised Training

1 INTRODUCTION . Few-shot learning , which aims to accomplish a learning task by using very few training examples , is receiving increasing attention in the machine learning community . The challenge of few-shot learning lies on that traditional techniques such as fine-tuning would normally incur overfitting ( Wang et al. , 2018 ) . To overcome this difficulty , a set-to-set meta-learning ( episodic learning ) paradigm was proposed ( Vinyals et al. , 2016 ) . In such a paradigm , the conventional mini-batch training is replaced by the episodic training , in term of that a batch of episodic tasks , each of which having the same setting as the testing environment , are presented to the learning model ; and in each episodic task , the model learns to predict the classes of unlabeled points ( the query set ) using very few labeled examples ( the support set ) . By this , the learning model acquires the transferable knowledge ( optimized feature representations ) across tasks , and due to the consistency between the training and testing environments , the model is able to generalize to novel but related tasks . Although this set-to-set few-shot learning paradigm has made great progress , in its current supervised form , it requires a large number of labeled examples for constructing episodic tasks , which is often infeasible or too expensive in practice . So , can we build up a few-shot learner in the paradigm of episodic training using only unlabeled data ? It is well-known that humans have the remarkable ability to learn a concept when given only several exposures to its instances , for example , young children can effortlessly learn and generalize the concept of “ giraffe ” after seeing a few pictures of giraffes . While the specifics of the human learning process are complex ( trial-based , perpetual , multi-sourced , and simultaneous for multiple tasks ) and yet to be solved , previous works agree that its nature is progressive and unsupervised in many cases ( Dupoux , 2018 ) . Given a set of unlabeled items , humans are able to organize them into different clusters by comparing one with another . The comparing or associating process follows a coarse-to-fine manner . At the beginning of learning , humans tend to group items based on fuzzyrough knowledge such as color , shape or size . Subsequently , humans build up associations between items using more fine-grained knowledge , i.e. , stripes of images , functions of items or other domain knowledge . Furthermore , humans can extract representative representations across categories and apply this capability to learn new concepts ( Wang et al. , 2014b ) . In the present study , inspired by the unsupervised and progressive characteristics of human learning , we propose an unsupervised model for few-shot learning via a self-supervised training procedure ( UFLST ) . Different from previous unsupervised learning methods , our model integrates unsupervised learning and episodic training into a unified framework , which facilitates feature extraction and model training iteratively . Basically , we adopt the episodic training paradigm , taking advantage of its capability of extracting transferable knowledge across tasks , but we use an unsupervised strategy to construct episodic tasks . Specifically , we apply progressive clustering to generate pseudo labels for unlabeled data , and this is done alternatively with feature optimization via few-shot learning in an iterative manner ( Fig . 1 ) . Initially , unlabeled data points are assigned into several clusters , and we sample a few training examples from each cluster together with their pseudo labels ( the identities of clusters ) to construct a set of episodic tasks having the same setting as the testing environment . We then train the few-shot learner using the constructed episodic tasks and obtain improved feature representations for the data . In the next round , we use the improved features to recluster data points , generating new pseudo labels and constructing new episodic tasks , and train the few-shot learner again . The above two steps are repeated till a stopping criterion is reached . After training , we expect that the few-shot learner has acquired the transferable knowledge ( the optimized feature representations ) suitable for a novel task of the same setting as in the episodic training . Using benchmark datasets , we demonstrate that our model outperforms other unsupervised few-shot learning methods and approaches to the performances of fully supervised models . 2 RELATED WORKS . In the paradigm of episodic training , few-shot learning algorithms can be divided into two main categories : “ learning to optimize ” and “ learning to compare ” . The former aims to develop a learning algorithm which can adapt to a new task efficiently using only few labeled examples or with few steps of parameter updating ( Finn et al. , 2017 ; Ravi & Larochelle , 2016 ; Mishra et al. , 2017 ; Rusu et al. , 2018 ; Nichol & Schulman , 2018 ; Andrychowicz et al. , 2016 ) , and the latter aims to learn a proper embedding function , so that prediction is based on the distance ( metric ) of a novel example to the labeled instances ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Ren et al. , 2018 ; Sung et al. , 2018 ; Liu et al. , 2018 ) . In the present study , we focus on the “ learning to compare ” framework , although the other framework can also be integrated into our model . Only very recently , people have tried to develop unsupervised few-shot learning models . Hsu et al . ( 2018 ) proposed a method called CACTUs , which uses progressive clustering to leverage feature representations before carrying out episodic training . This is different from our model , in term of that we carry out progressive clustering and episodic training concurrently , and the two processes facilitate with each other . Khodadadeh et al . ( 2018 ) proposed a method called UMTRA , which uti- lizes the statistical diversity properties and domain-specific augmentations to generate training and validation data . Antoniou & Storkey ( 2019 ) proposed a similar model called AAL , which uses data augmentations of the unlabeled support set to generate the query data . Both methods are different from our model , in term of that we use a progressive clustering strategy to generate pseudo labels for constructing episodic tasks . The idea of self-supervised training is to artificially generate pseudo labels for unlabeled data , which is useful when supervisory signals are not available or too expensive ( de Sa , 1994 ) . Progressive ( deep ) clustering is a promising method for self-supervised training , which aims to optimize feature representations and pseudo labels ( cluster assignments ) in an iterative manner . This idea was first applied in NLP tasks , which tries to self-train a two-phase parser-reranker system using unlabeled data ( McClosky et al. , 2006 ) . Xie et al . ( 2016 ) proposed a Deep Embedded Clustering network to jointly learn cluster centers and network parameters . Caron et al . ( 2018 ) further proposed strategies to solve the degenerated solution problem in progressive clustering . Fan et al . ( 2018 ) and Song et al . ( 2018 ) applied the progressive clustering idea to the person re-identification task , both of which aim to transfer the extracted feature representations to an unseen domain . None of these studies have integrated progressive clustering and episodic training in few-shot learning as we do in this work . 3 METHOD . In this section , we describe the model UFLST in detail . Let us first introduce some notations . Denote the model at the training round t as M t , { xi } the unlabeled dataset with the number of examples N , and { zi } t is the corresponded feature vector with dimentionalityD , which is given by fθt : x Mt−−→ z , where fθt representing the feature extracter and θt the training parameters of M t. { z̃i } t and { x̃i } t represent , respectively , the selected features and unlabeled data after removing outliers from the clustering results , and { ỹi } t the corresponding pseudo labels . 3.1 PROGRESSIVE CLUSTERING . 3.1.1 K-RECIPROCAL JACCARD DISTANCE . To cluster unlabeled data points , we adopt the k-reciprocal Jaccard distance ( KRJD ) metric to measure the distance between data points ( Qin et al. , 2011 ; Zhong et al. , 2017 ) , and it is done in the feature space rather than the raw pixel space . First , we calculate the k-reciprocal nearest neighbours of each feature point , which are given by , R ( z , k ) = { zj | ( zj ∈ N ( z , k ) ) ∩ ( z ∈ N ( zj , k ) ) } , ( 1 ) where N ( z , k ) denotes the k nearest neighbours of z. R ( z , k ) imposes that z and each element of R ( z , k ) are mutually k nearest neighbours of each other . Second , we compute KRJD between two feature points , which is given by Jij = 1− |R ( zi , k ) ∩R ( zj , k ) | |R ( zi , k ) ∪R ( zj , k ) | . ( 2 ) Compared to the Euclidean distance , KRJD takes into account the reciprocal relationship between data points , and hence is a stricter rule measuring whether two feature points matches or not . We find that KRJD is crucial to our model , which outperforms the Euclidean metric as demonstrated in Fig . 2 ( more comparisons of two metrics are given in Appendix A ) . 3.1.2 DENSITY-BASED SPATIAL CLUSTERING ALGORITHM . For the clustering strategy , we choose the density-based spatial clustering algorithm ( DBSCAN ) ( Ester et al. , 1996 ) , which performs better than other methods in our model . The reasons are : 1 ) other clustering methods such as k-means and hierarchical clustering are useful to find spherical shaped or convex clusters in the embedding space , while DBSCAN works well for arbitrarily shaped clusters ; 2 ) DBSCAN can detect outliers as noise points , which is very useful at the beginning of training when the distribution of data points in the embedding space is highly noisy ; 3 ) DBSCAN does not need to specify the number of clusters to be generated , which is appealing for unsupervised learning . After removing noisy points ( outliers ) as done in DBSCAN , pseudo labels ( i.e. , cluster identities ) of data points { x̃i } t can be expressed as , { ỹi } t = DBSCAN ( , ms , J ) , ( 3 ) where denotes the maximum distance for two points to be considered as in the same neighborhood , ms the minimum number of points huddled together for a region to be considered as dense , and J the KRJD matrix . The value of relies on the cluster density of feature points , which is set to be the mean distance of top P minimum distances in KRJD in this study ( see Appendix B ) . 3.2 EPISODIC TRAINING . 3.2.1 CONSTRUCTING EPISODIC TASKS . After each round of clustering , we construct a number of episodic tasks , denoted as T = { T1 , T2 , ... , TS } with S the number of tasks , from the pseudo-labeled data set { x̃i , ỹi } t. For each episodic task , we randomly sample NC classes and NE examples per class . Notably , the setting of each episodic task follows that of the test environment to be performed after training . We will apply two different ways to implement few-shot learning ( see below ) . One uses the prototype loss , which aims to learn the prototypes of each class and discriminate a novel example based on its distances to the prototypes . In this case , we further split NE into a support set S and a query set Q , i.e. , NE = NS +NQ . Following Snell et al . ( 2017 ) , we choose a larger value of NC in training than that in testing , but keeps NS the same . The other way to implement few-shot learning is to use the triplet loss or the hardtriplet loss , which separates examples from different classes with a positive marginm . In this case , no splitting support and query data is needed . To mine hard negative examples in triplets , we also use a larger NC in training than that in testing .

Summary: This paper aims to conduct few-shot learning on unlabeled data (instead of on training tasks with few-shot labeled data per task). The proposed algorithm is a trivial combination of existing clustering method and a few-shot learning method, i.e., the clustering provides pseudo labels, from which a series of few-shot training tasks are generated, and then traditional few-shot learning method can be applied afterward. Many existing techniques are integrated, e.g., k-reciprocal Jaccard distance, DBSCAN clustering, prototypical network, triplet loss, etc. The paper reported the experimental results on Omniglot and Market1501.