Unsupervised Learning via Meta-Learning

1 INTRODUCTION . Unsupervised learning is a fundamental , unsolved problem ( Hastie et al. , 2009 ) and has seen promising results in domains such as image recognition ( Le et al. , 2013 ) and natural language understanding ( Ramachandran et al. , 2017 ) . A central use case of unsupervised learning methods is enabling better or more efficient learning of downstream tasks by training on top of unsupervised representations ( Reed et al. , 2014 ; Cheung et al. , 2015 ; Chen et al. , 2016 ) or fine-tuning a learned model ( Erhan et al. , 2010 ) . However , since the downstream objective requires access to supervision , the objectives used for unsupervised learning are only a rough proxy for downstream performance . If a central goal of unsupervised learning is to learn useful representations , can we derive an unsupervised learning objective that explicitly takes into account how the representation will be used ? The use of unsupervised representations for downstream tasks is closely related to the objective of meta-learning techniques : finding a learning procedure that is more efficient and effective than learning from scratch . However , unlike unsupervised learning methods , meta-learning methods require large , labeled datasets and hand-specified task distributions . These dependencies are major obstacles to widespread use of these methods for few-shot classification . To begin addressing these problems , we propose an unsupervised meta-learning method : one which aims to learn a learning procedure , without supervision , that is useful for solving a wide range of new , human-specified tasks . With only raw , unlabeled observations , our model ’ s goal is to learn a useful prior such that , after meta-training , when presented with a modestly-sized dataset for a human-specified task , the model can transfer its prior experience to efficiently learn to perform the new task . If we can build such an algorithm , we can enable few-shot learning of new tasks without needing any labeled data nor any pre-defined tasks . To perform unsupervised meta-learning , we need to automatically construct tasks from unlabeled data . We study several options for how this can be done . We find that a good task distribution should be diverse , but also not too difficult : naı̈ve random approaches for task generation produce tasks that contain insufficient regularity to enable useful meta-learning . To that end , our method proposes tasks by first leveraging prior unsupervised learning algorithms to learn an embedding of the †Work done as a visiting student researcher at the University of California , Berkeley . input data , and then performing an overcomplete partitioning of the dataset to construct numerous categorizations of the data . We show how we can derive classification tasks from these categorizations for use with meta-learning algorithms . Surprisingly , even with simple mechanisms for partitioning the embedding space , such as k-means clustering , we find that meta-learning acquires priors that , when used to learn new , human-designed tasks , learn those tasks more effectively than methods that directly learn on the embedding . That is , the learning algorithm acquired through unsupervised meta-learning achieves better downstream performance than the original representation used to derive meta-training tasks , without introducing any additional assumptions or supervision . See Figure 1 for an illustration of the complete approach . The core idea in this paper is that we can leverage unsupervised embeddings to propose tasks for a meta-learning algorithm , leading to an unsupervised meta-learning algorithm that is particularly effective as pre-training for human-specified downstream tasks . In the following sections , we formalize our problem assumptions and goal , which match those of unsupervised learning , and discuss several options for automatically deriving tasks from embeddings . We instantiate our method with two meta-learning algorithms and compare to prior state-of-the-art unsupervised learning methods . Across four image datasets ( MNIST , Omniglot , miniImageNet , and CelebA ) , we find that our method consistently leads to effective downstream learning of a variety of human-specified tasks , including character recognition tasks , object classification tasks , and facial attribute discrimination tasks , without requiring any labels or hand-designed tasks during meta-learning and where key hyperparameters of our method are held constant across all domains . We show that , even though our unsupervised meta-learning algorithm trains for one-shot generalization , one instantiation of our approach performs well not only on few-shot learning , but also when learning downstream tasks with up to 50 training examples per class . In fact , some of our results begin to approach the performance of fully-supervised meta-learning techniques trained with fully-specified task distributions . 2 UNSUPERVISED META-LEARNING . In this section , we describe our problem setting in relation to that of unsupervised and semisupervised learning , provide necessary preliminaries , and present our approach . 2.1 PROBLEM STATEMENT . Our goal is to leverage unlabeled data for the efficient learning of a range of human-specified downstream tasks . We only assume access to an unlabeled dataset D = { xi } during meta-training . After learning from the unlabeled data , which we will refer to as unsupervised meta-training , we want to apply what was learned towards learning a variety of downstream , human-specified tasks from a modest amount of labeled data , potentially as few as a single example per class . These downstream tasks may , in general , have different underlying classes or attributes ( in contrast to typical semi-supervised problem assumptions ) , but are assumed to have inputs from the same distribution as the one from which datapoints in D are drawn . Concretely , we assume that downstream tasks are M -way classification tasks , and that the goal is to learn an accurate classifier using K labeled datapoints ( xk , yk ) from each of the M classes , where K is relatively small ( i.e . between 1 and 50 ) . The unsupervised meta-training phase aligns with the unsupervised learning problem in that it involves no access to information about the downstream tasks , other than the fact that they areM -way classification tasks , for variable M upper-bounded by N . The upper bound N is assumed to be known during unsupervised meta-training , but otherwise , the values of M and K are not known a priori . As a result , the unsupervised meta-training phase needs to acquire a sufficiently general prior for applicability to a range of classification tasks with variable quantities of data and classes . This problem definition is our prototype for a practical use-case in which a user would like to train an application-specific image classifier , but does not have an abundance of labeled data . 2.2 PRELIMINARIES . Unsupervised embedding learning . An unsupervised embedding learning algorithm E is a procedure that takes as input an unlabeled dataset D = { xi } and outputs a mapping from { xi } to embeddings { zi } . These embedded points are typically lower-dimensional and arranged such that distances correspond to meaningful differences between inputs , in contrast to distances between the original inputs , such as image pixels , which are not meaningful measures of image similarity . Task . An M -way K-shot classification task T consists of K training datapoints and labels { ( xk , ` k ) } per class , which are used for learning a classifier , and Q query datapoints and labels per class , on which the learned classifier is evaluated . That is , in a task there are K + Q = R datapoints and labels for each of the M classes . Meta-learning . A supervised meta-learning algorithmM ( · ) takes as input a set of supervised metatraining tasks { Tt } . It produces a learning procedure F ( · ) , which , in turn , ingests the supervised training data of a task to produce a classifier f ( · ) . The goal ofM is to learn F such that , when faced with a meta-test time task Tt′ held-out from { Tt } , F can learn a ft′ that accomplishes Tt′ . At a high level , the quintessential meta-learning strategy is to haveM iterate over { Tt } , cycling between applying the current form of Ft on training data from Tt to learn ft , assessing its performance by calculating some meta-loss L on held-out data from the task , and optimizing L to improve the learning procedure . We build upon two meta-learning algorithms : model agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) and prototypical networks ( ProtoNets ) ( Snell et al. , 2017 ) . MAML aims to learn the initial parameters of a deep network such that one or a few gradient steps leads to effective generalization ; it specifies F as gradient descent starting from the meta-learned parameters . ProtoNets aim to metalearn a representation in which a class is effectively identified by its prototype , defined to be the mean of the class ’ training examples in the meta-learned space ; F is the computation of these class prototypes , and f is a linear classifier that predicts the class whose prototype is closest in Euclidean distance to the query ’ s representation . Task generation for meta-learning . We briefly summarize how tasks are typically generated from labeled datasets { ( xi , yi ) } for supervised meta-learning , as introduced by Santoro et al . ( 2016 ) . For simplicity , consider the case where the labels are discrete scalar values yi . To construct an N -way classification task T ( assuming N is not greater than the number of unique yi ) , we can sample N classes , sample R datapoints { xr } n for each of the N classes , and sample a permutation of N distinct one-hot vectors ( ` n ) to serve as task-specific labels of the N sampled classes . The task is then defined as T = { ( xn , r , ` n ) | xn , r ∈ { xr } n } . Of course , this procedure is only possible with labeled data ; in the next section , we discuss how we can construct tasks without ground-truth labels . 2.3 UNSUPERVISED META-LEARNING WITH AUTOMATICALLY CONSTRUCTED TASKS . We approach our problem from a meta-learning perspective , framing the problem as the acquisition , from unlabeled data , of an efficient learning procedure that is transferable to human-designed tasks . In particular , we aim to construct classification tasks from the unlabeled data and then learn how to efficiently learn these tasks . If such tasks are adequately diverse and structured , then metalearning these tasks should enable fast learning of new , human-provided tasks . A key question , then , is how to automatically construct such tasks from unlabeled data D = { xi } . Notice that in the supervised meta-learning task generation procedure detailed in Section 2.2 , the labels yi induce a partition P = { Cc } over { xi } by assigning all datapoints with label yc to subset Cc . Once a partition is obtained , task generation is simple ; we can reduce the problem of constructing tasks to that of constructing a partition over { xi } . All that ’ s left is to find a principled alternative to human labels for defining the partition . A naı̈ve approach is to randomly partition the data D. While such a scheme introduces diverse tasks , there is no structure ; that is , there is no consistency between a task ’ s training data and query data , and hence nothing to be learned during each task , let alone across tasks . As seen in Table 3 , providing a meta-learner with purely random tasks results in failed meta-learning . To construct tasks with structure that resembles that of human-specified labels , we need to group datapoints into consistent and distinct subsets based on salient features . With this motivation in mind , we propose to use k-means clustering . Consider the partition P = { Cc } learned by k-means as a simplification of a Gaussian mixture model p ( x|c ) p ( c ) . If the clusters can recover a semblance of the true class-conditional generative distributions p ( x|c ) , creating tasks based on treating these clusters as classes should result in useful unsupervised meta-training . However , the result of k-means is critically dependent on the metric space on which its objective is defined . Clustering in pixel-space is unappealing for two reasons : ( 1 ) distance in pixel-space correlates poorly with semantic meaning , and ( 2 ) the high dimensionality of raw images renders clustering difficult in practice . We empirically show in Table 3 that meta-learning with tasks defined by pixel-space clusters , with preprocessing as directed by Coates & Ng ( 2012 ) , also fails . We are now motivated to cluster in spaces in which common distance functions correlate to semantic meaning . However , we must satisfy the constraints of our problem statement in the process of learning such spaces . To these ends , we use state-of-the-art unsupervised learning methods to produce useful embedding spaces . For qualitative evidence in the unsupervised learning literature that such embedding spaces exhibit semantic meaning , see Cheung et al . ( 2015 ) ; Bojanowski & Joulin ( 2017 ) ; Donahue et al . ( 2017 ) . We note that while a given embedding space may not be directly suitable for highly-efficient learning of new tasks ( which would require the embedding space to be precisely aligned or adaptable to the classes of those tasks ) , we can still leverage it for the construction of structured tasks , a process for which requirements are less strict . Thus , we first run an out-of-the-box unsupervised embedding learning algorithm E on D , then map the data { xi } into the embedding space Z , producing { zi } . To produce a diverse task set , we generate P partitions { Pp } by running clustering P times , applying random scaling to the dimensions of Z to induce a different metric , represented by diagonal matrix A , for each run of clustering . With µc denoting the learned centroid of cluster Cc , a single run of clustering can be summarized with P , { µc } = argmin { Cc } , { µc } k∑ c=1 ∑ z∈Cc ‖z− µc‖2A ( 1 ) We derive tasks for meta-learning from the partitions using the procedure detailed in Section 2.2 , except we begin the construction of each task by sampling a partition from the uniform distribution U ( P ) , and for xi ∈ Cc , specify yi = c. To avoid imbalanced clusters dominating the meta-training tasks , we opt not to sample from p ( c ) ∝ |Cc| , but instead sample N clusters uniformly without replacement for each task . We note that Caron et al . ( 2018 ) are similarly motivated in their design decision of sampling data from a uniform distribution over clusters . With the partitions being constructed over { zi } , we have one more design decision to make : should we perform meta-learning on embeddings or images ? We consider that , to successfully solve new tasks at meta-test time , a learning procedure F that takes embeddings as input would depend on the embedding function ’ s ability to generalize to out-of-distribution observations . On the other hand , by meta-learning on images , F can separately adapt f to each evaluation task from the rawest level of representation . Thus , we choose to meta-learn on images . We call our method clustering to automatically construct tasks for unsupervised meta-learning ( CACTUs ) . We detail the task construction algorithm in Algorithm 1 , and provide an illustration of the complete unsupervised meta-learning approach for classification in Figure 1 .

Summary: The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. SoTA meta-learning frameworks (MAML and ProtoNet) typically require rather large labeled datasets and hand-specified task distributions to define a sequence of tasks on which the algorithms are trained on. This paper proposes to unsupervised generate the sequence of tasks using multiple partitions as pseudo labels via k-means and other clustering variants on the embedding space. Empirical experiments show the benefit of the meta-learning on the M-way K-shot image classification tasks.  Also, “sampling a partition from U(P)” on page 4, the U(P) notation seems not defined.