Variadic Learning by Bayesian Nonparametric Deep Embedding

1 INTRODUCTION . In machine learning , classification problems span two important axes : the number of classes to recognize ( the `` way '' of the problem ) and the number of examples provided for each class ( the `` shots '' to learn from ) . At one extreme , there are large-scale tasks like ImageNet in which there are 1000 classes with roughly 1000 examples each ( a 1000-way , ∼1000-shot problem ) . At the other extreme , there are datasets for learning from few examples , such as Omniglot , which features a 5- or 20-way , 1-shot problem . State-of-the-art methods for these two learning regimes are substantially different , with the former dominated by standard parametric deep networks and the latter by episodic meta-learning techniques . Moreover , as shown in our experiments , many methods degrade when the shot and way vary between training and testing . By contrast , humans recognize both familiar and unfamiliar categories whatever the amount of data , and can even learn a new category from a single example ( Lake et al. , 2015 ) . To this end , we introduce a learning problem which requires generalization from few-way , few-shot problems to many-way , many-shot problems . We call this regime of variable shot and way the variadic learning regime , after variadic functions . Just as variadic functions are those which can take any number of arguments to produce a result , a good variadic learner must learn from any amount of data , whatever the number of examples and classes , and produce strong results across unknown data distributions during test . Meta-learning provides one potential avenue for pursuing a variadic learner . Meta-learning approaches generally use plentiful supervision from one distribution of tasks to learn an algorithm or metric that can be applied to more sparsely supervised tasks . Ideally , meta-learning approaches do not need knowledge of the specific setting in which they will be used . However , in practice , metalearning approaches have commonly been trained and evaluated in constrained circumstances , so their generalization properties are not fully known . Perhaps most significantly , meta-learning is usually carried out independently across settings so that a different learner is specialized to each n-way , k-shot task . This potentially limits their deployment to more diverse settings with variable shot and way that we address in this work . As a first step towards a strong variadic learner , we propose a multi-modal ( many-to-one ) semisupervised clustering approach which can adapt its capacity to the underlying class representations , and show that this is critical for modeling more complex data distributions . This innovation allows our model to perform inference with any amount of supervision ( from totally unsupervised to fully supervised ) after training , and adjust better to variable shot and way than existing approaches . Our bayesian nonparametric deep embedding ( BANDE ) model ( see Figure 1 ) extends prototypical networks to multi-modal clustering . Clustering with multiple modes is critical for complex classes , and multi-modality makes unsupervised clustering possible . BANDE generalizes across any-shot , any-way tasks better than existing methods . At the many-way extreme , when trained with 5-way 1-shot episodes , BANDE achieves 75 % accuracy for 1692-way 10-shot classification of Omniglot , improving on both few-shot and supervised learning baselines . At the many-shot extreme , BANDE approaches the accuracy of a standard supervised learner on CIFAR-10/100 . On standard few-shot benchmarks BANDE is state-of-the-art in the semi-supervised setting . 2 RELATED WORK . Prototypes and Nonparametrics Prototypical networks ( Snell et al. , 2017 ) and semi-supervised prototypical networks ( Ren et al. , 2018 ) are the most closely related to our work . Prototypical networks simply and efficiently represent each class by its mean in a learned embedding . They assume that the data is fully labeled . Ren et al . ( 2018 ) extend prototypes to the semi-supervised setting by refining prototypes through soft k-means clustering of the unlabeled data . They assume that the data is at least partially labeled . Snell et al . ( 2017 ) and Ren et al . ( 2018 ) are limited to one cluster per class . We define a more general and adaptive approach through bayesian nonparametrics that extends prototypical networks to multi-modal clustering , with one or many clusters per class , of labeled and unlabeled data alike . Through multi-modal representation and adaptive inference of the number of modes , our method is significantly more accurate on complex classes , does unsupervised clustering , and improves on standard semi-supervised few-shot learning benchmarks . For multi-modal clustering we incorporate DP-means ( Kulis & Jordan , 2012 ) in our method . DPmeans is a scalable , bayesian nonparametric algorithm for unsupervised clustering that creates new clusters when data points are more than a threshold λ away from existing clusters . Our full method handles labeled and unlabeled data , augments the clustering with soft assignments under a normalized Gaussian likelihood , and defines a procedure to choose λ during learning and inference . Metric Learning Learning a metric to measure a given notion of distance/similarity addresses recognition by retrieval : given an unlabeled example , find the closest labeled example . The contrastive loss and siamese network architecture ( Chopra et al. , 2005 ; Hadsell et al. , 2006 ) optimize an embedding for metric learning by pushing similar pairs together and pulling dissimilar pairs apart . Of particular note is research in face recognition , where a same/different retrieval metric is used for many-way classification ( Schroff et al. , 2015 ) . Our approach is more aligned with metric learning by meta-learning ( Koch , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Garcia & Bruna , 2018 ) . These approaches meta-learn a distance function by directly optimizing the task loss , such as cross-entropy for classification , through episodic optimization ( Vinyals et al. , 2016 ) for each setting of way and shot . While we likewise learn by episodic optimization , we differ from previous meta-learning work in our examination of generalization to variable numbers of examples and classes during testing , and show improvement in this regime . Unlike metric learning on either exemplars ( Schroff et al. , 2015 ) or prototypes ( Snell et al. , 2017 ; Ren et al. , 2018 ) , our method adaptively interpolates between exemplar and uni-modal prototype representation by deciding the number of modes during clustering . Learning Regimes Variadic learning is best explained in relation to few-shot learning , low-shot learning , and conventional supervised learning . Few-shot learning ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ) handles tasks of fixed , known , and small numbers of data points and classes . In contrast , variadic tasks have variable numbers of data points and classes that can shift across tasks . Low-shot learning ( Hariharan & Girshick , 2017 ; Qi et al. , 2018 ; Qiao et al. , 2018 ) addresses both densely supervised base classes and sparsely supervised novel classes , but presupposes which classes are in which set . Variadic learning also addresses these extremes of supervision , but requires no knowledge of how much or how little supervision each class has . Large-scale supervised learning ( Bottou , 2010 ) parameterizes the model by the number of classes , and is tuned to the amount of data by choosing capacity , optimization schedules , and so forth . Variadic learning requires accuracy without specialization to shot and way . Life-long learning ( Thrun , 1996 ; 1998 ) concerns variable shot and way for streams of non-stationary problems , while variadic learning is for one problem of unknown dimensions . Bridging life-long and variadic learning is sensible but out of scope for this work . 3 BAYESIAN NONPARAMETRIC DEEP EMBEDDINGS ( BANDE ) . Our method end-to-end learns a deep embedding network and jointly clusters labeled and unlabeled data points by bayesian nonparametrics . Crucially , our model is able to express a single class as multiple modes , unlike the uni-modal clustering approaches of prior work . Figure 1 gives a schematic view of our multi-modal representation and how it differs from prior prototypical representations . Algorithm 1 expresses one step of model optimization in pseudocode . Few-shot Meta-learning In few-shot classification we are given a support set S = { ( x1 , y1 ) , . . . , ( xK , yK ) } of K labeled examples and a query set Q = { ( x′1 , y′1 ) , . . . , ( x′K′ , y′K′ ) } of K ′ labeled examples where each xi , x′i ∈ RD is a D-dimensional feature vector and yi , y′i ∈ { 1 , . . . , N } is the corresponding label . In the semi-supervised setting , yi may not be provided for every example xi . The support set is for learning while the query set is for inference : the few-shot classification problem is to recognize the class of the queries given the labeled supports . Meta-learning is carried out by episodic optimization of the model parameters for the task loss . Episodes are comprised of support and query sets , constructed by randomly sampling a subset of classes , sampling examples from these classes , and then partitioning the examples into supports and queries . Optimization iterates by making one episode and one update . The update is defined by the task loss , which for classification could be the softmax cross-entropy loss . For deep metric learning models like ours , the model parameters are those of the embedding function hφ : RD → RM that is a deep network with parameters φ . The embedding of an example x is the M -dimensional feature vector taken from the last layer of the network . Meta-training proceeds by optimizing the model parameters φ with respect to a task loss . Meta-testing proceeds episodically like meta-training but without query labels or further optimization . Prototypes Prototypical networks ( Snell et al. , 2017 ) take the mean of the embedded support examples of a particular class to form a prototype : µn = 1|Sn| ∑ ( xi , yi ) ∈Sn hφ ( xi ) , with Sn denoting the set of support examples of class n. In conjunction with a distance function d ( xi , xj ) , this provides an inference scheme for a query point x as the softmax over distances to the prototypes : pφ ( y = n |x ) = exp ( −d ( hφ ( x ) , µn ) ) ∑ n′ exp ( −d ( hφ ( x ) , µn′ ) ) . φ is optimized by minimizing the negative log-probability of the true class of each query point by stochastic gradient descent in each episode . Prototypical networks defined in this way learn to create uni-modal class distributions for fully labeled supports . Multi-modal Clustering Our method defines multi-modal prototypes of both labeled and unlabeled data . That is , a single class is represented by a set of cluster modes . By deciding the number of modes , our method interpolates between exemplar and uni-modal prototype representations , in effect adjusting its capacity depending on the data . To create multi-modal prototypes , we extend the non-parametric clustering algorithm DP-means ( Kulis & Jordan , 2012 ) to make it compatible with end-to-end learning . DP-means iterates through all examples in a dataset , computing the example ’ s minimum distance to all existing cluster means . If this distance is greater than a particular threshold λ , a new cluster is created with mean hφ ( xi ) , the example assigned to it . If xi is labeled , the new cluster takes on its label . While we use DP-means for cluster creation , we include cluster variances for reassignment . Labeled clusters are assigned a variance σl and unlabeled clusters are assigned a variance σu . σl and σu are differentiable , and therefore learned along with the embedding parameters φ . ( We discuss the probabilistic interpretations of this choice in the next section . ) λ , the threshold for creating a new cluster , is the sole hyperparameter for DP-means clustering . It is non-differentiable , and so it can not be learned jointly . Instead , we set λ episodically as a function of the data . In Kulis & Jordan ( 2012 ) , λ is parameterized as −2σ log ( α ( 1+ ρσ ) d/2 ) . α is the relative probability of forming a new cluster in the Chinese Restaurant Process prior ( Aldous , 1985 ) , and ρ is a measure of the standard deviation for the base distribution from which clusters are assumed to be drawn . We estimate ρ as the variance in the labeled cluster means within an episode , while α is treated as a hyperparameter . In our experiments , we found a wide range of α values to give similar results , with the embeddings adjusting their overall magnitudes to match the magnitude of α .

Summary: This work proposes a learning method based on deep subspace clustering. The method is formulated by identifying a deep data embedding, where clustering is performed in the latent space by a revised version of k-means, inspired by the work [1]. In this way, the proposed method can adapt to account for uni-modal distributions. The authors propose some variations of the framework based on soft cluster assignments, and on cumulative learning of the cluster means.