Concept Learners for Few-Shot Learning

1 INTRODUCTION . Deep learning has reached human-level performance on domains with the abundance of large-scale labeled training data . However , learning on tasks with a small number of annotated examples is still an open challenge . Due to the lack of training data , models often overfit or are too simplistic to provide good generalization . On the contrary , humans can learn new tasks very quickly by drawing upon prior knowledge and experience . This ability to rapidly learn and adapt to new environments is a hallmark of human intelligence . Few-shot learning ( Miller et al. , 2000 ; Fei-Fei et al. , 2006 ; Koch et al. , 2015 ) aims at addressing this fundamental challenge by designing algorithms that are able to generalize to new tasks given only a few labeled training examples . Meta-learning ( Schmidhuber , 1987 ; Bengio et al. , 1992 ) has recently made major advances in the field by explicitly optimizing the model ’ s ability to generalize , or learning how to learn , from many related tasks ( Snell et al. , 2017 ; Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ) . Motivated by the way humans effectively use prior knowledge , meta-learning algorithms acquire prior knowledge over previous tasks so that new tasks can be efficiently learned from a small amount of data . However , recent works ( Chen et al. , 2019b ; Raghu et al. , 2020 ) show that simple baseline methods perform comparably to existing meta-learning methods , opening the question about which components are crucial for rapid adaptation and generalization . Here , we argue that there is an important missing piece in this puzzle . Human knowledge is structured in the form of reusable concepts . For instance , when we learn to recognize new bird species we are already equipped with the critical concepts , such as wing , beak , and feather . We then focus on these specific concepts and combine them to identify a new species . While learning to recognize new species is challenging in the complex bird space , it becomes remarkably simpler once the reasoning is structured into familiar concepts . Moreover , such a structured way of cognition gives us the ability to provide reasoning behind our decisions , such as “ ravens have thicker beaks than crows , with more ∗The two first authors made equal contributions . of a curve to the end ” . We argue that this lack of structure is limiting the generalization ability of the current meta-learners . The importance of compositionality for few-shot learning was emphasized in ( Lake et al. , 2011 ; 2015 ) where hand-designed features of strokes were combined using Bayesian program learning . Motivated by the structured form of human cognition , we propose COMET , a meta-learning method that discovers generalizable representations along human-interpretable concept dimensions . COMET learns a unique metric space for each concept dimension using concept-specific embedding functions , named concept learners , that are parameterized by deep neural networks . Along each high-level dimension , COMET defines concept prototypes that reflect class-level differences in the metric space of the underlying concept . To obtain final predictions , COMET effectively aggregates information from diverse concept learners and concept prototypes . Three key aspects lead to a strong generalization ability of our approach : ( i ) semi-structured representation learning , ( ii ) concept-specific metric spaces described with concept prototypes , and ( iii ) ensembling of many models . The latter assures that the combination of diverse and accurate concept learners improves the generalization ability of the base learner ( Hansen & Salamon , 1990 ; Dvornik et al. , 2019 ) . Remarkably , the high-level universe of concepts that are used to guide our algorithm can be discovered in a fully unsupervised way , or we can use external knowledge bases to define concepts . In particular , we can get a large universe of noisy , incomplete and redundant concepts and COMET learns which subsets of those are important by assigning local and global concept importance scores . Unlike existing methods ( Snell et al. , 2017 ; Vinyals et al. , 2016 ; Sung et al. , 2018 ; Gidaris & Komodakis , 2018 ) , COMET ’ s predictions are interpretable—an advantage especially important in the few-shot learning setting , where predictions are based only on a handful of labeled examples making it hard to trust the model . As such , COMET is the first domain-agnostic interpretable meta-learning approach . We demonstrate the effectiveness of our approach on tasks from extremely diverse domains , including fine-grained image classification in computer vision , document classification in natural language processing , and cell type annotation in biology . In the biological domain , we conduct the first systematic comparison of meta-learning algorithms . We develop a new meta-learning dataset and define a novel benchmark task to characterize single-cell transcriptome of all mouse organs ( Consortium , 2018 ; 2020 ) . Additionally , we consider the scenario in which concepts are not given in advance , and test COMET ’ s performance with automatically extracted visual concepts . Our experimental results show that on all domains COMET significantly improves generalization ability , achieving 6–15 % relative improvement over state-of-the-art methods in the most challenging 1-shot task . Furthermore , we demonstrate the ability of COMET to provide interpretations behind the model ’ s predictions , and support our claim with quantitative and qualitative evaluations of the generated explanations . 2 PROPOSED METHOD . Problem formulation . In few-shot classification , we assume that we are given a labeled training set Dtr , an unlabeled query set Dqr , and a support set S consisting of a few labeled data points that share the label space with the query set . Label space between training and query set is disjoint , i.e. , { Ytr } ∩ { Yqr } = ∅ , where { Ytr } denotes label space of training set and { Yqr } denotes label space of query set . Each labeled data point ( x , y ) consists of a D-dimensional feature vector x ∈ RD and a class label y ∈ { 1 , ... , K } . Given a training set of previously labeled tasks Dtr and the support set S of a few labeled data points on a novel task , the goal is to train a model that can generalize to the novel task and label the query set Dqr . 2.1 PRELIMINARIES . Episodic training . To achieve successful generalization to a new task , training of meta-learning methods is usually performed using sampled mini-batches called episodes ( Vinyals et al. , 2016 ) . Each episode is formed by first sampling classes from the training set , and then sampling data points labeled with these classes . The sampled data points are divided into disjoint sets of : ( i ) a support set consisting of a few labeled data points , and ( ii ) a query set consisting of data points whose labels are used to calculate a prediction error . Given the sampled support set , the model minimizes the loss on the sampled query set in each episode . The key idea behind this meta-learning training scheme is to improve generalization of the model by trying to mimic the low-data regime encountered during testing . Episodes with balanced training sets are usually referred to as “ N-way , k-shot ” episodes where N indicates number of classes per episode ( “ way ” ) , and k indicates number of support points ( labeled training examples ) per class ( “ shot ” ) . Prototypical networks . Our work is inspired by prototypical networks ( Snell et al. , 2017 ) , a simple but highly effective metric-based meta-learning method . Prototypical networks learn a non-linear embedding function fθ : RD → RM parameterized by a convolutional neural network . The main idea is to learn a function fθ such that in the M -dimensional embedding space data points cluster around a single prototype representation pk ∈ RM for each class k. Class prototype pk is computed as the mean vector of the support set labeled with the class k : pk = 1 |Sk| ∑ ( xi , yi ) ∈Sk fθ ( xi ) , ( 1 ) where Sk denotes the subset of the support set S belonging to the class k. Given a query data point xq , prototypical networks output distribution over classes using the softmax function : pθ ( y = k|xq ) = exp ( −d ( fθ ( xq ) , pk ) ) ∑ k′ exp ( −d ( fθ ( xq ) , pk′ ) ) , ( 2 ) where d : RM → R denotes the distance function . Query data point xq is assigned to the class with the minimal distance between the class prototype and embedded query point . 2.2 META-LEARNING VIA CONCEPT LEARNERS . Our main assumption is that input dimensions can be separated into subsets of related dimensions corresponding to high-level , human-interpretable concepts that guide the training . Such sets of potentially overlapping , noisy and incomplete human-interpretable dimensions exists in many realworld scenarios . For instance , in computer vision concepts can be assigned to image segments ; in natural language processing to semantically related words ; whereas in biology we can use external knowledge bases and ontologies . In many problems , concepts are already available as a prior domain knowledge ( Ashburner et al. , 2000 ; Murzin et al. , 1995 ; Wah et al. , 2011 ; Mo et al. , 2019 ; Miller et al. , 2000 ) , or can be automatically generated using existing techniques ( Blei et al. , 2003 ; Zhang et al. , 2018 ; Jakab et al. , 2018 ) . Intuitively , concepts can be seen as part-based representations of the input and reflect the way humans reason about the world . Importantly , we do not assume these concepts are clean or complete . On the contrary , we show that even if there are thousands of concepts , which are noisy , incomplete , overlapping , or redundant , they still provide useful guidance to the meta-learning algorithm . Formally , let C = { c ( j ) } Nj=1 denote a set of N concepts given/extracted as a prior knowledge , where each concept c ( j ) ∈ { 0 , 1 } D is a binary vector such that c ( j ) i = 1 if i-th dimension should be used to describe the j-th concept and D denotes the dimensionality of the input . We do not impose any constraints on C , meaning that the concepts can be disjoint or overlap . Instead of learning single mapping function fθ : RD → RM across all dimensions , COMET separates original space into subspaces of predefined concepts and learns individual embedding functions f ( j ) θ : RD → RM for each concept j ( Figure 1 ) . Concept embedding functions f ( j ) θ , named concept learners , are non-linear functions parametrized by a deep neural network . Each concept learner j produces its own concept prototypes p ( j ) k for class k computed as the average of concept embeddings of data points in the support set : p ( j ) k = 1 |Sk| ∑ ( xi , yi ) ∈Sk f ( j ) θ ( xi ◦ c ( j ) ) , ( 3 ) where ◦ denotes Hadamard product . As a result , each class k is represented with a set of N concept prototypes { p ( j ) k } Nj=1 . Given a query data point xq , we compute its concept embeddings and estimate their distances to the concept prototypes of each class . We then aggregate the information across all concepts by taking sum over distances between concept embeddings and concept prototypes . Specifically , for each concept embedding f ( j ) θ ( xq ◦ c ( j ) ) we compute its distance to concept prototype p ( j ) k of a given class k , and sum distances across all concepts to obtain a distribution over support classes . The probability of assigning query point xq to k-th class is then given by : pθ ( y = k|xq ) = exp ( − ∑ j d ( f ( j ) θ ( xq ◦ c ( j ) ) , p ( j ) k ) ) ∑ k′ exp ( − ∑ j d ( f ( j ) θ ( xq ◦ c ( j ) ) , p ( j ) k′ ) ) . ( 4 ) The loss is computed as the negative log-likelihood Lθ = − log pθ ( y = k|xq ) of the true class , and COMET is trained by minimizing the loss on the query samples of training set in the episodic fashion ( Snell et al. , 2017 ; Vinyals et al. , 2016 ) . In equation ( 4 ) , we use euclidean distance as the distance function . Experimentally , we find that it outperforms cosine distance ( Appendix B ) , which agrees with the theory and experimental findings in ( Snell et al. , 2017 ) . We note that in order for distances to be comparable , it is crucial to normalize neural network layers using batch normalization ( Ioffe & Szegedy , 2015 ) .

Summary: The paper presents a knowledge-driven prototypical learning strategy for few-shot classification tasks. The main idea of this work is to introduce a set of concepts defined in the subspaces of inputs and represent each class as a group of concept prototypes for few-shot learning. Following the prototypical networks, the method first computes the concept embeddings of an input, and then takes the summation of the distances between those embeddings and their corresponding concept prototypes in each class to estimate the class probability.  The experiments validates the proposed methods on 4 benchmarks in three different domains, including vision, language and biology. For the biology task, the authors also develop a new benchmark on cross-organ cell type classification.