Distribution Embedding Network for Meta-Learning with Variable-Length Input

1 INTRODUCTION . Deep learning has made substantial progress in a variety of tasks in image classification ( e.g. , He et al. , 2016 ) , object detection ( e.g. , Redmon & Farhadi , 2017 ; He et al. , 2017 ) , machine translation ( e.g. , Vaswani et al. , 2017 ) and natural language understanding ( e.g. , Devlin et al. , 2019 ) . These achievements rely on efficient gradient-based optimization algorithms ( e.g. , Duchi et al. , 2011 ; Sutskever et al. , 2013 ; Kingma & Ba , 2015 ) as well as a large number of labeled examples to train highly flexible deep learning models . However , in many applications , it is prohibitively expensive or impossible to collect a large amount of labeled training data , calling for techniques that can learn from small labeled data . Meta-learning aims to tackle the small data problem by training a model on labeled data from a number of related tasks , with the goal to learn a model that can perform well on similar , but unseen future tasks with only a small amount of labeled training data . In this work , we propose a meta-learning model for classification using Distribution Embedding Networks ( DEN ) . Unlike many existing meta-learning algorithms that assume a fixed feature set across tasks , DEN is designed for applications where both the distribution of features and the number of features could vary across tasks . For example , we may use DEN to learn the optimal aggregator of an ensemble of models to replace the naive majority vote , where in different aggregation tasks , the distribution of model outputs and the number of models in the ensemble could be different . On a high level , DEN first applies a learned feature transformation that allows features to be transformed into the same distribution family across tasks . It then uses a neural network to learn an embedding of the transformed data distribution . Finally , given the learned distribution embedding , together with the transformed features , DEN classifies examples using a Deep Sets architecture ( Zaheer et al. , 2017 ) , enabling it to be applied to variable-length inputs . To adapt the model on a new task , we only update the feature transformations with relatively few parameters . 2 RELATED WORK . There are multiple generic techniques applied to the meta-learning problem in the literature . The first camp of approaches learn similarities between pairs of examples . When presented with a new task with a small set of labeled examples , these methods classify unlabeled data based on their similarities with labeled ones . These methods include Matching Net ( Vinyals et al. , 2016 ) and Prototypical Net ( Snell et al. , 2017 ) , which learn a distance metric between examples . Siamese Net ( Koch et al. , 2015 ) and Relation Net ( Sung et al. , 2018 ) use twin towers to learn the relationship between examples . Learn Net ( Bertinetto et al. , 2016 ) proposes having class specific weights for the towers . Satorras & Estrach ( 2018 ) learns the similarity metric using a Graph Neural Network , and Transductive Propagation Network ( Liu et al. , 2019 ) classifies all unlabeled data at once by exploring the manifold structure of the new class space . The second camp of optimization-based meta-learning aims to find a good starting point model , so that when presented with a new task , the meta model can quickly adapt to perform well on the new task with a small number of gradient steps . MAML ( Finn et al. , 2017 ) designs a learning algorithm , such that the expected loss of the learned meta-model on new tasks after one gradient step is minimized . Meta-Learner LSTM ( Ravi & Larochelle , 2017 ) modifies the classical gradient steps with learned gradient update weights , which are trained to minimize the validation loss . More recently , LEO ( Rusu et al. , 2019 ) extends MAML and utilizes Relation Net to learn a low-dimensional latent embedding of model parameters and performs optimization-based meta-learning from this space . Another camp of methods use internal or external memory for meta-learning . They include MANN ( Santoro et al. , 2016 ) and Meta Net ( Munkhdalai & Yu , 2017 ) , which store the past knowledge in external memory and internal model activations , respectively . New examples are classified by retrieving relevant information from the memory . Our proposal does not take the above three routes . Rather , DEN is conceptually similar to topic modeling that learns the latent context variable . For example , Neural Statistician ( Edwards & Storkey , 2017 ) considers the hierarchical generative process and uses variational autoencoder ( Kingma & Welling , 2014 ) to learn the latent vector that summarizes the dataset in an unsupervised fashion . Similar proposals include variational homoencoder ( Hewitt et al. , 2018 ) and CNP ( Garnelo et al. , 2018 ) . In comparison , DEN is a supervised procedure , which learns an embedding of the data distribution . We then utilize this distribution embedding for classification on unseen examples . 3 NOTATIONS . In this paper , we use bold upper case letters to denote matrices ( e.g. , X ) , bold lower case letters to denote vectors ( e.g. , x ) , italic lower case letters to denote scalars ( e.g. , x ) , normal text to denote random variables ( e.g. , x ) , and bold normal text to denote random vectors ( e.g. , x ) . Let T1 , . . . , TM be M training tasks , following some task distribution P . In training task Ti , we observe a set of ni independent features and label pairs , ( XTi , yTi ) , whereXTi = [ x 1 Ti , . . . , x di Ti ] ∈ Rni×di is the di dimensional real-valued feature matrix , x j Ti ∈ Rni is the j-th feature of task Ti and yTi ∈ { 0 , 1 } ni is the binary label vector . We assume that the label is binary for simplicity of presentation . Our proposed model can be trivially extended to multiclass classification problems . We use PTi to denote the joint distribution of ( xTi , yTi ) . We first train DEN on the training tasks { T1 , . . . , TM } . Given a new task S with a small set of labeled examples , we fine-tune the trained model on S using these labeled examples . The final model is then applied to unlabeled examples in S for classification . The set of labeled examples is called the support set and the set of unlabeled examples is called the query set . 4 DISTRIBUTION EMBEDDING NETWORK . To motivate our proposal , we first consider the problem of minimizing the risk on a single task T : θ̂T = arg min θ∈Θ E ( xT , yT ) ∼PT [ L ( f ( xT ; θ ) , yT ) ] , ( 1 ) where f ( · ; θ ) is a model with parameter θ and L is the loss function . Lemma 1 . Assume the joint distribution PT has a probability density ( mass ) function q ( · ; ηT ) . Then the optimizer θ̂T is of the form φ∗L , f , q ( ηT ) , where φ ∗ L , f , q is some deterministic function depending on the loss L , the model f and the density q . The proof of Lemma 1 can be found in Appendix A . It suggests that , when the joint distribution PT is in a parametric family , the dependency of θ̂T on the task T is through two parts : the functional form of the density q and the distribution parameter ηT . Now , if the joint distributions of all training tasks { T1 , . . . , TM } are in the same parametric family , then there exists a common function φ∗L , f , q such that θ̂Ti = φ ∗ L , f , q ( ηTi ) for every i ∈ { 1 , . . . , M } . Hence , we may reparameterize f ( xT ; φ∗L , f , q ( ηT ) ) as f ′ ( xT , ηT ; γ ) , and learn the new model parameter γ from observations ( XTi , yTi ) and some estimate η̂Ti . One advantage of this reparameterization is that the optimal choice of γ is now task-independent , so , once learned from training tasks , it can be transferred to new unseen tasks whose data distribution falls in the same parametric family . Moreover , since the data distributions of all tasks are in the same parametric family , we can learn a shared model to estimate η̂T for all T. In practice , the data distribution could vary greatly across tasks . Hence , for each task T , we propose to apply a transformation cT to the original features and work with ( cT ( xT ) , yT ) , allowing it to unify the distribution family of ( cT ( xT ) , yT ) . To summarize , our proposal , DEN , consists of three building blocks . Firstly , we apply a transformation layer cT to the original features . Secondly , we use a distribution embedding module to obtain a distribution embedding sT of the transformed data ( cT ( xT ) , yT ) . This corresponds to the distribution parameter ηT discussed above . Finally , we use a classification module to output a prediction based on the transformed feature cT ( xT ) and the distribution embedding sT . Here the last two modules are shared across tasks . 4.1 LEARNING THE DISTRIBUTION EMBEDDING . To illustrate the idea of learning a compact distribution embedding , we first consider a special case that xT| ( yT = k ) ∼ Nd ( µT , k , ΣT , k ) for k ∈ { 0 , 1 } . With the normality assumption , PT is uniquely characterized by its first moment vectors E [ xT| ( yT = k ) ] , second moment matrices E [ xTx > T | ( yT = k ) ] , and the label positive rate P ( yT = 1 ) . Given an i.i.d . sample ( XT , yT ) where XT : = [ x 1 T , . . . , x d T ] ∈ Rn×d and yT ∈ { 0 , 1 } n , we may estimate these quantities by sample moments , leading to a compact distribution embedding sT = [ x1T,0 , . . . , x d T,0 , x 1 T,0 x1T,0 , . . . , x1T,0 xdT,0 , . . . , xdT,0 xdT,0 , x1T,1 , . . . , x d T,1 , x 1 T,1 x1T,1 , . . . , x1T,1 xdT,1 , . . . , xdT,1 xdT,1 , yT ] , ( 2 ) wherexjT , k denotes the j-th feature vector of examples with label k , x denotes the arithmetic average of its elements for a vector x , and is the element-wise product . There are two notable characteristics in this distribution embedding . Firstly , every element in sT is of an average form over examples within the same class . This motivates us to use a batch average layer in DEN when obtaining the distribution embedding . Secondly , the embedding sT only involves quantities of single features and pairs of features . As discussed in Section 4.1.2 , this allows us to decompose the whole distribution embedding vector of task-dependent dimension ( e.g. , 2d2 +2d+1 for Gaussian data ) into sub-vectors of fixed dimension , opening up an opportunity to handle variablelength features using a Deep Sets architecture ( Zaheer et al. , 2017 ) .

Summary: The authors proposed a method for meta-learning that produces a distributional embedding of the task, and then uses this information to perform few-shot classification. The method decouples feature extraction from feature aggregation & few-shot reasoning within the given task. The authors conduct experiments to show that their method is competitive with a few other methods from the literature on a number of datasets.