Distribution Embedding Network for Meta-Learning with Variable-Length Input

1 INTRODUCTION . Deep learning has made substantial progress in a variety of tasks in image classification ( e.g. , He et al. , 2016 ) , object detection ( e.g. , Redmon & Farhadi , 2017 ; He et al. , 2017 ) , machine translation ( e.g. , Vaswani et al. , 2017 ) and natural language understanding ( e.g. , Devlin et al. , 2019 ) . These achievements rely on efficient gradient-based optimization algorithms ( e.g. , Duchi et al. , 2011 ; Sutskever et al. , 2013 ; Kingma & Ba , 2015 ) as well as a large number of labeled examples to train highly flexible deep learning models . However , in many applications , it is prohibitively expensive or impossible to collect a large amount of labeled training data , calling for techniques that can learn from small labeled data . Meta-learning aims to tackle the small data problem by training a model on labeled data from a number of related tasks , with the goal to learn a model that can perform well on similar , but unseen future tasks with only a small amount of labeled training data . In this work , we propose a meta-learning model for classification using Distribution Embedding Networks ( DEN ) . Unlike many existing meta-learning algorithms that assume a fixed feature set across tasks , DEN is designed for applications where both the distribution of features and the number of features could vary across tasks . For example , we may use DEN to learn the optimal aggregator of an ensemble of models to replace the naive majority vote , where in different aggregation tasks , the distribution of model outputs and the number of models in the ensemble could be different . On a high level , DEN first applies a learned feature transformation that allows features to be transformed into the same distribution family across tasks . It then uses a neural network to learn an embedding of the transformed data distribution . Finally , given the learned distribution embedding , together with the transformed features , DEN classifies examples using a Deep Sets architecture ( Zaheer et al. , 2017 ) , enabling it to be applied to variable-length inputs . To adapt the model on a new task , we only update the feature transformations with relatively few parameters . 2 RELATED WORK . There are multiple generic techniques applied to the meta-learning problem in the literature . The first camp of approaches learn similarities between pairs of examples . When presented with a new task with a small set of labeled examples , these methods classify unlabeled data based on their similarities with labeled ones . These methods include Matching Net ( Vinyals et al. , 2016 ) and Prototypical Net ( Snell et al. , 2017 ) , which learn a distance metric between examples . Siamese Net ( Koch et al. , 2015 ) and Relation Net ( Sung et al. , 2018 ) use twin towers to learn the relationship between examples . Learn Net ( Bertinetto et al. , 2016 ) proposes having class specific weights for the towers . Satorras & Estrach ( 2018 ) learns the similarity metric using a Graph Neural Network , and Transductive Propagation Network ( Liu et al. , 2019 ) classifies all unlabeled data at once by exploring the manifold structure of the new class space . The second camp of optimization-based meta-learning aims to find a good starting point model , so that when presented with a new task , the meta model can quickly adapt to perform well on the new task with a small number of gradient steps . MAML ( Finn et al. , 2017 ) designs a learning algorithm , such that the expected loss of the learned meta-model on new tasks after one gradient step is minimized . Meta-Learner LSTM ( Ravi & Larochelle , 2017 ) modifies the classical gradient steps with learned gradient update weights , which are trained to minimize the validation loss . More recently , LEO ( Rusu et al. , 2019 ) extends MAML and utilizes Relation Net to learn a low-dimensional latent embedding of model parameters and performs optimization-based meta-learning from this space . Another camp of methods use internal or external memory for meta-learning . They include MANN ( Santoro et al. , 2016 ) and Meta Net ( Munkhdalai & Yu , 2017 ) , which store the past knowledge in external memory and internal model activations , respectively . New examples are classified by retrieving relevant information from the memory . Our proposal does not take the above three routes . Rather , DEN is conceptually similar to topic modeling that learns the latent context variable . For example , Neural Statistician ( Edwards & Storkey , 2017 ) considers the hierarchical generative process and uses variational autoencoder ( Kingma & Welling , 2014 ) to learn the latent vector that summarizes the dataset in an unsupervised fashion . Similar proposals include variational homoencoder ( Hewitt et al. , 2018 ) and CNP ( Garnelo et al. , 2018 ) . In comparison , DEN is a supervised procedure , which learns an embedding of the data distribution . We then utilize this distribution embedding for classification on unseen examples . 3 NOTATIONS . In this paper , we use bold upper case letters to denote matrices ( e.g. , X ) , bold lower case letters to denote vectors ( e.g. , x ) , italic lower case letters to denote scalars ( e.g. , x ) , normal text to denote random variables ( e.g. , x ) , and bold normal text to denote random vectors ( e.g. , x ) . Let T1 , . . . , TM be M training tasks , following some task distribution P . In training task Ti , we observe a set of ni independent features and label pairs , ( XTi , yTi ) , whereXTi = [ x 1 Ti , . . . , x di Ti ] ∈ Rni×di is the di dimensional real-valued feature matrix , x j Ti ∈ Rni is the j-th feature of task Ti and yTi ∈ { 0 , 1 } ni is the binary label vector . We assume that the label is binary for simplicity of presentation . Our proposed model can be trivially extended to multiclass classification problems . We use PTi to denote the joint distribution of ( xTi , yTi ) . We first train DEN on the training tasks { T1 , . . . , TM } . Given a new task S with a small set of labeled examples , we fine-tune the trained model on S using these labeled examples . The final model is then applied to unlabeled examples in S for classification . The set of labeled examples is called the support set and the set of unlabeled examples is called the query set . 4 DISTRIBUTION EMBEDDING NETWORK . To motivate our proposal , we first consider the problem of minimizing the risk on a single task T : θ̂T = arg min θ∈Θ E ( xT , yT ) ∼PT [ L ( f ( xT ; θ ) , yT ) ] , ( 1 ) where f ( · ; θ ) is a model with parameter θ and L is the loss function . Lemma 1 . Assume the joint distribution PT has a probability density ( mass ) function q ( · ; ηT ) . Then the optimizer θ̂T is of the form φ∗L , f , q ( ηT ) , where φ ∗ L , f , q is some deterministic function depending on the loss L , the model f and the density q . The proof of Lemma 1 can be found in Appendix A . It suggests that , when the joint distribution PT is in a parametric family , the dependency of θ̂T on the task T is through two parts : the functional form of the density q and the distribution parameter ηT . Now , if the joint distributions of all training tasks { T1 , . . . , TM } are in the same parametric family , then there exists a common function φ∗L , f , q such that θ̂Ti = φ ∗ L , f , q ( ηTi ) for every i ∈ { 1 , . . . , M } . Hence , we may reparameterize f ( xT ; φ∗L , f , q ( ηT ) ) as f ′ ( xT , ηT ; γ ) , and learn the new model parameter γ from observations ( XTi , yTi ) and some estimate η̂Ti . One advantage of this reparameterization is that the optimal choice of γ is now task-independent , so , once learned from training tasks , it can be transferred to new unseen tasks whose data distribution falls in the same parametric family . Moreover , since the data distributions of all tasks are in the same parametric family , we can learn a shared model to estimate η̂T for all T. In practice , the data distribution could vary greatly across tasks . Hence , for each task T , we propose to apply a transformation cT to the original features and work with ( cT ( xT ) , yT ) , allowing it to unify the distribution family of ( cT ( xT ) , yT ) . To summarize , our proposal , DEN , consists of three building blocks . Firstly , we apply a transformation layer cT to the original features . Secondly , we use a distribution embedding module to obtain a distribution embedding sT of the transformed data ( cT ( xT ) , yT ) . This corresponds to the distribution parameter ηT discussed above . Finally , we use a classification module to output a prediction based on the transformed feature cT ( xT ) and the distribution embedding sT . Here the last two modules are shared across tasks . 4.1 LEARNING THE DISTRIBUTION EMBEDDING . To illustrate the idea of learning a compact distribution embedding , we first consider a special case that xT| ( yT = k ) ∼ Nd ( µT , k , ΣT , k ) for k ∈ { 0 , 1 } . With the normality assumption , PT is uniquely characterized by its first moment vectors E [ xT| ( yT = k ) ] , second moment matrices E [ xTx > T | ( yT = k ) ] , and the label positive rate P ( yT = 1 ) . Given an i.i.d . sample ( XT , yT ) where XT : = [ x 1 T , . . . , x d T ] ∈ Rn×d and yT ∈ { 0 , 1 } n , we may estimate these quantities by sample moments , leading to a compact distribution embedding sT = [ x1T,0 , . . . , x d T,0 , x 1 T,0 x1T,0 , . . . , x1T,0 xdT,0 , . . . , xdT,0 xdT,0 , x1T,1 , . . . , x d T,1 , x 1 T,1 x1T,1 , . . . , x1T,1 xdT,1 , . . . , xdT,1 xdT,1 , yT ] , ( 2 ) wherexjT , k denotes the j-th feature vector of examples with label k , x denotes the arithmetic average of its elements for a vector x , and is the element-wise product . There are two notable characteristics in this distribution embedding . Firstly , every element in sT is of an average form over examples within the same class . This motivates us to use a batch average layer in DEN when obtaining the distribution embedding . Secondly , the embedding sT only involves quantities of single features and pairs of features . As discussed in Section 4.1.2 , this allows us to decompose the whole distribution embedding vector of task-dependent dimension ( e.g. , 2d2 +2d+1 for Gaussian data ) into sub-vectors of fixed dimension , opening up an opportunity to handle variablelength features using a Deep Sets architecture ( Zaheer et al. , 2017 ) .

Summary: This paper provides an interesting model that can be used for a meta-learning situation where both data distribution and the number of features vary across tasks. The proposed model referred to as ‘Distributed Embedding Network (DEN)’ transforms features to belong to the same distribution family, learn distribution embedding with them and process variable-length inputs by using DeepSets. Through experiments on binary classification tasks, this paper shows applying the proposed method on the problems outperforming meta-learning baselines.