Information Theoretic Meta Learning with Gaussian Processes

1 INTRODUCTION . Meta learning ( Ravi & Larochelle , 2017 ; Vinyals et al. , 2016 ; Edwards & Storkey , 2017 ; Finn et al. , 2017 ; Lacoste et al. , 2019 ; Nichol et al. , 2018 ) and few-shot learning ( Li et al. , 2006 ; Lake et al. , 2011 ) aim to derive data efficient learning algorithms that can rapidly adapt to new tasks . Such systems require training deep neural networks from a set of tasks drawn from a common distribution , where each task is described by a small amount of experience , typically divided into a training or support set and a validation set . By sharing information across tasks the neural network can learn to rapidly adapt to new tasks and generalize from few examples at test time . Several few-shot learning algorithms use memory-based ( Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ) or gradient-based procedures ( Finn et al. , 2017 ; Nichol et al. , 2018 ) , with the gradient-based model agnostic meta learning algorithm ( MAML ) by Finn et al . ( 2017 ) being very influential in the literature . Despite the success of specific schemes , one fundamental issue in meta learning is concerned with deriving unified principles that can allow to relate different approaches and invent new schemes . While there exist probabilistic interpretations of existing methods , such as the approximate Bayesian inference approach ( Grant et al. , 2018 ; Finn et al. , 2018 ; Yoon et al. , 2018 ) and the related conditional probability modelling approach ( Garnelo et al. , 2018 ; Gordon et al. , 2019 ) , meta learning still lacks of a general and tractable learning principle that can help to get a better understanding of existing algorithms and derive new methods . To this end , the main contribution of this paper is to introduce an information theoretic view of meta learning , by utilizing tools such as the mutual information and the information bottleneck ( Cover & Thomas , 2006 ; Tishby et al. , 1999 ) . Given that each task consists of a support or training set and a target or validation set , we consider the information bottleneck principle , introduced by Tishby et al . ( 1999 ) , which can learn a stochastic encoding of the support set that is highly informative about predicting the validation set . Such stochastic encoding is optimized through the difference between two mutual informations , so that the encoding compresses the training set into a representation that can predict well the validation set . By exploiting recent variational approximations to the information bottleneck ( Alemi et al. , 2017 ; Chalk et al. , 2016 ; Achille & Soatto , 2016 ) that make use of variational lower bounds on the mutual information ( Barber & Agakov , 2003 ) , we derive a general and tractable framework for meta learning . Such framework can allow us to re-interpret gradient-based algorithms , such as MAML , and also derive new methods . Based on the variational information bottleneck ( VIB ) framework ( Alemi et al. , 2017 ; Chalk et al. , 2016 ; Achille & Soatto , 2016 ) , we introduce a new memory-based algorithm for supervised fewshot learning ( right panel in Figure 1 ) based on Gaussian processes ( Rasmussen & Williams , 2006 ) and deep neural kernels ( Wilson et al. , 2016 ) that offers a kernel-based Bayesian view of a memory system . With Gaussian processes , the underlying encoding takes the form of a non-parametric function that follows a stochastic process amortized by the training set . Further , we show that VIB gives rise to gradient-based meta learning methods , such as MAML , when combined with parametric encodings corresponding to model parameters or weights , and based on this we derive a stochastic MAML algorithm . In an additional scheme , we show that our framework can naturally allow for combinations of memory and gradient-based meta learning by constructing suitable encodings , and we derive such an algorithm that combines Gaussian processes with MAML . We demonstrate our methods on few-shot regression and classification by using standard benchmarks such as Omniglot , mini-Imagenet and Augmented Omniglot . 2 META LEARNING WITH INFORMATION BOTTLENECK . Suppose we wish to learn from a distribution of tasks . During training for each task we observe a pair consisted of a task description represented by the support or training set Dt and task validation represented by the target or validation set Dv . At test time only Dt will be given and the learning algorithm should rapidly adapt to form predictions on Dv or on further test data . We wish to formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck ( Tishby et al. , 1999 ) . The idea is to learn a stochastic representation or encoding of the task descriptionDt that is highly informative about predictingDv . We introduce a random variable , Z , associated with this encoding drawn from a distribution qw ( Z|Dt ) parametrized by w. Given this encoding the full joint distribution is written as qw ( Dv , Dt , Z ) = qw ( Z|Dt ) p ( Dv , Dt ) , ( 1 ) where p ( Dv , Dt ) denotes the unknown data distribution overDt andDv . In equation 1 and throughout the paper we use the convention that the full joint as well as any marginal or conditional that depends on Z is denoted by qw ( emphasizing the dependence on the parametrized encoder ) , while corresponding quantities over data Dt , Dv are denoted by p. Eg . from the above we can express a Z-dependent marginal such as , qw ( Z , Dv ) = ∫ qw ( Z|Dt ) p ( Dv , Dt ) dDt . To tune w we would like to maximize the mutual information between Z and the target set Dv , denoted by I ( Z , Dv ) . A trivial way to obtain a maximally informative representation is to set Z = Dt , which does not provide a useful representation . Thus , the information bottleneck ( IB ) principle ( Tishby et al. , 1999 ) adds a model complexity penalty to the maximization of I ( Z , Dv ) which promotes an encoding Z that is highly compressive of Dt , i.e . for which I ( Z , Dt ) is minimized . This leads to the IB objective : LIB ( w ) = I ( Z , Dv ) − βI ( Z , Dt ) , ( 2 ) where β ≥ 0 is a hyperparameter . Nevertheless , in order to use IB for meta learning we need to approximate the mutual information terms I ( Z , Dv ) and I ( Z , Dt ) , which are both intractable since they depend on the unknown data distribution p ( Dv , Dt ) . To overcome this , we will consider variational approximations by following similar arguments to the variational IB approach ( Alemi et al. , 2017 ) that was introduced for supervised learning of a single task , which allows us to express a tractable lower bound on LIB ( w ) by lower bounding I ( Z , Dv ) and upper bounding I ( Z , Dt ) . 2.1 VARIATIONAL INFORMATION BOTTLENECK ( VIB ) FOR META LEARNING . To construct a bound F ≤ LIB ( w ) we need first to lower bound I ( Z , Dv ) , which is written as I ( Z , Dv ) = KL [ qw ( Z , Dv ) ||qw ( Z ) p ( Dv ) ] = Eqw ( Z , Dv ) [ log qw ( Dv|Z ) p ( Dv ) ] , ( 3 ) where KL denotes the Kullback-Leibler divergence and qw ( Dv|Z ) = ∫ qw ( Z|Dt ) p ( Dv , Dt ) dDt∫ qw ( Z|Dt ) p ( Dv , Dt ) dDtdDv is intractable since we do not know the analytic form of the data distribution p ( Dv , Dt ) . To lower bound I ( Z , Dv ) , we follow Barber & Agakov ( 2003 ) ( see Appendix A.1 ) by introducing a decoder model pθ ( Dv|Z ) to approximate the intractable qw ( Dv|Z ) where θ are additional parameters1 , I ( Z , Dv ) ≥ Eqw ( Z , Dv ) [ log pθ ( Dv|Z ) p ( Dv ) ] = Eqw ( Z , Dv ) [ log pθ ( D v|Z ) ] +H ( Dv ) , ( 4 ) where the entropy H ( Dv ) is just a constant that does not depend on the tunable parameters ( θ , w ) . Furthermore , to deal with the second intractable mutual information I ( Z , Dt ) so that to maintain an lower bound on LIB ( w ) we need to upper bound this term . Note that I ( Z , Dt ) = Eqw ( Z , Dt ) [ log qw ( Z|Dt ) qw ( Z ) ] , where qw ( Z ) = ∫ qw ( Z|Dt ) p ( Dt ) dDt is intractable since , e.g . it involves the unknown data distribution p ( Dt ) . By working similarly as before , we can approximate qw ( Z ) by a tractable prior model distribution pθ ( Z ) which leads to the following upper bound on the mutual information , I ( Z , Dt ) ≤ Eqw ( Z , Dt ) [ log qw ( Z|Dt ) pθ ( Z ) ] . ( 5 ) Then , by combining the two bounds we obtain the overall bound , F ( θ , w ) +H ( Dv ) ≤ LIB ( w ) : F ( θ , w ) = Eqw ( Z , Dv ) [ log pθ ( D v|Z ) ] − βEqw ( Z , Dt ) [ log qw ( Z|Dt ) pθ ( Z ) ] , where the constant H ( Dv ) is dropped from the objective function . Given a set of task pairs { Dti , Dvi } bi=1 , where each ( Dti , Dvi ) ∼ p ( Dv , Dt ) , during meta-training the objective function for learning ( θ , w ) reduces to the maximization of the empirical average , 1b ∑ i F̃i ( θ , w ) , where each F̃i ( θ , w ) is an unbiased estimate of F ( θ , w ) ( see Appendix A.2 ) and is given by F̃i ( w , θ ) = Eqw ( Zi|Dti ) [ log pθ ( D v i |Zi ) ] − βKL [ qw ( Zi|Dti ) ||pθ ( Zi ) ] . ( 6 ) The meta-training procedure is carried out in different episodes where at each step we receive a minibatch of task pairs and perform a stochastic gradient maximization step . The objective in equation 6 is similar to variational inference objectives for meta learning ( Ravi & Beatson , 2019 ) . In particular , it can be viewed as an evidence lower bound ( ELBO ) on the validation set log marginal likelihood , log ∫ pθ ( Dvi |Zi ) pθ ( Zi ) dZi , with the differences : ( i ) there is the hyperparameter β in front of the KL term and ( ii ) the variational distribution qw ( Zi|Dti ) in equation 6 is more restricted than in standard variational inference , since qw ( Zi|Dti ) now acts as a stochastic bottleneck that encodes the support set Dti ( i.e . it is amortized by Dti ) and via the term Eqw ( Zi|Dti ) [ log pθ ( D v i |Zi ) ] it is optimized to reconstruct the validation set .

Summary: The paper presents a method for Bayesian meta-learning. This method combines a NN feature extractor with a Gaussian Process on top. The GP kernel is linear. The information bottleneck is used to motivate a choice of approximate posterior. Using MAML to adapt the NN feature extractor weights improves performance of the composite model for few-shot learning.