Year: 2020

Abstract:
We formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training or support set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. We particularly develop new memorybased meta learning algorithms based on Gaussian processes and derive extensions that combine memory and gradient-based meta learning. We demonstrate our method on few-shot regression and classification by using standard benchmarks such as Omniglot, mini-Imagenet and Augmented Omniglot.

heading: 1 INTRODUCTION
text: Meta learning (Ravi & Larochelle, 2017; Vinyals et al., 2016; Edwards & Storkey, 2017; Finn et al., 2017; Lacoste et al., 2019; Nichol et al., 2018) and few-shot learning (Li et al., 2006; Lake et al., 2011) aim to derive data efficient learning algorithms that can rapidly adapt to new tasks. Such systems require training deep neural networks from a set of tasks drawn from a common distribution, where each task is described by a small amount of experience, typically divided into a training or support set and a validation set. By sharing information across tasks the neural network can learn to rapidly adapt to new tasks and generalize from few examples at test time.
Several few-shot learning algorithms use memory-based (Vinyals et al., 2016; Ravi & Larochelle, 2017) or gradient-based procedures (Finn et al., 2017; Nichol et al., 2018), with the gradient-based model agnostic meta learning algorithm (MAML) by Finn et al. (2017) being very influential in the literature. Despite the success of specific schemes, one fundamental issue in meta learning is concerned with deriving unified principles that can allow to relate different approaches and invent new schemes. While there exist probabilistic interpretations of existing methods, such as the approximate Bayesian inference approach (Grant et al., 2018; Finn et al., 2018; Yoon et al., 2018) and the related conditional probability modelling approach (Garnelo et al., 2018; Gordon et al., 2019), meta learning still lacks of a general and tractable learning principle that can help to get a better understanding of existing algorithms and derive new methods.
To this end, the main contribution of this paper is to introduce an information theoretic view of meta learning, by utilizing tools such as the mutual information and the information bottleneck (Cover & Thomas, 2006; Tishby et al., 1999). Given that each task consists of a support or training set and a target or validation set, we consider the information bottleneck principle, introduced by Tishby et al. (1999), which can learn a stochastic encoding of the support set that is highly informative about predicting the validation set. Such stochastic encoding is optimized through the difference between two mutual informations, so that the encoding compresses the training set into a representation that can predict well the validation set. By exploiting recent variational approximations to the information bottleneck (Alemi et al., 2017; Chalk et al., 2016; Achille & Soatto, 2016) that make use of variational lower bounds on the mutual information (Barber & Agakov, 2003), we derive a general and tractable framework for meta learning. Such framework can allow us to re-interpret gradient-based algorithms, such as MAML, and also derive new methods.
Based on the variational information bottleneck (VIB) framework (Alemi et al., 2017; Chalk et al., 2016; Achille & Soatto, 2016), we introduce a new memory-based algorithm for supervised fewshot learning (right panel in Figure 1) based on Gaussian processes (Rasmussen & Williams, 2006)
and deep neural kernels (Wilson et al., 2016) that offers a kernel-based Bayesian view of a memory system. With Gaussian processes, the underlying encoding takes the form of a non-parametric function that follows a stochastic process amortized by the training set. Further, we show that VIB gives rise to gradient-based meta learning methods, such as MAML, when combined with parametric encodings corresponding to model parameters or weights, and based on this we derive a stochastic MAML algorithm. In an additional scheme, we show that our framework can naturally allow for combinations of memory and gradient-based meta learning by constructing suitable encodings, and we derive such an algorithm that combines Gaussian processes with MAML. We demonstrate our methods on few-shot regression and classification by using standard benchmarks such as Omniglot, mini-Imagenet and Augmented Omniglot.

heading: 2 META LEARNING WITH INFORMATION BOTTLENECK
text: Suppose we wish to learn from a distribution of tasks. During training for each task we observe a pair consisted of a task description represented by the support or training set Dt and task validation represented by the target or validation set Dv . At test time only Dt will be given and the learning algorithm should rapidly adapt to form predictions on Dv or on further test data. We wish to formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck (Tishby et al., 1999). The idea is to learn a stochastic representation or encoding of the task descriptionDt that is highly informative about predictingDv . We introduce a random variable, Z, associated with this encoding drawn from a distribution qw(Z|Dt) parametrized by w. Given this encoding the full joint distribution is written as
qw(Dv,Dt, Z) = qw(Z|Dt)p(Dv,Dt), (1)
where p(Dv,Dt) denotes the unknown data distribution overDt andDv . In equation 1 and throughout the paper we use the convention that the full joint as well as any marginal or conditional that depends on Z is denoted by qw (emphasizing the dependence on the parametrized encoder), while corresponding quantities over data Dt,Dv are denoted by p. Eg. from the above we can express a Z-dependent marginal such as, qw(Z,Dv) = ∫ qw(Z|Dt)p(Dv,Dt)dDt.
To tune w we would like to maximize the mutual information between Z and the target set Dv , denoted by I(Z,Dv). A trivial way to obtain a maximally informative representation is to set Z = Dt, which does not provide a useful representation. Thus, the information bottleneck (IB) principle (Tishby et al., 1999) adds a model complexity penalty to the maximization of I(Z,Dv) which promotes an encoding Z that is highly compressive of Dt, i.e. for which I(Z,Dt) is minimized. This leads to the IB objective: LIB(w) = I(Z,Dv)− βI(Z,Dt), (2) where β ≥ 0 is a hyperparameter. Nevertheless, in order to use IB for meta learning we need to approximate the mutual information terms I(Z,Dv) and I(Z,Dt), which are both intractable since they depend on the unknown data distribution p(Dv,Dt). To overcome this, we will consider variational approximations by following similar arguments to the variational IB approach (Alemi
et al., 2017) that was introduced for supervised learning of a single task, which allows us to express a tractable lower bound on LIB(w) by lower bounding I(Z,Dv) and upper bounding I(Z,Dt).

heading: 2.1 VARIATIONAL INFORMATION BOTTLENECK (VIB) FOR META LEARNING
text: To construct a bound F ≤ LIB(w) we need first to lower bound I(Z,Dv), which is written as I(Z,Dv) = KL [qw(Z,Dv)||qw(Z)p(Dv)] = Eqw(Z,Dv) [ log qw(Dv|Z) p(Dv) ] , (3) where KL denotes the Kullback-Leibler divergence and qw(Dv|Z) = ∫ qw(Z|Dt)p(Dv,Dt)dDt∫
qw(Z|Dt)p(Dv,Dt)dDtdDv is intractable since we do not know the analytic form of the data distribution p(Dv,Dt). To lower bound I(Z,Dv), we follow Barber & Agakov (2003) (see Appendix A.1) by introducing a decoder model pθ(Dv|Z) to approximate the intractable qw(Dv|Z) where θ are additional parameters1,
I(Z,Dv) ≥ Eqw(Z,Dv) [ log pθ(Dv|Z) p(Dv) ] = Eqw(Z,Dv) [log pθ(D v|Z)] +H(Dv), (4)
where the entropy H(Dv) is just a constant that does not depend on the tunable parameters (θ, w). Furthermore, to deal with the second intractable mutual information I(Z,Dt) so that to maintain an lower bound on LIB(w) we need to upper bound this term. Note that
I(Z,Dt) = Eqw(Z,Dt) [ log
qw(Z|Dt) qw(Z)
] ,
where qw(Z) = ∫ qw(Z|Dt)p(Dt)dDt is intractable since, e.g. it involves the unknown data distribution p(Dt). By working similarly as before, we can approximate qw(Z) by a tractable prior model distribution pθ(Z) which leads to the following upper bound on the mutual information,
I(Z,Dt) ≤ Eqw(Z,Dt) [ log
qw(Z|Dt) pθ(Z)
] . (5)
Then, by combining the two bounds we obtain the overall bound, F(θ, w) +H(Dv) ≤ LIB(w):
F(θ, w) = Eqw(Z,Dv) [log pθ(D v|Z)]− βEqw(Z,Dt)
[ log
qw(Z|Dt) pθ(Z)
] ,
where the constant H(Dv) is dropped from the objective function. Given a set of task pairs {Dti ,Dvi }bi=1, where each (Dti ,Dvi ) ∼ p(Dv,Dt), during meta-training the objective function for learning (θ, w) reduces to the maximization of the empirical average, 1b ∑ i F̃i(θ, w), where each F̃i(θ, w) is an unbiased estimate of F(θ, w) (see Appendix A.2) and is given by
F̃i(w, θ) = Eqw(Zi|Dti) [log pθ(D v i |Zi)]− βKL[qw(Zi|Dti)||pθ(Zi)]. (6)
The meta-training procedure is carried out in different episodes where at each step we receive a minibatch of task pairs and perform a stochastic gradient maximization step.
The objective in equation 6 is similar to variational inference objectives for meta learning (Ravi & Beatson, 2019). In particular, it can be viewed as an evidence lower bound (ELBO) on the validation set log marginal likelihood, log ∫ pθ(Dvi |Zi)pθ(Zi)dZi, with the differences: (i) there is the hyperparameter β in front of the KL term and (ii) the variational distribution qw(Zi|Dti) in equation 6 is more restricted than in standard variational inference, since qw(Zi|Dti) now acts as a stochastic bottleneck that encodes the support set Dti (i.e. it is amortized by Dti) and via the term Eqw(Zi|Dti)[log pθ(D v i |Zi)] it is optimized to reconstruct the validation set.

heading: 2.2 INFORMATION THEORETIC VIEW OF MAML-TYPE METHODS
text: MAML (Finn et al., 2017) is a special case of our framework. To see this, suppose that the task encoding variable Zi for the i-th task coincides with a vector of some task-specific model parameters or neural network weights ψi, so that pθ(Dvi |Zi) ≡ p(Dvi |ψi) and pθ(Zi) reduces to a prior
1The lower bounds are valid even when the parameters w of the encoder qw(Z|Dt) and θ of the decoder pθ(Dv|Z) (and prior pθ(Z)) have shared components, e.g. are parameters of the same neural architecture.
pθ(ψi) over these parameters. The MAML approach (Finn et al., 2017) tries to find a shared initial parameter value θ so that few gradient steps based on the support set objective, log p(Dti |θ), lead to a task-specific parameter value ψi with good generalization on the validation set. MAML estimates the task parameters by ψi = θ+ ∆(θ,Dti), where ∆(θ,Dti) denotes the inner loop adaptation steps which for a single gradient step is just ρ∇θ log p(Dti |θ) and where ρ is a step size. By setting β = 0 and by using a deterministic Dirac measure encoder, δψi,θ+∆(θ,Dti), the VIB objective from equation 6 reduces to the standard MAML objective, F̃i(θ) = log p(Dvi |θ + ∆(θ,Dti)). We can construct a generalization of MAML by making the encoder stochastic, e.g. qθ,s(ψi|Dti) = N (ψi|θ + ∆(θ,Dti), sI) where s is a scalar variance parameter. Then, the objective becomes
F̃i(θ, s) = EN ( |0,I) [ log p(Dv|θ + ∆(θ,Dti) + √ s ) ] − βKL [ qθ,s(ψi|Dti)||pθ(ψi) ] ,
where we reparametrized the expectation suggesting the use of the reparametrization trick (Kingma & Welling, 2013) for stochastic optimization of the meta parameters (θ, s). In the experiments we will investigate an instance of the above where pθ(ψi) follows the hierarchical Gaussian form N (ψi|θ, sI), which views each task-specific parameter ψi as a randomized version of θ and with s being the same variance used by the encoder. For such prior the KL divergence term reduces to − 12s ||∆(θ,D t i)||2, which penalizes large values of the inner adaptation steps and small values of s.

heading: 2.3 VIB FOR SUPERVISED META LEARNING
text: Here, we explain how to adapt the VIB principle to supervised meta learning. Suppose the learning problem involves few-shot supervised learning where for each task we wish to predict outputs or labels given corresponding inputs. Let us denote the task support set as Dt = (Y t, Xt), where Y t = {ytj}n t j=1 and X t = {xtj}n t
j=1 denote the output and input observations. Similarly, we write Dv = (Y v, Xv) for the validation set. During meta-testing, for any novel task we observe the support set Dt∗ = (Y t∗ , Xt∗) together with the test inputs Xv∗ and the goal is to predict the test outputs Y v∗ . This suggests that we can construct a task encoder distribution of the form qw(Z|Y t, X) that depends on the training outputs Y t and generally on all inputs X = (Xt, Xv).2 We would like to train this encoder so that Z becomes highly predictive about the validation outputs Y v and simultaneously compressive about Y t. Then, a suitable VIB objective can be based on approximating the input-conditioned information bottleneck, I(Z, Y v|X)− βI(Z, Y t|X) i.e. where both I(Z, Y v|X) and I(Z, Y t|X) are conditional mutual informations (see Appendix A.3). By following similar arguments as those in Section 2.1, we can lower bound this objective and finally approximate it by an unbiased empirical average, 1b ∑b i=1 F̃i(θ, w), where
F̃i(θ, w) = Eqw(Zi|Y ti ,Xi) [log pθ(Y v i |Zi, Xi)]− βKL [ qw(Zi|Y ti , Xi)||pθ(Zi|Xi) ] , (7)
and where pθ(Y vi |Xvi , Zi) and pθ(Zi|Xi) are the decoder and prior model distributions introduced when applying the variational approximation. A detailed derivation can be found in Appendix A.3. Equation 7 provides the general form of the VIB objective suitable for supervised meta learning. The supervised version of MAML is expressed as a special case by following the arguments of Section 2.2. In Section 3, we particularize the above by combining it with a Gaussian process model.

heading: 3 SUPERVISED META LEARNING WITH GAUSSIAN PROCESSES
text: In this section we introduce VIB-based meta learning algorithms using Gaussian processes (GPs), which are suitable for few-shot supervised learning. In Section 3.1 we introduce a memory-based system, while in Section 3.2 we further combine it with gradient-based meta learning.

heading: 3.1 A GAUSSIAN PROCESS MEMORY-BASED METHOD
text: To use the VIB framework for few-shot supervised learning, as described in Section 2.3, for each i-th task we need to specify the encoding variable Zi together with the encoder qw(Zi|Y ti , Xi), the decoder over the validation outputs pθ(Y vi |Zi, Xi) and the prior model pθ(Zi|Xi). Here, we construct these quantities by using a GP model (Rasmussen & Williams, 2006).
2Dependence on all inputs can allow to explain both transductive and non-transductive settings for meta learning (Bronskill et al., 2020; Finn et al., 2017; Nichol et al., 2018) as special cases; see Appendix B.
We denote the unknown task-specific function that solves the i-th task by fi(x) and we assume that a priori (before observing any task data) this function is a draw from a GP, i.e. fi(x) ∼ GP(0, kθ(x, x′)), where kθ denotes the kernel function. Without loss of generality we shall use a deep kernel function where fi(x) is a linear function of a deep neural network feature vector φ(x; θ) with task-specific Gaussian weights, i.e. fi(x) = φ(x; θ)>θouti , θ out i ∼ N (θouti |0, σ2fI). Such function can be equivalently viewed as a GP sample: fi(x) ∼ GP(0, kθ(x, x′)), kθ(x, x′) = σ2fφ(x; θ)
>φ(x′; θ). In this functional space view the task-specific output weights θouti have been marginalized out and we are left with the feature vector parameters θ shared across tasks.3
Suppose now that we observe the task data, i.e. the support Dti = (Y ti , Xti ) and validation Dvi = (Y vi , X v i ) sets, so that we can evaluate the task function on all task inputs Xi = (X t i , X v i ). Let fvi,j ≡ f(xvi,j) denote the function value at the validation input xvi,j , associated with output yvi,j , while the vector of all such values is denoted by fvi = {fvi,j}n v
j=1. Similarly, the vector of function values at the training inputs Xti is f t i . In the VIB framework, we specify the task encoding variable Zi to be the full set of function values, Zi ≡ (fvi , f ti ), and we further choose the prior model for this encoding to be the GP prior, pθ(Zi|Xi) ≡ p(fvi , f ti |Xi), where
p(fvi , f t i |Xi) = p(fvi |f ti , Xi)× p(f ti |Xti ), (8)
= N (fvi |Kvti [Kti]−1f ti ,Kvi −Kvti [Kti]−1[Kvti ]>)×N (f ti |0,Kti). (9)
Here, Kti,K v i are square kernel matrices of size n t × nt and nv × nv on the training inputs Xti and validation inputs Xvi , while K vt i is the n
v×nt cross kernel matrix between these two sets of inputs. Note that the encoding is non-parametric since its size grows with the number of task data points.
To continue with the specification of the VIB objective, the second quantity we need to set is the decoder model pθ(Y vi |fvi , f ti , Xi) which is chosen to be a standard GP likelihood. Specifically, for i.i.d. observations Y vi given f v i becomes independent from f t i and Xi and the likelihood factor-
izes across data points, i.e. p(Y vi |fvi ) = ∏nv j=1 p(y v i,j |fvi,j), where each p(yvi,j |fvi,j) is a standard likelihood model, such as Gaussian density p(yvi,j |fvi,j) = N (yvi,j |fvi,j , σ2) suitable for standard regression problems or categorical/softmax likelihood for few-shot classification; see Appendix C.4. Finally, we specify the encoder distribution qw(Zi|Y ti , Xi) ≡ q(fvi , f ti |Y ti , Xi) as follows,
q(fvi , f t i |Y ti , Xi) = p(fvi |f ti , Xi)q(f ti |Dti), (10)
where p(fvi |f ti |Xi) is the same conditional GP prior from equation 8, while q(f ti |Dti) is a encoder of the training set that takes the form of a Gaussian distribution specified based on a very general amortization procedure, see Appendix C.1. Equation 10 shares a similar structure with a standard posterior Gaussian process where we first observe the training set, then we compute the (approximate) posterior distribution q(f ti |Dti), and finally we extrapolate/predict the validation set function values at inputs Xvi based on the conditional GP prior p(f v i |f ti , Xi). The above assumptions yield (see Appendix C.2) the following VIB objective for a single task,∑nv j=1 Eq(fvi,j)[log p(y v i,j |fvi,j)]− βKL [ q(f ti |Dti)||p(f ti |Xti ) ] , (11)
where q(fvi,j) = ∫ p(fvi,j |f ti , xvi,j , Xti )q(f ti |Dti)df ti is a univariate Gaussian over an individual validation function value fvi,j . Here, q(f v i,j) depends on the training set and the single validation input xvi,j , so that from the training set and the corresponding function values f t i we extrapolate (through the univariate conditional GP p(fvi,j |f ti , xvi,j , Xti )) to the input xvi,j in order to predict its value fvi,j .

heading: 3.2 COMBINATION WITH GRADIENT-BASED META LEARNING
text: In this section, we combine the GP memory-based meta learning method with a gradient-based approach, such as MAML (Finn et al., 2017). Based on the VIB principle we need to specify an encoding that will allow us to combine a memory-based with a gradient-based approach. As discussed in Section 2.2, gradient-based meta learning is associated with a parametric encoding that corresponds to a fixed-size task model parameter ψi. In contrast, as seen in Section 3.1, a memory system is associated with a function-space or non-parametric encoding that consists of the
3The kernel variance parameter σ2f (if learnable) is also considered to be part of the full set of parameters θ.
function values (f ti , f v i ) of all task input points. Therefore, a way to combine these techniques is to concatenate the encodings, i.e. Zi ≡ (ψi, f ti , fvi ). Here, ψi are the task-specific parameters of the GP kernel function kψi(x, x
′) (and possibly of the likelihood p(y|f)), which in our implementation are the parameters of the feature vector φ(x;ψi). Intuitively, a combination of the GP memorybased method with MAML will try to apply a short inner adaptation loop in order to adjust an initial feature vector φ(x; θ) to obtain a final φ(x;ψi) that can better solve the task. For the overall encoding (ψi, f ti , f v i ), the general form of the encoder distribution takes the form
q(f ti , f v i , ψi|Y ti , Xi) = p(fvi |f ti , ψi, Xi)q(f ti |ψi,Dti)q(ψi|Dti),
where p(fvi |f ti , ψi, Xi) is the conditional GP prior and q(f ti |ψi,Dt) is the amortized encoder (see Appendix C.1), where we have emphasized their dependence on ψi. The VIB objective becomes
Eq(ψi|Dti) nv∑ j=1 Eq(fvi,j |ψi)[logp(y v i,j |fvi,j)]−βKL[q(f ti |ψi,Dti)||p(f ti |ψi,Xti )] −βKL[q(ψi|Dti)||p(ψi)]. In practise, we can relax this objective and use different hyperparameters βf and βψ in front of the two KL terms. This is convenient as we would prefer to set βψ = 0 and use a deterministic MAML w.r.t. ψi rather than a stochastic MAML; see Section 2.2. This simplification avoids the need to specify a prior p(ψi) over the task-specific neural network parameters and at the same time it reduces the encoder q(ψi|Dti) to a Dirac delta, which simplifies the objective as follows,∑nv
j=1 Eq(fvi,j |ψi)[log p(y v i,j |fvi,j)]− βfKL
[ q(f ti |ψi,Dti)||p(f ti |ψi, Xti ) ] ,
where ψi = θ + ∆(θ,Dti). The inner loop adaptation term can be defined by an objective function on the support set Dti . In our case a suitable objective is the VIB for the GP memory-based method obtained by setting the validation set equal to the training set, i.e. Dvi = D t i in equation 11, which
gives ∑nt j=1 Eq(fti,j |θ)[log p(y t i,j |f ti,j)]− βfKL [q(f ti |θ,Dti)||p(f ti |θ,Xti )].

heading: 4 RELATED WORK
text: In this work, the VIB principle was used to formulate meta learning. VIB has been used before for different purposes, such as for regularization of single-task supervised learning (Alemi et al., 2017), sparse coding (Chalk et al., 2016), re-interpretation of β-VAEs and Dropout (Burgess et al., 2018; Achille & Soatto, 2016) and for compression of deep neural networks (Dai et al., 2018).
A meta learning method that connects with the information bottleneck was recently proposed by Hu et al. (2020). There the information bottleneck was used to analyze the generalization of a variational Bayesian inference objective suitable for transductive supervised few-shot learning. Note that the information bottleneck derived in Hu et al. (2020) (theorem 1) is not the same as the information bottleneck objective used here (i.e. the objective in equation 7 for the supervised learning case), since they differ in the second term. In addition, our framework expresses a general information bottleneck principle for meta learning applicable beyond transductive supervised learning.
Given the probabilistic nature of our framework, we can relate it to other probabilistic or Bayesian approaches and particularly with those that: (i) probabilistically re-interpret and extend gradientbased methods (Grant et al., 2018; Finn et al., 2018; Yoon et al., 2018; Nguyen et al., 2019; Gordon et al., 2019; Chen et al., 2019) and (ii) derive amortized conditional probability models (Garnelo et al., 2018; Gordon et al., 2019). The underlying learning principle in both (i)(ii) is to construct and maximize a predictive distribution (or conditional marginal likelihood) of the validation points given the training points, which, e.g. in supervised few-shot learning is written as pθ(Y v|Xv, Xt, Y t) = ∫ p(Y v|Xv, ψi)pθ(ψi|Xt, Y t)dψi = pθ(Y
v,Y t|Xv,Xt) pθ(Y t|Xt) . Here,
pθ(ψi|Xt, Y t) is a posterior distribution over the task parameters ψi, after observing the training points, and θ is a meta parameter which for simplicity we assume to be found by point estimation. However, this objective is very hard to rigorously approximate. Unlike the marginal likelihood on all task outputs pθ(Y v, Y t|Xv, Xt) for which we can easily compute a lower bound, there is no tractable lower bound on the predictive conditional pθ(Y v|Xv, Xt, Y t).4 This inherent difficulty
4To obtain such a bound we either need to have access to the intractable posterior pθ(ψi|Xt, Y t), or to upper bound the marginal likelihood on the training points pθ(Y t|Xt), which is very hard.
with computing the predictive distribution has led to several approximations, i.e. methods of category (i) above, ranging from MAP, Laplace, variational inference procedures (without rigorous bounds on the predictive conditional) and Stein variational gradient descent (Grant et al., 2018; Finn et al., 2018; Yoon et al., 2018; Nguyen et al., 2019; Gordon et al., 2019; Chen et al., 2019). The conditional probability modelling approaches (Garnelo et al., 2018; Gordon et al., 2019) try to directly model pθ(Y v|Xv, Xt, Y t) without considering this as an approximation to an initial joint Bayesian model. Our VIB framework differs significantly from the predictive distribution principle since VIB has an information theoretic motivation and it rigorously bounds an information bottleneck objective. VIB is also a fully tractable objective, avoiding the need to choose a particular approximate inference method and allowing us to rather focus on setting up the encoding procedure, as we did in the GP example in Section 3.
Regarding related works of GPs in meta learning, the ALPaCA method (Harrison et al., 2018) applied GPs and Bayesian linear regression to standard regression tasks with Gaussian likelihoods, while Tossou et al. (2019) used kernel-based methods (from a regularization rather than Bayesian perspective) again in standard regression. Closer to us, Patacchiola et al. (2019) and Snell & Zemel (2020) used GPs to perform few-shot classification together with deep neural kernels. However, our usage of GPs is different from these latter approaches, e.g. our general encoder amortization strategy can potentially deal with arbitrary likelihood functions and task output observations, while Patacchiola et al. (2019) assumes a Gaussian likelihood for the binary class labels and Snell & Zemel (2020) consider the Pólya-Gamma augmentation, which is very tailored to classification problems.

heading: 5 EXPERIMENTS
text: To evaluate the proposed algorithms, we consider a standard set of meta-learning benchmarks which include sinusoid regression and few-shot classification. As a baseline for comparison we consider MAML (Finn et al., 2017) and use exactly the same neural architecture for all methods and benchmarks as in Finn et al. (2017). The new methods we implemented are the following: (i) Stochastic MAML (S. MAML), as defined in Section 2.2, where the difference with MAML is the injected task-specific noise added to the outer loop update rule, together with the regularization term that comes from the VIB objective; see Section 2.2. The scalar noise parameter s is learned alongside the others. (ii) The memory-based GP method (referred as GP) trained by the VIB objective, as defined in Section 3, where the kernel feature vector φ(x; θ) is obtained from the last hidden layer of the same neural architecture as used in MAML. We use cosine and linear kernels (see Appendix D). For few-shot classification we report results separately for both kernels, while for sinusoid regression we only consider the linear kernel (the cosine kernel has similar performance). (iii) A combination of a memory-based and gradient-based approach referred to as GP+MAML where a MAML meta update rule is applied to the feature vector parameters θ; see Section 3.2.
Sinusoid regression. We first evaluate the method on a sinusoid regression as described by Finn et al. (2017) following their protocol. We train GP using VIB and evaluate its K-shot mean squared error performance. From Table 1, we observe that GP significantly outperforms MAML especially as K grows. Similarly, Figure 3 in Appendix illustrates this few-shot predictive ability of the GP, where posterior GP uncertainty reduces as K grows. Finally, from Figure 2 (left), we observe that GP drastically improves when more data is available and is more data efficient than MAML achieving near-optimal performance for K = 5. See Appendix D for more results and details.
Few-shot classification. The second domain is a standard few-shot meta-learning benchmark based on three datasets: Omniglot (Lake et al., 2011), mini-Imagenet (Ravi & Larochelle, 2017) and Augmented Omniglot (Flennerhag et al., 2018). For both Omniglot and mini-Imagenet we follow the experimental protocol proposed by Finn et al. (2017). For Augmented Omniglot, following Flennerhag et al. (2018); Chen et al. (2019), during meta-testing both MAML and S. MAML perform
100 steps of adaptation (resulting in 2000 data points seen by the model where each step processes a minibatch of size 20 points), while they are meta-trained by applying 20 adaptation steps (i.e. 400 training points seen per task). Both GP methods are meta-trained by memorizing the full N×K = 20×15 = 300 support points without further data augmentation, while during meta-testing we allow the GP methods to see up to 2000 points. See Appendix D for more details.
Classification accuracy performance for all methods are given in Table 2, while the corresponding negative log likelihood (NLL) scores are given in Table 3. We observe GP-based architectures outperform MAML and S. MAML in more complex scenarios such as Augmented Omniglot. From the NLL scores that depend on how well the predicted class probabilities are calibrated, we can observe that the GP methods perform significantly better in all cases where uncertainty matters i.e. on miniImagenet and Augmented Omniglot. On top of the described standard implementations of GP and GP+MAML, we implement ones with additional previously used architectures and tricks to improve results (see Appendx D.4). We observe that GP+MAML generally performs better than GP. Further, we notice that S. MAML has similar performance to MAML. One reason why S. MAML does not always outperform MAML could be related to hyperparameters and to the higher variance of the gradients caused by the reparametrization trick used to maximize the VIB objective in equation 7. This could imply that training a stochastic architecture could require longer training time, an issue deserving further investigation.
We also found that the qualitative behaviour of GP and GP+MAML are quite different as shown in Figure 4 in the Appendix D.2. In Appendix D.3 we provide ablative analysis for the impact of β on the performance of our architecture where the main result is that a large range of β values gives similar performance in practice.
Data efficiency in meta-testing of gradient-based vs GP-based meta learning. Having performed an expensive meta-training phase, the ultimate goal of a meta-learning system is to be deployed in practice and solve new tasks. In a real meta-testing scenario it is more likely the system to operate in a regime, where the training data for a given task is coming sequentially in mini-batches, and the system should continuously update itself. With this in mind, it is interesting to study the effect of K shots (or more generally the effect of the amount of processed data) in meta-testing performance. In Figure 2, we carry out an ablation study by comparing MAML and GP in sinusoid regression, mini-Imagenet and Augmented Omniglot by varying either K, for sinusoid regression and miniImagenet, or the amount of data augmentation, for Augmented Omniglot. We found that GP can be much more data efficient than MAML. This is because the GP predictive updates are based on Bayesian updating of sufficient statistics (for the linear kernels used here) which are similarly to Bayesian linear regression, as described in detail in Appendix C.3. Such updates are exchangeable (i.e. data order does not matter on the final prediction) and do not depend on learning rates. In contrast, a gradient-based method such as MAML can be less data efficient since stochastic gradient updates depend on the learning rate, mini-batch size, data order and the size of the inner loop.

heading: 6 CONCLUSION
text: We introduced an information theoretic framework for meta learning by using a variational approximation (Alemi et al., 2017; Chalk et al., 2016; Achille & Soatto, 2016) to the information bottleneck (Tishby et al., 1999). We derived a novel memory-based meta learning method with GPs, a stochastic MAML and a combination of memory and gradient-based meta learning.
While we have demonstrated our method in few-shot regression and classification, we believe that the scope of the information bottleneck for meta learning is much broader. For instance, an interesting topic for future research is to consider applications in reinforcement learning.

heading: A FURTHER DETAILS ABOUT VIB IN META LEARNING

heading: A.1 BOUNDS ON THE MUTUAL INFORMATION
text: Here, we review the standard variational bounds on the mutual information from Barber & Agakov (2003). Recall the definition of the mutual information,
I(x, y) = ∫ q(x, y) log q(x, y)
q(x)q(y) dxdy =
∫ q(x, y) log
q(x|y) q(x) dxdy.
By introducing p(x|y) that approximates q(x|y) we get
I(x, y) = ∫ q(x, y) log
p(x|y)q(x|y) p(x|y)q(x) dxdy =
∫ q(x, y) log
p(x|y) q(x) dxdy +
∫ q(y)KL[q(x|y)||p(x|y)]dy,
which shows that
I(x, y) ≥ ∫ q(x, y) log
p(x|y) q(x) dxdy, (12)
since ∫ q(y)KL[q(x|y)||p(x|y)]dy is non negative. An upper bound is obtained similarly. Suppose p(x) approximates q(x) then
I(x, y) = ∫ q(x, y) log
p(x)q(x|y) p(x)q(x) dxdy =
∫ q(x, y) log
q(x|y) p(x) dxdy − KL[q(x)||p(x)]dy,
which shows that
I(x, y) ≤ ∫ q(x, y) log
q(x|y) p(x) dxdy. (13)

heading: A.2 THE GENERAL VIB META LEARNING CASE
text: Consider the general case, where we work with the unconditional mutual information and we wish to approximate the IB: I(Z,Dv)− βI(Z,Dt). Recall that the joint distribution is written as
qw(Dv,Dt, Z) = qw(Z|Dt)p(Dv,Dt), (14)
from which we can express any marginal or conditional. In particular observe that
qw(Z,D v) = ∫ qw(Z|Dt)p(Dv,Dt)dDt.
If we have a function f(Z,Dv) and we wish to approximate the expectation,∫ qw(Z,Dv)f(Z,Dv)dZdDv = ∫ qw(Z|Dt)p(Dv,Dt)f(Z,Dv)dZdDvdDt, (15)
then given that we sample a task pair (Dvi ,Dti) ∼ p(Dv,Dt) we can obtain the following unbiased estimate of this expectation, ∫
qw(Z|Dti)f(Z,Dvi )dZ. (16)
We are going to make use of equation 15 and equation 16 in the derivation below.
To compute the variational approximation to IB, we need to lower bound I(Z,Dv) as I(Z,Dv) = ∫ qw(Z,Dv) log
qw(Z,Dv) qw(Z)p(Dv) =
∫ qw(Z,Dv) log
qw(Dv|Z) p(Dv) dZdDv
≥ ∫ qw(Z,Dv) log
pθ(Dv|Z) p(Dv) dZdDv, by using equation 12
= ∫ qw(Z,Dv) log pθ(Dv|Z)dZdDv +H(Dv),
where the entropyH(Dv) is just a constant. Subsequently, we upper bound I(Z,Dt) as follows,
I(Z,Dt) = ∫ qw(Z,Dt) log
qw(Z,Dt) qw(Z)p(Dt)
dZdDt = ∫ qw(Z|Dt)p(Dt) log
qw(Z|Dt) qw(Z) dZdDt,
≤ ∫ qw(Z|Dt)p(Dt) log
qw(Z|Dt) pθ(Z) dZdDt by using equation 13
Then we obtain the overall loss, F(θ, w) ≤ LIB(w): F(θ, w) = ∫ qw(Z,Dv) log pθ(Dv|Z)dZdDv − β ∫ qw(Z|Dt)p(Dt) log
qw(Z|Dt) pθ(Z) dZdDt,
where we dropped the constant entropic term H(Dv). Therefore, given a set of task pairs {Dti ,Dvi }bi=1, where each (Dti ,Dvi ) ∼ p̃(Dv,Dt), the objective function for learning (θ, w) becomes the empirical average, 1b ∑b i=1 Fi(θ, w), where
Fi(w, θ) = ∫ qw(Zi|Dti) log pθ(Dvi |Zi)dZi − β ∫ qw(Zi|Dti) log
qw(Zi|Dti) pθ(Zi) dZi, (17)
where for the first term we made use of equation 15 and equation 16 with f(Dv, Z) = log pθ(Dv|Z).

heading: A.3 THE SUPERVISED META LEARNING VIB CASE
text: For the supervised meta learning case the joint density can be written as
qw(Dv,Dt, Z) = qw(Z|Y t, Xt, Xv)p(Y t, Y v|Xt, Xv)p(Xv, Xt), = qw(Z|Y t, X)p(Y t, Y v|X)p(X), (18)
where X = (Xt, Xv) and the encoding distribution qw(Z|Y t, X) could depend on all inputs X but only on the training outputs Y t. The derivation of the VIB objective is similar with the general case with the difference that now we approximate the conditional information bottleneck I(Z, Y v|X)−βI(Z, Y t|X) where we condition on the inputs X . In other words, both I(Z, Y v|X) and I(Z, Y t|X) are conditional mutual informations, i.e. they have the form
I(z, y|x) = ∫ q(x) [∫ q(z, y|x) log q(z, y|x) q(z|x)q(y|x)dzdy ] dx = ∫ q(z, y, x) log q(z, y|x) q(z|x)q(y|x)dzdydx.
We can lower bound I(Z, Y v|X), as follows,∫ p(X) [∫ qw(Z, Y v|X) log qw(Z, Y v|X)
qw(Z|X)p(Y v|X) dZdY v
] dX
= ∫ p(X) ∫ qw(Z, Y v|X) log qw(Y v|Z,X)
p(Y v|X) dZdY vdX where qw(Z|X) cancels
≥ ∫ p(X) ∫ qw(Z, Y v|X) log pθ(Y v|Z,X)
p(Y v|X) dZY vdX by using equation 12
= ∫ qw(Z, Y v, X) log pθ(Y
v|Z,X) p(Y v|X) dZdY vdX
= ∫ qw(Z, Y v, X) log pθ(Y v|Z,X)dZdY vdX − ∫ p(Y v, X) log p(Y v|X)dY vdX (19)
Note that − ∫ p(Y v, X) log p(Y v|X)dY vdX is just a constant that does not depend on tunable parameters. Also
qw(Z, Y v, X) = ∫ qw(Z|Y t, X)p(Y t, Y v|X)p(X)dY t, (20)
so that if we have a task sample (Y ti , Y v i , Xi) ∼ p(Y t, Y v|X)p(X) an unbiased estimate of the expectation ∫ qw(Z, Y
v, X) log pθ(Y v|Z,X)dZdY vdX is given by∫
qw(Z|Y ti , Xi) log pθ(Y vi |Z,Xi)dZ. (21)
We upper bound I(Z, Y t|X) as follows,∫ p(X) [∫ qw(Z, Y t|X) log qw(Z, Y t|X)
qw(Z|X)p(Y t|X) dZdY t
] dX
= ∫ p(X) [∫ qw(Z, Y t|X) log qw(Z|Y t, X)
qw(Z|X) dZdY t
] dX, where p(Y t|X) cancels
≤ ∫ p(X) ∫ qw(Z, Y t|X) log qw(Z|Y t, X)
pθ(Z|X) dZdY tdX, by using equation 13
= ∫ qw(Z|Y t, X)p(Y t, X) log
qw(Z|Y t, X) pθ(Z|X) dZdY tdX,
Then we obtain the overall objective, F(θ, w) = ∫ qw(Z, Y v, X) log pθ(Y v|Z,X)dZdY vdX
− β ∫ qw(Z|Y t, X)p(Y t, X) log
qw(Z|Y t, X) pθ(Z|X) dZdY tdX,
where we dropped the constant term. Therefore, given a set of task pairs the objective becomes the empirical average, 1b ∑b i=1 Fi(θ, w), where
Fi(θ, w) = ∫ qw(Z|Y ti , Xi) log pθ(Y vi |Z,Xi)dZ − β ∫ qw(Z|Y ti , Xi) log
qw(Z|Y ti , Xi) pθ(Z|Xi) dZ,
(22)
where we made use of equation 21.

heading: A.4 CONNECTION WITH VARIATIONAL INFERENCE
text: As mentioned in the main paper, the VIB for meta learning (where we consider for simplicity the general case from A.2) is similar to applying approximate variational inference to a certain joint model over the validation set, pθ(Dv|Z)pθ(Z), where pθ(Dv|Z) is the decoder model, pθ(Z) a prior model over the latent variables and where the corresponding marginal likelihood is
p(Dv) = ∫ pθ(Dv|Z)pθ(Z)dZ.
We can lower bound the log marginal likelihood with a variational distribution qw(Z|Dt) that depends on the training set Dt,
Fβ=1(w, θ) = ∫ qw(Z|Dt) log pθ(Dv|Z)dZ − ∫ qw(Z|Dt) log
qw(Z|Dt) pθ(Z) dZ, (23)
which corresponds to the VIB objective with β = 1.

heading: B TRANSDUCTIVE AND NON-TRANSDUCTIVE META LEARNING
text: Here, we discuss how the transductive and non-transductive settings that appear in few-shot image classification (Bronskill et al., 2020; Finn et al., 2017; Nichol et al., 2018), due to the use of batchnormalization, can be interpreted under our VIB framework by defining suitable encodings. We shall use MAML as an example, but the discussion is more generally relevant.
The transductive case occurs when the concatenated support and validation/test inputs X = (Xt, Xv) of a single task (we ignore the task index i to keep the notation uncluttered) are used to compute batch-norm statistics (possibly at different stages) shared by all validation/test points, when predicting those points. For MAML this implies a deterministic parametric encoding, i.e. common to all individual validation inputs xvj ∈ Xv , obtained by a sequence of two steps: (i) Obtain first the task-specific parameter ψ in the usual way by the support loss, i.e. ψ = θ + ∆(θ,Dt). If batch-normalization is used here, then the statistics are computed only by Xt. (ii) Compute the validation loss by applying batch-normalization onXv or the unionX = Xt∪Xv (the union seems to be a better choice, but not used often in practice for computational reasons, e.g. Finn et al. (2017); Nichol et al. (2018) prefer to use only Xv). In both cases, the underlying encoder is parametric over the final effective task parameter ψ̃ = BN(ψ,X), where BN denotes the final batch-norm operation that outputs a parameter vector, that predicts all validation points and it is a deterministic delta measure.
In contrast, the non-transductive setting occurs when each individual validation input xvj is concatenated with the support inputs Xt to form the sets xvj ∪Xt, j = 1, . . . , nv . Then, each set xvj ∪Xt is used to compute point-specific batch-norm statistics when predicting the corresponding validation output yvj . Under the VIB framework this corresponds to a non-parametric encoding, which grows with the size of the validation set. The first deterministic step of this encoder is the same (i) above from the transductive case but the second step differs in the sense that now we get a validation pointspecific task parameter ψ̃j = BN(ψ, xvj ∪Xt) by computing the statistics using the set xvj ∪Xt. For MAML, this encoding becomes, Z ≡ {ψ̃j}n v j=1, and the encoder distribution is a product of delta measures. i.e. p({ψ̃j}n v j=1|Y t, X) ≡ ∏nv j=1 δψ̃j ,BN(θ+∆(θ,Dt),xvj∪Xt) .
Finally, note that under the VIB perspective it does not make much sense to meta-train transductively and meta-test non-transductively and via versa, since this changes the encoding. That is, whatever we do in meta-training we should do the same in meta-testing.

heading: C FURTHER DETAILS ABOUT THE GAUSSIAN PROCESS METHOD
text: For simplicity next we ignore the task index i to keep the notation uncluttered, and write for example f ti as f t and etc.
C.1 AMORTIZATION OF THE GP ENCODER q(f t|Dt)
A suitable choice of q(f t|Dt) is to set it equal to the exact posterior distribution over f t given the training set, i.e. p(f t|Dt) ∝ ∏nt j=1 p(y t j |f tj )N (f t|0,Kt). Interestingly, such a setting does not require to introduce any extra variational parameters w and it will depend only on the model parameters θ that appear in the kernel function and possibly also in the likelihood. For standard regression problems where the likelihood is Gaussian, i.e. p(ytj |f tj ) = N (ytj |f tj , σ2), the exact posterior has an analytic form given by
p(f t|Dt) = N (f t|Kt(Kt + σ2I)−1Y t,Kt −Kt(Kt + σ2I)−1Kt) (24)
and thus we can set q(f t|Dt) = p(f t|Dt). For all other cases where the likelihood is not Gaussian we need to construct an amortized encoding distribution by approximating each non-Gaussian likelihood term p(ytj |f tj ), with a Gaussian term similarly to how we often parametrize variational Bayes or Expectation-Propagation Gaussian approximation to a GP model (Hensman et al., 2014; Opper & Archambeau, 2009; Rasmussen & Williams, 2006) i.e.
p(ytj |f tj ) ≈ N (mtj |f tj , stj),
where mtj ≡ mw(ytj , xtj) ∈ R and stj ≡ sw(xtj) ∈ R+ are neural network amortized functions that depend on tunable parameters w and receive as input an individual data point (ytj , x t j) associated with the latent variable f tj . We made the simplification that the output point might only influence the real-valued mean mw(ytj , x t j), while the variance sw(x t j) can depend only on the input. Based on the above the amortized encoder is a fully dependent multivariate Gaussian distribution having the form
q(f t|Dt) = N (f t|Kt(Kt + St)−1mt,Kt −Kt(Kt + St)−1Kt), (25)
where St is a diagonal covariance matrix with the vector (st1, . . . , s t nt) in the diagonal and m t is the vector of values (mt1, . . . ,m t nt). This allows to re-write the VIB objective in equation 11 in the following computationally more convenient form (see Appendix C.2 next):
nv∑ j=1 Eq(fvj )[log p(y v j |fvj )]− β nt∑ j=1 Eq(ftj )[logN (m t j |f tj , stj)] + β logN (mt|0,Kt + St), (26)
where each marginal Gaussian distribution q(fj) when xj is either from the validation or the training set (or any other further test set) is computed by the same expression, q(fj) = N (fj |ktj(Kt + St)−1mt, kj − ktj(Kt + St)−1ktj > ), where ktj ≡ k(xj , Xt) is the nt dimensional row vector of kernel values between xj and the training inputs Xt and kj ≡ k(xj , xj). Classification. Here, we discuss how the above general amortization procedure can be particularized to classification problems, which is the standard application in few-shot learning. For notational simplicity we focus on binary classification, while multi-class classification is fully covered later in Appendix C.4.
Suppose a meta learning problem where each task is a binary classification problem where the binary class labels are encoded in {−1, 1}. To apply the method we simply need to specify the form of the amortized mean function mw(ytj , x t j) (recall that sw(x t j) is independent from the output y t j), which is chosen to be m(ytj , x t j) = y t j × m̃w(xtj),
where m̃w(xtj) is a real-valued function given by the neural network. Notice that the dependence on the output label ytj ∈ {−1, 1} simply changes the sign of m̃w(xtj). This latter function acts as a discriminative function that should tend towards positive values for data from the positive class and negative values for data from the negative class, while the product ytj × m̃w(xtj) should tend towards positive values. This amortization of the mean function is invariant to class re-labeling, i.e. if we swap the roles of the two labels {−1, 1} the amortization remains valid and it does not require any change. The multi-class classification case can be dealt with similarly, by introducing as many latent functions as classes, as discussed fully in Appendix C.4.

heading: C.2 DERIVATION OF THE VIB BOUND
text: The VIB objective for a single task from Eq. equation 22 in the main paper is computed as follows nv∑ j=1 Eq(fvj )[log p(y v j |fvj )]− β ∫ p(fv|f t, Xv, Xt)q(f t|Dt) log p(f v|f t, Xv, Xt)q(f t|Dt) p(fv|f t, Xv, Xt)p(f t|Xt) df tdfv
nv∑ j=1 Eq(fvj )[log p(y v j |fvj )]− β ∫ q(f t|Dt) log q(f t|Dt) p(f t|Xt) df t nv∑ j=1 Eq(fvj )[log p(y v j |fvj )]− βKL [ q(f t|Dt)||p(f t|Xt) ] , (27)
where q(fvj ) = ∫ p(fvj |f t, xvj , Xt)q(f t|Dt)df t is a marginal Gaussian over an individual validation function value fvj , as also explained in the main paper. Specifically, q(f v j ) depends on the training set (Y t, Xt) and the single validation input xvj , so intuitively from the training set and the corresponding function values f t we extrapolate (through the conditional GP p(fvj |f t, xvj , Xt)) to the input xvj in order to predict its function value fvj .
Given the specific amortization of q(f t|Dt):
q(f t|Dt) =
(∏nt j=1N (m t j |stj) ) N (f t|0,Kt)
N (mt|0,Kt + St) = N (f t|Kt(Kt + St)−1mt,Kt −Kt(Kt + St)−1Kt),
(28)
the VIB objective, by using the middle part of equation 28, can be written in the following form,
nv∑ j=1 Eq(fvj )[log p(y v j |fvj )]− β nt∑ j=1 Eq(ftj )[logN (m t j |f tj , stj)] + β logN (mt|0,Kt + St),
which is convenient from computational and programming point of view. Specifically, to compute this we need to perform a single Cholesky decomposition of Kt + St which scales as O((nt)3), i.e. cubically w.r.t. the size of the support set nt. This is fine for small support sets (which is the standard case in few-shot learning) but it can become too expensive when nt becomes very large. However, given that the kernel has the linear form kθ(x, x′) = φ(x; θ)>φ(x′; θ) (ignoring any kernel variance σ2f for notational simplicity), where φ(xi; θ) is M -dimensional and given that M nt, we can also carry out the computations based on a Cholesky of a matrix of size M ×M . This is because Kt = ΦtΦt
>, where Φt is an nt ×M matrix storing as rows the features vectors on the support inputsXt, and therefore we can apply the standard matrix inversion and determinant lemmas for the matrix ΦtΦt>+St when computing logN (mt|0,Kt +St). Such O(M3) computations also gives us the quantities q(fvj ) and q(f t j ), as also explained next.

heading: C.3 DATA EFFICIENT GP META-TESTING PREDICTION WITH CONSTANT MEMORY
text: Once we have trained the GP meta learning system we can consider meta-testing where a new fresh task is provided having a support setDt∗ = (Y t∗ , Xt∗) based on which we predict at any arbitrary validation/test input x∗. This requires to compute quantities (such as the mean value E[y∗]) associated with the predictive density
q(y∗) = ∫ p(y∗|f∗)p(f∗|f t∗, x∗, Xt∗)q(f t∗|Dt∗)df∗df t∗ = ∫ p(y∗|f∗)q(f∗)df∗
where q(f∗) is an univariate Gaussian given by
q(f∗) = N (f∗|kt∗(Kt + St)−1mt, k∗ − kt∗(Kt + St)−1kt∗ > ),
kt∗ = φ > ∗ Φ t, Kt = ΦtΦt > , k∗ = φ > ∗ φ∗, φ∗ = φ(x∗; θ).
Here, Φt is an nt∗ ×M matrix storing as rows the features vectors on the support inputs Xt∗. Note that if we wish to evaluate q(y∗) at certain value of y∗, and given that the likelihood p(y∗|f∗) is not the standard Gaussian, we can use 1-D Gaussian quadrature or Monte Carlo by sampling from q(f∗).
An interesting property of the above predictive density is that when the support set Dt∗ can grow incrementally, e.g. individual data points or mini-batches are added sequentially, the predictive density can be implemented with constant memory without requiring to explicit memorize the points in the support. The reason behind this that the feature parameters θ remain constant at meta-test time and the kernel function is linear, so we can apply standard tricks to update sufficient statistics as in Bayesian linear regression.
More precisely, what we need to show is that we can sequentially update the mean and variance of q(f∗) with constant memory. q(f∗) can be written as
q(f∗) = N (f∗|φ>∗ Φt > (ΦtΦt > + St)−1mt, φ>∗ ( I − Φt>(ΦtΦt> + St)−1Φt ) φ∗)
= N (f∗|φ>∗ (Φt > [St]−1Φt + I)−1Φt > [St]−1mt, φ>∗ (Φ t>[St]−1Φt + I)−1φ∗) (29)
where we applied the matrix inversion lemma backwards to write I − Φt>(ΦtΦt> + St)−1Φt = (Φt > [St]−1Φt+I)−1 and also used that Φt>(ΦtΦt>+St)−1 = Φt>(ΦtΦt>[St]−1+I)−1[St]−1 =
(Φt > [St]−1Φt + I)−1Φt > [St]−1 (based on the identity (AB+ I)−1A = A(BA+ I)−1). Now observe that the M -dimensional vector bt = Φt>[St]−1mt = ∑nt j=1 φ(x t j ; θ) mtj stj can grow incremen-
tally without memorizing the feature vectors φ(xtj ; θ) based on the recursion b t ← bt+φ(xtj ; θ) mtj stj (with the initialization bt = 0) as individual data points (similarly for mini-batches) are added in the support set: Dt ← Dt ∪ (xtj , ytj). Similarly, the M × M matrix At = Φt
> [St]−1Φt =∑nt
j=1 1 stj φ(xtj ; θ)φ(x t j ; θ) > can also be computed recursively with constant O(M2) memory.
Finally, note that the above constant memory during meta-testing can only be implemented when the feature vector θ remain fixed, which means that it is not applicable for the GP+MAML combination.

heading: C.4 MULTI-CLASS CLASSIFICATION
text: For multi-class classification meta learning problems we need to introduce as many latent functions as classes. For instance, when the number of classes for each task is N we will need N latent functions fn(x) which all are independent draws from the same GP. The marginal GP prior on the training and validation function values for a certain task factorizes as
N∏ n=1 p(fvn |f tn, Xv, Xt)p(f tn|Xt).
We assume a factorized encoding distribution of the form, N∏ n=1 p(fvn |f tn, Xv, Xt)q(f tn|Dt),
where each
q(f tn|Dt) = N (f tn|Kt(Kt + St)−1mtn,Kt −Kt(Kt + St)−1Kt). Here, mtn = Y t n ◦ m̃t and Y tn is a vector obtaining the value 1 for each data point xtj that belongs to class n and −1 otherwise. Note that the encoding distributions share the covariance matrix and they only have different mean vectors. The representation of mtn makes the full encoding distribution permutation invariant to the values of the class labels. Since also we are using a shared (i.e. independent of class labels) amortized functions m̃w(x) and sw(x) the terms (St, m̃t) are common to all N factors. This allows to compute the VIB objective very efficiently (in way that is fully scalable w.r.t. the number of classes N ) by requiring only a single Cholesky decomposition of the matrix Kt+St. Specifically, by working similarly to C.2 we obtain the VIB objective per single task,
nv∑ j=1 Eq({fvn,j}Nn=1)[log p(y v j |{fvn,j}Nn=1)]− β N∑ n=1 nt∑ j=1 Eq(ftn,j)[logN (m t n,j |f tn,j , stj)]
+ β N∑ n=1 logN (mtn|0,Kt + St),
where q({fvn,j}Nn=1) = ∏N n=1 q(f v n,j) and each univariate Gaussian q(f v n,j) is given by the same expression as provided in C.3. The last two terms of the bound (i.e. the ones multiplied by the hyperparameter β) are clearly analytically computed, while the first term involves an expectation of a log softmax since the likelihood is
p(yvj = n|{fvn′,j}Nn′=1) = ef v n,j∑N
n′=1 e fv n′,j
.
To evaluate this expectation we apply first the reparametrization trick to move all tunable parameters of q({fvn,j}Nn=1) inside the log-likelihood (so that we get a new expectation under a product of N univariate standard normals) and then we apply Monte Carlo by drawing 200 samples.
Finally, note that to compute the predictive density we need to evaluate, q(y∗) = Eq({fn,∗}Nn=1) [ p(y∗|{fn,∗}Nn=1) ] ,
which again is done by applying Monte Carlo by drawing 200 samples from q({fn,∗}Nn=1). This is precisely how the negative log-likelihood test performance was computed for the GP models in Table 3. To decide the classification label based on the maximum class predictive probability (in order to compute e.g. accuracy scores), we take advantage of the fact that all N univariate predictive Gaussians q(fn,∗) have the same variance but different means, thus the predicted class can be equivalently obtained by taking the argmax of the means of these N distributions.

heading: C.5 SPECIFIC GP IMPLEMENTATION AND AMORTIZATION FOR FEW-SHOT CLASSIFICATION
text: For all few-shot multi-class classification experiments in order to implement the GP and GP+MAML methods we need to specify the feature vector φ(x; θ) and the amortized variational functions m̃w(x) and sw(x). The feature vector is specified to have exactly the same neural architecture used in previous works for all datasets, Omniglot, mini-Imagenent and Augmented Omniglot; see for example Chen et al. (2019) for details on these architectures for all these three datasets. Note that when computing the GP kernel function the feature vector φ(x; θ) is also augmented with the value 1 to automatically account for a bias term in the kernel function.
Regarding the two amortized variational functions needed to obtain the encoder, we consider a shared (with the GP functions) representation by adding two heads to the same feature vector φ(x; θ): the first head corresponds to a linear output function m̃w(x) and the second applies at the end the softplus activation sw(x) = log(1+exp(a(x))) (since sw(x) represents variance) where the pre-activation a(x) is obtained by a linear function of the feature vector. For numerical stability we also apply a final clipping by bounding these functions so that m̃w(x) ∈ [−20, 20] and sw(x) ∈ [0.001, 20]. The bound −20 and 20 are almost never realized during optimization, so they are not so crucial, in contrast the lower bound 0.001 on sw(x) is rather crucial regarding numerical stability since it ensures that the minimum eigenvalue of the matrix Kt+St (i.e. the matrix we need to decompose using Cholesky) is bounded below by 0.001.

heading: D FURTHER EXPERIMENTAL DETAILS AND RESULTS

heading: D.1 EXPERIMENTAL SETTINGS AND HYPERPARAMETERS
text: The memory-based GP method (referred to in the Tables as GP) is trained by the VIB objective, as defined in Section 3, where the kernel feature vector φ(x; θ) is obtained by the last hidden layer of the same neural architecture as used in MAML. Based on this M -dimensional feature vector φ(x; θ) ∈ RM we consider two kernel functions: the standard linear kernel function kθ(x, x′) = 1 M φ(x; θ) >φ(x′; θ) (where the kernel variance σ2f is fixed to 1/M ) and the cosine kernel kθ(x, x ′) =
φ(x;θ)>φ(x′;θ) ||φ(x;θ)||||φ(x′;θ)|| . These kernels are used in the experiments.
The Omniglot dataset consists of 20 instances of 1623 characters from 50 different alphabets. Each instance was drawn by a different person. It is augmented by creating new characters which are rotations of each of the existing characters by 0, 90, 180 or 270 degrees. In Omniglot experiment, MAML, Stochastic MAML and GP+MAML run by applying one adaptation step for both metatraining and meta-testing. The mini-Imagenet involves 64 training classes, 12 validation and 24 tests classes. Following previous work on mini-Imagenet we meta-train Stochastic MAML with 5 adaptation steps, while 10 steps are used for meta-testing. GP+MAML uses 5 steps for both meta-training and meta-testing. For both Omniglot and mini-Imagenet we follow the experimental protocol proposed by Finn et al. (2017), which involves fast learning of N = 5-way classification with K = 1 or K = 5 shots. The problem of N -way classification is set up as follows: select N unseen classes, provide the model with K different instances per class, and evaluate the model’s ability to classify new instances within the N classes.
The Augmented Omniglot benchmark is a modified version of Omniglot which necessitates longhorizon adaptation and it is often considered as many-shots problem (Chen et al., 2019). For each alphabet, 20 characters are sampled to define a N = 20-class classification problem with K = 15 shots. Further, both train and validation images are randomly augmented, by applying transformations, which makes it even more challenging. Following Flennerhag et al. (2018); Chen et al. (2019), during meta-testing both MAML and Stochastic MAML perform 100 steps of adaptation (resulting in 2000 data points seen in total by the model where each step processes a minibatch of size 20
points), while they are meta-trained by applying 20 adaptation steps (i.e. 400 training points seen per task). Both GP methods are meta-trained by memorizing the full N ×K = 20× 15 = 300 support points without further data augmentation, while during meta-testing we allow the GP methods to see up to 2000 points. For all methods we perform a hyperparameter search using the train and validation subsets of all three benchmarks as detailed below.
For sinusoid regression, Omniglot and mini-Imagenet, meta-training of all methods consisted of 60000 iterations or episodes, where in each episode a learning update is performed based on a minibatch of tasks. For the sinusoid regression the meta batch-size was 5 tasks and for each task we have K = 10 examples. For N -way, K-shot classification in Omniglot and mini-Imagenet (with N = 5, and K = 1, 5 as mentioned in Table 2 in the main paper), the meta batch was 32 tasks in Omniglot and 4 tasks in mini-Imagenet. Also for these two datasets Stochastic MAML uses one gradient step (in the inner loop) for both meta-training and meta-testing in Omniglot, while in mini-Imagenet it uses 5 and 10 steps respectively, i.e. exactly as MAML by Finn et al. (2017) was applied in these datasets. As mentioned in the main paper GP+MAML considers one gradient adaptation step in Omniglot and 5 in mini-Imagenet (for both meta-training and meta-testing). The neural architectures of all these experiments is the same as in Finn et al. (2017). Specifically, for the Omniglot the architecture is from Vinyals et al. (2016), which has 4 modules with a 3 × 3 convolutions and 64 filters, followed by batch normalization, a ReLU nonlinearity, and 2 × 2 maxpooling. The Omniglot images are downsampled to 28×28, so the dimensionality of the last hidden layer is 64. For Omniglot, strided convolutions are used instead of max-pooling. For the GP methods this constructs 65-dimensional feature vector φ(x; θ) (64 plus one for the bias term which is included in φ(x; θ)). For mini-Imagenet, the network uses 32 filters per layer and the final layer the feature vector feature is obtained by flattening so that finally the feature vector φ(x; θ) is 801-dimensional.
For Augmented Omniglot, the meta-training of all methods consisted of 5000 iterations. The Augmented Omniglot dataset is a modified version of Omniglot which, for gradient-based methods like MAML, necessitates long-horizon adaptation and it is often considered as many-shots problem (Chen et al., 2019). For each alphabet, 20 characters are sampled to define a 20-class classification problem with K = 15 data points per class. Furthermore, both train and test images are randomly augmented, by applying transformations. Following Flennerhag et al. (2018); Chen et al. (2019), we use a 4-layer convnet and during meta-testing MAML and Stochastic MAML performs 100 steps of adaptation (resulting in 2000 data points where each step processes a minibatch of size 20 datapoints, i.e. per class), while they are meta-trained by applying 20 adaptation steps. GP+MAML uses 5 adaptation steps for both meta training and meta testing.
Additional hyperparameters details are included in Tables 4, 5 and 6. The ranges for the hyperparameters’ search are:
• Outer learning rate α:
[0.0001, 0.00025, 0.0005, 0.00075, 0.001, 0.002, 0.0025,
0.005, 0.0075, 0.01, 0.025, 0.05, 0.075, 0.1]
• Inner learning rate ρ:
[0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.5, 1.0, 1.5, 2.0]
• Bottleneck coefficient β:
[1e− 07, 1e− 06, 1e− 05, 5e− 05, 0.0001, 0.00025, 0.0005, 0.00075, 0.001, 0.0025, 0.005, 0.0075, 0.01, 0.1, 0.5, 1.0]

heading: D.2 ADDITIONAL EXPERIMENTAL RESULTS
text: Table 7 provides detailed few-shot sinusoid regression results, while Figure 3 illustrates how the GP model, after having been meta-trained on the sinusoid regression tasks, predicts a test task as the number K of the training examples increases.
We also found that the qualitative behaviour of GP and GP+MAML are quite different. In Figure 4, we report the performance of the GP and GP+MAML on mini-Imagenet tasks as a function of
inner steps executed at test time by GP+MAML. Note, that GP actually has only one step executed over all the test-time data in once, because there is no inner loop for it. Intuitively, we would expect GP+MAML to have similar performance to GP at the beginning of inner loop, but this is not what is happening. Instead, it starts quite low and peaks at exactly the number of inner steps used during meta-training and then the performance starts to deteriorate.

heading: D.3 ABLATIVE STUDIES
text: Bottleneck cost β ablation To understand the impact of β on learning we provide the ablation analysis in Figure 5 on sinusoid regression and two few-shot classification tasks. For a regression task, Figure 5 (a), we observe that having a large β = 1.0 is beneficial and provides the best mean squared error (MSE). For few-shot classification tasks Figures 5 (b)-(f), we observe that there is a large range of β values in between [10−7, 0.01] providing similar classification accuracy. Interestingly, for values of β close to 1, where for β = 1 the VIB corresponds to an ELBO, the performance deteriorates significantly.
D.4 IMPROVED VERSIONS FOR THE GP-BASED METHODS
For GP-based methods we consider additional popular tricks used in the community to improve performance.
For Augmented Omniglot dataset (Flennerhag et al., 2018), for the network architecture we add 4 additional layers similar to Flennerhag et al. (2020), where each layer is added after a convolutional layer. These additional layers are simple convolutions as considered in Flennerhag et al. (2020). In addition, we add a batch normalization. Simply using more layers in GP methods significantly improves the performance on this dataset. Note, that the results are achieved without a special warpgrad architecture as in Flennerhag et al. (2020). In Table 8 we show the results with and without batch normalization (BN).
In addition, we run additional experiments on mini-ImageNet (Ravi & Larochelle, 2017) and tieredImageNet (Ren et al., 2018) with using pre-trained embeddings from LEO (Rusu et al., 2019) paper as features. The embeddings are taken from their GitHub repository. On top of these features, we construct an MLP with two hidden layers and ReLU activations where each layer has 128 hidden dimensions. This defines the feature vector φ(x; θ) which is used by the GP methods in all the experiments. Moreover, we apply dropout to the LEO embeddings before passing them to the MLP. The results are given in Table 9. The hyperparameters are tuned in the same way as for the previous experiments. Their parameters are given in Table 10. Notice that the GP methods are comparable
with LEO, with no significant difference. We would like to point out that LEO uses a set of different tricks: dropout, label smoothing, `2 regularization and orthogonality penalty (Rusu et al., 2019), while we have only considered dropout.

Review summaries:
The paper derives a meta-learning framework based on the information bottleneck principle. By adapting the variational approximation proposed in [1] to the meta-learning setting, the authors come up with a tractable objective that generalises both gradient based and memory meta-learning methods. Based on this framework, the authors proposed a new memory based meta-learning algorithm by using a GP with a deep kernel and an extension that combines this memory based method with MAML. The authors show that the method outperforms MAML in several standard meta-learning  tasks, especially in regression and many shots classification problems.

The paper presents a method for Bayesian meta-learning. This method combines a NN feature extractor with a Gaussian Process on top. The GP kernel is linear. The information bottleneck is used to motivate a choice of approximate posterior. Using MAML to adapt the NN feature extractor weights improves performance of the composite model for few-shot learning.