Information Theoretic Meta Learning with Gaussian Processes
Authors: Michalis Titsias
Sotirios Nikoloutsopoulos
Alexandre Galashov
Venue:  (2021)
Keywords: Meta Learning
Information Bottleneck
Gaussian Processes
Few-shot learning
Variational Inference

Abstract: We formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training or support set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. We  particularly develop new memory-based meta learning algorithms based on Gaussian processes and derive extensions that combine memory and gradient-based meta learning. We demonstrate our method on few-shot regression and classification by using standard benchmarks such as Omniglot, mini-Imagenet and Augmented Omniglot.


Under review as a conference paper at ICLR 2021
INFORMATION THEORETIC META LEARNING WITH GAUSSIAN PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
We formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck. The idea is to learn stochastic rep- resentation or encoding of the task description, given by a training or support set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. We particularly develop new memory- based meta learning algorithms based on Gaussian processes and derive exten- sions that combine memory and gradient-basel meta learning. We demonstrate our method on few-shot regression and classification by using standard bench- marks such as Omniglot, Imagenett and Augmented Omniglot.
INTRODUCTION
Meta learning Raai & Larochelle 2017 Vinyals et al. 2016; Edwards & Storkey 2017 Finn et al, 2017 Lacoste et al. 2019 Nichol al. 2018) and few-shot learning (Li al.) 2006] Lake al. 2011 aim to derive data efficient learning algorithms that can rapidly adapt to new tasks. Such systems require training deep neural networks from of tasks drawn from common distribution, where each task is described by a small amount of experience, typically divided into a training or support set and validation set. By sharing information across tasks the neural network can learn to rapidly adapt to new tasks and generalize from few examples at test time.
Several few-shot learning algorithms use memory- beased (Vinyals et al. 2016 Ravi Larochelle 2017) or gradient-based procedures (Finn et al.) 2017, Nichol et al. (018). with the gradient-based model agnostic meta learning algorithm (MAML) by Finn et al. (2017) being very influential in the literature. Despite the success of specific schemes, one fundamental issue in meta learning is concerned with deriving unified principles that can allow to relate different approaches and invent new schemes. While there exist probabilistic interpretations of existing methods, such as the ap- proximate Bayesian inference approach (Grant et al. 2018 Finn et al., 2018 Yoon et al., 20188) and the related conditional probability modelling approach (Garnelo et al. 2018 Gordon et al.) 2019), meta learning still lacks of a general and tractable learning principle that can help to get a better understanding of existing algorithms and derive
methods.
To this end, the main contribution of this paper is to introduce an information theoretic view of meta learning, by utilizing tools such as the mutual information and the information bottleneck (Cover & Thomas [2066] Tishby et al. 1999). Given that each task consists of support or training set and target or validation set, consider the information bottleneck principle, introduced by Tishby al. (1999), which can learn stochastic encoding of the support set that is highly informative about pre- dicting the validation set. Such stochastic encoding is optimized through the difference between two mutual informations, so that the encoding compresses the training set into representation that can predict well the validation set. By exploiting recent variational approximations to the information bottleneck (Alemi et al.) 2017 Chalk et al.) 2016 Achille & Soatto 2016) that make use of varia- tional lower bounds on the mutual information (Barber & Agakov| 2003), we derive a general and tractable framework for meta learning. Such framework can allow to re-interpret gradient-based algorithms, such as MAML, and also derive new methods.
Based on the variational information bottleneck (VIB) framework (Alemi et al. 2017 Chalk al.) 2016 Achille & Soatto| 2016).. we introduce a new memory-based algorithm for supervised few- shot learning (right panel in Figure based Gaussian processes (Rasmussen & Williams 2006)

---

Under review as a conference paper at ICLR 2021
(1/b) E F -21(2)))33(222
Gradient-based
Param, encoder
Z s
Encoder: Gw Z2 (D')
Training set
Validation set
Y X
Y
D
Z
D
Nonparam. encoder
= f(x) ~ GP
tasks: =
Memory-based
Figure 1: (left) Meta learning with the information bottleneck. 2, qu (Z, (!) is the encoder wish optimize compress the task training Df (minimize the mutual information [(Z,, D))) and predict well the validation set D. (maximize [(,,, D))); see Section (right) Specialization to supervised few-shot learning where for cach task we have input-output data. Gradient- based algorithms, such as MAML, and Gaussian process memory-based methods (proposed in this paper) are instances of the framework; see Section 2.2 222and]
and deep neural kernels (Wilson et al. 2016) that offers kernel-based Bayesian view a memory system. With Gaussian processes, the underlying encoding takes the form of non-parumetric func- tion that follows a stochastic process amortized by the training set. Further, we show that VIB gives rise tradient-based meta learning methods, such MAML, when combined with parametric encodings corresponding to model parameters or weights, and based on this we derive a stochastic MAML algorithm. In additional scheme, show that framework combinations of memory and gradient-based meta learning by constructing suitable encodings, and derive such algorithm that combines Gaussian processes with MAML. We demonstrate our methods on few-shot regression and classification by using standard benchmarks such as Omniglot, mini- mini-Imagennt and Augmented Omniglot.
naturally allow for
META LEARNING WITH INFORMATION BOTTLENECK
Suppose we wish learn from a distribution of tasks. During training for each task we observe a pair consisted of a task description represented by the support or training set DD' and task validation represented by the target or validation set D At test time only D' will be given and the learning algorithm should rapidly adapt to form predictions on D or on further test data.
We wish to formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck (Tishby et al. 1999). The idea is to learn a stochastic representation or encoding the task description D' that is highly informative about predicting D We introduce a random variable, Z, associated with this encoding drawn from a distribution quu (Z(D*) parametrized by w. Given this encoding the full joint distribution is written as
aw (D', D'. 2) qu qh (Z)D' (DD" D').
where p(D",20 DD) denotes the unknown data distribution over DD' and D In equation and through- out the paper we use the convention that the full joint as well as any marginal or conditional that depends on Z is denoted by tu (emphasizing the dependence on the parametrized encoder), while corresponding quantities over data D', D are denoted by Eg. from the above we can express a D'')
Z- Z-dependent marginal such as, qu D")
qu
To tune w we would like to maximize the mutual information between Z and the target set D" de- noted by 1 1(2, D A trivial way to obtain maximally informative representation is to set 2 = D' which does not provide a useful representation. Thus, the information bottleneck (IB) principle Tishby al. 1999) adds model complexity penalty the maximization of I(Z, D") which pro- motes an encoding Z that is highly compressive of Dt, i.e. for which 1(2, D') is minimized. This leads to the IB objective:
(2) where 2 0 is a hyperparameter. Nevertheless, in order to use IB for meta learning we need to approximate the mutual information terms 1(2, D") and I(Z, D*), which are both intractable since they depend on the unknown data distribution p D'). To overcome this, we will consider variational approximations by following similar arguments to the variational IB approach (Alemi
Cıs(w) (=z,))=((ZDD)) -')
2

---

Under review as a conference paper at ICLR 2021
al. 2017) that was introduced for supervised learning of single task, which allows to express
a tractable lower bound on (IB(w) by lower bounding I(Z, and upper bounding [Z, Dr).
2.1
VARIATIONAL. INFORMATION BOTTLENECK (VIB) FOR META LEARNING
To construct a bound 2 LIB(W) we need first to lower bound I(2, D"), which is written
T(Z, = (g (2, D") (1) (Z)p(D") = (z,,( log Tu (D" [2)
(3)
(DD
J 4w (w(2" (PD" "(DD","")d"" D')dDt
where KL denotes the Kullback-Leibles divergence and qu (D" (2)
Tq Ip( JaD'an aD is
intractable since we do not know the analytic form of the data distribution p(D", D'). To lower bound (2, D we follow Barber Agakov (2003) (see Appendix [A.1) I by introducing decoder model po (D* to approximate the intractable Qu (D*I2) where 0 are additional parameter
(o( p(Do)
(Z2 D") 2 (q
log
L8 (2,D=)
[log pe (D"IZ) + H(D"),
(4)
where the entropy HDD") is just a constant that does not depend on the tunable parameters (6, w). Furthermore, to deal with the second intractable mutual information [[Z, D³) so that to maintain an lower bound on LiB (w) we need to upper bound this term. Note that
Tu (Z (1, Eg..
log Iw (Z)
where gar (2) ymn (Z is intractable since, e.g. involves the unknown data dis- tribution p(D)). By working similarly as before, we can approximate go (Z) by tractable prior model distribution (Z) which leads to the following upper bound on the mutual information,
Tu Th (2(D))
(Z, S (Z. log
(5)
(Z)
Then, by combining the bounds obtaín the overall bound, F(0, () H(D") s LIB(W):
log gu D4)
F(0, w) = (2,D) [log po (D" [Z)] - B8 (2,D9)
(o
where the constant H(D") is dropped from the objective function. Given a set of task pairs (D), Di where each (D), D.) p(D", Dt), during meta-training the objective function for learning (0, w) reduces to the maximization the empirical average, F. w), where cach F.(0, w) is an unbiased estimate of F(0, w) (see Appendix A.2 and is given by
(w, 0) = E (Z.12)) [log po (D. Z.)] - [q (Z) po
(6)
The meta-training procedure is carried out in different episodes where at each step we receive minibatch of task pairs and perform a stochastic gradient maximization step.
The objective in equation 6 is similar variational inference objectives for meta learning (Ravi & Beatsson 2019). In particular, it can be viewed as an evidence lower bound (ELBO) on the validation set log marginal likelihood, log f (D. | 2))pe (Z.)dZ,, with the differences:
(i) there is the hyperparameter / in front of the KI. term and (ii) the variational distribution (Z, D() in equation 6 is more restricted than in standard variational inference, since que (Z. now acts as stochastic bottlemeck that encodes the support set D (i.e. it is amortized by DD) and via the term E (Z.())))) [pp))pplooopppo (D:) 2.)] is optimized to reconstruct the validation set.
2.2
INFORMATION THEORETIC VIEW OF MAML-TYPE METHODS
MAML. Finn et al. 2017) special case of our framework. To see this, suppose that the task encoding variable Z; for the i-th task coincides with a vector of some task-specific model param- eters or neural network weights th, so that pa (D. Z.) = p(D))v)) and pe (2:) reduces to prior
'The lower bounds valid even when the parameters of the encoder qw (7\D') and 0 of the decoder (D (and prior (Z)) have shared components, e.g. are parameters of the neural architecture.
3

---

Under review as a conference paper at ICLR 2021
Po())) over these parameters. The MAML approach (Finn al. 2017 tries to find shared initial parameter value 9 so that few gradient steps based on the support set objective, log p( [0), lead the validation set. MAML esti- mates the task parameters by 1/; = + A (0, D where A(0, Dj) denotes the inner loop adaptation steps which for single gradient step just pVo log p(Di and where p is step size. By setting 3 = 0 and by using a deterministic Dirac measure encoder, 60. (()))) the VIB objective from cquation reeduces to the standard MAML objective, (0) = log (( A())
to task-specific parameter value v; with good generalization
We construct generalization of MAML by making the encoder stochastic, e.g. qo (1. (D() (1 + (0, D)), sT) where is a scalar variance parameter. Then, the objective becomes
s0 (elo..))) [logr(D"|0 (0,,D)) + Voe)] 3KL [ga.> (v:s (())))))loo()))))
where we reparametrized the expectation suggesting the of the reparametrization trick (Kingmal & Welling 2013) for stochastic optimization of the meta parameters (0, In the experiments instance of the above where pe (v4) follows the hierarchical Gaussian form (1bs)0, sT), which views each task- task-specific parameter 4/i as a randomized version of 9 and with being the same variance used by the encoder. For such prior the KL divergence term reduces to A (0, D21, which penalizes large values of the inner adaptation steps and small values of s.
we will investigate
2.3
VIB FOR SUPERVISED META LEARNING
Here, we explain how to adapt the VIB principle to supervised meta learning. Suppose the learning problem involves few-shot supervised learning where for each task we wish to predict outputs or labels given corresponding inputs. Let denote the task support set as D = (Y*, Xx),, where Y {リ} and X {a {
1 denote the output and input observations. Similarly, we write D" (Y", X²) for the validation set. During meta-testing, for any novel task we observe the sup- port set D
(.) together with the test inputs X"" and the goal is to predict the test outputs Y.. This suggests that we can construct a task encoder distribution of the form (n X) that depends on the training outputs yt and generally on all inputs X = (X*, X²) 2 We would like to train this encoder so that Z becomes highly predictive about the validation outputs Y and simulta- neously compressive about Y Then, suitable VIB objective can be based on approximating the input-conditioned information bottleneck, (")) - 3I(Z, Y+|X) i.c. where both ((Z, l and I [ZZ, Y* XX) are conditional mutual informations (see Appendix (A.3). By following similar guments as those Section 2.1
[. we can lower bound this objective and finally approximate it by an unbiased empirical average, E (,, w), where
(, w) = E (Z.)), x.) [log po (Y," 2,, X.)] 3KL
(7)
and where pe [X," Z.) and (x) are the decoder and prior model distributions introduced when applying the variational approximation. A detailed derivation can be found in Appendix A..3 7quation7prrove
the general form of the VIB objective suitable for supervised meta learning. The supervised version of MAML is expressed as a special case by following the arguments of Section we particularize the above by combining with Gaussian process model.
2.2 Section 3,
3 SUPERVISED META LEARNING WITH GAUSSIAN PROCESSES
In this section we introduce VIB-based meta learning algorithms using Gaussian processes (GPs), which are suitable for few-shot supervised learning. In Section 3.1 we introduce memory- system, while in Section 3.2
we further combine it with gradient-based meta learning.
3.1 A GAUSSIAN PROCESS HEMORY-BASED METHOD
To use the VIB framework for few-shot supervised learning, as described in Section 2.3 for each i-th task we need to specify the encoding variable Zü together with the encoder Iu (7. Y, X2). the decoder over the validation outputs pe (Y,", Z,,X,) and the prior model pe (Z, (X)). Here, we construct these quantities by using a GP model (Rasmussen & Williams 2006).
Dependence all inputs can allow to explain both transductive and -ransuaandeee settings for meta learning (Bronskill et al. 2020 Finn al. 2017 Nichol al. 2018) special cases; Appendix
4

---

Under review as a conference paper at ICLR 2021
We denote the unknown task-specific function that solves the i-th task by fz( and we assume that a priori (before observing any task data) this function is a draw from a GP, i... f((z) ~ GP(O, (z)))))), where Kg denotes the kernel function. Without loss of generality we shall use a deep kernel function where fi())) is a linear function of a deep neural network feature vector ((x;0) with task-specific Gaussian weights, i.e. f((x)) = obx;; 0) 0gue, 0put
(0000(00,07)) Such function can be equivalently viewed as GP sample: f((x) ~ GP(O, ko(2, z')), Ky (z, x²) 6f 0) p(z'; f). In this functional space view the task- -seeccfeciiic output weights 6: have been marginalized out and we are left with the feature vector parameters 0 shared across tasks f
Suppose now that we observe the task data, i.e. the support Dj = -Y, ()) and validation D. = (Y,", XX)) sets, so that we can evaluate the task function on all task inputs X; (X,, Let fly = f denote the function value at the validation input z associated with output y while the vector of all such values is denoted by f.' - U Similarly, the vector of function values at the training inputs X; is fl. In the VIB framework, we specify the task encoding variable Zi to be the full set of function values, Z; = (f. f.), and we further choose the prior model for this
encoding to be the GP prior, po [Z) x)) = p(ff' f/ X;), where
p(f (.) = p(f E. X²) x p x)))
(8)
= =N [ f - K (KT"" x ((F),,
(9)
Here, K KÝ are square kernel matrices of size n' x n' and n² n" on the training inputs X; and validation inputs X", while KV is the n" x n' cross kernel matrix between these two sets of inputs. Note that the encoding is non-parametric since its size grows with the number of task data points.
To continue with the specification of the VIB objective, the second quantity we need to set is the decoder model Pe (Y ( ff; X;) which is chosen to be standard GP likelihood. Specifically, for i..d. observations Y given ff' becomes independent from ff and X; and the likelihood factor- [T]=
where each ([,)) is standard likelihood model, such as Gaussian density p(v,,I/E, - =N WI, o²) suitable for standard re- gression problems categorical/softmax likelihood for few-shot classification; see Appendix C.4 Finally, we specify the encoder distribution qu (2,117 X.) = q(f." F as follows.
izes across data points, i.e. p(Y;"lf,")
(C,,, MIX, x,) - P(C" (F))))
(10)
where ((f)) is the same conditional GP prior from equation g while (f( is a encoder of the training set that takes the form of Gaussian distribution specified based very general amortization procedure, see Appendix | Equation I0] shares a similar structure with a standard posterior Gaussian process where we first observe the training set, then we compute the (approxi- mate) posterior distribution (f, DD),, and finally we extrapolate/predict the validation set function values at inputs X," based on the conditional GP prior p(f,") X,). The above assumptions yield (see Appendix the following VIB objective for single task,
E Ea (og Dogp(yi, (T,)1 - 3KL ((CE)))))))))
(1I)
where q(fT,) "", dation function value fLy. Here, q(fr,) depends on the training set and the single validation input " that from the training set and the corresponding function values ff we extrapolate (through
is a univariate Gaussian over an individual vali-
the univariate conditional GP p(f,,, [ft, a X()) to the input oY in order to predict its value fT-
3.2 COMBINATION WITH GRADIENT-BASED META LEARNING
In this section, combine the GP memory- based meta learning method with gradient-based approach, such as MAML (Finn et al. 2017). Based on the VIB principle we need to specify an encoding that will allow us to combine memory-basse discussed in Section 2.2 gradient-based meta learning is associated with a parametric encoding that corresponds to a fixed-size task model parameter v4, In contrast, as seen Section [3.]] a memory system is associated with a function-space or non-parametric encoding that consists of the
with a gradient- approach. As
"The kernel variance parameter of (if leamablee) is also considered be part of the full of parameters 6.
5

---

Under review as a conference paper at ICLR 2021
function values (f,, f(") of all task input points. Therefore, way to combine these techniques is to concatenate the encodings, ie. Z, = (v,, f.)). Here, 4/; are the task-specific parameters of the GP kernel function kg, 2z) (and possibly of the likelihood pfy]f)), which in our implementation are the parameters of the feature vector p( th). Intuitively, a combination of the GP memory- based method with MAML will try apply a short inner adaptation loop in order to adjust an initial feature vector o(z;0) () to obtain a final o(x; 16)) that can better solve the task. For the overall encoding (vi, fl ff), the general form of the encoder distribution takes the form
q(C,, G MY X.) p(f,'," di, Xx)a((,) [v,, (b)
where p(f ft, v/. X²) is the conditional GP prior and ((f//1//. D') is the amortized encoder (see Appendix
where we have emphasized their dependence on d/. The VIB objective becomes
Becv. (D)
E
( DDIlP())))))
In practise, we can relax this objective and use different hyperparameters 3 and B& in front of the two KL terms. This is convenient as we would prefer to set By 0 and use a deterministic MAML w.r.t. 3/ rather than a stochastic MAML; see Section [.2 This simplification avoids the need to specify prior p(t))) over the task-specific neural network parameters and at the same time it reduces the encoder ( (D)) to a Dirac delta, which simplifies the objective as follows,
Σ E RUC (o) flog p (I,,) Ky [9(E,) (w,, (P(()) (b,, [p X(],
where U/ = 0 A(0, ()). The inner loop adaptation term can be defined by an objective function on the support set DJ. In our case suitable objective the VIB for the GP memory- based method obtained by setting the validation set equal to the training set, i.e. D = Dद in equation 11
which
gives nn j== Eucst, (log plut, ( - 8yKL [0(1510, (f) X.)].
4 RELATED WORK
In this work, the VIB principle was used to formulate meta learning. VIB has been used before for different purposes, such for regularization of single-task supervised learning (Alemi al. 2017
sparse coding (Chalk et al. 2016), re-interpretation of 3- VAES and Dropout (Burgess et al. 2018 Achille & Soatto| 2016) and for compression of deep neural networks (Dai et 2018).
A meta learning method that connects with the information bottleneck was recently proposed by Hu (222). There the information bottleneck used analyze the generalization of a variational Bayesian inference objective suitable for transductive supervised few-shot learning. Note that the information bottleneck derived in Hu al. (2020) (theorem is not the same as the information bottleneck objective used here (i.e. the objective in equation 7 for the supervised learning case), since they differ in the second term. In addition, our framework expresses a general information bottleneck principle for meta learning applicable beyond transductive supervised learning.
Given the probabilistic nature of our framework, we can relate to other probabilistic or Bayesian approaches and particularly with those that: (i) probabilistically re-interpret and extend gradient- based methods (Grant al.] (20T8: Finn et al., 2018, Yoon et al.] 2018, Nguyen et al., 2019, Gordon et al.] 2019; Chen et al. 2019) and (ii) derive amortized conditional probability mod- els (Garnelo et al. 2018; Gordon et al., 2019). The underlying learning principle in both (i)- (ii) is to construct and maximize predictive distribution (or conditional marginal likelihood) of the validation points given the training points, which, e.g. in supervised few-shot learning is pY [x", tbx)po(ty)) Y')dy,
written po(Y"|X", XX, X))
Pu y"(x","") Here, p
pe X Yt) is a posterior distribution over the task parameters vhg after observing the training points, and is a meta parameter which for simplicity we assume to be found by point estimation. However, this objective is very hard to rigorously approximate. Unlike the marginal likelihood on all task outputs (Y", y*|X", X') for which we can easily compute a lower bound, there is no tractable lower bound on the predictive conditional po (Y"
X", X², Y This inherent difficulty
"To obtain such bound either need to have access to the intractable posterior po ( (Y,Y", or to upper bound the marginal likelihood on the training points po ( which is very hard.

---

Under review as a conference paper at ICLR 2021
with computing the predictive distribution has led to several approximations, i.e. methods of cat- egory (i) above, ranging from MAP, Laplace, variational inference procedures (without rigorous bounds on the predictive conditional) and Stein variational gradient descent (Grant et al. 20188: Finn et at 2018 Yoon et al., 2018 Nguyen et al. 2019, Gordon et al. 2019: Chen et al. (2119) The con-
ditional probability modelling approaches (Garnelo et al. 2018, Gordon al. 2019) try directly model pa (I"),, "x,',,,, X", Y`) without considering this as an approximation to an initial joint Bayesian model. Our VIB framework differs significantly from the predictive distribution principle since VIB has an information theoretic motivation and it rigorously bounds an information bottleneck objec- tive. VIB is also a fully tractable objective, avoiding the need to choose a particular approximate inference method and allowing us to rather focus setting up the encoding procedure, as we did in the GP example in Section 3
Regarding related works of GPs in meta learning, the ALPaCA method (Harrison et al., 2018) ap- plied GPs and Bayesian linear regression to standard regression tasks with Gaussian li likelihoods, while Tossou et al. (2019) used kernel-based methods (from a regularization rather than Bayesian perspective) again in standard regression. Closer to us, Patacchiola et al. (2019) and [Snell & Zemel 2020) used GPs to perform few-shot classification together with deep neural kernels. However, our usage of GPs is different from these latter approaches, e.g. our general encoder amortization strategy can potentially deal with arbitrary likelihood functions and task output observations, while while|Patacchi- Patacchi- ola et al. (2019) assumes a Gaussian likelihood for the binary class labels and [nell & 77mell consider the Pólya-Gamma augmentation, which is very tailored classification problems.
EXPERIMENTS
To evaluate the proposed algorithms, we consider a standard set of meta-learning benchmarks which include sinusoid regression and few-shot classification. As a baseline for comparison we consider MAML (Finn et al. 2017) and use exactly the same neural architecture for all methods and beneh- marks as in Finn et al. (017). The new methods we implemented are the following: (i) Stochastic MAML (S. MAML), as defined in Section [.2] where the difference with MAML is the injected task-specific noise added to the outer loop update rule, together with the regularization term that comes from the VIB objective; see Section 2.2 The scalar noise parameter 8 is learned alongside the others. (ii) The memory-based GP method (referred as GP) trained by the VIB objective, as defined in Section 3] where the kernel feature vector p(2; f) is obtained from the last hidden layer of the same neural architecture used in MAMII. We cosine and linear kernels (see Appendix (D). For few-shot classification we report results separately for both kernels, while for sinusoid regression we only consider the linear kernel (the cosine kernel has similar performance). (ii) A combination of a memory-based and gradient-basse based approach referred to as GP+MAML where MAML meta update rule is applied to the feature vector parameters 0: see Section 3.2
Sinusoid regression. We first evaluate the method on a sinusoid regression as described by Finn al. (2017) following their protocol. We train GP using VIB and evaluate its K-shot mean squared error performance. From T we observe that GP significantly outperforms MAML especially as K grows. Similarly, Figure in Appendix illustrates this few-shot predictive ability of the GP, where posterior GP uncertainty reduces as K grows. Finally, from Figure 2 (left), we observe that GP drastically improves when more data is available and is more data efficient than MAML achieving near-optimal performance for K = 5. See Appendix D for more results and details.
Model
K=5
K=10
K=20
Few-shot sinusoid regression re- sults. We report the results (mean values and 95% confidence intervals after 10 repeats) of all methods (GP. GP+MAML, MAML and MAML) for K 5, 10, 20. GP-based meth- ods use linear kemel.
Table 1:
MAMI.
0.280+00.033
0.096++0.005
0.043++0.003
S. MAML.
0.317+0.34
0.116+0.012 0.054++0.004
GP
0.02++0.014
0.002+00000 0.001+0.001
GP+MAML
0.058-1.054 0.002+0.001 0.002+0.0002
Few-shot classification. The second domain is standard few-shot meta-learning benchmark based
on three datasets: Omniglot (Lake et al. (011). mini-Immgenn (Ravi & Larochelle (017) and Aug- mented Omniglot (Flennerhag et al. 2018). For both Omniglot and mini- Imagene we follow the experimental protocol proposed by Finn et al. (2017). For Augmented Omniglot, following FFlen nerhag al. 2018]; Chen et al. (201)), during meta-testing both MAMI and S. MAML perform
7

---

Under review as a conference paper at ICLR 2021
GP, B=1
MAML: 1
p=
0.001
MSE
Accuracy
MAML-10
MAML: 10
6.
MAML:20
MAML: 20
0.55
MAML:40
MAML: 5
MAML:60
0
10
(a)
(b), meta-trained with N 5, K =
0.8
0.7
GP. P 33-00 0.001
Accuracy
0.6
Accuracy
MAML:10
GP, B 0.00075
MAML:20
9.4
MAML
MAML:40
0.5
MAML:60
0.2
500
1000
1500
2000
Data points (d)
(c) meta-trained with N = 5, K =5
Figure 2: (a) Sinusoid regression with GP and MAMI. in meta-testing as the number of shots K (x-axis) increases. On y-axis, report the MSE For MAML, report the performance with different number of inner loop steps, i.c. just SGD steps since we do meta-testing, specified in the legend. (b)-(c) Meta-testing classification accuracy (y-axis) on mini- Imageemm where each system has been meta-trained with either N 5, K = or N = 5, K = 5, as the number K of observed examples per class (while always N = 5) grows from 1 to 20, e.g. for K 20 each system sees N x K = 100 support examples. For MAML we show the performance for different inner loop sizes, which in meta-testing is just SGD updates, where each SGD step uses a random mini-batch of size 10 data points from the support set size x K. (d) Similarly to (b)-(c) for Augmented Omniglot, where instead of growing K we increase the amount of data augmentation in ppe-specified/fieed N = 20, K = 15 initial support set. This means in each SGD update of MAML or predictive density GP update we sample mini-batch from the fixed N K support set, apply random transformations this mini-hatch and then we it perform the actual update. The mini-hatch size 20 and hased on this data augmentation process we grow the amount of data (x-axis) processed hy each method from 20 up to 2000. Finally to create all plots we average performance under 10 repeats, where in each repeat the systems are meta-trained from scratch and then are evaluated in large number of meta-testing tasks.
Omniglol 5- way
mini- Imagenet 5 way
Augmented Omniglot 20 way
Model
K = 1 shut
K = 5 shot
K 1 shot
K 5 shot
K = 15 shot
Stochastic MAML
08.9444100030 % 00.745=4.01777
18.1139:0.3119
64.179::0.17 781
77.3 1-13%%
GP (lincar)
99.036::0 0311%
99.776:#0 019% 18.09740.2133
61.174100.0. 2.3%%
80.73:t0.77 7.76%
GP (cos)
99 00..059:00000 030%
99.774:10
0166
47,93410 293%
63 63.6601:002266
262%%
80.77:10.880
GP-MAML (linear)
99.771
15.509910.3440
64. 64..7.00.22266
81.33310..4888
GP+MAMT. (cns)
99.0511:0.028%
99.783:0.019%%
48.0755:0.44333
64.506:50. 177%
+81.79::0.671
MAML Finn at 2017
98.740.4%
99.9:t0.1%
48.7+1.84%%
63. 63.1110.92%
76.67:20.663 (our)
Table 2: Classification test accuracy on Omniglot, mini- mini-Imagenet and Augmented Omniglot. For all methods, mean performances with 95% confidence intervals are reported after repeating the experiments 10 times and each experiments performs meta-testing in 1000 tasks. Best performance is with bold, while * indicates statis- tically significant better performance than MAML in Augmented Omniglot. For Augmented Omniglot we MAML (see Appendix for the hyperparameters) since this dataset was not included in Fiin et al. (201)).
100 steps of adaptation (resulting in 2000 data points seen by the model where each step processes a minibatch of size 20 points), while they are meta-trained by applying 20 adaptation steps (i.e. 400 training points seen per task). Both GP methods N x = x0x15 300 support points without further data augmentation, while during meta-testing allow the GP methods see up to 2000 points. See Appendix for more details.
meta-trained by memorizing the full
8

---

Under review as a conference paper at ICLR 2021
Omniglot 5-way
mini- mnini-Inagenet 5-way
Augmented Omungglo 20-way
Model
= 1 shot
5 shot
K = shot
K = shot
K = 15 shot
Stochastic MAML.
0.031-10.000 % 0.008
1.27.10.008%
0.925.10.0133
0.673:10.02557
(lincar)
0.03640. 002%
0.01240.0019 001%
1.267=0.. 0008
0.90440..00 005%
0.67640. 034%
GP (cos)
0,015=0.00111
0.019=0.001 %
2262=00000%
0.921=0.006%
0.662=00.02 027%
GP+MAML (linear)
0.036:00. 0033
1.010=0.0011%
-1.246=000077
+0.900:1000099
0.671#00.0202
GP+MAMI. (206)
0.045.10.011
0.01910.0011% %
1.009%
0.902
~0.616.000022
MAML (our)
0. 0.03220.0013 0.008+0.001%
1.279=0.000%
0.926=0.011%
0.694+0.022%
Table 3: Classification negative log likelihood (NI.L) performance on Omniglot, mini-Imagen and Aug- mented Omniglot. obtain these scores for MAML we re-run MAML since only classification accuracy is reported in Finn et al. (0117) Again indicates statistically significant better performance than MAML.
Classification accuracy performance for all methods are given in Table while the corresponding negative log likelihood (NLL) scores are given in Table We observe GP-based architectures out- perform MAML and S. MAML in more complex scenarios such as Augmented Omniglot. From the NLL scores that depend how well the predicted class probabilities are calibrated, we can observe that the GP methods perform significantly better in all cases where uncertainty matters i.e. on mini- Imagenet and Augmented Omniglot. On top of the described standard implementations of GP and GP+MAML, we implement ones with additional previously used architectures and tricks to improve results (see Appendx D.4). We observe that GP+MAML generally performs better than GP. Further, we notice that S. MAML has similar performance to MAML. One reason why S. MAML does not always outperform MAML could be related hyperparameters and to the higher variance of the gradients caused by the reparametrization trick used to maximize the VIB objective in equation 7 This could imply that training a stochastic architecture could require longer training time, an issue deserving further investigation.
We also found that the qualitative behaviour of GP and GP+MAML are quite different as shown in Figure 4 in the Appendix D.2] In Appendix [.3 we provide ablative analysis for the impact of B on the performance of our architecture where the main result is that large range of / values gives similar performance in practice.
Data efficiency in meta-testing of gradient-based vs GP-based meta learning. Having performed an expensive meta-training phase, the ultimate goal of meta-learning system is to be deployed in practice and solve new tasks. In a real meta-testing scenario it is more likely the system to operate in regime, where the training data for given task is coming sequentially in mini-batches, and the system should continuously update itself. With this in mind, it is interesting to study the effect of K shots (or more generally the effect of the amount of processed data) in meta-testing performance.
In Figure 2 we carry out an ablation study by comparing MAML and GP in sinusoid regression, mini-Imagener and Augmented Omniglot by varying either K, for sinusoid regression and mini- Imagenet, or the amount of data augmentation, for Augmented Omniglot. We found that GP can be much more data efficient than MAML. This is because the GP predictive updates are based on Bayesian updating of sufficient statistics (for the linear kernels used here) which are similarly to Bayesian linear regression, as described in detail in Appendix C.3 Such updates are exchangcable (i.e. data order does not matter the final prediction) and do not depend on learning rates. In contrast, a gradient- based method such as MAML can be less data efficient since stochastic gradient updates depend on the learning rate, mini-bater size, data order and the size of the inner loop.
6 CONCLUSION
We introduced information thcoretic framework for meta learning by using variational approxi- mation (Alemi et al. 2017: Chalk et al.) Achillc Achille & Soatto (0166) to the information bottleneck
Tishby al. 1999). We derived novel memory- based meta learning method with GPs, stochas- tie MAML and combination of memory and gradient- based meta learning.
While we have demonstrated our method in few-shot regression and classification, believe that the scope of the information bottleneck for meta learning is much broader. For instance, an interest- ing topic for future research is consider applications in reinforcement learning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation, 2016.
Alex Alemi, lan Fischer, Josh Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
David Barber and Felix Agakov. The im algorithm: A variational approach to information maxi- mization. In Proceedings of the 16th International Conference Neural Information Processing Systems, NIPS 03, pp. 201-208, Cambridge, MA, USA, 2003. MIT Press.
John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tasknorm: Rethinking batch normalization for meta-lcarning, 2020.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in /3-ae, 2018.
Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational infor- mation bottleneck. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1957-1965. Curran Associates, Inc., 2016.
Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew W. Hoffman, and Nando de Freitas. Modular meta-learning with shrinkage, 2019.
Thomas M. Cover and Joy Thomas. Elements of Information Theory (Wiley Series Telecom- munications and Signal Processing). Wiley-Intersciene USA, 2006. ISBN 047 0491241954.
Bin Dai, Chen Zhu, Baining Guo, and David P. Wipf. Compressing neural networks using the vari- ational information bottleneck. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmüssan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp- 1143-1152. PMIR, 2018
Harrison Edwards and Amos Storkey. Towards a neural statistician. In 5th International Conference on Learning Representations (ICLR 2017), 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp- 1126-1133 1135, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning, In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 9516-9527. Curran Associates, Inc., 2018
Sebastian Flennerhag, Pablo G. Moreno, Neil D. Lawrence, and Andreas Damianou. Transferring knowledge across learning processes, 2018.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leaming with warped gradient descent, 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. volume 80 of Proceedings of Machine Learning Research, pp. 1704-1713, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference on Learning Representa- fions, 2018.
10

---

Under review as a conference paper at ICLR 2021
James Harrison, Apoorva Sharma, and Marco Pavone. Meta-learinng priors for efficient online bayesian regression, 2018.
James Hensman, Max Zwiessele, and Neil D. Lawrence. Tilted variational bayes. In Proceedings of the Seventeenth International Conference on Artificial Intelligncce and Statisties, AISTATS 2014, Reykjavik, Iceland, April 22-25. 2014, volume 33 of JMLR Workshop and Conference Proceedings, pp. 356-364. Jog.. 2014.
Xu Hu, Pablo Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, and Neil Lawrence. Empirical bayes transductive meta-learning with synthetic gradients. ICLR, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:13/2.6114, 2013.
Alexandre Lacoste, Boris Oreshkin, Wonchang Chung, Thomas Boquet, Negar Rostamzadeh, and David Krueger. Uncertainty in multitask transfer learning, 2019.
Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. Cognitive Science, 33, 2011.
Fei-Fei Li, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Truns. Pattern Anal. Mach. Intell., 28(():594 611, 2006.
Cuong Nguyen, Thanh-Toan Do and Gustavo Carneiro. Uncertainty in model-agnostic meta- learning using variational inference, 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Manfred Opper and Cédric Archambeau. The variational gaussian approximation revisited. Neural Comput., 21):788667792 March 2009. ISSN 0899-7667.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O' O'Boyle, and Amos Storkey. Deep kernel transfer in gaussian processes for few-shot learning, 2019.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenRe- view.net, 2019. URL. htt.:tppss///pppeeeiiieee
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In JCLR, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classifica- tion, 2018.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization, 2019.
Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each Pólya-Gamma augmented Gaussian processes, 2020.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference Communication, Control and Computing, pp- 368-377, 1999.
Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive deep kernel learning, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. koray kavukcuoglu, and Daan Wierstra. Match- ing networks for shot learning. In D. D. Lee, M. Sugiyama. U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3630-3638, Curran Associates, Inc., 2016.
11

---

Under review as a conference paper at ICLR 2021
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pp. 370-378, 2016.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn.
Bayesian model-agnostic meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 7332-7344 Curran Associates, Inc., 2018
12

---

Under review as a conference paper at ICLR 2021
Here, we provide additional details regarding our method. Appendix fully the derivation of the variational information bottleneck (VIB) objective for meta learning. Appendix explains how the transductive and non-tramsductive settings, often used in few-shot image classification, can be explained as particular cases the VIB framework under suitably defined encodings. Appendix Ciprovides full details regarding the GP meta learning method proposed in the main paper. Finally, Appendix D discusses experimental settings, and it presents additional experimental results and ablation studies.
A FURTHER DETAILS ABOUT VIB IN META LEARNING
A.1 BOUNDS ON THE MUTUAL INFORMATION
Here, review the standard variational bounds on the mutual information from Barber & Agakov 2003). Recall the definition of the mutual information,
I(x, v) (2x, y) log ((x,9) y) drdy q(>)g(V)
((>ly) drdy.
g(x,, v) log
g(z)
By introducing p(aly) that approximates ((>ly) we get
T(a.y) v) f ff ((x,, v) log p(zly)g(<())) drdy = / ((x,))) v) log p(xl))q(())
p(aly) dady + ((o))l/gg(oo///looobblll g(x)
which shows that
[(x, v) / g(2,, v) log drdy,
p(aly) g(z)
(12)
since I KL (p is non negative. An upper bound is obtained similarly. Suppose p(z) approximates g(x:) then
T(x,y)= = f q(2,, ((2,,)) y) log
p(z)a(zlv)) dody f q( v) log p(x) p(x)g(x)
((<lv)
drdy (L(g(=))lp<)))d
which shows that
[x, v) / g(2,, v) log ((x11) drdy. p(x)
(13)
A.2 THE GENERAL VIB META LEARNING CASE
Consider the general case, where we work with the unconditional mutual information and we wish approximate the IB: ((2, D (I Dt). Recall that the joint distribution is written
qu (D', D ,', 2) = qu qu (Z(D") (DD", D)),
(14)
from which we can express any marginal conditional. In particular observe that
qu (Z, (Z)
qw (Z(D'))(DD' DD))dD'
we have function f(Z, D*) and we wish to approximate the expectation,
flZ, / ((2 (Z ("" DZ"
-f
(15)
then given that sample task pair (D), D.) p(D" Dt) obtain the following unbiased estimate of this expectation,
/ f (w (1),
(16)
We going make of equation 15 and equation 16 in the derivation below.
13

---

Under review as a conference paper at ICLR 2021
To compute the variational approximation IB, need to lower bound [2, D*) as
1(2, D") = / qw (2, D")log
Io (2,
/ (") log Zu (D"IZ) ( dZdD'
Sw (Do))
> / qu (Z, lo) po (D 2) dZdD", by using equation 12 ((D))
qu (Z, D log po (D (D"))""""") H(D"),
where the entropy H(D")) is just a constant.
Subsequently, we upper bound (2, Dt) follows,
T(Z, T(Z = / qui (2, D') log 4n (2, Dt)
qu 2 (Di) dzdD' = qu (10 (((("))))(D)) log (u=()
dZdd'
que (2)
$ f Iu (z(D)))) log qw (D dZdD'
by using equation 13
[Z)
Then we obtain the overall loss, F(0, w) S (IB(w))
-f qu (Z, Do) (D - 3 f 9m
log qu (Z(D') [Z
F(O, w) =
dZdD'
where we dropped the constant entropic term H(D"). Therefore, given set of task pairs (D), D. where each (D), D P(D", D'), the objective function for learning (0, w) be- comes the empirical average, D Ei=1 F.(0, w), where
Fi(w, 0) = que (Z, lo (D; ( - 8/ qm ( (Z.IDt) log po(Z.)
dZ,,
(17)
where for the first term we made use of equation 15 and equation 16 with J(D", 7. = log po (D'1))
A.3 THE SUPERVISED META LEARNING VIB CASE
For the supervised meta learning case the joint density can be written as The (D", D'. 7) qu Y X X "D)((Y Y"IX, X"),XX", "'"XX"X""" X²), Iw (Z X)p((', ('(X)x(X),
(18)
where X = (x*, and the encoding distribution qn (2 could depend on all inputs outputs
X but only on the training outputs Yt. The derivation of the VIB objective is similar with the general case with the difference that now we approximate the conditional information bottleneck [[Z, (*[X)- - (I( Y*X) where condition the inputs X. In other words, both I(Z, ( and T(Z, x)) are conditional mutual informations, i.e. they have the form
1(x, vl) = ((x) |f ((2,)/x))
v/x)) log ((2,)|>) dzdy de I ((22,) log (22,)/z) dzdydæ. g(+)x))(()z) (x)
((+2)))(vlx))
We can lower bound I(Z, Y X)). as follows,
/ ((x)) 1 qn (2, log
(Z, dZdy dx
-f p(X) / / (2, log (Y X), dZdY"'X where qu (Z cancels
Po (Y, dZY"dX 2 f p(X) (xxl log
by using equation 12
{ Au (n, XY,,) p
ZZ dZdY"dx
qv qu, Y" X) Po Po (Y" Z, X X)dZdY ax -f 1 X(Y" (Y"" (19)
14

---

Under review as a conference paper at ICLR 2021
Note that - p(Y", log p( is just constant that does not depend on tunable parameters. Also
= f 10 (2 (Xp(", Y (p))))(()dd
qu
(20)
so that if we have task sample (Y YP, X;) p(Y,, (x())((X)) unbiased estimate of the expectation f qu Z, Y X) log po (Y Z, X) X)dZX)ZZYYYY vd given by
/ Iu X,) log 1
(21)
We upper bound T(Z,Y*|X as follows,
/D(X) [/ aw (,, log
au Y
9m Z X (X) dZdy! dX
-f /xXX) /
9u (Z .X) dZdy dx,
log
where xX) cancels
qu (2|x)
- i p(X) f (2l228911) log
9w tw (Z dZdY''X,
by using equation 13
(x(
/ qu (1)(("X))p((") () log
In (2r' dZdY'dX,
pa (Z
Then obtain the overall objective,
/ qr0 (z, Y XX) log Z. XZYYdddY
F(0, w)
B $ (x)
X)p((', X) log 2m (2 X) dZdY'
po(2|X)
where we dropped the constant term. Therefore, given set of task pairs the objective becomes the empirical average, & > Fi(0, w), where
F((0,)) w) = -f 9a (zl)
3 8/ 9 (u(Z Xo) (2 dZ,
po (Z)xx)
(22)
where we made of equation 21
A.4 CONNECTION WITH VARIATIONAL. INFERENCE
mentioned in the main paper, the VIB for meta learning (where we consider for simplicity the general case from A.2) is similar applying approximate variational inference to certain joint model over the validation set,
()))))))n(ZZ;
where Po (D" Z) is the decoder model, PZ(Z) prior model over the latent variables and where the corresponding marginal likelihood is
((D") = (D("")22)o0())d
We can lower bound the log marginal likelihood with variational distribution qu (Z(Dt) that de- pends on the training set D'
Fw-1(w.0) (no log pa (D² f (Z(DD) ((1D")log Je (Z(D')
po d7,
(23)
which corresponds the VIB objective with 8 1.
15

---

Under review as a conference paper at ICLR 2021
B TRANSDUCTIVE AND NON-TRANSDUCTIV META LEARNING
Here, we discuss how the transductive and non- crnttransductive settings that appear in few-shot image classification (Bronskill et al., 2020 Finn et al., 2017 Nichol et al., 2018), due to the use of batch- normalization, can be interpreted under our VIB framework by defining suitable encodings. We shall use MAML as an example, but the discussion is more generally relevant.
The transductive case occurs when the concatenated support and validation/test inputs X = (X', x*) of single task (we ignore the task index to keep the notation uncluttered) are used to compute batch-norm statistics (possibly at different stages) shared by all validation/test points, when predicting those points. For MAML this implies a deterministic parametric encoding, i.c. common to all individual validation inputs z E X", obtained by sequence of two steps: (i) OB- tain first the task-specific parameter in the usual way by the support loss, i.e. 1 0 4++ A0,, If soanch-ommaizzation is used here, then the statistics are computed only by X (ii) Compute the validation loss by applying patch-normalizatio
on X or the union X = X X (the union seems to be a better choice, but not used often in practice for computational reasons, e.g. -inn et al. (2017); Nichol al. (2018) prefer to use only X In both cases, the underlying encoder parametric over the final effective task parameter &i = BN X), where BN denotes the final batch-norm operation that outputs parameter vector, that predicts all validation points and it is a deterministic delta measure.
In contrast, the non-transductive setting occurs when each individual validation input zi is concate- nated with the support inputs X form the sets xy U UX', ,,...,n", n". Then, each set zi U is used to compute point-specific batch-norm statistics when predicting the corresponding validation output yf. Under the VIB framework this corresponds to a non- non-parametric encoding, which grows with the size of the validation set. The first deterministic step of this encoder is the same (i) above from the transductive case but the second step differs in the sense that now we get a validation point- specific task parameter 3 = ( 2ÿ UXX* by computing the statistics using the set æ"j U.,' For MAML, this encoding becomes. Z {6 by and the encoder distribution is a product of delta measures. i.e. p( p(( J Y X) - 03 (..
Finally, note that under the VIB perspective it does not make much sense to meta-train transductively and meta-test non-transductively and via versa, since this changes the encoding. That is, whatever we do in meta-training we should do the same in meta-testing.
C FURTHER DETAILS ABOUT THE GAUSSIAN PROCESS METHOD
For simplicity next we ignore the task index to keep the notation uncluttered, and write for example f as I' and etc.
C.1 AMORTIZATION OF THE GP ENCODER (D')
A suitable choice of q( (D') is to set it equal to the exact posterior distribution over f' given the training set, i.c. p(F' x II P()]) K'). Interestingly, such a setting does not require to introduce any extra variational parameters w and it will depend only on the model param- eters 0 that appear in the kernel function and possibly also in the likelihood. For standard regression problems where the likelihood Gaussian, i.e. p(uţ f3) = NWv o²), the exact posterior has an analytic form given by
p(F'(D') = -N( + + K'KK' TIKK'+00 +021)
(24)
and thus set (f'1DD') (D') For all other cases where the likelihood is not Gaussian we need to construct an amortized encoding distribution by approximating each non non- Gaussian like- lihood term If)), with Gaussian term similarly to how we often parametrize variational Bayes or Proppattonnnoaatatiiion Gaussian approximation to a GP model (Hensman et al.) 2014 Opper
& Archambeau 2009 Rasmussen & Williams 2006) i.e.
p(yl10)) NM(m)15),
16

---

Under review as a conference paper at ICLR 2021
where m. (f). R and s) = sw (x]) R4 are neural network amortized functions that depend tunable parameters w and receive input individual data point (y), associated with the latent variable fj. We made the simplification that the output point might only influence the realevalued mean (H). 2²), while the variance sn (x(()) can depend only on the input. Based on the above the amortized encoder is a fully dependent multivariate Gaussian distribution having the form
q(' D' N + + K' - K'KK' + +S*)
(25)
where St is a diagonal covariance matrix with the vector (si,..., a/)) in the diagonal and m' is the vector of values (mi..... ) This allows to re-write the VIB objective in equation in the following computationally more convenient form (see Appendix C.2 next):
n¹ Ruy Dog (mm sf)] + (om' 0, K +
Σ BougyDogg ((5)) Dogrg()op(()11
Σ
(26)
where each marginal Gaussian distribution (f{)) when zj is either from the validation or the training
set (or any other further test set) is computed by the same expression, q(G)) = NN [Jj (K' +
s')='m', kj - (/'K' + - "), kernel values between I, and the training inputs X and k, = kzz,
), where kj- = ,( X') is the n² dimensional row vector of
Classification. Here, we discuss how the above general amortization procedure can be particularized to classification problems, which is the standard application in few-shot learning. For notational simplicity we focus on binary classification, while multi-class classification is fully covered later in Appendix C.4
Suppose meta learning problem where each task is a binary classification problem where the binary class labels are encoded in 1.. To apply the method we simply need to specify the form of the amortized mean function mw 2)) (recall that sw (xx)) is independent from the output vÿ). which is chosen to be
mwy 2²) - v/ (x)))
where (xx) is a real-valued function given by the neural network. Notice that the dependence on the output label y E {-1, 1} simply changes the sign of (x5)) This latter function acts as diseriminative function that should tend towards positive values for data from the positive class and negative values for data from the negative class, while the product V/ x (=x should tend towards positive values. This amortization of the mean function invariant to class re-labeling, i.e. if we swap the roles of the two labels {-1, } the amortization remains valid and it does not require any change. The multi-class classification case can be dealt with similarly, by introducing as many latent functions as classes, as discussed fully in Appendix C.4
C.2 DERIVATION OF THE VIB BOUND
The VIB objective for a single task from Eq. equation 22 in the main paper is computed as follows
(ogp((1111 3 p(P")it, X", (')(("' log p(f" X", X",,""))((""" ('F df'df" n²
X",
Bur (og - 3/ (('))")) (( log ((P' dr x()
n"
Eg [ogg(( 3KL 3=1
Σ
(27)
(f )df a marginal Gaussian over an individual validation function value ff" as also explained in the main paper. Specifically, (ff)) depends on the training set (Y','') and the single validation input æj. so intuitively from the training set and the corresponding function values we extrapolate (through the conditional GP p(f (f', f."," x²)x)) order to predict its function value fz.
where ((f)) = p(fy)f Xx'))
) to the input æy in
17

---

can apply standard tricks to update sufficient statisties in Under review as a conference paper at ICLR 2021
Given the specific amortization of q("" (D'):
((" ([m3(o))))) (( N(m K
N((F (K' + +'s'"'m' "", K' K'(K' + -S')-"k")
K St)
(28)
the VIB objective, by using the middle part of equation 28 can be written in the following form,
Popr)oogp((]1 3 Σ
sf) + B Blog ((m")0.mm. K' + +S")
which is convenient from computational and programming point of view. Specifically, to compute this we need to perform a single Cholesky decomposition of K' + S' which scales as (( (nz))), i.e. cubically w.r.t. the size of the support n'. This is fine for small support sets (which is the standard case in few-shot learning) but it can become too expensive when n²' becomes very large. However, given that the kernel has the linear form (2 x²) = 0(2; 0) "<x'; (ignoring any kernel variance of for notational simplicity), where (xx;; 0) is M-dimensional and given that M < n', we can also carry out the computations based on Cholesky of matrix of size M X M. This is because K' = d'<' where d' is ne' x M matrix storing rows the features vectors the support inputs X and therefore we can apply the standard matrix inversion and determinant lemmas for the matrix O' S' when computing log (mm²)0, K' + S²). Such ( computations also gives us the quantities q(f}) and ((f)), as also explained next.
C.3 DATA EFFICIENT GP META-TESTING PREDICTION WITH CONSTANT MEMORY
Once we have trained the GP meta learning system we can consider meta-testing where new fresh task is provided having support set D = (Y X;) based on which we predict at any arbitrary val- idation/test input v.. This requires to compute quantities (such as the mean value Ely-)) associated with the predictive density
a(v)) = / pluu-ft)p(ffEEE X.) = / p(u-lff)9(ff)df
where q(f.) is an univariate Gaussian given by
((f.) = Nff (K' + + k. - -K' + S' - )
k = o. 4',
K' o' = o. Φ- = 0)r,;)).
Here, o' is an nł x M matrix storing as rows the features vectors on the support inputs X Note that if wish evaluate q(y=) at certain value of ye, and given that the likelihood pfy.lf.) is not the standard Gaussian, we can use 1-D Gaussian quadrature or Monte Carlo by sampling from {(f=).
An interesting property of the above predictive density is that when the support set D can grow incremenaally, e.g. individual data points or mini-batches are added sequentially, the predictive den- sity can be implemented with constant memory without requiring to explicit memorize the points in the support. The reason behind this that the feature parameters remain constant at meta-test time and the kernel function is linear, Bayesian linear regression.
More precisely, what need to show is that we can sequentially update the mean and variance of q(f=) with constant memory. q(f=) can be written as
((f+) = -N 4. (Ø' st) "m'. oo (I (1- -- T. (0'0² T
+ Φx)
= = ( (S +
Im' .o_ ( -1 + 1) -1(b)))
(29)
where we applied the matrix inversion lemma backwards to write / - x² ( + 5t) -1g =
(TeT - - and also used that क. ( + क. ($' (1)
18

---

Under review as a conference paper at ICLR 2021
(Ør ( [S"]-10' 101-'a++1--100 ( (based on the identity (AB+A)'AAA(BA+-) Now ob- A
serve that the M-di-n M-dimensional vector b' o² [S']-1m Σラー (z)); can grow incremen-
tally without memorizing the feature vectors <(a5; 0) based on the recursion b' + b' + 6aa): 0) m
(with the initialization b' 0) as individual data points (similarly for mini-batches) are added in the support set: D -- D U (zf. vf). Similarly, the M M matrix A ot [S Ej
can also be computed recursively with constant O memory.
Finally, note that the above constant memory during meta-testing can only be implemented when the feature vector remain fixed, which means that is not applicable for the combinatio
C.4 MULTI-CLASS CLASSIFICATION
For multi-class classification meta learning problems need to introduce many latent functions as classes. For instance, when the number of classes for each task is N we will need latent functions (2) which all are independent draws from the same GP. The marginal GP prior the training and validation function values for a certain task factorizes as
II PEME, x')(E())))
We assume factorized encoding distribution of the form,
Il p PE X (x, x')g(t))
where each
((EI"")=((E)))
+
K'(K' S'k++'''"K""
Here, m Y in and Y is vector obtaining the value 1 for each data point x² that belongs to class and 1 otherwise. Note that the encoding distributions share the covariance matrix and they only have different mean vectors. The representation of m makes the full encoding distribution permutation invariant to the values of the class labels. Since also we are using shared (i.e. indepen- dent of class labels) amortized functions (x) and (z)( the terms (St, m' are common all N factors. This allows to compute the VIB objective very efficiently (in way that is fully scalable w.r.. the number of classes N) by requiring only a single Cholesky decomposition of the matrix K' +S. Specifically, by working similarly to C.2 obtain the VIB objective per single task,
8
(loggol ( st)l
+
log N (m +
12
where q( (( expression as provided in C.3 hyperparameter 3) are clearly analytically computed, while the first term involves an expectation of a log softmax since the likelihood is
q (fr and each univariate Gaussian q( q(%%)) is given by the same The last two terms of the bound (i.e. the ones multiplied by the
plu = mniff m
IN =
To evaluate this expectation we apply first the reparametrization trick to move all tunable parameters of ( inside the log-like log-likeldhood (so that we get new expectation under a product of N univariate standard normals) and then apply Monte Carlo by drawing 200 samples.
Finally, note that to compute the predictive density we need to evaluate,
q(y.) Eyttf.. IN [p(y- )
19

---

Under review as a conference paper at ICLR 2021
which again is done by applying Monte Carlo by drawing 200 samples from q( f(f.fn,. ) This is precisely how the negative log-likelihood test performance was computed for the GP models in Table E To decide the classification label based on the maximum class predictive probability (in order to compute e.g. accuracy scores), we take advantage of the fact that all N univariate predic- tive Gaussians g(fn,) ) have the same variance but different means, thus the predicted class can be cquivalently obtained by taking the argmax of the means of these N distributions.
C.5 SPECIFIC GP IMPLEMENTATION AND AMORTIZATION FOR FEW-SHOT CLASSIFICATION
For all few-shot multi-class classification experiments in order to implement the GP and GP+MAML methods we need to specify the feature vector (xx; 0) and the amortized variational functions m (I) and sw)) The feature vector is specified to have exactly the same neural architecture used in pre- vious works for all datasets, Omniglot, mini-Imagenent and Augmented Omniglot; see for example Chen et al. 2019)) for details on these architectures for all these three datasets. Note that when computing the GP kernel function the feature vector (x;; 0) is also augmented with the value 1 to automatically account for a bias term in the kernel function.
Regarding the two amortized variational functions needed obtain the encoder, we consider a shared (with the GP functions) representation by adding two heads to the same feature vector o(æ; f): the first head corresponds to linear output function (x) and the second applies at the end the softplus activation sw (z) = log(1 + +cxp(a(z))) (since s (z) represents variance) where the pre-activation axx) is obtained by a linear function of the feature vector. For numerical sta- bility we also apply final clipping by bounding these functions so that mw E [-20, 20) and (x) E [0.001,20] The bound 20 and 20 are almost never realized during optimization, so they not erucial, in contrast the lower bound 0.001 on sw is rather crucial regarding numerical stability since it ensures that the minimum eigenvalue of the matrix K + S' (i.c. the matrix we need to decompose using Cholesky) is bounded below by 0.001.
D FURTHER EXPERIMENTAL DETAILS AND RESULTS
D.1 EXPERIMENTAL SETTINGS AND IYPERPAARMETERS
The memory-based GP method (referred to in the Tables as GP) is trained by the VIB objective, as defined Section 3 where the kernel feature vector oxx; 0) is obtained by the last hidden layer of the same neural architecture as used in MAML. Based on this M M-dimensional feature vector 6(x;0) E RR we consider two kernel functions: the standard linear kernel function ko x') 1
M (z; 0) (Txx;(())) (where the kernel variance of is fixed to 1/M) and the cosine kernel ky (o(xx=')
((x:0) "(x';9) These kernels are used in the experiments.
The Omniglot dataset consists of 20 instances of 1623 characters from 50 different alphabets. Each instance drawn by different person. It is augmented by creating rotations of each of the existing characters by 0, 90, 180 or 270 degrees. In Omniglot experiment, MAMI., Stochastic MAML and GP+MAML run by applying one adaptation step for both meta- training and meta-testing. The mini-Imagenet involves 64 training classes, 12 validation and 24 tests classes. Following previous work on mini-Imagenet we meta-train Stochastic MAML with 5 adaptation steps, while 10 steps are used for meta-testing, GP+MAML uses 5 steps for both meta-training and meta-testing. For both Omniglot and mini-Imagenet we follow the experimental protocol proposed by Finn et al. (2017), which involves fast learning of N 5-way classification with K 1 or K- 5 shots. The problem of N-way classification is set up as follows: select N unseen classes, provide the model with K different instances per class, and evaluate the model's ability to classify new instances within the N classes.
characters which are
The Augmented Omniglot benchmark is a modified version of Omniglot which necessitates long- horizon adaptation and it is often considered as many-shots problem (Chen et al. 2019). For each alphabet, 20 characters are sampled to define N 20-class classification problem with K - 15 shots. Further, both train and validation images are randomly augmented, by applying transforma- tions, which makes it even more challenging. Following Flemnerhag et al. (2018); Chen et al. (2019), FollowingFFemmerraag
during meta-testing both MAML and Stochastic MAML perform 100 steps of adaptation (resulting
in 2000 data points seen in total by the model where each step processes minibatch of size 20
20

---

Under review as a conference paper at ICLR 2021
points), while they are meta-trained by applying 20 adaptation steps (i.e. 400 training points per task). Both GP methods are meta-trained by memorizing the full x K = 20 x 15 = 300 sup- port points without further data augmentation, while during meta-testing we allow the GP methods to see up to 2000 points. For all methods we perform a hyperparameter search using the train and validation subsets of all three benchmarks detailed below.
For sinusoid regression, Omniglot and mini-Imagenet, meta-training of all methods consisted of 60000 iterations episodes, where in each episode a learning update is performed based on a mini- batch of tasks, For the sinusoid regression the meta batch-size was 5 tasks and for each task we have K 10 examples. For N-way, K-shot classification in Omniglot and mini-Imagenet (with N = 5, and K = 1, 5 as mentioned in Table 2 in the main paper), the meta batch was 32 tasks in Omniglot and tasks in mini-Imagenes one gradient step (in the inner loop) for both meta-training and meta-testing in Omniglot, while in mini-Image Imagenet it uses 5 and 10 steps respectively, i.e. exactly as MAMI. by |Finn et al. (2017) was applied in these datasets. As mentioned in the main paper GP+MAML considers one gradient adaptation step in Omniglot and neural architectures of all these experiments is the same in Finn al. (2017). Specifically, for the Omniglot the architecture is from Vinyals et al. (2016). which has 4 modules with a x 3 convolutions and 64 filters, followed by batch normalization pooling. The Omniglot images are downsampled to 28 x 28, so the dimensionality of the last hidden layer 64. For Omniglot, strided convolutions are used instead of max max--oooing. For the GP methods this constructs 65-dimensional feature vector (x;; 0) (64 plus one for the bias term which is included in (xx; 0)) For mini- Imageen, the network uses 32 filters per layer and the final layer the feature vector feature is obtained by flattening so that finally the feature vector Φ( 0) is 801-dimensional.
Also for these two datasets Stochastic MAML uses
mini-Imagenet (for both meta-training and meta-testing). The
ReLU nonlinearity, and x 2 max-
For Augmented Omniglot, the meta-training of all methods consisted of 5000 iterations. The Aug- mented Omniglot dataset is a modified version of Omniglot which, for mrudient-based methods like MAMI., necessitates long-horizon adaptation and often considered as many-shots problem (Chen et al. 2019). For each alphabet, 20 characters are sampled to define 20-class classification problem with K = 15 data points per class. Furthermore, both train and test images are randomly augmented, by applying transformations. Following Flennerhag et al. (2018) Chen et al. (2019), we use 4-layer convnet and during meta-testing MAML and Stochastic MAML performs 100 steps of adaptation (resulting in 2000 data points where each step processes a minibatch of size 20 data- points, i.e. per class), while they are meta-trained by applying 20 adaptation steps. GP+MAML uses 5 adaptation steps for both meta training and meta testing.
Additional hyperparameters details are included in Tables rameters' search are:
and 6
The ranges for the hyperpa-
Outer learning rate
[0.0001, 0.00025, 0.0005, 0.00075, 0.001, 0.002, 0.0025,
0.005, 0.0075, 0.01, 0.025, 0.05, 0.075, 0.1]
Inner learning rate pt
[0.0001, 0.0005, 0. 0.001, 0.002, 0.005, 0.01, 0.w00,0.000000 0.05, 0.1, 0.5, 1.0,000,,,
Bottleneck coefficient 8::
[le - 07, 1e - 06, 1e 05, 5e 05, 0.0001, 0.00025, 0.0005,
0000075,0000., 0.0025, 0.005, 0. 0.0075, 0.01, 0.1, 1.0]
D.2 ADDITIONAL EXPERIMENTAL RESULTS
Table 7 provides detailed few-shot sinusoid regression results, while Figure illlstrates how the GP model, after having been meta-trained on the sinusoid regression tasks, predicts a test task as the number K of the training examples increases.
We also found that the qualitative behaviour of GP and GP+MAML are quite different. In Figure we report the performance of the GP and GP+MAML mini- tasks as function of
21

---

Under review as a conference paper at ICLR 2021
Parameter
Stochastic MAML
Omniglot
K
K
Outer 1r.
0.005
0.001
Inner I.r. p
1.5
1.5
Bottleneck 8
10-
10-5
mini-Imagene'
K =
K 5
Outer a
0.0005
0.0005
Inner Ir.. p
0.05
0.1
Bottleneck |
10 10-3
10-4
Augmented Omniglot
Outer l.r. a
0.01
Inner I.r. P
0.5
Bottleneck 3
10
Sinusoid regression
Outer l.r.
0.0005
Inner l.r. p
0.1
Bottleneck /
10
Table 4: Hyperparameters Stochastic MAML.
Parameter
MAML
Augmented Omniglot
Outer a Inner l.r. P
0.01 0.5
Sinusoid regression
Outer Ir.. a Inner l.r.
0.002 0.005
Table 5: Hyperparameters for MAML in sinusoid regression and Augmented Omniglot.
22

---

Under review as a conference paper at ICLR 2021
Parameter
GP (cos)
GP (lincar)
GP MAML (cos)
GP + MAML (linear)
Omniglot
K =
K
K
K =
K
K = 5
K ==1
K
Outer Ir. a
0.001
0.0025
0.0025
0.005
0.00075
0.005
0.0025
0.001
Inner Ir. p
0.0005
0.002
0.002
0.0005
Bottleneck /
0.0005
0.0005
0.0001
0.0001
0.0005
0.0005
0.0001
0.00005
mini- -mageett
K
K
K
K
K
K 5
K
K
Outer I.r.
0.00075
0.0005
0.0025
0.00075
0.001
0.0005
0.001
0.00075
Inner lr.. p Bottleneck
0.0005
0.0005
0.002
0.002
0.005
0.005
0.005
0.005
0.001
0.0005
0.0001
0.0005
Augmented Omniglot
Outer l.r.
0.0025
0.05
0.005
0.005
Inner l.r. p
0.001
0.0005
Bottleneck 8
0.0005
0.00075
0.00025
0.00001
Sinusoid regression
Outer lr..
0.00075
0.002
Inner l.r. p
0.0001
Bottleneck
Table 6: Hyperparameters for the GP methods.
23

---

Under review as a conference paper at ICLR 2021
Model
K=5
K=10
K=20
MAML (our)
grads steps
0.600+00.662 0.359+0.015 0.228+0.018
grads steps 5
0.311=0.013
0.12+00006 0.06++0.004
grads steps 10
0.280110.013
0.09610.005 0.04310.003
Stochastic MAML
grads steps
0.662.10.04 0.38211.0211 0.244.10.014
grads steps 5
0.352=+0.08%
0.141=0.014 0.073 006
grads steps 10
0.317#0.34 0.116=0.012 0.054#0.000 004
GP
0.02:0.014
0.002=0.000
0.001
GP+MAMI.
0.058+00005
0.002+0.001
0.002:+0.002
MAML
Finn et al.
2017
0.35
Table 7: Detailed few-shot sinusoid regression results. We report the results of the GP model for K = 5, 10, 20 and for MAMI, assuming different gradient adaptation steps (including also a result reported in the original MAML paper by Finn et al. (2017).
K =
K
3.0
30
3.0
60
-30
K
4
K 5
K
1.0
0.58
325
025
300
-15
.0.5
-10
21
Figure 3: The red curve represents ground truth sinusoid function whereas the light blue is the prediction a GP. increases, the uncertainty (shown by shaded area) of the decreases its mean predictions become closer to the ground truth. Given already K points, the mean prediction matches well the sinusoid function.
mini-Imagenne (K
mini mini-Imagenet (K 5)
0.485
0.65
0.480
Accuracy
Accuracy
0.64
0.465
0.63
0.460
0.455
20
40
60
20
40
50
Inner step
Inner step
0.005
MAML
Figure 4: Qualitative difference between GP and GP+MAML on mini-Imagenet as a function of inner loop steps. Because there is no inner loop for the GP, we simply report it as reference to the GP4MAML.
24

---

Under review as a conference paper at ICLR 2021
inner steps executed test time by GP+MAML. Note, that GP actually has only step executed over all the test-time data in once, because there is no inner loop for it. Intuitively, we would expect GP+MAML to have similar performance to GP at the beginning of inner loop, but this is not what happening. Instead, starts quite low and peaks at exactly the number of inner steps used during meta-training and then the performance starts to deteriorate.
D.3 ABLATIVE STUDIES
Bottleneck cost B ablation To understand the impact of 3 on learning we provide the ablation analysis in Figure 5 on sinusoid regression and two few-shot classification tasks. For regression task, Figure 5 (a), we observe that having a large 3 = 1.0 is beneficial and provides the best mean squared error (MSE). For few-shot classification tasks Figures 3 (f)- observe that there is large range of values in between [10-1, 0.01] providing similar classification accuracy. Interest- ingly, for values of 3 close where for B the VIB corresponds ELBO, the performance deteriorates significantly.
(a) Sinusoid regression (K
(b) Omniglot (K 1)
(c) Omniglot (K 5)
6,055
Accuracy
Accuracy
8.6
0.4
0.4
0:052
34.6
(d) ceini-Imagene (K 1)
(c) mini-Imagene (K 5)
(f) Augmented Omniglot
0.8
Accuracy
Accuracy
02
02
62
36.6
Figure Ablation analysis for different values on sinusoid regression shot classification tasks, sinsusoid regression we report mean squared error (MSE): lower is better. For few-shot classification tasks, we report the accuracy: higher is better.
D.4 IMPROVED VERSIONS FOR THE GP-BASED METHODS
For GP-based methods we consider additional popular tricks used in the community to improve performance.
For Augmented Omniglot dataset (Flennerhag et al. 2018), for the network architecture we add additional layers similar to Flennerhag et al. (2020). where each layer added after convolutional layer. These additional layers are simple convolutions as considered in Flennerhag et al. (2200). In addition, add batch normalization. Simply using more layers in GP methods significantly improves the performance on this dataset. Note, that the results are achieved without special warp- grad architecture as in Flennerhag al. (200). In Table we show the results with and without batch normalization (BN).
In addition, we run additional experiments on mini- mini-ImageNet (Ravi & Larochelle, 2017) and tiered- ImageNet (Ren et al. (2018) with using pre-trained embeddings from LEO (Rusu et al. 2019) paper as features. The embeddings are taken from their GitHub repository. On top of these features, we construct an MLP with two hidden layers and ReLU activations where each layer has 128 hidden dimensions. This defines the feature vector d(x; f) which is used by the GP methods in all the experiments. Moreover, we apply dropout the LEO embeddings before passing them the MLP. The results are given in Table The hyperparameters are tumed the same way as for the previous experiments. Their parameters given in Table TO Notice that the GP methods comparable
25

---

Under review as a conference paper at ICLR 2021
Augmented Omniglot 20-way way
Model
GP (lineur)
86 86.299:0.849%
GP (006)
85. 6.t-6---06555
GP
6..%:11%%%
GP (c08), BN
..0.%%
GP+MAML (lineur)
84.59:10.559
GP+MAMI. (cDs)
84, 244:23:: 1.0%
GP+MAML (linear). BN
85. $5.944t0.779
GP+MAML (c0s), BN
+86. 1 17:: 1.0%%
Warp Flennerhag
2020
83.6.11.9%
Table 8: Classification test accuracy performance Augmented Omniglot for improved architectures. The GP based methods use 4 additional simple convolutional layers where each of these layers added after each network layer, similar Flennerhag al. (2020). We report results with and without batch normalization.
mini-ImageNNt 5-way
tiered-ImageeNet 5-way
Model
K
K =
K
(linear)
60.4100.28%
77.110.08%
64.1610.16%
81.610009%
GP (cos)
60.8:...1%%
76.8:2.08%%
65.3 16%
81.3 22%
GP + MAML (lineur)
60.5:10.550
77.08 13%
0.25%
1.6%
MAML (cns)
60.8:10.199
77.110.21%
65.6-10.2%
81.520.144
2019)
61.76.10.08%
77.5910.12%
81.44%10.09%
Table 9: Additional results mini-ImageNet and lmagee- We re-use LEO Rusu et al. 2019 embed- dings from their GitHub code and apply 2-layers MLP with ReLU activations 128 hidden dimensions to construct the feature vector.
with LEO, with significant difference. We would like point out that LEO uses of different tricks: dropout, label smoothing, l regularization and orthogonality penalty (Rusu et al. 2019). while we have only considered dropout.
(cos)
GP (lincar)
GP4MAML (cos)
GP-MANL (lincar)
=
K =
K =
=5
==1
K=5
=
==5
mini- ImageNet
Outer Lt.
5
2.5 10
0.00025
7.5 10
5 10-
7.5 x 10
2.5 10
0.0001
Inner Lt. p
0.001
0.001
0.001
0.001
Bottlenock Ø
0.0025
10-6
0.00075 0.1
10-6
0.0001 0.1
0.0001
0.0001
10-6
Dropout 0.1
0.1
0.2
0.2
0.1
0.2
tiered- tlered-Imagetee ImagetNet
7.5 10
0.0001
7.3 10 5
0.00025
10
7.5 10-
0.0001
0.0001
Inner Irr
0.001
0.001
0.001
0.001
Bottlencck Ø
10-5
10-5
5 10
10-6
10-8
10-8
10-7
10-7
Dropout p
0.1
0.1
Table 10: Hyperparameters for the additional results mini mini-ImageNNet and Mered-ImageNNo For dropout rate, we considered values: 0.0, O.I, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7.
26


Reviews:
[{"review_id": "VHWtzAG0Q5x", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates meta-learning using the information bottleneck. The method develops a meta-learning algorithm based on Gaussian Processes, which they also interpret as a memory-based algorithm. They further describe an extension that combines the GP-based method and MAML.\n\nThe paper seems to lack focus. It consists of two orthogonal components: (1) applying the information bottleneck to meta-learning and (2) a GP-based meta-learning algorithm. Each of these components is a separate meta-learning algorithm: (1) alone can extend MAML to \"stochastic MAML\" ($\\beta > 0$) and (2) alone is \"GP\" with hyperparameter $\\beta=0$. The paper does not make a sufficient case for why these two components should be used in tandem (e.g. GP w/ $\\beta > 0$), nor does it perform ablation experiments that each positively contributes to some measure of meta-learning performance.\n\nThe interpretation of MAML as a special case of information bottleneck (section 2.2) is not particularly novel: as the paper mentions in section 4, any previous works (LLAMA, probabilistic MAML, bayesian MAML, iMAML) have built on the interpretation of the initial parameter of MAML as encoding a prior distribution over parameters, and the gradient step as posterior inference. The fact that information bottleneck with $\\beta=0$ reduces to standard learning is well-known.\n\nThe experimental results are mixed, and the paper doesn't give much of an interpretation of these results. GP and GP+MAML outperform MAML by an order of magnitude on the sinusoid regression problem but is outperformed by MAML in two of the five classification tasks considered. The experiments in the appendix are similarly mixed. GP wins on Augmented Omniglot but loses on ImageNet variants. Is the GP approach vastly superior on only regression problems, or is the difference just due to tuning and the fact that sinusoid regression is a toy problem? \n\nGiven the relation to other probabilistic extensions of MAML, perhaps comparing against at least one of them would have made experiments more informative.\n\nThe experiments about the efficiency regarding the amount of data (Figure 2 right) are promising, but this claim would be more significant if evaluated on multiple tasks.\n\nminor\n\nThere is a link formatting issue btw pages 2 and 3", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pqf44PumjU1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for Bayesian meta-learning. This method combines a NN feature extractor with a Gaussian Process on top. The GP kernel is linear. The information bottleneck is used to motivate a choice of approximate posterior. Using MAML to adapt the NN feature extractor weights improves performance of the composite model for few-shot learning.\n\n---\nThings I liked:\n- This paper was interesting and on an important topic. \n- The information bottleneck seems some to fix some issues with variational inference takes on Bayesian meta learning (e.g. Amortized Bayesian Meta-Learning, Ravi & Beatson, 2019). The VI approach in that paper doesn't distinguish train/support and validation/query sets, and doesn't motivate the weight beta which is placed on the prior, while the information bottleneck does.\n- The particular information bottleneck motivated posterior used with the GP is not specific to classification, regression, or other particular structure of prediction problem.\n- I liked that the paper had plots of performance vs number of data points, as opposed to only reporting error on the full set of training tasks\n- The proposed method leads to a bump in accuracy over MAML for augmented omniglot\n\n---\nThere are potential areas of concern which I would like the authors to respond to. If the concerns are sufficiently addressed (or turn out to be due to my misunderstanding), I could be willing to significantly increase my score.\n\nMain concerns:\n1. What is the objective of the proposed method? Is it to get better test error than MAML, or to quantify uncertainty, or something else? If the point is to get better test error, it seems unfair to compare only to vanilla MAML given the considerable amount of literature and number of improvements in the years since. To build a case for an advantage in terms of accuracy, the paper should compare to a wider range of stronger/more recent baselines (e.g. \"Bayesian Few-Shot Classification with One-vs-Each\nPólya-Gamma Augmented Gaussian Processes\", Snell & Zemel 2020, does a good job of this).\n2. If the motivation for the current proposed method is more than just improving the test error on these benchmarks, this motivation should be clearly explained and/or measured. The method proposed in the paper adds a lot of complexity on top of MAML and variants, so I think the paper needs to build a strong case and clarify exactly which settings this method is preferable for.\n3.  One paper the authors could compare to and cite is \"Meta-learning with differentiable closed-form solvers\", Bertinetto et al 2018, which -- as with the current paper -- combines NN features with a closed-form linear model, and which significantly outperforms MAML.\n4. The authors could also compare to the other papers they cite in Section 4, e.g. Snell & Zemel 2020 significantly outperform all the baselines and the proposed method from the current paper's results.\n5. Can the authors discuss the lack of data augmentation used for Augmented Omniglot for the GP methods? It strikes me that the data augmentation is likely part of what makes Augmented Omniglot hard. In this case, removing the data augmentation for the GP methods would be giving the GP methods an unfair advantage, which could explain almost all of the GP methods' superiority on this task. A fair comparison should also compare to MAML(+variants) without data aug, and/or GP methods with data aug\n6. Can the authors comment on the choice of a linear kernel for the GP? This surprised me, as for regression using a GP with a linear kernel is the same thing as doing Bayesian linear regression. Bayesian linear regression can be done exactly in O(N) time instead of O(N^3) for linear-kernel GP, so it seems strange not to use the LR formulation. It also seems surprising to have to jump through hoops to use an information bottleneck-motivated posterior in a case where we could do exact Bayesian inference cheaply. Of course, in a classification setting one needs to use approximations like Polya-Gamma augmentation, variational inference, or similar.\n7. Why only show accuracy vs data points (fig 2 right) for GP not GP-MAML, and why only for augmented omniglot? This is an interesting plot which I would like to see more of in few-shot learning papers.\n\n---\nOther points of feedback:\n1. I think that whenever the GP method is referred to (title, abstract, text, results tables, plots...), it should be made clear that this is a composition of a neural network feature extractor and a GP. Yes, the NN feature extractor can be seen as part of a very particular choice of kernel for the GP, and this is explained in Section 3. However, someone skimming the paper might not realize the existence of this NN at all, which would make the results very surprising. I think the paper should endeavor to give readers the correct impression on a skim. \n2. The set up in Section 2 (and \"Stochastic MAML\") is extremely similar to Amortized Bayesian Meta-Learning, Ravi & Beatson, 2019. As mentioned, the current submission fixes some issues with that paper, but due the similarity it wouldn't hurt to cite them.\n3. The paper should explain the Augmented Omniglot benchmark better in the text (or at least refer the reader to the appendix, where it is explained). Flennerhag et al who introduced the benchmark did not use the term \"Augmented Omniglot\", so simply citing that paper when the term is mentioned is insufficient and left me confused before I found the discussion in the appendix, although I do think this is a good name for the benchmark.\n4. How was the NN+GP method trained? I imagine this was done by backpropogating through the GP fitting procedure to get the gradient of the query-set error for training tasks (or of the IB objective for training tasks) and using this to optimize the NN weights. It would be nice to have this spelled out, as well as how this gradient was computed (whether it is simple or required some non-trivial tricks).", "extra_scores": {}, "extra_text": {}}, {"review_id": "KNPt__knvzr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper derives a meta-learning framework based on the information bottleneck principle. By adapting the variational approximation proposed in [1] to the meta-learning setting, the authors come up with a tractable objective that generalises both gradient based and memory meta-learning methods. Based on this framework, the authors proposed a new memory based meta-learning algorithm by using a GP with a deep kernel and an extension that combines this memory based method with MAML. The authors show that the method outperforms MAML in several standard meta-learning  tasks, especially in regression and many shots classification problems.\n\n## Comments\n\nThe motivation of the paper is really clear and the paper is extremely well-written. \n\nFrom the technical perspective, the paper has two main contributions:\n\n1. The development of a general meta-learning framework based on the information bottleneck that generalizes gradient and memory based meta-Learning\n2. Two new methods that resort to GPs with deep kernels to implement the encoder in the aforementioned framework\n\nRegarding the framework, the extension of the variational information bottleneck to the meta-learning setting is not particularly challenging being a straightforward extension of [1]. However, it is an interesting observation that this extension subsumes gradient based methods (by using parametric encodings) and memory based ones (by using nonparametric encodings). One of the things that it is missing is that the authors failed to mention that the connection between meta-learning and the information bottleneck principle had been previously explored in the literature (see [2]). While the setting is slightly different, it would make sense for the authors to address how the two papers are related, especially since the GP method proposed by the authors can be used in the transductive settings as well. \n\nRegarding the new methods, I think that the use of a GP encoder with a deep kernel is interesting but again not particularly challenging from the technical perspective. However, I think the combination of MAML + GP it is a very interesting observation that can open new research avenues.\n\nThe weakest part of the paper is the experiments. The authors use MAML as the unique baseline ignoring more up to date meta-learning methods which undermines the message of the paper. It would be really interesting to see how the method generalises and compares against other state of the art methods. For example, there has been several methods that try to leverage uncertainty into MAML (e.g. [3]). This would be an interesting comparison since it would help to disentangle the impact of a parametric/nonparametric encoder versus probabilistic/deterministic encoders. In addition, there are several recent methods specifically developed for transductive meta-learning that would be interesting to include in the comparison (e.g. [4]). Finally, I would appreciate a more in depth analysis of why the combined approach works better or worse than the GP one in different settings (see some questions below).\n\n## Questions/Minors\n\n* In section 2.2 the authors cast MAML under their framework. Two do so, they assume Z_i=\\psi_i leading to p_\\theta(\\mathcal{D}_i^v|\\psi_i) and p_\\theta(\\psi_i). However, the second approximation is irrelevant since further on they assume \\beta=0.\n* To build the stochastic extension of MAML, the authors use a stochastic encoder. However, in addition they use \\beta \\neq 0. It would be nice to see an ablation study to see the contribution of these separate extensions (there is an ablation study in D.3 but it is not specifically addressing this and it is not clear what is the model they use for the study). Also it would be interesting a small discussion about the importance of tying the variances of the prior and the encoder\n* In section 3.1 it should be mention that it is a degenerate GP.\n* Toward the end of section 3.2 a bunch of simplifications are done in order to come up with the final version of the combined algorithm: \\beta to \\beta_f and \\beta_\\psi and \\beta_psi=0 which leads to a deterministic encoder and remove the need of defining a prior. However, these simplifications goes in the opposite direction of the extensions made to come up with the stochastic version of MAML. It would be great if the authors could elaborate on why that is the case and the contribution to the final performance of each of this simplifications.\n\n## References\n\n[1] Deep variational information bottleneck\n[2] Empirical Bayes Transductive Meta-Learning with Synthetic Gradients\n[3] Recasting Gradient-Based Meta-Learning as Hierarchical Bayes\n[4] Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4S-Ag-rNZl1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposed variational approximations to the information bottleneck objective functions for meta-learning.\nThe authors then provided three different settings using their variational loss functions, namely SMAML, GP, and GP + MAML.\nThe authors' motivations for these three settings were to study the effect of stochastic gradient-based method, non-parametric method, and the combination of gradient-based method with non-parametric method.\n\nReason for the score:\n\nI find the paper interesting for establishing a variational information theoretic objective function for meta-learning.\nI also enjoyed the comparisons among purely gradient-based method with non-parametric method and their hybrid.\nYet, I did find some details where the motivations are a bit difficult to understand. It would be great if the authors could\nclarify them.\n\nPro:\n- The authors provided clear derivations for their variational objective functions.\n- The authors provided clear kernel setups for their GP models.\n- The authors provided evaluations for their three settings on both regression and classification tasks.\n- The paper is easy to follow and I did not find typos.\n\nCon / Questions:\n- For the objective function in Eq.1, is there a particular reason for assuming conditional independence for $q(Z| D^{t}) $?\n- For Eq.5 and Eq.6, are the models of $p_\\theta (D_i^v | Z_i)$ and $p_\\theta (Z_i)$ related in some ways? It seems that they are sharing the parameters $\\theta$. \n- I did not find Section 3.2's motivation very convincing. The authors mentioned that the hope for combining MAML with GP is to reduce inner steps for MAML and possibly achieve better results. Yet, in Figure 4, it seems that pure GP is basically on-par with GP+MAML and outperforms GP+MAML when more inner steps are introduced for MAML. Could the authors help me with some insights for this observation?\n- It seems that SMAML performed worse than MAML consistently for regression tasks and occasionally better for classification tasks. I observe that SMAML differ from MAML in two aspects, being stochastic and having the extra KL regularizer controlled by $\\beta$. Are the worsened performance due to being stochastic or too much regularization from the KL term?\n- On the classification tasks, it seems that GP models perform better than the standard MAML mostly when $K$ is large. It is difficult to conclude that the results are better thanks to GP being non-parametric or the variational loss function being better. This is also somewhat true for the regression tasks because we know GP is smooth and interpolates better.\n\n-----------------------------------------------------------------\nPost Rebuttal:\n\nMany thanks for the authors to update their original paper addressing some of my questions and concerns.\nUnfortunately, I still think that some aspects could be better analysed,\nit is not crystal clear to me if the improvements come from the GP or their proposed variational information\ntheoretic function. I am keeping my original score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Information bottleneck is a well-known principle that is used for clustering, dimensionality reduction, and recently deep learning. It finds a compressed representation of input X while retaining most information on the response Y. This paper addresses an attempt to interpret the meta-learning using the information bottleneck. In addition, a GP-based meta-learning method is also proposed. 
The topic itself is interesting without any doubt. However, most of reviewers have serious concerns about this work, which is summarized below. First of all, two components of this paper (IB and GP-based meta-learning) do not provide a coherent message.  While the IB interpretation is emphasized in the beginning of this paper, the main point seems to that GP-based methods can be more data efficient than gradient-based meta learning. There does not much point to GP+MAML or IB interpretation of MAML.  Experiments are not strong enough, although a few ones are added during the author responses. During the discussion with reviewers, no one support this work, so I do not have choice but to suggest rejection.

Decision: Reject