Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
Authors: Shell Xu Hu
Pablo Garcia Moreno
Yang Xiao
Xi Shen
Guillaume Obozinski
Neil Lawrence
Andreas Damianou
Venue:  (2020)
Keywords: Meta-learning
Empirical Bayes
Synthetic Gradient
Information Bottleneck

Abstract: We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging the unlabeled query set in addition to the  support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning.   The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior on the query set of each task.
We derive a novel amortized variational inference that couples all the variational posteriors via a meta-model, which consists of a synthetic gradient   network and an initialization network. Each variational posterior is derived from synthetic gradient descent to approximate the true posterior on the query  set, although where we do not have access to the true gradient.
Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods. Besides, we conduct two zero-shot learning experiments to further explore the potential of the synthetic gradient.

Published as a conference paper at ICLR 2020
EMPIRICAL BAYES TRANSDUCTIVE META-LEARNING WITH S SYNTHETIC GRADIENTS
Shell Xu Hu' Pablo G. Moreno Yang Xiao
Xi Shen'
Guillaume Obozinski
Neil D. Lawrence"
Andreas Damianou²
'colee des Ponts ParisTech Cuapps-sur-aanee France
AAmazon
Cambridge, United Kingdom
(xu.hu, yang. xiao, xi.shen] @enpe fr
(morepabl, dami | @amazon. com
Swiss Data Science Center Lausanne, Switzerland wecoozinssin wnwiiiiikkiiiiie
*University of Cambridge Cambridge, United Kingdom ndl @dl21ecam... ac..a...u.. .uk
guillaume. oboz
ABSTRACT
We propose a meta-learning approach that learns from multiple tasks in a trans- ductive setting, by leveraging the unlabeled query set in addition to the support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as sum of local KL divergences between the variational posterior and the true posterior on the query set of each task. We derive a novel amortized variational inference that couples all the variational posteriors via meta-model, which consists of a synthetic gradient network and an initialization network. Each variational posterior is derived from synthetic gradient descent to approximate the true posterior on the query set, although where we do not have access to the true gradient. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous wate-of-the-an methods. Besides, conduct two zero-shot learning experiments to further explore the potential of the synthetic gradient.
INTRODUCTION
While supervised learning of deep neural networks can achieve even surpass human-level per- formance (He et al.) 2013 Devlin et al. 2018). they can hardly extrapolate the learned knowledge beyond the domain where the supervision provided. The problem of solving rapidly new task after learning several other similar tasks is called meta-learning (Schmidhuberi 1987; Bengio et al. 1991 Thrun & Pratt 1998); typically, the data presented in a two-level hierarchy such that each data point at the higher level is itself a dataset associated with a task, and the goal is to learn a meta-model that generalizes across tasks. this paper, we mainly focus on few-shot learning (Vinyals al. 2016). an instance of meta-learning problems, where a task t consists of query set de == { Vt.s Vt.;) |i
serving as the test-set of the task and a support set d):=((<i...b). meta-testing"] one is given the support set and the inputs of the query set := {xr,)) ) and asked predict the labels 1/2 = { of few-shot learning is summarized in Table
serving as the train-set. In
meta-training, Yt provided as the ground truth. The setup
important distinction to make whether task solved in a transductive or inductive manner, that is, whether J, is used. The inductive setting is what was originally proposed by Vinyals et al. (2016), in which only de is used generate model. The transductive setting, as an alternative, has the advantage of being able see partial all points in 2, before making predictions. In fact,
'To distinguish from testing and training within task, meta-testing and meta-training are referred to as testing and training over tasks.

---

Published as a conference paper at ICLR 2020
Support set
Query set
dj = {( Di
Tt == { = (u
Meta-training Meta-testing
X
Table 1: The setup of few-shot learning. If task & is used for meta-testing, y/ not given to the model.
Nichol et al. (2018) notice that most of the existing meta-learning methods involve transduction unintentionally since they use 24 implicitly via the batch normalization (Toffe Szegedy 2015). Explicit transduction is less explored in meta-learning, the exception Li et al. (2018), who adapted the idea of label propagation (Zhu et al. 2003) from graph-based semi-supervised learning methods. We take totally different path that meta-learn the "gradient" descent on I, without using y/-
Due to the hierarchical structure the data, it natural to formulate meta-learning by hierarchical Bayes (HB) model (Good, 1980; Berger 1985), alternatively, an empirical Bayes (EB) model (Robbins) 1985 Kucukelbir & Blei 2014). The difference is that the latter restricts the learning of meta-paramett to point estimates. In this paper, focus the EB model, it largely simplifies the training and testing without losing the strength of the HB formulation.
The idea of using HB EB for meta-learning is not new: Amit & Meir (2018) derive objective similar to that of HB using -ay-Baysi analysis; Grant et al. (2018) show that MAML (Finn et al. 2017) can be understood and compute posteriors via amortized variational inference. However, unlike our proposal, these methods do not make full use of the unlabeled data in query set. Roughly speaking, they construct the variational posterior as a function of the labeled set di without taking advantage of the unlabeled set 2,, The situation is similar gradient based meta-learning methods (Finn al. 2017 Ravi
EB method: Ravi & Beatson (2018) consider a HB extension MAML
& Larochelle, 2016 Li et al.) 2017b] Nichol et al.) 2018: Flennerhag et 2019) and many other meta-learning methods [Vinyals al. 2016 Snell et al. 2017 Gidaris & Komodakis 2018), where the mechanisms used to generate the task-specific parameters rely groundtruth labels, thus, there no place for the unlabeled set to contribute. We argue that this is suboptimal choice, which may lead to overfitting when the labeled set is small and hinder the possibility of zero-shot learning (when the labeled set is not provided).
|
In this paper, propose to use synthetic gradient (Jaderberg al. 2017) to enable transduction, such that the variational posterior is implemented as a function of the labeled set de and the unlabeled set Fy. The synthetic gradient is produced by chaining the output of gradient network into auto- differentiation, which yields a surrogate of the inaccessible true gradient. The optimization process is similar the inner gradient descent in MAML, but it iterates the unlabeled sy rather than on the labeled di, since it does not rely on Yt to compute the true gradient. The labeled set for generating the model for an unseen task now optional, which only used to compute the initialization of model weights in our case. In summary, our main contributions are the following:
In section 2 and section 3 we develop a novel empirical Bayes formulation with transduction for meta-learning. To perform amortized variational inference, we propose parameteriza- tion for the variational posterior based on synthetic gradient descent, which incoporates the contextual information from all the inputs of the query set.
2. In section 4 we show in theory that transductive variational posterior yields better general- ization performance. The generalization analysis done through the connection between empirical Bayes formulation and multitask extension of the information bottleneck princi- ple. In light of this, we name our method synthetic information bottleneck (SIB).
3. In section 5| verify our proposal empirically. Our experimental results demonstrate that our method significantly outperforms the state-of-the-art meta-learning methods on few-shot classification benchmarks under the one-shot setting.
2

---

Published as a conference paper at ICLR 2020
Intremnee
8
g
Bg
Pldd
02
0
0g
Bid
8
(a) Graphical model of EB
(b) MAML
(c) Our method (SIB)
Figure 1: (a) The generative and inference processes of the empirical Bayes model are depicted in solid and dashed arrows respectively, where the meta perta-arsseterss are denoted by dashed circles due to the point estimates. comparison between MAML (6) and our method (SIB) (10) shown (b) and (c). MAML is inductive method since, for task t, it first constructs the variational posterior (with parameter g* as a function of the support set d', and then test on the unlabeled 24; while SIB uses better variational posterior as function of both d' and x;; it starts from initialization 0g (d))
generated using d' and then yields 0.* by running synthetic gradient steps on Tt.
META-LEARNING WITH TRANSDUCTIVE INFERENCE
The goal of meta-learning is to train meta-model collection of tasks, such that it works well on another disjoint collection of tasks. Suppose that we are given a collection of tasks for training. The associated data is denoted by D := {dz (Ix, y1 J In the case of few-shot learning, we are given in addition a support set dÿ in each task. In this section, we revisit the classical empirical Bayes model for meta-learee letaalearning. Then, propose to a transductive scheme in the variational inference by implementing the variational posterior as a function of 2T.
2.1 EMPIRICAL BAYES MODEL
Due the hierarchical structure among data, it is natural consider hierarchical Bayes model with the marginal likelihood
N [I
[ -
PT(D) =
Py(de (e.dd/ww)p((u.lf p(b).
The generative process is illustrated in Figure [ (a, in red arrows): first, a mela-parameter xp (i.e., )yper-parameter) is sampled from the hyper-prior p(t); then, for each task, task-specific parameter Wi is sampled from the prior p(wr (b);; finally, the dataset is drawn from the likelihood py (d, we). Without loss of generality, we assume the logeliieldhood model factorizes as
ly
log (Mr. (2,,, w.) log p(xx,,)w)))
ee (or, (Fres)) wx)s ve.) constant.
It is the setting advocated by Minka (2005), in which the generative model p(z),)/w;) can be safely ignored since it irrelevant to the prediction of yv. To simplify the presentation, we still keep the notation py(d//uc /w:)) for the likelihood of the task and use E, to specify the discriminative model, which is also referred to as the task-specific loss, e.g., the cross entropy loss. The first argument in & is the prediction, denoted by It, (f w.), which depends on the feature representation (ar,,) and the task-specific weight wt-
Note that rather than following fully Bayesian approach, we leave some random variables to be estimated in frequentist way, e.g., f is a meta-parameter of the likelihood model shared by all tasks, for which use point estimate. As such, the posterior inference about these variables will be largely simplified. For the same reason, we derive the empirical Bayes (Robbins 1985 Kucukelbir & Blei 2014) by taking point estimate 1b. The marginal likelihood
3

---

Published as a conference paper at ICLR 2020
now reads
N Pv.s - II
Pr(dg ( (w))
(3)
We highlight the petaaparrmeters as subscripts of the corresponding distributions distinguish from random variables. Indeed, are not the first to formulate meta-learning as empirical Bayes. The overall model formulation is essentially the same as the ones considered by (Amit Meir (2018); Grant et al. (2018); Ravi & Beatson (2018). Our contribution lies in the variational inference for empirical Bayes.
2.2 AMORTIZED INFERENCE WITH TRANSDUCTION
As in standard probabilistic modeling, we derive an evidence lower bound (ELBO) on the log version of (3 by introducing a variational distribution qo, (w,) for each task with parameter 8,:
log PV. (D)
R wur (oggp(()//w)) DEL (20, (w; (wn))lp((w:))]
(4)
The variational inference amounts to maximizing the ELBO with respect to 8 - ON. which together with the maximum likelihood estimation of the meta-parameters, we have the following optimization problem:
N
min min v.S
Ba log (de/w::) + DxL (wr (ww))l)
(5)
P1.....N
N
However, the optimization in (5), as N increases, becomes more and more expensive in terms the memory footprint and the computational cost. We therefore wish to bypass this heavy optimization and to take advantage of the fact that individual
terms indeed share the same structure. To this end, instead of introducing N different variational distributions, we consider parameterized family of distributions the form of (ø(-), which defined implicitly by deep neural network Φ taking as input either de or di plus Tt. that is, lo(d)) or (o)d),a.) Note that we cannot use entire d,, since we not have access to y4 during meta-testing. This amortization technique was first introduced in the case of variational autoencoders (Kingma & Welling 2013 Rezende al.) 2014), and has been extended to Bayesian inference in the case of neural processes Garrnelo et al.) 2018).
Since d and disjoint, the inference scheme is inductive for variational posterior lo(d))- As an example, MAML (Finn et al. (017) takes (a(a)) as the Dirac delta distribution, where o(d) = 0K the K-th iterate of the stochastic gradient descent
8 = 0 + Ve log p(d)/w:)) with 8 = learnable initialization.
(6)
In this work, we consider a transductive inference scheme with variational posterior (o(d) The inference process is shown in Figure Tla in green arrows). Replacing each qo, (5) by (o)d) the optimization problem becomes
N
min min $
Bwirg log 1y + DKL (qood) (wo)ll
(7)
N
In nutshell, the meta-model to be optimized includes the feature network f. the hyper-parameter e from the empirical Bayes formulation and the amortization network from the variational inference.
UNROLLING EXACT INFERENCE WITH SYNTHETIC GRADIENTS
It is however non-trivial design a proper network architecture for o(d 22). since d' and a, are both sets. The strategy adopted by neural processes (Garnelo et al.) 2018) is to aggregate the information from all individual examples via averaging function. However, pointed out by Kim al.
4

---

Published as a conference paper at ICLR 2020
(2019), such strategy tends to underfit 2; because the aggregation does not necessarily attain the most relevant information for identifying the -ask-ipecifie parameter. Extensions, such as attentive neural process (Kim et al. 2019) and set transformer (Lee et al. 2019a), may alleviate this issue but come at a price of O(n²) time complexity. We instead design @f d' 34 to mimic the exact inference arg ming, DKI(go, (go,(w,) p (w]]d)) by parameterizing the optimization process with respect A..
More specifically, consider the gradient descent on Og with step size
0f+11 0: Ve, DKL (g0t (w) Pu.r(w) (u))((ldd)).
(8)
We would like to unroll this optimization dynamics up to the K-th step such that 0, = odd x1) while make sure that 0 is a good approximation to the optimum 07. which consists parameterizing
(a) the initialization og and (b) the gradient DKL (ge, (w)) Pp. (we d4)).
By doing so, 0.5 becomes a function of o ta and 2: we therefore realize (o(d),..) as 90%
For (a), we opt to either let 00 = to be global data-independent initialization as in MAML (Finn et al. 2017) or let 0º = A(d)) with a few supervisions from the support set, where À can be implemented by a permutation invariant network described in Gidaris & Komodakis (0188. In the second case, the features of the support set will be first averaged in terms of their labels and then scaled by a learnable vector of the same size.
For (b), the fundamental reason that we parameterize the gradient is because we do not have access yr during the meta-testing phase, although we are able to follow (8) in meta-training to obtain 40z (we)) xx Py (dg) (ey) Pu (wx). )- To make a consistent parameterization in both meta-training and meta-testing, thus do not touch Yr when constructing the variational posterior. Recall that the true gradient decomposes as
Ol, No.s) Div Sw, (0r,e) + Vo, DKI. (go, (lw)))))))
Vo, DKL. go. llpv.. Ee
iür, Owa
00,
under reparamcterization We we( c) with p(c). where all the terms can be computed ah.." Thus, we introduce deep neural network {(tr,) to synthesize it. The idea of synthetic gradients was originally proposed by Daderberg al. (2017) to parallelize the hack-propagation. Here, the purpose of E()r.)) is to update Og regardless of the groundtruth labels, which is slightly different from its original purpose. Besides, we do not introduce an additional loss
without Y/ except for 86,
between E((.,) and 86, since x( will be driven by the objective in (7). As an intermediate Bū.,;
computation, the synthetic gradient is not necessarily a good approximation to the true gradient.
To sum up, we have derived a particular implementation of ((d),z)) xx) by parameterizing the exact inference update, namely (8), without using the labels of the query set, where the meta-model ø includes initialization network \ and a synthetic gradient network ç, such that ((x+)) 0 the K-th iterate of the following update:
8 0¹
Ec
1 n
E(Or.)
Tin, Ow.((t,e) owr
Vo, DKL [00 [Pp)]
(10)
06,
The overall algorithm is depicted in Algorithm I We also make side-by-side comparison with MAML shown in Figure precise to think of it as a part of the computation graph created in the prwward-roppgattoo.. The computation graph of the amortized inference is shown Figure 2
Rather than viewing 10 as an optimization process, it may be more
As an extension, if we were deciding to estimate the feature network f in Bayesian manner, we would have to compute higher-order gradients as the case MAMI.. This is inpractical from a computational point of view and needs technical simplifications (Nichol et al.] (018). By introducing a series of synthetic gradient networks in a way similar to Jaderberg et al. (2017). the computation will be decoupled into computations within each layer, and thus becomes more feasible. We see this as a potential advantage of our method and leave this to our future workf"
R0 is also dependent of f. We deliberately remove this dependency to simplify the update of f. We do not insist on Bayesian estimation of the feature network hecause most Bayesian versions of CNNs underperform their deterministic counterparts.
5

---

Published as a conference paper at ICLR 2020
Classifier forward
Synthetic gradient
grad
(k).detach)
module
VoD
Classifier backward
dc
dÿ
init A
0
SGD
0
SGD
SGD
0
KL
(orlp)))
y/
X7
f(x))
Classifier forward
y
X,
f(x))
Loss
ELBO
Figure 2: The computation graph to compute the negative ELBO, where the input and output of the synthetic gradient module are highlighted in red. The detach() is used to stop the pack-propagation down to the feature network. Note that we do not include every computation for simplicity.
Algorithm 1 Variational inference with synthetic gradients for empirical Bayes
1: Input: the dataset D; the step size y;; the number of inner iterations K; pretrained f.
2: Initialize the meta-models vh, and @ = (A, E)
3: while not converged do
4: Sample a task t and the associated query set d, (plus optionally the support set d)).
5: Compute the initialization 0
0 A(d)).
6:
for k 1..... K do Compute 67 via (10) end for
7:
8:
9:
Compute wy = wt (0 {) with p(e).
10:
Update t <- 1 n V DKL (1og (2) )
11:
Update & + 17 V DKL (gp(ze) (ps pv))
12:
Optionally, update - + V ly wy).
end while
GENERALIZATION ANALYSIS OF EMPIRICAL BAYES VIA THE CONNECTION TO INFORMATION BOTTLENECK
The learning of empirical Bayes (EB) models follows the frequentist's approach, therefore, use frequentist's tool to analyze the model. In this section, we study the generalization ability of the empirical Bayes model through its connection to a variant of the information bottleneck principle Achille & Soatto 2007); Tishby et al. (2000).
Abstract form of empirical Bayes simpler joint distribution since
From
3
we see that the empirical Bayes model implies
log f (W1, W.N D) = log (delw:) + log (we),
(11)
which is equal to the log-density of N iid samples drawn from the joint distribution
p(w, d, t) = Pv. (w, d,t) py (d/w, t)p.. (w)p(t)))))))
(12)
up constant if introduce random variable represent the task and assume p(t) is an uniform distribution. We thus see that this joint distribution embodies the generative process of empirical Bayes. Correspondingly, there is another graphical model of the joint distribution characterizes the

---

Published as a conference paper at ICLR 2020
inference process of the empirical Bayes:
q(w, d, () = qo (e d, - (g(d))))))))()
(13)
where go (w]l, 1) is the abstract form of the variational posterior with amortization, includes both the inductive form and the transductive form. The coupling between p(w, d. t) and g(w, d, t) is due to p(t) = q(t)
only have access to tasks through sampling.
We are interested in the case that the number of tasks N + oc, such as the few-shot learning paradigm proposed by Vinyals et al. (2016), in which the objective of can be rewritten in an abstract form of
(t) (d((t) Bo(mldd)) [
log p(d w.t)]
DxL (q(w)d, ())l
(14)
In fact, optimizing this objective is the same as optimizing from stochastic gradient descent point of view.
The learning of empirical Bayes with amortized variational inference can be understood as a varia- tional EM in the sense that the E-step amounts to aligning q(w) d, t) with (( [d, t) while the M-step amounts to adjusting the likelihood p(d[w, t) and the prior p(w).
Connection to information bottleneck The following theorem shows the connection between (14) and the information bottleneck principle.
Theorem 1. Given distributions q (w d, t). (dl)t), q(t), p(w) and p(d w. t), we have
14
(w: dlt) + H2 (dlww.t))
(19)
where Ig(w;d10 = Dxr(g(us (a(ue, ([()] (())))(()())((())))) is the conditional mutual information and Hg(wld, == Bg(w,d,e)-- log q( (wd,, t)] is the conditional entropy. The equality holds when
Vt: (x)((( = 0 and Dxl(q(d) w. t)]lp(d w. (t) =0
In fact, the lower bound on 14) is an extention of the information bottleneck principle (Achille & Soatto 2017) under the multi-task setting, which, together with the synthetic gradient based variational posterior, inspire the name synthetic information bottleneck of our method. The tightness of the lower bound depends on both the parameterizations of py (dl ur, () and (w) well as the optimization of (14). It thus can be understood as how well we can align the inference process with the generative process. From an inference process point of view, for a given g d. t). the optimal likelihood and prior have been determined. In theory, we only need to find the optimal q(wld, t) using the information bottleneck in However, in practice, minimizing
is more straightforward.
Generalization of empirical Bayes meta-learning The connection to information bottleneck suggests that we can eliminate p(d[w, 1) and p(w) from the generalization analysis of empirical Bayes meta-learning and define the generalization error by g(w, d, l) only. To this end, we first identify the empirical risk for a single task with respect to particular weights w and dataset d as
n 1 1 Σ ((s.)
Le d) =
(15)
The true risk for task with respect to w is then the expected empirical risk Ed-g(d))) (ww d). Now, we define the generalization error with respect to g(w, d, f) as the average of the difference between the true risk and the empirical risk over all possible t, d, w:
gen(g) (g(t) d.1) Ea
(e) d) E0n--Eggthaa Eattrgaansseelaan
(16)
where q(wlf) is the aggregated posterior of task
Next, we extend the result from Xu & Raginsky 2017) and derive a data-dependent upper bound for gen(g) using mutual information.
Theorem 2. Denote by (x, v). If (f w). ys) = le(w, 22) is o-subgaussian under ((2()) then Le(w, d) of /n-sugauussian under ql ((d()) due to the iid assumption, and
2202 gen(g)) S
n [ dlt).
(30)
7

---

Published as a conference paper at ICLR 2020
Plugging this back to Theorem obtain different interpretation for the empirical Bayes ELBO. Corollary 1. If & is chosen to be the negalive log- bog-likelihodd minimizing the population objective of empirical Bayes meta-leat neeanng amounts to minimizing both the expected generalization error and the expected empirical risk:
14
20 gen (o(0t)((d(((((ldd..) (ww,, (w,d)
(17)
The Corollary D suggests that 44) amounts to minimizing regularized empirical risk minimization.
In general, there tradeoff between the generalization error and the empirical risk controlled by the coefficient 20 where
|d] is the cardinality of d. If is small, then are in the overfitting regime. This is the case of the inductive inference with variational posterior g(w d'' t). where the support set d' is fairly small by the definition of few-shot learning. On the other hand, if we were following the transductive setting, we expect achieve a small generalization error since the implemented variational posterior is a better approximation to q(wld, t). However, keeping increasing n will potentially over-regularize the model and thus yield negative effect. An empirical study on varying can be found in Table 5 in Appendix
EXPERIMENTS
In this section, we first validate our method on few-shot learning, and then on zero-shot learning (no support set and no class description are available). Note that many meta-lcarning methods cannot do zero-shot learning since they rely on the support set.
5.1 FEW-SHOT CLASSIFICATION
We compare SIB with state-of-the-art methods on few-shot classification problems. Our code is available at
5.1.1
SETUP
Datasets We choose standard benchmarks of few-shot classification for this experiment. Each benchmark is composed disjoint training, validation and testing classes. MinilmageNett pro- posed by Vinyall et al. (2016), which contains 100 classes, split into 64 training classes, 16 validation classes and 20 testing classes; each image is of size 84 84, CIFAR-FS is proposed by Bertinetto et al.. (2018), which is created by dividing the original CIFAR-100 into 64 training classes, 16 validation classes and 20 testing classes; each image is size 32x32.
Evaluation metrics In few-shot classification, a task (aka episode) consists query set d, and support set d, When we say the task is k-way-n' -shht -shot mean that de is formed by first sampling classes from a pool of classes; then, for each sampled class, n'' examples are drawn and a new label taken from {O...., k 1} is assigned to these examples. By default, each query set contains 15k examples. The goal of this problem is to predict the labels of the query set, which are provided as ground truth during training. The evaluation is the average classification accuracy over tasks.
Network architectures Following Gidaris & Komodakis 2018); Qiao et al. (2018) Gidaris et al. (2019), we implement f by a 4-layer convolutional network (Conv-4-64 or Conv-4- or a WideResNet (WRN-28-10) (Zagoruyko & Komodakis 2016). We pretrain the feature network f(.) on the 64 training classes for a stardard 64-way classification. We reuse the feature averaging network proposed by Gidaris & Komodakis (0018) as initialization network X(-), which basically averages the feature vectors of all data points from the same class and then scales each feature dimension differently by learned coefficient. For the synthetic gradient network E(-), ), implement a three-layer MLP with hidden-layer size 8k. Finally, for the predictor Dty (-, w:), we adopt the sissie-similarity based classifier advocated by Chen et al. (2019) and Gidaris & Komodakis (2018).
$Conv-4-66 4-64 consists of 4 convolutional blocks each implemented with a 3 x 3 convolutional layer followed by BatchNorm ReI.U + 2 x 2 max-pooling units. All blocks of Cony-4-64 have 64 feature channels. Conv-4-128 has 64 feature channels in the first two blocks and 128 feature channels in the last two blocks.
8

---

Published as a conference paper at ICLR 2020
MinilmageNet, 5-way
CIFAR-FS, 5-way
Method
Backbone
1-shot
5-shot
1-shot
5-shot
Matching Net Vinyals
2016
Conv-4-64
44.2%
57%
MAML (Finn ct al. 2017
Conv-4-64
48. 18.7=188%
63.1+:09%%
58.9=11 1.9%
1..5=
Prototypical Ner (Snel cl
2017
Conv-4-64
49.4=0.8%
68.2+t0.7%
55.5=0.7%
72.0=0.6%
Relation Net (Sung al.
2018
Conv-4-64
50.410.885
65.31:0.%%
55.0 11:2:10055 1.0%
69.31.0.8%
GNN (Satorras & Bruna
201
Conv-4-64
50.3%
66.4%
61.9%
75.3%
(Bcrtnetto et al 2018
Conv-4-64
49.5=0022%
65.4+...%
62.3=0.2%%
77.42022%
TPN (Luu et al 2018
Comv-4-64
55.5%
69.9%
Gidaris et al. ) 2019
Conv-4-64
54.8+0.4%
0..9+
63.5++03%%
79.8++0.2%
SIB R
=k=0 (Pre-trained (eatur))
Conv-4-64
5..0%
67.0%00.4%
0.52=
0..4%
SIB 1=lc-3, K=3
Conv-4-64
58.0=0.6%
70.7+0.4%
68.7+40.6%
77.1=0.4%
SIB 1=le-3, K=0
Conv-4-128
53.62 + 0.79%
71.48 + 0.64%
SIB 1----3 K=1
Conv-4-128
58.74 = 0.89%
74.12 + 0.63%
SIB 1e-3,, K=3
Conv-4-128
62.59
1.02%
75.43 0.67%
SIB 1--3-, K=5
Conv-4-128
63.26
1.07% %
75.73 +: 0.71 %
TADAM (Oreshkin et al.
2018
ResNet-12
58.5+
76.7++0.3%
SNAIL (antoro ct al. 2017
ResNct-12
55.7=100%
68.9 68.9900.995 9%%
MetaOpINet-RK
Lee
el al
ResNet-12
61.41066%
77.910.5%
72.610.7%
84.31..5%
2019h
MctaOptNet-SWW
ResNet-12
62.62006%
78.6%
72.0-0.7%
84.2+0.5%
CTM [Li 2019
ResNet-18
14..%%
0..5.10.1%
Qiao
WRN-28-10
59.6=0.4%%
73.7+t0.2%
L60 Rusu
2019
WRN-28-10
61811.1%
77.6100.1%
Gidaris et 219
WRN-28- 10
62.9+-05%%
79.9+-0.33%
73.6++0.9%
8..1+
SIB R (Pre-frateed feature)
WRN-28-10
60.6=0.4%
77.5%033%
70.0=0.5%
83.5==0.5%
SIB le-3, K=1
WRN-28-10
67.310.5%
78.8.10.5%
76.8_1.55%
84.91104%
STB le-,, K=3
WRN-28-10
69.6+0.66 %
78.9++0.4%
78.4-0.6%
85.3++0..4%
SIB 1e-3, K=5
WRN-28-10
70.0=0.6%
79.2:20.4%
80.0:10.6%
85.3-2044%
Table 2: Average classification accuracies (with 95% confidence intervals) the test-set of Mini- ImageNet and CIFAR-FS. For evaluation, we sample 2000 and 5000 episodes respectively for MinilmageNet and CIFAR-FS and test three different architectures the feature extractor: Conv-4- 64, Conv-4- 128 and WRN-28- 10, We train SIB with learning rate 0.001 and try different numbers of synthetic gradient steps K.
We run SGD with batch size 8 for 40000 steps, where the learning rate is fixed to 10 10-3, During training, we freeze the feature network. To select the best hyper-paan parameters, we sample 1000 tasks from the validation classes and reuse them at each training epoch.
Training details
5.1.2 COMPARISON TO STATE-OF-THE-ART META-LEARNING METHODS
In Table we show comparison between the state-of-the-ard approaches and several variants of method (varying K or /(.)). Apart from SIB, TPN (Liu et al. 2018) and CTM (Li et al. 2019) are also transductive methods.
First of all, comparing SIB (K 3) to SIB (K 0), we observe a clear improvement, which suggests that, by taking few synthetic gradient steps, we do obtain a better variational posterior as promised. For 1-shot learning, SIB (when K = 3 or K = 5) significantly outperforms previous methods on both MinilmageNet and CIFAR For 5-shot learning. the results are also comparable. It should be noted that the performance boost is consistently observed with different feature networks, which suggests that SIB is a general method for few-shot learning.
However, we also observe potential limitation of SIB: when the support set is relatively large, e.g.. 5-shot, with good feature network like WRN-28-100 10, the initialization 0 may already be close to some local minimum, making the updates later less important.
For 5-shot learning, SIB is sligtly worse than CTM (Li et al. 2019) and/or Gidaris et al. 2019). CMT (Li al. 2019) can be seen similarity between a query example and the support set while making use of intra- and inter-class relationships. Gidaris et al. (2019) uses in addition the self-supervision richer and more transferable feature model. Both ideas are complementary to SIB. We leave these extensions to future work.
alternative way to incorporate transduction it measures the
auxilary loss to learn

---

Published as a conference paper at ICLR 2020
5.2
ZERO-SHOP REGRESSION: SPINNING LINES
SIB vnliuntion
SIB dysssiss
Predistin comparacn
Sil jat 6P
GT
GT tock
(kwd))
podkiticn
prlitiion
Dandd (o1b
GT
SIB poeditions
0.3
10
15
10
0.4
0.2
010
0.3
-15
+2.
-10
Figure 3: Left: the mean mquanequare emrors Dieas Ex (9ok (w)))l)((,,)d)), DKI (p (w))[l((w))))))))))))))))))))( and the estimate of I (w:; d) = DxL (wy) Ip (wx)). Middle: the predicted y's by = 0 for O...... 4. Right: the predictions SIB.
Since our variational posterior relies only on Te. SIB is also applicable zero-shot problems (i.c., no support set available). We first look at a toy multi-task problem, where [ww; de) is tractable.
Denote by Duain = {de H the train set, which consists of datasets of size n: d {(x;, 1: We construct dataset d by firstly sampling iid Gaussian random variables as inputs: 2; ~ N(,, o"). Then, we generate the weight for each dataset by calculating the mean of the inputs and shifting with a Gaussian random variable ew:
Σ, + Ew, for N ((po: o.) The output for æs is V: = w Zz- We decide ahead of time the hyperparameters Jl, a, Phee Tw for generating æ; and yi- Recall that weighted sum of iid Gaussian random variables is still a Gaussian random variable. Specifically, w = E; q,,, and H; ~ NN ), then w ~ N(E, c,p,, >, C.a)). Therefore, we have p(w) = NN ww: n ood +o On the other hand, if we are given dataset of size n, the only uncertainty about comes from Ew, that is, should consider x; as constant given d. Therefore, the posterior p( ( (d) ( > D; + Mms o We use a simple implementation for SIB: The variational posterior is realized by
g0. (w) (O ow), ofti = - 100000 10-3 Σ x 20E(0)) and 0₂ X IR;
(18)
4 is a mean squared error, implies that p(yl:, w) = N(wx, 1): Po (w) a Gaussian distribution with parameters I The synthetic gradient network & is a threc-layer MLP with hidden size 8.
In the experiment, we sample 240 tasks respectively for both Dmain and Drest. We learn SIB and BNN on Dinin for 150 epochs using the ADAM optimizer (Kingma Ba 2014). with learning rate 10 and batch size 8. Other hyperparameters are specified as follows: = 32, K 3, 0, 1. mw = 1, 0.1. The results are shown in Figure 3 On the left, both DKL (9 (w]]lp(w, de)) and (p& (w) (lw()) are close to zero indicating the success of the learning. More interestingly, in the middle, we see that 6f.6%... B!.................. of evolves gradually towards the ground truth, which suggests that the synthetic gradient network is able to identify the descent direction after meta-learning.
CONCLUSION
We have presented an empirical Bayesian framework for meta-learning. To enable an efficient varia- tional inference, we followed the amortized inference paradigm, and proposed to use transductive scheme for constructing the variational posterior. To implement the transductive inference, we make use of two neural networks: synthetic gradient network and an initialization network, which together enables a synthetic gradient descent on the unlabeled data to generate the parameters of the amortized variational posterior dynamically. We have studied the theoretical properties of the proposed framework and shown that yields performance boost on MinilmageNet and CIFAR-FS for few-shot classification.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentangling in deep represen- tations. arXiv preprint 1.Xi::17006013500 2017.
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In International Conference on Machine Learning. pp. 205-214, 2018.
Y Bengio, S Bengio, and J Cloutier. Learning synaptic learning rule. In IJCNN-91-Senne International Joint Conference on Neural Networks, volume 2, pp. 969-vol. IEEE, 1991.
James O. Berger. Statistical Decision Theory and Bayesian Analysis. Springer-Verke Verlag New York, 1985.
Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. ArXiv, abs/1805 111118050008000 00..08133 08136, 2018.
Fabio Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2229,2238,222 2019.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint arXiv: 04232, 2019.
Jacob Devlin, Ming- Wiig-Wei&ee Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04860 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 1335-1135. JMLR. org, 2017.
Sebastian Flennerhag, Pablo G Moreno, Neil D Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. International Conference on Learning Representations (ICLR), 2019.
Marta Gamelo,, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv: 087.02 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings the the IEEE Conference Computer Vision and Pattern Recognition, pp. 4367 2018.
4167-43777 4375,
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07728, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthiuu Cord. Boosting few-shot visual learning with self-supervision. arXiv preprint arXiv: (xXiv:/906.00 /906.05 16, 2019.
Irving John Good. Some history of the hierarchical bayesian methodology. Trabajos de estadística de investigación operativa, 31()):489 1980.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learnig as hierarchical bayes. arXiv preprint arXiv: 11001101199011 08930, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
Max Jaderberg. Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1627-1635.
JMLR, 2017.
11

---

Published as a conference paper at ICLR 2020
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05661, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 442.0690, 2014.
Diederik P Kingma and Max Welling. 13Xiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alp Kucukelbir and David M Blei. Population empirical bayes. arXiv preprint 1r1.0214 2014.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Tch. Set trans- former: A framework for attention- based permutation inntuuttatiioniviiianttt neural networks. In International Conference on Machine Learning, pp. 3744-3753 2019a.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019b.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5542-5550, 201
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding Task- Relevant Features for Few-Shot Learning by Category Traversal. In CVPR, 2019.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot 2017b.
learning. arXiv preprint
Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint 100. 2018
Tom Minka. Discriminative models, not discriminative training. Technical report, Technical Report MSR-TR-2205-114 Microsoft Research, 2005.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 1=Xxv:8803.0299)) 2018
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems (NIPS), 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7229-7238, 2018.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. International Conference Learning Representation, 2018
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. International Conference on Learning Representation, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint arXiv:1401.403: 2014.
Herbert Robbins. An empirical bayes approach to statistics. In Herbert Robbins Selected Papers, pp. 41-47. Springer, 1985.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learming with latent embedding optimization. In International Confer- ence on Learning Representations, 2019. URL
1JackTAckT
12

---

Published as a conference paper at ICLR 2020
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Timothy P. Lillicrap. simple neural network module for relational reasoning. In NIPS, 2017.
Victor Garcia Satorras and Joan Bruna. Few-shot learning with graph neural networks. ArXiv, 0t5/17111040445 2017.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. on learning now learn: The mela-meea--meta..-toon -ook. Phd thesis, Technische Universitat Munchen, Germany, 1987. URL html
/tttp://ww. idsia. ch/-jjuergen/cipplcccc
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Flood Sung, Yongxin Yang. Zhang, Tao Xiang, Philip HS Torr, and Timothy Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Sebastian Thrun and Lorien Pratt. Learning to learn. Kluwer Academic Publishers, 1998.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems 29, pp. 3630-3638. Curran Associates, Inc., 2016.
Aolin Xu. Information-theorer limitations of distributed information processing. PhD thesis, University of Illinois at trrban-Champpaie 2016.
Aolin Xu and Maxim Raginsky. Information-theorretie analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems, pp. 2524-2533 2017.
Jiaolong Xu, Liang Xiao, and Antonio M López. Self-supervised domain adaptation for computer vision tasks. IEEE Access, 7:156694-156706, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.077 2016.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference Machine learning (ICML-03), pp. 912-919, 2003.
APPENDIX
PROOFS
Theorem 1. Given distributions q(wld, t), ((d)t), q(t), p(w) and w( t), we have
14
T (w: d(t) Hg (d)
(19)
where (dwwdlt) Dxe (g(w,
is the conditional mutual information and H (ulld, Beg(w,,,,) log q(wld, (d, ()] is the conditional entropy. The equality holds when
VL: Dea and Dxa(g(d) w. ()]lp(d w. 1)) 0.
13

---

Published as a conference paper at ICLR 2020
Proof. Denote by ((wlt)) = (wdd the aggregated posterior of task t. decomposed as
14 can be
E (e(t) Bg Be(wddda)]
log p(d/w, t)] + DxL (a(wld, ((ll()
(20)
log q(wld d, (w)())
= (ddtt) ((wri
(21)
p(d/ ()p(w)g(w(())
log ((wld, t)
= Bo(t)) (o
+ (e Butolae))))) log log p(d
(ww(t)
log g(vllt)
+ (e (2(
(22)
p(w)
Lww d|t) + H (o, + (kL(
(23)
2 [w; d|() + (d|wu,))
(24)
The inequality because (g(((((1l) (l)(()) 2 for all I's. Besides, used the notation Hg.p. which is the conditional cross entropy. Recall that Dx (g(dlw, ()lp(d/ f)) -d
Hy.(ddlu t) 2 0. We attain the lower bound as desired if this inequality is applied replace H.pp d/w, t) by Hg [d],w,tt t).
The following lemma and theorem show the connection between Lu(;; d[t] and the generalization error. We first extend Xu (2016) Lemma 4.2).
Lemma 1. If. for all t, Y) is o-subgaussain under Px @ Pr, then
Ep(ry (rxx,)))
(FPxII)p htt.://m........mEmr//m///////11 & (2)0=((X;YTT)
(25)
Proof.
The proof is adapted from the proof Xu (2016) Lemma 4.2).
LHS S B B
(26)
s <EPIT B
(27)
S 20 (PP
(28)
(2x2
(29)
The second inequality was due to the Donsker-Varadhan variational representation of KL divergence and the definition of subgaussain random variable.
Theorem 2. Denote by v). If (f w) ys) E,(w,22) is o-subgaussian under ((w)t)q(=[tt, then L, (w, d) is af / Vn-subgaussian under q ((w((t)((dtt) due to the iid assumption, and
20 gen(a)) sv (w: dlt).
(30)
Proof. First, if (f(((((()) w). y) is -ubbaaussia under ((w(())(=11)) by definition,
[g(w)))(((1)) exp (Ae, (w, 2)) < exp((A 2)) exp(X20212 (xx(((
(31)
It is straightforward to show (ww,, d) is a / subgausssia since
E E (o(dle) exp exp(AL.(w, d)) = II
Ev
exp( A (e(w,z))
(32)
1=1
I exp 1=1
( Br,,
Lw 2=)+ + 2n²
X202
(33)
X²0²
= =exp (Em,,s((uu22)) expl 2n
(34)
X2
\2(0//11)
=exp XE (Xequilbatdi))Loauu exp
(35)
2
14

---

Published as a conference paper at ICLR 2020
Method
Art
Cartoon
Sketch
Photo
Average
JiGin (Carlucci et al. 2019 Rot (Xu al. 2019
84.9%
81.1%
79.1%
98.0%
85.7%
88.7%
86.4%
74.9%
98.0%
87.0%
SIB-Rol
85.7 75.7%
86.6%
80.3%
98.3%
87.7%
SIB-Rot K
88.9
89.0%
82.2%
98.3%
89.6%
Table Multi-souree domain adaptation results on PACS with ResNet- features. Three domains are used as the source domains keeping the fourth one as target.
By Lemma have
gen(g)) = (et)) Ee (2tt)))))) (ws,
(angsdd--Bywlaeen (etw))a(dd) Le(
(36)
2 [(w:
S n
(37)
desired.
ZERO-SHOT CLASSIFICATION UNSUPERVISED MULTI-SOURCE DOMAIN ADAPTATION
more interesting zero-shot multi-task problem is unsupervised domain adaptation. We consider the case where there exists multiple source domains and a unlabeled target domain. In this case, treat each minibatch as task. This makes sense because the difference in statistics between two minibatches are much larger than in the traditional supervised learning. The experimental setup is similar to few-shot classification described in Section 5.1 except that we do not have a support set and the class labels between two tasks are the same. Hence, it possible to explore the relationship between class labels and self-supervixed labels to implement the initialization network without resorting support set. We reuse the same model implementation for SIB as described in Section 5.1 The only difference is the initialization network. Denote by == {= the set of self-supervised labels of task t, the initialization network À is implemented follows:
0.- = - VoL. (2 (6t (f(xz), wo ()), f(xi))
(38)
where ii global initialization similar to the one used by MAML; Lg is the self-supervised loss, & is the set of predictions of the self-supervised labels. One may argue that 09 À would be simpler solution. However, it is insufficient since the gap between two domains can be very large. The initial solution yielded by (38) more dynamic in the sense that 0f is adapted taking into account the information from Zz.
In terms of experiments, we SIB on the PACS dataset (Li et 2017a), which has object categories and 4 domains (Photo, Art Paintings, Cartoon and Sketches), and compare with state- of-the-art algorithms for unsupervised domain adaptation. We follow the standard experimental setting (Carlucci et al. 2019). where the feature network is implemented by ResNet-18. We assign self superviedd label 2t,i image by rotating the image by predicted degree. This idea was originally proposed by Gidaris et al. (2018) for representation learning and adopted by Xu et al. (2019) for domain adaptation. The training is done by running ADAM for 60 epochs with learning rate 10 4 We take each domain in turns as the target domain. The results shown in Table 3 can be seen that SIB-Rot (K = 3) improves upon the baseline SIB-Rot (K 0) for zero-shot classification, which also outperforms tffe-offthe-ane methods when the baseline is comparable.
IMPORTANCE OF SYNTHETIC GRADIENTS
To further verify the effectiveness of the synthetic gradient descent, we implement an inductive version of SIB inspired by MAML, where the initialization 80 is generated exactly the same way as SIB using A(d)), ). but it then follows the iterations in 6 as in MAML rather than follows the iterations in (1O) as in standard SIB.
'A is overloaded he both the network and its parameters.
15

---

Published as a conference paper at ICLR 2020
We conduct experiment CIFAR-FS using Conv-4-64 feature network. The results are shown in Table can be seen that there is no improvement over SIB (K' = 0) suggesting that the inductive approach is insufficient.
inductive SIB Training on 1-shot
SIB
Training on 1-shot Testing on
Training on 5-shot Testing on 5-shot
Testing on 5-shot
K
1-shot
I-shot 5-shot
1-shot
0
59.7110.5%
75.5=
59.2.10.5%
75.4410.4%
59.2-10.5%
75.4.1:0.4%
1
le-1
59.8-10.5%
71.2=0.4%
65.3110.6%
75.8:10.4%
64.5+00.6%
77.34.0.%%
3
le-1
59.6++0.5%
75.9=0.4%
65.0:10.6%
75.0:10.4%
64.0+10.6%
77.0::0.4%
59.9+0.5%
74.9-0.4%
66.0++0.6%
76.3+t0.4%
64.0++0.5%
76.8++0.4%
1e-2
39.77-0.5%
73.3-50.4%
67.8+006%
74.37-0.4%%
63.6-F006%
77.5440.7%
3
le-2
5..5+
75.8-0.4%
68.6++0.6%
77.4+0.4%
67.8+00.%
78.5+004%
5
le-2
59.8+
75.7=0.4%
67.4+0.6%
72.6+00.%
67.7+0.7%%
77.7+0.4%
1e-3
59.510.5%
75.62004%
66.210.6%
75.71.0.4%
64.6.E0.6%
78.120.4%
3
le-3
59.9+0.5%
75.9-0.4%
68.7+0.6%
77.1+0.4%
66.8+0.6%
78.4+044%
5
le-3
59.4+0.5%
75.7=0.4%
69.1+-0.%%
76.7+004%
66.7+006%
78.5+0.4%
1e4
58.810.5%
75.3_0.4%
59.010.5%
75.7100.%
59.3.005%
75.7110.%
le-4
59.4+0.5%
75.9=0.4%
58.9+
75.6+0.4%
59.3+0.5%
75.9+t0.4%
5
lc-4
59.3+
75.3=0.4%
60.1%005%
76.0+t0.4%
60.5+t0.5%
76.4+0.4%
Table 4: Average 5-way classification accuracies (with 95% confidence intervals) with Conv-4-64 on the test set of CIFAR-FS. For each test, we sample 5000 episodes containing 5 categories (5-way) and 15 queries in each category. We report the results with using different learning rate 7 well as different number of updates K. Note that K is the performance only using the pre-trained feature.
D VARYING THE SIZE OF THE QUERY SET
We notice that changing the size of de (i.e., n) during training does make a difference on testing. The results are shown in Table 5
5-way, 5-shot
5-way, 1-shot
Validation
Test
Validation
Test
77.97 0.34%
75.91
0.66%
63.60 0.52%
61.32
1.02%
78.14 0.35%
76.01
0.66%
64.67 + 0.55%
62,50 + 1.02%
10
78.30 + 0.35%
76.22 + 0.66%
65.34 + 0.56%
63.22 + 1.04 %
15
77.53 + 0.35%
75.43
0.67%
65.14 + 0.55%
62.59 + 1.02%
30
76.21 0.35%
74.04 0.67%
63.37 + 0.53%
60.96 + 0.98%
45
75.65
0.36%
73.27 0.66%
62.08 0.51%
$9.59 0.93%
Table 5: Average classification accuracies on the validation set and the test set of Mini-ImageNet with backbone Conv-4-128. We modify the number of query images, i.e., n, for each episode study the effect on generalization.
16


Reviews:
[{"review_id": "SJedK9owqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the issue of meta-learning in a transductive learning setting. That is, it aims to learn a model from multiple tasks and make it generalise to a new task in order to solve it efficiently. In the transductive setting, the query set (i.e., containing the unlabeled test data) of the new task is taken into account when learning the model. \n\nThis paper takes the empirical Bayes approach to meta-learning. In order to utilise the test data that do not access to groundtruth labels, it proposes to use synthetic gradient to implement the tranductive learning. A multi-layer perceptron network is used to systhesize the gradient. Theoretical analysis is conducted to demonstrate the generalization capability of the proposed model and reveal its connection to the information bottleneck principle in the literature of neural networks. \n\nOverall, this is a well organised and nicely presented work. The idea on how to utilise the unlabeled test data to realise tranductive learning is novel; the analysis is thorough; and experimental study is provided to show the effectiveness of the proposed method. Meanwhile, this work can address the following issues:\n\n1. The first paragraph on page 5, which describes the key step of syntheising gradient, can be made clearer; \n2. In the experimental study, Table 1 compares various methods with the proposed one. It will be helpful to clearly indicate for each method in comparison whether/how it also utilises the query set. This will give more context in interpreting the comparison results;\n3. The advantage of the proposed method seems to diminish quickly from 1-shot to 5-shot settings. Does this mean in the case of 5 (or more)-shot setting, considering unlabeled test data with the proposed method could even adversely affect the meta-learning performance? Please comment. \n4. Since the proposed method works in a tranductive manner, it is presumed that the whole model needs to be retrained/updated once a new set of query data (e.g., for the same task or another new task) is given? In other words, how does the trained model generalise to unseen unlabeled test data? Please provide some discussion on this issue. \n5. Finally, how is the computational complexity of training the proposed EB model? \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1gFwdapYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors argue for the importance of transduction in few-shot learning approaches, and augment the empirical Bayes objective established in previous work (estimating the hyperpior $\\psi$ in a frequentist fashion) so as to take advantage of the unlabelled test data.\nSince the label is, by definition, unknown, a synthetic gradient is instead learned to parameterize the gradient of the variational posterior and tailor it for the transductive setting. The authors provide an analysis of the generalization ability of EB and term their method _synthetic information bottleneck_ (SIB) owing to parallels between its objective of that of the information bottleneck. SIB is tested on two standard few-shot image benchmarks in CIFAR-FS and MiniImageNet, exhibiting impressive performance and outperforming, in some cases by some margin, the baseline methods, in the 1- and 5-shot settings alike, in addition to a synthetic dataset.\n\nThe paper is technically sound and, for the most part, well-written, with the authors' motivations and explanation of the method conceived quite straightforwardly. The basic premise of using an estimated gradient to fashion an inductive few-shot learning algorithm into a transductive one is a natural and evidently potent one. The paper does, however, at times feel to be disjointed and, to an extent, lacking in focus. The respective advantages of EB and the repurposing of synthetic gradients to enable the transductive approach are clear to me, yet while they might indeed be obviously complementary, what is not obvious is the necessity of the pairing: it seems there is nothing prohibiting the substitution of the gradient for a learned surrogated just as well under a deterministic meta-initialization framework. As such, despite sporting impressive results on the image datasets, I am not convinced about how truly novel the method is when viewed as a whole. \n\nOn a similar note, while the theoretical analysis provided in section 4 was not unappreciated, and indeed it was interesting to see such a connection between EB with information theory rigorously made, it does feel a little out of place within the main text, especially since it is not specific to the transductive setting considered, nor even to the meta-learning setting more broadly. Rather, more experiments, per Appendix C, highlighting the importance of transduction and therein the synthetic gradients and its formulation would be welcome. Indeed, it is stated that an additional loss for training the synthetic gradient network to mimic the true gradient is unnecessary; while I agree with this conclusion, I likewise do not think it would hurt to explore use of the more explicit formulation.\n\nConsidering the authors argue specifically for the importance of transduction in the zero-shot learning regime, I think it would be reasonable to expect experiments substantiating this, and the strength of their method in this regard, on non-synthetic datasets. As far as the toy problem is concerned, I am slightly confused as to the choice of baseline, both in the regard to its training procedure and as to why this was deemed more suitable than one purposed for few-shot learning, so that we might go beyond simple verification to getting some initial sense for the performance of SIB. Moreover, it is not clear from the description as to how $\\lambda$ is implemented here. As it stands, Section 5, for me, offers little in the way of valuable insights. The experiments section on the whole, results aside, feels somewhat rushed; the synthetic gradients being a potential limiting factor for instance feels \"tacked on\" and seems to warrant more than just a passing comment.\n\nMinor errors\n- Page 7: the \"to\" in \"let $p_\\psi(w)$ to be a Gaussian\" is extraneous\n- Page 8: \"split\" not \"splitted\".\n- Further down on the same page, \"scale\" in \"scale each feature dimension\" should be singular and Drop\" is misspelled as \"dropp\".\n- Page 9: \"We report results **with using** learning rate...\"\n- _Equation 17_ includes the indicator function $k \\neq i$ but $i$ is not defined within the context.\n\nEDIT: changed score", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgSFawkqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for transductive few-shot learning. The method is derived by taking a Bayesian perspective and recasting meta-learning as amortized variational inference, showing that results in a transductive scheme, and then using maml-style approximation of the inference (i.e., based on truncated stochastic gradient). While the idea of the paper seems intuitive, I find the writing quite confusing throughout (see my comments below) and I believe it must be improved before publishing the paper. Regarding the empirical evaluation, the results on standard benchmarks (miniImageNet and CIFAR-FS) seem reasonably strong; however, I would not call it \"significantly outperform previous state-of-the-art\" (as the authors claim in the abstract), since really all the top methods are in the same ballpark (the provided 95% CI overlap).\n\n\nComments:\n\n1. In Eq. 2, if the task-specific losses are arbitrary, the whole construction is no longer a log-likelihood but rather just a loss. The authors also denote the distribution over the meta-training datasets as p_\\psi(d_t), where d_t includes both inputs and targets. However, the concrete instantiations of the framework use discriminative models. Adjusting and clarifying the notation would improve the paper.\n\n2. The way KL divergence is used in Eq. 5 is misleading since the arguments are two distributions over different sets of random variables. I would recommend keeping expected log conditional probability as a separate term (which is common in the literature).\n\n3. Relatedly, going from ELBO to amortized VI (Eqs. 4-6) is a standard widely used VAE trick, so the derivation itself is not that informative. On the other, it would be great to include the inductive inference scheme mentioned right before Eq. 7 and compare it side-by-side with the standard amortized VI (Eq. 6). The way that part is presented now leaves the reader to derive all the details on their own.\n\n4. Sec. 3, paragraph 1: While the original neural processes tend to underfit the data as pointed by the authors, more recent versions of the model such as attentive neural processes might work well, and perhaps worth mentioning.\n\n5. Difference between Eq. 7 and 8 -- I believe I am misunderstanding this, but the updates look identical to me up to KL between q_\\theta and a prior p_\\psi. How exactly does \\phi(x_t) parametrize the optimization process? I don't see how it enters into the equations. Generally, I feel deriving the method through a Bayesian perspective is quite confusing (as it is presented now) and way less clear than what is illustrated in Figure 1c.\n\n6. Re: theoretical analysis -- it seems like the more than half a page spent on defining what generalization error is in the given setup (where all the definitions are quite standard), but then the discussion of the result, discussion of specific cases, connection to the information bottleneck bounds are all compressed down to in 1-2 sentences. This makes the \"analysis\" section really useless. Exemplifying the result of Thm. 1 and significantly elaborating the discussion would improve the paper.\n\n\nMinor:\n\n- The paragraph before Theorem 1: \"Proposition\" -> \"Theorem\"\n- [UPD] Figure 3: titles, labels, ticks are all too small to be readable.\n\n---------\n\nThanks to the authors for a detailed response. Most of my points have been addressed satisfactorily. I've updated my review accordingly.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers have assessed this paper and they have scored it 6/6/6 after rebuttal. Nonetheless, the reviewers have raised a number of criticisms and the authors are encouraged to resolve them for the camera-ready submission.
Decision: Accept (Poster)