A Hierarchical Bayesian Model for Few-Shot Meta Learning
Authors: Minyoung Kim
Timothy Hospedales
Venue: ICLR 2024 spotlight (2024)
Keywords: Bayesian models
Meta learning
Few-shot learning
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: A novel hierarchical Bayesian model for the few-shot meta learning problem, with the efficient one-time episodic learning algorithm that can scale up to modern architectures (eg, ViT).

Abstract: We propose a novel hierarchical Bayesian model for the few-shot meta learning problem. We consider episode-wise random variables to model episode-specific generative processes, where these local random variables are governed by a higher-level global random variable. The global variable captures information shared across episodes, while controlling how much the model needs to be adapted to new episodes in a principled Bayesian manner. Within our  framework, prediction on a novel episode/task can be seen as a Bayesian inference problem. For tractable training, we need to be able to relate each local episode-specific solution to the global higher-level parameters. We propose a Normal-Inverse-Wishart model, for which establishing this local-global relationship becomes feasible due to the approximate closed-form solutions for the local posterior distributions. The resulting algorithm is more attractive than the MAML in that it does not maintain a costly computational graph for the sequence of gradient descent steps in an episode. Our approach is also different from existing Bayesian meta learning methods in that rather than modeling a single random variable for all episodes, it leverages a hierarchical structure that exploits the local-global relationships desirable for principled Bayesian learning with many related tasks.

Published as a conference paper at ICLR 2024
A HIERARCHICAL BAYESIAN MODEL FOR FEW-SHOT META LEARNING
Minyoung Kim & Timothy M. Hospedales Samsung Al Center Cambridge, UK mi mikkm210gmm (@gma 1. com
University of Edinburgh, UK hossedaaessee. ac. uk
ABSTRACT
We propose novel model parametrisation and inference algorithm in a hierarchical Bayesian model for the few-shot meta learning problem. We consider episode wise random variables to model episode-specific generative processes, where these local random variables are governed by higher-level global random variable. The global variable captures information shared across episodes, while controlling how much the model needs be adapted to new episodes manner. Within our framework, prediction on novel episode/task can be seen as Bayesian inference problem. For tractable training, we need be able relate each local episode-specific solution to the global higher-level parameters. We propose a Normal-Inverse. Wishart model, for which establishing this local- global relationship becomes feasible due to the approximate closed-form solutions for the local posterior distributions. The resulting algorithm is more attractive than the MAML in that it does not maintain costly computational graph for the sequence of gradient descent steps in an episode. Our approach also different from existing Bayesian meta learning methods in that rather than modeling a single random variable for all episodes, it leverages a hierarchical structure that exploits the local- local-global relationships desirable for principled Bayesian learning with many related tasks.
principled Bayesian
INTRODUCTION
Few-shot learning (FSL) aims to emulate the human ability learn from few examples (Lake et al.) 2015). It has received substantial and growing interest (Wang et al. (020b) due to the need to alleviate the notoriously data intensive nature mainstream supervised deep learning. Approaches to FSL are all based on some kind of knowledge transfer from set of plentiful source recognition problems to the sparse data target problem of interest. Existing approaches are differentiated in terms of the assumptions they make about what is task agnostic knowledge that can be transferred from the source tasks, and what is task-specific knowledge that should be learned from the sparse target examples. For example, the seminal MAML (Finn et al. (017)) and ProtoNets (Snell et al. (017)) respectively assume that the initialisation for fine-tuning, or the feature extractor for metric-based recognition should be transferred from source categories.
One of the most principled and systematic ways to model such sets of related problems are hierarchical Bayesian models (HBMs) (Gelman et al., 2003). The HBM paradigm is widely used in statistics, but has seen relatively less use in deep learning, due to the technical difficulty of bringing hierarchical Bayesian modelling to bear on deep learning. HBMs provide powerful way to model set of related problems, by assuming that each problem has its own parameters (e.g, the neural networks that recognise cat vs dog, or car vs bike), but that those problems share a common prior (the prior over such neural networks). Data-efficient learning of the target tasks is then achieved by inferring the prior based on source tasks, and using it to enhance posterior learning over the target task parameters.
A Bayesian learning treatment of FSL would be appealing due to the overfitting resistance provided by Bayesian Occam's 's razor [MacKay 2003), as well as the ability to improve calibration of inference so that the model's confidence is reflective of its probability of correctness mission critical applications (Guo et al., 2017). However the limited attempts that have been made to exploit these tools in deep learning have either been incomplete treatments that only model single Bayesian layer within the neural network (Zhang et al.) 2021 Gordon et al., 2019), or else fail to scale up to modern neural architectures (Finn et al. 2018 Yoon al. 2018).
crucial property in

---

Published as a conference paper at ICLR 2024
In this paper we present the first complete hierarchical Bayesian learning algorithm for few-shot deep learning. Our algorithm efficiently learns a prior over neural networks during the meta-train phase, and efficiently learns posterior neural network during each meta-test episode. Importantly, our learning is architecture independent. It can scale up to the state-of-the-art backbones including ViTss (Dosovitskiy et al., (021), and works smoothly with any few-shot learning architecture spanning simple linear decoders (Finn et al. set-based decoders such as FEAT (Ye et al. 2020) and CNP(C et al. /018)/AP((K) (ii et al.
2017: Snell et al. (2017). to those based on sophisticated
2019). We show empirically that our HBM provides improved performance and calibration in all of these cases, as well as providing clear theoretical justification.
Our analysis also reveals novel links between seminal FSL methods such as ProtoNet (Snell et al. 2017). MAMI. (Finn et al., 2017), and Reptile (Nichol et al., 2018), all of which are different special cases of our framework despite their very different appearance. Interestingly, despite its close relatedness to ffMil-famlly algorithms, our Bayesian learner admits an efficient closed- form solution to the task-specific and task-agnostic updates that does not require maintaining the computational graph for reverse-mode backpropagation. This provides a novel solution to famous meta-learning scalability bottleneck. In summary, our contributions include: (i) The first complete hierarchical Bayesian treatment of the few-shot deep learning problem, and associated theoretical justification. (ii) An efficient algorithmic learning solution that can scale up to modern architectures, and plug into most existing neural FSL. meta-learners. (iii) Empirical results demonstrating improved accuracy and calibration performance on both classification and regression benchmarks.
2 PROBLEM SETUP
Let p(T) be the (unknown) task/episode distribution, where each task T~ p(T) is defined distribution (T( y) for data (r. y) where x is input and is target. For training, we have a large number of episodes. Tí, T TN - P(T) sampled i..d., but we only observe small number of labeled samples from each episode, denoted by D; {xt), w/) ~PT (z. y). where n. = [1,] is the number of samples in D,. The goal of the learner, after observing the training data D1..... DN from a large number of different tasks, is to build predictor p" (y|z) for novel unseen tasks T ~ p(T). We will often abuse the notation, e.g., i ~ T refers to the episode sampled, i.e., D; ~ PT; (2. y) where Te ~ p().. At the test time we are allowed to have some hints about the new test task T in the form of a few labeled examples from T", also known as the support set denoted by D ~ P (I. v).
In Bayesian perspective, the goal is to infer the pos- terior distribution with all training episodes and a test support set as evidence, i.e., p(y| D* D1.v). A ma- jor computational challenge, compared to conventional Bayesian learning, is that the training episodes (evi- dence) may not be stored/replayedrrvviiitedd
(a)
DOWEED
Figure 1: (a) IID episodes. (b) Individual episode. (c): FSL as Bayesian inference (grey nodes evidences, red target to infer). D = support set for test episode.
MAIN APPROACH
We introduce two types of latent random variables, o and (0.) Each episode i uses neural network weights 0 for modeling the data D, (i (u=1, 1,..., N). Specifically, D; is generated (input x given and only p(y]æ) modeled) by 0, as in the likelihood model in (.. The variable & can be viewed as a globally shared variable that responsible for linking the individual episode-wise parameters 0,. We assume conditionally independent and identical priors, p((0,,, ))))) = II, (().))). Thus the prior for the latent variables (Ø. J.V is formed in a hierarchical manner as follows. (For background on Bayesian modeling, refer to (ourphy/2022))
(Prior) p(0, 01.N) IT
(Likelihood) ((D) (I(..))ED, ptyp (z 02) (1)
where p(y xz 0.) a conventional neural network model. See the graphical model in Fig. Ta).
Given the training data { {Di the posterior is p(ó, AIN DINN) o p(o) I p(O, (e.)). and we approximate it with variational inference. That is, q(0, AIN: L) p(o, 0 DINN) (.N) where
( 01:;; L)= = g(0; Lo) II (0;; Lz),
where the variational parameters consists of Ln (parameters for ())) and [ 's (parameters of q: (0.))ss for episode i). Note that although 0, 's are independent across episodes under (2) they
2

---

Published as a conference paper at ICLR 2024
differently modeled (note the subscript i in notation q:), reflecting different posterior beliefs originating from heterogeneity of episodic datasets D,.
Saaall-nnvere--iihaae (NIW) model. We consider NIW distributions for the prior and varia-
tional posterior. First, the prior is modeled as a conjugate form of Gaussian-NIW. With o = (H, 2),
IW(E; Eg, vo), p(0,l)) (O μ. Σ), i= I....,... N, (3) where A = {uo, Co, An, /n is the parameters of the NIW. We do not need to pay attention to the choice of values for A since p(ő) has vanishing effect posterior for a large amount of evidence as we will see shortly. Next, our choice of the variational density family for g( is the NIW, mainly because it admits closed-fore expressions in the ELBO function due to the conjugacy, allowing efficient local episodic optimisation, as will be shown. For (:()))'s 's we adopt Gaussians. That is,
p(Φ) [N: po. A Σ)-
g(; Lo) == NN; mo. 47 I)) IE; Vo, mo). q(0q;; Li) (0,; me, Vi). So, Lo = {mo, Vo. lo, no} with Vo restricted to be diagonal, and L; = {m,, Vi). Learn- ing (variational inference) amounts to finding Lo and {L,),= that makes the approximation q(0, (I:N; 2 p(o, 81 ( ), as tight as possible.
(4)
Variational inference. The negative marginal log- likelihood (NMLL) has the following upper bound (Appendix [. for derivations):
-logp(DI.)) I (L(g())lp(o)) E' (E,, (E, (o,, (,, (1)((())))
(5)
where ((0))== log p( (.) is the negative training log-likelihone of 0g in episode i. By dividing both sides by N, the LHS naturally becomes the effective episode-averaged NMLL N log p( (1:.) ).
The first KL term in the RHS, 1 (KL(g())lp(e) E [fi] for any expression fú. the ELBO learning (approximately) reduces the following:
diminishes for large N. Using + Σ fi Ro
min Lo.(L:)
E (..00...)44400) + (o( (0,; (.)l(())))
(6)
Local episodic optimisation (whose solution as function of global parameters Lo). Note that (6) is challenging due to a large number of optimisation variables {L.) and the nature of episode sampling T~ Applying conventional SGD would simply fail since each L, will never be updated more than once. Instead, effectively representing the optimal solutions as functions of Lo, namely {L, (Lo)
tackle it by finding the optimal solutions for L3's for fixed Lg, thus
N i-1- Plugging the
optimal L, (Lo)'s 's back to (6) leads the optimisation problem over Ly alone. The idea is just like solving: min..y y f2, y) = min, f( y² (I)) where y²(x) = arg min,, f(( y) with æ fixed.
Note that when fix Lo (i.e., fix (()), the objective 6 is completely separable over i, and can optimise individual independently. More specifically, for each 2 1,
min (4: + E& (0;; [.)]l((0,10)]
As the expected KL. term in 7 admits closed form due to NIW-Gaussiar conjugacy (Appendix B.2 for derivations), we can reduce to the following optimisation for L, = (m, V.):
Eyle log [V]]
+ no (m;-ma) /Vo TV TV./VV)))
[ (To) == arg min m..V.
(8)
2 2
with Lo = {mo, Vo, lo, no} fixed. Here (m,-mp)) ) and .VV are all elementwise operations.
Quadratic approximation of episodic loss via SGLD. To find the closed-form solution L (Lo)
in {8) we make quadratic approximation of (,0,)) = log p( (D)))). In general, -log (D))) as a function of 0. can be written as:
- -log (210) R - (0- A (1(0mm;) mm)) + const.,
99)
for some (VT,, A:) that are constant with respect to 0. One may attempt to obtain (M,, A,) via Laplace approximation (e.g., the minimiser of -log p( (.)6) for m. and the Hessian at the minimiser for A,). However, this involves computationally intensive Hessian computation. Instead, using the fact that the log-posterior log log D.) equals (up to constant) log p( ())0) when we use uninformative prior p(f) x we can obtain samples from the posterior (() using MCMC sampling, especially the stochastic gradient Langevin dynamics (SGLD) (Welling & Teh 2011). and estimate sample mean and precision, which become m. and A,, respectively"" Note that this amounts to performing
Similar approaches include the stochastic weight averaging Izmaiilov et al. 2018 Maddox et al. 2019
3

---

Published as a conference paper at ICLR 2024
several SGD iterations (skipping a few initial for burn-in), and unlike MAML (Fin et al. 2017) no computation graph needs to be maintained since (M, are constant. Once we have (m,, At), the optimisation (8) admits the closed-form solution (Appendix [or4 derivations),
m (Lo) (A, + na/ [V)) -1 (A,m, + ngmo/ /Vo,, (La) = (A; + no/oo) (10)
Computation in (10) is cheap since all matrices are diagonal.
Final optimisation. Plugging 100 back to (6). the final optimisation is (Appendix B.5 for details):
min BENT [i(To) + 200((0) + d
2hn s.t. fi(TT)) = (EEw((..) [bs (m. (Ta) + V;
Lo
IVol (La))
+ no TrV (Lo)VV) + mo (Lo) mma)² /Vo du (2)) (11)
gi(Lo) log
where vha(-) is the multivariate digamma function and d= di) As lo only ap- pears in the term 2h the optimal value 1o 00. We use SGD to solve (T). repeating the two steps: i) Sample i T: iii Lo + Lo ( (Lo)) Note that L (Ji(To)) + (Lo)) is an
Algorithm 1 Our few-shot meta learning algorithm.
Initialise: Lo = {mo, Vo, no } of g(Ø; Lo) randomly.
4_
for episode = 1, 1, do
Perform SGLD iterations on D, to estimate (m.. A.).
Compute the episodic minimiser L; (Lo) from [0).
Update Lo by the gradient of f((Lo) + 59 (o) as in end for
unbiased stochastic estimate for the gra-
Output: Learned Lo.
dient of the objective E-- in [T].. Furthermore, our learning algorithm above (pseudocode
in Alg. D is fully compatible with the online/batch episode sampling nature of typical FSL. After
training. we obtain the learned Ly, and the posterior g(0; Lo) will be used at the meta test time, where we show in Sec. 3.2 that this can be seen as Bayesian inference as well.
We emphasise that our framework is completely flexible in the choice of the backbone p(y) 0). It could be the popular instance-based network comprised of feature extractor and a prediction head where the latter can be either a conventional learnable readout head or the parameter-free one like the nearest centroid classifier (NCC) in ProtoNet (Snell et al., 2017). i.e., P(D(0) (1ff)=ppl =p(Q]S, 0) where D= UUSUUG Q and p(y xr S, 0) is the NCC prediction with support S. We can also adopt the set-based networks (Ye et al. 2020 Garnelo et al. 2018 Kim et al. 2019) where p(y| [ S, 0) itself is modeled
by neural net y G(z, 8; with input S).
3.1 INTERPRETATION
We show that our framework unifies seemingly unrelated seminal FSL algorithms into one perspective.
MAML (Finn et al. 2017) as special case, Suppose we have spiky variational densities, V/2 7 (constant). The local episodic optimisation (8) reduces to: arg minm, + R(m,) where R(m,) is the quadratic penalty of m. deviating from mg. One reasonable solution is to perform a few gradient steps with loss 4, starting from ma to have small penalty 0 initially). That is, mn; mig and few steps of m, < m, a (l,,))) to return m. (L)). Plugging this into [1] and disregarding the Ii term, leads to the MAML algorithm. Obviously, the main drawback is m, (Lo) is function of my E Lo via full computation graph of SGD steps, compared to our lightweight closed forms (IO).
special case. With V; - O, if we ignore the negative log- likelihood term in 8 then the optimal solution becomes m (Lo) = my- If we remove the gi term, we can solve (11 by simple gradient descent with V log p( (D. (m)m))). We then adopt the NCC head and regard ma as sole feature extractor parameters, which becomes exactly the ProtoNet update.
ProtoNet (Snell et al., 2017)
Reptile (Nichol et al. 2018) as special case. Instead, if we ignore all penalty terms in (8) and follow our quadratic approximation ( with V; 0, then m (Tnn) ,ेद It is constant with respect to Lo = (mo, Vo, mg)), and makes the optimisation (II) very simple: the optimal my the average of M, for all tasks i, i.e., mi = EET [77.]] (we ignore Vo here). Note that Reptile ultimately finds the exponential smoothing of m This can be seen as an online/running estimate of Ei-- [7..].
(k) over T where m
(k) is the iterate after SGD steps for task i.
3.2 META TEST PREDICTION AS BAYESIAN INFERENCE
At meta test time, we need to be able to predict the target y of a novel test input x= TT* sampled from the unknown distribution T p(T) In FSL, have the test support data D* {(x, v)) ~T* The
4

---

Published as a conference paper at ICLR 2024
Table 1: Three competing Bayesian models for the toy experiment. (Fig. 3 for graphical models)
Model
Model II
Model III (Ours)
y = 0 x + 8 + Ey.
y= x x 8 Ey-
y = 0. x + 8 + cv
p(0,, 3.) = NN4, Vi
p(0, B) = N(,,
p(6) = (mm, V): p(0., 3.10) = N(o, Vi
test-time prediction can be seen as posterior inference problem problem with additional evidence of the sup- port data D" (Fig. [()). More specifically, p(y" D", Di.v) p(y" |=",') p(8)D", Di.N) d8. So, boils down to D" DI.N), ) the posterior given both the test support data D* and the entire training data DI.N. Under our hierarchical model, exploiting conditional independence (Fig. [c)),
we can link it to our trained q(() as:
/ p(0)D", 6) p( DINN) do = = () (()) do = p(0 o"), (12)
D(0 DI:N) ~
where in the first approximation in (2) we disregard the impact of D on the higher-level e given the joint evidence, i.e., p(, DI.N) Rs p(o) D1:N). ). due to dominance
DI.V compared to smaller D We use the delta function approximation in the last part of (12 with the mode o* of g()), where 6- = (p², , Σ*) has a closed form u = my. Σ = Vo/(no - d+2).
Next, since D", Φ²)) involves difficult marginalisation p ( f p( o- d0,, adopt variational inference, introducing a tractable variational distribution v(0) = p(A) o*). With the Gaussian family as in the training time (4), i.e., u(0) = N(, V) where (m, V) are the variational parameters optimised by ELBO optimisation,
min Eo) -- log (D + KL (L()))) p(0)o")) where o" == (mo, (o
(13)
m.V
The detailed derivations for (13 can be found in Appendix [B.6]
Once we have the optimised model v, our predictive distribution can be approximated by the Monte-Carlo average. ply" D" DI.N) R (1/Ms) ΣM p(v"'=", ;(())), where ((*) ~(()) for 8 = 1,.... Ms samples. which simply requires Geed-forwrdiig x² through the sampled networks g(*) and averaging. Our meta-test algorithm is also summarised in Alg. 2 (Appendix). Note that we have test-time backbone update as per (I3). which can make the final m deviate from the learned mean mg. Alternatively, if we drop the first term in [13. the optimal v0) equals ))) (O0 mo, Vo (mo This can be seen as using the learned model my with some small random perturbation as a test-time backbone B.
TOY EXPERIMENT: WHY HIERARCHICAL BAYESIAN MODEL?
To demonstrate why our hierarchical Bayesian modelling is effective for few-shot meta learning problems, we devise simple toy synthetic experiment as proof of concept. We consider multi-task (Bayesian) linear regression problem. The data pairs (zr E 2R, .22,y y IR) for each episode i are generated by the following process: y (Wahrred (w)) x: + by()) + Ey where Wstured is the episode-agnosticc shared weight vector Vi, and we have episode-dependent intercept by(0) among the three candidates (b,, b2. by}. we select j(i) ~ (1.2.3) uniformly at random for each episode i. Please refer to Appendix. [iror full details and derivations. In this way we ensure that the resulting episodes only related to one another through the shared weight vector, but they are differentiated by potentially different intercepts. We sample N = 40 episodes for training and 10 episodes for test. Each training episode has D = 3 samples, and the support set at test time also has [D| 3 labeled samples.
not
Three competing models. We consider three Bayesian models with differrnt levels/degrees of flexibility and regularisation as outlined in Table [] Model I has episode-wise parameters (0,, 3.), thus highly fiexible. However, these parameters are all independent across episodes, hence lacking regularisation. Model II is conventional (non-hierarchical Bayesian model where single parameter set (0, B) is shared across episodes, thus too much regularisation with lack of flexibility. Model III our hierarchical Bayesian model which imposes balanced flexibility and regularisation episode- wise (0,, 3,)'s allows high flexibility, but unlike Model I, the higher-level variable & regularises the episode-specific parameters, and captures the inter-episodic shared information.
Results. After training (learning μ for Model I and II: learning (m, V) for our Model III), at test time, for each of 10 test episodes, we obtain the posterior means of the weights and intercept parameters Eie,5 D D1:N] for the three models. They all admit flosed-fomm solutions as detailed in Appendix C and predict the outputs of ~ 50 unseen test inputs. The mean absolute errors (MAE) averaged over 10 test episodes are: Model I = 2.87, Model Il = 3.13, and Model III (ours) our model to other competing methods. Fig. visualises
1.28, clearly showing the superiority of
5

---

Published as a conference paper at ICLR 2024
Emore Visel
1.33 Model I1 0.45, Node B1 0.00
Cronc Modell 1,45, Model
0:10, Model II 0.10
Dron: Nodel 2 2.22, Nodel
2.31, Mosel i 1,21
Moceal
-4.0
-50
-5.
1000 FRE 1O0 21L 20 sproodees
Foe entenoagt IDs (OA5)) 10 wphones
bue erierugg 10s (uded)) 33 for spino
(a) Weight dim-1
(b) Weight dim-2
(c) Intercept
Figure 2: Toy experiments. Visualisation of the learned posterior means compared to the true values (blue-circled). (a) weight dim- 1 (0[0) vs. [0]) (b) weight dim-2 (0[1) vs, Wahared [1) and (c) intercept (3 vs. (.))) In each plot, the X-axis shows the indices of the true intercepts sampled, that is, j(*) E {1, 3). for 10 test episodes. In the titles we also report the distances (errors) between the true values and the posterior means for the three methods, averaged over 10 episodes.
the results. First, Model II's posterior means rarely change over test episodes, meaning the impact of test support data D. is limited. This behavior is expected since the model imposes too much regularisation with little flexibility, and the test prediction is dominated by the mean model obtained from training data DI.N. Model I exhibits highly sensitive predictions over test episodes, which mainly originates from little regularisation the posterior too sensitive to the current episode's support data, thus being vulnerable to overfitting especially when the support data size is small, typical the few-shot learning. The model failed to capture useful shared information Wshared- Our Model III balances between these two extremes, imposing proper amount of regularisation and endowing adequate flexibility. Our posterior estimation best extracts the shared spisode-agnostoe information (the weight parameters fluctuate less the test episodes), and captures the episode-specioe features the most accurately (the estimated intercepts are aligned the best with the true values).
5 THEORETICAL ANALYSIS: GENERALISATION ERROR BOUNDS
We offer two theorems for the generalisation error of the proposed model. The first theorem relates the generalisation error to the ultimate ELBO loss (6) that we minimised in our algorithm, and we utilise the recent PAC-Bayes-A bound (Thiemann et al. 2017 7 Rivasplata et al. 2019). The second theorem is based on the recent regression analysis technique (Pati et al.) 2018 Bai et al, 2020).
Without loss of generality assume D n for all episodes i. We let (q (0), of (0.) F be the optimal solution of (6). We leave technical details and the proofs for both theorems in Appendix A
Theorem 5.1 (PAC-Bayes-A bound). Let R2(0) (0) be the generalisation error of model 0 for the task i, more specifically, (0) = ((n,y) TT, log ply z, 0)]. As the number of training episodes N + oo, the following holds with probability least 8 for arbitrary small 8 >> 0:
2e
B Ey (0:) (R. (0.)
S
where e" = the optimal value of 6
(14)
Theorem 5.2 (Bound derived from regression analysis). Let diy (Po,, P() be the expected squared
Hellinger distance between the true distribution P'(y(ar) and model's 's Po, (ylx)) for task i. As the number of training episodes N oo, the following holds with high probability:
E (0.) (Po. P') < 0 (2+ +
(15)
where X' = =EiT X7). x; mingee [Eo | E' 111-1113 is the lowest possible regression ervor within O and Fas En are decreasing sequences vanishing to 0 n. increases.
6 RELATED WORK
Due to limited space it infeasible review all general FSL and meta learning algorithms here. We refer the readers to (Hospedales et al.) 2022 Wang et al. (2020)), the excellent comprehensive surveys on the latest techniques. We rather focus on discussing recent Bayesian approaches and their relation to ours. Although several Bayesian FSL approaches have been proposed before, most of them dealt with only a small fraction of the network weights (e.g.. a readout head alone) as random variables (Gamelo al.] [2018] Kim et al.) 2019 Requeima al.) 2019 Gordon et al.) 2019 Patacchiola et al. 2020 Zhang et 2021). This considcrably limits the benefits from uncertainty modeling of full network parameters.

---

Published as a conference paper at ICLR 2024
Table 2: Classification accuracies with standard backbones on minilmageNeet and lageee
(a) minninageNee
(b) dierodmnggNe
Model
Backbone 1-Shot
5-Shot
Model
Backbone 1-Sbor
5-Shot
AM3 Xing el al.
2019
ResNer 65.21 +0.49 75.20 0.58
TapNet (Yoon el
2019
ResNet 63.08 63.08 +- 80.26 +2.12
RelationNer2 Zhang et 2020
ResNort2 20.92 77.15 77.09
RelationNet2
ResNert# 68.58 29.558.0..0 80. 65 t0.91
MetaOpt LLee et al 2019
ResNert2 61.09:00.22
50.00=0.45
MetaOpt
ResNet"2 65.81 20.77 81. 31.75 20.30.
SimpleShot Wang el al 2019
ResNet" 62.85200.2 20 80. 02.022-:14
SimpleShot
ResNet" 69.09-20.22 84.58 to
S2M2 Mangla al.
2020
ResNat" 64.06400.11 16 80.58
MetaQDA
RexNet" 69.9740.52 s5.55 20.50
MetaQDA (Zhang el al
2021
ResNet" 65.12 80.98 -0.75
NIW-Meta (Ours)
ResNet* 70.52++0.19 85.83 t0.17
NIW-Mela (Ours)
ResNer 45.4 81.71 70.17
LEO Rusu ot al. 2019
WRN 66.33-2000 81. $1.44 2110000
SimpleShot
WRN 63.50=00:00 30 60.33=0.14
SimpleShot
WRN28-10 69.75 85.31 10.10
$2M2
WRN2S 64.93++0. 18 83.18 18
S2M2
WRN28-10 73.71 # 88.50 58.50+0..1 10
MetaQDA
WRN28 07 83 +0000.04 04 24.28 00.00
MetaQDA
WRN28-10 +-0.0 89 56 to
NIW-Meta (Ours)
WRN28 68.54 20.20 84.81 #0028
NIW-Meta (Ours)
WRN28-10 74.59=0.33 79.
Bayesian approaches to MAML (Finn et al. 2018: Yoon et al. 2018 Ravi & Beatson 2019: Nguyen
et al. 2020) are popular probabilistic extensions the madient-based adaptation in MAMI. (Finn 2017) with known theoretical support (Chen & Chen, 2022). However, they are weak in several aspects to be considered as principled Bayesian methods. For instance, Probabilistic MAML (PMAML) (Finn et al. 2018 Grant et al. 2018) has similar hierarchical graphical model structure as ours, but their learning algorithm considerably deviates from the original variational inference objective. Unlike the original derivation of the KL term measuring the divergence between the posterior and prior on the task-specific variable f;, namely Eg( [L. (0, [o]]o]]p p(0. [o)]] as in ().
et al.
in PMAML they measure the divergence on the global variable é, aiming to align the two adapted models, one from the support data only ((b)S:) and the other from both support and query (( SS Q.).
VAMPIRE (Nguyen et al. (020) incorporates uncertainty modeling to MAML by extending MAML'S point estimate to a distributional one that is learned by variational inference. However, it inherits all computational overheads from MAML, hindering scalability. The BMAML (Yoon et al. 2018 not a hierarchical Bayesian model, but aims to replace MAML's gradient- based deterministic adaptation steps by the stochastic counterpart using the samples (called particles) from p(t)) thus adopting stochastic ensemble-base, adaptation steps. If a single particle is used, it reduces exactly to MAML. Thus existing Bayesian approaches are not directly related to our hierarchical Bayesian perspective. A related but different line of research studied Bayesian neural processes (Volpp et al.) [023 2021
Qi Wang 2020 Garnelo et al. (0188) by treating the support set embedding as random variates.
EVALUATION
7.1 FEW-SHOT CLASSIFICATION
Standard benchmarks with ResNet backbones. For standard benchmark comparison using the popular ResNet backbones, R8 (He et al.. 2016) and WideResNet (Zagoruyko & Komodakis 2016), we test our method on: minilmagenet and magee (Table 2. We follow the standard protocols (details of experimental settings in Appendix D). Our NIW- exhibits consistent improvement over the SOTAs for different settings in support set size and backbones.
Large-scale ViT backbones. We also test our method on large-scale (pretrained) ViT
Table 3: Classification accuracies with large-scale ViT backbones.
Model
Backbone
minilmageNet
CIFAR-FS
tirredImageNet
/Peetriin
1-bbot
5-shot
I-shot
3-shot
T-shot
5-shot
backbones DINO-small (Di-
ProtoNet
DINOSS
90.1 98.0
t1.1
92.5 89.0 t0.0
no/s) and DINO-base (DI-
MetaOpt
DINOS
92.210022 97.6
70.2 20.22.
$1.1 10.27
87.510.25
91.710.20
MetaQDA
DINOS 92.0+0.31
97.0
77.2
+0.1
87.8+0.27
90.6+0.10
NO/b) (Caron al., 2021 1) :
NIW-Meta
DINOS
93.4+0.17
98.2#0.15
82.8+
92.9=0.11
89.3+0.11
96.0
similarly as the setup in (Hu
ProtoNet
DINOS
93.3t0.13
98. 1t.11
84.3
92.2
91.3
t6.5
al. 2022). In Table 3 we re-
MetaOpt
DINOS 94.4-0.11
98. 44.040
72.0-..29
$6.2-0.11
89
89.54..27
95.7-0.15
port results on the three bench-
MetaQDA
DINO 94.7+0.21
98.7#0.14
80.9+0.91
93.8=0.11
89.7 89.7+0.21
+6.0
NIW-Meta
DINO/
95.5to.1s
98.770.12 81.7t0.13
03.2 t1.1 t0.7
marks: minilmagenet, CIFAR-
FS, and rieredlmageNet. Our NIW-Meta adopts the same NCC head as ProtoNet after the ViT feature extractor. As claimed in (Hu et al. 2022), using the pretrained feature extractor and further finetuning it significantly boost the performance of few-shot learning algorithms including ours. Among the competing methods, our approach yields the best accuracy for most cases. In particular, compared to the shallow Bayesian MetaQDA (Zhang et al.) 2222I), treating all network weights as random variates in our model turns out to be more effective than the readout parameters alone.
Set-based adaptation backbones. We also conduct experiments using the set- based adaptation architecture called FEAT introduced in (Ye et al.) 2020). The network is tailored for
Table 4: FEAT
method. Classification accuracies.
mintImageNet
nieredImageNer
Model
T-ooot
3-shet
T-boot
3-shot
FEAT
66.
82.05
70.50 10.50-0.11
84.79-0.11
NeW- (Ours)
66.91 66.9149.10
82.28 40.15
70.93
85.20

---

Published as a conference paper at ICLR 2024
few-shot adaptation, namely 2 = G where the network G takes the entire support set S and query image x as input. Note that our NIW-Meta can incorporate any network architecture, even the set-based one like FEAT. As shown in Table 4 the Bayesian treatment leads to further improvement over YYeet et al.
2020) with this set-based architecture.
Error calibration. Bayesian models are Table 5: ECEs on minilmageNet. "ECE+TS" indicates known to be better calibrated than determin- extra tuning of the temperature hyperparameter.
istic counterparts. We measure the expected calibration errors (ECE) (Guo et al. (017)) to judge how well the prediction accuracy and the prediction confidence are aligned-
Model
Backhone
ECE
ECE+TS
I-shot
S-shot
I-shot
S-shor
Lineur classifier
Conv44
8.54
7.48
3.56
2.85
SimpleShot
Couv44
33.
45.81
3.82
3.35
McuQDA-MAP
Con-44
8.03
5.27
2.75
0.89
No aacc())con f(b) where ECE= b-t NN'
MetiQDA-FR
Cons44
4.32
2.92
2.33
0.45
we partition test instances into B bins along the model's prediction confidence scores, and cb). acc(b) are the average con- fidence and accuracy for the b-th bin, re- spectively. The results on maielmageNeet
NIW- (Ours)
Conv-4
2.68
1.88
1.47
0.32
SimpleShot
WRN-28-10
39.56
55.65
4.05
1.80
S2M2 | Linear
WRN-28-10
33.23
36.84
4.93
2.31
MetaQDA-MAP
WRN-28-10
31.17
17:37
3.94
0.94
MetaQDA-ra
WRN-28-10
30.68
15.86
2.71
0.74
NIW-Meta (Ours)
WRN-28-10
10.79
7.11
2.03
0.65
are shown in Table[5] We used 20 bins and optionally performed the temperature search on validation
sets, similarly as (Zhang et al., (02T). Again, Bayesian inference of whole network weights in our NIW-Meta leads to far better calibrated model than the shallow Meta-QDA (Zhang et al. 2021).
7.2 FEW-SHOT REGRESSION
Sine-Line dataset (Finn et al., 2018). The 1D (z,y) data pairs are generated by ran- domly selecting either linear or sine curves with different scalesslopesfiieeeneeesssppppssp For the episodic few-shot learning setup, we follow the standard protocol: each episode is comprised of k- = 5-shot support and 45 query samples randomly drawn from a random curve (regarded as a task). To deal with real-valued targets, we adopt the so-called RidgeNet, which has a parameter-free readout head derived from the support data via (closed- form) estimation of the linear coefficient matrix using the ridge regression (the L2 regularisation coefficient 0=0.1).
Table 6: Sine-Line results. PMAML w/
5 inner steps incurred numerical errors.
Model
Mean squared
R-ECE
RidgeNet
0.8210
N/A
(I-sep))
0.8206
N/A
MAML (5-step)
0.8309
N/A
PMAML (1-step)
0.9160
0.2688
NIW-Meta (Ours)
0.7822
0.1728
is analogous to the ProtoNet (Snell et al.) 2017) in classification which has a parameter- free head derived from NCC on support data. A simi lar model was introduced in (Bertinetto et al.) 2019) but mainly repurposed for classification. We find that RidgeNet leads to much more accurate prediction than the conventional trainable linear head. For instance, the test errors are: RidgeNet = 0.82 vs. MAML with linear head = 1.86. Furthermore, we adopt the ridge head in other models as well, such as MAML, PMAML (Finn et al.) (2018), and our NIW-Meta. See Table/6 for the mean squared errors contrasting our NIW-Meta against competing meta learning methods. The table also contains the regression- ECE (R- BCF) calibration erron?? Clearly our model is calibrated the best.
Object pose estimation on ShapeNet datasets. We consider the recent few-shot regression benchmarks (Gao et al., 2022; Yin al., (020) which introduced four datasets for object pose estimation: Pascal-ID, ShapeNet-ID, ID, ShapeNet-2D and Distractor. In all dataset=]] the main goal is to estimate the pose (positions in pixel and/or rotation angles) of the target object in an image. Each episode is specified by: i) selecting target object randomly from pool of objects with different object categories, and ii) rendering the same object in an image with several different random poses (position/rotation) to generate data instances. There are k support samples (input images and target pose labels) and ky query samples for each episode. For ShapeNet- ID, for instance, 1: is randomly chosen from 3 15 while k, = 15. Except Pascal- some object categories are dedicated solely for
"The definition of the R-ECE is quite different from that of the classification ECE in Sec. 7.1 the notion of goodness of cumulative distribution matching used in (Tran et al. 2020 Cui et al. 2020).
We follow
Specifically, denoting by @p(z) the p-th quantile of the predicted distribution p(y[=)) we mcasure the deviation
of Ptruecy (y Σ Qx(x)(a [z) from p by absolute difference. So it is 0 for the ideal case p(y[æ) Ptrss (ylæ). Note that by definition we can only measure R-ECE for models with probabilistic output p(u|=))
'Pascal- 1D and ShapeNet- 1D are relatively easier datasets than the rest two as we have uniform noise- free backgrounds. To make the few-shot learning problem more challenging, ShapeNet-2D and Distractor datasets further introduce random (real-world) background images and/or so called the distractors which are objects randomly drawn and rendered that have nothing to do with the target pose to estimate.
8

---

Published as a conference paper at ICLR 2024
Table 7: Pose estimation results. Mean squared errors in rotation angle differences (Pascal- ID and ShapeNet-1D), quaternion differences x x10-2 (ShapeNet-2D) and pixel errors (Distractor). The dataset-wise different augmentation schemes are shown in the parentheses.
Pascal-ID (TA)
ShapeNet-ID (TA+DA) ShapeNet-2D (TA+DD+DD)
Distractor (DA)
Model
Intra-calegory Clos-culegory Intrarca Cros-calegary
Iutrary Cross-cllggryy
MAML
1.02 + 0.06
17.96
18.79
CNP (Gamelo et
2018
1.98 = 0.22 7.06
7.66 0.18 8.66 = 0.19
14.2040
13.5640.25
2,45
3.75
CNP+BB BA Volpo al.
2021
14.1640.08
13.5640.18
3.97
CNP4+PC +FC (aa
3.05
ANP Kim al.
1919)
1.36 - 0.25 5.81 + 0.23 6.23 0.11
14.12+0.14
13.59
2.65
4.08
ANP+FCLL Goo al
2023
14.01 +0.00
13.32
NIW-Mela w/ C+R
0.89 + 0.06 5.62
1 0.38 6.57
1 0.39
21.2520.76
20.8220.44 43
8.90 26
17.3120.30 20.38
NIW
Meta Net-Meta
CNP
0.94 1 0.15 5.74
1 0.17 6.91
1 0.18
13.86+00.20
13.04
1.80
2.94
NeW- w/ ANP
0.95 1 0.09 5.47 + 0.12 6.06 + 0.18
18.74
12.95
3.10
5.20
meta testing and not revealed during training, thus yielding two different test scenarios: intra-category and cross-category, in which the test object categories are seen and unseen, respectively.
In (Gao et al. 2022). they test different augmentation strategies in their baselines: conventional data augmentation on input images (denoted by DA), fask augmentation (TA) (Rajendran et al.) 2020 which adds random noise to the target labels to help reducing the memorisation issue (Yin et al. 2020), and domain randomisation (DR) (Tobin et al. 2017) which randomly generates background images during training. Among several possible combinations reported in (Gao et al., 2022). we follow the strategies that perform the best. For the target error metrics (e.g.. position Euclidean distances in pixels for Distractor, rotation angle differences for ShapeNet-1 we follow the metrics used in (Gao al. 2022). For instance, the quaternion metric may sound reasonable in ShapeNet-2ID due to the non-uniform, non-symmetric structures that reside in the target space (3D) rotation angles).
The results are summarised in Table 7 In (Gao et al. 2022), they have shown that the set-based backbone networks, especially the Conditional Neural Process (CNP) (Garnelo et al. 2018) and Attentive Neural Process (ANP) (Kim al. 2019) outperform the conventional architectures of the conv-net feature extractor with the linear head that are adapted by MAML (Finn et al. 2017) (except for the Pascal- ID case). Motivated by this, we adopt the same set-based CNP/ANP architectures within our NIW-Meta. In addition, we also test the ridge-head model with the conv-net feature extractor (denoted by C+R)). Two additional competing models contrasted here are: the Bayesian context aggregation in CNP (CNP+BA) (Volpp et al. 202 l) and the use of the functional contrastive learning loss as extra regularisation (FCL) (Gao et al. 2022).
For P and ShapeNet-1D, there is dataset regime where MAML clearly outperforms (Pascal- ID) and underperforms (Shape ID) the CNP/ANP architectures. Very promisingly, our NIW-Meta consistently performs the best for both datasets, regardless of the choice of the architectures: not just CNP/ANP but also conv-nct feature extractor + ridge head (C+R). For ShapeNet-2D and Distractor where MAML is not reported due the known computational issues and poor performance, our NIW-Meta still exhibits the best test performance with CNP/ANP architectures. Unfortunately. the conv-net + ridge head (C+R) did not work well, and our conjecture is that the presence of heavy noise and distractors in the input data requires more sophisticated modeling of interaction/relation among the input instances, as is mainly aimed (and successfully done) by CNP/ANP.
Computational complexity, running time and memory footprint. We have analysed the computational complexity of our NIW-Meta compared to the simple feed-forward workflows (c.g.. ProtoNet). Our method incurs only constant- factor overhead compared to the minimal-cost ProtoNet, as summarised in Table & in Appendix E Also in Fig. Fig-5]thee therrin. we also report the memory footprints and running times of MAML and our NIW-Meta on real datasets, which show that NIW-Meta has far lower memory requirement than MAML. MAML suffers from heavy use of memory and thecomputational overhead of keeping track of a large computational graph for inner gradient descent steps. Our NIW-Meta has a much more efficient strategy of local episodic optimisation that is linked to global parameters, without storing the full optimisation trace. Please see all details in Appendix E
8 CONCLUSION
We have proposed a new hierarchical Bayesian perspective to the episodic FSL problem. By having a higher-level task-agnostic random variate and episode-wise task-specific variables, we formulate principled Bayesian inference view of the FSL problem with a large number of tasks (evidence). The effectiveness of our approach has been verified empirically in terms of both prediction accuracy and calibration, on a wide range of classfiication/eegressio tasks with complex backbones including ViT and bet-based adaptation networks.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jincheng Bai, Qifan Song, and Guang Cheng. Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee. In Advances in Neural Information Processing Systems, 2020.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In International Conference on Learning Representations, 2019.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer. 2006.
S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: Nonasymptotic Theory of Independence. Oxford University Press, 2013.
Michael Braun and Jon McAuliffe. Variational inference for large-scale models of discrete choice. arXiv preprint arXiv:0712.2526, 2008.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the International Conference on Computer Vision (ICCV), 2021.
Lisha Chen and Tianyi Chen. Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?, 2022. Al and Statisties (AISTATS).
Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. In Advances in Neural Information Processing Systems, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic Model-Agnostic Meta-Learning In Advances in Neural Information Processing Systems, 2018.
Ning Gao, Hanna Ziesche, Ngo Anh Vien, Michael Volpp, and Gerhard Neumann. What matters for meta-learning vision regression tasks? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14776-14786, June 2022.
Marta Garnelo, Dan Rosenbaum, Chris Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional Neural Processes, In International Conference on Machine Learning, 2018.
Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Texts in statistical science. Chapman Hall/ / CRC, 2nd edition, 2003.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Represen- lations, 2019. URL. htt.:tttts///peereeeeeee
net. www.fooormmmddissmmcooom
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Tom Griffiths. Recasting pradient-based meta-learning as hierarchical bayes. In ICLR, 2018.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta-learning. 2019.
arXiv preprint
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2016.
10

---

Published as a conference paper at ICLR 2024
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44: 5149-5169, 2022.
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference. In CVPR, 2022.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Uncertainty in Artificial Intelligence, 2018.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. In International Conference on Learning Representations, 2019.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
John Langford and Rich Caruana. (Not) Bounding the True Error. In Advances in Neural Information Processing Systems, 2001.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2019.
David MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A Simple Baseline for Bayesian Uncertainty in Deep Learning. arXiv preprint arXiv:1902. 1902.02476.. 2019.
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vi- neeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning, In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020.
Andreas Maurer. Note on the PAC Bayesian Theorem. arXiv preprint arXiv:0411099, 2004.
David McAllester. Some pac-bayesian theorems. Machine Learning, 37:355-363, 1999.
Kevin P. Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL
probml.32
Cuong Nguyen, Thanh-Toan Do, and Gustavo Carneiro. Uncertainty in model-agnostic meta-learning using variational inference. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3090-3100, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Leaming Algorithms. In arXiv preprint arXiv: 1803. 029999 2018
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, and Amos Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In Advances in Neural Information Processing Systems, 2020.
D. Pati, A. Bhattacharya, and Y. Yang. On the Statistical Optimality of Variational Bayes, 2018. AI and Statistics (AISTATS).
Herke van Hoof Qi Wang. Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables. In International Conference Machine Learning, 2020.
11

---

Published as a conference paper at ICLR 2024
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-Learning Requires Mett-ntugennttionn In Advances in Neural Information Processing Systems, 2020.
Sachin Ravi and Alex Beatson, Amortized Bayesian meta-learning. In International Conference on Learning Representations, 2019.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and Flexible Multi- Task Classification Using Conditional Neural Adaptive Processes. In Advances in Neural Information Processing Systems, 2019.
Omar Rivasplata, Vikram M Tankasali, and Csaba Szepesvari. PAC-Bayes with Backprop. arXiv preprint Ar0i 07380, 2019.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero. and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
Matthias Seeger. PAC-Bayesian Generalization Error Bounds for Gaussian Process Classification. Journal of Machine Learning Research, 1:233-269) 2002.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. CoRR, abs/1700300515, 2017. httttt:://aae htttt::://xxxi..o///aa...
Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul P. Srinivasan, Jonathan Bar- ron, and Ren Ng. Learned Initializations for Optimizing Coordinate- Baased Neural Representations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Niklas Thiemann, Christian Igel, Olivier Wintenberger, and Yevgeny Seldin. A strongly quasiconvex PAC- May-siaesian bound. In International Conference Algorithmic Learning Theory, 2017.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pp. 23-30, 2017. doi: 10.1109/ROS..00 201...002....2..1.
Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, and Zachary W Ulissi. Methods for comparing uncertainty quantifications for material property predictions. Machine Learning: Science and Technology, 1(2):025006, 2020.
Michael Volpp, Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian Context Aggregation for Neural Processes. In International Conference on Learning Representations, 2021.
Michael Volpp. Philipp Dahlinger, Philipp Becker, Christian Daniel, and Gerhard Neumann. Accurate Bayesian Meta-Learning by Accurate Task Posterior Inference. In International Conference on Learning Representations, 2023.
Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearestneighbor classification for few-shot learning. In arXiv preprint 19X1v:1911.04663, 2019.
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3):1-34, 2020a.
Yaqing Wang, Quanming Yao. James Kwok, and Lionel M Ni. Generalizing from a few examples: A survey few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020b.
Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference on Machine Learning, 2011.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O. Pinheiro. Adaptive cross-min cross-modal few-shot learning. In Advances in Neural Information Processing Systems, 2019.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8808 8817, 2020.
12

---

Published as a conference paper at ICLR 2024
Mingzhang Yin. George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-Learning without Memorization. In International Conference on Learning Representations, 2020.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian Model-Agnostie Meta-Learning. In Advances in Neural Information Processing Systems, 2018.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. TapNet: Neural network augmented with task- adaptive projection for few-shot learning. In International conference on Machine Learning. 2019.
S. Zagoruyko and N. Komodakis. Wide residual networks. In arXiv preprint 1605: 2016.
Xueting Zhang, Yuting Qiang. Sung Flood, Yongxin Yang, and Timothy M. Hospedales. RelationNet2: Deep comparison columns for few-shot learning. In International Joint Conference on Neural Networks (IJCNN, 2020.
Xueting Zhang, Debin Meng, Henry Gouk, and Timothy Hospedales. Shallow Bayesian Meta Learning for Real- Real-World Few-Shot Recognition. In International Conference on Computer Vision, 2021.
13

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Proofs for Generalisation Error Bounds (Sec.
A
Proof for PAC-Bayes-A Bound (Sec. A.1)
Proof for Regression Analysis Bound (Sec. A.2
Detailed Derivations (Sec.
Toy Experiment: Why Hierarchical Bayesian Model? (A Detailed Version) (Sec.
Implementation Details and Experimental Settings (Sec. D
Computational Complexity, Running Time and Memory Footprint (Sec.
Training Stability and Impact of Number of Training Episodes (Sec.
Additional Discussions (Sec. G
Comparison to Bayesian Neural Processes (Sec. G.
Justification of Model and Algorithm Choices (Sec. G.2)
Limitations and Future Works (Sec. G.3
A PROOFS FOR GENERALISATION ERROR BOUNDS
We prove the two theorems Theorem 5.1 and Theorem 5.2 in the main paper that upper-bound the generalisation error of the model that is averaged over the learned posterior q(db, 01.N). Without loss of generality we assume D = n for all episodes i. We let (q" (Ø). : {qī (()));=))) be the optimal solution of 66. In these theorems we often make the assumption of N co, that the number of training episodes tends to infinity, mainly for mathematical convenience. This assumption may not be true for practical situations, however, the theorems still be applicable with approximate guarantees where we provide several justifications for this in Sec.
A.1 PROOF FOR PAC-BAYES-A BOUND
First, Theorem reiterated below as Theorem
A.I relates the generalisation error to the ultimate
ELBO loss 66) that we minimised in our algorithm.
Theorem A.1 (PAC-Bayes-A bound). Let R, (0) be the generalisation error of model for the task i, more specifically, (0) (( TT, [- log pl (ylx, (rr,)) with the assumption of (0. boouded errors. As the number of training episodes + co. the following holds with probability at least 8 for arbitrary small 0:
E (R. &
(16)
where e" the optimal value of
Proof. We utilise the recent PAC- PAC-Bayes-A bound TThiemann et al. 2017 Rivasplata et al. 2019)
variant of the traditional PAC-Bayes bounds (McAllester) 1999 Langford & Caruanal 2001 Seeger
2002 Maurer 2004). It states that for any À E (0,2), the following holds with probability least 1-8:
1
(L(((3)(Pp(3) log(22/m/)
E B < X/2
E Bgtm Rm (B)]
(17)
A(1-1/2)
where 3 represents all model parameters (random variables), R(3) is the generalisation emror/lsss for given model B, and (3) is the empirical error/loss on the training data of size m. It holds for any data-independent (c.g.. prior) distribution p(3) and any distribution (possibly data-dependent, e.g., posterior) q()).
14

---

Published as a conference paper at ICLR 2024
Now, with N oo, rewrite ( in equivalent form as follows:
min Tor(1. (
Q(Lo, (4)))))
Bg(d:Lo) II. (.0..0..))) (E, +
(18)
N
KI. ( In) [o) L.) | (1000
N-+
Then we 3= 01.N]: ( ((.)) and p(3) p( p(
We also define
the generalisation loss and the empirical loss as follows:
N
N (2(
1
R(3)
E (..))
(19)
N
N i=1
11 N N
Rm (3) =
E
(a.)))D.
ply x. 0)] |
log pl (D.(6.)
Σ
((,,) (20)
n
n2
Note that the empirical data size m n.N in our case. Plugging these into
17 with X leads to:
E (0.) (R.(O)) S
+ + 1og(2V/N/0) (21)
2 N Taking N do in 21 makes i) the LHS become Ev. (R. ii) the complexity term log(2//NN88 (8) in the RHS vanish, and iii) the RHS converge " QQQ {Li)e That is, nN
i= g..0. +
n.
E E (0,) (0.) S Q(LLo (4.),)))
2
(22)
7
Since 22
holds for any g. we take the minimiser q² of 6 which completes the proof.
A.2 PROOF FOR REGRESSION ANALYSIS BOUND
Theorem 5.2 reiterated below as Theorem A. in a more detailed form, is based on the recent regression analysis techniques (Pati et al., 2018 Bai et al. 2020). Before we prove the theorem, we formally state some core assumptions and notations. Let P' y) be the true data distribution for episode/task where = 1..... N and N + cc. We consider regression- basidd data modeling, assuming that the target y is real vector-valued (y E R.S). Also it is assumed that there exists true regression function f S= + Sy for each i, more formally P (yylx) = N(y: (z)) where o² is constant Gaussian output noise variance.
For easier analysis we assume that the backbone network is an MLP with L width-M hidden layers, and all activation functions (()) are Lipschitz continuous with 1. We consider the bounded dim(0) and is the maximal
parameter space, E θ = {0 E RC ||0]|= < B), where norm bound. Then the prediction (regression) function fo S.S. + R is induced from O as: Po(y|æ) = Nyr So (x), o?? I), where the true noise variance is assumed to be known. The expressions Eo(:) and E' [-] refer to the expectations with respect model's 's Pa and the true Pt, respectively. The generalisation error measure that we consider is the expected squared Hellinger distance between the true pi and the model Ps. more specifically,
1 (-
[1fo(x)
d (Po, Pi) E (o) [H ("()))) P'(y(=))] Plo
807
(23)
Now we state our theorem.
Theorem A.2 (Bound derived from regression analysis). Let dPo, P*) be the expected squared Hellinger distance between the true distribution P (y)xz) and model's 's Po, (v)xz) for task/episode i. Then the following holds with high probability:
Co +C C2(rn + +*),
By (d) P')] <
(24)
15

---

Published as a conference paper at ICLR 2024
where C. > 0 constant, X [[]T[A]] with X; minøge max; [Ea]y|22] - E EEyy/=>11 is
the lowest possible regression error within the underlying network O In G (I. 1) log M +
log ( (5.VE) and cn /rolog (n) for 6 constant.
Proof. We utilise the Donsker- Varadhan's ' (DV) theorem (Boucheron et al. 2013) to relate the variational ELBO objective function to the Hellinger distance. The DV theorem says that the following inequality holds for any distributions p. q and any (bounded) function h(2):
log (ep()) max (By(s)[[(=]] KL(glpp))-
(25)
In our case, we define: p(z) p (1) ((2) (:.))), h(2) = log 1/s (9:) with
n: (0,0)) == exp (o(Po, (D:): (D)))) + nd (Po. P'))
(26)
where (P,), P'(,)) log PTD.)) is the log-ratio. Note that P(D.) = P(Y] Plugging
(D.)
these into 25 leads the following inequality which holds for any Φ:
n c,, (d (Pots P')] S - (D:) (D.),, P((D))) (L)):(())ll((AAA ((0 (0)) + log (mo0))))((...).
(27)
We take the expectation with respect to (()), which yields:
(o, (d. Pi)]
B.(t.)
(po (D,) (DD)))
E (())
KL q: (0: (0:))l (0: B.ts)) [log 10) 1o) (m:)]] (28)
From the regression theorem (Pati et al. Cnne for any distribution s(0) with high probability. We apply this result to the last term of (28).
2018 (Theorem 3.1 therein), it is known that (b)(() [n()]] <
Summing it over i = 1,..., N leads to:
B (0.) [d (Po, P(] s
E (m -=
n
N
Eg(o) (L NCnd
(29)
By dividing both sides by N and sending N + 0o, we have:
n E, (0,) ld (Po. (Po. P']] < ENT B -p (D.), ((.0))))))
(Poo
Cne (30)
EL)BO g) log (iD;))
As indicated, the right hand side is composed of -ELBO(g) (the objective function of (6). the constant log (D(D)). and the complexity term Cne
The next step is to plug in the optimal q² to have a meaningful upper bound. To this end, we introduce/define (6.) and ((Φ) as follows:
(0.) = N(O, 00,00,0000 I,,), ((0) = arg min E 1 (()) KL (L(
where
(31)
(())
07 = arg min max [1fo(z) - (x)()) 13, of
8n A,
(32)
dEB 2
A log(3S, M) (2BM) (S5+1+
BM BM- (2BM
(2BM - 1)2
(33)
16

---

Published as a conference paper at ICLR 2024
Since (6.)), 1==1 (Ø)) is the minimiser of the negative ELBO (6) clearly have
=LBBO(9") I (9) We plug q" into (30) and apply this ELBO inequality to have:
n E Bug (0,) /d [d P') I E E (o.
p Po. (D)) Pt (D))) +
Bu-TERe) KL (q: (e)) p (()))))) Cne.
(34)
The second term of the right hand side of (34 is constant (independent of n) and denoted by Ć. For the first term of the right hand side, we use the following fact from the proof of Lemma 4.1 in (Bai al. 2020), which says that with high probability,
((Po) PD())) C'n(rn + x7).
(35)
for some constant C' 0. Using this bound, (34) can be written as follows:
71 E T (0.) (d P')] s Ò C'n + EuT((ii) + One
(36)
The proof completes by dividing both sides by n.
B DETAILED DERIVATIONS
B.1 ELBO DERIVATION FOR
We derive the upper bound of the negative marginal log- likkliihkoood for our Bayesian FSL model, that is, deriving (5) in the main paper.
(( II, (0.) pDIN:))
(L 01:N) )aa)]l ll p( (r:N(D::N)) = Ey log p()) II. ((0 IL.,00,).))
(37)
= =oggp(Di:)) +
N
(l(((())P((0))
(B.(o.)l- loggooopp + By( [KL((
(38)
=:C(L)
Since KL divergence is non-neg non-negative, -L(L) must be lower bound of the data log- ligeliiel log p( DI.N), rendering C(L) upper bound of log D.NN)).
B.2 DERIVATION FOR Ey(() (() IN 6
We will derive the full closed-form formula for Eg(o) [L (0.) (0.)) [p())))) which not only leads to equivalence between 7 and (8) but also used in deriving (TI). In a nutshell, the formula that we will prove is follows:
Eo( (0:( [o))]
(39)
- d2og(22) + log Vil
+ d
101
VV
v/d
2
To
no (m,-mo) (m, + TT(VVo
where va(a) = Σド wba + (1 - j)/2) is the multivariate digamma function, and vb(+) is the
digamma function.
We begin with the definition of the KL divergence,
Eg(e)
KL g: (,) (0 [o))] (H() + (o))) p(0,()))
(40)
where the first term is the negative entropy which admits a closed form due to Gaussian (.,))) N( m, Vi),
d log(( (mTe log [V].
-I(())(()))
(41)
17

---

Published as a conference paper at ICLR 2024
Next expand the second term of 40) using p(0=(o) -N((;; ps, Σ) as follows:
E 1ite).t00ll-oop0..110 = 2
(() log ]0]
+
2 (00 - (1) T (0, [1) + log(2
T2
(42)
Using the following facts from (Bishop 2006; Braun & McAuliffe 2008):
EIw(r, log |از = ldlog 2 + log // - va(l/2)
(43)
WI E uv
(44)
we can derive the two terms Ti and T2 follows (Recall: (()) = NN mo. - Vo. no)):
1 { dlog 2 + log |Vo 4// (%) (%)))
(Ti =) E ((0) [lo Σ]
(45)
2
(T2
2 E (00)) (0: 14) - - mn] - (ecosa Tr (0, p))s- --11)
1
(46)
2
= Tr (Eut)) Bare [(0: - -)(00-1))))
- -w) -2)
(47)
(Roto) (mam) umm maje T mu +V:)
Tr
(48)
Tr (IW(EE:O.RO) BNNimmooto [mmm] pmm mip²" + uu T vo]=~11
(49)
= Tr
(IIWEE:VONO) (mmm) - -mom? - -mmo To + momo + 1 + vi)x-1)
(50)
= - Tn
Tr
++ ((m; - mo) mo) + Vi)noVo
(51)
1/d d to
+ no mmm mo) Tv (m; mo) + ToT
(52)
Combining all the above results yields the formula (39).
B.3 DERIVATION FOR 8 FROM 7
Using the result (39 we can easily show that the local episodic optimisation in the main paper (53) below) reduces to (8) (54) below).
min Li
(0.:L)) [4((0)] + Egco) (0)40(0;;
(53)
min Bx - log VV + m..V
70 (m= m)) [Vo - mo) + no Tr (V.VV (54)
2
Recall that the optimisation is with respect L, = (m4, Vi) with Lo
{mo, Vo. 40 no fixed.
Plugging (39 into (3) and removing the terms other than (m, Vj) leads to (4
B.4 DERIVATION FOR
10
For the quadratic approximation of 1, (0,) - log (D.10)) ~
+ const., here
we show that the minimiser of (8) (54) above) can be obtained by the closed-form formula II0 (55) below).
m (Lo) = (A, no Vo (Am, noVo mo),
(L. (A, no noVo
(55)
By replacing (,,),) by the quadratic approximation, the expected loss term in 8 or 54 can be written follows:
Ex( v.)l4(4))) ~ Ex(O, v4) (0. - m ( m + const.
(56)
2 Tr(8 [A,)) m. A,m, - m. A,m, + m, A,m. + const.
(57)
18

---

Published as a conference paper at ICLR 2024
Algorithm 2 Meta-test prediction algorithm.
Input:
Test support
and learned g(;; Lo) where Lo {mo, Vo, no). My = number test-time variational inference steps.
My = number of test-time model samples.
Compute the mode o² (n mo, ======== (no
Initialise (m,, V) with (,²."").
for I...., .My do
Take gradient descent update for (ma, V) with the objective in 664 end for Sample g(*) (0,; m, V) for I,..., Ms. Output: Samplc-averaged predictive distribution, la"" 1 D', D1:00) ss E.n ply" 0(0)).
[r((V + m A,m; m. A,m; m. A,m: + m; A.ms)
const.
(58)
2 (T((V; + (m; m²) A, - m,) + const.
(59)
After plugging this back to (54). take the derivatives of the objective with respect to m, and V; and set them to 0:
mm. (+) = A:(m; - m + r4o Vo-aona (m, mma) = 0
(60)
Vv(() { V/ + no Vo
(61)
The solution becomes 10 or
55
B.5 DERIVATION FOR
It is quite straightforward that by plugging TT0)
(55) and also 39 in (6 have final
optimisation problem (11) in the main paper. It is reiterated below:
min BiTT [f((Lo) + 20i(Lo) + 21o
s.t. fi(Lo) = EmNiOon 1. m (Lo) + V.- (Lo)
Lo
VVo
(LooVV))) + (mo) 2/V/== via 110
gi(Lo) log [V,
(62)
B.6 FORMULAS FOR TEST-TIME ELBO OPTIMISATION 13
63 below). For the test-time variational density (B) (6; m. V) to approximate p(0 )-) for test support data D and learned
We provide formulas for the test-time ELBO in
13
o (1" mm, Σx = /o (no + 22)),, we had
min E.(o) log log1( + (L((())l)((0))))
(63)
Using the closed-form Gaussian KL divergence and the reparametrised sampling trick, we can express as:
{
(EaNviOLn (loyoo.n-loop(D"1i + v2/22)] - log [V| +
min m.
+ +(m ma) TV (m ma)) (Tr(V-- dd+
(64)
Also, our meta-test prediction algorithm is summarised pseudo code in Alg.
C ToY EXPERIMENT: WHY HIERARCHICAL BAYESIAN MODEL? (A DETAILED VERSION)
To demonstrate why our hierarchical Bayesian modelling is effective for few-shot meta learning problems, we devise simple toy synthetic experiment as proof of concept.
19

---

Published as a conference paper at ICLR 2024
(a) Model I
(b) Model II
(c) Model III (Ours)
Figure 3: (Toy experiment) Graphical models for the three competing Bayesian models. Here
= [0, []] is the the concatenation of the weight and intercept random variables.
The problem that we consider is basically a multi-task (Bayesian) linear regression problem. First, generate the multi-cask/episodic data by the following process: The input-output data pairs (xr E R2 & R) are generated from a linear model with shared normal vector and episode- specific intercepts. More specifically, let Wshared R2 be the episode-agnostic shared weight vector, and {b1, b2, b3} (b, E R) be the three candidate intercepts among which each episode can take one randomly. The actual values of the true parameters are: Wshared (-5.4282, (.98677 by 1.4149, b2 = - by = 22.8900 The true data distribution T; for the episode i (= 1, 2, N) defined by the following linear process:
1. Sample the intercept ID for this episode, j(i) {1, 21, uniformly at random.
2. Repeat the following to collect (x, v) pairs (so that (ar, y) ~ TT):
y = (Wshared + c x + by(0) + Eyy
(65)
where NNO, I), Gr NO 10-110 and cy ~ ((0 10-1).
In this way we ensure that the resulting episodes are not only related to one another through the shared weight vector Wshared, but they are differentiated by potentially different intercepts. We generate 50 episodes where N = 40 episodes each training episode i, we have three (z, y) samples as an episodic training set D; (all available to training algorithm, where we make distinction between support and query sets). At test time, we take three samples as a (labeled) support set D, (* denotes each of the 10 test episodes), and test performance is measured on about 50 unseen samples from the same distribution T...
used for training and the rest 10 episodes serve test data. For
Three competing Bayesian methods. We consider three Bayesian models which exhibit different levels/degrees of flexibility and regularisation. The first one is highly flexible by modeling each individual episode independently with its own parameters, thus with lack of regularisation. The second case is conventional (non-bierarchical) Bayesian model where we consider single parameter set shared across episodes, thus too much regularised with lack of flexibility. At last, our hierarchical Bayesian model imposes balanced flexibility and regularisation by introducing the higher-level variable e that captures the inter-episode shared information.
Model I: This model has episode-wise parameters while they are all independent with minimal regularisation. More formally,
= x B
(66)
where 8:) is the parameters for the episode i. We place the prior p(O., 3.) N(a, 10-40) with the model parameter l shared over episodes. The training amounts to learning the parameter μ E 3 via marginal likelihood maximisation (i.e., max, log p(D,..(..... Drlp)). At test time we do inference p(0., B.| DI:N) which boils down p(0., D.) due the cross-episode independence assumption. The graphical model diagram for the model shown in Fig. (a).
2. Model I: Unlike introducing episode-specific variables, this model has a single set of variables shared across all episodes. More specifically,
y - x. + +
(67)
where 30 is the episode-agnostic parameters, endowed with the prior p(0, 3) N N(,, 10 10-41). Due to this parameter sharing. this model is highly regularised but at the expense of significantly reduced flexibility. Once y is trained, the inference at test time is done by p(0, D., Di.v) which is not simplified further and has to take into account all training and test data. The graphical model diagram in Fig. (b).
20

---

Published as a conference paper at ICLR 2024
Enoes Modell 2.55
Node
Emers Model1 LA5, Modid
-2.
Modell
-30
Moow
Model (Durs)
b.
Truc
Model
Model
-35
Model
Model
Model (Dun))
Model 11 (Ount)
Interoegt DO 111.2.00 apinatiies
The intercopt De (12222)) opicndie
The merrept (2-22333 eptiodes
(a) Weight dim-1
(b) Weight dim-2
(c) Intercept
Figure 4: Toy experiments. Visualisation of the learned posterior means compared to the true values (blue-circled). (a) weight dim- 1 (0[0] vs. (sharee(O)), (b) weight dim-2 (0[1] vs. Wahared [1] and (c) intercept (3 vs. bz(z)). In each plot, the X-axis shows the indices of the true intercepts sampled, that is, j(+) E {1, 2, 3), for 10 test episodes. In the titles we also report the distances (errors) between the true values and the posterior means for the three methods, averaged over 10 episodes.
3. Model III (Ours): This is our hierarchical model where each episode has its own parameters (like Model I) but there is globally governing variable & to regularise the episode-wise parameters. That is. we have the same regression form as (66) with episode- wise (0,, B.) parameters, but our prior distributions are defined hierarchically as:
(68) where (m, V) is the model parameters to be learned. At test time, we infer p(0., B. D., DI.N). however, unlike the Model II case, each training data D; and the test data D. do not have equal, symmetric contribution. Note that the training data affect the posterior indirectly only through the higher-level variable follows:
p(0)= (m, V), p(O,, 8 () = NV(é, 10 *1),
p(0., B.|D., DI:N) / p(0_, B.lo, D.) p( Dz, D1:N) dg.
(69)
That is, while our model is as flexible as Model due to the episode-wise parameters, their impacts on the test prediction are controlleddreggulrisse in very sensible manner. The graphical model diagram in Fig. c)..
Note that all the posterior inferences of the above three models can be done in closed forms due to the Linear-Gaussial properties. The details of the posterior distributions and derivations as well as the model training can be found in Sec. C.I
Results. For the 10 test episodes, we obtain the posterior means of the weights and intercept parameters E00, B3 D., D1:N] ] for the three models via the closed-form solutions as detailed in Sec. C.I I.I The test support set size |D.| = 3. With these posterior means, we predict the outputs of about 50 unseen test inputs. The mean absolute errors (MAE) of the three models averaged over test episodes are: Model I 2.87, Model II 3.13. and Model III (ours) 1.28, clearly showing the superiority of our model to other competing methods. In Fig, we also visualise the posterior means to check how much they deviate from the true weights and intercept, namely the difference between the posterior mean of (8, () and the true (Watared b)(.)). First, we see that Model II's posterior means rarely change over the test episodes, in other words. the impact of test support data D. is diminished. This behavior is expected since the model imposes too much regularisation with little flexibility, and the test prediction is dominated by the mean model obtained from training data DI.N- Secondly, Model I exhibits highly sensitive predictions over the test episodes, which mainly originates from little regularisation. In other words, the posterior is affected too sensitively by the current episode's support data, thus being vulnerable to overfitting especially when the support data size is small, typical in the few-shot learning. The model failed to capture useful shared information, in this case Wahared, from diverse training episodes. On the other hand, our Model III takes the balance between the above two extremes, imposing proper amount of regularisation and endowing adequate flexibility. Our posterior estimation best extracts the shared episode-agnostic information (the weight parameters fluctuate less over the test episodes), and at the same time, captures the ppisode-specific features the most accurately (the estimated intercepts are aligned well with the true values).
C.1 DERIVATIONS FOR TRAINING AND POSTERIORS IN TOY EXPERIMENTS
For the convenience in notation, we let θ = [0.3] E R be the concatenated random variables (subscripts can be applied accordingly). The training data D; for each episode consists of the inputs
21

---

Published as a conference paper at ICLR 2024
with the constant 1 appended, denoted X, R and the outputs Y, R". With this notation, the linear regression model can be succinctly written as Y; = ex,,, Similarly, the test-time support data D, is decomposed into (X., Y.). The noise standard deviation is denoted by 10- ?
C.1.1 MODEL
Model I (the episode-wise independent model) can be formally written as:
p(0,) = 14, I²I,, p(D, = NY;; X10,, V..
(70)
The posterior 1 ..... OI. Dy) is fully factorised over and we can deal with individual terms p(O; D.) where
((O)DD) & p(O)) p(D,)O,) = NN(O; XYYY; o²1).
(71)
Due to the product-of-Gausians form, we have the closed-form posterior:
1 o2 ( (I+ by =
1
(.) = NV(O;; A, A,") where A;
02 (u x, Yi). (72)
The training amounts to maximising the data (log-)likelihoon) max, log pl (D,,.... Dx(()) where the objective is fully decomposed over as follows:
N
log DN(P) Σ
log p(D)) where
(73)
1=1
(D:) = logp(00; O2 D:) - log ((0 D,) (for any O,)
(74)
=logp(O) O.) + log θ:) logr(O;(D.) (for any θ:).
(75)
Using the Gaussian posterior form 72 we can easily evaluate log logp(D....... Nx (a) and also optimise it with respect J.. At test time, the posterior of θ, given all the training data and the test support data, p(e, Di,..., Dv) equals (O.]D.) due to the independence assumption, and admits the same Gaussian form as 722) with the test support data (X., Y.) in the place of (X;, Y.).
C.1.2
MODEL II
Model II (the shared model across episodes) can be formally written as:
p(0) = 12, p(D,()) = NYY; 020.
(76)
The posterior .., D2) can be derived as follows:
N
DI,... Dx) x ((O) II (D.0)) =
(77)
Again, due the pooduct-of-Gausiian form, we have the elosed-form posterior:
p(O)D,..... Dx) N(O; 'b A-1) where
1 1(1 [ x b- o² 1++ ( = σ²
x, x,YY,
(78)
Likewise, the training amounts to maximising the data (log))likel,hod,, max,, log p( D Dwlp)
where the objective becomes:
log p( DD...., DNlp) log (O,, DI..... D2)
Dx) (for any Q)
(79)
N
=log(()) + Σ logg(D,(() - log DO...,) Dx) (for any θ). (80)
Again, using the Gaussian posterior form 78 we can easily evaluate logp( (D.............................. Dx (x) and also
optimise it with respect μ. At test time, the posterior of O given all the training data and the test support data, p(O, D., .,...., Dv) admits a Gaussian form, derived similarly as (78) with the test support data statistics X_ X, and X Y, additionally added the training statistics.
22

---

Published as a conference paper at ICLR 2024
C.1.3 MODEL III (OurS)
Our Model III (the hierarchical Bayesian model) can be formally written as:
p()) = (d; V):
(81)
p(O,jo) 0f)1,
- -(Xx Vi.
(82)
The posterior p(o ...... DN) can be derived as follows:
p( p(. Dx) p(Ø) / p(O,,.....
p(D1,
d6
(83)
= -(()) /
II p(0,1o) (...))))((D.1.)) d0I.N
(84)
i=1
N
=
II
(()))))) (D.(O.) dO:
(85)
N
/ ((O00,,01) -NY); X(0, de;
= mo V)
II
(86)
N
=N V)
II NYY; 0 ((X,),
(87)
where we use the property of the product of Gaussians for the derivation from 886 to 87 Now, in 87), due to the Ionduct-of-Gaussian form, we have the closed-form posterior:
P Dx) = (o; A A1 where
N
1 N
A + Σ 1=1
Σ x² x (x -,, b= V-1 m - x² i=1
(X, + (88)
The training amounts to maximising the data (log-)likelihoode maxm, log pl Dı.... Dv V) where the objective becomes:
log p(g DD........
DN mm, V) logp(d, Dı,.... D2) logp(ø) DN) (for any Φ) (89)
log (()) + log p( Di.... Dxlo) (og DN) (for any Φ), (90)
and the é-conditioned data log- le-likkelihood log pl (D,,.... DNl0) can be derived as follows:
log logp(Dr,... D....., DNld) log /
N
II
p(O.)))-p((Ditt (D..).)) dOI.
(91)
i=1
NN
log
II
((O,)o) O(), do;
(92)
==1
N
log
p(O,jo)) (116))p((D0)) p(D.)0)) d0,
(93)
==1
(Y;; o² xxx,))
(94)
So we can easily evaluate logp(D),..... Dv/m, V) and also optimise it with respect (m, V). At test time, the posterior of O, given all the training data and the test support data, p(O. Di...., DN) can be derived as follows. We start with:
DN) p(O,oo, D.) p(o
DI...., DN) do,
(95)
and the first term in the integration, p(O. (0b D.). since it is proportional to the product of two Gaussians p(O.ló. 66 D.) x p(D.(O.) p(e. ()) = X.O., g2T) (N(O:; d o² it admits Gaussian (from the Gaussian properties).
10,, D.) (N( Cp E),, where
(96)
23

---

Published as a conference paper at ICLR 2024
C=I- KY,, E=o²C, K
(97)
The second term of (95) becomes a Gaussian following the derivation similar to 88 with the test
support data X, and Y. included. Consequently we let p(o) Di...., Dx) = No A, A, At last, (95) is the marginalisation of the product of two Gaussians, which admits the following closed
form:
p(O,)D,, Di,..., Px) = NVO,,; Cd. +))
(98)
D IMPLEMENTATION DETAILS AND EXPERIMENTAL SETTINGS
We implement our NIW-Meta using PyTorch (Paszke et al. 2017) and the Higher (Grefenstette et al. 2019 library. The latter makes the implementation of the backpropagation through the functional network weights in PyTorch modules very easy, Real codes for the synthetic SineLine regression dataset and the large-scale VIT are also attached in the Supplement Material to help understanding of our algorithm. For all few-shot classification experiments, we use the ProtoNet-like parameter- free NCC head in our NIW-Meta. Some important implementation details on the SGLD iterations for quadratic approximation of the local episodic optimisation include: we have either steps without burn-in (for large-scale backbones ViT) or 5 steps with 2 burn-in steps (for smaller backbones ConvNet, ResNet-18, 18, and CNP). Before starting SGLD iterations, the network is initialised with the current model parameters mg. For reliable variance estimation of A,, small regulariser is added to the diagonal entries of the variances.
For the standard benchmarks with ConvNet/ResNet backbones, we follow the standard protocols of (Wang et al.] 2019; Mangla et al., 2020) Zhang et al., (021): With 64/16/20 and 391/97/160 train/validation/tesl class splits for minilmageeet and tieredImageNet datasets, respectively, the images are resized to 84 pixels. We initialise the mo parameters from the pretrained models: checkpoints from (Wang al. (019) for Conv-4 and ResNet- 18 and checkpoints from (Mangla et al., 2020) for WRN-28-10. 10. With the stochastic gradient descent (SGD) optimizer, we set momentum 0.9, weight decay 0,0001, and initial learning rate 0.01 for minilmageNet and 0.001 for tieredImageNet. We have learning rate schedule by reducing the learning rate by the factor of 0. at epoch 70.
For the large-scale ViT backbones, we utilise the code base from (Hu et al., 2022). We use the self-supervised pretrained checkpoints from (Caron et al., (221)) to initialise the mo parameters. The CIFAR-FS dataset is formed by splitting the original CIFAR- T00 into 64/16/20 train/validationntest classes. For training, we run 100 epochs, each epoch comprised of 2000 episodes. We follow the same warm-up plus cosine annealing learning rate scheduling as (Hu ct al. 2022). For test evaluation, we have 600 episodes from the test splits.
For the few-shot regression experiments with ShapeNet datasets, we basically follow all experimental settings and CNP/ANP network architectures from (Gao al. 2022). For instance, in the ShapeNet- ID dataset, we run our algorithm for 500K iterations with learning rate 10-4 where each batch iteration consists of 10 episodes. The CNP backbone, for instance, in the Distractor dataset case, has ResNet image encoder and a linear target encoder, where the concatenated instance-wise embeddings then go through three-layer fully connected network followed by max pooling. The decoder has similar architecture and converts the support set embedding and query image into a target label. For the conv-net plus ridge-regression head backbone (C+R) tested for our method, the conv-net feature extractors are formed by taking the encoder parts of the CNP architectures in (Gao et al., (022) while discarding the pooling operations and decoders. Also the regressinn L.2 regularisation coefficient is set to À = 1.0 for all datasets.
E COMPUTATIONAL COMPLEXITY, RUNNING TIME AND MEMORY FOOTPRINT
Although we have introduced principled Bayesian model/frammework for FSL with solid theoretical support, the extra steps introduced in our training/test algorithms appear to be more complicated than simple feed-forward workflows (e.g., ProtoNet (Snell et al.) 2017)). To this end, we have analysed the time complexity of the proposed algorithm contrasted with ProtoNet (Snell et al., 2017). For fair comparison, our approach adopts the same NCC head on top of the feature space as ProtoNet.
24

---

Published as a conference paper at ICLR 2024
Table 8: (Per-episode) Time complexity of our NIW-Meta vs. ProtoNet. We denote by FD and Bp the forward-pass and backpropagation times with data D Support or Query. In our algorithm. My My. and My indicate the numbers of SGLD iterations, test-time variational inference steps for (13 or 66 64 and the number of test-time model samples ((s), respectively. The costs required for reparametrised sampling in model space and regulariser computation in or 622
are denoted by
((d) where number of backbone parameters.
Training time
Test time
(F' - F. Bq) (1)++) + O(d)
(Fs + Bg) My
NIW-Meta (Ours)
(Fs FQ) Ms + O(d)
ProtoNet
Fi's Fo+
F's + FQQ
Table Effect of the finite number of episodes (N) on the generalization error gap in terms of the sample complexity gap log(2/nN (6) in 21 Here 8
N (#episodes)
n (#shots x #ways)
Sample complexity error gap (1)
10a
1x5 / 5x5
2.4x
5.0x 10-4
104
1x5 5x5
3.0x10 / 5.5 10-5
10a
1x5 5x5
2.8x10 ~ /600 10 -6
The computational complexity is summarised in Table Despite seemingly increased complexity in the training/test algorithms, our method incurs only constant-factor overhead compared to the minimal-cost ProtoNet.
As we claimed in the main paper, one of the main drawbacks of MAML (Finn et al. 2017) is the computational overhead to keep track of a large computational graph for inner gradient descent steps. Unlike MAML, our NIW-Meta has much more efficient episodic optimisation strategy, i.e., our local episodic optimisation only computes the (constant) firs//secon--rrdr moment statistics of the episodic loss function without storing the full optimisation trace.
To verify this, we measure and compare the memory footprints and running times of MAMI. and NIW- Meta two real-world classfficationn/eeresssoo datasets: minilmageNee 1-shot with the ResNet-18 backbone and ShapeNet-ID ID with the ConvNet backbone. The results in Fig. 5 show that NIW-Meta has far lower memory requirement than MAML (even smaller than l-inner-step MAML) while MAML suffers from heavy use of memory space, nearly linearly increasing as the number of inner steps. The running times of our NIW-Meta are not prohibitively larger compared to MAML where the main computational bottleneck is the SGLD iterations for quadratic approximation of the local episodic optimisation. We tested two scenarios with the number of SGLD iterations 2 and 5, and have nearly the same (or even better) training speed as the l-inner-step MAML.
F TRAINING STABILITY AND IMPACT OF NUMBER OF TRAINING EPISODES
In our theoretical analysis of the generalisation error (Sec. in the main paper and our proofs in
Sec. A). we regard the number of training episodes N as infinity. In practice, N is finite, but large enough (N~ 100K in typical FSI.), and we simply take it as infinite N for mathematical convenience (c.g., to have the first KL term in (5) vanish; reduction to the task population mean from (29) to
(30)). To see the effect of finite N on the generalization performance, we exemplify several typical
N values and corresponding generalization (sample complexity) error gaps (21) in Table We see that even for relatively small N, error gaps are small/negligible.
In addition, investigate the impact of the number of training episodes N on training stability. We illustrate it in Fig. [6] which shows that our method works stably well even for a small number of initial episodes, convergence being fast ProtoNet with far better generalization performance.
25

---

Published as a conference paper at ICLR 2024
miniimage
ShapeNet-1D
GRU mntrrrr footprint (MB)
6000
NIW-Meta (Ours)
GPU memorry footprint (MB)
1600
NIW Meta (Ours)
MAML
0300
1400
5000
1200
+500
:0000
800
4000
800
3500
Number steps
steps
(a) GPU memory footprints
minllmageNet
ShapeNet-1D
Per-episodc training time (mseec)
180
NiW-Meta (#5GLD=2)
Per-epiieede training time (rrvec)
NiW-Meta (#SGLD=2)
160
NIV-Meta (05CLD=5)
20
(#SGLD=5)
MAML
MAML
140
300
80
3
Number inner stops in MAML
Number of inner stops MAML
(b) Per-episode training times
Figure 5: Computational complexity of MAML (Finn et al. 2017) and NIW-Meta. (a) GPU memory footprints (in MB) for a single batch. (b) Per-episode training times (in milliseconds). For NIW-Meta models, the time for the number of burn-in steps (2 steps in this case) also included. That is, NIW-Meta ##SGLD 2) runs 2 + 2 SGLD iterations, and NIW-Meta (# #SGLD=5) runs inner iterations. We use the ResNet-18 backbone
5 + 2, respectively, compared MAML with for minilmageNet in 1-shot classification and the ConvNet backbone for ShapeNet- 1D regression (10 episodes per batch).
Training losses training episodes
Validation performance training episodes
76
(NIY Meta) loss
1.05
030225
(NW Meta) Train CE loss
1.2
(ProtoNet) Train CE loss
1.00
0,5000
1.0
0.95
0.6975
W-Metal
70
Valicition
(Pro Val loss
(Ni--Meta)
0.85
(N W-Meta) Val acc
(ProcoNet) Val acc
66
0.4225
0,4900
0.6
0.75
8.3375
02
20
60
80
20
40
80
Train epoch (1 epoch 2000 episodes)
Train epach (1 epoch 2000 episoces)
Figure 6: CIFAR I-shot learning with ViT (DINO/s). (Left) Training losses vs. training episodes (2000 episodes per epoch). We plot the training loss of our NIW-Meta (i.e., (TI) in blue, and super- impose the training cross-entropy loss in red, where the latter is comparable to ProtoNet's training CE loss in magenta. We see that NIW-Meta training is pretty stable, where the convergence speed is as fast as ProtoNet. (Right) Performance on the validation set as training episodes increases. The validation losses of our NIW-Meta (blue) and ProtoNet (cyan) are comparable, while also compare the validation accuracy of our NIW-Meta (red) against ProtoNet (magenta).
26

---

Published as a conference paper at ICLR 2024
Table 10: Predictive log-marginal--ikelihooo scores (LMLHHD on the Sine-Line dataset. The higher the better uncertainty quantification and generalisation.
Context size
2
12
16
Volpp et al. 2023
-18.38
-15.98
-13.69
-12.74
-11.75
-11.14
-10.23
NIW-Meta (Ours)
-17.02
-14.74
-12.60
-8.14
-3.51
-1.55
-1.33
G ADDITIONAL DISCUSSIONS
G.1 COMPARISON TO BAYESIAN NEURAL PROCESSES
An interesting question is what are the relative merits of a complete Bayesian treatment but with restricted Gaussian forms as in our model, versus a shallow Bayesian but GMM-like rich variational forms such as (Volpp et al. 2023). We do not know which absolutely better than the other. Further theoretical and empirical study needs to be carried out in this regard. But we have some experimental evidence as demonstrated in our paper, the comparison with MetaQDA (Zhang et al. 2021) shallow Bayesian approach that only places prior distribution on the model head parts while freezing the feature extractor (Table 3 5 As shown, our complete Bayesian treatment outperformed it, in both test generalisation performance and uncertainty calibration. This is one supporting evidence of why complete Bayesian treatment could be more promising than a shallow Bayesian treatment.
For the experimental demonstration, we have done additional experiments the Sine-Line dataset to report the predictive marginal test likelihood score, which is directly comparable to the shallow Bayesian embedding models, esp., the Bayesian Neural Process model of (Volpp et al., 2023). The predictive Nog-marginal-iikelioos
scores (LMLHD) are shown in Table I0 We try to match the experimental settings from (Volpp et al. 2023) that the results are comparable. More specifically, we test on 256 tasks, each of which consists of 64 samples with varying number of context samples (1, 2, 4, 6, 8. 12 and 16). The number of posterior samples used to compute/approximate the predictive marginal likelihood is 1024. As shown, our NIW-Meta has higher scores, especially for larger context size, implying that capturing uncertainty in full model parameters important for generalisation capability.
G.2 JUSTIFICATION OF MODEL AND ALGORITHM CHOICES
Motivation for distributional estimate (()). If o were directly linked to the observed data D, 's in our graphical model Fig. [ then & can tend to be determined deterministically. as the number of data N becomes large. However, is linked to latent variables 0,'s so the belief on & also needs to capture and accumulate the uncertainty in 0 's, which amounts to marginalising out the 8; variables. So, may not not be appropriate treat the posterior for delta function (0 uncertainty), and it is better to follow the Bayesian inference principle, i.e., let the posterior be computed from the observed evidence.
Why distributional estimate (()) if we only use the mode of (()) at meta-test time. Using the mode of (()) only for practical convenience and simplicity. Although we used the mode, the distributional form ((Φ) would take into account uncertainty in its optimisation, and thus lead to different solution from the deterministic one.
Why not the same inference approach for meta testing as meta training. For meta testing,
we may attempt to solve the optimisation problem similar (7) or (8) for meta training with Lo fixed, which also results in the same meta test solution as our derivation in Sec. 3.2 This can also be verified by inspecting the similarity between (13) and 7 or (8). However, this approach requires that
Lo be fixed at the trained value, which do not know for sure in the pure optimisation perspective. In Sec. 3.2 we aimed to derive the meta-test optimisation problem from the Bayesian perspective from the outset, which offers us reasonable justification for why Lo be fixed.
Quality of the quadratic approximation. In 9.. we have made the quadratic approximation for the negative likelihood, llog (() } (0 (0-m;) (0 -7..) const. To see the quality of this approximation, we empirically evaluated the true value (the left hand side of the approximation) and the quadratic approximation (the right hand side) the mageem 5-shot dataset with
27

---

Published as a conference paper at ICLR 2024
the ResNet--18 backbone. We take random 10 perturbations of 0 around the mode/mean m, with perturbation radius 0.1. The relative error the quadratic approximation is 0.0050 + 0.0004. This shows that the negative log-liielihood is well approximated by our quadratic function in the vicinity of the mode/mean.
G.3 LIMITATIONS AND FUTURE WORKS
Limitations. 1) Our NIW-Meta introduces some extra hyperparameters (e.g,, the number of
SGLD iterations, the number of burn-in steps). These are currently estimated empirically, but more rigorous study on how select them automatically needs empirically verified that our quadratic episodic loss optimisation is effective, more theoretical analysis on the quality of this approximation as well as its impact on the final results, needs to be done.
addressed. 2) Although
Future works. We have quite an extensive evaluation on popular few-shot classification and regres- sion benchmarks. However, would like to evaluate approach on new emerging applications of few-shot learning such as efficient learning the implicit neural representations such as NeRF, e.g.. TTncikk al. 2021).
28


Reviews:
[{"review_id": "O0XdMIVrqs", "reviewer": "Reviewer_wcX1", "summary": "This paper introduces a novel approach to few-shot meta-learning using hierarchical Bayesian models. In particular, a global random variable governs the shared information across different tasks, whereas local random variables model the generative process for a particular task. Using Normal-Inverse-Wishart distributions allows for closed-form expressions of the ELBO, allowing local episodic optimization. The authors show that many seminal few-shot learning approaches can be seen as special cases of the hierarchical Bayesian framework while their proposed algorithm does not require a full computation graph. The hierarchical Bayesian model is motivated by a toy example that highlights the need for both shared parameters across tasks and task-specific parameters. In addition, the authors provide generalization error bounds using the PAC-Bayesian framework and evaluate the proposed hierarchical framework on a range of regression and classification tasks.", "questions": "- Could distributions other than NIW be considered for the prior and variational posterior? What are the limitations of using NIW? In other words, what limitations does the choice of the prior impose on the effectiveness of the approach and how does it compare to existing few-shot learning frameworks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper presents the first comprehensive treatment of the hierarchical Bayesian framework. The framework is motivated using a simple toy experiment. Both the derivations and theoretical guarantees are presented in the paper (as well as the computational complexity in the appendix), and the effectiveness of the approach is demonstrated on a number of regression and classification tasks.\n- The proposed method is architecture-independent and the authors demonstrate its effectiveness in conjunction with other meta-learning approaches such as the neural process family. In addition, seminal works on few-shot learning are shown as special cases of the hierarchical Bayesian model, demonstrating the unifying nature of this framework.", "weaknesses": "- The possible limitations of the approach / venues for future work are currently not discussed in the paper. Providing insights into the limitations and investigations left for future work could provide a better context for the paper in the few-shot learning community.\n- Though this is potentially a matter of personal preference, the flow of the paper could be improved by putting the toy experiment either before the derivations or after the theoretical analysis. Otherwise, it seems a bit out of place. Moreover, adding a proof sketch for Theorems 5.1 and 5.2 in the main paper would be helpful even though the full proofs are provided in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nbO0hBys3r", "reviewer": "Reviewer_ZG8Z", "summary": "The manuscript proposes a hierarchical Bayesian meta learning model for the few-shot learning problem, where the Normal-Inverse-Wishart (NIW) model is used to model the globally shared variables and the individual episode-wise task-specific variables. Using the NIW-Gaussian conjugacy and the stochastic gradient Langevin dynamics (SGLD), the authors propose the approximate closed-form solutions for the local posterior distributions, which makes the proposed methods scalable to large backbone networks.", "questions": "1.\tRegarding Section 5, what kind of information about the proposed method can we get from the final forms of the theorems, eq. (14) and eq. (15)?  For example, can we say anything about the choice of the hyperparameter \\Lambda or about the relationship between the quality of the approximation of the posterior distributions and the generalization error bound? \n\n2.\tRegarding Figure 5, first the figures can be misleading because the computational complexities of your method look constant with the common parameter with MAML (which is not, the parameters for only MAML) in the current presentation. NIW-Meta, more precisely SGLD, requires multiple iterations as it skips the first burn-in iterations. How about comparing the actual training times of both methods when they are optimized to show the similar performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript proposes a complete hierarchical Bayesian model for the few-shot meta-learning problem and a scalable training algorithm. \nThe manuscript provides the theoretical analysis for the proposed method.", "weaknesses": "The authors claim that the proposed method shows the improved prediction accuracy and calibration performance. \n- What aspect of the proposed method improves the prediction performance is not yet clearly stated in the text. The Bayesian treatment? It should be made more explicit and clearer in the text.\n- It seems that the experiments do not include other Bayesian meta-learning methods. The hierarchical Bayesian model is the standard way to extend existing methods in Bayesian learning. I am not an expert on meta-learning but found several publications on this extension using simple keyword search. How about this paper [Reference]? I think that not comparing the proposed method with other Bayesian meta-learning methods limits the significance of the proposed method. \n- The calibration performance is validated using only a single data set. \n\n[Reference] Zhang, Z.; Li, X.; Wang, S. “Amortized Bayesian Meta-Learning with Accelerated Gradient Descent Steps” Appl. Sci. 2023, 13, 8653. https://doi.org/10.3390/app13158653", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVgXUWCceA", "reviewer": "Reviewer_PGLR", "summary": "The paper proposes a Bayesian framework to explicitly leverage the hierarchical structure of the generative process in few-shot learning. The Normal-Inverse-Wishart model-based variational inference and quadratic approximation allow efficient learning. The theoretical results link the essential values in the approximations to the ultimate performance.", "questions": "Could the author comment on the choice of the parameters introduced by the proposed method, including the number of SGLD iterations, test-time variational inference steps for (13) or (63,64), and the number of test-time model samples?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Theoretical results on the PAC-Bayes-$\\lambda$ bound and regression analysis reveal the convergence of the proposed method, though there is no convergence rate.\n\n2. The proposed framework offers a unified interpretation for different established FSL methods.\n\n3. The toy example in section 4 clearly motivates the proposed hierarchical model.", "weaknesses": "1. There are multiple approximations, most importantly the ELBO and the quadratic approximation, in the proposed framework. Though well-motivated, the lack of empirical evolution of the effectiveness of the approximations other than the downstream performance undermines the significance of the work. Especially given that the theoretical analysis in section 5 also relies on the quality of the approximation.\n\n2. In section 7, the empirical results for some tasks only show average values rather than showing confidence intervals. And some superior performances lack statistical significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fd4QZQRWyf", "reviewer": "Reviewer_kRtC", "summary": "This paper proposes a novel hierarchical Bayesian model for the few-shot meta-learning problem. They consider episode-wise random variables to model the generation process, where these local random variables are governed by higher-level global random variables. The global variable captures information shared across episodes while controlling how much the model needs to be adapted to new episodes in a principled Bayesian manner. Prediction on a novel task can be seen as a Bayesian inference problem. They propose a Normal-Inverse-Wishart model, for which establishing the local-global relationship becomes feasible due to the approximate closed-form solutions for the local posterior distributions.", "questions": "1. From Figure 3, (c) seems can be modelled via recurrent VAEs. which were proposed before, so what are the novelties of this paper on the few-shot learning task? \n2. If possible, not only few-shot learning but other related tasks could be tested to validate the generalisation ability of the model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper actually proposes a unified framework for hierarchical Bayesian learning for few-shot problems. \n2. They interpret the model in a way that MAML, ProtoNet and Reptile are special cases of the hierarchical Bayesian learning. \n3. A detailed introduction and explanation of the methods can be seen in the paper,  also good experimental results have been obtained.", "weaknesses": "1. Please provide more background knowledge for the Normal-Inverse-Wishart model, and why this is justified in an application for this model.\n2. This model is similar to hierarchical recurrent VAE, if so, what are the advantages of this model?\n3. More general interpretation is needed (in the sense that not only in the Bayesian learning but for deep learning in general) so that the readers can benefit more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8j0B5y5o0a", "reviewer": "Reviewer_EN3o", "summary": "The paper studies the problem of few-shot learning in a model-based meta-learning setting. The authors employ a hierarchical Bayesian meta-model, variants of which have been studied extensively in the literature [1-10]. They parametrize the model by Gaussian priors for the task-specific variables and conjugate Gaussian-inverse-Wishart hyper-priors for the task-global variables, and consider various neural network based likelihood models from prior work. The authors propose a novel variational inference procedure to meta-learn the task-global posterior, which notably does not employ task-amortization in contrast to most prior work. Similarly, they determine the meta-test task-posterior using a variational approach starting from a mode of the task-global posterior. The authors include a toy experiment to demonstrate the relevance of hierarchical modeling for few-shot meta-learning. Furthermore, they evaluate their approach against several competing methods on standard benchmarks like meta-image classification, meta-regression, and pose estimation.\n\n\n-------------------------\n\n\nI increased my score from 5 to 6 after reading the author's response.", "questions": "cf. weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is mostly well-written and presents a conceptually interesting approach for inference in hierarchical Bayesian meta-models. The derivation of the inference procedures for the task-global variables at meta-train time as well as the task-specific variables at meta-test time appears to be executed well, but gets technically quite dense at times and leaves open several important points, cf. weaknesses. While the toy experiment in Sec. 4 seems to be unrelated to the novel inference procedure developed in Sec. 3, it can serve as a nice pedagogical illustration of the efficiency of the known hierarchical Bayesian modeling approach for few-shot learning.", "weaknesses": "My main concern is that central claims of the paper are not sufficiently supported by theoretical arguments nor by experimental evidence, so I encourage the authors to provide further justification:\n- The authors claim in the abstract to \"propose a novel hierarchical Bayesian model for the few-shot meta learning problem\". This is not accurate as the model and variants thereof have been studied extensively in the literature [1-10].\n- The authors claim to provide \"the first complete hierarchical Bayesian treatment of the few-shot deep learning problem\". I ask the authors to elaborate on this claim. \n  - In fact, as noted above, the model has been considered before for few-shot meta learning by a range of papers, all of which propose different parametrizations and inference approaches. The related work section mentions a few of them, but describes them as improper solutions of the Bayesian few-shot learning problem.\n  - In particular, the authors claim that a Bayesian treatment of \"only a small fraction\" of the model's weights \"considerably limits the benefits from uncertainty modeling\". The authors cite (among others) and compare to the conditional neural process [11], which is a conditional model and, thus, not a Bayesian treatment of the few-shot learning problem, as well as to the attentive neural process [12], which has been shown in [6] to  revert to a mainly deterministic approach due to the deterministic, attentive computation path. Therefore, I do not consider these methods suitable baselines to support the author's claim, i.e., to show that their approach yields improved uncertainty modeling.\n  - Unfortunately, the authors neither discuss nor compare to the Bayesian neural process versions proposed in [3] and further studied in, e.g., [4-6]. I consider such types of neural process based approaches the main competitors to the proposed inference scheme. Therefore, I ask the authors to discuss in which sense their approach is more \"complete\", and also to provide experimental evidence for their claim. In fact, performing inference over \"only a small fraction\" of the model's parameters opens the door for sophisticated variational inference methods and expressive variational approximations, which I consider a major benefit of such approaches [4,6]. In contrast, \"complete hierarchical treatments\" are typically restricted to simple Gaussian variational distributions (as also used in the proposed approach) due to the high dimensionality of the parameter space. I ask the authors to theoretically motivate and discuss, as well as to experimentally demonstrate that performing inference over all model parameters still outweighs these downsides.\n\nFurthermore, as said above, I have doubts regarding the empirical evaluation. In particular, the authors claim that their approach is superior in terms of uncertainty modeling, cf. above, but I do not consider their experimental approach suitable to support this claim:\n- Most of the experiments are classification tasks on which the authors report accuracy. I'd ask the authors to elaborate in which sense these experiments and metrics really measure uncertainty quantification. \n- I acknowledge that the authors measure calibration in Tab. 6, but still think that most of the presented experimental evidence is unsuitable to support the central claims of the paper.\n- Therefore, I encourage the authors to provide more experimental evidence for their claims. I am have no strong opinion on what experiments shall be presented, as long as they are suitable for measuring the quality of predictive uncertainty quantification, e.g.,  \n  - provide results also in terms of predictive log marginal likelihood [2,4-6,10],\n  - provide calibration measurements for further experiments, possibly also in terms of reliability diagrams [8],\n  - provide further experiments that better measure uncertainty quantification, e.g., \n    - further probabilistic regression experiments (such as the presented line-sine),\n    - experiments that require well-calibrated uncertainties for exploration-exploitation trade-off, e.g., Bayesian optimization or Bandits as in [3,6,8].\n\nLastly, I ask the authors to elaborate on some more technical issues as well as on some algorithmic choices of their proposed inference scheme that did not become fully clear to me:\n- I'm a bit confused about the inference procedure. In Sec. 2 the authors describe the standard meta-learning setting which considers a \"large number of episodes\". By construction, the task-global parameters $\\phi$ are informed by all this data, which is why related work often argues that $\\phi$ is determined with good accuracy, which allows empirical Bayes estimates, e.g., [2-6]. I'd ask the authors to elaborate on the following:\n  - What is the theoretical motivation for distributional estimate $q(\\phi)$ of the posterior of $\\phi$?\n  - Furthermore, why do we need a distributional estimate $q(\\phi)$ if at meta-test time we only use the mode of $q(\\phi)$?\n  - Along the same lines, I wonder why the authors do not use the same inference approach for the $L_*$ at meta-test time (for a test episode indexed by \\*) as they use during meta training, i.e., why don't they use the (now fixed) $L_0$ obtained from meta-training and solve Eq. (8) for $m_*$, $V_*$?\n- I'd ask the authors to add some more details on whether the two-stage optimization approach of (i) reducing the optimization problem over $(L_{1:N}, L_0)$ to one over $L_0$ alone by solving for $L_i^*$ for fixed $L_0$ and (ii) solving for $L_0$ is guaranteed to find the same solution as the original joint optimization problem.\n- I'd be interested in some more architectural details about how the combination of the authors proposed approach with set-based backbones like CNP works, cf. end of Sec. 3. \n\nFurther assorted issues:\n- Not all results come with error bounds. How were the error bounds of the presented results computed?\n- The boldfaced results do not always seem to be better within error bounds. What is the author's \"protocol\" to boldface results? \n- Table 3/4 captions do not say that classification accuracy is reported.\n\nTo summarize: I consider the proposed inference approach an interesting and technically sound contribution that is relevant for the community. Nevertheless, I recommend to reject the paper in its current form, because I find that central claims the authors make require further discussion and experimental evidence. I'm looking forward to the author's response and I'm willing to increase my score after the rebuttal period in case my concerns are adressed.\n\n[1] Bakker & Heskes, \"Task Clustering and Gating for Bayesian Multitask Learning\", JMLR 2003\n\n[2] Gordon et al., \"Meta-Learning Probabilistic Inference for Prediction\", ICLR 2019\n\n[3] Garnelo et al., \"Neural Processes\", ICML Workshop 2018\n\n[4] Volpp et al., \"Bayesian Context Aggregation for Neural Processes\", ICLR 2021\n\n[5] Wang & van Hoof, \"Double Stochastic Variational Inference for Neural Processes with Hierarchical\n\nLatent Variables\", ICML 2020\n\n[6] Volpp et al., \"Accurate Bayesian Meta-Learning by Accurate Task Posterior Inference\", ICLR 2023\n\n[7] Amit & Meir, \"Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory\", ICML 2018\n\n[8] Ravi & Beatson, \"Amortized Bayesian Meta-Learning\", ICLR 2019\n\n[9] Grant et al., \"Recasting Gradient-Based Meta-Learning as Hierarchical Bayes\", ICLR 2018\n\n[10] Finn et al., \"Probabilistic Model-Agnostic Meta-Learning\", NeurIPS 2018\n\n[11] Garnelo et al., \"Conditional Neural Processes\", ICML 2018\n\n[12] Kim et al., \"Attentive Neural Processes\", ICLR 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxgAAymtDu", "reviewer": "Reviewer_Whmq", "summary": "This work presents a hierarchical Bayesian model aimed at addressing the few-shot meta-learning problem. The task-specific parameters are influenced by a higher-level global random variable, responsible for capturing shared information across tasks. Within the framework, the prediction tasks on new tasks is viewed as a Bayesian inference problem.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method leverages conjugate properties to provide closed-form solutions for both task-specific and task-agnostic updates. Additionally, this method seamlessly integrates with a wide range of existing neural few-shot learning meta-learners, enhancing its versatility within the field. The empirical results are robust, showcasing improved accuracy and calibration performance across various benchmarks, encompassing tasks related to classification and regression.", "weaknesses": "I have partially reviewed the article, particularly focusing on the derivation of mathematical formulas, and the technical aspects appear to be solid. I did not identify any significant errors or weaknesses. The only minor concern is that the overall approach and the core ideas behind this work are somewhat similar to most Bayesian meta-learning approaches in the context of few-shot learning. These methods generally utilize priors to relate the parameters of multiple tasks, and they apply variational inference for posterior inference on task-level or meta-level parameters. In this regard, the level of innovation in the article may be somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)