Meta-Learning Bayesian Neural Network Priors Based on PAC-Bayesian Theory
Authors: Jonas Rothfuss
Martin Josifoski
Andreas Krause
Venue:  (2021)
Keywords: meta-learning
life-long learning
transfer
bayesian neural networks
prior
few-shot learning
pac-bayes
generalization bound

Abstract: Bayesian deep learning is a promising approach towards improved uncertainty quantification and sample efficiency.
Due to their complex parameter space, choosing informative priors for Bayesian Neural Networks (BNNs) is challenging. Thus, often a naive, zero-centered Gaussian is used, resulting both in bad generalization and poor uncertainty estimates when training data is scarce. In contrast, meta-learning aims to extract such prior knowledge from a set of related learning tasks. We propose a principled and scalable algorithm for meta-learning BNN priors based on PAC-Bayesian bounds. Whereas previous approaches require optimizing the prior and multiple variational posteriors in an interdependent manner, our method does not rely on difficult nested optimization problems and is agnostic to the variational inference method in use. Our experiments show that the proposed method is not only computationally more efficient but also yields better predictions and uncertainty estimates when compared to previous meta-learning methods and BNNs with standard priors.

Under review as a conference paper at ICLR 2021
META-LEARNING BAYESIAN NEURAL NETWORK PRIORS BASED ON PAC- THEORY
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian deep learning is a promising approach towards improved uncertainty quantification and sample efficiency. Due to their complex parameter space, choosing informative priors for Bayesian Neural Networks (BNNs) is challenging. Thus, often a naive, zero-centered Gaussian is used, resulting both in bad generalization and poor uncertainty estimates when training data is scarce. In contrast, meta-learning aims to extract such prior knowledge from a set of related learning tasks. We propose principled and scalable algorithm for meta-learning BNN priors based on MaC-Bayesian bounds. Whereas previous approaches require optimizing the prior and multiple variational posteriors in interdependent manner, our method does not rely on difficult nested optimization problems, and moreover, is agnostic to the variational inference method in use. Our experiments show that the proposed method is not only computationally more efficient but also yields better predictions and uncertainty estimates when compared to previous meta-learning methods and BNNs with standard priors.
INTRODUCTION
Bayesian Neural Networks (BNNs) offer a probabilistic interpretation of deep learning by inferring distributions over the model's weights (Neal 1996). With the potential of combining the scalability and performance of neural networks (NNs) with framework for uncertainty quantification, BNNs have lately received increased attention (Blundell et al. 2015 Gal & Ghahramani 2016). In particular, their ability to express epistemic uncertainty makes them highly relevant for applications such as active learning (Herrández--Lobtto & Adams 2015) and reinforcement learning (Riquelme et al. 2018).
However, BNNs face two major issues: 1) the intractability of posterior inference and 2) the difficulty of choosing good Bayesian priors. While the former has been addressed in an extensive body of literature on variational inference (e.g. Blundell et al.) 2015] Blei al.) a [016]] Mishkin al.) 2018 Liu & Wang, 2016), the latter has only received limited attention (Vladimirova et al.) 2019 Ghosh Doshi Velez, 2017). Choosing an informative prior for BNNs is particularly difficult due to the high- digh-diimeniioal and hardly interpretable parameter space of NNs. Due to the lack of good alternatives, often a zero-centered, isotropic Gaussian is used, reflecting (almost) no a priori knowledge about the problem at hand. This does not only lead to poor generalization when data is scarce, but also renders the Bayesian uncertainty estimates poorly calibrated (Kuleshov et al. 2018).
Meta-learning (Schmidhuber 1987] Thrun & Prati 1998) acquires inductive bias in data-driven way, thus, constituting an alternative route for addressing this issue. In particular, meta-learners attempt extract shared (prior) knowledge from in the face of a new, related task. Our work develops a principled and scalable algorithm for meta- learning BNN priors. We build on the PAC-Bayesian framework (McAllester) 1999), a methodology from statistical learning theory for deriving generalization bounds. Previous PaC-Bayesian bounds for meta-learners (Pentina & Lamperi 2014 Amit & Meir 2018) require solving a difficult optimization problem, involving the optimization of the prior as well as multiple variational posteriors in nested manner. Aiming to overcome this issue, we present a PAC-Bayesian bound that does not rely on nested optimization and, unlike (Rothfuss et al. (202)), can be tractably optimized for BNNs. This makes the resulting meta-learner referred to as PACOH-NN, not only much more computationally efficient and scalable than previous approaches for meta- learning BNN priors (Amit & Meir 2018), but also agnos- tie to the choice of approximate posterior inference method which allows us to combine it freely with recent advances in MCMC (e.g. Chen et al. 2014) variational inference (e.g. Waan et al. 2019).
of related learning tasks (i.e., datasets), aiming to learn

---

Under review as a conference paper at ICLR 2021
Our experiments demonstrate that the computational advantages of PACOH-NN do not result in degraded predictive performance. In fact, across several regression and classification environments, PACOH-NN achieves a comparable or better predictive accuracy than several popular meta-learning approaches, while improving the quality of the uncertainty estimates. Finally, we showcase how meta- learned PACOH-NN priors can be used in real-world bandit task concerning the development of vac- cines, suggesting that many other challenging real-world problems may benefit from our approach.
RELATED WORK
Bayesian Neural Networks. The majority of research on BNNs focuses on approximating the intractable posterior distribution (Graves) 2011 Blundell et al.] 2013: Liu & Wang 2016 Wang
et al. 2019). In particular, employ the approximate inference method of Liu & Wang (0066).
Another crucial question is how to select good BNN prior (Vladimirova et al. 2019). While the majority of work (e.g. Louizos & Welling 2016 Huang et al., 2020) employs a simple zero-centered, isotropic Gaussian, Ghhosh (aessshn,,GGhhh
& Doshi Velez (2017) and Pearce et al. (020) have proposed other prior distributions for BNNs. In contrast, we go the alternative route of choosing priors in a data-driven way.
Meta-lcarning, A range of popular methods in meta-learning attempt to learn the "learning program" " in form of a recurrent model (Hochreiter al., 200T Andrychowicz et al.] 2016 Chen et al., 2017).
learn an embedding space shared across tasks (Snell et al. [ et al. (0166) or the initializa-
tion of NN such that it can be quickly adapted to new tasks (Finn et al. [017]] (007ool
al. 2018
Rothfuss et al. 2019b). A group of recent methods also uses probabilistic modeling to also allow for uncertainty quantification (Kim et 20188] Finn et al.) 2018 Garnelo et al.) 2018). Although the mentioned approaches are able to learn complex inference patterns, they rely on settings where meta- training tasks are abundant and fall short of performance guarantees. In contrast, we provide formal assessment of the generalization properties of our algorithm. Moreover, PACOII-NN allows for prin- cipled uncertainty quantification, including separate treatment of epistemic and aleatoric uncertainty. This makes it particularly useful for sequential decision algorithms (Lattimore & Szepesvari 2020).
PAC-Bayesian theory. Previous work presents generalization bounds for randomized predictors, as- suming a prior to be given exogenously (McAllester 1999 Catoni 2007 Germain et a 2016 Alquier et al. 2016). More recent work explores data-dependent priors -errrddd-H et al. 2012][Dziu
gaite & Roy (0166) or extends previous bounds the scenario where priors are meta-learned (Pentina & &Lmppert 2014] Amit & Meir 2018). However, these meeta-geneallization bounds are hard to mini- mize as they leave both the hyper-poce poseerrort and posterior unspecified, which leads to nested optimiza- tion problems. Our work builds on the results of Rothfuss et al. (2020) who introduce the methodology to derive the closed-form solution of the PAC-Bayesian meta-learning problem. However, unlike ours, their approach suffers from (asymptotically) non-vanishing terms in the bounds and relies on closed- form solution of the marginal log- likelhooddd By contributing a numerically stable score estimator for the generalized marginal log- lidhooo,, we are able to overcome such limitations, making PAC- Bayesian meta- meta-learning both tractable and scalable for a much larger array of models, including BNNs.
BACKGROUND: THE PAC-BAYESIAN FRAMEWORK
Bayesian Neural Networks. Consider a supervised learning task with data S = {[z,, 177 drawn from unknown distribution D. In that, X {-))
x denotes training in- puts and Y = (y5 )= E ym the targets. For brevity, we also write zj- = (x),0;) E Z. Let hg : X + )' be a function parametrized by a NN with weights E O. Using the NN mapping, we define conditional distribution p(yla, 0). For regression, we set p(y|z, 0) = N(y (x). where 0² is the observation noise variance. For classification, we (rrgorrica(softmax(ht
For Bayesian inference, one presumes prior distribution p(0) over the model parameters 0 which is combined with the training data S into posterior distribution p(0[X Y) x p(0)p(Y X, 0). For unseen test data points x", we form the predictive distribution as p(y" [x" X, Y) p(y" x= ())( Y)do.
choose p(y) x. 8)
The Bayesian framework presumes partial knowledge of the data-generating process in form of a prior distribution. However, due to the practical difficulties in choosing appropriate BNN prior, the prior is typically strongly misspecified (Syring & Martin 2018). As a result, modulating the influence of the prior relative to the likelihood during inference typically improves the empirical performance of BNNs and is thus a common practice /Weneel et al. 2020). Using such a "tempered" posterior p. (+(0) [0)x Y) xc p(0)p(( (YJ), 0)" with > 0 is also referred as generalized Bayesian learning (Guedj 2019)
2

---

Under review as a conference paper at ICLR 2021
The PAC-Bayesian Framework. In the following, we introduce the most relevant concepts of PAC- Bac--aaeii learning theory. For more details, we refer to Guedj (2019). Given a loss function 1:0 x 2 + I. we typically want minimize the generalization error [ D) E...D 1(0,2"). Since D is unknown, the empirical error [(0, S) = J -i= 1(0, z,) is usually employed, instead. In the PAC-Bayesian framework, we are concerned with randomized predictors, i... probability measures on the parameter space θ allowing us to reason about epistemic uncertainty. In particular, we consider two such probability measures, a prior P E MM()) and a posterior Q M()). In here, by M()), we denote the set of all probability measures on Ô. While in Bayesian inference, the prior and posterior are tightly coupled through Bayes' theorem, the PAC-Bayesian framework only requires the prior to be independent of the data S. Using the definitions above, the so-called Gibbs error for a randomized predictor Q is defined as ((Q, D) E [h,, Similarly, we define its empirical counterpart as É(Q, S) = En~Q [(h, S). The PAC-Bayesian framework provides upper bounds for the unknown Gibbs error in the following form:
m
Theorem 1. (Alquier et al. 2016) Given a data distribution D, a prior P E M((),, a confidence level 6 E (0, 11, with probability at least å over samples S Dm,, we have:
1
1
AQ E : L)Q, S (Q,
DxI(QIP)) + In [((m)
(1)
Vm
exp [vm (((). ((e,
where (Vm = In
In that, (Vmm) is log moment generating function that quantifies how strong the empirical error deviates from the Gibbs error. By making additional assumptions about the loss function we can bound V (()m) and thereby obtain tractable bounds. For instance, if I(0, z) is bounded in [a, b), we obtain V (Vmm)= ) S ((b //8 8 by Hoeffding's 's lemma. For unbounded loss functions, it is common to assume bounded moments. For instance, a loss is considered nub-ganma with variance factor s² and scale parameter c, under prior P and data distribution DD, if its deviations from the mean can be characterized by random variable := [hh, D) I(h, whose moment generating function is upper bounded by that of a Gamma distribution [(s, c) (Boucheron et al. 2013). In such case, we obtain (vm) 8² (2 20
Connecting the PAC-Bayesien framework and generalized Bayesian learning. In PAC- Bayesian learning we aim to find the posterior that minimizes the bound in (1 which is in general challenging optimization problem over the space of measures M()). However, to our benefit, it can be shown that the Gibbs posterior is the probability measure that minimizes (I.. For details refer
to Lemma 2 in the Appendix or Catonni (207) and (Germain et al. 2016). In particular, this gives us
(0) == arg min VmL(Q, + (KI(O) P(0)e / QEM(O)
(2)
where Z(S, P) is a normalization constant. In a probabilistic setting, our loss function is the negative log-likellng legliikli,ood, i.e. 1(0,2)) == - log p( 2; (0). In this case, the optimal Gibbs posterior coincides with the generalized Bayesian posterior Q (0: P,S) x P(0))((10))/v//// /Z, P) where Z(5, P) Je ( di is the generalized marginal likelihood of the data sample S.
4 PAC-BAYESIAN BOUNDS FOR META-LEARNING
This section describes the PAC-Bayesian meta-lcarning setup and discusses how we can obtain generalization bounds that can be transformed into practically useful meta-learning objectives. In that, we draw on the methodology Rothfuss et al. (2020) which allows us to derive closed form solution of the PAC-Bayesian meta-learning problem.
In the standard supervised learning setup (see Sec. 3 the learner has prior knowledge in the form of prior distribution P. given exogenously, over the hypothesis space H. When the learner faces a new task, it uses the evidence, provided dataset S, to update the prior into posterior distribution Q. We formalize such a base learner QSS, P) as a mapping Q 2 x M(H) + M(H) that takes in dataset and prior and outputs posterior.
In contrast, in meta- learnnen we aim to acquire such a prior P in a data-drived manner, that is, by con- sulting set of statistically related learning tasks {Ti, .... }. We follow the setup of Baxter (2000) in which all tasks T := (D,, S,) share the same data domain 2 == X' V. parameter space and loss function 1(0, z), ), but may differ in their (unknown) data distributions D; and the number of points
3

---

Under review as a conference paper at ICLR 2021
m; in the corresponding dataset S ~ D. For our theoretical expositions, we assume w.l.o.g that m = m, vi. Further, each task T; ~ T is drawn i.id. from an environment T, a probability distribu- tion over data distributions and datasets. The goal is extract knowledge from the observed datasets which can then be used as a prior for learning on new target tasks 7 ~ T [Amit & Meir] 2018). To extend the PAC-Bayesian analysis the meta-learning setting, again consider the notion of proba- bility distributions over jypothhses/parameetee
However, while the object of learning has previously been the NN parameters 0, it now the prior distribution P E M()). Accordingly, the meta-learner presumes a hyper-prior P E M(M(H)), i.e., a distribution over priors P. Combining the hyper-prio P with the datasets S1 .... S, from multiple tasks, the meta-learner then outputs hyper-posterior Q over priors. The p performance/quality is measured in form of the expected Gibbs error when sampling priors P from Q and applying the base learner, the so-called transfer-error:
( = Er-o [E(D, [ES-D" P)) P)]]]
While the transfer error unknown in practice, we can estimate using the empirical multi-task error
L S)) = Ep~e
Σ ():
1=1
Similar to the PAC-Bayesian guarantees for single task learning. we can bound the transfer error by its empirical counterpart Ł(Q, Si..., S.) plus several tractable complexity terms: Let Q z x M(H) + .M(H) be base learner and P e MMM(HH) some fixed hyper-prior. For all hyper-posteriors Q E (M(((H)) and E (0,
Theorem 2.
nn/m (()=++))
1
((2,T) S SI..,, S.)
(2IPP))P)
Vn
(3)
1 Ep~o (DL, + n(6,n.m) mm
holds with probability 1 1- 8. If the loss function is bounded, that is l H x Z - [a, b). the above
inequality holds with C(6, n, m) (2a-
8/m (ba-ax)²
Svn vn In 8. If loss function is sub-gamma
with variance factor oe and scale parameter Cj for data distributions Dj and sij en for the task 1 In
distribution T, the inequality holds with C(ő, n, n) 2(ym-o) 2[Vn-au)
Vn
Under bounded loss assumption, Theorem 2 provides structurally similar, but tighter bound than Pentina & Lampert 2014) and Rothfuss et al. (2020). In particular, by using a improved proof methodology, we are able to forgo union bound argument, allowing to reduce the negative influence confidence parameter 8. Compared to Rothfuss et al. (220)). the DKz P) term has opposed to 1/vm 1/v7.. Importantly, the bound in (3) is consistent, i.e. C(,, n, m) - as n, m + co. Unlike Pentina & Lampert (2014) and Amit & Meir (2018), the theorem also provides guarantees for unbounded loss functions under moment constraints (see Appendix A for details). This makes Theorem particularly relevant for probabilistic models such as BNNs in which the loss function coincides with the inherently unbounded negative liceliod....d.
an improved decay rate, that is, 1/(n/m) 1/Vr
2
Amit & Meir (0018) propose meta-learn NN priors by directly minimizing a bound similar (3) However, the posterior inference for BNNs, i.e. obtaining Q; = Q(S, P). is stochastic optimization problem in itself whose solution in turn depends on P. Hence, minimizing such meta-level bound w.r.t. P constitutes computationally infeasible two-level optimization problem. To circumvent this issue, they jointly optimize P and approximate posteriors Q; that correspond the different datasets S,, leading to an unstable and poorly scalable meta-learning algorithm.
To overcome these issues, we employ the methodology of Rothfuss et al. (2020) and assume the
Gibbs posterior (S, P) as base learner. As discussed in Section 3] Q'(S,, P) does not only constitute a generalized Bayesian posterior but also minimizes the PAC- BByessis bound. Thus, the resulting bound in Corollary [ is tighter than 3 More importantly, the bound can be stated in terms of the partition function (S, P) which allows us to forgo the explicit reliance on the task posteriors Q. This makes the bound much easier to optimize
meta- meta-learning objective than previous bounds (c.g. Pentina & Lampert 2014: Amit & Meir 2018), since it no longer constitutes two-level optimization problem. Moreover, renders the corresponding meta-learner agnostic
4

---

Under review as a conference paper at ICLR 2021
to the choice of approximate inference method used to approach the Gibbs/Bayes posterior (S, P) when performing inference on a new task.
Corollary 1. When choosing the Gibbs posterior Q" (S,, P) := P(O) exp(-Vmm2(s,, ()
(S;, P)
as a base learner, under the same assumptions as in Theorem 2 it holds with probability 5 that
1 Σ Ep (In Z(S;, P)] +
1 DxI(Q11P) + n( (4)
C(2, T)
mvm)
vn
wherein Z(S,, Eamp (xp((-vm((s, 0))
is the generalized marginal log- -ilkeiihhodd
A natural way to obtain PAC- meta-learning algorithm could be minimize (4) w.r.t. Q. Though, in general, this is hard problem since it would require a minimization over M (H)). the space of all probability measures over priors. Following Rothfuss et al. (2020), we exploit once more
the insight that the minimizer of A can be written as Gibbs distribution (c.f. Lemma 2}. allowing us to to derive such minimizing hyper-posterion 2", i.e. the PACOH, in closed form:
Proposition 1. (PAC-Optimal Hyper- ) Given hyper-prior P and datasets Si,..., Sn, the hyper-posterior Q that minimizes the CC-Baysiaan meta- meta-learning bound (4) is given by
@' = P(P) exp
Vnm++ Σi=1 In 7(S,, P))
(5)
("
Sny P)
with the partition function defined as z Eppp
Vam++ E In Z(5,, PI)]
exp
We refer to (P) as PAC-optimal since, among all meta-leamers, it gives the best possible PAC-Bayesiie guarantees induced by Theorem 2
5 PACOH-NN: A SCALABLE ALGORITHM FOR LEARNING BNN PRIORS
We discuss main contribution, that is, how translate the PACOH (Prop. I into practical algorithm for meta-learning BNN priors. To this end, we first specify various components of the generie meta-learning setup presented in Sec. and then discuss how to obtain tractable approximation of Q"
The setup. First, we define a family of priors {Po Φ E 4 over the NN parameters 0. For computational convenience, we employ Gaussian priors with diagonal covariance matrix, i.c. Po =Nup, ( with Φ := [up,logor] to avoid additional positivity constraints. In fact, any parametric distribution such as normalizing flows (Rezende & Mohamed, 2015) that allows for re-parametrized sampling and has a tractable log-density could be used. As typical in the Bayesian framework, our loss function is the negative log-likelihood, i.e. =00,))= = In p(ylar, 0) for which we assume an additive Gaussian noise model p(y|z, 0) N(r he (z))) az) in regression and categorical softmax distribution in case of classi- fication. Moreover, we use zero-centered, spherical Gaussian over the parameters & that specify the prior. In our setup, the hyper-prior acts form of meta-level regularization that penalizes complex priors.
Note that we represent dp in the log-space
(O, ob/) as a hyper-prior
Approximating the hyper-posterior. Given the hyper-prior and (level-I) log-partition function In (S,, P). we can compute the PACOH Q" up to the normalization constant z Such a setup lends itself to approximate inference methods (Blei et al. 2016). In particular, we employ Stein Variational Gradient Descent (SVGD) (Liu Wang (016) which approximates Q* as a set of particles Q = {0) ΦK). Initially, the particles Φx ~ P (i.e. the priors' parameters) are sampled randomly. Subsequently, the method iteratively transports the set of particles to match Q", by applying a form of functional gradient descent that minimizes Dxl(Ql"* in the reproducing kernel Hilbert space induced by a kernel function k(.,)). In each iteration, the particles are updated by Φ Øk mb mpp"((x) (Øx) with the step size and R
4 (Ø) = K
Σ
kkOw
V) on In (ow) + Vay k(ow 5)]
(6)
W=1
In that, V In Q" (dx) = Vo, In P(Ok)
V In 7(,, Po.) is the score of Q*.
V
Approximating the generalized marginal log-likelihood. The last remaining issue towards a viable meta-learning algorithm is the intractable generalized marginal likelihood In 7(.,, Pa)
5

---

Under review as a conference paper at ICLR 2021
Traning with camples
Trasiing win Stamples
Training with 10 camples
Trariig with camples
Traising with samples
Trainig i sample
Teans
Trae
28 -2
50
33 41 60
25
50
40
-25
86
25
56
(a) BNN with isotropic, zero-centered Gaussian prior
(b) BNN with meta-lcarned PACOH-NN prior
Figure 1: BNN posterior predictions with (a) standard Gaussian prior vs. (b) meta-learned prior. Meta-learning with PACOH-WN-SVGD was conducted on the Sinusoids environment.
In E (0,, Estimating and optimizing In (S.. Po) is not only challenging due to the high- nigh-diimessional expectation over è but also due numerical instabilities inherent in computing e" fm, when m; is large. Aiming to overcome these issues, we compute numerically stable Monte Carlo estimates of V In Z(S,, Po. by combining the LogSumExp (LSSE) with the re- parametrization trick (Kingma Welling 2014). In particular, we draw samples 02 == f( et)
μP + op EJi El N(0,1) 1) and compute the forward pass as follows:
In ((s, Po) = In 1
Σ
vm,L(o,.S.) = LSE (-vm; - I, 01 ~ P 1=1
(7)
L
The corresponding gradients follow softmax-weighted average of score gradients:
e e-vm;
Vo in Z(S,, = -mm
f() Vo, (o. S.)
(8)
e mm
1=1
re-param.
Jacebian
softmax
Note that In (S,, Po) a consistent but not an unbiased estimator of In Z(S, Ps). The following proposition ensures us that we still minimize valid bound (see Appx. B.3 for details).
Proposition 2. In expectation, replacing In Z(S, Pb) Ps) in (4) by the estimate In (S,, P) still yields a valid upper bound of the transfer error C(2, T) for any L
Moreover, by the law of large numbers, we have that In Z(S,, P) In Z(S, P) as L + co, that is, for large sample sizes L, we recover the original PAC-Bayesian bound in 44). In the opposite edge case, i.e. I. 1, the boundaries between tasks vanish meaning that the meta-training data (Si,..,, Sn) is treated as if it were one large dataset U, S, (see Appx. B.3 for further discussion).
The algorithm. Algorithm M in Appendix B summarizes the proposed meta-learning method which we henceforth refer to PACOH-NN. To estimate the score Ve,, In ("((v) (6) we can even use mini-batching on the task level. This mini-batched version, outlined in Algorithm 2, maintains K particles to approximate the hyper-posterior, and in each forward step samples L NN-parameters (of dimensionaly @ per prior particle that deployed on mini-batch of n&. tasks estimate the score of Q As a result, the total space complexity in the order computational complexity of the algorithm for single update (c.f. (6 is O(K2 + KTmbs).
[[e] + L) and the
key advantage of PACOH-NN over previous methods for meta-learning BNN priors (e.g. Pentina & LLmmpert 2014 Amit & Meir 2018) is that it turns the previously nested optimization problem into a much simpler stochastic optimization problem. This makes meta-learning not only much more stable but also more scalable. In particular, posteriors Q; and can do mini-batching over tasks. As a result, the space and compute complexity do not depend on the number of tasks n. In contrast, .LAP Amit & Meir 2018) has a memory footprint of O([O)n r2) making meta-learning prohibitive for more than 50 tasks.
do not need to explicitly compute maintain the task
central feature of PACOH-NN is that comes with principled meta-level regularization in form of the hyper-prion P which combats overfitting to the meta- tetantraining tasks (Qin et al. 2018). As we show in our experiments, this allows us successfully perform meta-learning with little as This is unlike the the majority of the popular meta-learners (Finn et al. 2017; Kim et al. 20188 Garnelo et al. (2018] e.g.) which rely on a large number of tasks to generalize well on the meval
tasks.
(Qin et al. 2018 Rothfuss et al. 2020).
EXPERIMENTS
We empirically evaluate the method introduced in Section 5 in particular, two variants of the algorithm: PACOH-NN-SVGH, with K 5 priors as particles and the edge case K = which

---

Under review as a conference paper at ICLR 2021
Cauchy
SwissFel
Physionet-GCS
Physionst-HCT
Berkeley-SSnsor
BNN (Lu & Wang 2016
0.327 1: 0.008
0.329 0.022
2.664
0.274
3.938
0.869
0.109 :: 0.004
PACORENN Tours)
0.195 0.001
0.372 0.000
1.561
2,405 0.017
NACOH-NN-MAP (ours)
0.202 + 0.003
0.375 + 0.004
1.564 0.2000
2.480 + 0.042
0.017 = 0.001
PACOH GP
Rothfuss et al. 2020)
0.209 + 0.008
0.376 + 0.024
1.498 0.081
2.361 + 0.0047
0.065 + 0.005
MLAP.M
Amil & Meir 2018
0.219
0.492 0.009
2.232 0.261
2.541 0.114
0.032 = 0.003
MLAP-S
Amiil & Mei 2018
0.219
0.486 + 0.026
2.009 002218
2.170 + 0.039
0.050 = 0.005
FOMAMI
Nicholl et al.
2018
0.200+0.007
0.897 + 0.071
2.545 0.615
2.408 + 0.064
0.059 + 0.005
MAML
Pulu
0.219 0.004
0.057
2.413 0.113
0.003
BMAML
Kim
al.
2018
0.225 + 0.001
0.577 + 0.041
1.991
0.062
2.500
+ 0.002
0.073 = 0.014
NPs (Garnelo ot al.
2018]
0.224 + 0.008
0.471 1 0.053
2.056 = 0.207
2.504 1 0.107
0.079 = 0.007
Table 1: Comparison of standard and meta-learning algorithms in terms of test RMSE in 5 meta-
learning environments for regression. Reported are mean and standard deviation across 5 seeds.
Cauchy
SwissFel
Physionet-HCT Physionct-GCS
Berkeley-Sensor
BNN (Lān & Wang 2016
0.055 0.006
0.085 + 0.008
0.277 + +0011
0.507 + 0.009
0.179 + 0.002
PACOH-NN SVGD (Ours)
0.046
0.001
0.027 0.003
0.267 0.005
0.302 1 0.003
0.067 0.005
PACOH-NNAA NN-MAP (ours)
0.051 0.002
0.031 0.003
0.268 0.015
0.306 + 0.003
0.063 0.016
PACOH-GP Rothiuss et al. 2020
0.056 0.004
0.038 + 0.006
0.262 + 0.000
0.296 + 0.003
0.088 0.005
MLAPPM Amml & Menr 2018
0.088 0.004
0.104 0.015
0.339 0.012
0.297 1 0.007
0.077 0.010
MI.AP.S (Amil Meir
0.086 0.015
0.090 0.021
0.343 0.017
0.344 0.016
0.108 0.024
BMAML Kim et al. 2018
0.061 2 0.007
0.115 1 0.036
0.279 1 0.010
0.423 1 0.106
0.161 1 0.013
NPs (aameoo et al.
2018
0.037 = 0.009
0,131 = 0.056
0.259 : 0.012
0.319 = 0.004
0.210 0.007
Table 2: Comparison of standard and meta-learning algorithms in terms of test calibration error in 5
meta-learning environments for regression. Reported are mean and standard deviation across 5 seeds.
(MAP) approximation of Q", thus referred to as PACOH- NN-MAP. In order to evaluate the quality of the meta-learned prior, i.e. meta-lesting, we need to do approximate the BNN posterior Q*(S, P) for which we use SVGD with 5 particles, too.
coincides with a maximum-a-posternn
Comparing to various NN-based meta-learning approaches on various regression and classification environments, we demonstrate that PACOH-NN (i) outperforms previous meta-learning algorithms in terms of predictive accuracy, (ii) improves the quality of uncertainty estimates and (iii) is much more scalable than previous PAC-Bayesial meta-learners. Finally, we showcase how meta-learned PACOH- NN priors can be harnessed in a real-world bandit task concerning peptide- peptide-based vaccine development.
6.1 META-LEARNING BNN PRIORS FOR REGRESSION AND CLASSIFICATION
Figure fillustrate BNN predictions on sinusoidal regression task with standard Gaussian prior as well as a PACOH-NN prior meta-learned on sinusoidal functions of varying amplitude, phase and slope (details can be found in Appendix C.1 In Figure Ta we can see that the standard Gaussian prior provides poor inductive bias, not only leading to bad mean predictions away from the testing points but also to poor 95% confidence intervals (blue shaded areas). In contrast, the meta-learned PACOH-NN prior encodes useful inductive bias towards sinusoidal function shapes, leading to good predictions and uncertainty estimates, even in face of minimal training data (i.e. I training point).
Meta-learning benchmark. In the following, we present a comprehensive benchmark study. First, we use a BNN with zero-centered, spherical Gaussian prior and SVGD posterior inference (Liu & Wang 2016) as a bascline. Second, we compare our proposed approach against various popular meta-learning algorithms, including model-agnostic meta-learning (MAML) (Finn al.. (017). its first-order version (FOMAML) (Nichol et al. 2018). Bayesian MAML (BMAML) (Kim et af. 2018) and two variants of the PAC-Bayesian approach by Amit & Meir 2018) (MLAP). For experiments with regression tasks, we also include into our comparison neural processes (NPs) (Garnelo et al. 2018) and the GP based meta-learner of Rothfuss et al. (2020) (PACON-GP)). The latter, similar to our method as it also approximates a form of the PAC-optimal Hyper-Posterior with SVGD. However, unlike PACOH-NN marginal log- likellihoood. Among all, MLAP is the most similar to our approach as it is neural network based and minimizes PAC-Bayesian bounds of the transfer error. Though, unlike PACOH-NN, relies on nested optimization of the task posteriors Q, and the hyper-posterior Q.
uses Gaussian Processes (GPs) as base learners and relies on a closed-form
Regression environments. In our experiments, we consider one synthetic and four weal-world meta- learning environments for regression. As a synthetic environment we follow Rothfuss et al. (2020), employing a 2-dimensional mixture of Cauchy distributions plus random GP functions. As real-world environments, we use datasets corresponding to different calibration sessions of the Swiss Free Electron Laser (SwissFEL) (Milne et al.) 2017 Kirschner et al. 2019b), as well as data from the PhysioNer 2012 challenge, which consists of time series of electronic health measurements from
7

---

Under review as a conference paper at ICLR 2021
Accuracy
Calibration error
Omniglot 2-sbot
Omniglot 5-shot
Omniglot 2-shot
Omniglot 5-shot
BNN (Läu & Wang 2016
0.6704 0.006
U.795 0.006
0.173 0.009
0.135 0.009
PACOH-NN--SVG (OUTS)
0.733 + 0.009
0.885 = 0.090
0.094 0.004
0.091 0.000
PACOH NN MAP (ours)
0.735 + 0.010
0.866 + 0.005
0.099 0.009
0.075 + 0.006
MLAP-M (Amit Meir 2018
0,635 0 0.015
0.801 0.0168
0.108
0.00S
0.119 = 0.0193
MLAP-S Amit & Meir 2018
0.615 - 0.037
0.700 0.0135
0.018 0.129
0.108 + 0.010
FO-MAML Nichol et al
2013
0.129 = 0.017
0.590 - 0.010
NIA
N/A
MAML (Fun et di 2017
0,571 = 0.018
0.693 + 0.013
NIA
N/A
BMAML
(Kim et ale 2018
0.651 0.008
0.764 0.025
0.132 0.007
0.191 0.018
Table Comparison of meta- meta-learnnng algorithms in terms of test accuracy and calibration error on the Omniglot environment with 2-shot and 5-shot w-way-clasiifiatiion tasks.
intensive care patients (Silva et al. 2012); in particular the Glasgow Coma Scale (GCS) and the hematocrit value (HCT). Here, the different tasks correspond to different patients. Moreover, we employ the Intel Berkeley Research lab temperature sensor dataset (errkeley-Sensor) (Madden 2004) where the tasks require aulo-regressive prediction of temperatures measurements corresponding to sensors installed in different locations of the building. Further details can be found in Appendix
Table reports the results of our benchmark study in terms of the root mean squared error (RMSE) on unseen test tasks. Among the approaches, PACOH-NN consistently performs best or is among the best two methods, demonstrating that the introduced meta-learning framework is not only sound, but also endows us with an algorithm that works well in practice. Only for owwdimensional and small-scale regression environments like Physionet, we find that the GP-based meta-learner PACOH-GE GP which is build on a similar theoretical foundation as our method works better than PACOII-N.
Further, we hypothesize that by acquiring the prior in principled data-driven manner (e.g.. with PACOH-NN), we can improve the quality of the BNN's uncertainty estimates. To investigate the effect of meta-learned priors on the uncertainty estimates of the BNN, we compute the probabilistic predic- tors' calibration error, reported in Table 2 The calibration error measures the discrepancy between predicted confidence regions and actual frequencies of test data in the respective areas (Kuleshov et al. 2018). Note that, since MAML only produces point predictions, the concept of calibration does not apply to it. First, we observe that meta-learning priors with PACOH-NN consistently improves the standard BNN's uncertainty estimates. For meta-learning environments where the task similarity is high, i.e. SwissFel and Berkeley-Senen -ensorr, the improvement is substantial. Surprisingly, while improving upon the standard BNN in terms of the RMSE, we find that NPs consistently yields worse-calin calibrated predictive distributions than the BNN without beaanng.. This may be due meta-level overfitting as NPs lack any form of meta-level regularization (cf. Qin et al. 2018 Rothfuss et al. 2020).
Classification environments. We conduct experiments with the multi-tast classification environment Omniglot (Lake et al. (0115), consisting of handwritten letters across 50 alphabets. Unlike previous work (e.g. Finn et al. 2017)) we do not perform autteetgmenttoionn and do not re-combine letters of dif- ferent alphabets, preserving the data's 's original structure. In particular, one task corresponds to 5-way classification of letters within an alphabet. This leaves us with much fewer tasks (i.e. 30 train and 20 test tasks), making the environment more challenging and more interesting for uncertainty quantifica- tion. This also allows us to include MLAP in the experiment which hardly scales to more than 50 tasks.
In Table 3 we report both the accuracy and calibration error for 2-shot and 5-shot classification test tasks. Again, PACOH- NN yields the most accurate classification results and the lowest calibration error. Note that MAML fails to improve upon the standard BNN. i.e. demonstrating negative transfer. This is consistent with previous work (Qin et al., 2018; Rothfuss al.) 2020) raising concerns about overfitting to the meta-training tasks and observing that MAML requires large number of tasks to generalize well. In contrast, by its very construction on meta- geteraeeneiiioooionnn bounds, PACOH-NN is able to achieve positive transfer even when the meta- meta-training tasks are diverse and small in number.
Moximum mamery conoumptionn [in (B)
Memory
Execution time
2.5
20
PACOM-NN
2.0
15
Execuptio tmeelnn
1.5
2.0
0.0
20
40
60
80
20
60
Number tosies
Number of tasks
Figure 2: Scallbilityy comparison of PACOH-NN and MLAP-S in memory footprint and compute time, the number of meta-training task grows.
8

---

Under review as a conference paper at ICLR 2021
Scalability. Unlike the MLAP (Amit & Meir 2018), PACOH-NN does not need to maintain posteriors Q; for the meta- mheta-trinnnng tasks and can use mmiibatching on the task level. As a result, is computationally much faster and more scalable than previous PAC-Bayesian meta-learners. This is reflected in its computation and memory complexity, discussed in Section [] 2 showcases this computational advantage during meta-training with PACOH-NN-MA environment with varying number of tasks, reporting the maximum memory requirements, as well as the training time. While MLAP's memory consumption and compute time grow linearly and becomes prohibitively large even for less than 100 tasks, PACOH-NN maintains a constant memory and compute load as the number of tasks grow.
and MLAP-S in the Sinusoids
6.2 -EETA-LE FOR BANDITS- VACCINE DEVELOPMENT
We showcase how relevant real-world application such as vaccine design can benefit from our proposed method. In particular, the goal is to discover peptide sequences which bind to major histocompatibility complex class-I molecules (MHC-I). MHC-I molecules present fragments of proteins from within a cell to T-cells, allowing the immune system to distinguish between healthy and infected cells. Following the bandit setup Krause & Ong (01T), each task corresponds to searching for maximally binding peptides. vital step in the design of peptide-based vaccines. The tasks differ in their targeted MHC-I allele, i.e., correspond to different genetic variants of the MHC-I protein. We use data from Widmer al. (2010) which contains the binding affinities (IC50 values) of many peptide candidates to the MHC-I alleles. The peptide candidates are encoded as 45-dimensional feature vector and the binding affinities were standardized.
We use 5 alleles (tasks) to meta-learn BNN prior with PACOH-NN and leave the most genetically dissimilar allele (A-6901) for our bandit task. In each iteration, the experimenter (i.e. bandit algorithm) chooses to test one peptide among the pool of more than 800 candidates and receives its binding affinity reward feedback. In particular, we employ UCB (Lattimore & Szepesvari (020)) and Thompson- Sampling (TS) [Thmmsonn] 1933) as bandit algorithms, comparing the BNN-based bandits with meta-learned prior (PACOI--CB/TS)) against a zero-centered Gaussian BNN prior (BNN-UCB/TS) and a Gaussian process (GP-UCB) (rinivas et al. 2009
GP-UCB
BNI--UCB
simpple regret
BAIN-TS
FACOH-UCB
averac
PACOH-TS
22 0
20
20
40
iteration
iteration
Figure 3: MHC-I peptide design task: Regret for different priors and bandit algorithms. A meta- learned PACOH-NN prior substantially improves the regret, compared to a standard BNNGG prior.
Figure reports the respective average regret and simple regret over 50 iterations. Unlike the bandit algorithms with standard BNN/GP prior, PACOH- UCBTS reaches near optimal regret within less than 10 iterations and after 50 iterations still maintains a significant performance advantage. This highlights the importance of transfer (learning) for solving real-world problems and demonstrates the effectiveness of PACOH-NN to this end. While the majority of meta-learning methods rely on large number of meta-training tasks QQin et al. 2018), PACOH-NN allows to achieve promising positive transfer, even in complex real-world scenarios with only a handful (in this case 5) tasks.
7 CONCLUSION
Based on PAC-Bayesian theory, we present novel, scalable algorithm for meta-learning BNN priors, that overcomes previous issues of nested optimization, by employing the closed-form solution of the PAC-Bayesian meta-learning problem. Experiments show that our method, PACOFI-NN, does not only come with computational advantages, but also achieves comparable or better predictive accuracy than several popular meta-learning approaches. while improving the quality of the uncertainty estimates- a key aspect of our approach. The benefits of our principled treatment of uncertainty showcased in the real-world vaccine development bandit task are particularly amenable to interactive machine learning systems. This makes the integration of PACOH-NN with Bayesian optimization and reinforcement learning potentially promising avenue to pursue. While our experiments are limited to diagonal Gaussian priors and SVGD as approximate inference method, we hope that future work will build on the added flexibility of our framework to possibly explore more recent approaches in variational inference (c.g. Wang et al. 2019) or consider more expressive priors such as normalizing flows (Rezende & Mohamed 2015).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pierre Alquier, James Ridgway, Nicolas Chopin, and Yee Whye Tch. On the properties of variational approximations of Gibbs posteriors. Journal of Machine Learning Research, 2016.
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In International Conference Machine Learning, 2018
Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. arXiv,, 2016.
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 2000.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International Conference on Machine Learning, 2013.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. arXiv, 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv, 2015.
Stephane Boucheron, Gabor Lugosi, and Pascal Massart. Concentration inequalities nonasymptotic theory of independence. chapter 2.4, pp. 27-30. Oxford University Press, 2013.
Olivier Catoni. PAC-Bayesian supervised classification: the thermodynamics of statistical learning. arXiv, 2007.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International Conference Machine Learning, 2014.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando De Freitas. Learning to Learn without Gradient Descent by Gradient Descent. In International Conference Machine Learning. 2017.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Uncertainty in Artificial Intelligence, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learnig for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, 2018.
Vincent Fortuin, Gunnar Rätsch, and Stephan Mandt. Multivariate time series imputation with
variational autoencoders. arXiv preprint arXiv:1907.04115: 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model
uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv, 2018
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julies Pac-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, 2016.
Soumya Ghosh and Finale Doshi-Veles Model selection in bayesian neural networks via horseshoe priors. arXiv, 2017.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011.
10

---

Under review as a conference paper at ICLR 2021
Benjamin Guedj. A primer PAC-Bayesian learning. arXiv, 2019.
José Miguel Hernández-Lobate and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Learning, 2015.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning To Learn Using Gradient Descent. In International Conference on Artificial Neural Networks, 2001.
Chin- Wei Huang, Ahmed Touati, Pascal Vincent, Gintare Karolina Dziugaite, Alexandre Lacoste, and Aaron Courville. Stochastic neural network with kronecker flow. In International Conference on Artificial Intelligence and Statistics, 2020.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian -adel-agnostic meta-learning. In Advances in Neural Information Processing Systems, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe bayesian optimization in high dimensions via one-dimensional subspaces. In International Conference on Machine Learning, 2019a.
Johannes Kirschner, Manuel Nonnenmacher, Mojmir Mutný, Nicole Hiller, Andreas Adelmann, Rasmus Ischebeck, and Andreas Krause. Bayesian optimization for fast and safe parameter tuning of swissfel. In International Free-Electron Laser Conference (FE112019), 2019h.
Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. In Advances in Neural Information Processing Systems, 2011.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference Machine Learning, 2018.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Tor Lattimore and Csaba Szepesvari. Bandit Algorithms. Cambridge University Press, 2020.
Qiang Liu and Dilin Wang. Stein Variational Gradient Descent: General Purpose Bayesian Inference Algorithm. arXiv, 2016.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning, 2016.
Samuel Madden. Intel lab data. httpp:/dbb. htt.ll..bccaa....// wwt..m/e///abbbaaaaaaaaaaaa
htm)
2004. Accessed: Sep 8, 2020.
David A McAllester. Some PAC-Bayesian theorems. Machine Learning, 1999.
Christopher J Milne, Thomas Schietinger, Masamitsu Aiba, Arturo Alarcon, Jürgen Alex, Alexander Anghel, Vladimir Arsov, Carl Beard, Paul Beaud, Simona Bettoni, et al. Swissfel: the swiss x-ray free electron laser. Applied Sciences, 2017.
Aaron Mishkin, Frederik Kunstner, Didrik Nielsen, Mark Schmidt, and Mohammad Emtiyaz Khan. Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient. In Advances Neural Information Processing Systems, 2018.
Radford M. Neal. Bayesian Learning for Neural Networks. Springer- Verlag, 1996.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Leaming Algorithms. arXiv, 2018.
Emilio HarradoHHernande Amiran Ambroladze, John Shawe-Taylor, and Shiliang Sun. PAC-Bayes Bounds with Data Dependent Priors. Journal of Machine Learning Research, 2012.
11

---

Under review as a conference paper at ICLR 2021
Tim Pearce, Russell Tsuchida, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Expressive priors in bayesian neural networks: Kernel combinations and periodic functions. In Uncertainty in Artificial Intelligence, 2020.
Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In Interna- tional Conference on Machine Learning, 2014.
Yunxiao Qin, Weiguo Zhang, Chenxu Zhao, Zezheng Wang, Hailin Shi, Guojun Qi, Jingping Shi, and Zhen Lei. Rethink and redesign meta learning. arXiv, 2018.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference Machine Learning, 2015.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling, In International Conference on Learning Representations, 2018.
Jonas Rothfuss, Fabio Ferreira, Simon Boehm, Simon Walther, Maxim Ulrich, Tamim Asfour, and Andreas Krause. Noise Regularization for Conditional Density Estimation. arXiv preprint arXiv: 1907.08907. 2019a.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal mota-policy search. In International Conference Learning Representations, 2019b.
Jonas Rothfuss, Vincent Fortuin, and Andreas Krause. PACOH: Bayes-Optimal Meta-Leaming with PAC-Guarantees. arXiv, 2020.
Juergen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The meta-meta-.. hook. PhD thesis, Technische Universitaet Munchen, 1987.
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In Computing in Cardiology, 2012.
Jake Snell. Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In Advances in Neural Information Processing Systems, 2017.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning, 2009.
Nicholas Syring and Ryan Martin, Calibrating general posterior credible regions. Biometrika, 2018.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of samples. Biometrika, 1933.
Sebastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, 1998.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in Bayesian neural networks at the unit level. In International Conference on Machine Learning, 2019.
Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. arXiv, 2019.
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swikatkowski, Linh Tran, Stephan Mandt, Jasper Snock, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good the bayes posterior in deep neural networks really? arXiv, 2020.
Christian Widmer, Nora C Toussaint, Yasemin Altun, and Gunnar Rätsch. Inferring latent task structure for multitask learning by multiple kernel learning. BMC bioinformatics, 2010.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A PROOFS AND DERIVATIONS
A.1 PROOF OF THEOREM 2
Lemma 1. (Change of measure inequality) Let f be random variable taking values in set A and let X1, .... X₁ be independent random variables, with X E A with distribution pik. For functions Ok A A + IR, k 1, ...g 1, let Ex(f) E Ex [g (f, Xx)] denote the expectation of 9k under jal for any fixed f E A. Then for any fixed distributions n, P E MM(A) and any > O, we have that
Es~p
Σ sx(f) - (x)
$ (DKi(OlIm) + In (E Ea(l) 6a(1) (a
Proof of Theorem 2
To prove the Theorem, we need to bound the difference between transfer error L(2,7) T) and the empirical multi-tusk error L(Q, S1, Sn). To this end, we introduce an intermediate quantity, the expected multi-task error:
1 Σ E. [S( D.)
Dn) = Ep
(10)
In the following we invoke Lemma I twice. First, in step 1, we bound the difference between Ž(Q Di.,, Dn) and É(Q, S1. ..., S.). then, in step 2, the difference between C(Q, T) and [(2, Di..., Finally, in step 3, we use a union bound argument to combine both results.
Step (Task specific generalization) First, bound the generalization error of the observed tasks т, = (D;,m,), i = n. when using a learning algorithm Q M x z + M, which outputs posterior distribution Q, (S,, P) over hypotheses 8, given prior distribution P and dataset S, D of size m,. In that, define m = (E m ) the harmonic mean of sample sizes.
In particular, we apply Lemma the union of all training sets S U St with ( - m. Hence, each X corresponds to one data point, i.e. X₃ = and μ4x = D,. Further, we set == (P, hi,..., h.,) h. to be tuple of prior and n base hypotheses. This be understood as a two-level hypothesis, wherein constitutes a hypothesis of the meta-learning problem and h; hypothesis for solving the supervised task Tæ. Correspondingly, we take = (Q. Q") P. II P and p (2, )") Q II;=1 Qi joint two-level distributions and g (f, mm,, 1(h,, Zij) as summand of the empirical multi-tast error. We can invoke Lemma to obtain that [1 and 14
n
e [C(Q,, D.)] < Σ
Ep [C(Q,, ( (De
n
(11)
+ In EpapBan [e (.1.))))...)))0)..
Using the above definitions, the KL-divergence term can be re-written in the following way:
(RL((Q."1)) (OPPP))((PPPP))) = Ep~e EnNQ, In (PP) (1)(1)
(12)
P(P) I P(h)
In Q(P)
In Qi(h)
=Ep
Ep_e En-q.
(13)
P(P)
P(h)
= =Kk((0(P)) + Ep (2:I(P
(14)
13

---

Under review as a conference paper at ICLR 2021
Using 11
and 14)
bound the expected multi-task error follows:
(i, (1 S1. S.) + DKL(211P) + Σ E (PL((Q))))))
(15)
E e (ith,, ((h,,,)
(A)
Step 2 (Task environment generalization) Now, we apply Lemma on the meta-level. For that, treat each task as random variable and instantiate the components Xx Ti, l = and jlg 'T Furthermore, we set p = 2. π = P. P and (x) = [Q; SEQQ,,,, D.). This allows us to bound the transfer error as
C(Q.T) S ( E(O.D) Dz) + 3 + (B)
(16)
wherein ("(3) In Epp
Combining
15 with
16 we obtain
- - (xl(())P))
((Q,T)
(17)
n
Epwe [Dxz T'(A) (3)
Step 3 (Bounding the moment generating functions)
(x)+)))" =EpaP
en (4EE) 1/A
E Ep-pEEpr
1/8
(18)
= EP-P
1/a
E Ep-PEEmp
eam,
(C(N,
Case I: bounded loss
If the loss function ((h,, Zij) is bounded in [ak, by], we can apply Hoeffding's 's lemma to each factor
in (18). obtaining:
(3)+)
S E Eppp [
e (br -a)?
1/8
EErppEEp coom (by -a 1/2
(19)
el 8m 8mm
(bbe 2
(20)
Next, we factor out /n from and 8, obtaining
(())+( = - (crlA)))+TM(Vm)) to2
(21)
Using
[ ((AN))+T" S se( ( + (b)-
E...E....
(22)
apply Markov's inequality w.r.t. the expectations the task distribution T and data distributions D, to obtain that
B (br -a-)))² )x)2
8nm (bx 2 1 In
T(()) (8) <
(23)
8n
Vn
v1(18)
*a())
14

---

Under review as a conference paper at ICLR 2021
with probability at least - 8.
Case II: sub-gamma loss
- U(h,, First, assume that, Vi ...,, the random variables V - 25,)) are sub- sub-gamma with variance factor of and scale parameter under the two-level prior (P, P) and the respective data distribution D, That is, their moment generating function can be bounded by that of Gamma distribution [(ff,c) cr):
7 2((1 97)
EgnDD CepppBBnr (e(()....)) & exx
by (0,1//))
(24)
Second, we assume that, the random variable V == (.DS) (CQQP,)))) ((QQ)P,)) sub-gamma with variance factor si and scale parameter cn under the hyper-prior P and the task distribution T. That is, its moment generating function can be bounded by that of Gamma distribution [(s cu):
2
((p ler Bww.swwtLLLKeeee))00---....
exp
Vy E (0, 1/cn)
2((1
am7)
(25)
These two assumptions allow to bound the expectation of 188 as follows:
E [10)
As
< exp
2mmm(1 aa/(annni
eep
2m
(/33m)
(26)
Next, we factor out v from and 8, obtaining
e (()++( -( (oTLANA)+*M()N)) tt2
(27)
Finally, by using Markov's inequality we obtain that
As
As
T'()) rT)
In
(28)
2nii (1 ag/(nmm)
2n(1 3/33n)
Vn
(()
with probability at least - 8.
To conclude the proof, we choose À mmm and B vn.
A.2 PROOF OF COROLLARY
Lemma 2. (Catoni 2007) Let be a set, g A - R a function, and E M(A) and a E M(A)
probability densities over A. Then for any and Va E A.
(a(a)= = TTaee Ag(a)
T(a)e Ao(a)
(29)
Z
F Earn e Ag(aa]
is the minimizing probability density
arg BEarp [9(a)] Dxz(pllT).
(30)
pEM(1A
15

---

Under review as a conference paper at ICLR 2021
Proof of Corollary [
When we choose the posterior Q as the optimal Gibbs posterior Q7
Q" (S,, P). follows that
2(2,5), Sn)
1
Σ
18
2 [RI((]]1P)]
(31)
11
Vmn
S()h
1
(Epwe (PxL (QI(PP))))
(32)
vm
(h)
Bp-elBinei
Vin St) + In P(h)
(33)
m
U(h, - In P(hje ..n Ei= ((h,zz))
1
ER-oBEnni
(34)
Vm j=1
P(h) P)
7
(-Ep~e [In 2(S,,
(35)
n
This allows to write the inequality in (3) as
1 +
((2,T)
E In Z(S;, P)]+ +
(n
DKL( + C(ō, n, m)
n
nvm
(36)
According to Lemma
2 the Gibbs posterior (*, P) is the minimizer of 33 in particular VP
M(H), vi = 1,...n
Q', P) P(h)e-Viican VML((N..))
arg min Ea-Q (ch, + QEM(H)
1 - (QIPP)
(37)
7(S;, P)
mm
Hence, we can write
1 +
((2,7) $
Ep-o In 2(S,, P)]+
+
(x C(6,,, m)
11
m
Vn
nym
(38)
1
E
min QEM(H)
((Q,S)) + DKL (Q11()
(39)
71
Vm
1
+ nC(,n,,))
(40)
nvm
Epae ((0,)))
(41)
Vm
1
1
DKL(O]IP) + D n))
(42)
nvm
(1+= + n (x
1
S1, Sn)
(43)
Ep-o +
(44)
which proves that the bound for Gibbs- base learners in 36 and is tighter than the bound in Theorem 2 which holds uniformly for all E M(H)..
16

---

Under review as a conference paper at ICLR 2021
meta-training
target training
target testing
= I...,P
T
S
D
S
k
Meta-
Base learner
P
y
learner
Figure S1: Overview of our meta-learning framework with environment T, meta-task distributions D; target task distribution D, hyper-prior P. hyper-posterior Q, target prior P, target posterior Q datasets S, and data points
(z,y).
A.3 PROOF OF PROPOSITION D PAC-OPTIMAL HYPER-POSTERIE
An objective function corresponding to reads as
J(Q)= Eg mu In Z(,, P) + (KI(P))P).
(45)
To obtain J(Q), omit all additive terms from that do not depend and multiply by the scaling factor nym Since the described transformations are monotone, the minimizing distribution of J (Q),, that is,
Vmn+1*
arg min (()),
(46)
(EM(M((1))
is also the minimizer of 4.. More importantly. J( is structurally similar to the generic minimization problem (30). Hence, we can invoke Lemma with = M(H), g(a) = - -E In (S,,
Vnñ+ii" to show that the optimal hyper-posterior is
(P(P) = P(P) exp Vnm+1 Eis In In,
(47)
wherein
In Z(S,, 2(S p))
(S,, P) = Ep-p
exp
Vnm
1=1
Technically, this concludes the proof of Proposition result:
However, we want to remark the following
we choose Q = Q", the PAC-Bayes bound in 44 can be expressed in terms of the meta-level partition function 2 that
(2h
1 (S (S,...., Sn, P) + C(o, m)
( T) <
(48)
Vn nvm
We omit detailed derivation of
48 since it is similar the one for Corollary
B PACOH-NN: A SCALABLE ALGORITHM FOR LEARNING BNN PRIORS
In this section, summarize and further discuss proposed meta-leaming algorithm PACOH-NN An overview of our proposed framework is illustrated in Figure ST Overall, it consists of two stages meta-training and meta-festing which explain in more details in the following.
17

---

Under review as a conference paper at ICLR 2021
B.1 META-TRAINING
The hyper-posterior distribution Q that minimizes the upper bound on the transfer error is given by
(P) P(P) exp
Vnm++ Ei= In ZS, P)
(49)
(" Sn P)
Provided with a of datasets $1 ..., S2, the meta-learner minimizes the respective meta-objective, in the case of PACOH-SVGD, by performing SVGD on the Q Algorithm | outlines the required steps in more detail.
Algorithm 1 PACOH-NN-SVGIE meta-training
Input: hyper-prior P, datasets Si,..., Sn, kernel k(:,)). step size n, number of particles
(ØI...., Øx ~ p
// Initialize prior particles
while not converged do for k 1,.., K do
{0, ........ 0r.] } Pon
I/ sample NN-parameters from prior
for 1...,, do
2(,; Par)) + LSE (- (-Vm,L(0),ss)) In
// estimate generalized MLL
6z (ox) Vo. In P(0k) Vnm+1 Els Vez In Z(,, Pot) klow, Vou In (dw) + V oz K(k ox)]
// compute score
vk E [K] 6x Φx N Ex=1
// SVGD
Output: set of priors (Po Pax
Alternatively, to estimate the score of V (ov) we can use mini-batching both the task and the dataset level. Specifically, for given meta-bater batch size of and a batch size of mis, we Algorithm
Algorithm 2 PACOH-NN-SVGII mini-batched meta-training
Input: hyper-prior P. datasets St,..., S,
Input: kernel function k(,,)) :), SVGD step size number of particles K
(ó1...., ΦK} P
Initialize prior particles
while not converged do (T..... Tno, C
sample Bhs tasks uniformly random // sample més datapoints from Sy, uniformly at random
for i 1, ..., Mbs do
5; + [21,... 2mso S Sr:
for k 1,. K do
[0,, .... 0n} ~ Por
I1 sample NN-parmmeters from prior
for 1,... Tbbs do
In 7(55, Pon. LSE (- (-V,LL(0,,.)) - In T // estimate generalized MLLI
V 6k (06) f Vol In P( Vnm++1 n E Vo, In (s,, Poc) // compute score
Φx Ox Σk K
[K(w, dz) In (Øx) V k(o 2x) Vk [K] update
part.
Output: set of priors Poxt
B.2 META-TESTING
The meta-learned prior knowledge is now deployed by a base learner. The base learner is given a training dataset ~ D pertaining to an unseen task t = (D.m) m) ~ T. With the purpose of approximating the generalized Bayesian posterior Q* P), the base learner performs (normal) posterior inference. Algorithm 3 details the steps of the approximating procedure referred to as target training when performed via SVGD. For data point x=" the respective predictor outputs
18

---

Under review as a conference paper at ICLR 2021
probability distribution given as p(y" |=",,, K-L E Els p(y" (ho)(=)), We evaluate the
quality of the predictions on a held-out test dataset S D from the same task, in targer testing phase (see Appendix
Algorithm 3 NN-SGD- meta-testing
Input: set of priors Input: kernel function )), SVGD step size v, number of particles
Pox }. target training dataset S, evaluation point x
for k 1,.... K do
10% 02_ Por
// initialize NN posterior particles from k-th prior
while not converged do for = I,.., do
QO t Vel In Pox (0+)) + v V L),
I/ compute score
of c Of F, E k(0p 0f Vo Q + Voy k10 04) 44 [4]
|/ update
particles
Output: set of NN parameters 10
10 0
B.3 PROPERTIES OF THE SCORE ESTIMATOR
Since the marginal log likelihood of BNNs is intractable, have replaced it by numerically stable Monte Carlo estimator In (S,, Po) in 7 in particular
L Σ
In (S, Pb) = In L e (
LSE (-Vm,((0),s)) -in L, O1 ~ Po. (50)
Since the Monte Carlo estimator involves approximating expectation of exponential, it is not unbiased. However, we can show that replacing In Z(S,, Po) by the estimator In (S:,, P&), we still minimize a valid upper bound on the transfer error (see Proposition 3).
4 by the Monte Carlo estimate In (S,, P) = In Ei= e Or P still yields valid upper bound of the transfer enror. In particular, it holds that
Proposition 3. In expectation, replacing In (S.,, Pb)
1 "
E In 7(S,, + (1) +
1
((2,T) S
DKI(2 + C(6, (51)
avm
n1
< 1
Epae Eo ED In 2( P)]]
n
(52)
(kI(2 C(8,n.
Vn nvm)
Proof. Firsts, we show that:
Eg Br-P In Z(S, P)]
1 Σ
e -m)
In
L
Eaaw (
In
In Eg- [ e-mii((o.s.)
In (S
(53)
which follows directly from Jensen's 's inequality and the concavity the logarithm. Now, Proposition 3
follows directly from 53
19

---

Under review as a conference paper at ICLR 2021
In fact, by the law of large numbers, is straightforward to show that L oc, the In (S,,, P) -n, In Z (S;; P), that is, the estimator becomes asymptotically unbiased and we recover the original PAC- Bayesian bound (i.e. 52
a.s. 51 ). Also it is noteworthy that the bound in (2 we get by our estimator is, in expectation, tighter than the upper bound when using the naive estimator
(S. : - -mm L
Si)
Bt ~ Ps
1=1
which can be obtained by applying Jensen's 's inequality to In Eo-Po (m-
In the edge = our LSE estimator In (S,, P) falls back to this naive estimator and coincides expectation with [I 2(S,, P)] = -mm Eo St)- As a result, we effectively minimize the looser upper bound
Eg (((0,S)) + V
+ (++++ / +(((
C(2, 7) <
(54)
1=1
In phplusy 0) 0)
n
+ - +v²+ nym) ) (kL(21IP)+ + C(6, m)
= -
nm
j=1
(55)
As can see from 55 the boundaries between the tasks vanish in the edge case of L 1, that is, all data-points are treated as if they would belong to one dataset. This suggests that should be chosen greater than one. In experiments, used L 5 and found the corresponding approximation be sufficient.
C EXPERIMENTS
C.I META-LEARNING ENVIRONMENTS
In this section, we provide further details on the meta-learning environments used in Section Information about the numbers of tasks and samples in the respective environments can be found in ST
Sinusoid
Cauchy
SwissFEL
Physionet
Berkeley
n
20
20
100
36
m;
5
20
200
4-
288
Table S1: Number of tasks and samples per task m, for the different meta-learning environments.
C.1.1 SINUSOIDS
Each task of the sinusoid environment corresponds to a parametric function
fa,b,,o(z) = * x - a sin(1.5 (a - -b)) +
(56)
which, in essence, consists of affine well sinusoid function. Tasks differ in the function parameters (a, b, 3) that are sampled from the task environment T as follows:
1(07, b~ b~N(00,..13. c~~((5.000.1))
(57)
Figure S2a depicts functions Ja, with parameters sampled according to (57 To draw training samples from each task, we draw x uniformly from ((-55, 5) and add Gaussian noise with standard deviation 0.1 to the function values f(x):
x u(-5,5)
y ~ N(Jaaa..(s))
(58)
20

---

Under review as a conference paper at ICLR 2021
30-
2.0
15
(a) Sinusoid tasks
(b) Cauchy tasks
Depiction of tasks (i.e., functions) sampled from the Sinusoid and Cauchy task environ- ment, respectively. Note that the Cauchy task environment is two-dimensional (dim(X') 2), while (b) displays a nieediinniionll projection.
Figure S2:
C.1.2 CAUCHY
Each task of the Cauchy environment can be interpreted two dimensional mixture of Cauchy distributions plus function sampled from a Gaussian process prior with zero mean and SE kernel
function k(z, x²) exp
||===111 with 0.2. The (unnormalized) mixture of Cauchy densities 21
defined as:
3
m(x) T. (1 + || 11 11))2) T (1
(59)
1 122
with μi (-1,-1) and μ22 (2, 2)
Functions from the task environments are sampled follows:
f(x) m=) + g(x),
GP(O, ((x,,)(
(60)
Figure S2b depicts a one-dimensional projection of functions sampled according to (60) training samples from each task, draw x from truncated normal distribution and add Gaussian noise with standard deviation 0.05 to the function values fx)):
To draw
I = min mmax 2, 2}. -3}. ă ~ NN(,
~~ ((f(()))0005)
(61)
C.1.3 SWISSFEL
Free-electron lasers (FELs) accelerate electrons to very high speed in order to generate shortly pulsed laser beams with wavelengths in the X-ray spectrum. Thess X-ray pulses can be used map nanometer scale structures, thus facilitating experiments in molecular biology and material science. The accelerator and the electron beam line of FEL consist of multiple magnets and other adjustable components, each of which has several parameters that experts adjust to maximize the pulse energy Kirsshhnrr et al. 2019a). Due do different operational modes, parameter drift, and changing (latent) conditions, the laser's ' pulse energy function, in response to its parameters, changes across time. As result, optimizing the laser's parameters recurrent task.
Overall, our meta-learning environment consists of different parameter optimization runs (i.c., tasks) on the SwissFEL, an 800 meter long laser located in Switzerland (Milne et al.) 2017). A picture of the SwissFEL is shown in Figure S3 The input space, corresponding to the laser's parameters, has 12 dimensions whereas the regression target the pulse energy (I-dimensional), For details on the individual parameters, we refer to Kirschner et al. (2019b). For each run, we have around 2000 data points. Since these data-points are generated with online optimization methods, the data are non-ii.d. and get successively less diverse throughout the optimization. As meta-learning with limited data and want to avoid issues with highly dependent data points, we only take the first 400 data points per run and split them into training and test subsets of size 200. Overall,
concerned with
21

---

Under review as a conference paper at ICLR 2021
Figure S3: Accelerator of the Swiss Laser (SwissFEL.)
we have 9 runs (tasks) available. 5 of those runs are used for meta-training and the remaining 4 runs are used for meta-testing.
C.1.4 PHYSIONET
The 2012 Physionet competition (Silva et al. 2012) on the intensive care unit (ICU). Each patient stay consists of a time series over 48 hours, where up 37 clinical variables are measured. The original task in the competition patient mortality, but due to the large number of missing values (around 80 % across all features), the dataset also popular as a test bed for time series prediction methods, especially using Gaussian processes (Fortuin et al. 2019).
published open-access dataset of patient stays
binary classification of
In this work, we treat each patient as a separate task and the different clinical variables as different environments. We use the Glasgow coma scale (GCS) and hematocrit value (HCT) as environments for our study, since they are among the most frequently measured variables this dataset. From the dataset, we remove all patients where less than four measurements of CGS (and HCT respectively) are available. From the remaining patients we use 100 patients for meta-training and 500 patients each for meta-validation and meta-testing. Here, each patient corresponds to task. Since the number of available measurements differs across patients, the number training points m, ranges between and 24.
C.1.5 BERKELBY-SENSOL
We use data from 46 sensors deployed in different locations at the Intel Research lab in Berkeley (Madden 2004). The dataset contains 4 days of data, sampled at 10 minute intervals. Each task corresponds to one of the 46 sensors and requires auto-regressive prediction, in particular, predicting the next temperature measurement given the last 10 measurement values. In that, 36 sensors (tasks) with data for the first two days data for the last two days are employed for meta-testing. Note, that we separate meta-training and -tstting data both temporally and spatially since the data non-i.i.d. For the meta-testing, we the 3rd day as context data, i.c. for target training and the remaining data for target testing.
for meta-training and whereas the remaining 10 sensors with
C.2 EXPERIMENTAL METHODOLOGY
In the following, we describe our experimental methodology and provide details on how the empirical results reported in Section were generated. Overall, evaluating meta-learner consists of two phases, meta-training and meta-testing, outlined in Appendix B The latter can be further sub-divided into target training and target testing. Figure SI illustraaes these different stages for our PAC-Bayesian meta-learning framework.
The outcome of the training procedure is approximation for the generalized Bayesian posterior Q', P) (see Appendix ), pertaining to an unseen task 7 = (D, m) ~ T from which we observe a dataset 5 D. In target-testing, we evaluate its predictions on a held-out test dataset 5* ~ D from the same task. For PACOH-NN, NPs and MLAP the respective predictor outputs a probability distri- bution p0yy ((y"a"" [2x"" .5) for the x² in S*. The respective mean prediction corresponds to the expectation
22

---

Under review as a conference paper at ICLR 2021
of p, that is i E(y" (,", In the of MAML, only prediction is available. Based on the mean predictions, we compute the root mean- squan- evvor (RMSE):
RMSE
Σ (="
(y* -
(62)
Note that unlike e.g. Rothfuss et al. (2019a) who report the test log-likelihood, we aim to measure the quality of mean predictions and the quality of uncertainty estimate separately, thus reporting both RMSE and calibration error.
and the calibration error (see Appendix C.2.1
The described meta-training and meta-testing procedure is repeated for five random seeds that influence both the initialization and gradient-estie -estimatess of the concerned algorithms. The reported averages and standard deviations are based on the results obtained for different seeds.
C.2.1 CALIBRATION ERROR
The concept of calibration applies to probabilistic predictors that, given new target input produce a probability distribution p(y (z;) over predicted target values yi. Corresponding the predictive density, we denote a predictor's 's cumulative density function (CDF) as F(y,læ,) = f plylæ:)dy.
For confidence levels gn <.. < qH I 1, we can compute the corresponding empirical frequency
Huj (yg)(r)) S gh. 1, m
4n.
(63)
m
based on dataset S = {xx 4/) ) of m samples. If we have calibrated predictions we would expect that Gr. + qh as m + oc. Similar to (Kuleshov et 2018), we can define the calibration error as function of residuals @n- Qh. in particular,
H 9g. 9h.
1
calib-err
(64)
Note that we while (Kuleshov et al. 2018 reports the average of squared residuals 14x 2h 12 we report the average of absolute residuals Q gal in order to preserve the units and keep the calibration error easier interpret. In our experiments, compute (64) with K 20 equally spaced confidence levels between 0 and 1.
C.3
SELECTION
For each of the meta-environments and algorithms, we ran a separate hyper-parameter search to select the hyper-paranee parampaareettrs. In particular, we use the prrperopt[ppccagee (Bergstra et al. 2013) which performs Bayesian optimization based on regression trees. As optimization metric, we employ the average log- likelihood, evaluated on separate validation set of tasks.
The scripts for reproducing the hyper-parameter search are included in our code repository For reported results, we provide the selected hyper-parameters and detailed evaluation results under [Link wi 11 be made added upon acceptance]
C.4 META-LEARNING FOR BANDITS VACCINE DEVELOPMENT
In this section, we provide additional details the experiment in Section 6.2
We use data from Widmer et al. (210)) which contains the binding affinities (ICso values) of many peptide candidates to seven different MHC-I alleles. Peptides with ICso 500nM considered non-binders, all others binders. Following Krause & Ong (2011), we convert the IC5o values into negative log-scale and normalize them such that 500nM corresponds to zero, i.e. - log + (500) with is used as reward signal of our bandit.
'htt::/hhyproopt.gii ittthh..//hyerrop/// [I Link will be made added upon acceptance]
23

---

Under review as a conference paper at ICLR 2021
Allele
A-0202
A-0203
A-0201 3088
A-2301
A-2402
m1 1446
1442
103
196
Table S2: MHC-I alleles used for meta-training and their corresponding number of meta-training samples my.
We use 5 alleles to meta-learn a BNN prior: The alleles and the corresponding number of data points, available for meta-training are listed in Table S2 The most genetically dissimilar allele (A-6901) is used for our bandit task. In each iteration, the experimenter (i.c. bandit algorithm) chooses to test one peptide among the pool of 813 candidates and receives r as a reward feedback. Hence, we are concerned with a 813-arm bandit wherein the action a, E {1,..., 813) A in iteration t corresponds to testing ar-th peptide candidate. In response, the algorithm receives the respective negative log-IC.0 reward r(ar).
As metrics, we report the average regret
T Σ
Reeg.
maxr(a)
r(as)
aeA
T
1=1
and the simple regret
R
Reimple
max maxx(())
max r(az) I=....,?
aEAA
To ensure a fair comparison, the prior parameters of the GP for GP-UCB and GP-TS are meta-learned by minimizing the GP's marginal log- ligelike on the five meta-trainin veta-training tasks. For the prior, we use constant mean function and tried various kernel functions (linear, SE, Matern). Due to the 45-dimensional feature space, we found the linear kernel to work the best. So overall, the constant mean and the variance parameter of the linear kernel are meta- learned.
24


Reviews:
[{"review_id": "mbx-2Ozp4x", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission has significant overlap with the following reference: \n\nJonas Rothfuss, Vincent Fortuin, and Andreas Krause. PACOH: Bayes-Optimal Meta-Learning with\nPAC-Guarantees. arXiv, 2020.\n\nAssumably, the submission is from the same authors as theoretical analyses and experiment setups are almost identical. The main difference I can see is the Bayesian baseline model is Bayesian neural nets while the reference used Gaussian processes. The rating I gave is because of this main concern on overlapping content. \n\nThe analysis of PAC-Bayesian bounds led to the conclusion that Gibbs posterior can help minimize the bounds in the meta-learning settings. With Gibbs base learner, the bound expressed with the corresponding partition function can be done more efficiently compared to existing PAC-Bayesian meta-learners, including MLAP. \n\nMy other concerns are as follows: \n\n1. The authors may need to make sure about the main contributions of this submission with most of theoretical results already presented in the referred paper. If Bayesian neural nets are indeed showing much better empirical performances compared to GPs, the authors may need to emphasize that and discuss thoroughly with performance comparison. \n\n2. The final solution to derive Gibbs posterior is just deriving the model posterior given the hyper-prior using the data from all training tasks. If that is the case, the authors may present it clearly and discuss the connections as well as pros and cons to other existing meta-learners, for example, the existing metric-based methods that reweighting the loss across training tasks. \n\n3. The experiments may need to be done more consistently. For example, it is not clear why figure 1 only compares PACOHNN with a vanilla BNN using a simply prior instead of, for example, comparing with MLAP or PACOH with GPs? For classification examples, the authors did not compare with neural processes as claimed at the beginning of section 6.1. In section 6.2, why max regrets are smaller than average regrets in figure 3? How many training samples were used for the five tasks to get the reported results? \n\n4. The writing of theoretical analyses (main text, but especially appendix) can be improved. There are typos and confusing math notations. For example, in section 5,  above equation 7, \"\\epsilon_l j \", where j appears to be a typo? In A.1, \"to proof the theorem\" should be \"to prove the theorem...\"; in step 1 and 2 of the proof, X_k was used to denote the data point and then task, respectively, which is indeed confusing. It is not clear either what the subscript k was used for. Cleaner math notations, better organization, and careful rewriting may significantly help the readability of the paper. \n\n5. The authors may want to add complexity analysis of PACOH-NN. It is in fact quite confusing to me why it \"maintains a nearly constant memory and compute load as the number of tasks grow.\" as stated since the number of tasks does factor in based on Algorithm 1 in Appendix B. Clearer setup descriptions with complexity analysis and discussion are needed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "kwcSlote-9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for learning BNN priors based on optimising a PAC-Bayes bound for meta-learning, which they call PACOH-NN. This extends previous work on optimising PAC-Bayes bounds (PACOH), and is attractive for being principled in construction and effective in practice. The main claims are that fewer tasks are needed to achieve good performance in terms of both utility and calibration, and that the algorithm comes with performance guarantees.\n\nIn the empirical analysis, they firstly show that the method is competitive in terms of utility (for both classification and regression) with baseline methods, but with superior calibration properties. Meanwhile the method is also superior in terms of computational scaling. They also demonstrate the effectiveness on a bandit learning task for vaccine development. Along with proofs, extensive details are given for the experimental setup and training algorithms.\n\n\nStrengths:\n- Principled approach to meta-learning for BNNs\n- Efficient algorithm with good scaling properties\n\nWeaknesses:\n- A little hard to read in places\n- Performance guarantees never shown\n\nSpecific comments:\n- One of the main motivations for the paper is that the method comes with performance guarantees. E.g. in the related work section\n- The use of Hoeffding’s lemma could be very loose, particularly in the cases where values close to 0 or 1 are unlikely\n- In the use of the Hoeffding’s lemma, it looks like there should be an extra $m^2$ term in the numerator. Where has this term gone? Similarly, I would expect a factor of $m$ for the sub-Gamma bound in the numerator. This would carry over to the derivations for Thm2 as well, and would also affect the convergence properties\n- It seems that you have hyper-priors for the means of the BNN priors only. Are the variances always set to a default value for new tasks? Did you consider putting a hyperprior on the variances/precisions as well? This would feel more in keeping with the Bayesian flavour, and might give the ability to selectively “freeze” (softly) any weights that are stable across all tasks.\n- In Figure 1, with only a single training example, I don’t understand how PACOH-NN is able to transfer the knowledge of sinusoidal functions. Since the transfer is only at the hyperprior level, it seems to me that the only possibility is that the tasks are similar enough that the hyperpriors are actually capturing the entire distribution over tasks, so that there is almost no learning required at all on the target task. Can you comment on this?\n- You cite the Bayesian MAML paper by Yin et al. However, in the experiments, you only use the regular MAML and first order version, both of which give point estimates. A comparison to BMAML seems to be essential here\n- The experiment in 6.2 is a nice addition, and the results are quite impressive on first glance. However, since none of the baselines are meta-learners, we can’t be sure here if the difference is due to the effect of transfer learning or the superior uncertainty quantification. Another meta-learner should be included. There are several meta-bandit algorithms that are available.\n- Figure S1 in the appendix is really helpful! If possible, I would prefer to see this in the main paper\n\n\nMinor comments/Typos:\n- P1 variation -> variational\n- P2 in the BNN description, the regression model have a noise parameter but the classification model does not. Wouldn’t that imply that the classes would have to be separable (i.e. no mislabelled examples)?\n- P3 $\\mathcal{H}$ is undefined on first use. Also, in the description the prior is defined as $P \\in \\mathcal{M}(\\mathcal{H})$, but in (1) you have $P \\in \\mathcal{\\Theta}(\\mathcal{H})$ and $\\mathbb{E}_{\\theta \\sim P}$. On page 4, you then have $P \\in \\mathcal{M}(\\mathcal{\\Theta})$\n- Where does Theorem 1 appear in Alquier et al 2016? Is this the correct reference?\n- For the sub-Gamma case, you reference Boucheron et al , which is a book. Please make the reference more specific\n- Tasks are defined as $\\tau_i := (\\mathcal{D}_i, S_i)$, but it would seem more accurate to define them as $\\tau_i := (\\mathcal{D}_i, m_i)$. Not that later you have an expectation where you draw $(\\mathcal{D}, m) \\sim \\tau$ which is also consistent with the latter\n- In (7) you use LSE - mention this is LogSumExp\n- P8 you say “nearly constant memory and compute”. I’m not sure what “nearly constant” means - is it really linear, but with better scaling constants? Or is it truly constant?\n- P12 $p$ undefined in lemma 1\n- P18 (53) second line should be $\\theta_l$ rather than $\\theta_L$\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I8-dlxiWKO6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of learning data-driven priors for Bayesian neural networks. Assuming zero-centered Gaussian priors for BNNs often results in poor generalization and uncertainty quantification, whereas choosing informative priors is challenging due to limited interpretability of network weights. To be able to overcome these issues, the authors propose a meta-learning framework based on PAC-Bayesian theory, in which they optimize a PAC bound called PACOH in the space of possible posterior distributions of BNN weights. Unlike the previous approaches in the literature, their approach doesn’t rely on nested optimization schemes, instead they directly minimize PAC bound via a variational algorithm called PACOH-NN which is based on SVGD and reparameterization trick.\n\nThe main paper and its comprehensive appendix are well written mostly, especially the PAC-Bayes review is very concise and elaborate for a conference paper. After the introduction of related concepts about PAC-Bayes bounds and meta-learning, their application to the derivation of PACOH bound for finding a hyper-posterior expressed so naturally. Proposed methodology sounds, and claims of the authors seem correct. Experimental results both on synthetic and real-world data seem fine, and there seems to be no fallacy in empirical evaluations. Also, experimental methodology is discussed thoroughly both in the main paper and appendix. Empirical results from various experimental setups are sufficient to show that their framework improves both predictive accuracy and the uncertainty estimates compared to several popular meta-learning approaches with additional computational advantages.\n\nAs a researcher with superficial knowledge of the field, I didn't have a major difficulty in following the paper, despite the use of heavy mathematics in methodology. Mathematical notation is clear and consistent (except the usage of $M(\\Theta)$ and $M(H)$ interchangeably). Derivations in both the main paper and the appendix are easy to follow. Also, relation to prior work is clearly addressed, relative advantages and disadvantages of the proposed methodology are discussed.\n\nIn my opinion, the problem of meta-learning informative priors for Bayesian neural networks will certainly be of interest to the ICLR community. On the other hand, paper’s contribution to the work of Rothfuss et al. seems limited. Authors state that the main contribution of the paper is their practical variational algorithm for PACOH bound. In order to tractably optimize PACOH for a BNN, they employ a Stein gradient variational descent algorithm by utilizing reparameterization trick. However, the specifics of their algorithm are presented very quickly, which makes it difficult to grasp the actual novelty of PACOH-NN. For instance, neither the advantages of this particular optimization scheme over the previous methods, nor the theoretical guarantees regarding convergence and approximation quality for PACOH-NN aren’t discussed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BmQha9LMPcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\n\nOne of the main issues that BNNs have to face is the choice of good informative priors in order to provide precise information about the uncertainty of predictions. The present work connects BNNs and PAC theory to formulate a new system to obtain a general-purpose approach for obtaining significative priors for Bayesian NNs. This is done by employing the closed-form solution for the PAC-Bayesian meta-learning problem. The meta learner here learns a hyper-posterior over the priors of the parameters of the BNN by using the closed-form expression PACOH  (PAC Optimal Hyper-Posterior). This is applied in the context of NNs, where the priors are to be defined over the NN parameters. Extensive experiments are carried out to show the performance both in regression and classification datasets, as well as its scalability. In all of these regards, the system seems to be competitive, improving on the results of previous state-of-the-art methods and producing promising results in real-world problems. \n\n##### Pros:\n\n* The method does not employ nested optimization problems, therefore avoiding all the issues related to these approaches altogether.\n* The usage of Bayesian seems to point in the right direction since the Bayesian framework allows for an easy formulation of the different levels needed here to formulate the system. \n* The construction of the PACOH closed-form expression seems innovative and relevant \n* The final system improves on the previous state-of-the-art in most of the cases here shown, and in some experiments, the improvement is very clear. Thanks to the fact that it is agnostic to the inference method in use, it presents itself as a very general-purposed approach, able to improve the predictive qualities of previous methods. \n* The article is very complete and detailed, both in the main body and in the appendix. The appendix is particularly extensive, providing insight on many of the main points of the main text. \n* The experiments conducted are exhaustive and complete, providing a very wide scope of the capabilities of the method. Detailed results can be found for every experiment and task. \n* The text is well written and comprehensible\n\n##### Cons:\n\n* The choice in section 5 of using Gaussian priors with diagonal covariance matrices is motivated by the convenience in the computations. Moreover, the hyper-prior is also modeled by a zero-centered spherical Gaussian. How does choosing these distributions affect the final results? Are there been any experiments on which the parametric distributions chosen here are different from the ones presented? Please, describe how do you think these choices may affect the results and bias the final distributions obtained. \n* The calibration error is all the information provided to quantify the quality of the predictive intervals for regression. It would be helpful to include some other quantification of this, such as the usage of CRPS or other strictly proper scoring rules. In the same line, using metrics such as the Brier score for classification tasks would help getting a more complete picture. \n* How does the complexity of the BNNs employed affect the final results? Does using more layers and/or more units improve the results? There have been recent works (e.g. Functional Variational BNNs) regarding artifacts that arise when using large BNNs which make the system not able to properly learn the data. Could these problems be solved as well with this approach since the final prior is constructed using the data? \n\n###### Other comments: \n\n* As an optional suggestion, the article is well written but can be difficult to comprehend at some points. To that end, I would suggest trying to provide qualitative descriptions of some of the quantities in the expressions that later prove to be of importance. As an example, providing some intuition on $\\psi(\\sqrt{m})$ would help to understand the expression (1). In general, I think the article would benefit from extending a bit the explanations of some parts, especially section 4. \n* Typo: section 5, paragraph 2 - \"categorigal\" \n* To have a clearer explanation of figure 1 I would try to include the description of the sinusoidal functions that is already present in the appendix section C.1, since it seems relevant to the text in section 6.1 as well.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of prior selection in Bayesian neural networks by proposing a meta-learning framework based on PAC-Bayesian theory. The authors optimize a PAC bound called PACOH in the space of possible posterior distributions of BNN weights. The method does not rely on nested optimization schemes, instead, they directly minimize PAC bound via a variational algorithm called PACOH-NN which is based on SVGD and the reparameterization trick.  The method is evaluated on experiments with both synthetic and real-world data showing improvements in both predictive accuracy and uncertainty estimates.

Initially many reviewers were positive about the paper. However, it was noticed by one reviewer that the submitted paper presents a very significant overlap with

Jonas Rothfuss, Vincent Fortuin, and Andreas Krause. PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees. arXiv, 2020.

Another reviewer mentioned that they were actually reviewing for AISTATS the above manuscript by Rothfuss et al. The ICLR program chairs were contacted for a possible violation of the dual submission policy for ICLR: 

"Submissions that are identical (or substantially similar) to versions that have been previously published, or accepted for publication, or that have been submitted in parallel to this or other conferences or journals, are not allowed and violate our dual submission policy." 

The ICLR program chairs decided that the similarities between the two papers are not enough to issue a desk-rejection. However, in the discussion period,  three reviewers out of 4 pointed out that, even though the authors did revise Sections 4 and 5 in the current version, these modifications do not seem to be strong enough to make up for the really strong overlaps between the two papers. The reviewers agreed on rejection and stated that this paper should either be merged with the Rothfuss et. al. one (assuming the authors are the same), or its content should be developed to the point of making both of them clearly distinct. 
Decision: Reject