Meta-Learning Reliable Priors in the Function Space
Authors: Jonas Rothfuss
Dominique Heyn
jinfan Chen
Andreas Krause
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transfer
Meta-Learning
Life-Long Learning
Bayesian Optimization
AutoML
Hyper-Parameter Tuning
Hyper-Parameter Optimization
Continual Learning

Abstract: Meta-Learning promises to enable more data-efficient inference by harnessing previous experience from related learning tasks. While existing meta-learning methods help us to improve the accuracy of our predictions in face of data scarcity, they fail to supply reliable uncertainty estimates, often being grossly overconfident in their predictions. Addressing these shortcomings, we introduce a novel meta-learning framework, called F-PACOH, that treats meta-learned priors as stochastic processes and performs meta-level regularization directly in the function space. This allows us to directly steer the probabilistic predictions of the meta-learner towards high epistemic uncertainty in regions of insufficient meta-training data and, thus, obtain well-calibrated uncertainty estimates. Finally, we showcase how our approach can be integrated with sequential decision making, where reliable uncertainty quantification is imperative. In our benchmark study on meta-learning for Bayesian Optimization (BO), F-PACOH significantly outperforms all other meta-learners and standard baselines.  Even in a challenging lifelong BO setting, where optimization tasks arrive one at a time and the meta-learner needs to build up informative prior knowledge incrementally, our proposed method demonstrates strong positive transfer.

Meta-Learning Reliable Priors in the Function Space
Jonas Rothfuss
Dominique Heyn
ETH Zurich rothfusstnf www.oofsssssff.eth.....mm ethz. ch
ETH Zurich heynddstudent. ethz. ch
Jinfan Chen
Andreas Krause ETH Zurich krausea@ethz.com ch
ETH Zurich
georgcjf0@mail.com
Abstract
When data are scarce, meta-learning can improve a learner's 's accuracy by harness- ing previous experience from related learning tasks. However, existing methods have unreliable uncertainty estimates which are often overconfident. Addressing these shortcomings, we introduce a novel meta-learning framework, called F-PACOFI, that treats meta-learned priors as stochastic processes and performs meta-level regularization directly in the function spuce. This allows to directly steer the probabilistic predictions of the meta-learner towards high epistemic uncer- tainty in regions of insufficient meta-training data and, thus, obtain well-calibrated uncertainty estimates. Finally, we showcase how our approach can be integrated with sequential decision making, where reliable uncertainty quantification is imper- ative. In our benchmark study on meta-learning for Bayesian Optimization (BO), F-PACOH significantly outperforms all other meta-learnrrs and standard baselines.
Introduction
Learning concepts and skills from small number of examples well as adapting them quickly in face of changing circumstances a key aspect of human intelligence. Unfortunately, our machine learning algorithms lack such adaptive capabilities. Meta-Learning 2 has emerged promising avenue towards enabling systems to learn much more efficiently by harnessing experience from previ- related learning tasks (318) By meta-learning probabilistic prior beliefs, not only make more accurate predictions when given small amount of training data, but also improve the self-assessment machine learning algorithm in the form of epistemic uncertainty estimates (H12). Such uncertainty estimates are critical for sequential decision problems such as Bayesian optimization (BO) 13 14 and reinforcement learning [6] which require efficient information gathering and exploration.
However, in most practical settings, there are only few tasks available for meta-training. Hence face the risk of overfitting to these few tasks [17] and, consequently impairing the performance on unseen tasks. To prevent this, recent work has proposed various forms of regularization on the
meta-level [8 12 {. While these methods are effective in preventing meta-overfitiing for the mean predictions, they fail to do so for the associated uncertainty estimates, manifested in gross overconfidence. Such overconfidence is highly detrimental for downstream sequential decision tasks that rely on calibrated uncertainty estimates 19 20 to perform sufficient exploration. For instance, the global optimum of the true target function in BO lies outside the model's 95 % confidence bounds the acquisition algorithm may never query points close to the optimum and get stuck in sub-optimal solutions. We hypothesize that previous methods yield overconfident predictions since they do not meta-regularize the predictive distribution directly. Instead, they perform meta- regulaaiiaiiiiooonnn in some latent space, for example the method parameters, which is non-trivially associated with the resulting predictive distribution and thus may not have the indented regularization effects.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To overcome the issue of overconfident predictions in meta-learning we develop novel approach that regularizes meta-learned priors directly in the function space. We build on the PAC-Bayesian PACOH framework [1] which uses a hyper-prior over the latent prior parameters for meta-regularivationn However, we propose to deñin the hyper-prior as stochastic process, characterized by marginal distributions in the function space, and make the associated meta-learning problem tractable by using an approximation of the KL-divergence between stochastic processes [21]. The functional KL allows us to directly steer the meta-learned prior towards high epistemic uncertainty in regions of insufficient meta-training data and, thus, obtain reliable uncertainty estimates. When instantiating our functional meta-leariing framework, referred to as F-PACOH, with Gaussian Processes (GPs), we obtain a simple algorithm that can be seamlessly integrated into sequential decision algorithms.
In our experiments, we showcase how F-PACOH can facilitate transfer and life-long learning in the context of BO, and unlike previous meta-learning methods, consistently yields well-calibrated uncertainty estimates. In our benchmark study on meta-learning for BO and hyper-parameter tuning, F-PACOH significantly outperforms all other meta-learners and standard baselines. Finally, we consider lifelong BO, where the meta-BO algorithm faces a sequence of BO tasks and needs build-up prior knowledge iteratively. In this challenging setting, F-PACOHI is the only method that is able to significantly improve its optimization performance as it gathers more experience. This paves the way for exciting new applications for meta-learning and transfer such as the recurring re-optimization and calibration of complex machines and systems under changing external conditions.
2 Background
In this section, we formally introduce IAC-Bayesian meta-learning, the foundation of the functional meta-learning framework that we develop in Section Moreover, we provide a brief description of Bayesian Optimization, which serves as the main testing ground for our proposed method.
PAC-Bayesian Meta-Learning, Meta-learning extracts prior knowledge (i.e., inductive bias) from set of related learning tasks to accelerate inference in light of a new task. In the context of supervised learning, the meta-learner is given n datasets 1, 1 Pn,T.. Each dataset D.T, = (X,
consists of T; noisy function evaluations Yi = fi(Xi,)) + corresponding to a function ft : X +
for the matrix of function inputs and y? = (Ir,,, .... M.,T.) ) for the vector of corresponding observations. The functions ft ~ T are sam- pled from a task distribution T, which can be thought of as a stochastic process that governs random function f X + Y. In standard Bayesian inference, we exogenously presume an- - often carefully manually designed prior distribution Phh) over learning hypotheses h X + V, h H and combine it with empirical data to form a posterior (hh) = P(h]D). In contrast, in meta-learning we endogenously infer the prior P(h) in a driven manner, by using the provided meta- veta-training data.
and additive noise e. In short, we write X = (x1,1 ...,,,).,,,,,;;)) J
While there exist different approaches, we focus on PAC-Bayesian meta-learning 22 12 due to its principled foundation in statistical learning theory. The PACOH framework by Rothfuss et al. 12] presumes a loss function 1(h, x, y) and a parametric family (P& l E } of priors (h) with hyperparameter space T. In a probabilistic setting, one typically uses the negative log- likelloodoood as the loss function, i.e., (hh, x. y) = - In (x)). Given the meta-training data DI,..., Dz and a hyper-prior distribution P(Ø) over 4 PACOH aims to infer a hyper-posterior distribution (()) over the parameters & of the prior. Using PAC-Bayesian learning theory (24), they derive a gigh-robabbiltyy bound on the transfer error, i.e., the generalization error for posterior inference on an unseen task f ~ T with priors sampled from the generalization bound also serves as an objective for the meta-learner, minimized w.r.t. Q:
Q(Ø). This meta-level
((2) = - Σ
1
E E [In Lp(D Po)1 + (++ (++1) nf KLIQ P| + const.
(1)
+ 1= is the geometric mean of the dataset sizes, In (D..,, P) = P(h) exp (-ET, (,,, w..) dh the generalized marginal log- likelihood and KL(Q))) the Kullback-Leible (KL) divergence between hyper-posterior and hyper-prior. To obtain asymptotically consistent bounds, À is typically chosen as X = Vn.
Here, A > 0, T = (T
Bayesian Optimization. Bayesian Optimization (BO) aims to find the global maximizer x* arg maxxex fxx of a function f X' ++ R over a bounded domain X. To obtain an estimate of x*, the BO algorithm iteratively chooses points Xi...., XT E X' at which to query f. and observes noisy

---

Vanilla GP
PACOH-MAP
F-PAGOH-MAN MAP (ours)
Samples of Task Distr. f
2.00
1.5
1.5
15-
1.75
1.0
1.25
2.0
1.00
Posterior 1.5
15
-10
0
10
1.0
10
1.0
-10
mean prediction
95% confidence Interval
target furcciinn
X
trainigg points
Figure 1: Prior posterior predictions for a Vanilla GP and PACOH / F-PACOH GP meta-traine on functions from the task distribution displayed left. While PACOH yields over-confident and the Vanilla GP under-confident predictions, F-PACOH provides the well-calibrated confidence intervals.
feedback VI. ER form a Bayesian surrogate model the function I' based on previous observations D. ((x,, M+)] Typically, Gaussian Process (GP) GP(m(x)) k(x, x') with mean vx(x) and kernel function x() employed to form a posterior belief p( (x)(()) (x) over function values (2). In each iteration, the next query point X++1 == arg max (x(x) is chosen maximizer of an acquisition function (x)x) that is typically based on the pxf(x)(D) and trades-off exploration and exploitation (26-29)
ER,eeg e.g., via y/. f(x)) +e, ~ (0, o²) with O² 13 14]. BO methods
3 Related Work
Meta-Learning.
Common approaches in meta-learning attempt to learn shared embedding space 3 32 amortize inference by a meta-learned recurrent model 35 or learn the initialization of a NN so it be quickly adapted to new tasks 6 336] A growing body of work also uses probabilistic modeling to enable uncertainty quantification meta-learning [
37 However, when only given small number of meta-training tasks such approaches tend to overfit and fail to provide reliable un- certainty estimates. The problem of overfitting to the meta- veetattaiinng tasks has been brought to attention by [7 33], followed by work that discusses potential solutions in the form of meta-regularizztion [8] Our meta-learning framework builds on a sequence of work on PAC-Bayesian meta-learning However, instead of using hyper-prion perir over the latent parameters of the learnable prior, we define the hyper-prior -riior as stochastic process in the function space which gives us better control over the behavior of the predictive distributions of our meta-learned model the absence of data.
12
1 22
Learning hypotheses in the function space. Recent work bridges the gap between stochastic processes and complex parametric models by developing variational inference in the function space 21 39 Our framework in the function space heavily builds on these ideas; in particular, the
proposed approximations of the KL-divergence between stochastic processes [2]
Meta-Learning and Transfer for Bayesian Optimization. To improve the sample efficiency of BO based previous experience, basic approach is to warm-start BO by initially evaluating configurations that have performed well on similar tasks in the past 42 KEl
Another common theme to form global GP based model across tasks [5 However, such global GPs quickly become computationally infeasible as they scale cubically in the number of tasks times the number samples per task or require hand-designed task features. Conceptually, the most similar
work is the approach of learning a neural-network based feature map that shared across tasks and used for BO based on Bayesian linear regression model in the shared feature space 49 50) While these methods are highly scalable, they lack any form meta-level regularization and thus, in face of data-scarcity, are prone overfitting and over-overconfident uncertainty estimates. In contrast, our method overcomes these issues thanks to its principled meta-regularization in the function space.
4 PAC-Bayesian Meta-Learning in the Function Space
In this section, we present our main contribution: a principled framework for meta-learning in the function space. First, we introduce the general framework, then we discuss particular instantiations and components our approach and describe the resulting algorithm. Finally, we elaborate how our proposed meta-learner can be effectively applied in the context of Bayesian Optimization.
3

---

4.1 The F-PACOH Framework: Meta-Learning with Stochastic Process Hyper-Priors
We are interested in settings where the number of available meta-training tasks as well as observations may be small, and thus generalization beyond the meta-training data is challenging. In such scenarios, regularization on the meta-level plays a vital role to prevent meta-overfitinng and ensure positive transfer [8] [7]. Due to its principled meta-level regularization grounded in statistical learning theory, we build on the PACOH meta-learning framework of [12] which has been introduced in Section 2
A critical design choice when attempting to meta-learn with the PACOH objective in (T) is the hyper-prior P which, through the KL-divergence, becomes the dominant force that shapes the learned prior P& in the absence of sufficient meta-training data. Rothfuss et al. [12] define the hyper- prior as distribution over the prior parameters @ and, in particular, use a simple Gaussian P(ø) (d;; 0, 7) While this may act as a form of smoothness regularization on the prior, it unclear how such a hyper-prior shapes the predictions of our meta-learned model in the function space, especially in regions where no training data is available. In such regions of data scarcity, we ideally want our meta-learned model make conservative predictions, characterized by high epistemic uncertainty. As we can observe in Figure [] PACOH fails to do so, yielding grossly over-confident uncertainty estimates. This is particularly problematic in sequential decision making, where data is typically non-i.i.d. and we heavily rely on well-calibrated epistemic uncertainty estimates to guide exploration.
Hyper-prior in function space. Aiming to make meta-learned models behave more predictably outside the support of the data, we devise hyper-prio prrirrroorrr in the function space. We assume that the hyper-prior P is stochastic process, indexed in X' and taking values in y i..., a random function h 1' Furthermore, assume that for any finite measurement set X [x1 Xi....,, x4] E X k N. exists and
the corresponding marginal distribution of function values p(h*) X := fulfills the exchangability and consistency conditions of the Kolmogorov Extension Theorem (5]). Similarly, we treat the prior Ps as stochastic process from which we can either sample parameters 0 that correspond to functions ha X' + y or even have direct access its finite marginals p(h*) = p(h(x)..... .... h(xx)), e.g., multivariate normal distributions in the case of a GP. Likewise, function samples from the hyper-posterior can be obtained by hierarchical sampling (he(.) with 0 ~ Pa. 6 Q) and finite marginals by forming mixture distribution (h>) = E e [p<(h>)
Characterizing the prior, hyper-prior, and hyper-posterior as stochastic processes, we need to re- consider how to define and compute K [ For this purpose, we build on the result of Sun et al. (1 who show that the Ki-divereence between two stochastic processes y and p can be expressed as a supremum of Kl.-divergences between their finite marginals:
K k[
sup "EN,XEX"
KL(g(h*) [11((1))1))(h
This supremum is highly intractable, and without further specifications on the measurement set X it does not present viable optimization objective [40 41. We thus follow a sample-based approach to computing (2), for which Sun et al. [21] provide consistency guarantees.
Enforcing the hyper-prior. A-priori, want our meta-learned priors to match the structure of the hyper-prior both near the meta-training data and in regions of the domain where no data is available at all. Thus, for each task, we build measurement sets X; P X.M by selecting random subset XP, of the meta-training inputs X.P as well random points M itd ((()) sampled independently and uniformly from the bounded domain X. In expectation over these random measurement sets, then compute the KL-divergence between the marginal distributions of the stochastic processes, giving us [x, [KL L [g(hx.) [[)(())))]]]]] ]]] as approximation of (2) Intuitively, to obtain a low expected KL-divergence, q must closely resemble the behavior our stochastic process p hyper-prior across the entire domain. Hence, in the absence of meta-training data, the hyper-prior gives us direct control over the a- a-priori behavior of our meta-learner in the function space. In Figure a Vanilla GP as hyper-prior shapes the predictions of our meta- meta-learned model.
L we can observe how
In summary, by defining the hyper-prior in the function space and using sampling- sampling-based measurement sets X, to compute the functional KI. divergence, we obtain the following meta-learning objective:
(x( n Σ
12
el Pa)1+
1 Vn
1 nT:
Ex [KL[g(h> X [
(3)
T:
marginal log likelihoud
functional KL-divergence

---

In here, the marginal log- liellikelhood forces the meta-learned prior towards good fit of the meta-training data while the functional KL-divergence pushes the prior towards resembling the hyper-prior's behavior in the function space. Since Jr(2) is inspired by the PACOH framework of [2] but works with meta-learning hypotheses in the function space, we refer algorithms that minimize (Q) as Sunctional-PACOP (F-PACOH).
4.2 Instantiations and Components of the F-PACOH framework
In the following, we discuss various instantiations of the introduced F-PACOH framework and their implications on the two main components of the meta-learning objective in 3 the (generalized) marginal log-likelihood and the functional KL-divergence.
Representing the hyper-posterior Q. To optimize the functional meta-learning objective Jr w.r.t Q we may choose variational family of hyper-posteriors (0) E =} from which we can sample & in a re- parameterrizalle manner. This allows us to obtain low-variance gradient estimates of the expectations Vg Vg [In (DD,,,, Po)] and Ve In qe (h^) E In Exg-a (px)))
Alternatively, we can form a maximum a posteriori (MAP) estimate of the hyper-posterior which approximates Q by a Dirac delta function ((6)) = 8(Φ - -) in a single prior parameter Ø. As a result, the corresponding expectations become trivial to to solve, turning ( into much simpler objective to minimize and making the overall meta-learning approach more practical. Thus, we focus on the MAP approximation in main body of the paper and discuss variational hyper-posteriors in Appx. A
The marginal log-likelihodd.
case of GPs, the marginal log-likelihood Z(D,,T, : Pa) = In p(y? XP: ②) can be computed in closed form as (see Appx. A.1 most other cases, e.g. when the hypothesis space H = {ho, 0 E )) corresponds to the parameters 0 of neural network, we need to form an approximation of the (generalized) marginal log-likelihood p(y xXP
In Eg (e- ΣT (x.)).))))
For further discussions on this matter, we refer
12 52 53
Gradients of the KL-ddvergenne. Generally, we only require re-parametrizable sampling of functions from prior. Following 21 we can write the gradients of the KL-divergence
V VoKLLp(h* Ilp(h>))
E, Voh (Vn In - Vn In p(hx)))))
(4)
Here, Voh is the Jacobian of a function sample h w.r.t. the prior parameters 0. Thus, it remains to estimate the score of the prior Vn In p. (hx) and the score of hyper-prior Vn In ps (h>),. In various scenarios, the marginal distributions p(hx) ) p(hx) in the function space may be intractable and we can only sample from it. For instance, when our prior (o) a known distribution over neural network (NN) parameters 6, associated with NN functions he : X + V, its marginals pa (h in the function space are typically intractable. In such cases, (55] to estimate the respective score from samples. Whenever the marginal densities available in closed form,
either the Spectral Stein Gradient
Estimator (SSGE) (54) or sliced score matching
automatic differentiation to compute their score. Finally, if both the prior and the hyper-prion pprio are GPs, we use the closed-form KL-divergence between multivariate normal distributions.
4.3
The F-PACOH-MAP Algorithm
When focusing on the MAP approximation of the prior's parameter vector o, arrive at simple algorithm for meta-lcarning reliable priors in the function space. After initializing Pg. we iteratively perform stochastic gradient steps on the functional meta-learning objective (() in (3) In each step, we iterate through all the meta-training tasks, com- pute the corresponding (generalized) marginal log-likelihood In ((D,,T; Pa), sample a measurement set X, and estimate the gradient of the functional KL-divergence DKL D (llpx))))l(>)))) based on X,- In accordance with (3), we compute weighted gradients over the tasks. Finally, we use our gradient estimate V rr)r(()) to perform a gradient update on ş). The overall procedure, which we denote by F-PACOH-MAP, summarized in Algorithm [
where we directly meta-learn the
of these components and average the resulting
While the proposed function-space approach brings us many benefits, also comes at a cost: Estimat- ing the expectation over measurement sets X, by uniform sampling and Monte Carlo estimation is subject to the curse of dimensionality. Hence, we do not expect it to work well for high-dimensiooal data (d > 50) such as images. For such purposes, future work may investigate alternative approxima- tion / sampling schemes that take the data manifold into account.
5

---

Algorithm 1 I-PACOH--MPP: Meta-Learning Reliable Priors
Input: Datasets ..., D,. T., parametric family {Pal E ) of priors, learning rate Input: Stochastic process hyper-prior with marginals p(-)
1: Initialize the parameters & of prior Ps
2: while not converged do
3: for i 1, 12 do
Iterate over meta-training tasks
4:
X, X Las x where X C
C X ~ ((t)
uid
4> Sample measurement set
5: Estimate or compute V [ Pa) and V V.KLipp (h* (ll(((x))))))
6:
V VoJF;: T. V In Z(,,T Po) (T + प..
V V LL Ilo(h(<))))
7: end for
8:
a Di=1 V JOJF.
Update prior parameter
9: end while
4.4 Application: Meta-Learning Reliable GP Priors for Bayesian Optimization
To harness the reliable uncertainty estimates of our proposed meta-learner towards improving sequential-decision making, we employ it in the context of BO. We assume that we either face sequence of related BO problems corresponding to target functions f1.... fn T or have access to the function evaluations from multiple such optimization runs. We use the previously collected function evaluations for meta-training with F-PACOH. Then we employ the UCB algorithm [26 27 together with our meta-learned model to perform BO on new target function T.
To be able to extract sufficient prior knowledge from the data of previous BO runs, we need to choose sufficiently rich parametrization of the GP prior (h( GP (h) (h)ma(r)), (o(2,, x')). Hence, following [56] (38), we instantiate m& and ki as neural networks (NN), where the parameter vector ő corresponds the weights and biases of the NN. To ensure the positive-defnittnnss of the kernel, use the neural network as feature map (2) on top of which we apply squared exponential kernel.
In the standard BO setting, would usually use Vanilla GP with constant mean function and a conservative SE or Matérn kernel. Hence, in the absence of sufficient meta- trainin data, we ideally want to fall back the behavior of such Vanilla GP. For this reason, with zero-mean and SE kernel with a small lengthscale as hyper-priof] Correspondingly, the marginals P (hx) (mx,, kx,²) of the prior and the hyper-prior p(h²) (O, Kx.sE) are multivariate normal distributions. Thus, the marginal log-likelihood (see Section [A. and the KL-divergence are available in closed form. We provide more details in Appx. [A.4 and Algorithm 3 Due to the closed-form computations, the resulting FPACOH-MAP algorithm for GPs has an asymptotic runtime complexity of Onn nT.) per iteration. In that, T max; T, is the maximal number of observations in one of the meta-training datasets and L = max; [X, is the number of points per mcasurement set which can be chosen freely. our experiment, we use However, BO is most relevant when function evaluations are costly and data is scarce, i.e., both T' and n are small. Thus, the cubic runtime is hardly a concern in practice. Alternatively, sparse GP approximations can be used to reduce the runtime complexity [57 58
GP prior
20.
Experiments
First, investigate the effect of our functional meta-regularizztion on the uncertainty estimates of the meta-learned model. To assess the utility of the uncertainty for sequential decision making, we then evaluate our F-PACOH approach in meta-learning for BO as well as a challenging life-long BO setting. Details on the experiments can be found in Appendix
B
5.1 Calibration and Uncertainty Quantification
To study and illustrate our proposed meta-rrggulaizzation in the function space, we use a simulated task distribution of Idiimensional functions. Random samples from this task distribution are displayed
left in Fig. [ Using F-PACOH-MAN and PACOH-MAP [12], we meta-train GP with n = 10 tasks and T 10 function evaluations per task. collected with Vanilla GC- 27). PACOH-MAP corresponds to a MAP approximation of ( with Gaussian hyper-prior on the prior parameters ø.
Note that we standardize the inputs x and observations based on the empirical mean and variance of the meta-training data. The GP prior is applied in the standardized data space

---

Fig. [ displays the prior and posterior predic- tions of the meta-learned models along those of a Vanilla GP with zero mean and SE kernel. We observe that the 95 confidence intervals of the PACOH posterior are strongly concentrated around the mean predictions, even far away from the training points. Unlike Vanilla GP whose confidence regions contract locally around the training points, the uncertainty estimates of PACOH contract uniformly across the domain, even far away from the training data. The fact that the true target function lies mostly outside of the confidence intervals reflects the over-confidence of PACOH-MAP. In contrast, while also reflecting useful meta-learned prior knowledge, the predictions of the F-PACOH model exhibit the behavior we desire, i.c., small uncertainty close to the training points, higher uncertainty far away from the data.
posterior (T=1)
posterior (T=8)
1.00
emperical coverage
0.75
0.50
0.25
0.00
0.0
0.5
1.0
0.0
0.5
10
confidence level
confidence
F-PACOH- (ours)
Vanilla GP
PACOH-MAP
perfect calibration
Figure 2: Calibration plots of posterior predictions corresponding to = and training points. Only F-PACOH yields well-calibrated predictions.
How reliable uncertainty estimates are can be quantified by the concept of calibration 19 20]. We say that a probabilistic predictor is calibrated, if, in expectation, its a % confidence intervals contain a% of the true function values. Fig- visualizes this by plotting the fraction of the true function values covered by the posterior's 's confidence intervals at varying levels of Here, the Vanilla-GP model consistently cnder-confident, while PACOH-MAL makes grossly over-confident predictions. In contrast, F-PACOH yields well-calibrated uncertainty estimates across the entire range of confidence levels. For a more thorough analysis, Table 5 in Appendix B.3. reports the calibration error for all the methods and BO environments, presented Section 5.2 There, F-PACOH yields significantly lower calibration errors than the other methods in the majority of the environments. All in all, this empirically supports our claim that, through its meta-regularizztion in the function space, F-PACO1I is able to meta-learn priors that yield reliable uncertainty estimates.
5.2 Meta-Learning for Bayesian Optimization: Setup of the Benchmark Study
We present comprehensive benchmark study on meta-learned priors and multi-task methods for BO in which we compare our proposed method F-PACOH-MAP against various related approaches. We use the UCB aquisition algorithm 266 27 across all the models.
Baselines. We compare against approaches that also meta-learn a GP prior. PACOH-MAP 12 is the most similar to our approach as it meta-learns a neural-network based GP prior. However, it uses a hyper-prior over prior parameters ø instead of our stochastic process formulation. Similarly, ABLR [50] mecanl the feature map and prior of a Bayesian linear regression model. As simple bascline, we meta-train GP prior with constant mean and SE kernel by maximizing the sum of marginal log- oggliiklihoods across tasks (Learned GP). We also compare with neural processes (NP) [11 and rank- weighted GPs (RWGP) 47 Finally, we use a Vanilla GP as a baseline that does not perform transfer across tasks and, if the regret definition permits" also report the performance of Random Search.
Simulated Benchmark Environments. We use three simulated function environments as well as three hyper-parameter optimization environments as benchmarks. Two of the simulated environments are based on the well-known Branin and Harlmannó functions for global optimization (59). Following [60]. we replace the function parameters with distributions over them in order to obtain task distribution. Another simulated function is based on the Camelback function [6]] which we overlay with a product of sinusoids of varying amplitude, phase and shifts along the two dimensions.
Hyper-Parammetr Optimization for machine learning algorithms. As practical application of meta-learning for BO, we consider different datasets. In this setting, the domain X' is algorithm's 's lyper-parameter space and the target function f(x) represents the test performance under the hyper-parameter configuration x. The different functions fi ~ correspond to training and testing the machine learning algorithm on different datasets. The goal is to gather knowledge from hyper-parameter optimizations on different datasets so that tuning of the same algorithm on new dataset can be done more efficiently.
tuning of machine learning algorithms on
Since random search performs model-based inference, can only report its simple regret.
7

---

Random Branin
Camelback Sin-Noise
Random Hartmann6
10a
10y
Simplees reghet
20
201
10-1
10
10"1
10
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Stop
stop
Step
GLMNET
PPart
XGBaost
Smple reghet
10
10-2
10-2
10:4
10"7
20
30
10
20
5.0
30
30
step
step
step
1-PACOR-MAP (ours)
PACOH- MMP
RWGG
ABLR NP
Learned
Vanilla GP
Random Search
Figure 3: Performance of BO with meta-learned models on simulated function environments (top) and pyper-parameter tuning (bottom). Reported is the simple regret, averaged over seeds and function samples, alongside 95% confidence intervals. While other methods improve slowly or saturate in sub-optimal solutions, BO based on F-PACOH consistently finds near-optimal solutions quickly.
In particular, consider three machine learning algorithms for this purpose: Generalized linear models with elastic NET regularization (GLMNET) [62]. recursively partitioning trees (RPart) 63 [64] and XGBoost 665 Following previous work [e.g. 50 60], we replace the costly training and evaluation step by a cheap table lookup based on a large number of
evaluations [66] on 38 classification datasets from the OpenML platform (67). In these aaperrparameter experiments, we use the area under the ROC curve (AUROC) as test metric to optimize.
5.3 Meta-Learning for BO: Offline Neta-Training
The first scenario consider is where have offline access to data of previous BO runs. In particular, using Vanilla GP-UCB, we collect T; function evaluations on n tasks sampled from the task distribution T. Depending the dimensionality of X' we vary and T. across the task distributions (see Tab. ii Appx. B With this meta-training data Di,T:,..., Di,T., we meta-train F-PACOH and the considered baselines. To obtain statistically robust results, perform independent BO runs on 10 unseen target functions/ tasks and we repeat the whole meta- meta--raining & -testing process for 25 random seeds to initialize the meta-leamnr. To assess the performance, we report the simple regret Tfa = f(x)) max fxx) as well as the inference regret f,, = f(x') - f({)) ). wherein x* arg max x=x (x) is the global optimum, x, the point the BO algorithm chooses to evaluate in iteration t and X2 is the optimum predicted by the model at time t.
Fig. 3 displays the results. The inference regret is reported in Fig. [ in Appx. B and reflects the same patterns. While PACOH-MAP yields relatively low regret after few iterations, it saturates in performance very early on and gets stuck in sub-optimal solutions. This supports our hypothesis that meta-learning with a hyper-prior in the parameter space leads to unreliable uncertainty estimates that result in poor BO solutions. Similar early saturation behavior can be observed for the other the other meta-learners, i.e., ABLR, NPs and Learned GPs. F-PACOH also yields good solutions quickly, but then continues improve throughout the course of the optimization. Across all the environments, it yields significantly lower regret and better final solutions than the other methods. This demonstrates that, due to our novel meta-level regularization the function space, F-PACOH achieves strongly positive transfer in BO without loosing the reliability and long-run performance of welllestablished algorithms like GP-UCB
5.4 Lifelong Bayesian Optimization
Finally, we consider the scenario of lifelong BO where we face a sequence fo...,, T of related target functions which we want to optimize efficiently, one after another. This a common problem setup when complex systems machines such as the Swiss Free-Flectron laser (SwissFEL.) (68) that are subject to external and internal effects (e.g., drift, hysteresis). As a result, the system/ / machine responds differently to parameter configurations over time and needs to be
8

---

Random Branin
Camelback Sin-Noise
RPart
150
3.00
100
2300
1000
50-
0-
50
100
150
200
1000
150
200
50
1.00
150
200
cunrubtive steps
cumulotive stops
cumultative steps
Stople regret on nuh
1.0
10-3-
k00
10-
TUT
F PACOH MAP (ours)
PACCH NAP
RW/GP
ABLR
NP
Leancd GP
Vanilia GP
Figure 4: Lifelong BO performance two simulated function environments and one hyper- prrraer tuning benchmark (RPart). The reported results are averages over seeds and random sequences of tasks alongside 95% confidence intervals. While other meta-learners struggle to achieve positive transfer, F-PACOH able to significantly improve the BO perfomance gathers more experience.
de-calibrated//ppimiized on a frequent basis. Our aim is to incrementally improve our performance from run to run by meta-learning with data collected in previous BO runs. This setup is more challenging than gathering meta- meta-trainin training offline for two reasons: 1) in the beginning, the meta-learner has much fewer meta-training tasks and 2) the meta-learned models and the collected meta-train data become interdependent, resulting in a feedback-loop between predictions and data collection spploratonnexxplottatio trade-off the meta-level,
on the task level. In short, this can be which our calibrated uncertainty helps to effectively navigate.
Overall, we sequentially conduct 10 BO runs with T; 20 steps each. In the initial BO run (i 0), we start without meta-training data, i.e. .Mo = 9 and thus use Vanilla GP-UCB. After each run, we add the collected function evaluations to the meta-training training data, i.e., M,,1 M₁ U (D.r). ). For the following runs (i > 0), we first perform meta-trai meta-training with M, and then run BO with the meta-learned model. As performance metric, compute the cumulative inference regret, i.c., the sum of inference regrets of all the previous steps and runs as well as the simple regret r 2, at the end cach run. We repeat the experiment for random sequences of target functions and 5 random model seeds each.
Figure 4 displays the results of our lifelong BO study for the Random Branin, Camelback Sin-Noise and RPart environment. Similar results for more environments can be found in Appx. B At first glance, meta-learning seems to result lower cumulative inference regret overall. However, this mainly due to the fact that the meta-learned models start with better initial predictions which, in case of most meta-learning baselines, hardly improve within a run. Similarly, we observe that the majority of meta- letannla baselines fail consistently find better solution than Vanilla GP. F-PACOH in stark contrast is able to significantly improve the BO perfomance as it gathers more experience, and finds better solutions by the end of run than the other methods. This further highlights the reliability of our proposed method. Finally, the fact that F-PACOH shows strong positive transfer despite this challenging setting is highly promising, many real-world applications may benefit from it.
Conclusion
We have introduced a novel meta-learning framework that treats meta-learned priors stochastic processes and performs meta-level regularization directly in the function space. This gives us much better control over the behavior of predictive distribution of the meta-learned model beyond the training data. Our experiments empirically confirm that the resulting F-PACOH meta-learning method alleviates the major issue of over-confidence plaguing prior work, and yields well-calibrated confidence intervals. Our extensive experiments on lifelong learning for Bayesian Optimization demonstrate that F-PACOH is able to facilitate strong positive transfer in challenging sequential decision problems - well beyond what existing meta-learning methods are able to do. This opens many new avenues for exciting applications such as continually improving the efficiency with which we re-calibratel-optiiize complex machines and systems.

---

Broader Impact
Our work focuses on meta-learning reliable priors with a small number of meta-tasks and thus a the potential to implact applications that can be cast in such as setting. Since it ensures more reliable uncertainty estimates, the proposed method is of particular interest for interactive machine learning systems that actively gather information, c.g., active learning, Bayesian optimization and (.:ioon5533))
reinforeement learning. For instance, the meta-learning for BO method featured in Section [53 and 5.4 be applied in robotics for tuning controllers more efficiently, or in biochemistry molecular medicine develop novel therapeuties through protein optimization. Other potential applications include recommender systems and targeted advertisement. While improving medical applications and the control of robots promises positive impact, misuse never be avoided.
Acknowledgments and Disclosure of Funding
This project received funding from the Swiss National Science Foundation under NCCR Automation under grant agreement 5 54NF40 180545, the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation program grant agreement no. 8 815943, and was supported by Oracle Cloud Services. Morcover, we thank Sebastian Curi and Lars Lorch for their valuable feedback.
References
[1] Juergen Schmidhuber. Evolutionary principles in self-referential learning. On learning how learn: The meta-meta-... hook. PhD thesis, Technische Universitaet Munchen, 1987.
[2] Sebastian Thrun and Lorien Pratt, editors. Learning to Learn. Kluwer Academic Publishers, 1998.
[3] Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 2000.
[4] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICMI. deep learning workshop, volume Lille, 2015.
[5] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap, and Google Deepmind. Meta-Learning with Machine Learning, 2016.
Neural Networks. In International Conference on
[6] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
[7] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal Meta- Policy Search. In International Conference on Learning Representations, 2019.
[8] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference on Learning Representations, 2020.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic agneliag meta-leaming. Advances
Neural Information Processing Systems, 2018
[10] Jaesik Yoon, Taesup Kim, Ousmane Dia. Sungwoong Kim. Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-leaming. In Advances in Neural Information Processing Systems, pages 7332-7342 2018.
[11] Marta Gamelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, volume 80, pages 1704 1713, 2018.
[12] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: Bayes-optimal meta- learning with PAC-guarantees. In International Conference for Machine Learning (ICML), 2021.
[13] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104(1):148-175, 2016.
[14] Peter 1. Frazier. A Tutorial on Bayesian Optimization. arXiv preprint arXiv:1807.0283 2018
[15] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
[16] Leslie Pack Kaelbling, Michael I. Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of Machine Learning Research, 4:237-285, 1996.
[17] Yunxiao Qin, Weiguo Zhang, Chenxu Zhao, Zezheng Wang, Hailin Shi, Guojun Qi, Jingping Shi, and Zhen Lei. Rethink and redesign meta learning. arXiv, 2018.
[18] Yogesh Balaji. Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards Domain Generalization using Meta- RettaaReguaiizatioon In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Resa-Bianchh, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates,
Inc., 2018
10

---

[19] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts. calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 69(2):243-268 2007.
[20] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate Uncertainties for Deep Learning
Using Calibrated Regression. In International Conference Machine Learning, 2018.
[21] Shengyang Sun. Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks. In International Conference on Learning Representations, 2019.
[22] Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In International Conference on Machine Learning, 2014.
[23] Ron Amit and Ron Meir. Meta-leaming by adjusting priors based on extended PAC-Bayes theory. In International Conference on Machine Learning, 2018.
[24] Benjamin Guedj. A primer on PAC-Bayesian learning. In 2nd Congress of the French Mathematical Society, 2019.
[25] Carl Edward Rasmussen and Christopher K. L. Williams. Gaussian processes in machine learning. MIT Press, 2006.
[26] Peter Auer, Nicolo
and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.
[27] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design. In International Conference on Machine Learning, pages 1015-1022, 07 2010.
[28] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4))285-294 1933.
[29] Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In Interna- tional Conference on Machine Learning, pages 5627-3635. PMLR, 2017.
[30] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 2017.
[31] Oriol Vinyals. Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Procexsing Systems, 2016.
[32] James Harrison, Apoorva Sharma, and Marco Pavone. Meta-learning priors for efficient online bayesian regression. In International Workshop on the Algorithmic Foundations of Robotics, pages 318-337. Springer, 2018.
[33] Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell, Learning To Learn Using Gradient Descent. In International Conference on Artificial Neural Networks, 2001.
[34] Marcin Andrychowicz. Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffmann David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. arXiv, 2016.
[35] Yutian Chen, Matthw W Hoffmann, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando De Freitas. Learning to Learn without Gradient Descent by Gradient Descent. In International Conference on Machine Learning, 2017.
[36] Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Leaming Algorithms. arXiv. 2018.
[37] Tacsup Kim, Jacsik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning In Advances in Neural Information Processing Systems, 2018.
[38] Vincent Fortuin and Gunnar Rätsch. Deep mean functions for meta-learning in gaussian processes. arXiv preprint w-Xxi:9901008090 2019.
[39] Chao Ma, Yingzhen Li, and José Miguel Hermández-Lohaar Variational implicit processes. In International Conference on Machine Learning, pages 4222-4233. PMLR, 2019.
[40] Alexander Matthews, James Hensman, Richard Turner, and Zoubin Ghahramani. On sparse variational methods and the kullback- leibler divergence between stochastic processes, Journal of Machine Learning Research, 51:231 239, 2016.
[41] David Burt, Sehastian W Oher, Adrià Garriga-Alonso, and Mark van der Wilk. Understanding variational inference in function-space. arXiv preprint trXiv:2011.0942 2020.
[42] Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-learning initialize bayesian optimization hyperparameters. In MetaSel@ ECAI, pages 3-10. Citeseer, 2014.
[43] Marius Lindauer and Frank Hutter. Warmstarting of model-based algorithm configuration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018
11

---

[44] Y. Wei, P. Zhao, Huaxiu Yao, and J. Huang. Transferable Neural Processes for Hyperparameter Optimiza- tion. urXiv preprint vrXiv:1909003209 2019.
[45] Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. In C. J. C. Burges,
L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[46] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google Vizier: A Service for Black-Bo Box Optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 1487-1495, New York, NY, USA,
2017. Association for Computing Machinery.
[47] Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian optimization using ranking-weighted gaussian process ensembles. In AutoML Workshop at ICML, volume 7, 2018.
[48] Ho Chung Leon Law, Peilin Zhao, Lucian Chan, Junzhou Huang, and Dino Sejdinovic. Hyperparameter learning via distributional transfer. In Advances in Neural Information Processing Systems, 2019.
[49] Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust Bayesian neural networks. Advances in Neural Information Processing Systems, 34-4414, 2016.
[50] Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable Hyperparameter Transfer Learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, 2018.
[Si] Bernt Øksendal. Stochastic Differential Equations: An Introduction with Applications, volume 82. 01
2000.
[52] Anne-Marie Lyne, Mark Girolami, Yves Atchadé, Heiko Strathmann, and Daniel Simpson. On Russian Roulette Estimates for Bayesian Inference with Intratlet Likelihoods. Statistical Science, 30(4):443 467,2015. 2015.
[53] Yueen Luo, Alex Beatson, Mohammad Norouzi, Jun Zhu, David Duvenaud, Ryan Adams, and Ricky TQ Chen. SUMO: Unbiased cstimation of log marginal probability for latent variable models. In International Conference on Learning Representations, 2020.
[54] Jiaxin Shi, Shengyang Sun, and Jun Zhu, spectral approach to gradient estimation for implicit distribu- tions. In International Conference on Machine Learning, pages 4644-6533, PMLR, 2018.
[55] Yang Song. Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach density and score estimation. In Uncertainty Artificial Intelligence, pages 574-584, PMLR, 2020.
[56] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics, pages 17700 PMLR, 2016.
[57] Joaquin Quiñionero -anerela and Carl Edward Rasmussen. A Unitying View of Sparse Approximate Gaussian Process Regression. Journal of Machine Learning Research, 6(65):1939 1959, 2005.
[58] Matthias Bauer, Mark van der Wilk, and Carl Edward Rasmussen. Understanding probabilistic sparse gaussian process approximations. In Advances in Neural Information Processing Systems, 2016.
[59] L. C. W. Dixon and G. P. Szego. The Global Optimization Problem: An Introduction. In Towards Global Optimisation 2. North-Holland Pub. Co, 1978.
[60] Felix Berkenkamp, Anna Fivazi, Lukas Grossberger, Kathrin Skubch, Jonathan Spirz,Christtan Stefan Falkner. Probabilistic Meta-Learning for Bayesian Optimization. https://openrevierr net/
Christian Daniel, and
addfid=fd2wIPnsY 2021.
[61] Marcin Molga and Czeslaw Smutnicki. Test functions for optimization needs. Test fumctions for optimiza- tion needs, 101:48, 2005.
[62] Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1);1, 2010.
[63] Leo Breiman, Jerome Friedman, Charles J Stone, and Richard A Olshen. Classification and regression frees. CRC press, 1984.
[64] Terry Therneau, Beth Atkinson, Brian Ripley, and Maintainer Brian Ripley. rpart: Recursive partitioning and regression trees, 2018.
[65] Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[66] Daniel Kühn, Philipp Probst. Janek Thomas, and Bernd Bischl. Automatic exploration of machine learning experiments on openml. arXiv preprint arXiv:1806. 10961, 2018
[67] B. Bischl, Giuseppe Casalicchio, Matthias Feurer, F. Hutter, Michel Lang, R. Mantovani, J. N. Rijn, and
J. Vanschoren. Openml benchmarking suites and the openml100. arXiv preprint 1rXiv:1708.03731. 2017.
12

---

[68] Christopher Milne. Thomas Schietinger. Masamitsu Aiba, Arturo Alarcon, Jürgen Alex, Alexander Anghel. Vladimir Arsov, Carl Beard, Paul Beaud, Simona Bettoni, al. Swissfel: the swiss x-ray free electron laser. Applied Sciences, 2017.
[69] Georg Ch Pflug. Optimization of stochastic models: the interface between simulation and optimization, volume 373. Springer Science & Business Media, 2012.
[70] Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu accelcration. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[71] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[72] Jonas Rothfuss, Fabio Ferreira, Simon Boehm, Simon Walther, Maxim Ulrich, Tamim Asfour, and Andreas Krause, Noise Regularization for Conditional Density Estimation, arXiv preprint arviv 1907.08922, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work introduces new meta-leaming approach which is very general and can be applied in many different scenarios. Our results suggest that is makes the uncertainty estimates of meta-learned models more reliable and robust which promises overall positive impact. However, misuse and potentially harmful applications of our method can never be ruled out.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Appendix
B.3
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section B.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "k-9aHAuFyZB", "reviewer": "Reviewer_N7j1", "summary": "This paper observes that meta-learning algorithms typically underestimate epistemic uncertainty because of a failure of regularization to respect the structure of meta-learning.\nThey propose a meta-regularization step that builds on functional KL-divergence to a GP functional prior.\nTheir experiments in sequential BO suggest that this functional prior outperforms alternatives.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Thank you for this paper, which attacks an important problem for meta-learning: getting good uncertainty estimates that incorporate the 'meta-knowledge' as well as the task-knowledge in a principled way.\nThe general approach you take makes sense and feels like a promising strategy for dealing with this problem.\n\nNevertheless, I am not currently recommending acceptance.\nI felt that the theoretical contribution is comparatively modest, as the paper mostly combines two previously established tools (adding the functional KL on top of a meta-learning formulation).\nGiven that, the paper needed to make some real progress on empirical demonstrations showing the usefulness of the approach.\nHere, I would have preferred to either see good results on higher-dimensional spaces or some discussion/investigation of failures in high-dimensional spaces.\nI would change my mind either if you clarified that I am undervaluing your theoretical contribution, or that the experimental support you offer has broader implications than I currently understand.\n\n# Major comments\n\nOne major difficulty with Sun et al.'s approach is the estimation of this supremum over measurement sets.\nThe approach you take, similar to theirs, is to estimate the expectation over measurement sets (rather than the supremum over a sample).\nTo the best of my understanding, this might improve stability, but doesn't really match the derivations.\nAnd the estimation of this term is quite difficult in more challenging data-sets.\nBut if I've misunderstood this, please correct me.\n\nMore generally, I worry that the approach of uniformly sampling data to fill in the off-distribution prior will only work in low-dimensional datasets.\nThis is not something that your experiments are in a position to evaluate.\nIn order to feel confident that this work is of relevance to the broader NeurIPS community, it would be good to examine settings with more than dozen or so dims (or to explicitly state the limitation if it is not possible).\n(My current understanding is that the highest-dim experiment is xgboost with 10 dims, is that right?)\nHere, it would be very valuable to push the limits of your approach into domains where it might be harder to get a functional approach to work.\n\n## Originality\nI have not seen the use of functional hyper-priors on this formulation of meta-learning before.\nHowever if I have a slight reservation, it is that this paper represents a somewhat limited application to Sun et al.'s Functional VI to Rothfuss et al.'s PACOH.\nIt would be helpful to build further on these ideas, perhaps by analysing the strengths and limitations of the approach in different settings.\n\n## Quality and Clarity\nI felt that the writing could have been much more concise and that this would have improved the presentation of ideas.\n\nI am slightly confused by how uncorrelated the ranking of the non-F-PACOH methods in Figure 3 is.\nI think this either means that all the other methods are very sensitive to the dataset in their performance or that they might not be fully tuned to the task?\n\n## Significance\nThis work could be significant in offering new strategies for dealing with uncertainty in meta-learning.\nI suspect that the work will have much greater impact if it is able to apply to a broader range of data types.\n\n# Minor remarks\nAbstract could probably be cut significantly without losing meaning. E.g., the first sentence is probably not necessary and second could be \"Meta-learning can improve accuracy when data are scarce, but existing methods have unreliable uncertainty which is often overconfident.\"\nMore generally, it felt like you could cut a lot of text in a way that would clarify the arguments and contributions.\n\nLine 174 you are missing the KL divergence symbol inside your expectation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the paper would have benefited from more discussion of the limitations of the functional KL estimation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Xot3l2_cH-9", "reviewer": "Reviewer_iGTP", "summary": "This work introdcues a new way to regularize priors in meta-learning through the use of stochastic processes as hyper-priors. The authors rely on sampling-based measurements to estiamte the KL divergence between the meta-learned prior and a stochastic process hyper-prior. \n\nThrough experiments on toy datasets, the algorithm, F-PACOH, demonstrates principled uncertainty estimates. It provides higher levels of uncertainty far from training data, which is in contrast to previous methods for imposing meta-learning priors and hyper-priors. Finally, the authors apply their method to a Bayesian Optimization meta-learning problem and demonstrate decreased regret compared to other methods benchmarked. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, this paper is a strong combination of several existing ideas in meta-learning and stochastic processes. The application of KL divergence sampling-based estimates from Sun et al. in conjunction with the PACOH framework demonstrated principled uncertainty estimates and strong performance on Bayesian optimization tasks. The combination of these ideas yielded a method that has strong empirical justifications and theoretical motivations. \n\nThroughout, the authors were very clear in their presentation. One thing that may improve clarity is an appendix section (didn't have time to check those, so sorry if this already exists) on stochastic processes. I consulted Sun et al. for a brief review and that was helpful, but making the paper more self contained for someone coming from an adjacent area could be useful. Another point to that effect is that the authors keep referring to \"regularization in the function space\" and my brain keeps asking \"the function space of <what>?\" That could just be how one refers to function spaces but a few examples of writing \"the function space of the prior\" would have defintely helped me get the concept quicker. \n\nOne error (I believe) is on Line 174. The authors missed the \"KL\" in the expression $$\\mathbb{E}_{X_i}KL[q(h^{X_i}||\\rho (h^{X_i}))]$$ \n\nIn terms of experiments, I would be interested in more demonstrations that F-PACOH generates principled uncertainty estimates. The 1-D toy datasets did show that property but more empirical validation would be welcome. I'd be particularly interested in uncertainty estimates on out-of-distribution data that has undergone dataset shift (i.e. Ovadia et al.). \n\nUltimately, my impression is that this paper represents a solid advance in (hyper-)priors for metalearning and combines several ideas into a useful new method. Given it's strong performance on the BO tasks, I could see other researchers building on these ideas. \n\nUpdate after responses: \nI think that a major limitation of the work is its restriction to lower dimensional settings. This will fundamentally limit its applications and is a larger issue than I originally thought. I still think this is a good paper but am slightly lowering my score because of this. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors didn't spend much time on the limitations of their work, beyond addressing the cubic runtime of their methods. Some things that may or may not be limitations that could be addressed:\n* Are you always able to get good sampling estimates for the KL-divergence and its gradient?\n* Can this method scale in parallel at all? Can it handle large batches of evaluations?\n\nAdditionally, societal impacts weren't addressed. Could you talk about how BO can develop novel therapeutics through protein optimization, for instance? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WyG623xfCDx", "reviewer": "Reviewer_RiXp", "summary": "This paper discusses a meta-learning method with priors defined on function spaces. Building on PACOH, a meta-learning framework with PAC-Bayesian bounds, the paper proposes F-PACOH where the hyper-priors and priors defining meta-learning models in function spaces. In original PACOH, the hyper-priors are set to be parametric distributions (usually Gaussians), and the authors argue that such parametric priors cannot effectively capture the functional structure of the tasks that we ultimately want to enforce for meta-learners. On the other hand, functional priors directly regularize functions of interest to be close to prior knowledge, e.g., standard Gaussian processes, so result in more reliable uncertainty estimates. \n\nThe inference is done by optimizing the PAC-Bayesian bound, and the caveat here is the computation of KL-divergence between functional hyper-priors and hyper-posteriors. This can be approximated by the technique introduced in Sun et al., 2021, where a set of measurements points are sampled and the KL-divergence is approximated with the evaluations on those measurement points.\n\nThe paper mainly focuses on the application of the proposed method to Bayesian optimization. On several benchmarks, the proposed F-PACOH is demonstrated to quickly meta-learn and improves on similar BO tasks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is clearly written and easy to follow. The motivation is well set, and the idea to use functional priors is a reasonable solution to the problem. The BO experiments, especially the sequential BO, are interesting. I'm no expert in BO literature, so my assessment may be wrong, but at least for me, these experiments seem novel.\n\nI cannot say the proposed method is novel enough so that the paper should be accepted solely based on its novelty; the paper is definitely well-executed, but the core contribution is rather an incremental combination of well-established frameworks; PACOH and functional variational Bayesian neural nets (FBNNs). Roughly speaking, F-PACOH is an FBNN version of PACOH which uses the same inference techniques as the original FBNNs. The application to meta-learning is new, but it does not incur additional technical difficulty.\n\nAlthough the framework is generic so virtually one can apply any functional hyper-priors and priors, the paper only presents the case when both of them and priors are set to be Gaussian processes (GPs). GP is easy to deal with due to its closed-form marginals. Have you actually tested with other types of priors or hyper-priors, for instance, implicit processes? In such a case, as noted in the paper, the gradient of KL-divergence should be approximated via SSGE or sliced score matching. Personally, I find the gradient estimates with such techniques unreliable, so wonder if the proposed method works well with them.\n\nThe measurement points should be sampled for the KL-divergence approximation. This might be trivial for 1D real line for instance, but what if the target domain is high-dimensional space such as images or sequences? Would the uniform sample method still work well? I guess this would be important because the proposed meta-learning will be more useful for such a high-dimensional setting for which the performance of a vanilla GP based BO is expected to degrade.\n\nIn the experiments, a straightforward baseline I can think of is, during meta-training gather all the training task points  $D_{1:T}$,\nconstruct a posterior GP conditioned on $D_{1:T}$, and use it as a hyper-prior for subsequent tasks. I guess this can be a strong baseline despite having to store all the training task data $D_{1:T}$.\n\nIt would be good to directly assess the uncertainty estimates for BO tasks as well (with calibration error plots for instance).\n\nHow sample efficient the proposed method is? For instance, how are the performances of the algorithms w.r.t. different numbers of training tasks? As far as I know, NP requires quite many tasks to be reliably trained, even for simple GP regression tasks. Judging from the simulation results where F-PACOH is trained with only 10 tasks and still is able to reliably adapt to new tasks, I think F-PACOH to be much more sample-efficient than NP.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed both limitation and potential negative societal impact of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "QXzdehu-DeH", "reviewer": "Reviewer_VyPu", "summary": "This paper considers improving uncertainty estimations for meta-learning. Previous work [1] has done this via using a hyper-prior over the prior parameters for meta-regularization. This hyper-prior, however, took the form of a simple Gaussian distribution over the prior parameters, which imposes smoothness regularization on the prior but may not be ideal for enforcing uncertainty in regions where no meta-training data is available. To remedy this, the authors propose a hyper-prior in the function space in the form of a stochastic process. They then show how the meta-learning algorithm can be defined so that it uses stochastic processes for the prior, hyper-prior and hyper-posterior. The experiments are conducted on a Bayesian optimization setup, where meta-training involves using a sequences of related BO problems corresponding to n target functions and evaluation involves using the meta-learned model to perform BO on a new target function. For example, one such setting is hyper-parameter optimization for machine learning models, where the training task functions correspond to training and testing the machine learning model on different datasets. Across different benchmarks in this setting, they show their proposed algorithm performs better than [1].\n\n[1] Rothfuss et al. PACOH: Bayes-optimal meta-learning with PAC-guarantees. ICML 2021.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the paper is proposing a modification to [1] to improve its behavior and performance; however, I believe the proposed change is interesting and does seem to lead to better performance.\n\nQuality: the paper seems technically sound. There was strong justification provided for the proposed method and the experimental results seem to show that it does provide the type of benefit in uncertainty estimation that was discussed in the motivation for the work.\n\nClarity: the paper is very well-written. Explains the proposed method and its justifications well and the experiments are thoroughly described.\n\nSignificance: I think Bayesian optimization is an interesting application of meta-learning and the experiments & benchmarks considered for meta-learning for Bayesian optimization in this work could be interesting to other researchers and help build more momentum in this area.\n\n[1] Rothfuss et al. PACOH: Bayes-optimal meta-learning with PAC-guarantees. ICML 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not explicitly discussed but should not be different than existing work in this area of meta-learning.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses PAC-Bayes meta-learning to do meta-learning using the functional KL.  They demonstrate this in a series of experiments where they meta-learn hyperpriors of Gaussian processes for the purpose of doing Bayesian optimization among related tasks.

The reviewers thought the paper was well written, technically strong and interesting.  The review scores were 6, 7, 4, 6.  The main concern here seems to be novelty.  The reviewer arguing for reject expressed that they thought it was a straightforward extension of functional KL and meta-learning ("The combination of functional-KL and meta-learning problem seem fairly marginal.”).  

One major criticism shared by all reviewers seems to be the scale of the experiments, in that they’re limited to low-dimensional problems due to limitations of the function space view.  One reviewer lowered their score from a 7 to a 6 as a result of this during the discussion period. It seems that the authors are aware of this concern and have promised to address this in their discussion.  From the authors: “We agree with the reviewers’ concern about high-dimensional data. For this reason, we have added a discussion of the method’s limitations to lower-dimensional data domains to the updated version of the paper.”

The majority vote among the reviewers was to accept the paper, citing that the paper is well written, technically well-executed and empirically strong (although limited to low-D problems). The application to meta-Bayesian optimization seems well motivated and practically useful.  Thus the recommendation is to accept the paper as a poster.  Hopefully the reviewers feedback can be incorporated into the camera ready version (including discussion of low-D limitations) to make the paper stronger.
Decision: Accept (Poster)