MARS: Meta-learning as Score Matching in the Function Space
Authors: Krunoslav Lehman Pavasovic
Jonas Rothfuss
Andreas Krause
Venue: ICLR 2023 notable top 25% (2023)
Keywords: score estimation
meta-learning
bayesian neural networks

Abstract: Meta-learning aims to extract useful inductive biases from a set of related datasets. In Bayesian meta-learning, this is typically achieved by constructing a prior distribution over neural network parameters. However, specifying families of computationally viable prior distributions over the high-dimensional neural network parameters is difficult. As a result, existing approaches resort to meta-learning restrictive diagonal Gaussian priors, severely limiting their expressiveness and performance. To circumvent these issues, we approach meta-learning through the lens of functional Bayesian neural network inference which views the prior as a stochastic process and performs inference in the function space. Specifically, we view the meta-training tasks as samples from the data-generating process and formalize meta-learning as empirically estimating the law of this stochastic process. Our approach can seamlessly acquire and represent complex prior knowledge by meta-learning the score function of the data-generating process marginals instead of parameter space priors. In a comprehensive benchmark, we demonstrate that our method achieves state-of-the-art performance in terms of predictive accuracy and substantial improvements in the quality of uncertainty estimates.

MARS: META-LEARNING AS SCORE MATCHING IN THE FUNCTION SPACE
Krunoslav Lehman Pavasovic" ETH Zurich
Jonas Rothfuss
Andreas Krause ETH Zurich Switzerland k krauseaßethz.chh
ETH Zurich Switzerland
Switzerland k ZlehmanBethz.ch
rojonaseethz.ch
ABSTRACT
Meta-learning aims to extract useful inductive biases from set related datasets. In Bayesian meta-learning, this is typically achieved by constructing a prior dis- tribution over neural network parameters. However, specifying families of com- putationally viable prior distributions over the high-dimensional neural network parameters is difficult. As a result, existing approaches resort to meta-learning restrictive diagonal Gaussian priors, severely limiting their expressiveness and performance. To circumvent these issues, we approach meta-learning through the lens of functional Bayesian neural network inference, which views the prior as a stochastic process and performs inference in the function space. Specifically, we view the meta-training tasks as samples from the data-generating process and for- malize meta-learning empirically estimating the law of this stochastic process. Our approach can seamlessly acquire and represent complex prior knowledge by meta-learing the score function of the data matu--ennraaing process marginals instead of parameter space priors. In a comprehensive benchmark, we demonstrate that method achieves state-of-the-art performance in terms of predictive accuracy and substantial improvements in the quality of uncertainty estimates.
INTRODUCTION
Using data from related tasks is of key importance for sample efficiency. Meta-learning attempts to extract prior knowledge (i.e., inductive bias) about the unknown data generation process from these related tasks and embed into the learner so that it generalizes better to new learning tasks (Thrun & Pratt, 1998; Vanschoren, 2018). Many meta-learning approaches try to amortize or re-learn the entire inference process (c.g., Santoro et al., 2016; Mishra et al., 2018; Garnelo et al., 2018) or significant parts of it (c.g., Finn et al., 2017; Yoon et al., 2018). As result, they require large amounts of meta-training data and are prone to meta-overfitting (Qin et al., 2018; Rothfuss et al., 2021a).
The Bayesian framework provides sound and statistically optimal method for inference by combining prior knowledge about the datt-ggeeratinn process with new empirical evidence in the form of dataset. In this work, adopt the Bayesian framework for inference the task level and only focus on meta-learning informative Bayesian priors. Previous approaches (Amit & Meir, 2018; Rothfuss et al., 2021a) meta-learn Bayesian Neural Network (BNN) prior distributions from a set of related datasets; by meta-lcarning the prior distribution and applying regularization at the meta-level, they facilitate positive transfer from only a handful of meta-traine ttainiin tasks. However, BNNs lack a parametric family of (meta-) llarnabbee priors over the high- high--immesiooal space of neural network (NN) parameters that both computationally viable and, simultaneously. flexible enough to account for the over-parametriizttoo of NNs. In practice, both approaches use a Gaussian family of priors with diagonal covariance matrix, which is too restrictive to accurately match the complex probabilistic structure of the data-generating process.
To address these shortcomings, we take a new approach to formulating the meta-learning problem and represent prior knowledge in novel way. We build on recent advances in functional approx- imate inference for BNNs that perform Bayesian inference in the function space rather than in the parameter space of neural networks (Wang et al., 2018; Sun al., 2019). When viewing the BNN prior and posterior as stochastic processes, the perfect Bayesian prior is the (true) data-generating
"Equal contribution.

---

stochastic process itself. Hence, view the meta-training datasets samples from the meta-data- generating process and interpret -eatning as empirically estimating the law of this stochastic process. More specifically, then directly be used as a source of prior knowledge when performing approximate functional BNN inference on
meta-leart the score function of its marginal distributions, which can
target task. This ultimately allows to flexible neural network models for learning the score and overcome the issues of meta-learning BNN priors in the parameter space.
In our experiments, we demonstrate that our proposed approach, called Meta-learning via Attention- based Regularised Score estimation (MARS), consistently outperforms previous meta-learners in predictive accuracy and yields significant improvements in the quality of uncertainty estimates. Notably, MARS enables positive transfer from only handful of tasks while maintaining reliable uncertainty estimates. This promises fruitful future applications to domains like molecular biology or medicine, where meta-training data is scarce and reasoning about epistemic uncertainty is crucial.
RELATED WORK
Meta-Learning. Common approaches in meta-learning amortize the entire inference process (San- toro et al., 2016; Mishra et al., 2018; Ravi & Beatson, 2018; Garnelo et al., 2018), learn a good neural network initialization (Finn et al., 2017: Rothfuss et al., 2019; Nichol et al., 2018; Kim al., 2018) or a shared embedding space (Baxter, 2000; Vinyals et al., 2016; Snell et al., 2017). Although these approaches can meta-learn complex inference patterns, they require large amount of meta- training data and often perform poorly in settings where data is scarce. Another line of work uses hierarchical Bayesian approach to meta-learn priors over the NN parameters (Pentina & Lampert, 2014; Amit & Meir, 2018; Rothfuss et al., 2021a). Such methods perform much better on small data. However, they suffer from the lack of expressive families of priors for the digensinns and complex parameter space of NNs, making too restrictive assumptions to represent complex in- ductive biases. Our approach overcomes these issues by viewing the problem in the function space and directly learning the score, which can easily be represented by a NN instead of a prior distribu- tion. Also related to our stochastic process approach are methods that meta-learn Gaussian Process (GP) priors (Fortuin et al., 2019; Rothfuss et al., 202 lb: 2022). However, the GP assumption is quite limiting, while MARS can, in principle, match the marginals of any data-generating process.
Score estimation. We use score estimation In particular, we use a parametric approach to score matching and employ an extended version of the score matching objective of Hyvärínen & Dayan (2005). For high-dimensional problems, Song et al. (2020); Pang et al. (2020) propose randomly sliced variations of the score matching loss. Alternatively, there is body of work nonparametric score estimation (Canu & Smola, 2006; Liu et al., 2016; Shi et al., 2018; Engl et al., 1996; Zhou et al., 2020). Among those, the Spectral Stein Gradient Estimator (Shi al., 2018) has been used for estimating the stochastic process marginals for functional BNN inference in a setting where the stochastic prior is user-defined and allows for generating arbitrarily many samples (Sun et al., 2019). Such estimators make it much harder to add explicit dependence on the measurement sets and prevent meta-overfitting via regularization, making them less suited to our problem setting.
central element of meta-learning method.
3 BACKGROUND
Bayesian Neural Networks. Consider a regression task with data D = (XP, P) that consists of m i.idd. noisy function evaluations y/ = fx,) + E; of an unknown function f X' + V. Here, X2 = } E x denotes training inputs and yD = { E ym the corre- sponding noisy function values. Let hep X' + y be a function parametrized by a NN with weights 0 E e. For regression, where y C R. we can use he to define a conditional distri- bution over the noisy observations p(y|xx x, 8) = (x))ho(x), o²), with flexible mean represented by he and observation noise variance o². Given prior distribution p(0) over the model param- eters 8, Bayes' theorem yields a posterior distribution p(0]x".y' x "(x" 8)p(0), where p(yP|XP, 0) I[m p(yy)>( xx, For an unseen test point x", compute the predictive dis- tribution, which is defined as px"" .P PP) = J p(y"(x" () ydD and obtained by marginalizing out the posterior over parameters A.
BNN inference in the function space. Posterior inference for BNNs is difficult due to the high-dimensional parameter space θ and the over-parameterized nature of the NN mapping (x)).
2

---

An alternative approach views BNN inference in the function space, i.c., the space of regression functions : + V. yielding the Bayes rule p(h[X", yy") x p(yP|x" h)p(h) (Wang et al., 2019; Sun et al., 2019). Here. p(h) is stochastic process prior with index set X, taking values in V
Stochastic processes can be understood as infinitediimensional random vectors, and, thus, are hard to work with computationally. However, given finite measurement sets X == X1,-..,XX, E A k N, the stochastic process can be characterized by its corresponding marginal distributions of function values )(h*) :- ((h(x))), (xx((()))) (ef. Kolmogorov Extension Theorem, Oksendal (2013)). Thus, we can break down the functional posterior into a more tractable form by re-phrasing it in terms of posterior marginals based on measurement sets X: p(h |X X", Py x p(y? h )p(h*).
This functional posterior can be tractably approximated to achieve functional BNN inference (Sun et al., 2019; Wang et al., 2019). We briefly describe how this can be done via functional Stein Variational Gradient Descent (fSVGD, Wang et al., 2019). The procedure is explained in more detail in Appendix A.I. The FSVGD algorithm approximates the posterior using set of L NN pa- rameter particles
}r.]- To optimize posterior approximation, fSVGD iteratively re-samples measurement sets X from measurement distribution v, supported on X, computes SVGD up- dates (Liu & Wang, 2016) in the function space, and projects them back into the parameter space, in order update the parameter-space particle configuration approximating the posterior. To achieve this, ISVGD uses the functional posterior score, i.c., the gradient of the log-density Vn> In p(h 11h XX, _P) Vnx In p(y h>²) + Vnx Inp(h*) ) to guide the particles towards areas of high posterior probability. Formally, for all / = I,.., L the particle updates are computed as
8' 7
Kv ng In x, x y Vnx of
Kii
I
SVGD update the function space
where K = [k: h2 JI is the kernel matrix between the function values in the measurement points based on a positive semi-definite kernel function k(--)) A key insight that will later draw upon in the paper is that such functional approximate inference techniques only use the prior scores and, in principle, do not require full prior density that integrates to I.
4 META-LEARNING AS SCORE ESTIMATION IN THE FUNCTION SPACE
4.1 PROBLEM STATEMENT: META-LEARNING
Meta-learning extracts prior knowledge (i.e., inductive bias) from a set of related learning tasks accelerate inference on new and unseen learning task. The meta-leamer is given n datasets DI...., Dr, where each dataset D; = (XP: yyP) consists of m, noisy function evaluations Vı, = f((x,,)) + e corresponding noise E. In short, we write X
a function f. X' + y S R and additive sub-Gaussiaa (x., X.,m;) for the matrix of function inputs and y (yt.1, St. mt: T for the vector of corresponding observations. Following previous work (e.g.. Baxter, 2000; Pentina & Lampert, 2014; Rothfuss et al., 2021a), we assume that the functions fi ~ T are sampled i..d. from a task distribution T, which can be thought of as a stochastic process p(f) that governs the random function f X' y.
Our goal is to extract knowledge from the observed datasets, which can then be used as a form of prior for learning new, unknown target function f ~ TT from a corresponding dataset D*. By performing Bayesian inference with a meta-learned prior that attuned to the task distribution T. we hope to obtain posterior predictions that generalize better.
4.2 SHORTCOMINGS OF META-LEARNING PRIORS IN THE PARAMETER SPACE
Previous work phrases meta-learning as hierarchical Bayesian problems and tries to meta-learn a prior distribution p(0) on the NN weight space that resembles p(f) (e.g., Amit & Meir, 2018; Rothfuss et al., 2021a). This approach suffers from the following central issues: Due to the over- parameterized nature of neural networks, large set of parameters 0 correspond to exactly the same mapping ha )). This makes specifying a good prior distribution p(0) typically very difficult. At the same time, there is lack of sufficiently rich parametric families of distributions over the high- dimensional parameter space e that are also computationally viable. So far, only the very restrictive Gaussian family of priors with diagonal covariance matrices has been employed. As result, the
3

---

meta-learned posterior lacks the necessary flexibility to match the complex probabilistic structure of the geeneratiee process accurately.
4.3 META-LEARNING AS SCORE ESTIMATION ON THE AATA-GENERATING PROCESS
Aiming to address these issues, we acquire prior knowledge in data-driven manner with new perspective. We develop a novel approach to meta-learning which hinges upon three key ideas:
First, we view BNN inference in the function space (see Sec. 3), i.e., as posterior inference p[h[X". y²) .y")
x p(y"I x h)p(h) over neural network mappings he X + y instead of pa- rameters 0. From this viewpoint, the prior, which is the target of our meta-learning problem. is p(h), a stochastic process on the function space. This alleviates the aforementioned problem of over-parametrization,, which arises when considering priors on the parameter space.
Second, we ask ourselves what is a desirable prior: from a Bayesian perspective, the best possible prior is the stochastic process of the task generating distribution T itself, i.e., p(h) = p(f). Hence, we would want to meta-learn a prior that matches p(f) as closely as possible. Since the meta-training datasets Di, Dn constitute noisy observations of the function draws fi,..., ~ p(f), we can use them to estimate the stochastic process marginals p(fx). Crucially, we tractably represent and estimate the stochastic process p(.)) through its finite marginals p(fx) in measurement sets X.
Our third insight is that all popular approximate inference methods for BNNs only use the prior score, i.e., the gradient of the log-prior, and not the prior distribution itself. In addition, parametrizing and estimating the prior score is computationally easier than the prior distribution itself since it does not have integrate to 1, allowing for much more flexible neural network representations. For this reason, our meta-learning approach directly forms estimate of the scores s(fx, X) = Vex In ((fx) of the data-generating process marginals.
In summary, our high-level approach is to meta-learn / estimate the prior score Vhx In ((>) that matches data- getta-genrratig stochastic process from meta-training data D1, .... D,, At meta-test time, the meta-learned prior marginals used in the approximate functional BNN inference dataset D' to infuse the acquired prior knowledge into the posterior predictions. In the following section, we discuss in more detail how implement this general approach as a concrete, computa- tionally feasible algorithm with strong empirical performance.
target
THE MARS META-LEARNING ALGORITHM
In Section 4, we have motivated and sketched the idea of meta-learning via score estimation of the data-generating process marginals. We now discuss particular challenges and outline the design choices for translating this idea into a practical meta-learning algorithm.
Score estimation of stochastic process marginals has previously been used in the context of existing functional BNN approaches ( (Wang et al., 2019; Sun et al., 2019). However, they presume a known stochastic process prior with oracle access arbitrarily many sampled functions that can be evalu- ated in arbitrary locations of the domain X' Compared to such a stochastic process prior with oracle access, we face two key challenges which make the score estimation problem much harder:
1. We only have access to a finite number of datasets {D. each corresponding to one function fa drawn from the data-generating stochastic process. This means, we have so esti- mate the marginal scores V,x In p(hx() from only function samples without over-fitting
2. Per function fi. we only have a limited number of m; noisy function evaluations yP in input locations X? which are given to us exogenously, and we have no control over them. However, to perform score estimation for the marginal p(h*) distributions, for each function, need vector of function values f.X evaluated in the measurements points X.
In the following subsections, we present our approach to estimating the stochastic prior scores and discuss how solve these challenges. In addition, we discuss how to perform functional approxi- mate BNN inference with the meta-learned score estimates.
5.1 PARAMETRIC SCORE MATCHING FOR STOCHASTIC PROCESS MARGINALS
Performing functional approximate inference with ISVGD updates as in (1) requires estimation of the prior marginal scores Vnx In p(hx) for arbitrary measurement sets X. A key property of
4

---

stochastic process marginal scores is their permutation equivariance: if permute the order of the points in the measurement set, the rows of the score permute in the same manner. Similarly, the mea- surement sets can be of different sizes, implying score vectors/matrices of different dimensionalities.
Aiming to embed these properties into a parametric model of the prior marginal scores, we use a transformer encoder architecture (Vaswani et al., 2017), that takes
input a measurement set X E xd of k points and corresponding query function values h (Rdm(())) x and performs attention over the second dimension, i... the &s columns corresponding to measurement points. Our score network model is denoted as sah X) with trainable parameters ó and outputs estimate of the score, i.e., 5y (h* X) + Vn,x(h>>), matrix of the same size as the query function values. The model is permutation equivariant w.r.t. the columns corresponding to measurement points and supports inputs/outputs for varying measurement set sizes k. We schematically illustrate the score network architecture in Figure 4 and provide details in Appendix A.3.
We train the score network with a modified version of the score matching loss (Hyvärinen & Dayan, 2005; Song et al., 2020). This loss minimizes the Fisher divergence between the data-generating pro- cess marginals and our score network, based on samples f/x ...., fx from the data- data-generating process marginals. In contrast to the standard score matching loss, modified loss takes the dependence of stochastic process marginals on their measurement sets into account. It uses an expectation over randomly sampled measurement sets, similar the functional approximate inference in Section 3.
[()) = Ex
n Σ
tr( Lr(V X)) 7 (F X31
(2)
As noted previously, meta-training data only provides noisy function evaluations for lim- ited, exogenously given, set of input locations per task ft. The input locations X? per task might even differ in number and be non-overlapping. Thus, fx...., f2 for arbitrary measurement sets X which we require for our score matching loss in (2). In the following section, we discuss how resolve this problem.
do have access the function values
5.2 INTERPOLATING THE DATASETS ACROSS X
In our meta-learning setup (see Section 4.1), we only receive noisy function evaluations y? in my locations X.P for each meta- meta-training task. However, to minimize the score matching loss in (2), need the corresponding function values in arbitrary measurement locations in the domain X.
Hence, interpolate each dataset D; by regression model. Importantly, the regression model should be able to quantify the epistemic uncertainty that stems from having observations at only a finite subset of points X X. While range of method could be used, we employ a Gaussian Process (GP) with Matérn- 5/2 kernel or Bayesian neural network (BNN) based on Monte Carlo dropout (Gal& & Ghahramani. 2016) for this purpose. We fit the respective Bayesian model indepen- dently on each dataset D; which gives us a posterior p(f," XX X, yP) over function values f.X for arbitrary measurement sets X. In Appx. A.4 we provide more details on how we fit each model.
While we could simply use posterior mean values for computing the score matching loss in 2. this would not reflect the epistemic interpolation uncertainty. Instead, for each task, we sample function values fx p(fx' |X xx.x?? .yP) from the corresponding posterior and use them as an input to the score matching loss in (2). Correspondingly, the score matching loss modifies to
[(0) = Ex
1
Σ
E
PEPS S (PX X)) + Is 1ls X)13
The further a measurement point x away from the closest point in XP, the higher is the (epistemic) uncertainty of the corresponding function value f((x)). By sampling from the posterior, in expectation, the loss in (3) effectively propagates the interpolation uncertainty into the score matching procedure, preventing over-confident score estimates for areas of the domain X with scarce data. This is illustrated in Appx. D, Figure 6.
5.3 PREVENTING OETAHTVVEHIITING OF THE PRIOR SCORE NETWORK
The final challenge we need to address is over vertffittin to the meta-training tasks (Yin et al., 2020; Rothfuss et al., 2021a). As we only have meta-training data {D.) corresponding functions
5

---

Algorithm MARS: Meta-Learning the Data-Generating Process Score
Input: datasets D1,... 2R, measurement point distribution v, step size r) Initialize score network parameters Ø
for i I,... n
fit GP or BNN on D. obtain posterior p y?!)
while not converged do
X 3id
|| sample measurement set
for i 1,..., nt do
P.x ~ |( XP
sample function values from posterior marginal score matching loss I/ score network gradient update
(()) - +. Ei= tr( (r(Vixss x)) -
(1,, x)
④ 1 - (L)
Output: trained score network So
fi drawn from the data-generan generating stochastic process, i.e., only have samples for estimation of marginal scores Vn In p(h²), making us prone to overfitting the prior score network. If we per- formed functional BNN inference with such overfitted score estimate, the BNN's posterior predic- tions are likely to be over-confident and too biased towards the functions seen during meta-training.
To counteract the tendency overfit, we regularize the score network via spectral normalization (Miyato et al., 2018) of the linear layers in the transformer encoder blocks. Spectral normalization controls the Lipschitz constant of the neural network layers by dividing their weight matrix W by its spectral norm W|], i..., re-parametrizing the weights as W //| Hence, by applying spectral normalization, we bias our score network towards smoother score estimates corresponding to marginal distributions p(h*) with higher entropy. Empirically we find that spectral normal- ization effectively combats meta-overfitting and prevents the estimated prior score from inducing over-confident posterior predictions when employed in functional BNNs. In our experiments, we examined several other regularization methods, e.g., gradient penalties (Gulrajani et al., 2017) or spectral regularization penalty in the loss, but found spectral normalization to work the best.
5.4 THE FULL MARS ALGORITHM
We now summarize our meta- leattal algorithm MARS. It consists of two stages:
Stage 1: Meta-Learning the Prior Score Network. After initializing the parameters of the score network, we fit a GP or BNN to each of the meta-training datasets D;. Then, we train the score network by stochastic gradient descent on the modified score matching loss Σ(Φ) in (3). In each iteration, we first sample a measurement set X i.i.d. from the measurement distribution = ((2), chosen as uniform distribution over the hypercube X C X conservatively covering the data X. Based on the measurement set, we sample a vector of functions values f.X from the corresponding GP BNN posterior marginals p(f X, .,,,PP? .yP), I....,?! n. Based on these samples, we form an empirical estimate (()) of the score matching loss and perform a gradient update step on the score network parameters Φ. This is repeated till convergence and summarized in Alg. 1. Depending on whether we use GP or BNN as interpolator, we refer to our method as MARS-GE or MARS-BNN.
Stage 2: Functional BNN Inference with the Prior Score Network. When concerned with target learning task with a training dataset D = (X? y we can infuse the meta-leamned inductive bias into the BNN inference by using the prior score network S, from Stage 1 for the approximate inference. In particular, we can either perform functional VI (Sun et al., 2019) or ISVGD (Wang al., 2019). using the score network sy (h> X) predictions as swap-in replacement for the marginal scores V, in p(hx) of user-specified stochastic process prior. In our experiments, we use ISVGD (see Section 3) as functional approximate inference method. The resulting fSVGD BNN inference procedure with our leareld priors score network summarized in Alg. 2 in Appx. A.
6 EXPERIMENTS
We provide a detailed benchmark comparison with existing meta-learning methods, demonstrating that MARS: (i) achieves state-of-the-art performance in terms of predictive accuracy, (ii) yields

---

true far pred mean
(a) Vanilla BNN
(b) fBNN with GP prior
(c) MARS (ours)
Figure 1: BNN posterior predictions with (a) zero-centered Gaussian prior on the NN parameters
(b) Gaussian process prior and (c) meta-learned MARS prior scores.
SwissFRl.
Physionet-
Physionet-HCT
Berkeley Sensor
Argus-Contril
Vanilla
+: 0.000
2.240 0.000
2.768
0.000
0.258
+ 0.000
Vanilla BNN
0.529 + 0.022
2.664 + 0.274
3.938
0.869
0.151 + 0.018
0.016 + 0.002
MAML
0,730 +: 0.057
1.895 + 0.141
2.413 + 0.113
0.121 +: 0.027
0.017 + 0.001
BMAML
0.577 +: 0.044
1.894 + 0.062
2,500
0.002
0.222 0.032
0.037 + 0.003
NP
0.053
0.173
0.001
PACOH-GP
0,376 + 0.024
1.498 = 0.081
2.361 0.047
0.197 + 0.058
0.016 + 0.005
PACOH-NN NN
0.437 +: 0.021
1.623 + 0.057
2.405 + 0.017
0.160 0.070
0.018 + 0.002
MARS-GP (ours)
0,391 + 0,011
1.471 + 0,083
2,309
0.041
0.116
+ 0.001
MARS BNN (ours)
0.407 0.061
1.307 0.065
2.248 0.057
0.113 4.113:0.00
0.017 0.003
Table 1: Meta-Leerning benchmark results in terms of the test RMSE. Reported are the mean and standard deviation across five seeds. MARS consistently yields the most accurate predictions.
well-calibrated uncertainty estimates. In comprehensive ablation study, we shed more light on the central components of MARS and provide empirical support for our design decisions.
6.1 EXPERIMENT SETUP
Environments. We consider five realistic meta-learning environments for regression. The first environment corresponds to data of different calibration sessions of the Swiss Free Electron Laser (SwissFEL) (Milne al., 2017). Here, task requires predicting the laser's beam intensity based on undulators parameters. We also consider electronic health measurements (PhysioNer) from in- tensive care patients (Silva al., 2012), in particular, the Glasgow Coma Scale (GCS) and hematocrit value (HCT). Here, each task correspond to patient. We also use the Intel Berkeley Lab temperature sensor dataset Berkeley-Senson (Madden, 2004), requiring auto-regressive prediction of temperature measurements of sensors in different locations of the building. Finally, the Argus- Control environment requires predicting the total variation robot from its target position based on its PID controller parameters (Rothfuss et al., 2022). Here, tasks correspond to different step sizes between the source and target location. See Appendix B for details.
Baselines. As non-meta-lean meta-leaming baselines, with a zero-mean, spherical Gaussian prior and SVGD inference (Liu & Wang, 2016). To compare previous meta-learners,
Vanilla GP with RBF kernel and a Vanilla BNN
report results for model agnostic meta-learning (MAML) (Finn al., 2017). Bayesian MAML (BMAML) (Yoon et al., 2018) and neural processes (NPs) (Garnelo et al., 2018). In addition, we include PACOH (Rothfuss al., 2021a) which performs hierarchical Bayes inference to meta-learn GP prior (PACOH-GP) or prior over NN parameters (PACOH-NN).
6.2 EMPIRICAL BENCHMARK STUDY
Qualitative illustration. Fig. illustrates the posterior predictions of BNN with Gaussian Prior in the parameter space. b) functional BNN with GP prior (Wang et al., 2019) and c) IBNN trained with the meta-learned MARS-GP marginal scores. For the meta-training we use n 20 tasks with each m = 8 data points, generated from a student-t process with sinusoidal mean function 2x + sin (2x). Similar to previous work (e.g., Fortuin et al., 2022) we observe that weight-space BNN priors provide poor inductive bias and yield grossly over-confident uncertainty estimates. Approach- ing the BNN inference problem in the function space (see Fig 1b) results in much better uncertainty estimates. In this case. however, the uninformative GP prior does not result in accurate mean predic- tions. Finally, the meta- learnle scores in the function space provide useful inductive biased towards the linear+sinusoidal data-genegating pattern while yielding tight but reliable confidence intervals.

---

SwissFEL
Physionet-Ger GCS
Physionet-IICT
Berkeley-Sensor
Argus Conutol
Vanilla GP
0.135 0.000
0.268 0.000
0.277 = 0.000
0.119 + 0.000
0,090 0.000
Vanilla BNN
0,085
0.008
0.277 = 0.013
0.307 = 0.009
0.206 = 0.025
0.104 = 0.005
BMAML
0.115 0.036
0.279 1 0.010
0.423 - 0.106
0,154 1 0.021
0.068 1 0.005
NP
0.131 0.056
0.299 + 0.012
0.319 = 0.004
0.140 + 0.035
0.094 + 0.015
PACOH-GP
0.038 0.006
0.262 - 0.004
0.296 = 0.003
0.251 = 0.035
0.102 = 0.010
PACOH-NN
0.037 0.005
0.267 0.005
0.302 1 0.003
0.223 1 0.012
0.119 1 0.005
MARS-GP (ours)
0.035 0.002
0.263 0.001
0.136 = 0.007
0.080 + 0.005
0.055 0.002
MARS-BNA (ours)
0.054 0.009
0.268 0.023
0.231 0.029
0.078 0.020
0.076 0.031
Table 2: Meta-learning benchmark results in terms of the calibration error. Reported are the mean and standard deviation across five seeds. MARS provides the best-calibrated uncertainty estimates.
Estimator
SwissFEL
Physionet-GCS
Physionct-HCT
Berkeley-
Argus-Conmoo
MARS-GP
0.391
0.011
1.471 0.083
2.309
0.041
0.116 0.024
0.013 0.001
SSGE score estimates
0.449 = 0.027
3.292
0,562
2.784 0.257
1.105 0.562
0.030 0.003
No spectral reg.
0.420 0.060
2.208
0.338
2.560 0.341
1.734 0.169
0.014 0.001
No GP sampling
0.471 0.059
2.994 1: 0.363
5.995 1.108
1.253 0.112
0.073 0.003
Table 3: Ablation study results for MARS components in terms of the RMSE.
MARS provides accurate predictions. We perform a comprehensive benchmark study with the environments and baselines introduced in Sec. 6.1. Table reports RMSE on unseen meta-test tasks. Both MARS-GP and MARS-BAN BNNN yield substantial improvements in the RMSE compared to the non-meta-learning baselines and significantly outperforms all other meta-learning baselines in most environments. This shows that MARS can acquire valuable inductive biases from data reliably. We hypothesize that MARS performs substantially better than PACOH-NN, which meta-learns restric- tive diagonal Gaussian priors over θ since it has much more flexibility in expressing prior knowl- edge about the data-generating process. In the majority of enviromments, MARS also outperforms PACOH-GP which meta-learns GP prior. In contrast to Gaussian marginals, MARS can meta-learn any stochastic process marginal, and, thus has much more flexibility to express inductive bias.
MARS yields well-calibrated uncertainty estimates. We hypothesize that by performing meta-learning in the function space, we avoid the pitfalls of NN over-parametrization,, which often lead over-confidence. To investigate the quality of uncertainty estimates, we compute the calibration error, which measures how much the predicted confidence regions deviate from the actual frequencies of test data in the respective regions. We list the results Table 2¹. While, in most cases, still better than the baselines, the uncertainty estimates MARS-BNN are on average worse than those of MARS-GP. This not surprising as the MC-dropout uncertainty estimates typically not reliable those of a GP. MARS-GP consistently yields the best-calibrated uncertainty estimates and, for some environments, reduces the calibration error by more than 30 % compared to the next best baseline. This provides further evidence for the soundness and efficacy of our function space approach to meta-learming Finally, the superior calibration performance suggests that the spectral regularization, together with the posterior uncertainty sampling in Sec. 5, effectively prevents meta-overfitting and accounts for epistemic uncertainty.
6.3 ABLATION STUDY
We empirically investigate the algorithm components introduced in Sec. 5 and provide supporting empirical evidence for our design decisions. First, we perform a quantitative ablation study where we vary/remove components of our algorithm. We consider MARS-GP with 1) nonparametric score estimator SSGE (Shi et al., 2018) instead of the parametric score network + score matching from Sec. 5.1, 2) without spectral regularization, and 3) without GP posterior sampling, i.e., using the GP posterior mean instead of samples for the score matching. Table 3 and 4 report the quantitative results of this ablation experiment. In the following, we discuss the three aspects separately:
Parametric score matching outperforms nonparametric score estimation. In Sec. 5. we have introduced a parametric score network which we train with the score matching loss. Alternatively, for each measurement set X, we could apply the nonparametric score estimator SSGE (Shi et al., 2018) to the function values fx ..., fx sampled from the posteriors, and directly use the resulting
Note that we omit MAML since does not provide uncertainty estimates.
8

---

Estimator
SwisFEL
Physionet-GGG
Physionet-HCT
Berkeley-
Argus Control
MARS-GP
0.035 + 0.002
0.263 0.001
0.136
0.007
0,080 :: 0.005
0.055 = 0.002
SSGE score estimates
0,151 + 0.001
0.249
0.002
0.246
0.007
0.232 4 0.010
0.210 = 0.011
No spectral reg.
0.233 + 0.041
0.265
0.012
0.244
0.009
0.192 + 0.018
0.187 0.028
No GP sampling
0.204 = 0.013
0.225 0.021
0,237 = 0.018
0.141 : 0.029
0,216 = 0.066
Table 4: Ablation study results for MARS components in terms of the calibration error.
5508
Ioo rep.
True
1.
1.0
MARS
9/0 ptel
0.5
p01
0.5
0.
0.0
0.06
-03
-0.5
0.04
-15
-1.5
0.02
-20
-2.0
0.00
-2
(a) SSGE vs. score network
(b) spectral regularization
(c) Corresponding density
Figure 2: (a) Underconfident MARS and overconfident SSGE score predictions on ten samples from a Gaussian. (b) Score estimates of MARS with and without regularization on ten function samples from zero mean GP with SE kernel. (c) Numerically integrated score predictions corresponding to the scores in (b). Overall, SSGE and MARS w/o regularization overfit and underestimate the variance, whereas spectral normalization biases MARS towards higher entropy.
score estimates during the ISVGD posterior inference in (1), This produces score estimates 'ad-hoc' and does not require us train an explicit score network. In contrast, our score network is global function that explicitly considers the measurement set and thus can exploit similarity structure across X. which SSGE cannot. To SSGE we also cannot simply add inductive bias towards higher entropy as we do through spectral normalization. Overall, the experiment results in Table 3 and 4 suggest that instantiating our general approach with SSGE performs worse than MARS. We can also observe this visually in Fig. 2a, where we plot the score estimate of MARS and SSGE for GP marginal where the true score is known. While the MARS score network slightly overestimates the variance of true generative-proeess marginal, SSGE implicitly underestimates the prior variance, leading to over-confident predictions. Finally, Table 7 in Appx. D quantitatively shows that the MARS score network provides the most accurate estimates compared to variety of nonparametric estimators.
Spectral normalization prevents meta-overfitting. In Sec. 5.3, we added spectral normalization to our score network. Here, we empirically investigate what happens when remove the spectral normalization from MARS. Figure 2b and 2c illustrate how our score network over-fits and under-estimates the true data-generating variance when we do not use spectral regularization. Quantitatively, we observe substantial increases in calibration errors and a consistent worsening of the predictive accuracy. Overall, this highlights the empirical importance of spectral normalization for preventing meta-ovvriitting and biasing the score estimates toward higher entropy.
Accounting for epistemic uncertainty of the GP interpolators is crucial. In MARS, we in- terpolate each dataset with a GP or BNN and, when performing score estimation, use samples from the posterior marginals to account for the epistemic uncertainty of the interpolation. Here, we empirically study what happens if we ignore the epistemic uncertainty and take each GP's mean predictions. When doing so, we observe detrimental effects on the RMSE and calibration error in the majority of the environments in Table and 4. This affirms that propagating the epistemic uncertainty of the interpolation into the score estimates is a critical component of MARS.
7 CONCLUSION
We have introduced a novel meta-learning approach in the function space that estimates the score of the data sata--enerating process marginals from a set of related datasets. When facing new learning task, we use the meta-learned score network as a source of prior for functional approximate BNN inference. By representing inductive bias as the score of a stochastic process, our approach is versa- tile and can seamlessly acquire/represent complex prior knowledge. Empirically, this translates into strong performance when compared to previous meta-leaming methods. The substantial improve- ments of MARS in terms of the quality of uncertainty estimates open up many potential extensions toward interactive machine learning where exploration based on epistemic uncertainty is vital.

---

ACKNOWLEDGMENTS
This research was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program grant agreement no. 815943 and the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545. Jonas Rothfuss was supported by an Apple Scholars in AI/ML fellowship. We thank Alex Hägele, Parnian Kassraie, Lars Lorch and Danica J. Sutherland for their valuable feedback.
REFERENCES
Ron Amit and Ron Meir. Meta-learning by adjusting priors based extended pac-bayes theory. In International Conference on Machine Learning, pp. 205-214. PMLR, 2018.
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 2000.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //github. com/ google/
Stéphane Canu and Alex Smola. Kernel methods and the exponential family. Nearocomputing, 69 (7-9):714-720, 2006.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. kernel test of goodness of fit. In International Conference Machine Learning, 2016.
Brian Coyle, Daniel Mills, Vincent Danos, and Elham Kashefi. The born supremacy: quantum advantage and training of an ising born machine. upj Quantum Information, 6(1):1-11, 2020.
Charles-Alban Deledalle, Samuel Vaiter, Jalal Fadili, and Gabriel Peyré. Stein unbiased gradient estimator of the risk (sugar) for multiple parameter selection. SIAM Journal on Imaging Sciences, 7(4):2448-248) 2014.
Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffmann, and Rif A. Saurous. Tensorflow distributions, 2017. URL https::p//// wwwii. wwr.viioorr/oooooooo abs/ 1711.11111 10604.
Heinz Werner Engl, Martin Hanke, and Andreas Neubauer. Regularization of inverse problems, volume 375. Springer Science & Business Media, 1996.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Vincent Fortuin, Heiko Strathmann, and Gunnar Rätsch. Meta-leaming mean functions for gaussian processes, 2019. URL https: /ttp://aaavv.//////aa/eeeeaaaava wwwp/iii.oroooooooooe..e abs
Vincent Fortuin, Adrià Garriga-Alonso, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. In Interna- tional Conference on Learning Representations, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050~1059, 2016.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint urXiv:1807.01622. 2018
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
10

---

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015.
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
URL http::ttp//. / thub com deepmi nd dm-ham haaiuu ku.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances Neural Information Processing Systems, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Johannes Kirschner, Mojmir Mutný, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adap- tive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces. In International Conference on Machine Learning. 2019a.
Johannes Kirschner, Manuel Nonnenmacher, Mojmir Mutný, Andreas Krause, Nicole Hiller, Ras- mus Ischebeck, and Andreas Adelmann. Bayesian optimisation for fast and safe parameter tuning of swissfel. In FEL2019, Proceedings of the 39th International Pree-Electron Laser Conference, pp. 707-710. JACoW Publishing, 2019b.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. 2704-2804.
PMLR, 2018.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based sermutation-invarian neural networks. In Interna- tional conference on machine learning, pp. 3744-3753. PMLR, 2019.
Qiang Liu and Dilin Wang. Stein Variational Gradient Descent: A General Purpose Bayesian Infer- ence Algorithm. In Advances in Neural Information Processing Systems, 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In International Conference on Machine Learning. PMLR, 2016.
Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopf. Amortized inference for causal structure learning. arXiv preprint 2mXi::2205.12934 2022.
Samuel Madden. Intel lab data. http://db. esail mit. wdweeddd/daaea wttp:eeeaaaaaaaaaaaaaaaeaaaaa httm,, 2004. Accessed: Sep 8, 2020.
Christopher J Milne, Thomas Schietinger, Masamitsu Aiba, Arturo Alarcon, Jürgen Alex, Alexander Anghel, Vladimir Arsov, Carl Beard, Paul Beaud, Simona Bettoni, et al. Swissfel: the swiss x-ray free electron laser. Applied Sciences, 7(7):220, 2017.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A Simple Neural Attentive Meta- Learner. In International Conference on Learning Representations, 7 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Leaming Algorithms. arXiv preprint arXiv: /003. 02999, 2018
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
11

---

Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learn- ing of generative models via cittediiffeeence score matching. Advances in Neural Information Processing Systems, 33:19175-19188 2020.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumaaeaa, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In Inter- national Conference on Machine Learning, 2014.
Thomas Pinder and Daniel Dodd. Gpjax: A gaussian process framework in jax. Journal of Open Source misc, 7(75):4455, 2022. doi: 10.21105/joss004455. 04455. URL https: /ttps://doi.oo org/100..........
21105/joss: 04455.
Yunxiao Qin, Weiguo Zhang, Chenxu Zhao, Zezheng Wang, Hailin Shi, Guojun Qi, Jingping Shi, and Zhen Lei. Rethink and redesign meta learning. arXiv preprint arXiv:1812.04955 2018.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In International Conference on Learning Representations, 2018.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal Meta-Policy Search. International Conference on Leurning Representations, 2019.
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. Pacoh: Bayes-optimal meta-learning with pac-guarantees. In International Conference on Machine Learning, pp. 9116 9126. PMLR, 2021a
Jonas Rothfuss, Dominique Heyn, Jinfan Chen, and Andreas Krause. Meta-learning reliable priors in the function space. Advances in Neural Information Processing Systems, 34, 2021b.
Jonas Rothfuss, Christopher Koenig, Alisa Rupenyan, and Andreas Krause. Mcta-learning priors for safe bayesian optimization. In Conference on Robot Learning, 2022.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In International Conference on Machine Learning. pp. 1842-1850, 2016.
Amar Shah, Andrew Wilson, and Zoubin Ghahramani. Student-t processes as alternatives to gaus- sian processes. In Artificial intelligence and statistics, pp. 877-885. PMLR, 2014.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions. In International Conference on Machine Learning, pp. 4644-4653. PMLR, 2018.
Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physsionetccomputing in cardiology challenge 2012. In Computing in Cardiology, 2012.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020.
Ingo Steinwart. A sober look at neural network initializations. arXiv preprint arXiv: /903. / //42,, 2019.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger B. Grosse. Functional variational bayesian neural networks. In International Conference Learning Representations, 2019.
12

---

Danica J Sutherland, Heiko Strathmann, Michael Arbel, and Arthur Gretton. Efficient and princi- pled score estimation with nyström kernel exponential families. In International Conference on Artificial Intelligence and Statistics, pp. 652 652-660. PMLR, 2018.
Scbastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, 1998.
Joaquin Vanschoren. Meta-learning: survey, arXiv preprint arXiv:1810.03544, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Dilin Wang, Zhe Zeng, and Qiang Liu. Stein variational message passing for continuous graphical models. In International Conference on Machine Learning, pp. 5219-5227, PMLR, 2018.
Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks. International Conference on Learming Representations, 2019.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference Learning Representations, 2020.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning, Advances in neural information processing systems, 31, 2018
Yuhao Zhou, Jiaxin Shi, and Jun Zhu. Nonparametric score estimators. In International Conference Machine Learning, pp. 11513-11522. PMLR, 2020.
13

---

A MARS IMPLEMENTATION DETAILS
This section focuses on a detailed explanation of the proposed framework. We first elaborate on the ISVGD inference process and then provide information regarding the network architecture and the hyperparameter configuration.
A.1 FSVGD INFERENCE
To perform approximate BNN inference with the meta-learned score network, we use functional Stein Variational Gradient Descent (JSVGD) (Wang et al., 2018). In the following, we explain our ISVGD implementation and how it interplays with the score network in our MARS framework.
In context, the goal of ISVGD is approximate the posterior p(hg .y²) p(y" X he hg)p(ha) over neural network mappings hg. Recall from Section 3, that X {x th E Xm are the training inputs, yd {y, 1= ) the corresponding noisy function values for new, unseen learning tasks. The FSVGD algorithm approximates the posterior using set of L NN parameter particles 0r. where each O, corresponds to the weights and biases of neural network. The particles (i.e., weights and biases) are initialized based the scheme of Steinwart (2019) (see Appendix A.2 for details).
To make the BNN inference in the function space tractable, in each iteration. ISVGD samples measurement set X from a measurement distribution y and performs its particle updates based on the posterior marginals [X, .P.yP) p(y? )p(h*) corresponding to X. We uniform distribution over the bounded domain X' as measurement distribution v and sample the L measurement points i.i.d. from it.
For each NN particle, we compute the NN function values the measurement points, i.e. h (x)( (xxxx) I = I,..., L and the corresponding posterior marginal scores:
R In X.
Vn a²
Inply Dr
V hxx In ((
(4)
likelibood
1m R0 (h2g X)
N N( - lihood variance. Unlike in Wang et al. (2018) where the stochastic process prior is exogenously given, and its marginals phh*) are approximated multivariate Gaussians,
Here, we use a Gaussian likelihood p(y p(yPPh**)
where o² is the like-
use our meta-learned marginal prior scores. In particular, we use the score networks so (h X) output as a swap-in for the prior score V, In p(h Finally, based on the score in (4) and the function values h we can compute the SVGD updates in the function space, and project them back into the parameter space via the NN Jacobian V bg..
X.
L C K wnpphhhli,ta.. X
8¹ 8' 0¹
(o NN Jacobian
V gf
Ki;
(5)
SVGD update function space
Here, 7 the SVGD step size and K (hg, h Ja the kernel matrix between the function values the measurement points based on a kernel function k(s-) y x + R. We use the RBF kernel khh, h³) exp ((-ljh h211²)/(2(x)) where la is the bandwidth hyper-parameter. The particle update in (5) completes one iteration of ISVGD. We list the full procedure in Algorithm 2.
ISVGD hyperparameter selection. For the fBNN training using ISVGD, among other parame- ters, we need to choose the step size y, kernel bandwidth le and likelihood standard deviation as we found these three to have the most substantial impact on training dynamics. We fix the number of particles to L 10 and perform 10000 ISVGD update steps. We always standardize both the input and output data based on the meta-training data's 's empirical mean and standard deviation. For the NNs, we three hidden layers, each of size 32 and with leaky ReLU activations. To initialize the NN weights, we use He initialization with a uniform distribution (He et al., 2015) and the bias initializer of Steinwart (2019) (see Section A.2 for details). Generally, choose the
14

---

Algorithm 2 Approximate BNN Inference with ISVGD (Wang et al., 2019)
Input: SVGD kernel function : (k),, bandwidth Eg step size y
Input: dataset "D' ,x",""" for target task, trained score network Sy (-:
Initialize set of BNN particles (01;-.,,1)
while not converged do X sid
sample measurement set
for 1 = 1, 1,... L do
hg + (hig (x1).... .... (...,,/((() (xx)) where X (X), xx.)),)))))))) xx.) // compute NN function values in X
Vb In ph XX yp)
Vny Inply? (hyy? (1x))) So (h X)
// posterior marginal score
0¹ - 0 (V hpg
+ KR V In ph yP)
b K₁
fSVGD update
Output: Set of BNN particles ...,... Or that approximate the BNN posterior process
15
pred
10
train data
pred std
-5
-10
-10
-15
-7.5
-5.0
-2.5
0.0
-7.5
-5.0
-2.5
0.0
2.5
-7.5
-5.0
-2.5
0.0
2.5
5.0
(a) Data generation process
(b) Constant bias initializer
(c) Steinwart bias initializer
Figure 3: a) The underlying data generating process and corresponding fBNN prediction after train- ing using the fSVGD algorithm for 2000 iterations with b) constant bias initializer, and c) Steinwart bias initializer. With the Steinwart bias initializer, we get the desirable non-linear behavior of the ISVGD BNN much faster.
kernel bandwidth & via a random hyper- hyper-parameter search over the values range of [0.1, When comparing to the original FSVGD implementation with SSGE, we fix the SSGE lengthscale to 0.2 for comparability. Note that we also experimented with using the median heuristic (as proposed by Shi et al. (2018)); however, we observed this to yield inferior performance.
A.2 BIAS INITIALIZATION
When initializing the biases to zeros or small positive constants, we find that the learned neural network maps behave like linear functions further away from and that it takes many SVGD iterations for them to assume non-linear behavior at the boundaries of the domain.
To address this issue, use the bias initialization scheme of Steinwart (2019), which initializes the biases such a way that the kinks of the leaky ReLU functions are more evenly distributed across the domain and less concentrated around zero. More specifically, initialize each bias b, as
b = - (w;,
where we is uniformly sampled from a sphere by taking w, = | 0g ~ UO, 1). x are sampled uniformly: x ~ Umninz maxx), where miny and maz, are the minimum and maximum points the input domain respectively.
By using Steünwart initialization, the neural networks show much more non-linear behavior after initialization. Compared to constant bias initialization, learning non-linear functional relationships happens much more quickly. We showcase this in Figure 3: Figure 3a displays simple data generation process of sinusoids varying amplitude, frequency, phase shift, and slope. Figure 3b displays corresponding BNN predictions after 2000 iterations with constant bias initializer (where the constant equals 0.01). and Figure 3c shows the result the same training dynamics, using Stein- wart's bias initializer instead of the constant initializer. The BNNs with the Steinwart initialization
15

---

EvaCenNter SPS
Figure 4: Architecture of MARS score estimation network. From left to right: & measurement sets consisting of input-output concatenations with 2; E R f(x;) E RR, I,.... k; inputs/outputs embeddings using spectrally normalized linear layers; two identical blocks of scaled dot product at- tention, residual layers and feed-forward position-wise NN with spectrally normalized linear layers; the final linear layer, not spectrally normalized.
assume desirable non-linear behavior much earlier during training, thus, significantly speeding up training. However, if trained for large enough number of iterations, the performance of the two networks with different bias initializations becomes much less distinct. For further details on the implications the Steinwart initializer on the training dynamics, we refer to Steinwart (2019).
A.3 SCORE ESTIMATION NETWORK
We now give overview of the score estimation network. We start by providing motivation for the required architecture, detailing the permutation equivariance properties of the network. We construct the proposed architecture step-by-step, giving the architectural details and mentioning additional architectural designs we experimented with. Finally, we comment on the optimization method and acknowledge the libraries we used in our implementations.
Incorporating task invariances. The proposed network is permutation equivariant across the k dimension: reordering the measurement inputs would result reordering the network's prediction in the same manner: Formally, for any permutation of the measurement set indices have that sy (h..( (a(i:)) V V ())))))))))))))))))))))) (()) To impose permutation equivariance, use the self-attention mechanism (Vaswani et al., 2017). Permutation equivariance is obtained by concatenating inputs (measurement points) and the corresponding functional evaluations (i.e., the concatenation constructs an object corresponding to Transformer tokens), which are then embedded and inputted the attention mechanism. The architecture is displayed in Figure 4.
Constructing the network architecture. The core of our model is composed of two identical blocks, each consisting of two residual layers, the first one applying multi-head self-attention and the second one position-wise feed-forward neural network, similar to the vanilla Transformer en- coder (Vaswani et al., 2017). Since the multi-head attention is permutation equivariant over the measurement point (i.e., token) dimension, the representation is permutation equivariant all times (Lee et al., 2019). Finally, to minimize training time, we select attention embedding di- mensions proportional to the data dimension of the environments; (c.g.. SwissFEL, detailed in Appendix B) correspond to the higher number of attention parame- ters/embedding dimensions and lower environments (e.g., the Sinusoid and Berkeley environment) to lower embeddings. Furthermore, we tune the step size and report the chosen configurations under https: / /tiinynul..... com/ 37 376wpxx. We describe the corresponding implementation details in the following subsection.
environments
Constructing the score network. We train the score network on the input/output pair concate- nations, which are then embedded onto higher-dimensional space. After this, we perform self- attention. Specifically, we fix the initialization scale self-attention weights to 2.0. As mentioned, the size of the model and embeddings varies across meta-leariing environments. Next follows position- position-wise wise feed-forward neural network, for which we use Exponential Linear- (ELU) as the activation, as implemented in Haiku's 's vanilla Transformer encoder (Hennigan et al., 2020). After- ward, we apply the self-atte attention mechanism again, following a position-we wise feed-forward neural network, after which apply the final linear layer.
16

---

Variants of the attention mechanism. We experimented with multiple variations of the attention mechanism, all being permutation equivariant in the & dimension, similar to the work of Lorch et al. (2022). We mention two other architectural designs: in the first design, rather than using the em- beddings of the concatenation of input-output points as keys, queries, and values (as performed in the vanilla Transformer encoder), we experimented with using the embeddings of the inputs the keys and queries and the embeddings of functional outputs. However, this design yields varying performance: on several low-dimensional tasks, the performance was slightly better, whereas per- formance on tasks with gigher-dimensional inputs was substantially worse. In another attempt, we experimented with learning different embeddings for inputs and functional outputs, which we then concatenated and used as keys, queries, and values. The difference in performance in this method was marginal, and we decided against it for simplicity.
Optimizer setup and employed libraries. Finally, across all experiments, we use gradient clip- ping of the prior score together with the ADAM optimizer (Kingma & Ba, 2014), with the default values set in Jax (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020). For the gradient clip- ping, we use values of 1., 10., or 100., depending on the task and the underlying properties of the data generating process. For example, for the first experiment in Appendix D. larger clipping values (50 or 100) performed better for the heavier-tailed Student's t-f process, whereas clipping at 10. re- sulted in good performance of the GP task. We train the score network for 20000 iterations. For the GP and Student-/ process implementations, we use GP-Jax (Pinder & Dodd, 2022), scikit-learn (Pedregosa al., 201 and TensorFlow Distributions packages (Dillon al., 2017).
A.4 INTERPOLATING THE DATASETS ACROSS X
Interpolation with Gaussian Processes. In GP regression, each data point corresponds to a featere-target tuple Zg. (x yt.z) Rdd TR. For the i-th dataset, write D, = (X,, y.), where X; = (Zi.1..... Ti.m.) and y; = (yi.I..... m.) ) GPs are a Bayesian method which the prior P(h) GP (h) m(rr), k l (Ir,x') is specified
positive definite kernel k X' X IR
and a mean function m : X + R. In this section, we assume a zero mean GP and omit writing the dependence on m(-). As the GP kernel, we use the Matérn covariance function
-' k(x, X2,x T(v)
x'
x'
K,
2v
with degree v, lengthscale f, and I'(-) representing the gamma function, and K, the modified Bessel function of the second kind. We use fix the degree of the Matérn kemnel to v = 5/2 and choose the lengthscale f via 4-fold cross-validation (CV) from 10 log-uniformly spaced points in [0.001, 10). We select the lengthscale that maximizes the 4-fold CV log marginal likelihood, averaged across the n tasks. In Alg. 3, we give the full procedure of selecting the lengthscale and fitting GPs to the meta-training tasks.
Algorithm Fitting the GPs the meta tasks
Input: n datasets (Dr)i where D; {(xi)) }1r)) m. 1=1 Input: set of p Matérn lengthscale value candidates = {
Input: zero-mean GP prior Pff) = gP (f be (x)xx),, specified by the Matérn kernel ke
X' x X
+ R
for i --r=l 1,.;,,,.,, do (GP(S ke) (score: CVa P)
|/ cross-validation D,
j = arg max, = I....p Σ score;,j Output: n GP posteriors (gP(( ((r,,)))))=
selecting the optimal kernel parameters
Interpolation with Bayesian Neural Networks. For MARS-BN BNNN in order to interpolate the datasets, the Monte-Carke dropout (MC-dropout) approach by Gal & Ghahramani (2016) which trains neural networks with dropout and also uses dropout at inference/test time to generate random forward passes through the network.
Consider a neural network with L layers. For each layer I, let us denote as M, the (random) weight matrix of dimensions K,x K2-1- MC-dropout samples these weight matrices by randomly dropping
17

---

TOWS:
M Wr diag (#w))
3 Bernoulli (pi) for 1,...,, L, -,.j-...... Ki-1
given some dropout probabilities pt and trained weight matrices W, variational parameters. We write w (Wi) for the set variational parameters. The binary variable 21,j 0 corresponds to neuron j in layer / being dropped. We can view these random weights M, as samples from some variational distribution with parameters w.
For every dataset D; we fit the respective MC-dropout BNNs to the dataset D; resulting in set of variational parameters w,. This yields the posterior distribution [X, over function values f, for arbitrary measurement sets X. Sampling from p(f," X, wi) corresponds to a random (i.e., with randomly dropped neurons) forward pass with inputs X. Analogously to (3) case, the score matching loss modifies to
(( Ex
E ((XI)))))
Ir(Vpxs (C. X))
order to approximate the inner expectation in Eqn. (7), we sample one realisation of p(fx X, wi) per iteration by performing random forward pass through the BNN. We fix the dropout probabilities of all layers P1 = = PL = p. across all datasets to a value p. which is a tunable hyperparameter.
In the experiments, we use fully connected neural network with 3 hidden layers of size 32 each. As the activation, we use the leaky-relu. To train each network we minimize the MSE and with the Adam optimizer with weight decay.. Weights and bias initialization set to default initializations by Haiku (Hennigan et al., 2020), i.e., truncated normal distiribution for the weights and constant initialization for the biases.
We fix the number of epochs to 100. The hyper-parameters are selected from the following:
learning rate: {le le 1e le 1.-4, 1.5e- 5,555 ()
weight decay: {0., le 12, 1e 1e-3, 11--4 35 5c
batch size: (1, 2, 4, 8, 16)
dropout: {0.1, 0.25, 0.5, 0.7, 0.8)
Note that for Physionet dataset, the datasets varying and some have less than 16 data points, the selected batch size is 16 and the dataset contains less than 16 points, we select the full dataset.
A.5 CHOOSING THE MEASUREMENT DISTRIBUTION V
The measurement distribution should be supported relevant parts of the domain X' from which may queries at test time. In our experiments, choose (x))) as uniform distribution over the hypercube X' X C Rd which conservatively covers the data. In particular for each dimension k = 1,.... d we compute the minimum and maximum value that occurs in the meta- training data, i.e.,
min
min min
(k) max
max
max
1....
I.....r.s
and expand the respective ranges by 20% on each side
2 low
(k)
(k)
0.2 (xr(A) max
I high
=
x mux
(k)
+ 0.2 (x
We construct the hypercube & x 1ow high
(1) (1)
Noo mngh from the Cartesian product of x
x
(d) (d)
(k) (k) low
the expand ranges
I
2 high
B META-LEARNING ENVIRONMENTS
In the following, we provide details on the meta-learning environments used in our experiments in Section 6. We list the number of tasks and samples in each environment in Table 5.
18

---

Sinusoid
SwissFEL
Physionet
Berkeley
Berkeley"
Argus-Control
20
5
100
10
36
20
m2
8
200
2-
30
288
500
Table 5: Number of tasks and samples per task m, for the different meta-learning environments.
B.1 SINUSOIDS (SYNTHETIC ENVIRONMENT)
The sinusoid environment corresponds to a simple 1-dimensional regression problem with a sinu- soidal structure. It is used for visualization purposes in Figure and Figure 3a. Each task of the sinusoid environment corresponds to parametric function
fa.b,c,a()) = * a (1.5 * b)) +c,
(8)
yielding sum of affine and sinusoid function. Tasks differ the function parameters (a, b, c, ()) that are sampled from the task environment T as follows:
((0.77113)) b~ b~N(00,..1", c~,(05000019) 8~ 8~NN00.,00.3) 0.2²).
Figure 3a displays functions fa.b.e,8 with parameters sampled according to (9). To draw training samples from each task, uniformly sample x from ()-5,55) and add Gaussian noise with stan- dard deviation 0.1 to the function values f(x):
x ~ (<(5,)))) y ((
(10)
B.2 SWISSFEL
Free-electron lasers (FELs) accelerate electrons to a very high speed to generate shortly pulsed laser beams with wavelengths in the X-ray spectrum. The X-ray pulses from the accelerator map nanometer-scale structures, thus facilitating molecular biology and material science experiments. The accelerator and the electron beam line of FEL consist of multiple magnets and other ad- justable components, which have several parameters that experts adjust in order to maximize the pulse energy (Kirschner al., 2019a). Due different operational modes, parameter drift, and changing (latent) conditions, the laser's 's pulse energy function, in response to its parameters, changes across time. As a result, optimizing the laser's parameters recurrent task.
Meta-learning setup. The meta- meta-learning environment represents different parameter optimization runs (i.e., tasks) on SwissFEL, an 800-meter-long free electron laser located in Switzerland (Milne al., 2017). The input space is 12-dimensional and corresponds to the laser parameters, whereas the regression target corresponds to the one-dimensional pulse energy. We refer to Kirschner et al. (2019b) for details on the individual parameters. Each optimization run consists of roughly 2000 data points generated with online optimization methods, yielding non-i.i.d. data, which becomes successively less diverse throughout the optimization. To avoid issues with highly dependent data points, we take the first 400 data points per run and split them into training and test subsets of size 200. As we have total of runs (tasks) available, we use 5 of them for meta- taatt-triiinngg and the remaining 4 for meta-testing.
B.3 PHYSIONET
In the context of the Physionet competition 2012, Silva et al. (2012) have published an open-access dataset of patient stays in the intensive care unit (ICU). The dataset consists of measurements taken during the patient stays, where up to 37 clinical variables are measured over the span of 48 hours, yielding time series of measurements. The intended task for the competition was the binary clas- sification of patient mortality. However, the dataset is also often used for time series prediction methods due to large number of missing values (around 80 across all features).
Meta-learning setup. To set up the meta-learning environment, treat each patient separate task and the different clinical variables as different environments. Out of the 37 variables, we picked the Glasgow coma scale (GCS) and hematocrit value (HCT) environments for study since
19

---

RMSE
Calib, enror
Full dataset
Partial dataset
Full dataset
Partial dataset
Vanilla GP
0.276 + 0.000
0.258 + 0.000
0.109
0.000
0.119 + 0.000
Vanilla BNN
0.109 0.004
0.151 = 0.018
0.179 0.002
0.206 = 0.025
MAML
0.045 + 0.003
0 0.027
/
BMAML
0.073 + 0.014
0.222 0.032
0.161
0.013
0.154 + 0.021
NP
0.079 + 0.014
0.173
0.210
0.000
0.140 = 0.035
MLAP
0.050 0.034
0.348 0.034
0.108
0.024
0.183 = 0.017
PACOH-NN
0.130 + 0.009
0.160 + 0.070
0.167 + 0.005
0.223 + 0.012
MARS
0.093 + 0.002
0.116 + 0.024
140 0.002
0.080 + 0.005
Table 6: Prediction accuracy and uncertainty calibration full and partial Berkeley-Sensor dataset. On the full dataset, MARS performance is less competitive due to the strong auto-correlation the data which is not taken into account in the BNN likelihood. On the partial dataset, which has less dependency among the data points, MARS outperforms all other meta-learning methods.
they are among the most frequently measured variables in this dataset. From the dataset, we remove all patients where less than four measurements of CGS (and HCT, respectively) are available. From the remaining patients, we used 100 patients for meta-training and 500 patients for meta-validation and meta-testing, Since the number of available measurements differs across patients, the number of training points my ranges between and 24.
B.4 BERKELEY-SENSOR
The Berkeley dataset contains data from 46 temperature sensors deployed in different locations at the Intel Research lab in Berkeley (Madden, 2004). The temperature measurements are taken over four days and sampled at 10-minute intervals. Each task corresponds to one of the 46 sensors and requires auto-regressive prediction, particularly predicting the subsequent temperature measurement given the last ten measurements.
Meta-learning setup. The Berkeley environment, used in Rothfuss et al. (2021a), uses 36 sen- sors (tasks) with data for the first two days for meta-training and the last 10 for meta-testing. The meta-training and meta-testing are separated temporally and spatially since the data is non-i.i.d. Data are abundant, and the measurements are taken at very close intervals. Thus, the features and the train/context data points are strongly correlated, violating the i.i.d. assumption that underlies our factorized Gaussian likelihood and Bayes rule in Section 3, causing the BNN to over over-weight the empirical evidence and making over-confident predictions. To alleviate this problem, we subsample the data. In particular, we randomly select 10 out of the 36 training tasks, and instead of using all measurements, we randomly sample 30 of them. This has two effects: First, makes the data less dependent/correlatte and thus more compatible with our Bayesian formulation. Second, we increase the epistemic uncertainty by using less data, making the calibration metrics more meaningful. The results reported in Section 6 correspond to the sub-sample sampled data.
For completeness, we also report the results for the full dataset as in Rothfuss et al. (2021a) in Table 6. Other meta-learning baselines, such MAML MLAP,, perform better than MARS the full dataset since they do not explicitly use the Bayes rule with i.i.d. assumption or weight the empirical evidence less. Note that MARS performs worse due the Bayesian inference meta-test time rather than our meta-learning approach. Accounting for the strong auto-correlation of the data in the likelihood would most likely resolve the issue. On the sub-sampled environment, MARS again performs best.
B.5
The final environment we use in our experiments robot case study. In particular, it aims at tuning the controller of the Argus linear motion system by Schneeberger Linear Technology. The goal is to choose the controller parameters so that the position error is minimal. In our setup, each task is a regression problem where the goal is to predict the total variation (TV) of the robot's position error
20

---

signal when controlled by PID controller in simulation. The regression features the three PID controller gain parameter parameters.
Meta-learning setup. Overall, the environment consists of 24 tasks, which 20 are used for meta-training and the remaining 4 for meta-testing. Each task corresponds to a different step size for the robot to move, ranging from 10pm to 10mm. At different scales, the robot behaves differently in response to the controller parameters, resulting in different target functions. This presents good environment for transferring similarities across different scales while leaving enough flexibility in the prior adjust the target function at step size.
C EXPERIMENTAL METHODOLOGY
In the following, describe our experimental methodology used in Section 6.
C.I. OVERVIEW OF THE META-TRAINING AND META-TESTING PHASES
Evaluating the performance of meaanleearer consists of two phases, mela-training and mela-testing. The latter phase, mela-testing, be further split into larget training and larget lesting. In particu- lar, for MARS the phases consist of the following:
Meta-training: The meta- meta-training datasets 1 used to train the score estimator net- work (see Algorithm 1).
Equipped with knowledge about the underlying data-generation process, i.e., the score network, we perform BNN inference ing context dataset D In particular, we run ISVGD with the score network as a swap-in for the marginal scores of the stochastic process prior (see Algorithm 2). As result, obtain set of NN particles that approximates the BNN posterior in the function space.
Target training:
target task with correspond-
Target testing: Finally, we evaluate the approximate posterior predictions on a test set Dt corresponding the same target task. In particular, compute the residual mean squared error (RMSE) and the calibration error as performance metrics. We describe the evaluation metries more detail in Section C.2.
The target training and testing are performed independently with the meta-lcarned score network for each test task. The metries reported as averages over the test tasks.
The entire meta-training and meta-testing procedure are repeated for five random seeds that influ- ence the score network initialization, the sampling-based based estimates in Algorithm well as the initialization of the BNN particles for target training. The reported averages and standard deviations based on the results obtained for different seeds.
C.2 EVALUATION METRICS
During target-testing, we evaluate the posterior predictions on a held-out test dataset Dt Among the methods employed in Section 6, MARS, PACOH-NN, NPs, MLAP, Vanilla BNN and Vanilla GP yield probabilistic predictions p(y" D*) for the test points rf E Dt, For instance, in the case of MARS. PACOH-NN, and Vanilla BNN where the posterior approximated by a set of NN particles {0) ..., 0r.) and we use Gaussian likelihood, the predictive distribution is an equally weighted mixture of Gaussians:
1 (x))) L
ply [ D*)
(11)
The respective mean prediction corresponds to the expectation of p. that is & = B(y" D*). In the case of MAML,, only the mean prediction is available.
"=""""""""""
21

---

RMSE
GP-2D
GP-3D
TP-2D
TP-3D
KEF
8.836 + 0.000
6.625 + 0.000
8.852 + 0.000
13.694 + 0.000
NKEF
4.322 + 0.000
4.402 0.000
5.936 + 0.000
8,402 + 0.000
KEF-CG IMQ"
3,009 + 0.000
5.197 + 0.000
0.000
5,473 + 0.000
KEF-CG RBF*
1.145 0.000
2.237 0.000
0.000
4.370 0.000
y-method IMQ'
4.682 + 0.000
9.171 0.000
0.000
10.81 + 0.000
y-method IMQp"
4.682 = 0.000
9.171 + 0.000
11.53 0.000
5.872 0.000
w-method RBF*
1.718 + 0.000
4.035 + 0.000
4.278
0,000
5,200 + 0.000
SSGE
1.166 + 0.000
1.989 + 0.000
1.753
0.000
4.822 0.000
Stein estimator
0.912 = 0.000
4.058
+ 0.000
3.519 0.000
7,251 + 0.000
MARS network
0.664 + 0.179
1.375 + 0.215
1.093 + 0.241
3.199 0.418
Table 7: RMSE between the true and predicted scores of sitthree-dimensionall marginal distribu- tions of GP and TP with an RBF kernel and sinusoidal mean function with linear trend. MARS score network significantly outperforms all nonparametric score estimators. Estimators with cor- respond to curl-free estimators (Zhou et al., 2020) with either an IMQ or RBF kernel.
Evaluating prediction accuracy (RMSE) mean-squared error (RMSE)
Based the mean predictions, we compute the root-
RMSE
Σ
(ut B(ytlot, [t))
(12)
D
(ED
which quantifies how accurate the mean predictions are.
Evaluating uncertainty calibration (Calibration error) In addition to the prediction accuracy, also assess the quality of the uncertainty estimates. For this purpose, calibration, iee, a probabilistic predictor is calibrated if the predicted probabilities are consistent unseen data. We regression calibration error similar to Kuleshov et al. (2018) in order to quantify how much the predicted probabilities deviate from the empirical frequencies.
the concept of
with the observed frequencies
Let denote the predictor's cumulative density function (CDF) I x, )ddj where p(ū)x, D' Jd the predictive posterior distribution. For confidence lev- els qh ... qu S compute the corresponding empirical frequency
f 1
(yl
D*) <
(xt
v!)
E D
Di
based on some test dataset DD_ If the predictions calibrated, expect that Ín Following Kuleshov et al. (2018), we define the calibration eтror metric as a function of the residuals ğn- Qh
Qh.
calib-err Σ
1 9n1-
Note that report the average of absolute residuals Iân ml. rather than reporting the average of squared residuals 14, qul 2 as done by Kuleshov et al. (2018). This is done to preserve the units and keep the calibration enror easier to interpret. In our experiments, we compute the empirical frequency using M 20 equally spaced confidence levels between 0 and 1.
C.3
SELECTION
For each of the enctanenvisossntss and algorithms. lect the hyper
separate dyper-paramette search syype--arameters. parameters. In particular, we use 30 randomly sampled hyperparameter configurations
across five randomly selected seeds and select the best-performing in terms of either RMSE or
22

---

Cosine similarity
GP-2D
GP-3D
TP-2D
TP-3D
KEF
0.475 + 0.000
0.410 + 0.000
0.532 + 0.000
0.485
0.000
NKEF
0.556 + 0.000
0.434 + 0.000
0.384 + 0.000
0.318 = 0.000
KEF-CG IMQ'
0.286 + 0.000
0.308 + 0.000
0.303 + 0.000
0.538 0.000
KEF-CG RBF
0.621 = 0.000
0.457 0.000
0.479 0.000
0.428 0.000
v-method IMQ"
0.337 + 0.000
0.286 0.000
0.251 + 0.000
0.384 = 0.000
v-method IMQp"
0.337 = 0.000
0.286 0.000
0.374 0.000
0.411 0.000
e-method RBF
0.809 +: 0.000
0.759 + 0.000
0.628 + 0.000
0.786 + 0.000
SSGE
0.943 + 0.000
0.601 + 0.000
0.661 0.000
0.521 0.000
Stein estimator
0.778 + 0.000
0.851 = 0.000
0.626 + 0.000
0.590 = 0.000
MARS network
0.956 = 0.110
0.889 + 0.048
0.737 0.097
0.706 Σ 0.061
Table 8: Cosine Similarity between the true and predicted scores of two/three siithreediimessionall marginal distributions of GP and TP with an RBF kernel and a sinusoidal mean function with a linear trend. estimators. Estimators with
MARS score network performs competitively to nonparametric correspond to curl-free estimators (Zhou et al., 2020) with either an IMQ or RBF kerne..
calibration error. For the reported results, we provide the chosen hyperparameters and detailed eval- uation results in attps://tiinee com/ 37 376 6xp8xe. The code scripts for reproducing the experimental results are provided in our repository?.
D FURTHER EXPERIMENT RESULTS
We provide further empirical evidence for the proposed method. In particular, we showcase that:
1. Our parametric score matching approach performs favorably to many nonparametric score estimators.
2. Regularization via spectral normalization does not hinder the flexibility of the network.
3. Sampling from the GP posteriors when training the score network successfully incorporates epistemic uncertainty in areas of the domain where meta-training data is scarce.
D.1 PARAMETRIC VS NONPARAMETRIC SCORE ESTIMATION METHODS
We start by comparing the performance of MARS to that of several nonparametric score estimators, described in Zhou et al. (2020).
Nonparametric score estimators. There are several theoretically motivated nonparametric score estimation methods with well-understood properties and straightforward implementations. Their simplicity and fiexibility make them popular choice (Coyle et al., 2020; Wang et al., 2019; Deledalle et al., 2014). KEF (Canu & Smola, 2006) performs regularized score matching inside kernel exponential family; this can further be efficiently approximated by the Nyström method (NKEF) (Sutherland et al., 2018). Liu et al. (2016): Chwialkowski et al. (2016) propose a Stein estimator, and Shi al. (2018) propose Spectral Stein Gradient Estimator (SSGE) by expanding the score function in terms of the spectral eigenbasis. Together with iterative methods (the y-method and Landweber iteration (Engl al., 1996)), these approaches can naturally unified through a reg- ularized, nonparametric regression framework (Zhou et al., 2020). in which the conjugate-gradient version of the KEF estimator is also proposed (KEF-CG). our evaluations, we reports results for curl-free sccore estimators with both an inverse multiquatric (IMQ) or RBF kernel.
MA RS vs nonparametric estimators. To compare the performance of MARS to nonparametric methods, we consider score estimation of marginal distributions of Gaussian Processes (GP) (Ras- mussen, 2003) and Student's s-t processes (TP) (Shah et al., 2014). In both cases, as the stochastic
23

---

True
1.0
v,20
0.
-05
-1.0
-10
-15
-30 -5
-20
-30
10
20
Figure 5: Predictions of the MARS score network, trained on 45 samples from Laplace (left) and Student's s-t distribution (right). The network approximates the score functions sufficiently well, showcasing that spectral normalization of linear layers does not hinder its flexibility.
process mean function, we use the sinusoidal mean function 2r + (2x), and for the covariance kernel, we choose the radial basis function kernel, with the lengthscale parameter set use Tensorflow Probability's 's (Dillon et al., 2017) implementation of both GP and TP and set all other parameters to their default values.
We
We perform four experiments, the first two estimating the marginal scores of a GP and the last two on the In all experiments, we sample one measurement set X containing either two (X = {=1,, 222)) or three points (X [2, of dimension one, i.e., 2y E IR, Vi. For all ex- periments, the z; follow a uniform distribution: for the GP experiment, I; ~ U[[-,, 5)) whereas for the TP experiment, 2, 1[), vi. In the experiments with measurement sets consist- ing of two points, the score network and the nonparametric estimators are trained on 50 samples from the corresponding wioddimesioona marginal distributions. For the measurement sets of size 3, we train the score network and nonparametric estimators on 200 samples from the corresponding three-dimensional marginal distribution. When evaluating the performance of MARS, average performance over five different seeds after training the network for 2000 iterations in the nvo-dimensional marginal case and 5000 iterations in the dimeessinnii case. We measure the quality of the score estimates via the RMSE and cosine similarity between the estimates and true scores. The corresponding evaluation results are reported in Tables 7-8.
take the
The RMSE measures deviations of the estimated scores in both direction and magnitude of the gradients. For the SVGD particle estimation, it is more important that the gradients in the vector field point in the correct direction than having the correct magnitude. Thus, we also report the cosine similarity, which only quantifies how well the directions in the estimated vector field match the directions in the true vector field while neglecting errors in the score magnitude. As we in Table and 8, MARS consistently outperforms the nonparametric score estimators. Among the SSGE in our ablation study nonparametric methods, SSGE performs best average. Hence, in Section 6.3.
D.2 FLEXIBILITY OF THE REGULARIZED NETWORK
Recall from section 5.3 that, order to prevent overfitting, we perform spectral normalization the weights of the linear layers by re- re-parameterizing the weights by W == W //|l/// // To investigate whether this hinders the flexibility of the network's outputs, we visually examine the network's 's predictions on the following two tasks, in which we estimate the scores of one-dimensional Laplace and Student's-I distribution.
In order to estimate score Vz log p(x) for some unknown distribution p(z) (rather than marginal
distribution scores Vxx In p(fx) for some unknown stochastic process p(f)), use 45 samples X {" x445 from: (i) one-dimensional Student's s-t distribution with v = degrees of freedom, location parameter μ 0, and scale parameter = and (ii) one-dimensional standard Laplace distribution, i.e., with location parameter = O and scale parameter o 1.
To perform distributional score estimation, need to make slight modification to the original score network architecture; rather than concatenating input-output pairs (i.c., concatenating x; and only the distributional samples inputs. To be more precise,
f(x:), as shown in Fig. 4), at every iteration, we randomly select (without replacement) & = 8 inputs input the score network. We train the network using the regularized score matching loss, i.e.,
of} C X, which
24

---

perform spectral normalization of the layers. The network is trained for 2000 iterations, using learning rate of 0.001. For this experiment, we set the attention embedding dimension to 32, key size to 16, and use 8 attention heads. The remaining hyperparameters respective predictions are displayed in Figure 5. We observe that the score network approximates the score functions well and that the regularization does not hinder flexibility too much.
set according to Sec. A.3. The
D.3 INCORPORATING UNCERTAINTY THROUGH GP INTERPOLATION
In addition to the ablation study performed in Section 6.3, we visually investigate the implications of sampling functions from the GP posteriors during score matching. In particular, we do so using the sinusoid environment, detailed in Appendix B.I.
Experiment setup. In this experiment, we sample 10 functions from the environment and evaluate each function at five randomly selected inputs in the [-5,- -2] [2, range. We investigate whether the proposed GP interpolation method promotes uncertainty in the [-2, 2] part of the domain.
In order do so, let recall Algorithm in which learn the trained score network 8ş- The algorithm first fits a GP to each of the datasets Di...., D.,. Then, at every step, the algorithm samples a measurement set X sid v, and then, for every dataset D; (i.e., for every collection of input/output pairs x,P yP)),, the algorithm samples function values from the GP posterior marginal: pK~ XX, yP). We compare this approach training the score network Sø on the mean predictions of the posterior marginal p(fx' X X?, yyP), i.e., where f. = MMrr? xx, sP)) order to distinguish between the two approaches, we denote the score network trained on the posterior marginal mean predictions with sy and use FX := P'te?" |x.f x as shorthand notation for the mean predictions of the posterior marginals. Observe that, for measurement sets X close to X the samples fx and fX will be very similar, whereas when X is far from XP, the variability of the samples f.x will be mueh larger. In the following, we showcase that this variability successfully incorporates uncertainty about the areas of the domain X' with little no data.
In order to compare the two approaches, we visually compare the overall framework when the FSVGD network is trained with s (the score network trained on fxx, is the GP mean predictions),
and when trained with Sy (the score network trained on f.X the samples from the GP posterior marginal), as performed in the original MARS algorithm.
Empirical findings. We plot the corresponding ISVGD-BNN (fBNN) predictions, trained using the two score networks s and s.. Both fBNNs fitted to four points, where inputs 1; 24 sampled uniformly from 5-5, (i.e., the whole input domain), and their functional evaluations are obtained according to the sinusoid environment.
The results are visualized in Figure 6. The first two plots on the left Figure correspond to posterior predictions of two randomly selected GPs fitted to the tasks, where no task contains information in the [-2, input range. The middle plot corresponds to the fBNN network predic- tions, where the network was trained using the ISVGD algorithm and the score network 8 The last plot corresponds to the (BNN network predictions, where fBNN is trained using the FSVGD algorithm and the score network Sg. We observe a clear difference between the two approaches: MARS (trained using $y) successfully incorporates the epistemic uncertainty in the [- part of the input domain into the IBNN posterior, yielding less confident predictions in the area where no data was available during meta-training. In contrast, when use GP posterior means instead of samples when fitting the score network, the resulting BNN predictions entirely ignore the epistemic uncertainty that arises due to the fact that don't 't know the function values in [-2, 2]. This may lead to overconfident posterior predictions.
D.4 COMPARISON OF INTERPOLATION METHODS
So far, we have experimented with using GPs (MARS- GA) and MC-dropout BNNs as interpolators (MARS-BNN) and sample random random function values from the corresponding posteriors when training our score network. In the ablation study (performed in Sec. 6.3), we also experimented with
25

---

11a Art pnal TED
-10
Figure 6: MARS prediction samples from the sinusoid environment, with data in the [- range. Left: posterior predictions of two randomly selected GPs fitted to the meta-tasks. Middle: fBNN predictions, fitted using s the score estimation network trained GP mean predictions. Right: fBNN predictions, fitted using $p. the score estimation network trained samples from the GPs. Training IBNN with Sy (by sampling from the GPs) successfully incorporates uncertainty about the [-2, 2] part of the input domain.
SwissFEL
Physionet-GGG GCS
Physionet-HHe HCT
Porkeley-Sensee
Argus-
Argus-Control Control
GP mean
0.471
0.059
2.994 + 0.363
5.995 + 1.108
1.253 0.112
0.073 + 0.003
MARS-GP
0.391 + 0.011
1.471 = 0.083
2.309 + 0.041
0.116 + 0.024
0.013 + 0,001
determ. NN
0.380 + 0.032
2.891 0.042
2.530 = 0.049
0.306 + 0.068
0.017 = 0,003
NN Ensemble
0.423 + 0.031
1.539 0.074
2.281 = 0.014
0.134 + 0.045
0.015 + 0.001
MARS-BNN
0.407 + 0.061
1.307 0.065
2.248 + 0.057
0.113 0.015
0.017 + 0.003
Table 9: Comparison of various interpolation methods in terms of the RMSE. Reported are the mean and standard deviation across five seeds. MARS-BNI and MARS-GI GGP perform the best.
using GP's mean predictions rather than sampling from the GP posterior. Here, we provide empirical analysis of using two further interpolation methods: (i) deterministic NNs, and (ii), ensembles of NNs in generic MARS pproach.
Both the deterministic NN and the NN ensemble interpolators are trained similarly to the MC- dropout BNNs, described in Section A.4. For the NN ensembles, choose the number of of ensemble members as hyper-parameter from {5, 10, 20). In case of the deterministic NN, we do perform any posterior sampling in (3) and just the NN's predictions function values for the score estimation. In case of the ensemble, we randomly sample the function values corresponding one ore multiple ensemble member. The number of function value samples from the ensemble is chosen as hyper- parameter and selected from {1, 2, 3, 4}. The results for employing MARS with different interpolators are reported in Table 9-10. Not surprisingly, the not accounting for epistemic uncertainty (i.e., GP mean and deterministic NN) performs much worse in the majority of environ- ments, in particular, w.r.t. the calibration error. Among the neural network based interpolators, the MC-dropout BNNs consistently perform the best. This is why suggest their usage and evaluate them in the main part of the paper.
SwissFEL
Physionet-GCS
Physionet-HCC HCT
Berkeley-Sensoo
Argus- Control
GP mean
0.204 + 0.013
0.225
0.021
0.237 + 0.018
0.141 + 0.029
0.216 1 0.066
MARS-GP
0.035 + 0.002
0.263 + 0.001
0.136 + 0.007
0.080 :: 0.005
0,055 1 0.002
determ. NN
0.070 + 0.025
0.249
+ 0.014
0.233 : 0.016
0.112 + 0.022
0.088 + 0.028
NN Ensemble
0.081 + 0.010
0.238 0.018
0.248 + 0.021
0.129 +: 0.028
0.118 + 0.014
MARS-BNT
0.054 + 0.009
0.268 0.023
0.231 + 0.029
0.078 0.020
0.076 + 0.031
Table 10: Comparison of various interpolation methods in terms of the calibration error. Reported
are the mean and standard deviation across five seeds. MARS-GP performs best, with MARS-BNN second best performing.
26


Reviews:
[{"review_id": "aKeUvznmc3t", "reviewer": "Reviewer_Wct9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for meta learning in function space, denoted MARS.\n\nTheir approach cosists of two stages:\nfirst, the authors train a GP on task-specific data to interpolate between different function values.\nThey then use this to be able to train a score function model given by a transformer which estimates the score of the meta learning model per task.\nSecond, the authors use their learned score matching network to perform functional BNN inference and meta-learn a prior.\n\nIn their experiments the authors show good performance compared to various baselines.\n", "summary_of_the_review": "The authors propose an interesting, complicated, and ultimately empriically useful way to perform meta-learning in function space using BNNs.\n\nWhile the approach has some modeling drawbacks by its dependence during training on GPs which I feel makes the approach somewhat less likely to generalize than I would have hoped, I am very excited about the ideas around amortizing the score estimate and the overall positioning and technical execution here.\n\nAs such, I believe this paper has value to the community for its  technical ideas and the strong presentation, even if I have doubts about the core model.", "strength_and_weaknesses": "Strengths:\n- The approach is very elegant conceptually, since the authors strive to overcome the specific problems of meta-learning and functional BNN estimation via score functions jointly and develop an integrated approach which utilizes amortization of the score estimate using a transformer based on a graceful objective.\n- the performance of the approach seems to be good in the empirical part of the paper\n\n\nWeaknesses:\n- There is one aspect in this paper that feels dissatisfactory: the fact that the authors require a GP to be learned on their datasets in order to be able to learn a NN, since this inherently assumes the same inductive bias as the GP and the score function estimator will be biased towards the types of interpolations of the GP. It is basically somewhat suspicious that the authors rely on the GP as the key to overcome here but in their model they cannot overcome the GP's limitations in case it is misspecified as it directly is upstream of their score estimator.\n- Given that the GP is fit directly on the observations, this also means the authors inherit all the GP limitations in terms of computation and so on, which here of course is only relevant for training and not for testing, but is still somewhat dissatisfactory.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FhaK2_ZDgL", "reviewer": "Reviewer_HXi8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new Bayesian meta learning algorithm which aims to relax the traditional restricted task prior. The idea is to use a stochastic process prior for all tasks rather than plain diagonal Gaussian prior for over-parameterized Bayesian neural networks. The new prior is claimed to be with less possibility of over-fitting and improved uncertainty estimation. Further, instead of estimating the prior, the authors propose to only fit the score function by a transformer encoder and then use fSVGD in meta training and test. Specifically, a GP is fitted for each task in order to generate the values on all possible measurement set and spectral normalization is used to further reduce the tendency of overfitting. ", "summary_of_the_review": "The paper is well written and easy to follow. The overall idea is novel and interesting. There are still some weaknesses regarding the design (please see Weakness for more details). ", "strength_and_weaknesses": "Strength\n\n1. The paper is well scoped and design flow is clear. Each new idea is given a clear motivation or observation. \n2. The idea of fitting score function by a network for meta learning is novel and interesting. \n3. The experimental results show the consistently good performance comparing benchmark methods. \n\nWeakness\n\n1. The authors use fSVGD rather than functional BNN, but no comparative evaluations are given in the experiments to verify this choose. Since fSVGD is particles-based, it may affect the computational efficiency comparing normal fBNN. It is expected to what the additional advantages comparing with fBNN with GP prior.\n\n2. A transformer encoder is used to model score function considering the permutation equivariance property of the process. However, there is another property of the process that is marginal consistency. Is there any control of the encoder to ensure such property? If not, what is the possible effect when loss such property? \n\n3. In 5.2, the authors propose to fit each task by a GP in order to use eq. (2). If you can use a GP to model each task, do you still need the score network? Can we just fit a prior for all GPs? like a Hierarchical GP? Have you tested on such option? \n\n\n\n\n\n\n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1tiLF7nWoCy", "reviewer": "Reviewer_VdFm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for Bayesian meta learning by learning priors in the function space. The key insight that distinguishes this work from prior literature is that functional inference techniques only make use of the prior score function and do not require a normalized prior density; therefore, instead of fitting a restricted form of prior distribution during meta training, one can fit a flexible score model of the function outputs and use it to transfer knowledge to new tasks. ", "summary_of_the_review": "Overall I am positive about this paper and would be happy to raise my score if the rebuttal sufficiently addresses the weakness part. The idea is novel and addresses an important problem in Bayesian meta-learning. The empirical study is very careful and demonstrates the effectiveness of the proposed method.", "strength_and_weaknesses": "## Strengths\n\n* The paper leverages ideas from several different areas (meta learning/GPs/BNNs/score estimation) and combines them in a creative way. The review paragraphs are very well-written and clearly convey the author's line of thought. \n\n* The idea of using score models instead of normalized priors for Bayesian meta learning is a clever one and overcomes many challenges faced by existing methods. \n\n* The experimental results are strong, if not extensive, and come with careful ablation study that shows the benefit of each component. \n\n## Weaknesses\n\n* One big limitation I could see from the formulation is that different tasks must share the same input space and output space. I am not an expert in meta learning so not sure if this is a commonly-made assumption, but it does feel significantly limiting the application scope. \n\n* Using a user-specified measurement distribution seems unnatural to me. Why not just rely on the empirical distribution of X from the training data? \n\n* Relying on GPs to interpolate at places where there is no training data is also a bit unsatisfying. I could imagine the GP's behavior will have a great influence on the performance. Could we do an ablation study by remove the GP? Also, it might be difficult to answer this question--why not use a GP prior if we are relying on a GP anyway?\n\n* In the ablation study, it would be nice to compare to the nonparametric score estimator using curl-free kernels (i.e., those based on kernel exponential families) which were shown to have drastically better performance in Zhou et al.\n\n* Some experiment details are missing, e.g., what is the measurement distribution being used in experiments? What kind of algorithms are used to fit the GP? Do you use a sparse approximation or the dataset is small enough?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%