FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning
Authors: Tristan Cinquin
Marvin Pförtner
Vincent Fortuin
Philipp Hennig
Robert Bamler
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian neural networks
Laplace approximation
uncertainty quantification
Gaussian processes
Primary area: probabilistic_methods
TL;DR: We propose a method to endow the Laplace approximation in Bayesian neural networks with informative Gaussian process priors.

Abstract: Laplace approximations are popular techniques for endowing deep networks with epistemic uncertainty estimates as they can be applied without altering the predictions of the trained network, and they scale to large models and datasets. While the choice of prior strongly affects the resulting posterior distribution, computational tractability and lack of interpretability of the weight space typically limit the Laplace approximation to isotropic Gaussian priors, which are known to cause pathological behavior as depth increases. As a remedy, we directly place a prior on function space. More precisely, since Lebesgue densities do not exist on infinite-dimensional function spaces, we recast training as finding the so-called weak mode of the posterior measure under a Gaussian process (GP) prior restricted to the space of functions representable by the neural network. Through the GP prior, one can express structured and interpretable inductive biases, such as regularity or periodicity, directly in function space, while still exploiting the implicit inductive biases that allow deep networks to generalize. After model linearization, the training objective induces a negative log-posterior density to which we apply a Laplace approximation, leveraging highly scalable methods from matrix-free linear algebra. Our method provides improved results where prior knowledge is abundant (as is the case in many scientific inference tasks). At the same time, it stays competitive for black-box supervised learning problems, where neural networks typically excel.

FSP-LAPLACE: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning
Tristan Cinquin"
Marvin Pförtner'
Tübingen Al Center, University of Tübingen
Tübingen Al Center, University of Tübingen marvin, iewerrtterrunnttuubinneen
tristan. cinquin@uni-tuueiinge
Vincent Fortuin Helmholtz AI, TU Munich de
Philipp Hennig
Tübingen AI Center, University of Tübingen de robert. bamlerduni Bamlerruni-tuebingeen -tuebingen.ddcc. de
Robert Bamler
Tübingen Al Center, University of Tübingen
Abstract
Laplace approximations are popular techniques for endowing deep networks with epistemic uncertainty estimates they can be applied without altering the predic- tions of the trained network, and they scale to large models and datasets. While the choice of prior strongly affects the resulting posterior distribution, computa- tional tractability and lack of interpretability of the weight space typically limit the Laplace approximation to isotropic Gaussian priors, which are known cause pathological behavior as depth increases. As remedy, we directly place a prior on function space. More precisely, since Lebesgue densities do not exist on infinite- dimensional function spaces, we recast training as finding the so-called weak mode of the posterior measure under a Gaussian process (GP) prior restricted to the space of functions representable by the neural network. Through the GP prior, one can express structured and interpretable inductive biases, such as regularity or periodic- ity, directly in function space, while still exploiting the implicit inductive biases that allow deep networks to generalize. After model linearization, the training objective induces negative log- pog-posterior density to which we apply Laplace approximation, leveraging highly scalable methods from matrix linear algebra. Our method provides improved results where prior knowledge is abundant (as is the case many scientific inference tasks). At the same time, stays competitive for black-box supervised learning problems, where neural networks typically excel.
Introduction
Neural networks (NNs) have become the workhorse for many machine learning tasks, but they do not quantify the uncertainty arising from data scarcity-the epistemic uncertainty. NNs therefore cannot estimate their confidence in their predictions, which is needed for safety-critical applications, decision making, and scientific modeling [1-3]. As solution, Bayesian neural networks (BNNs) cast training as approximating the Bayesian posterior distribution over the weights (i.e., the distribution of the model parameters given the training data), thus naturally capturing epistemic uncertainty. Various methods exist for approximating the (intractable) posterior, either by set of samples (e-g., MCMC) or
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Matern-1/2 kernel
Matern-3/2 kernel
Matern-5/2 kernel
RBF kernel
Periodic kernel
-2
-2
(a) LSP-LAPLACE posterior predictive of a BNN under different choices of GP prior
FSP-Laplace (ours)
FVI
Laplace
MAP
GP
Sparse GP
-2
(b) ESP-LAP compared to related approximate inference methods and models
Figure I: FSP-LAPLACE allows for efficient approximate Bayesian neural network (BNN) inference under interpretable function space priors. Using method, it is possible to encode functional properties like smoothness, lengthscale, or periodicity through a Gaussian process (GP) prior. The gray data points in the plots are noisy observations of periodie function.
by a parametric distribution (e.g., variational inference or Laplace approximations), While sampling methods can be asymptotically exact for large numbers of samples, they are typically expensive for large models, whereas variational inference (VI) and the Laplace approximation are more scalable. The Laplace approximation is particularly appealing as it does not alter maximum-a-posteree predictions. Compared to VI, its differential nature allows the Laplace approximation to scale to large datasets and models, and it has been shown to provide well-calibrated uncertainty estimates [4, 5].
(MAP)
The need for function-space priors in BNNs. While the choice of prior strongly influences the uncertainty estimates obtained from the posterior [6, 7], there are hardly any methods for elicitation of informative priors in the literature on BNNs, with the notable exceptions of Sun et al. [8] and Tran et al. [9]. This is not just conceptual issue. For instance, the default isotropic Gaussian prior, commonly thought of as uninformative, actually carries incorrect beliefs about the posterior (uni-modality, independent weights, etc.) [6, 10] and has known pathologies in deep NNs [9]. As network weights are not interpretable, formulating a good prior on them virtually impossible. Current methods work around this issue by model selection, either through expensive cross-validation or type-II maximum likelihood estimation with a Laplace approximation under an isotropic Gaussian prior, which can only be computed exactly for small networks, and has known issues with normalization layers [11, 12]. As a solution, Sun et al. [8] proposed VI method to specify priors directly on the function implemented by the BNN, with promising results. Function- sescci priors incorporate interpretable knowledge about the variance, regularity, periodicity, or length scale, building on the extensive Gaussian Process (GP) literature [13]. But it turns out that the method by Sun et al. [8] is difficult to use in practice: the Kullback-Leiibee (KI.) divergence regularizing VI is infinite for most function-space priors of interest [14), and estimating finite variants of it is challenging [15].
This paper addresses the lack of a procedure to specify informative priors in the Laplace approximation by proposing a method to use interpretable GP priors. Motivated by MAP estimation theory, we first derive an objective function that regularizes the neural network in function space using GP prior, and whose minimizer corresponds to the MAP estimator of a GP on the space of functions represented by the network. We then apply the Laplace approximation to the log- posterior density induced by the objective, allowing us to effectively incorporate beliefs from a GP prior into deep network (see Figure 1). Efficient matrix-fres methods from numerical linear algebra allow scaling our method to large models and datasets. We show the effectiveness of our method by showing improved results in cases where prior knowledge is available, and competitive performance for black-box regression and classification tasks, where neural networks typically excel. We make the following contributions:
1. We propose a novel objective function for deep neural networks that allows incorporating interpretable prior knowledge through GP prior in function space.
2. We develop an efficient and scalable Laplace approximation that endows the neural network with epistemic uncertainty reflecting the beliefs specified by the GP prior in function space.
3. A range of experiments shows that our method improves performance on tasks with prior knowledge in the form of kernel, while showing competitive performance for black-box regression, classification, out-of-disttibution detection, and Bayesian optimization tasks.

---

2 Preliminaries: Laplace approximation in weight space
For given dataset DD = (X,Y) of inputs X = )= E X" and targets Y = (y ) E Y", consider model of the data that is parameterized by neural network f: : X x w + R with weights w E W C RR, a likelihood p(Y f(X, w)) = P f w)), and a prior p(w).
We seek the Bayesian posterior p( w D) p(Y f(X, w)) p(w). As it is intractable in BNNs, approximate inference methods have been developed. Among these, the Laplace approximation [16] to the posterior is given by Gaussian distribution q(w w) W (w; w A-"A"" whose parameters are found by MAP estimation w* arg win (w) of the weights, where
(wws(w): log p(w
D) log p(w)
Σ logp(y f(x(() w)) const.
(2.1)
and computing the Hessian of the negative log- peg-posterior A H log
|D) w= E RRP.,
The linearized Laplace approximation. Computing A involves the Hessian of the neural network w.r.t. its weights which is generally expensive. To make the Laplace approximation scalable, is common to linearize the network around w* before computing A [17].
f(z, w) f(x, w") + Jw- (x)(w w²) flin (xx w)
(2.2)
with Jacobian Jw (ar) = D f1r w) w= Thus, the approximate posterior precision matrix A is
A ~ Ho logp(w) w=w Σ Jw- (x)()) H (T Jw (x)
(2.3)
where (+)) Hy ( If) f- f(rtt) w Thus, under the linearized network, the Hessian of the negative log- likelihood (NLL) coincides with the generalized Gauss-Newton -Newton matrix (GGN) the NLL. Crucially, the GGN is (semidd-(eem even if Hw log P(Y f(X, w)) w= is not.
3 FSP-LAPLACE: Laplace approximation under function-space priors
conventional Laplace approximation in neural networks requires a prior in weight space, with the issues discussed in Section 1. We now present FSP-LAPLACE, method for computing Laplace approximations under interpretable GP priors in function space. Section 3.1 introduces an objective function that is log-density under local linearization. Section 3.2 proposes scalable algorithm for the linearized Laplace approximation with function-space prior using matrix-free linear algebra.
3.1 Laplace approximations in function space
We motivate our Laplace approximation in function space through the lens of MAP estimation in an (ifinite-dimensionaa) function space under GP prior. As Lebesgue densities do not exist in inffiite-dimensionall function spaces, cannot minimize Equation (2.1) find the MAP estimate. We address this issue using a generalized notion of MAP estimation, resulting in minimizer of a regularized objective on the reproducing kernel Hilbert space Hls, We then constrain the objective to the set of functions representable by the neural network and minimize it using tools from deep learning. Finally, local linearization of the neural network turns this objective into valid negative log-density, from which we obtain the posterior covariance by computing its Hessian.
MAP estimation in neural networks under Gaussian process priors. The first step of the Laplace approximation is to find estimate of the neural network weights w. i.e., the minimizer of the negative log-density function in Equation (2.1) w.r.t. the Lebesgue measure as a "neutral" reference measure. In our method, we regularize the neural network in function space using d' -outuuu GP prior values in an infiniteddimensional (Banach) space BB of functions, where no such reference Lebesgue measure exists [18]. Rather, the GP induces a prior measure Pg on E [19, Section B.2). We are thus interested in the "mode" of the posterior measure PY under P3 defined by the Radon-Nikoday
GP (p, Σ) with index set X. However, a (nonparametric) GP takes its
3

---

derivative (d.) & exp (df) Pa(df) where Φy is the potential, in essence the negative log-likelihood functional of the model. In our case, Φ (f)= logp(y()) f())))). Similar to Bayes' rule in finite dimensions, this Radon-Nikodym derivative relates the prior measure to the posterior measure by reweighting. Since there is no Lebesgue measure, the (standard) mode is undefined, and we therefore follow Lambley [20], using so-called weak modes.
Definition 1 (Weak Mode [see e.g., 20, Definition 2.1]). Let IB be separable Banach space and let P be a probability measure on (LB, B (1B)). A weak mode of P is any point f supp P such that
lim sup ((B,(f)) S1 for all f fEb B. (f*))
(3.1)
Above, B (B) denotes the Borel o-algebra on B and (f) C. B is an open ball with radius r centered at point f E B. The intuition for the weak mode is the same as for the finite-dimensional mode (indeed they coincide when Lebesgue measure exists), but the weak mode generalizes this notion to infinite-dimensional (separable) Banach spaces. Under certain technical assumptions on the potential d (see Assumption A.2 in the appendix), Lambley [20] shows that any solution to
arg min (
2 II -
(3.2)
felly
2
(f)
is weak mode of the posterior probability measure pl-
Equation (3.2) casts the weak mode of the posterior measure Pÿ as the solution of an optimization problem in the RKHS Hyg We can now relate it to an optimization problem in weight space W, and apply tools from deep learning. Informally speaking, we assume that the intersection of the set of
functions represented by the neural network F = {f(., w): w E W) C B C (I&')² and T is non-empty, and constrain the optimization problem in Equation (3.2) such that f belongs to both sets
arg min (fs
(3.3)
fEHxnF
Unfortunately, the framework by Lambley [20] cannot give probabilistic meaning to optimization problems with hard constraints of the form f E F. To address this, we adopt and elaborate on the fll the distance of a function f E to the set F C B. Under Assumption A.3, ds (S, F) = 0 and only I E IF by Lemma A.2. Hence, we can relax the constraint f E F by adding 2x di (f, F) with X >> 0 to the objective in Equation (3.2). Intuitively, the resulting optimization problem is the MAP problem
informal strategy from Chen et al. [21, Remark 2.4). Denote by (f. f. F) == inf serll
for the measure P ' obtained by conditioning PY on the observation that df F) + = O, where Ex is independent centered Gaussian measurement noise with variance X Formally:
Proposition 1. Let Assumptions A.I to A.3 hold. For X > 0, define x ,y,>; B - f (f)
(df) x exp (- Pa(df) has at least one weak mode f* E Hyy, and the weak modes of P coincide with the minimizers of
2x (f, F). Then the posterior measure P DY
RØsp: Hy - RR क Y (f) (
} llf -
(3.4)
As x + O, the term 2t d (f forces the minimizers of Resp to converge to functions in Hg F that minimize Rysp:
Proposition 2. Let Assumptions A.I to A.3 hold. Let (da (AnhNeN C Eso with Xn + 0, and {fi hEE C Hys such that f*, is minimizer of Res. Then ifiineN has an HHy convergent subsequence with limit f E Hy F. Moreover, f* is a minimizer of Rysp on Hy E..
We prove Propositions 1 and 2 in Appendix A.2. To provide some intuition about the mode of convergence in Proposition 2. we point out that weak convergence of the subsequence (ftt hEen implies hoth (strong/norm) convergence in the path space of the Gaussian process (i.e.,
lim f+t - ff || = 0) and pointwise convergence (i.e., lim4 ft. (x) = (x) Vo E X).
Finally, under certain technical assumptions, Theorem 4.4 from Cockayne et al. [22] can be used to show that, as x + 0, P converges² to P defined by the Radon-Nikodym derivative Py (df) x exp (- (df)) (f( where Pr is the regular conditional probability measure P f E E).
w..tt. integral probability metric (IPM)
4

---

Summarizing informally, we address the nonexistence of a density-based based MAP estimator by con- structing a family {fā of weak modes of the related "relaxed" posteriors P (d.f) P (df dp (f, +A = 0) (Proposition 1). These weak modes converge to minimizers f of the optimization problem in Equation (3.3) as X + 0 (Proposition 2). Moreover, under certain technical assumptions, the relaxed posteriors P converge to the "true" posterior P X 0. Given the above, conjecture that the f are weak modes of P., and leave the proof for future work. This motivates using the objective in Equation (3.3) to find the weak mode of the posterior measure P that wish to suaplaceapproxinng
The next paragraph shows how this objective becomes a valid log-density.
The FSP-LAPLACE objective
unnormalized log-density. As first step, we use Algo- rithm 1, discussed in detail below, to train the neural network using the objective function
1 [l.(( ,.) -u -lligg (3.5) Risp(w) () == ResP (f(-, w)) = Σ logp(y()) f(x!() w)) 2
We then intuitively want to use the same objective function to compute an approximate posterior over the weights using Laplace approximation. For this to be well-defined, REsp( (w) needs to be valid unnormalized negative log-density, i.e., ur -> exp(-Rrst (ur) needs to be integrable. However, without weight-space regularization, this often fails to be the case (e.g.. due to continuous symmetries in weight space). Our method works around this issue by linearizing the network locally around w" after training (see Equation (2.2)) and then applying a Laplace approximation to Relig (w) = RFsp(fling (" w)). this case, the RKHS regularizer in Rip can be rewritten as
1 1 ( (w)))) ww- ulli = 2(( (ww) Eiu- (w - (ww)) + const., 2
(3.6)
where (E) )ij ((Jwr) w )is (Fwi Di) (here, Eng is the ((Jw )c, ", - Ug pww = w - Ew U. Crucially, Ew- is positiv-(semiddeeiiic This means that space. Note that this approximation also induces a (potentially degenerate) Gaussian "prior" w ~ N (mw- Ew over the weights.
pseudoinverse), v;
SP (-)) normalizable over im (Ew-). i.c., we don't integrate over the null
Using model linearization, we can also establish a correspondence between the Gaussian process prior in function space and the induced prior over the weights. Namely, the resulting expression is the Lebesgue density of the Horoorhooonal projection of the GP prior onto the finite-dimensional subspace spanned by the "feature functions" (Ju-) ); learned by the neural network. Hence, our model inherits the prior structure in function space on the features induced by the Jacobian, zeroing out the probability mass in the remaining directions.
3.2 Algorithmic Considerations
Training with the LAP-LAP objective function. Evaluating the FSP-LAPLACE objective proposed in the previous section requires computing the RKHS norm of the neural network. Unfortu- nately, this does not admit closed-form form expression in general. Hence, we use the approximation
())(,
pllig R (J(C, (C()) T C) (J w) - m())),
(3.7)
= =llbo (amhc( ,
where C' e X is set of no context points and ha w) == E(x, C)E(CCC)- w) (((C)) E H The function he is the minimum-norm interpolant of f(: w) at C Hy. Hence, the estimator of the RKHS norm provably underestimates, i.e., hc w) S flf( w)
During training, need to compute | w) every optimizer step. Since this involves solving a linear system in ne unknowns and, more importantly, evaluating the neural network on the nic context points, we need keep nc small for computational efficiency. We find that sampling an i..d. set of context points at every training iteration from a distribution Pc is an effective strategy for keeping ne small while ensuring that the neural network is appropriately regularized. The resulting training procedure is outlined in Algorithm 1.
In order to simplify notation, will assume F C Hy for the remainder of the main paper.
5

---

Algorithm I RKHS-reguulriie model training
1: function FSP-LAPLACE-TRAINGG
w, GP (1,, E), Pc. D, b)
2:
for all minibatch = (XB, B) ~ of size do
3:
[ (1)p(w(()) t - En= log logg(us
(1) [[ : w)
Negative log-likelihood
4:
C+ ( 1e Pc
Sample ne context points
5:
R (f((C,w) w) pc( ((C
μ(C)) b> Eqqation (3.7)
6:
w +- OPTIMIZERSTEP RESSP R w)
7:
return w
Algorithm 2 Linearized Laplace approximation with Gaussian process priors
1: function -AP-LLPL GP (H, 2), C.D. w*)
2: f Ju. (C)1/1/3 (C)1122
Initial Lanczos vector (using forward-mode autodiff) & min(ned .p) and LL Σ(C, C)t using backward-mode autodiff D MM = UM DMM where UM E TRpAY H J (zo10) using forward-mode autodiff D UA DA E R.xy
3:
L LANCZOS [C,, C), Ruoddxr xr
4:
M + (C) LOILLEE Rexr
5:
(UM, DM,...) t SV)(M)
6:
At D Ei= U J (1 (xx10 (T
7:
(UA, ( (A)
8:
St UMMAADA Rexr
DSS A
9: Find smallest is such that diag (Jw- (C,).. (S. Jw (C,) < diag E(C, C,) Vi
return N (w", (S.
Efficient linearized Laplace approximations of the FSP-LAPLACE objective. Once minimum un of Resp is found, we compute a linearized Laplace approximation at w". The Hessian of Resp is then
A Σ w.
Σ Jw (x) (+) Jwo (x!1))
(3.8)
Again, the RKHS inner products in the entries of D do not admit general closed-form expressions. Hence, we use the same strategy as before estimate Σ Ju ECC C) Ju. (C) at another set C' of context points. Unlike above, for a Laplace approximation, it is vital to use large number of context points to capture the prior beliefs well. Luckily, Σ only needs to be computed once, at the end training. But for large networks and large numbers of context points, is still infeasible to compute or even represent Σ in memory. To address this problem, we devise an efficient routine for computing (a square root of) the approximate posterior covariance matrix A outlined in Algorithm 2. Our method is matrix-free, i.c. it never explicitly constructs any big (i.c., p X P.P. P X TiC or nc X Tic) matrices memory. This allows the method to scale to large models.
rnseudo-)inverted We start by computing rank < min(ncd',, approximation of the (pseudo- iinverted kernel Gram matrix [C,, C) z LL" using fully-reorthogonalized Lanczos iteration [23, Section 10.1] with an embedded LDL ffctorization [23, Section 11.3.5). This only needs access to metrix-vector products with ECC, C), which can be implemented efficiently without materializing the matrix in memory. Moreover, kernel Gramians typically exhibit rapid spectral decay [see e.g., 24], which makes the low-rank approximation particularly accurate. The low-rank factors L then yield rank
approximation of Σ - MM with M Ju (C) L. which is embarassingly parallel and can be computed using backward- mode autodiff to avoid materializing the network Jacobians in memory. An eigendecomposition of MM T UM D , M can be computed in Oppr²) time from thin SVD of M D V The eigenvectors UM define an orthogonal projector = U onto the range and orthogonal projector Po := I UMU onto the nullspace of MM This means that we have the decomposition A PAPT T + P.APA UMAUT P.A.P. with
UM
n
U AUM D - Σ
UNg T
J (
(3.9)
i=1

---

We find that Po is negligible in practice, and hence, we approximate A R UM M (see Table C.2).
Finally, by computing an eigendecomposition of A = UADAUA E E in ( time, we obtain an eigendecomposition of A x UMAUM = (UMUA)DA(UMUU) obtain matrix- free representation of square root of) the (pseudo-)invedd of A. Unfortunately, due to numerical imprecision, it is difficult to distinguish between zero eigenvalues of A and those with small positive magnitudes. Since we need to pseudo-invert the matrix to obtain the covariance matrix, this can explode predictive variance. As remedy, we use heuristic based on the observation that, in a Gaussin model, the marginal variance of the posterior is always upper-bounded by the marginal variance of the prior. Hence, we impose that diag (Jw. (C.) J.w (C,) < diag [,,, C,) for all i I,.... no by successively truncating the smallest cigenvalues in D until the condition is fulfilled. This turns out to be an effective strategy to combat exploding predictive variance in practice.
which can be used to
Choice of context points. Methods for regularizing neural networks in function space rely on set of context points to evaluate the regularizer [8, 9, 25, 26]. The context points should cover the set of input locations where it is plausible that the model might be evaluated when deployed. Popular strategies include uniform sampling from a bounded subset of the input space [8, 9, 25, 26], from the training data [8], and from additional (possibly unlabeled) datasets [25, 26]. For MAP estimation, we choose a uniform context point distribution Pc or sample from other datasets for high-dimensional inputs. We compute the posterior covariance using samples C' from oow-diccepyancy sequence, which effectively cover high-dimensional spaces.
4 Experiments
We evaluate
synthetic and real-world data, demonstrating that our method effec- tively incorporates beliefs specified through a GP prior; that it improves performance on regression tasks for which we have prior knowledge in the form of kernel (Mauna Loa and ocean current modeling); and that it shows competitive performance on regression, classification, -uitof-distribution detection, and Bayesian optimization tasks compared to baselines.
Baselines. We compare ESP-LAPLACE to deterministic neural network fit using maximum
a posteriooi estimation with an isotropic Gaussian prior on the weights (MAP) and neural network for which we additionally compute the standard linearized Laplace approximation (Laplace) [17]. We further compare our method to FVI [8], which uses GP priors with VI, to Gaussian process (GP) [13] when the size of the dataset allows it, and to a sparse Gaussian process (sparse GP) [27]. We use the full Laplace approximation when the size of the neural network allows it, and otherwise consider
the K-FAC or diagonal approximations [28]. All neural networks share the same architecture.
Qualitative evaluation on synthetic data. We consider two synthetic data tasks: 1-dimensional regression task with randomly drawn noisy measurements of the function y sin( (2n(2.z), and the 2-dimensional two-moons classification task from the scikit-learn library [29]. For the regression task, data points are shown gray circles, functions drawn from the posterior as green lines and the inferred mean function as a red line (see Figures and C. to C.4). For the classification task, we plot the predictive mean and 2-standard-deviiations of the predictions for class (blue circles) in Figures C.5 and C.6. We apply FSP-LAPLACE to the data with different GP priors and find that it successfully adapts the specified beliefs. For instance, by varying the GP prior, can make our method generate functions that are periodic within the support of the context points (Figure C.2), and control their smoothness (Figure 1) and length scale (Figure C.3) without modifying the network's architecture, or adding features. Such flexibility is impossible with the isotropic Gaussian weight- space priors used in the Laplace and MAP baselines. These results carry over classification, where our method shows smooth decision boundary when equipped with an RBF kernel (Figure C.6). and a rough decision boundary when equipped with a Matern- 1/2 kernel (Figure C.5). Unlike the Laplace and MAP baselines, whose predictions remain confident beyond the support of the data, the FSP-LAPLACE'S posterior mean reverts to the zero-mean prior, and its posterior variance increases. Our method also captures the properties specified by the GP prior better than FVI, especially for rougher Matern-1/2 1/2 kernels (Figures and C.5). While a sufficient number of context points necessary to capture the beliefs specified by the GP prior, we find that, even with very small number
7

---

Ground Truth
GP
LSP-aae
FVI
Laplace
1.2
27
Lattud
0.9
26
0.6
0.3
0.0
-40
-88
-06
-54
Longitude
1.1
Lahause
27
0.8
Ocean current
26
Drifting buoy
25
0.3
Lastude
0.9
0.3
84
900
-88
86
Longitude
Longitude
Longitude
Longibude
Results for the ocean current modeling experiment. We report the mean velocity vectors, the norm of their coandard-deviaiion and the squared errors of compared methods. Unlike the Laplace, we find that FSP-LAPI.CC accurately captures ocean current dynamics.
Figure 2:
Table Results for the Mauna Loa CO2 prediction and ocean current modeling tasks. Incorporating knowledge via the GP prior in our FSP-LAPLACE improves performance over the standard Laplace.
LaFLACS
PSP-LARLACE (408))
LaR-cc
WOLL 54 1EL.3
TE ONE 1333.35
LETTT SS 30133.00
22 37133 90 8,80
8,9 8:15
15 38 3
36.20 6.83
15.00 D43
of context points, our method produces useful uncertainty estimates and reverts to the prior mean (Figures C.7 and C.8).
4.1 Quantitative evaluation real-world data
We now move on to investigate FSP-LAPLACE on two real-world scientific modeling tasks: fore- casting the concentration of CO2 at the Mauna Loa observatory and predicting ocean currents in the Gulf of Mexico. We then assess the performance of our method on standard benchmark regression, classification, nut-of-distributio detection, and Bayesian optimization tasks. When reporting results, we bold the score with the highest mean as well as any scores whose standard-error bars overlap.
Mauna Loa dataset. We consider the task of modeling the monthly average atmospheric CO2 concentration the Mauna Loa observatory in Hawaii from 1974 2024 using data collected from the NORA global monitoring laboratory' [30]. This 1-dimensional dataset is very accurately modeled by combination of multiple kernels proposed in Rasmussen and Williams [13, Section 5.4.3]. We equip and compare to FVI with the same prior and additional periodic features, to a GP with the same prior, and to the linearized Laplace with the same additional periodic features. Using periodic features (partially) reflects the prior knowledge carried by the kernel. Results are presented by Table 1 and Figure C.9 in the Appendix. We find that incorporating prior beliefs both via informative prior and periodic features in FSP-LAPLACE significantly reduces the mean squared error (MSE) compared to Laplace and GP baselines, and that method is also more accurate than FVI, which variational inference. In terms of expected liglliie,,oodd, all neural networks under-estimate the likelihood scale, which results in poorer scores than the GP.
with this informative kernel and with additional periodic features,
Ocean current modeling. We further evaluate ESP-LAPLACE's ability to take into account prior knowledge by considering observations of velocities ocean drifter buoys, and we are interested in estimating ocean currents further away from the buoys. For this, we consider the GulfDrifters dataset [31] and we follow the setup by Shalashilin [32]. We incorporate known physical properties of ocean currents into the considered models by applying the Helmholtz decomposition the GP prior's kernel [33] as well as to the neural network. Results are shown in Table and in Figure 2. We find that FSP-LAPLAR
current modeling task where are given sparse 2-dimensional
8

---

Branin (d-2)
Ackley (d-4)
PDE (dm4)
Hartmann (d=6)
Polynomial (d=8)
(d=10)
-7.5
0.05
3,0
4,0
25
Max Roeard
35
20
12.5
0.15
2.0
3.0
15
-5.00
0.20
2.5
17.55
2.0
-20.0
0.5
150
200
Function evaluations
Function evaluations
Function avaluntions
Function evalustions
Fanction evaluktions
Function evaluations
PSPLLaplace
Laplace
Figure 3: Results using our method (FSP-LAPLACE) as surrogate model for Bayesian optimization. We find that FSP-LAPLACE performs particularly well lower-dimensional problems, where converges more quickly and to higher rewards than the Laplace, obtaining comparable scores as the Gaussian process (GP).
Table 2:
Test expected log-likeling log-likelihood, accuracy, expected calibration error and OOD detection accu- racy MNIST and FNSS. FSP-LAPLACE performs strongly among baselines matching the accuracy of best-performing baselines and obtaining the highest expected log-likelihood and -iittr-ddittitttiinn detection accuracy.
DATAMPT
MaTRIC
PSP-LAPLACH (PPLLPLAECCMNISTTT
PSP-LAR ACS (RNT)
(VU(NNIIST)
FVE
LAPLaCE
MAP
SHARSS GP
L.6--IKBLIBOOD
-0.043
-0.145 0.005
0.001
0.001
MNIS
D.03 -.000
0,054
0.001
0.013
0.003
0.035 0.001
0OD ACCURACT (1)
8,977
9.083
0.345 2: 0.001
0.91=00000
0.89 6 0.010
0.907 = 0.016
0,889 = 0,00
0.908-2 0.020
0.003
0.002
-0.479 0.00
(CB())
0.018
0.060
0.600 0.003
0.044
0.002
0.027 0.005
0.814
8.062
0,016
0,082
0.007
0.00)
OOD ACCURACY (T)
8,994 9.081
0.797 0.000
0:975
0.002
0.925 0.005
0.501
0.014
0.794 0.000
0.938 0.007
strongly improves Laplace, FVI and GP in terms of expected log-likelihood, and performs competitively in terms of mean squared error (MSE). FSP-LAPLACE also improves MSE and test expected log-likelihood over FVI, which strongly underestimates the predictive variance.
Image classification. We further evaluate our method on the MNIST [34] and FashionMNIST [35] image classification datasets using a convolutional neural network. We compare our model FVI, Laplace, MAP, and Sparse GP baselines, as the scale of the datasets forbids exact GP inference, For FSP-LAPLACE and FVI, we compare using context points drawn from uniform distribution (RND)
and drawn from the KuzushijiMNNITT dataset (K MNIIS) following the setup by Rudner et al. [25]. Results are presented Table Although these datasets are particularly challenging to our method, which is regularized in function space, we find that FSP-LAPLACE performs strongly and matches or exceeds the expected likelikll and predictive accuracy of best-performing baselines. also yields well-calibrated models with low expected calibration error (ECE).
Out-of-distribution detection. We now investigate whether the epistemic uncertainty FSP- LAPLACE is predictive for out-of-distribution detection (OOD). We follow the setup by Osawa et al. [36] and report the accuracy of single threshold to classify OOD from in-distribution (ID) data based the predictive uncertainty. Additional details are provided in Appendix B.2. FSP-LAPL.AS with context points sampled from KMNIST performs strongly, obtaining the highest out-of-distribuue detection accuracy (see Table 2, note that FSP-LAPLACE makes assumption whether context points are in or out of distribution). This can be further observed in Figure C. in the Appendix, where the predictive entropy of ID data points tightly peaked around whereas the predictive entropy of OOD data points is highly concentrated around the maximum entropy of the softmax distribution (In(10) 2.3). With RND context points, our method performs comparably to the Laplace baseline.
Bayesian optimization. We finally evaluate the epistemic uncertainty of FSP-LAPLACE as surrogate model for Bayesian optimization. We consider setup derived from Li et al. [7], comparing to FVI, the linearized Laplace, and GP. Results are summarized in Figure 3. We find that our method performs particularly well on lower-dimensional tasks, where it converges both faster and to higher rewards than Laplace, and noticeably strongly improves over Gaussian process on PDE. nigher-dimensienn tasks, method performs comparatively well to Laplace and GP basclines.

---

5
Related work
Laplace approximation in neural networks. First introduced by MacKay [37], the Laplace approximation gained strong traction in the Bayesian deep learning community with the introduction of scalable log-posterior Hessian approximations [28, 38], and the so-called linearized Laplace, which solves the underfitting issue observed with standard Laplace [17, 39-41]. In addition to epistemic uncertainty estimates, the Laplace approximation also provides method to select prior parameters via marginal likelihood estimation [11, 12]. Recent work has made the linearized Laplace more scalable by restricting inference to subset of parameters [42], by exploiting its GP formulation [17] to apply methods from the scalable GP literature [43-45], or by directly sampling from the Laplace approximation without explicitly computing the covariance matrix [46]. While these approaches use the GP formulation to make the linearized Laplace more scalable, we are unaware of any method that uses GP priors to incorporate interpretable prior beliefs within the Laplace approximation.
BNNs with function-space priors. In the context of variational inference, function-space priors in BNNs demonstrate improvements in predictive performance compared to weight-space priors [8]. While this idea might seem sound, it turns out that the KL divergence in the VI objective is infinite for most cases of interest due mismatching supports between the function-space prior and BNN's predictive posterior [14], which therefore requires additional regularization to be well defined [8]. Due to this issue, other work [47] considers generalized VI [10] using the regularized KL divergence [48] or abandons approximating the neural network's 's posterior and instead uses deterministic neural networks as basis functions for Bayesian linear regression [15] or for the mean of a sparse GP [49]. In contrast, our method does not compute a divergence in function space, but only the RKHS norm under the prior's kernel, thus circumventing the issue of mismatching support. Alternatively, rather than directly placing a prior on the function generated by BNN, researchers have investigated methods find weight-space priors whose pushforward approximates a target function-space measure by minimizing divergence [9, 50], using the Ridgelet transform [51], or changing the BNN's s architecture [52].
Regularizing neural networks in function space. Arguing that one ultimately only cares about the output function of the neural network, has been proposed to regularize neural networks in function space, both showing that norms could be efficiently estimated and that such regularization schemes performed well in practice [26, 53-55]. Unlike FSP-LAPLACE, none of these methods allow to specifyy informative beliefs via GP prior. Our method uses the same RKHS norm estimator as Chen et al. [54] (however, under different kernel) by sampling a new batch of context points at each update step. Similarly, Rudner et al. [26] propose empirical prior on the weights that regularizes the neural network in function space, which they use for MAP estimation or approximate posterior inference. The MAP objective resembles ours, but unlike our method, uses the kernel induced by the last layer of the neural network and includes an additional Gaussian prior on the weights.
Conclusion
We propose a method for applying the Laplace approximation to neural networks with interpretable Gaussian process priors in function space. This addresses the issue that conventional applications of approximate Bayesian inference methods to neural networks require posing prior in weight space, which is virtually impossible because weight space is not interpretable. We address the non-existence of densities in dimiiiioedmmsiinaa) function space by generalizing the notion of a MAP estimate to the limit of a sequence of weak modes of related posterior measures, leading us to propose simple objective function. We further mitigate the computational cost of calculating high-dimensional curvature matrices using scalable methods from matrix-free linear algebra. By design, our method works best in application domains where prior information can be encoded in the language of Gaussian processes. This is confirmed by experiments on scientific data and Bayesian optimization. In high-dimensional spaces, where explicit prior knowledge is difficult to state, Gaussian process priors are naturally a disadvantage. While we do demonstrate superior performance on image data, it is yet unclear how to find good function-spant space priors in such nigh-dimensional spaces.
10

---

Acknowledgments and Disclosure of Funding
TC, MP. PH, and RB are funded by the DFG Cluster of Excellence "Machine Learning New Perspectives for Science", EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FKZ: 01IS 8039A); A); and funds from the Ministry of Science, Research and Arts of the State of Baden-Württenmen WWrtteemerg.. MP and PH gratefully acknowledge financial support by the European Research Council through ERC StG Action 757275 PANAMA, and ERC CoG Action 101123955 ANUBIS. VF was supported by Branco Weiss Fellowship. RB further acknowledges funding by the German Research Foundation (DFG) for project 448588364 of the Emmy Noether Programme. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting TC and MP.
Bibliography
[1] Nina Effenberger, Marvin Pförtner, Philipp Hennig, and Nicole Ludwig. Probabilistic wind speed downscaling for future wind power assessment. URL hctpss///netttnggrraanzzzr copernicus cus.. PPrg/EGU4/ECGUE /EE//4-222224-- html.
[2] Abhaya Indrayan. Aleatory and epistemic uncertainties can completely derail medical research results. Journal of postgraduate medicine, 66, 03 2020. doi: 100:10.4033/pgmJJPGG 585_11 19,
[3] Kenza Tazi, Jihao Andreas Lin, Alex Gardner, ST John, Hong Ge, and Richard Turner. Towards more interpretable and robust geospatial modelling with gaussian processes. AGU23, 2023.
[4] Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization, 2021.
[5] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux effortless bayesian deep learning, 2022.
[6] Vincent Fortuin. Priors in bayesian deep learning: A review, 2022.
[7] Yucen Lily Li, Tim G. J. Rudner, and Andrew Gordon Wilson. A study of bayesian neural network surrogates for bayesian optimization, 2024.
[8] Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks, 2019.
[9] Ba-Hien Tran, Simone Rossi, Dimitrios Milios, and Maurizio Filippone. All you need is good functional prior for bayesian deep learning. 2022.
[10] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. Generalized variational inference:
Three arguments for deriving new posteriors, 2019.
[11] Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scalable marginal likelihood estimation for model selection in deep learning, 2021.
[12] Javier Antorán, David Janz, James Urquhart Allingham, Erik Daxberger, Riccardo Barbano, Eric Nalisnick, and José Miguel Hernández-Lobato. Adapting the lincarised laplace model evidence for modern deep learning. 2022.
[13] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X.
[14] David R. Burt, Sebastian W. Ober, Adrià Garriga-Alonso and Mark van der Wilk. Understanding variational inference in function-space, 2020.
[15] Chao Ma and José Miguel Hernández-Lobale Functional variational inference based on stochas- tic process generators. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21795-21807. Curran Associates, Inc., 2021. URL htpp:/// neurips.... cc/ paper_ ioper_files//
Paper pdff.
[16] David J. C. MacKay. The Evidence Framework Applied to Classification Networks. Neural Computation, 4(5):720-736, 09 1992. ISSN 0899-7667. doi: 10. www.1116e.e./99/00 1992.4.5.72. URL
https://doi. wtt.:o................ www.//ee1ooeoo// 19w.eee............

---

[17]
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization, 2021.
[18] John Oxtoby. Invariant measures in groups which are not locally compact. Transactions of the American Mathematical Sociery, 60:215-237, 1946. doi: 10. (00000000000029997799966666 818-5. 188-.
[19] Marvin Pförtner, Ingo Steinwart, Philipp Hennig, and Jonathan Wenger. Physics-informed
Gaussian process regression generalizes linear PDE solvers, 2022.
[20]
Hefin Lambley. Strong maximum a posteriori estimation in Banach spaces with Gaussian priors. Inverse Problems, (9())): (9(11):120011 125010, 2023. 10t:10.11881100 1666--4////07244
[21] Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M. Stuart. Solving and learning nonlinear PDEs with Gaussian processes. Journal of Computational Physics, 447, 2021. doi: 20.::000016/..ep2221.111001
[22] Jon Cockayne, Chris J. Oates, T. J. Sullivan, and Mark Girolami. Bayesian probabilistic numerical methods. SIAM Review, 61(4):756-789, 2019.
[23] Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. The Johns Hopkins University Press, Baltimore, fourth edition, 2013.
[24] Francis Bach. Unraveling spectral properties of kernel matrices 2024. URL https:
//francisbach. cww http//ppetuummmmeeee kernel- paarnee-matrices--11
[25] Tim G. J. Rudner, Zonghao Chen, Yee Whye Teh, and Yarin Gal. Tractable function-space variational inference in bayesian neural networks, 2023.
[26] Tim G. J. Rudner, Sanyam Kapoor, Shikai Qiu, and Andrew Gordon Wilson. Function-space regularization in neural networks: A probabilistic perspective, 2023.
[27] James Hensman, Nicolo Fusi, and Neil D. Lawrence. Gaussian processes for big data, 2013.
(28) Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations, 2018. URL attps://oppereeveet
[29] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[30] Charles D Keeling and T Whorf. Monthly carbon dioxide measurements on Mauna Loa, Hawai from 1958 to 1998. 2000. URL htttps://oi..... org/oo. 10. 1594/PANGAEA, 56536.
[31] Jonathan M. Lilly and Paula Pérez-Brunius. GulfDrifters: consolidated surface drifter dataset for the Gulf of Mexico, January 2021. URL https: org/10 w0....//eeeooo........ 4421585.
[32] Ivan Shalashilin. Gaussian processes for vector fields and ocean current modelling, March 2024. httpstttp:///oes wax.ussiiappoocessscooooo
com/ www.applees/ceemmdddllinng..
[33] Renato Berlinghieri, Brian L. Trippe, David R. Burt, Ryan Giordano, Kaushik Srinivasan, Tamay Özgökmen, Junfei Xia, and Tamara Broderick. Gaussian processes at the helm(holtz): A more fluid model for ocean currents, 2023.
[34] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: //yann.lecaa lecun. com/ exdb/nnist/.
[35] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
[36] Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with bayesian principles, 2019.
[37] David John Cameron MacKay. Bayesian methods for adaptive models. 1992. URL https:
/ //tpp...eaanticceee....eea/aasas www.nticscholaarrrr org/Corpus :TTDD:2231444 123141880.
[38] James Martens. New insights and perspectives on the natural gradient method, 2020.
[39] Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate inference turns deep networks into gaussian processes, 2020. URL. https: //aariv////iv//a//a org/
abs/1906. 1tt.//9966..01....
12

---

[40] Andrew Y. K. Foong, Yingzhen Li, José Miguel Hernández-Lobtto, and Richard E. Turner. 'in-between' uncertainty in bayesian neural networks, 2019.
[41] Mohammad Emtiyaz Khan and Siddharth Swaroop. Knowledge adapeedddeeadattttionn priors, 2021. URL https://arxiv. org/ abs/2106. 08769.
[42] Erik Daxberger, Eric Nalismick, James Urquhart Allingham, Javier Antorán, and José Miguel Hernández-Lobato. Bayesian deep learning via subnetwork inference, 2022.
[43] Zhijie Deng, Feng Zhou, and Jun Zhu. Accelcrated linearized laplace approximation for bayesian deep learning, 2022.
[44] Luis A. Ortega, Simón Rodriguez Santana, and Daniel Hernández-Lobato. Variational linearized laplace approximation for bayesian deep learning, 2024.
[45] Aidan Scannell, Riccardo Mereu, Paul Edmund Chang, Ella Tamir, Joni Pajarinen, and Arno Solin. Function-space parameterization of neural networks for sequential learning.
In The Twelfth International Conference on Learning Representations, 2024. URL https: / /oopnrevieww
[46] Javier Antorán, Shreyas Padhy, Riccardo Barbano, Eric Nalisnick, David Janz, and José Miguel Hernández-Lobato. Sampling-based inference for large linear models, with application to linearised laplace, 2023.
[47] Tristan Cinquin and Robert Bamler. Regularized kl-divergence for well-defined function-space variational inference in bayesian neural networks, 2024. URL https://arzii: ortpa////rr////////////a/aa 2406. 04317.
[48] Minh Ha Quang. Regularized divergences between covariance operators and gaussian measures on hilbert spaces, 2019.
[49] Veit D. Wild, Robert Hu, and Dino Sejdinovic. Generalized variational inference in function spaces: Gaussian measures meet bayesian deep learning, 2022.
[50] Daniel Flam-Shepherd. Mapping gaussian process priors to bayesian neural networks. 2017.
URL https://appp www.mticsccooleeee...e..c.ccccee org/Corpusiaaieooooooo : 160026812.
[51] Takuo Matsubara, Chris J. Oates, and François-Xavier Briol. The ridgelet prior: A covariance function approach to prior specification for bayesian neural networks, 2022.
[52] Tim Pearce, Russell Tsuchida, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Expressive priors in bayesian neural networks: Kernel combinations and periodic functions, 2019.
[53] Ari S. Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space, 2019.
[54] Zonghao Chen, Xupeng Shi, Tim G. J. Rudner, Qixuan Feng, WEIZHONG ZHANG, and Tong Zhang. A neural tangent kernel perspective function-space regularization in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop). 2022. URL https://openreviis
[55] Shikai Qiu, Tim G. J. Rudner, Sanyam Kapoor, and Andrew Gordon Wilson. Should we learn most likely functions or parameters?, 2023.
[56] Nathaël Da Costa, Marvin Pförtner, Lancelot Da Costa, and Philipp Hennig. Sample path regularity of Gaussian processes from the covariance kernel, 2023.
[57] Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, and Bharath K. Sriperumbudur. Gaus- sian processes and kernel methods: A review on connections and equivalences, 2018.
[58] M. Dashti, K. J. H. Law, A. M. Stuart, and J. Voss. MAP estimators and their consistency in Bayesian nonparametric inverse problems. Inverse Problems, 29(9). September 2013. doi: 10. (088/0266-5611/299999
[59] John Conway. Course in Functional Analysis, volume 96 of Graduate Texts in Mahhmmtics. Springer, New York, NY, second edition, 1997. 10110/99----4777-43388888 1-4757-438338.
[60] Vladimir Igorevich Bogachev. Gaussian Measures, volume 62 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, Rhode Island, 1998.
[61] Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone. Dirichlet-basee gaussian processes for large-scale calibrated classification. In
13

---

S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. itors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL.htps htppp///////.oooeciiiiss.so...s neurips. wtt.cc/ppppppee.. _ fiw.s/lllpaeee plles/paperr221//ttiee (9077999e9967776ccabbbblel----p pdf.
and R. Garnett, ed-
[62] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Yamamoto Kazuaki, and David Ha. Deep learning for classical japanese literature, 12 2018.
[63] J.H. HALTON, On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals. Numerische Mathematik, 2:84-90, 1960. URL http: //edmm.
org/doc/131es 111444.
[64] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[65] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: / /wwww wandb. com/ Software available from wandb.com.
[66] Hao Chen, Lili Zheng, Raed Al Kontar, and Garvesh Raskutti. Gaussian process inference using mini-batch stochastic gradient descent: Convergence guarantees and empirical benefits, 2021.
[67] M. Mckay, Richard Beckman, and William Conover. A comparison of three methods for selecting vales of input variables the analysis output from a computer code. Technometrics, 21:239-245, 05 1979. doi: 000:10.1000/00 10800004001110000000000000000000 706. 1979. 179991544755
[68] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton. Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL
http:// http://arxiv org /t/aa/ss///11 19403. 06403.
[69] Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. URL WtRLhhtt://arccive ics. uci. edu/m1.
[70] Andrey Malinin, Liudmila Prokhorenkova, and Alcksei Ustimenko. Uncertainty gradient boosting via ensembles, 2021.
[71] Thierry Bertin-Mahieur Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011), 2011.
[72] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://gittuu..com com/ google/jax.
[73] Tom Hennigan, Trevor Cai, Tamara Norman, Lena Martens, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URL http: //gthbb../ com homm/deepmndd/// cemeemmindddm--ee dm-haiku.
[74] Aleksandar Botev and James Martens. KFAC-JAX, 2022. URL https://github. com/
geogle-deeminndkkfa--
[75] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty, 2018. URL https::////// https://arxiv ieeeiiiiorgg/aae http//bb/1110..111
14

---

Appendix
A Theory
A.1 Assumptions and their applicability
Assumption A.1. f GP (), is d' d'-output Gaussian process with index set X on (St, A. such that
ii the paths of f lie (P-almost surely) in real separable Banach space B of Rd valued functions on X with continuous point evaluation maps 0z BB Rd and
w f(; w) a Gaussian random variable with values (B, (B)).
We denote the law of w >> f(:,w) by P3-
For this paper, focus on B C(X), with X being compact metric space. In this case. Assumption Ai) can be verified from regularity properties of the prior covariance function Σ [see, e.g., 56]. Moreover, the sufficient criteria from Pförtner et al. [19, Section B.2] show that Assumption A. (ii) also holds in this case.
Assumption A.2. The potential Φ> B + R is coontn-contiuuuus and, for each rj > there is K (()) E I such that 4 (f) 2 K (n) - fllff li for all f E E.
This holds if the negative log-likelihood functions g(i); TR'd' + R. f() + -logggg p(y {() are continuous and, for all 0, there is (n) E IR such that ()(()) > (()())
( + m m/Ilff (1) 2 for all
f() Rdd, For instance, this is true for a Gaussian likelihood (f(()) 2x (z f
Assumption A.3. (i) Hy n nonempty, (ii) Hy n is closed 3 Hy, and (iii) Hy nF CB has the Heine-Bote property, i.e., all closed and bounded subsets of Hy n F are compact in B.
Assumption A.3(i) can be verified using a plethora of results from RKHS theory. For instance, for Sobolev kernels like the Matérn family used in the experiments, the RKHS is norm- norm-equivalent to a Sobolev space [see, e.g., 57J. In this case, we only need the NN to sufficiently often (weakly) differentiable on the interior of its compact domain X. The closure property from Assumption A.i( more difficult to verify directly. However, we assume that W compact and that the map f(, w) continuous, then IF is compact (and hence closed) as the image of compact
W + B, set under a continuous function. This is reasonable assumption, since, in practice, the weights of neural network are represented as machine numbers with maximal magnitude, meaning that W is always contained an &oo ball of fixed radius. Incidentally, compactness Heine-Borel property from Assumption Aii(i Alternatively, IF also has the Heine-Borel property it is a topological manifold (e.g., a Banach manifold), since it is necessarily mnite-dimensionll.
also entails the
A.2 Proofs
Lemma A.1. Let (X, d) be metric space, A C X nonempty, and d(,A):: X R20, inf d(x, a). aeh
Then d(,,A) is I-Lipschitz
Proof. For all æ1, 22 E X we have
d(z2, = inf d(z2, a)
S d((z 21) inf d(x1,a)
aEA
= = v2) + d( A)
by the triangle inequality and hence d(æ2, A.) - d(z), A) < d(z1, Since this argument is
symmetric in in and 22, this also shows that
-dd(z, A) d(x1, A)) = d( A) d(rz A) S x1) = d(I1)
All in all, we obtain d(æ2, A) d(x1, A)) I d(z1,02).
15

---

Lemma A.2. Let (X, d) be a metric space and A S X a closed, nonempty subset with the Heine-Borel property. Then inf d(x, a) attained for all X.
Proof. Let & E X and > Ya infaga d(x, a). Then & (x) + 0 as well as d(x, a) > r for all a E A (x) and thus infaea d(æ, a) aE, d(x; a). Moreover, An (x) is compact by the Heine-Borel property and d(z, -) is continuous, Hence, the claim follows from the Weierstrass extreme value theorem.
Proposition Let Assumptions A.I to A.3 hold. For X O, define x B RR Φ (f) 222
1 df(fF)) F). Then the posterior measure P. (dff x exp (- Pa(df) has at least one
weak mode f" Hiy, and the weak modes of P 2 coincide with the minimizers of
Røsp: Hy - R. $ (f)
2 I -
(3.4)
Proof. 1F(.,1F is (globally) I-Lipschitz by Lemma A.1 and bounded from below by 0. Hence, Φ. A σbx 2xz FF) is continuous and for all r > there is K K(n) E R such that
( 2 K()) fll
((,F)) 2 K(n) nlllf
2X
20
by Assumption A.2. The statement then follows from Theorem 1.1 in Lambley [20].
Proposition 2. Let Assumption A.l to A.3 hold. Let (dainet C R>0 with A, + and (f+ hee C Hi such that f is minimizer of Rysp. Then (fä)neN has an weukly convergent subsequence with limit f E Hyx R.F. Moreover, f is minimizer of Resp Hx n.
Our proof makes use ideas from Dashti et al. [58] and Lambley [20].
Proof. Without loss generality, we assume μ4 0. By Assumption A.2, there are constants K, a >> O such that RFsp(s) 2 K + I for all E Hy [20, Section 4.1]. Now fix an arbitrary
Hy F. Then
Resp( = (F)
(fsp(f())
(FsP(fr) + (fi,
(A.1)
2X2
K + a f
de F)
(A.2)
2x
K + allfillisss
and hence
1 < (RPse K)
i.c. the sequence {fä] C. Hy bounded. By the Banach-Alaoglu theorem [59, Theorems V.3.1 and V.4.2(d)) and the Eberlein-Smulian theorem [59, Theorem V.13.1], there is a weakly convergent subsequence {fr, TREN with limit f E Hyy.
We need show that f F. From Equation (A.2), it follows that
0 S dg (ftt F) & Ann (fF - K - a II 12 11 S Anz 2( K),
where the right-hand side converges to as + oc. Hence, lim4-1o0 ds ( F) = 0. The embedding e: TH + B is compact [60, Corollary 3.2.4] and, by Lemma A. dis (,, - R is continuous, which implies that da (1{ (, F): IH] -> IR is sequentially weakly continuous. Hence, ds F) = limg-zoo da(F df = and, by Assumption A.3 and Lemma A.2, it follows that f" E F.
16

---

Finally, we show that f is minimizer of RESP on Hy F. Since xY: B - R is con- tinuous and is compact, Φy o Hy + R sequentially weakly continuous. Moreover. we have lim - Ji ll 2 II since Hilbert norms are sequentially weakly lower- semicontinuous. Hence, lim su Risp 2 Risp(f"). Now, by Equation (A.1),
1
Resp (f) 2 (FsP(f") + lim sup k-000 2
sup ( F);
for all f E Hy R.B. For f f this implies that lim supk-coo 2 arrive at Rf) 2 Rf) for all f E Hy OF, f is minimizer of Resp-
1 B). 0. All in all,
B Experimental setup
B.1 Qualitative experiments with synthetic data
Regression. We sample points from the corresponding generative model
yi sin(2:rx:)) with (0,
(B.1)
using o, = 0.1 and draw x2 ~ -, 1)). We plot data points as gray circles, functions sampled from the approximate posterior green lines, the empirical mean function as red line and its empirical esstadard-devvattin have the same two hidden-layer architecture with 50 neurons per layer and hyperbolic tangent (lanh) activation functions. For FSP-LAPLACE, we use 100 context points placed on a regular grid and run maximum of 500 Lanezos iterations. For FVI, we sample 100 context points drawn from 2([-2,2) 2] at each update. Except when stated otherwise, we consider a centered GP prior and find the parameters of the covariance function by maximizing the log ong-magginal likelihood [13]. For the Laplace, we use the full generalized Gauss-Newton matrix and an isotropic Gaussian prior with scale op The MAP estimate uses the same prior. We find the parameters of the Gaussian process priors by maximizing the enargiala likelihood and the parameters of the sparse GP by maximizing the evidence lower bound [13].
interval around the mean as a green surface. All neural networks
Classification. We sample randomly perturhed data points from the two moons data [29] with noise level 02 0.1. We plot the the data points from class 0 as red dots and those from class blue dots. We show the mean (upper row) and E-standard-deviale sample æ belongs to class under the approximate posterior, which we estimate using K 100 samples. We consider a two hidden-layer neural network with 100 neurons per layer and hyperbolic tangent activation functions. For FSP-LAPLACE, we use 100 context points placed on regular grid over [-3.75, 3.75] x -3775, 3.75] and limit Lanczos to run for at most 500 iterations. For FVI. we sample 100 context points from (1 (.173,,,..755 3.75)²) at each update. We consider a centered GP prior and find the parameters of the covariance function by maximizing the log-marginal likelihood [13] using the reparameterization of classifications labels into regression targets from Milios et al. [61]. For the Laplace, we use the full generalized Gauss-Newton matrix and an isotropic Gaussian prior with scale a, 1. The MAP estimate uses the same prior. For the Gaussian process, we auplace-approximale the intractable GP posterior and find the prior parameters by maximizing the log-marginal likelihood [13]. Sparse GP parameters are found by maximizing the ELBO [13].
(bottom row) of the probability that
B.2 Quantitative experiments with real-world data
Mauna Loa. We consider the Mauna Loa dataset which tracks the monthly average atmospheric CO, concentration at the Mauna Loa observatory in Hawaï from 1974 2024 [30]. We consider the first 70% of the data in chronological order as the train set (from 1974 to 2005) and the last 30% as the test set (from 2009 2024). We standardize the features (time) and regression targets (CO2 concentration). We use two hidden-layer neural networks with hyperbolic tangent activations and 50 units each. We augment the input of the neural networks with an additional sine and cosine transformation of the features i.e., we use the feature vectors (f,, wiissiinnttt. (2tt. (T), cos(2tt,/TT) where I, the time index and T is the period used in Rasmussen and Williams [13]. For FSP-LAPLACE, we use 100 context points placed on regular grid and limit Lanczos to run at most 500 iterations. For FVI, we draw uniformly 100 context points at each update. For Laplace, we use the full generalized Newton matrix and find the prior scale by maximizing the marginal likelihood [11].
17

---

Ocean current modeling. We consider the GulfDrifters dataset [31] and we follow the setup by Shalashilin [32]. We use as training data 20 simulated velocity measurements (red arrows in Figure 2). and consider as testing data 544 average velocity measurements (blue arrows in Figure 2) computed over a regular grid on the [-90.8, - 83.8) x 24.0,27.55 longitude-latiuude interval. We standardize both features and regression targets. We incorporate physical properties of ocean currents into the models by applying the Helmholtz decomposition to the GP prior's covariance function [33] as well as to the neural network/ f using the following parameterization
f(, w) = -grad (t. w1) + rot V (,, W2)
(B.2)
where w = [wi, w22 and $(:, w1) and (V(,, w22) are two hidden-layer fully-conneche neural networks with hyperbolic tangent activation functions and 100 hidden units per layer. We use 96 context points placed on regular grid for the ESP-LAPPACC and limit Lanczos to run at most 500 iterations. For FVI, we use the same 96 context points. For the linearized Laplace, we use the full generalized Gauss-Newton (GGN) and fix the prior scale to Op 1.
Image classification. We consider the MNIST [34] and FashionMNIST [35] image classification datasets. We standardizing the images, fit the models on random partition of 90% of the provided train splits, keeping the remaining 10% as validation data, and evaluate the models on the test data. We report mean and standard-erron across 5 such random partitions of the train data with different random seeds. We compare our model to FVI, Laplace, MAP, and Sparse GP baselines, as the scale of the datasets forbids exact GP inference. The expected log- aigliikeld and expected calibration error are estimated by Monte Carlo integration with 10 posterior samples. All neural networks have the same convolutional neural network architecture with three convolutional layers (3 x 3 kernels and output channels 16, 32 and 64) interleaved with a max-pooling layer, before two fully connected layers (with output size 128 and 10). For FSP-LAPLACK we sample context points from the Kuzushiji-MNin
(KMNIST) dataset [62] of 28 x 28 gray-scale images during training following Rudner et al. [26] and use 250 points from the Halton low discrepancy sequence [63] to compute the covariance. We also consider sample context points uniformly over the range h.w.. , P 1,,=11 where H, W and C are respectively the height, width and number of h.w.c min - 0.5 x A h,wr. and pmnae = w.warc + 0.5 X A h,we. where A h,, = 1 maaz - min is the difference between the minimal (c (1)5 and maximal (1l values of the data set at pixel index (h, w. c). For FVI, we use the same context point distributions as FSP-LAPLACE. For the Laplace, we use the K-FAC approximation of the generalized Gauss-Newton matrix. We use Categorical likelihood, the Adam optimizer [64] with a batch size of 100 and stop training early when the validation loss stops decreasing. We optimize hyper-parameter Bayesian optimization tool provided by Weights and Biases [65] and select the parameters which maximize the average validation expected log-likelhhood across 1 random partitioning of the provided training split into training and validation data. We find covariance function parameters by maximizing the log-marginal likelihood from batches [66] using the reparameterization of classifications labels into regression targets from Milios et al. [61]. We optimize over kernel, prior scale, learning-rate -rate, de (introduced by Milios et al. [61]] and activation function and select covariance functions among the RBF, Matern- 1/2, Matern-3/2, Matern-57 and Rational Quadratic.
H,W.C'
channels of the images, and P b.w.c
N.w.C
using the
Out-of-distribution detection with image data. We consider out-of ant-of-distribution detection with image data and a Categorical likelihood following the setup by Osawa et al. [36]. We aim to partition in-distribution (ID) data from out-of-distribution (OOD) based on the mean of entropy of the predictive distribution with respect to 10 posterior samples using a single threshold found by fitting a decision stump. We evaluate model fit on MNIST using its test data set as in-distribution data (ID) and the test data set of FashionMNIST as out-of -f-dirribbuttionn (OOD), and vice-versa for model fit on FaShionMNISTT. We use the same models and apper-prameterss as for image classification and report mean and standard-cne -eroor of our scores across the same 5 random partitions of the data.
Bayesian optimization. We consider Bayesian optimization (BO) problems derived from Li et al. [7]. More specifically, we use the same setup but change the dimension of the feature space of the tasks. We report mean and standard error of repetitions of the tasks across different random seeds. We use two hidden layer neural networks with hyperbolic tangent activations and 50 hidden units each. ESP-LAPLACE uses Matern-5/2 covariance function with constant zero mean function whose parameters are found by maximizing the marginal likelihood [13]. We use 400 context points during
18

---

training and 10'000 to compute the posterior covariance sampled using latin hypercube sampling
[67]. We use the same prior for FVI and the same number of context points during training. The Gaussian process uses zero mean function and Matern-5/2 covariance function following Li et al.
[7]. For the Laplace approximation, we find the prior scale by maximizing the marginal likelihood
[1]]. Unlike FSP-LAPLACE which uses low-rank factors to parameterize the posterior covariance, we found that repeatedly computing the covariance and predictive posterior of the linearized Laplace with the full and K-FAC generalized Gewton (GGN) matrices was often prohibitively slow in the BO setup. We therefore use the K-FAC approximation to the GGN where possible (Branin and PDE) and the diagonal approximation otherwise (Ackley, Hartmann, Polynomial and BNN). We implemented this experiment using the BO Torch library [68].
Regression with UCI datasets. We consider tabular regression datasets from the UCI repository [69]. Specifically, we perform leave-one-out 5-fold cross validation, considering 10% of the training folds validation data, and we report the mean and standard-error of the average expected log- likelihood on the test fold. We report the mean rank of the methods across all datasets by assigning rank 1 to the best scoring method as well as any method who's standard error overlaps with the highest score's error bars, and recursively apply this procedure to the methods not having yet been assigned a rank. We estimate the expected log-likelihood using 10 posterior samples. We encoding categorical features as one-hot vectors and standardizing the features and labels. We consider two hidden-layee -ayer neural networks with 50 hidden units each and hyperbolic tangent activations. All models have a homoskedastic noise model with a learned scale parameter. FSP-LAPLACE uses context points drawn uniformly over the range [P'min P Pmazli=1+ where d is the dimension of the feature space, and Pinin miin - 0.5 x A and Pimaz = Vmmax 0.5 x A where A v'mar - the difference between the minimal (Mrmn) ) and maximal (mmaz) values of the data set at feature index i. For the Laplace, we use the full generalized Gauss-Newton matrix. FVI uses the same context points as FSP-LAPLACE Neural networks are using the Adam optimizer [64] and we stop training early when the validation loss stops decreasing. Hyper-parameters are found just as for the image classification experiment.
Out-of- distribiittiiio detection with regression data. We consider out-of-distributio (OOD) detection with tabular regression data from the UCI datasets [69] following the setup from Malinin et al. [70]. We aim to separate test data (in-distribution) from a subset of the song dataset [71] (out- of-distribution) with the same number of features based on the variance of the predictive posterior estimated from 10 posterior samples using single threshold obtained via decision stump. We process the data just like in the regression experiments, use the same model report mean and standard-error of the scores across the same 5 random partitions of the data.
and
B.2.1 Software
We use the JAX [72] and DM-Haiku [73] Python libraries implement neural networks. The generalized Gauss-Newton matrices used in the Laplace approximations are computed using the KFAC-JAX library [74]. We implemented the GPs and sparse GPs using the GPyTorch library [68]. We conducted experiments the Bayesian optimization experiments using the BOTorch library [68].
B.2.2 Hardware
All models were fit using single NVIDIA RTX 2080Ti GPU with 1IGB of memory.
C
Additional experimental results
C.1 Additional qualitative results
Regression. We show additional results for the synthetic regression task described in Section 4. We find that FSP-LAPLACE successfully adapts to the beliefs specified by different Gaussian process priors in terms of periodicity (Figure C.2), smoothness (Figure 1) and length scale (Figure C.3) without modifying the neural network's 's architecture or adding features. We also find that our method effectively regularizes the model when the data is very noisy (Figure C.4).
19

---

FSP-Laplace (ours)
FVI
Laplace
MAP
GP
Sparse GP
0
2
2
Figure C.I: Just like the Gaussian process (GP) and sparse GP. FSP-LAPLACE captures the smooth-
ness behavior specified by the RBF covariance function of the Gaussian process prior.
FSP-Laplace (ours)
GP
Laplace
0
-2
-2
-1
2
2
-1
2
Figure C.2: Unlike the linearized Laplace, FSP-LAPLACE allows to incorporate periodicity within the support of the data using periodic prior covariance function and without additional periodic features.
À = 0.15
À = 0.35
À = 0.5
adapts to the length scale provided by the RBF covariance function of the Gaussian process prior.
Figure C.3:
On = 0.1
Gn
Figure C.4: FSP-LAPLACE is effectively regularized under strong label noise.
Classification. We show additional results for the two-moons classification task described in Section 4. Similar the GP and sparse GP baselines, we find that our method captures the behavior of the prior, showing a smooth decision boundary when equipped with a RBF covariance function (Figure C.6) and rough decision boundary when equipped with a Matern-1/2 covariance function (Figure C.5). FSP-LAPLACE also reverts to the zero-valued mean outside of the data support.
Effect of context points. We provide additional details on the role of the context points in FSP. LAPLACE. The goal of the context points is to regularize the neural network input locations which includes any point where we would like to evaluate the model. During MAP estimation (see Algorithm 1), context points are resampled at each update step to amortize the coverage of the feature space. During the posterior covariance computation (sce Algorithm 2), context points are fixed and define where regularize the model. Context points bare similarity with inducing points in variational GPs [27] in this later step as both define where the model is regularized.
finite set of
We show additional results demonstrating the behavior of our model in the low context point regime. Figure C.7 shows the effect of the number context points on a 1-dimensional regression task with GP priors equipped with RBF and Matern 1/2 kernel. Figure C.8 shows the same experiment on a 2-dimensional classification task. The M context points are randomly sampled uniformly during training, and we use the Halton low discrepancy sequence context points to compute the covariance. Even with a very small number of context points (M- 3 and M = 5), our model still produces useful uncertainty estimates even if it cannot accurately capture the beliefs specified by the
20

---

FSP-Laplace (ours)
FVI
Laplace
MAP
GP
Sparse
1.0
-2
22
-06
00.
-04
0.3
2
Figure C.5: ESP-LAPLACE with a Matern 1/2 covariance function against baselines in the two-moons classification task. Similar to the Gaussian process (GP) and sparse GP, our method shows a rough decision boundary.
FSP-Laplace (ours)
Laplace
Sparse
FO.
X
0.5
0.44
2
2
Figure C.6: FSP-LAPLACE with RBF covariance function against baselines in the two-moons classification task. Similar to the Gaussian process (GP) and sparse GP, our method shows smooth decision boundary.
prior. We also note that our method requires more context points capture the beliefs of rougher priors than smooth priors (see Figure C.7).
C.2 Additional quantitative results
Mauna Loa. We here show the figure associated with the Mauna Loa dataset experiment in Section 4.1. Figure C.9 shows the predictions of LSP-LAPLACE and the baselines on the Mauna Loa dataset. We find that incorporating prior beliefs both via an informative prior and periodic features in FSP-LAPLACE results in improved fit over FVI, Laplace and GP baselines.
RBF kemnl
Matern-1/2 kernel
M-3 3
M-5
M-10
M-100
M-3
M-5
M-10
M-100
W
M
W
MM
W
W
W
2
2
2-0 2- 21012
21012 21012
Figure C.7: FSP-LAPLACE with smooth RBF covariance function and a rough Matern- 1/2 with varying amounts of context points M. Given a very small number of context points, our method still produces useful uncertainty estimates.
21

---

Rotated MNIST and FashionMNIIST. We provide an additional experiment studying the behavior of PSP-LAPLACE under out-of-distrine of--iitribution data. We consider the setup by Rudner et al. [25], Sensoy et al. [75] and track the predictive entropy of models trained on MNIST and FashionMNIST for increasing angles of rotation of the test images. We expect the predictive entropy of well-calibrated neural network to grow as the inputs become increasingly dissimilar to the training data with higher find that RBF kernel
Matern-1/2 kernel
M=3
M=5
M=3
M=5
2
-0.6 2
-0.5
-0.4
-0.3
0.2
-2
0.1 X
Figure C.8: FSP-LAPLACE with a smooth RBF covariance function and a rough Matern- 1/2 with varying amounts of context points M. Given a very small number of context points, our method still produces useful uncertainty estimates and reverts to the prior's zero valued mean.
GP
FSP-Laplace (ours)
FVI
Laplace
440
440
440
440
CD, cancentration (ppmn)
380
380
360
360
360
360
Test
2000
2020
1990
2000
1980
Year
Year
Year
Year
Figure C.9: Regression on the Mauna Loa dataset. Incorporating prior knowledge via a kernel tailored specifically to the dataset, our method (FSP-LAPLACE) results in strong decrease in mean square error over baselines.
Out-of-distribution detection with image data. We present additional results for the out-of- distribution detection experiment presented in Section 4.1. Figure C.10 shows the distribution of the predictive entropy of in-distribution (ID) and out-of aattof-distribution (OOD) data under the sparse GP, FSP-LAPLACE. FVI and Laplace models. For both MNIST and FashionMNIST, we find that predictive entropy of ID data under FSP-LAPLACE is tightly peaked around nats and that the predictive entropy of OOD data strongly concentrates around its maximum In 10 x 2.3 nats.
angles of rotation. Similar FVI, sparse GP and the linearized Laplace baselines, FSP-LAPLACE yields low predictive entropy for small rotation angles and that the predictive entropy increases with the angle on both MNIST and FaShionMNIST which what expect from well calibrated Bayesian model.
Regression with UCT datasets. We further evaluate our method on regression datasets from the UCI repository [69] and compare FSP-LAPIAAC ACE to FVI, Laplace, MAP, GP, and Sparse GP baselines. We perform leave-one-out 5-fold cross- vals-vali,,t,on, keeping 20% the remaining train folds as validation data. We report the mean and standard-ers of the expected log-likelihood with respect to samples
22

---

SVGP
FSP-Laplace (ours)
FVI
Laplace
MNIST
2
2.5
0.0
Fashion MNIST
0
0
1
1
2
2
2
1
Predictive entropy
Predictive entropy
Predictive entropy
Predictive entropy
Figure C.10: Distribution of in-distribution (ID) and out-of-distribution (OOD) samples for the MNIST and Fashion MNIST image datasets. The predictive entropy produced by FSP-LAPLACE nearly perfectly partition ID and OOD data. This reflected in OOD accuracy in Table 2.
MNIST
Fashion MNIST
2.0
PSPN aplace
Jeplace
SVCP
Predictvv entrpy
1.5
1.0
0.5
0.0
20
120
140
180
Angle of rotation
Angle of rotation
Figure C.11: Input distribution shift experiment. We find that FSP-LAPLACE shows greater predictive entropy as the input becomes increasingly dissimilar to the training data.
from the posterior across the 5-folds. Additional details can be found in Appendix B.2. Results are presented in Table C.I. We find that our method performs well compared to baselines, matching or improving over the mean rank of all Bayesian methods (FVI, Laplace, GP, and Sparse GP) but slightly lower mean rank than the MAP baseline (1.636 vs. 1.363). In particular, our method is noticeably more accurate than the Laplace.
Table C.I:
Test log-likellbbn ligeliikl and out-of-distribne distrribution accuracy of evaluated methods on regression datasets from the UCI repository. We find that our method matches or improves over the mean rank of Bayesian baselines in terms of expected oog-likelhhood and obtains out-of distribbiitnnn detection accuracies similar the best performing BNN.
EXPECIIN ING- FEMIS
WEC-R-EATTEE DIS DNTECTION AECERACY (T)
ESP-LAURACS (SU30)
211
LARAIS
SAAF
GF
StADia CP
F3PNAYRUMS (0284)
FV1
LAFANCE
6F
SPACIE
45.23
2112
-1.95 11.274
45352 EJF7
01325 0.112
0.36 1.115
3.17
EONE
1440 -1.23
1.45 0.250
1442 2362
L.154 L RUSS
1370....00 633
1 6.003
1,800 1.00
0/22 3.00
L449 FU800
L,KE 8.000
1899 EUK
1483 11818
15433 2-1155
1 8105
1.88 5
5.674 8.119
1999 8760
1,00 8148
1898 (8882-1180)
189 KU9
POOTTEES
.8588 16992
1.30
5.01
8.92- 61891
8114
832
41334 5.000
0.511
0.51 2312
8,515
Wese
111 01000
0.30--33 276
1.10 4 B.196
033 5/00
8.812 8.7%
6.0 8811
01513 1/00
Out-of-distribution detection on tabular data. We also investigate whether the epistemic uncer- tainty of FSP-LAPLACE is predictive of oitto---iitt data in the regression setting by evaluating on out-of-distribution detection following Malinin al. [70]. We report the accuracy of a single threshold to classify OOD from in-distribution (ID) data based on the predictive uncertainty. More details are presented Appendix B.2. the context of tabular data, we find that FSP-LAPLACE performs second best among BNNs and is almost as accurate as the Laplace approximation (see Table C) which first. We note that FSP-LAPLACE systematically outperforms FVI terms of out-of-distribution detection and obtains higher mean rank (1.818 vs. 3.091).
23

---

| IF is negligible. We provide evidence that the term P F in Section 3.2 is negli- gible compared to | in four different configurations. We consider the synthetic regression and classification setups described in Appendix B.1.
Table C.2: PP |///|//| for different combinations of priors and tasks.
KERNEL
REGRESSION
CLASSIRCATION
RBF
1.026 x 10
3.548 x 10
MATERN-1/2
10-
1.299 10
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experimental results support the fact that the our methods captures belief's specified by the GP prior.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in sections Sections 4 and
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

We rigorously detail assumptions in Appendix A.1 and proofs can be found in Appendix A.2 and the references.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We do so in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
Code will be published at https://gittie http://cccttttsttnnnniiio httptp/////tthhhoc//ttttttaaaaaa
fsplaplace.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
Ponnccy)))) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cc/ ccc/pbblcc/pee Seliides/ooeeubbee sssssssosiiyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report standard error along the mean scores across cross validation folds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http://///eeuip/eeeeeee
Answer: [Yes]
Justification:
To the best of our knowledge, we have fully conformed with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We believe that there is no direct societal impact of the work perform that needs discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The methods developed in the paper have risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets (code and data) used in this publication are publicly available, cited in Section and Appendix C.I, and their respective licenses are, the best of our knowledge, properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwodeettoo....m com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not conduct any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not conduct any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "xMVCwyd8qR", "reviewer": "Reviewer_Lja8", "summary": "This paper proposes a method to calculate the Laplace approximation of the posterior of neural networks with a prior defined directly in the functional space (Gaussian processes). Due to the absence of Lebesgue densities in infinite-dimensional functional spaces, the notion of weak mode is used to obtain an analogy to MAP estimation in the functional space. Through model linearization, the Laplace approximation is performed at the MAP parameter of the neural network. Experiments on both synthetic and real-world data demonstrate that the proposed method effectively captures the epistemic uncertainty in function fitting and classification.", "questions": "- What will happen if the same context points are used for both training and approximating the Hessian?\n\nAdditionally, please also see my questions raised in the Weakness section regarding the relation between this work and Sun et al., 2019, and the challenges of the context point sampling strategy for high-dimensional datasets.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses the posterior approximation in Bayesian deep learning, an important area for quantifying predictive uncertainty of models. The proposed method is well-motivated by the limitations of placing prior distributions over the parameter space of neural networks. The techniques used in this paper are solid.\n- The paper is generally very well-written, with clear and easy-to-follow notation.", "weaknesses": "My main concern is the relation between this paper and Sun et al. (2019) [8]. Sun et al. also consider Bayesian inference of function posteriors with a functional prior directly defined in the function spaces (e.g., Gaussian processes). The main difference seems to be that Sun et al. perform fully Bayesian inference, while this work aims to obtain a MAP estimate followed by a Laplace approximation to obtain the posterior. Sun et al.'s work should be included as a baseline, and more discussion on the relation between the two works is needed.\n\nSimilar to the difficulty in evaluating the functional KL divergence in Sun et al., when evaluating the \"prior probability\" (i.e., RKHS norm) of $\\\\mathbf\\{f\\}$, a set of context points must be specified to approximate it. My second concern is that this sampling strategy could be inefficient and impractical for high-dimensional datasets. For 1D or 2D datasets, the context points can be sampled uniformly or fixed to grid points. However, for higher-dimensional cases, this approach becomes less effective. This is a critical issue for the proposed method, as in high-dimensional spaces, there is little chance that the context points will adequately cover the support of the data distributions. For example, using KMNIST as the context points for MNIST may not always be feasible in general cases.\n\nMinor:\n- L 89, Section 3.1 -> Section 3.2.\n- L271, L533, L597: References are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback and time. We are glad that they found our \"method is well-motivated\", our \"techniques [...] are solid\" and that the paper was \"very well-written\" and \"easy-to-follow\".\n\n> \"My main concern is the relation between this paper and Sun et al. (2019) [8]. Sun et al. also consider Bayesian inference of function posteriors with a functional prior directly defined in the function spaces (e.g., Gaussian processes). The main difference seems to be that Sun et al. perform fully Bayesian inference, while this work aims to obtain a MAP estimate followed by a Laplace approximation to obtain the posterior. Sun et al.'s work should be included as a baseline, and more discussion on the relation between the two works is needed.\"\n\nWhile Sun et al. (2019) and our method both specify GP priors on the function represented by the neural network, Sun et al. use variational inference (VI), whereas our method uses the Laplace approximation. VI approximates the posterior by a parametric distribution that maximizes the ELBO (Eq. 3 in Sun et al.), while the Laplace approximation approximates the posterior by a Gaussian centered at the MAP estimate of the parameters. Note that the Laplace approximation and VI are both parametric *approximate* posterior inference methods that are commonly used in the machine learning and statistics literature. Neither one of them can be considered more \"fully Bayesian\" than the other. We agree that a comparison with Sun et al. is interesting, and we therefore include experiments using this method (labeled \"FVI\") in our overall rebuttal above. Our method FSP-Laplace generally outperforms FVI. This can possibly be explained by a complication in the FVI method: evaluating the ELBO requires access to the density of the variational distribution, which is not available in function space. FVI addresses this issue with implicit score function estimators, which introduce another approximation and makes FVI difficult to use in practice (Ma and Hernández-Lobato, 2021). On a more theoretical level, note that the KL divergence between measures (Eq. 4 in Sun et al.) is actually infinite when using GP priors (Burt et al., 2020) and the function-space ELBO (Eq. 3 in Sun et al.) is undefined. Our Laplace approximation does not suffer from either of these issues and approximates a well-defined objective.\n\n> \"Similar to the difficulty in evaluating the functional KL divergence in Sun et al., when evaluating the 'prior probability' (i.e., RKHS norm) of $f$, a set of context points must be specified to approximate it. My second concern is that this sampling strategy could be inefficient and impractical for high-dimensional datasets. For 1D or 2D datasets, the context points can be sampled uniformly or fixed to grid points. However, for higher-dimensional cases, this approach becomes less effective. This is a critical issue for the proposed method, as in high-dimensional spaces, there is little chance that the context points will adequately cover the support of the data distributions. For example, using KMNIST as the context points for MNIST may not always be feasible in general cases.\"\n\nMethods that regularize a neural network in function space (both for approximate Bayesian inference or regularized empirical risk minimization) always use a set of context points to evaluate the regularizer (Sun et al. 2019, Tran et al. 2022, Rudner et al. 2023). As you say, relying on a set of context points is fine for datasets with low-dimensional features, which are precisely the cases for which we have prior beliefs in the form of a GP prior (GP priors are much harder to specify with high-dimensional data). Our FSP-Laplace method was precisely designed with these scenarios in mind and the scalable matrix-free linear algebra methods can perfectly cope with a very large number of context points (> 20'000). For high-dimensional data, using context points is manageable if we have prior knowledge about where the training data lies (for example, a manifold in the ambient feature space). We can then use a set of context points that has approximately the same support, as shown in our image classification examples where we use a set of context points from the KMNIST dataset. Nevertheless, Table D.2 in the PDF linked in the general rebuttal also shows that our method remains competitive on MNIST image classification when using context points drawn from a uniform distribution during training. Due to limited time during the rebuttal period, results for Fashion MNIST are not yet available (N.As in Table D.2) and we will update the reviewers during the discussion phase with the results. Another example where prior beliefs can often be formulated in the form of a GP is scientific data (see, e.g., our ocean current experiment). For many scientific data, context points could also be generated by a simulation that does not need to be very accurate (since it is only meant to generate points in the vicinity of the true data distribution). We will add more details in the camera-ready paper. \n\n> \"What will happen if the same context points are used for both training and approximating the Hessian?\"\n\nThis is fine. It is only important that the set of context points covers a finite set of input locations containing all the training data as well as any points where we would like to evaluate the model. During training, we can amortize the coverage by resampling the context points at each update step (step 5 in Algorithm 2). When computing the covariance, we can no longer amortize and must use a set of points that covers the space well, such as a low discrepancy sequence (Latin hypercube sampling, for example).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83XmLKFROU", "reviewer": "Reviewer_wqcS", "summary": "The paper proposes a new method (FSP-Laplace) to place priors in the function space of deep neural nets and develop scalable Laplace approximations on top of it. The ideas are inspired in MAP estimation theory and the fact that an objective function that is actually regularising the neural network in the function space can be developed. Additionally, to face scalability and mitigate the issues of the computational costs in calculating large curvature matrices, the authors introduce methods from matrix-free linear algebra (i.e. Lanczos). Experimental results show that the FSP-Laplace method works well in such problems were makes sense to encode information in the form of a GP prior, and in general, overcomes the standard Laplace approximation.", "questions": "**Q1** -- In L201, it is mentioned that there is an observation on the fact that the trace of the posterior is 'always' upper-bounded by the trace of the prior covariance. Due to this, a little condition is imposed as a remedy. Is this an empirical observation? it is always true? or is there a proof for this?\n\n**After rebuttal comments:** Thanks to the authors for their detailed comments and clarifications to the few questions and points I raised in my first review. I am, in general, excited with the contributions that the paper brings to the community, particularly on Bayesian NNs and topics related to the (linearised) Laplace approximation. I do think that the paper is strong and should be accepted, so I thereby increase my score to 8.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and in general, well polished to give the right and clear details of a methodology that could be very difficult to follow otherwise. In that regard, the effort on synthesis and clarification is a big strength. The quality of the manuscript is therefore high (including the technical methods considered and the problem faced).\n\nGetting more in detail, I particularly like the spirit of the work and the idea of building a Laplace approximation directly on the functional space of the deep NN. Despite I don't see exactly how feasible could be to compute the potential \\phi and Eq. (3.1), I see what the authors actually do with the idea of the constraint in Propositions 1 and 2. \n\nDespite I have a little question here, I also see a point of strength in the way that the local linearisation fixes the problem of having valid unnormalized negative log-densities.\n\nThe use of the matrix-free methods is also thorough and despite the fact that it adds an extra point of complexity, it is clear what the advantages are and how it positively affects the scalability and performance of the method in practice. Last but not least, experimental results seem to show a good performance, and despite being somehow short, they are kind of sufficient and standard for the Laplace approximation considered and also in comparison with the rest of Laplace SOTA papers in the last years.", "weaknesses": "Even if I (honestly) don't find many weaknesses in the work, I think that the following points make the approach a bit weaker:\n\n- Context points in section 3.2 are kind of an obscure part of the Algorithm to me. Despite it is indicated that the best strategy is to use iid samples and that it is an effective way to compute the FSP-Laplace objective, I think a bit of related work and connections in this point could be needed. In general, it reminds me of inducing points in GPs and kind of pseudo-coresets. I also see that some limitations or issues with this sampling and the number of context points could appear, and somehow it is not really considered in the manuscript (in my opinion).\n\n- For instance, Laplace Redux [Daxberger 2021] states clearly the advantage of using Laplace approximation that it is feasible for both regression and classification problems, once the MAP estimate is obtained. In the case of FSP-Laplace, it is not clear to me if the work is also fully applicable to both classification and regression problems in an easy way, to which likelihood models, and if doing that makes things more difficult on the FSP-Laplace evaluation part or the Laplace approximation side. Perhaps, a bit more of clarification in this direction could be useful.\n\n\n**References.** \n\n[Daxberger 2021] -- Laplace Redux – Effortless Bayesian Deep Learning, NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and constructive feedback. We are glad that they found that the \"quality of the manuscript is [...] high\", that they \"like the spirit of the work\" and that our method has \"good performance\".\n\n> \"Context points in section 3.2 are kind of an obscure part of the Algorithm to me. [...] I think a bit of related work and connections in this point could be needed. In general, it reminds me of inducing points in GPs and kind of pseudo-coresets. I also see that some limitations or issues with this sampling and the number of context points could appear, and somehow it is not really considered in the manuscript (in my opinion).\"\n\nMethods for regularizing neural networks in function space always rely on context points to evaluate the regularizer (Sun et al. 2019, Tran et al. 2022, Rudner et al. 2023). Popular strategies include uniform sampling from a distribution over a bounded subset of the feature space (Tran et al. 2022, Rudner et al. 2023), from the data (Sun et al., 2019), and from additional (possibly unlabeled) datasets (Rudner et al. 2023).\n\nThe goal of the context points is to regularize the neural network on a finite set of input locations which includes the training data and any point where we would like to evaluate the model. During the MAP estimation phase of our method (see Algorithm 2), context points are different from inducing points in GPs: context points are resampled at each update, unlike inducing points of a GP, which are optimized or kept fixed. When computing the posterior covariance, however, context points indeed share some similarity with inducing points in a GP: at this point, they are fixed and define where we regularize the model.\nUnlike pseudo-coresets, context points do not aim to summarize the training data but rather to cover a finite set of input locations that includes the training data and any point where we wish to evaluate the model. \n\nRelying on a set of context points is fine for datasets with low-dimensional features, which are precisely the cases for which we have prior beliefs in the form of a GP prior (GP priors are much harder to specify with high-dimensional data). Our FSP-Laplace method was precisely designed with these scenarios in mind and the scalable matrix-free linear algebra methods can perfectly cope with a very large number of context points (> 20'000). For high-dimensional data, using context points is manageable if we have prior knowledge about where the training data lies (for example, a manifold in the ambient feature space). We can then use a set of context points that has approximately the same support, as shown in our image classification examples where we use a set of context point from the KMNIST dataset. Nevertheless, Table D.2 in the PDF linked in the general rebuttal also shows that our method remains competitive on MNIST image classification when using context points drawn from a uniform distribution during training (RND). Due to limited time during the rebuttal period, results for Fashion MNIST are not yet available (N.As in Table D.2) and we will update the reviewers during the discussion phase with the results. Another example where prior beliefs can often be formulated in the form of a GP is scientific data (see, e.g., our ocean current experiment). For many scientific data, context points could also be generated by a simulation that does not need to be very accurate (since it is only meant to generate points in the vicinity of the true data distribution).\n\nA sufficient number of context points is necessary to capture the beliefs specified by the GP prior (see Figure D.3 and D.4 in the PDF link to the general rebuttal). If the number of context points is too small, the beliefs are not accurately captured, and the function draws will not look like the prior but the uncertainty should not collapse (see Figures D.3 and D.4 in PDF in the general rebuttal).\n\n> In the case of FSP-Laplace, it is not clear to me if the work is also fully applicable to both classification and regression problems in an easy way, to which likelihood models, and if doing that makes things more difficult on the FSP-Laplace evaluation part or the Laplace approximation side.\n\nFSP-Laplace works in exactly the same settings as the standard Laplace approximation, e.g., for classification and regression settings using Gaussian and Categorical likelihoods, respectively (see regression and classification examples in Section 4.1). Just like Laplace, we require that the likelihood function is twice differentiable (we need to compute its Hessian). Note that for the classification setting with $c$ classes, we place a GP prior on every logit such that we have $c$ priors. Therefore, $M$ is the $p \\times rc$ matrix containing the concatenation of the $c$ $(J_i^T L_i)_{i=1}^c$ matrices in line 4 of Algorithm 1. We will make this clearer in the camera-ready version. \n\n> \"In L201, it is mentioned that there is an observation on the fact that the trace of the posterior is 'always' upper-bounded by the trace of the prior covariance. Due to this, a little condition is imposed as a remedy. Is this an empirical observation? it is always true? or is there a proof for this?\"\n\nThis follows from the expression of the covariance of the posterior in a Gaussian process, which is the prior covariance minus a term corresponding to the conditioning on the training data. The latter is always non-negative.\n\n> \"I don't see exactly how feasible could be to compute the potential \\phi and Eq. (3.1)\"\n\nIntuitively, the potential can be understood as the negative log-likelihood.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MTZMqf70dY", "reviewer": "Reviewer_ospn", "summary": "This paper proposes a functional Laplace approximation approach which is able to incorporate function space prior instead of weight space prior used by existing Laplace approximation methods. The function space prior is more meaningful than weight space ones in terms of expressing prior knowledge about the underlying functions. The core idea of this paper is, on top of [20], to resolve the problem of functional probability density issue of naïve extension to functional MAP. The results comparing to weight space LA show the effectiveness of the new functional LA.", "questions": "Can you please explain the equation in Line148 in more details?\n\nWhat is the meaning of 'rigorous probabilistic interpretation' in Line 128? Does that mean the new P_B(df) is a properly defined functional posterior?  \n\nThe objective function is similar to the kernel ridge regression. Can you please explain the difference?\n\nCan you please add comparative results with functional BNN?", "limitations": "n/a", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and it is easy to follow the idea. There are sufficient details and background information to understand each part.\nThe method is reasonably designed and the flow is smooth.", "weaknesses": "The comparative experiments are not sufficient. Various efficient LA methods should be compared. \n\nOrtega, L.A., Santana, S.R. and Hernández-Lobato, D., 2023. Variational linearized Laplace approximation for Bayesian deep learning. arXiv preprint arXiv:2302.12565.\n\nMcInerney, J. and Kallus, N., 2024. Hessian-Free Laplace in Bayesian Deep Learning. arXiv preprint arXiv:2403.10671.\n\nDeng, Z., Zhou, F. and Zhu, J., 2022. Accelerated linearized Laplace approximation for Bayesian deep learning. Advances in Neural Information Processing Systems, 35, pp.2695-2708.\n\nIt would be better to include some functional BNN as well. \n\nThere are some typos, like Line89 and Line94.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and positive feedback. We are glad that they found our paper was \"well-written\", \"easy to follow\" and had a \"smooth flow\".\n\n> \"The comparative experiments are not sufficient. Various efficient LA methods should be compared. [...] It would be better to include some functional BNN as well.\"\n> \"Can you please add comparative results with functional BNN?\"\n\nWe agree with the reviewer that a comparison with a functional BNN would be useful. Therefore, we ran our experiments on FVI (Sun et al., 2019) and present the results in the general rebuttal (see Figures D.1 and D.2 and Tables D.1, D.2 and D.3). We find that our method generally outperforms FVI.\n\nWe use the full Laplace posterior covariance for regression, Mauna Loa, and ocean current modeling; due to computational limitations, we use only the KFAC approximation for the classification experiments and for Bayesian optimization. KFAC is standard in Laplace approximations for neural network (Ritter et al. 2018, Immer et al. 2021) and has shown to perform very competitively. The methods proposed by the reviewer offer more scalable alternatives when the full GGN is not available. While these are interesting in themselves, the point of this paper is to discuss the influence of a function-space prior and not of the GGN approximation. Exploring the use of the methods proposed by the reviewer to make our method more scalable sounds like an interesting direction for future work.\n\n> \"Can you please explain the equation in Line148 in more details?\"\n\nThe equation follows from inserting the definition of $f^\\text{lin}$ (Eq. 2.2) into the left-hand side and completing the square. We note that the $\\dagger$ symbol designates the pseudo-inverse.\nIn detail:\n\n<!--$$\\frac{1}{2} ||f^{\\text{lin}}(\\,\\cdot\\ , \\mathbf{w}) - \\mathbf{\\mu}||_{\\mathbb{\\mathbf{\\Sigma}}}^2 = \\frac{1}{2} (\\mathbf{w} - \\mathbf{w}^*)^\\top \\mathbb{\\mathbf{\\Sigma}}_{\\mathbf{w}^*}^\\dagger(\\mathbf{w} - \\mathbf{w}^*) + \\mathbf{v}\\mathbf{v}^\\top (\\mathbf{w} - \\mathbf{w}^*) + \\frac{1}{2} ||f(\\,\\cdot\\ ,  \\mathbf{w}^*) - \\mathbf{\\mu}||_\\mathbb{\\mathbf{\\Sigma}}^2$$ -->\nfrom which we complete the square to obtain the right-hand side of line 148.\n\n> \"What is the meaning of 'rigorous probabilistic interpretation' in Line 128? Does that mean the new P_B(df) is a properly defined functional posterior?\"\n\nWe mean that $\\frac{1}{\\lambda} d_B(f, F)$ can be interpreted as a negative log likelihood encoding that we observe $d_B(f, F) = 0$ with Laplace distributed noise (with parameter $\\lambda$). The proposition shows that the optimization problem is the corresponding MAP estimation problem. \n\n> \"The objective function is similar to the kernel ridge regression. Can you please explain the difference?\"\n\nGood observation! The kernel ridge regression (KRR) can be seen as the maximum a-posteriori of a GP.\nThus, the first part of our algorithm, which learns an RKHS-regularized neural network, bears some similarity with KRR.\nHowever, KRR is a nonparametric method wherease the first part of our algorithm optimizes the parameters of a neural network (this is motivated by the generalization performance of deep neural networks, as mentioned in the abstract).\nFurther, the second part of our algorithm estimates uncertainties of the neural network parameters, which has no analog in KRR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DfSWFsQDtt", "reviewer": "Reviewer_KV8g", "summary": "This work takes a function space view of the posterior distribution, and places a GP on the likelihood with neural network predictors. This generate a posterior distribution, and the authors attempt a Laplace approximation as a which they combine with matrix-free linear algebraic methods to aid in computational tractability. Numerical examples are provided demonstrating their method performs well.", "questions": "Can you please address the weaknesses above?", "limitations": "No, limitations were not adequately addressed. However, there is little view for negative societal impact in work of this nature.", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "This paper is considers an important problem -- high quality uncertainty estimates for neural models are an important area of machine learning research, and crucial when deploying models in critical systems. \nThe work is generally well written and presented. \nThe method performs well in the numerics section.", "weaknesses": "Laplace approximations are justified by the Bernstein-von Mises theorem, which states that (under appropriate assumptions) the posterior distribution concentrates around its mode \\emph{independent of the prior distribution} in the large-data limit. \nIn order to use a Laplace approximation out of the box, it's important for these assumptions to be met, and the model to be in the appropriate pre-limit. \nSingular learning theory is one approach that addresses this for singular models such as neural networks (see https://www.routledge.com/Mathematical-Theory-of-Bayesian-Statistics/Watanabe/p/book/9780367734817). \nHowever approximating the Hessian, or regularizing it and claiming it's because there's a prior not only does nothing to fix the problem, but perpetuates a line of ill-posed research. \nWhile taking a function space view opens up the possibility of priors that concentrate with the dataset size and therefore cannot be discarded in the BvM limit, the authors claim that the eigenvalues of their prior term rapidly decay and discard them, killing any possibility that they have considered this concentration. \nother forms of Laplace approximations may be more suitable (see https://arxiv.org/pdf/2110.12922, https://arxiv.org/abs/2307.07785 for related methods using expansions in data-space). \nHaving said this, the work does produce decent empirical results as a quadratic approximation of the posterior \\emph{inspired by} Laplace approximations, however the current reasoning used to get there is ill-considered.\n\nThe dependence of $f$ appearing in Propositions 1 and 2 should be made explicit.\n\nThe proofs given in the paper are not sound. In particular, the proof of Proposition 2 is \"Analogous to the proof of Theorem 5.2(b) from Lambley [20].\". However, Theorem 5.2(b) in [20] is not proved, where it is stated that the proof is similar to another reference. That reference also points to the proof of another corollary, stating it is analogous. A chain this long suggests not only that the proof needs to be written out in the current context to concretely check that the 3 analogies used are sound, but that the authors have not bothered to check the references themselves, let alone write out the proofs internally. The proof of Proposition 1 is also not clear, and needs to be written out, so that it is not burdensome for the reader to confirm the claims.\n\nThe claim that the term appearing in line 195 is negligible should be confirmed numerically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback. We are happy that they found the paper was \"well written\" and that our method \"performs well\".\n\n> \"Laplace approximations are justified by the Bernstein-von Mises theorem  which states that [...] the posterior distribution concentrates around its mode *independent of the prior distribution* in the large-data limit. [...] However approximating the Hessian, or regularizing it and claiming it's because there's a prior not only does nothing to fix the problem, but perpetuates a line of ill-posed research.\"\n\nWe respectfully disagree. It is true that the Bernstein-von Mises theorem only holds in the large-data limit. However, there is a long line of empirical work suggesting that the linearized Laplace approximation works surprisingly well for neural networks despite the lack of theoretical justification [(Papamarkou et al. 2024, Section 3.1 references many such papers)](https://proceedings.mlr.press/v235/papamarkou24b.html). Our paper provides additional empirical evidence supporting this observation. Calling this entire line of research \"ill-posed\" needs to be measured against the large amount of empirical evidence that clearly demonstrates the value of such research.\n\nMore generally, the point of empirical research is precisely to explore regimes in which thoeretical statements are not possible. Rigorous theoretical statements can often only be proven under strong assumptions, but this does not mean that the opposite of the statement is true as soon as assumptions are violated (absence of proof is not proof of absence). Empirical research is indispensable to explore how far away from a perfect limiting case remnants of a theory still apply. For example, the entire field of deep learning relies on optimization methods for which most of the theory applies only to convex functions, and yet methods motivated by this theory are successfully used to train highly non-convex deep neural networks because empirical studys show that they work surprisingly well in certain useful regimes far beyond convexity. This is not an argument against the value of theory. On the contrary: it would have been impossible for us to \"guess\" the empirically successful approximation scheme that we propose had we not been guided by theory that strictly holds only in a limiting case that is admittedly relatively far away from our applications.\n\n> \"the work does produce decent empirical results as a quadratic approximation of the posterior *inspired by* Laplace approximations, however the current reasoning used to get there is ill-considered.\"\n\nWe thank the reviewer for acknowledging our empirical results. We agree that these results were obtained with a method that is inspired by Laplace approximations (see our proofs of Propositions 1 and 2 in the additional comment for clarification on this connection). As with most forms of scalable Bayesian inference, several additional approximations were necessary to make our method usable in practice. We believe that our paper clearly highlights the additional approximations, e.g., by explicitly grouping them into Section 3.2. These additional approximations do not change the fact that the resulting algorithm approximates the posterior with a Gaussian distribution whose mean is the MAP (of the RKHS-regularized neural network) and whose precision matrix is obtained by approximating the curvature at this point. Methods for approximate posterior inference that follow this scheme are generally called Laplace approximations in the literature.\n\n> \"The dependence of f appearing in Propositions 1 and 2 should be made explicit.\"\n\nWe do not understand your inquiry, would you mind elaborating on what you mean by the \"dependence on f appearing in Propositions 1 and 2?\nFrom our point of view, there is no dependence on any particular $\\mathbf{f}$ in Propositions 1 and 2, since in these propositions $\\mathbf{f}$ is an unbound variable denoting an arbitrary element of the sample space $\\mathbb{B}$ of the GP prior (or an element of the RKHS $H_{\\mathbf{\\Sigma}} \\subset B$ depending on the context).\nMoreover, the $\\mathbf{f}^\\star_n$ in Proposition 2 are explicitly defined as minimizers of a sequence of optimization problems, and $\\mathbf{f}^\\star$ is also explicitly defined as the weak limit of a certain subsequence of $( \\mathbf{f}^\\star_n )_{n \\in N}$.\n\n> \"The proofs given in the paper are not sound. In particular, the proof of Proposition 2 [...]. The proof of Proposition 1 is also not clear, and needs to be written out\"\n\nWe agree that the proof of Propositions 2 is difficult to follow as one has to combine several ideas in the referenced works in a nontrivial manner.\nHence, we conducted the proof of Proposition 2 in full detail and attach it as a separate comment below.\nIt will also be included in the appendix in the camera-ready version of the paper.\n\nUnder Assumptions A.1 to A.3 detailed in the paper, Proposition 1 is a pretty direct corollary of Theorem 1.1 in (Lambley, 2023).\nThe given proof in the paper verifies the assumptions of Lambley's Theorem 1.1 and is hence complete.\nHowever, we will make an effort in the camera-ready version of the paper to add more explanatory comments to make the proof easier to follow.\n\n> \"The claim that the term appearing in line 195 is negligible should be confirmed numerically.\"\n\nWe provide evidence that the term $P_0 \\Lambda P_0^\\top$ is negligible in four different configurations.\nFirst, considering the regression setup described in Appendix B.1, we have:\n* RBF kernel: $\\frac{||P_0 \\Lambda P_0^\\top||_F}{||\\Lambda||_F} = 1.026 \\times 10^{-7}$\n* Matern-1/2: $\\frac{||P_0 \\Lambda P_0^\\top||_F}{||\\Lambda||_F} = 4.305 \\times 10^{-6}$\n\nConsidering the classification setup described in Appendix B.1, we have:\n* RBF kernel: $\\frac{||P_0 \\Lambda P_0^\\top||_F}{||\\Lambda||_F} = 3.548 \\times 10^{-4}$\n* Matern-1/2: $\\frac{||P_0 \\Lambda P_0^\\top||_F}{||\\Lambda||_F} = 1.299 \\times 10^{-2}$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and feedback. Some reviewers requested a comparison with FVI (Sun et al., 2019) and more details on the effect of the context points. We ran these additional experiments and wish to share them with all the reviewers. 

**Comparison with FVI (Sun et al., 2019):** We ran our suite of experiments on FVI and report results in the Tables D.1 to D.3 and Figures D.1 and D.2 in the linked PDF file.
Due to limited time during the rebuttal period, we are unable to provide results for FSP-Laplace using the RND context points on Fashion MNIST (N.As in Table D.2). We will update the reviewers with the results during the discussion phase.  

We find that FVI is less accurate than FSP-Laplace and baselines on the Mauna Loa $CO_2$ forecasting task (see Table D.1). Due to limited space in the 1 page PDF, we do not show Figure C.10 updated with FVI. On the ocean current modeling task, FVI performs similarly to our FSP-Laplace method in terms of mean squared error but strongly underestimates the predictive variance (see Figure D.1) and therefore incurs a lower test expected log-likelihood than FSP-Laplace (see Table D.1). We also find that FSP-Laplace shows higher expected log-likelihood and accuracy on the image classification tasks and produces models with lower expected calibration error (ECE) than FVI (see Table D.2). FSP-Laplace also obtains higher OOD detection accuracy than FVI when using the KMNIST context point distribution. Preliminary results using context points uniformly sampled from a subset of the features space (RND) on MNIST, also show improved performance compared to FVI. On the Bayesian optimization task, we find that FSP-Laplace converges faster than FVI on Branin, PDE, and BNN, that FVI converges faster than FSP-Laplace on Polynomial, and that both methods perform similarly on Ackley and Hartmann (see Figure D.2).
Finally, on the UCI regression tasks in Table D.3, we find that FSP-Laplace and FVI perform similarly in terms of expected log-likelihood with a slight advantage for FSP-Laplace (mean rank of 1.545 vs. 1.700). However, FSP-Laplace almost systematically performs best in terms of out-of-distribution detection (mean rank of 2.182 vs. 3.000). 

**Effect of context points:** We show additional results demonstrating the behavior of our model in the low context point regime. 

Figure D.3 shows the effect of the number of context points on a 1-dimensional regression task with GP priors equiped with a RBF and a Matern 1/2 kernel. Figure D.4 shows the same experiment but on a 2-dimensional classification task. The M context points are randomly sampled uniformly during training, and we use the Halton low discrepancy sequence as context points to compute the covariance.

We show that even with a very small number of context points (M=3 and M=5), our model still produces useful uncertainty estimates even if it cannot accurately capture the beliefs specified by the prior. We also note that our method requires more context points to capture the beliefs of rougher priors than smooth priors (see Figure D.3). 

Table D.2 shows the results of FSP-Laplace when using the Kuzushiji-MNIST dataset (KMNIST) and draws from a uniform distribution (RND) as context points during training. We use the Halton low-discrepancy sequence as context points to compute the covariance. Due to limited time during the rebuttal period, we are unable to provide results for FSP-Laplace using the RND context points on Fashion MNIST (N.As in Table D.2). We will update the reviewers with the results during the discussion phase. On MNIST, we find that FSP-Laplace with RND context points obtains similar expected log-likelihood, accuracy, and expected calibration error as with the KMNIST context point distribution. OOD detection accuracy is lower than with context points from KMNIST but remains very competitive with respect to baselines.

Meta-review: The submission considers using a function space prior in place of standard weight priors for the Laplace approximation in neural networks. After several approximations, this boils down to a tractable regularisation that enforces the neural network's prior predictive distribution to match that of a Gaussian process prior at a small number of context points. The empirical results are generally promising, showing the utilities of the new method. The suite of experiments is, however, not as comprehensive as recent Bayesian deep learning / Laplace papers.  

A reviewer raised a substantial concern, resulting in an unsettled debate between the reviewer and authors in the rebuttal. The issue is the theoretical justification of Laplace and its independence from the prior. The AC noted the concern is nitpickingly technically correct. However, what the authors proposed and the approximations to make things tractable, similar to other recent `Laplace` methods, is approximate and no longer strictly Laplace and seems to give good empirical results. Other reviews are generally positive about the submission. The AC thus suggests acceptance.

One thing that the reviewers have surprisingly missed is Rudner et al's tractable functional VI scheme (NeurIPS 2022). The AC encourages the authors to compare FSP-Laplace to this.
Decision: Accept (poster)