Approximate Bayesian Inference with Stein Functional Variational Gradient Descent
Authors: Tobias Pielok
Bernd Bischl
David Rügamer
Venue: ICLR 2023 poster (2023)

Abstract: We propose a general-purpose variational algorithm that forms a natural analogue of Stein variational gradient descent (SVGD) in function space. While SVGD successively updates a set of particles to match a target density, the method introduced here of Stein functional variational gradient descent (SFVGD) updates a set of particle functions to match a target stochastic process (SP). The update step is found by minimizing the functional derivative of the Kullback-Leibler divergence between SPs. SFVGD can either be used to train Bayesian neural networks (BNNs) or for ensemble gradient boosting. We show the efficacy of training BNNs with SFVGD on various real-world datasets.

Published as a conference paper at ICLR 2023
APPROXIMATE BAYESIAN INFERENCE WITH STEIN FUNCTIONAL VARIATIONAL GRADIENT DESCENT
Tobias Pielok, Bernd Bischl, David Rügamer
Department of Statistics, LMU Munich, Munich, Germany Munich Center for Machine Learning, Munich, Germany (tobias. (tobias.pielok, bernd. www.biichhhh bishl, david. ruegamer @asatt unn -muenchen.de
ABSTRACT
We propose general-purpor variational algorithm that forms a natural analogue Stein variational gradient descent (SVGD) in function space. While SVGD successively updates set of particles to match target density, the method intro- duced here of Stein functional variational gradient descent (SFVGD) updates a set of particle functions to match a target stochastic process (SP). The update step is found by minimizing the functional derivative of the Kullback-Leible divergence between SPs. SFVGD can either be used to train Bayesian neural networks (BNNs) or for ensemble gradient boosting. We show the efficacy of training BNNs with SFVGD on various real-world datasets.
INTRODUCTION
Bayesian inference can be treated as a powerful framework for data modeling and reasoning under uncertainty. However, this assumes that we can encode our prior knowledge in a meaningful manner. Typically, this is done by specifying the prior distribution of the model parameters. However, in machine learning (ML), models potentially consist of millions of parameters with potentially highly complex interactions (e.g.. very large neural networks (NNs)). Furthermore, the parameter structure of the models itself is allowed to change during training, e.g., the number of parameter grows when using gradient boosting (GB). This makes defining meaningful prior assumptions for parameter spaces difficult or nearly (practically) infeasible. As we usually do not care about single parameters but complete resulting function, it seems intuitive to directly express function space by, c.g., specifying the characteristic length scale, periodicity, or smoothness in general. Fortunately, Bayesian inference can also be formulated in function space. In this case, the prior and posterior distributions are stochastic processes (SPs). The most prominent representative the Gaussian process (GP), for which the posterior GP can be analytically computed. However, training GPs scale cubically in the number of observations, and the implicit Gaussian likelihood assumption often violated in reality. this paper, we introduce Stein functional variational gradient descent (SFVGD). This method provides general gradient descent method in function space that enables practitioners train ML models to approximate the posterior SP, assuming certain regularity conditions of the prior SP and the likelihood function hold.
prior knowledge in hypothesis
1.1 RELATED WORK
Kernelized Stein Methods These methods combine Stein's 's identity with a reproducing kernel Hilbert space (RKHS) assumption. Based on finite particle set, they either be used to find the optimal transport direction to match a target density or to estimate the score gradient of the empirical distribution of the particles. The former is called Stein variational gradient descent (SVGD) Liu Waan (2066), and approaches of the latter category are called (non-pate coon-paraettric score estimators (Zhou al. 2000). Our method internally uses SVGD and forms natural analogue in function space. Several extensions to SVGD exist, e.g., approaches incorporating second-order information such Leviyev et al. (2022) and the more general matrix-kernel valued approach by Wang al.) (2019a). While these extensions usually outperform SVGD, their computational costs are also higher.
Bayesian Neural Networks (BNNs) Typically. BNNs are NNs with weight priors that are trained via variational inference. The prominent representatives are BNNs using Bayes by Backprop (Blundell

---

Published as a conference paper at ICLR 2023
et al. 2015) and scalable probabilistic backpropagation (Hernandez-Lobate & Adams 2015), Re- cently, Immer et al. (2020) proposed transforming BNNs into generalized linear models with inference based on Gaussian process equivalent to the model. While Markov Chain Monte Carlo (MCMC) methods often are prohibitively expensive to be used for BNNs, some variants, e.g., Chen et al. (2014) account for noisy gradient evaluations and can be used in this setting. However, MCMC-based methods are still usually employed for relatively low-dimensional problems.
Functional BNNs (FBNNs) Sun et al.] (2019) proposed to use functional priors to train BNNs. Training BNNs with our descent is closely related to their method, but while they use score-based approach for the estimation of the derivative of the Seillback-eeibler divergence DKL between the
prior SP and the variational SP, estimate this derivative directly via SVGD. Waag al. (2019b)
also use SVGD for FBNNs but apply SVGD directly to the DKL. between the posterior SP and the variational SP. Furthermore, their work does not show that this in fact maximizes lower bound for the log marginal likelihood. Recently, Ma & Hernández- proposed different FBNN approaches that also build upon the results of Sun et al. (2019). but while their methods are specific to training NN function generators, our method can be used to update set of particle functions in general.
(202T) and Rudner et al. (002T)
Repulsive Deep Ensembles Repulsive Deep Ensembles deep ensembles that incorporate repul- sive terms in their gradient update, forcing their members' weights apart. A variety of repulsive terms presented in (D' Angelo & Fortuin 202T) and (D'Angelo et al., (021], outperforming the approach by Wang et al. (2019b) However, these approaches mainly focus on weight priors, and empirical findings also only relate to the weight space. In contrast to our work, functional priors can only be applied if posterior SP with analytical marginal density exists.
GB with Uncertainty The closest neighbor of our approach applied to GB is the ensemble GB scheme proposed by Malinin al. (202)) which is based on Bayesian ensembles. In contrast to functional approach, their method is based on approximating the posterior of the model parameters. Another GB-based method is NGBoost proposed by Duan al. (2019), which directly learns the predictive uncertainty; however, prior knowledge can not be taken into account.
1.2 OUR CONTRIBUTION
We propose a novel natural extension of SVGD in function space (Section B) which enables the practitioner to match a target SP. This approach can be implemented in BNN or as GB routine (Section 3.3). Using real-world benchmarks, we show that the resulting generator training algorithm competitive while having less computational costs than the approach Sun et al. (2019). In contrast to other existing uncertainty-aware GB algorithms, a GB ensemble, when trained via SFVGD, can naturally incorporate prior functional information. These versatile applications of our framework are made possible by providing a unifying view of NNs and GB from a functional analysis perspective.
2 BACKGROUND
2.1 SUPERVISED ML. FROM A FUNCTIONAL ANALYSIS PERSPECTIVE
Given a labeled dataset D E (X' 37)" Phy of n N independent and identically distributed (i..d.) observations from an unknown data generating process Pxy. a supervised ML algorithm tries to construct risk optimal model under a pre-specified loss L. In this case, the function f defines a mapping from the feature space X' to the target space V. The learning algorithm I to construct / is function mapping from the set of all datasets Unen( x 3/)" to hypothesis space H. which is a subset of the set of all functions mapping from X' to the model output s D C R with E N. In order to specify the goodness-of fit of a function f. one can define a loss function L. y' x ) + RR (y. fxx) - L(y, f(x)), which measures how well the output of a fixed model H fits an observation (x, 3) xxx- In the following, we present supervised ML from functional analysis perspective. Here, we fix the observation and associate the loss L with the loss functionalL 7 + R. + L(y. 0..55... f(x)). Based on this loss functional, we can define the risk functional of model f.
R[ B(xx,))= Tx(x,1)
If y is numeric, 5 Otherwise, 5 is numerical encoding of y.
2

---

Published as a conference paper at ICLR 2023
which measures the expected loss of f, and is used to theoretically identify optimal models. In the following, we will assume that the expectation in Eq. (I) exists and is finite. we knew the usually unknown data generating process and hence the risk functional, we could update any model f e H. in the direction of the steepest descent in H w.r.t. R. by following the negative functional gradient of R. The negative functional gradient of R, -V f RR[f], is itself a mapping from X' to D. For every input location x, this gradient returns the direction in model output space V, which points the locally steepest descent w.r.t. R. In the following, unless otherwise stated, the functional derivative is taken in the L space.
Proposition 2.1 Assuming sufficient regularity and that L(y, f(x)) is partially continuously differ- entiable w.r.t f(x), we observe for numeric inputs and model output that
aLyy,
V)R[[((x== -px((x) Eny-P u/x 0f(x)
where px is the marginal density xf
The proof is given in A.I.I In practice, we usually do not know Px. and since our dataset finite, we only have access to n realizations of (L()./(x)) If the feature space is at least partially continuous, its size | oo, and we thus cannot estimate -V, R[f](x) without additional assumptions.
0f(x)
However, we have access to the functional empirical risk Remp.D|f] == Lxx.g.)eD (x)))))) [f]. for
which we assume that converges in mean to R. as n - co. Its negative functional gradient can be expressed via the chain rule such that
aL f(x)) 0f(x.)
-V RymmmppDlf[() -- E
[(x:)]
(3)
(X.9:)E
where y [xx)] is the functional gradient of the evaluation functional of f at x;, which evaluates to the Dirae delta function 8x,- However, since we take the functional gradient in H. V, [xx,) becomes the projection of 8x, into H. For example, if H. is an RKHS with associated kernel k, then V [fxx)] (x) = k(x,, x), i.e., our choice of directly influences the "bumpiness" of V y Reemp, [..). Furthermore, we can interpret V /eRmmppD p[f] as jump-)continnour functional representation the dataset 0Dr, = ((x;, _Lyy,/(xill) ) (xi, ye) E D) C (X' x D)", which also implicitly defines a learner. In the following, we show how two core supervised ML algorithms (gradient boosting and neural networks) naturally incorporate this functional gradient while training.
0f(x,)
Gradient Boosting (GB) For GB (Friedman) 200I), the situation is usually reversed, and we choose (base) learner Ia that implicitly defines H and with which fit model to the data set aDL, GB uses these approximations of the negative functional gradient of the empirical risk to successively update model f(u) such that
|ل+ 1 + 1969 with d
Io (OD ft.
where 1/1 Rso is the learning rate and possibly depends on the iteration / N. For further details see Appendix (A.2).
Neural Networks (NNs) If f is an NN with parameters Φ, then the parameter gradients w.r.t. the empirical risk functional needed for backpropagation can be obtained via the chain rule such that
OL(y,, f(xi)) Vof(x;), (5)
V * Remp.
V RyRemp.Dlf(e x
(xf(x)x =
Σ
0f(x,)
(x.D:))D
where the second equality holds, since here we do not restrict H, i.e., V, [f(sxx &x..
However, these procedures only assure that we can find an optimal model f E H. w.r.t Remp- which does not imply that is optimal w.n.t. R. In practice, tune the hyperparameters of the algorithms ie., use data withheld from learning for subsequent model selection - and apply early stopping to find a model approximately optimal w.r.t. R.
3

---

Published as a conference paper at ICLR 2023
2.2 STOCHASTIC PROCESSES
In this section, we will shortly introduce stochastic processes (SPs) that can be used to represent distributions over functions and thereby allow to express the uncertainty of models independent of their specific parameter structure. We will regard X' as an index set and let ()², 9) be a measurable space with o-algebra on the state space V. Forx x E X' and a given probability space (1,, PP) based on the sample space S2 F is a o-algebra on Q and probability measure I.. Let Q(x) be a random variable projecting from Q to y. An SP Q the family {Q(x); x E } of all random variables Qxx) (Lamperti] 1977). With this, we can define sample function fw X - V.x + (x))(W for a fixed w E S2. Olten, it easier to look at SPs from this sample function view: For every A E F, set of functions (Sw; w E A} with an associated measure P(A) can be identified i.e. SPs define a distribution over functions projecting from X' to y. For a finite index set X == X1:m E X we denote the mnite-dimensional marginal joint distribution over function values (Q(x))..... ((xm) as Qx In the following, we assume that for every Qx exists a corresponding density function PQx 3 Rgo, Px + PQx (PX), where fx == (f(x)),...- fxxx)) ) are the function values at X1:m based on a sample function f where we suppressed the w to ease the following notation. We will denote the associated functional this density function with pQx [f].
The DKL is a measure of distance between two distributions over the same probability space. Since SPs are distributions over functions, the DKL can also be used for distances between two SPs. Unfortunately, computing this quantity is non-trivial (Matthews al., 2015). However, for two consistent and ergodic SPs Q and P, i.e. Q and P can be characterized by marginals over all finite index sets (e.g.. GPs), /Snn et al. (2019) showed that the DKL between these SPs can be solely expressed in terms of their marginals, i.e.,
(Q
sup MEN,XEX"
DKL (xxllxx
This expression enables to find differentiable distance measure between two stochastic processes.
2.3 STEIN VARIATIONAL GRADIENT DESCENT
SVGD (Liu & Wang 2016) is a variational Bayesian method. Variational methods can be used to approximate the generally intractable posterior density of a continuous random variable 0
PD o D 0 (0)
Pe ( J PD o D 0 yo (O)do
where PDie and pe are the likelihood and the prior density function, respectively. SVGD tries to
match the posterior Pays with a density g represented via a fixed number r E N of pseudo-samples
so-called particles- - and iteratively updates them by minimizing (II(o) = f (() log(a(0)) In RKHS with associated kernel k the optimal update direction is found by considering the negative functional derivative
(0) dO.
V DKt (trll//rrlllloopppps>
Ea-g [vo log (0)
Vok(0,))]
where T(0) + ((0), and yız] is the density of 0' T(0) when 0 q. We can estimate this functional gradient based on the particles in unbiased manner, as we are able to evaluate the score function of D (i.c., V log po)p), although log PoyD might be intractable.
STEIN FUNCTIONAL VARIATIONAL GRADIENT DESCENT
In this section, we develop functional version of SVGD which we will call Stein functional variational gradient descent (SFVGD). While SVGD can be used to approximate the posterior distribution of continuous random variable, SFVGD can be applied when we are interested in the posterior SP PrID defined by its Radon-Nikodyy derivative (Schervish) 1995) w.r.t. the prior SP Pj.
dPrI [f] J Ppis(DIfddP) dP,
where prl is the likelihood functional, which measures how likely it is to observe D, given a sample function f. In the following, we assume that the posterior PAID exists and also that it is ergodic and consistent SP. Analogously to SVGD, we try to approximate PAID with a distribution Q represented by pseudo-samples. However, for SFVGD, these particles are now functions.
4

---

Published as a conference paper at ICLR 2023
3.1 OBJECTIVE FUNCTION
Since analytical solutions for the differential Eq. (9) only exist in special cases (e.g.. if the prior Pj is a GP and the likelihood is also Gaussian), we use the DEL between two SPs formulate an optimization objective. More specifically, the goal of our framework is to construct an approximating measure Q" for which it holds that
Q' E arg min DRL(Q]1P/R); QEO
(10)
where Q is the set of representable variational posterior processes. Here, represent Q via r E sample functions fi..... fr from Q. which act as pseudo-samples and which we also call particle functions. It can be shown (Matthews al. 2015) that minimizing Eq. (TO) is equivalent to maximizing the functional evidence lower bound (HLBO) LD, i.e,
Q" E arg max E [(D]/]] /1] DRL. (1)) QEQ
(11)
m:Cp(Q)
[ The advantage of formulation over (10) is that Eq. (1) only depends known quantities. In the following, apply Eq. 6 I.e., the results of [Sun et al. 2019) regarding the DKI. of ergodic and consistent SPs, yielding
where ('Dif] := log Pri [D]f].
Q' E arg max inf QEQ MEX,XEX"
E DKI, (xx(((xx11Ppp
(12)
=Q().((Q)
In contrast to Sun et al. (019)) however, we do not unfold the DKL term, since we are able to directly take its functional gradient via SVGD, The resulting maximin game formulation of Eq. (12) proves to be challenging to solve, especially since we need to minimize over discrete sets X and the infimum also does not ensure a finite m. Hence, we follow Sun al. (2019) by replacing the inner minimization with sampling-based based approach, i.e.,
Q* arg max Ep, Exx~Cx (D., QEQ
(13)
where D, is a random subsample of size [D. = drawn from D. Xp, are the associated feature vectors of D.,, and XMr [x,,.... XM E a are M stacked random feature vectors drawn from sampling distribution Cx with support X. If X' is bounded, Sun et al. (2019) proposes uniform distribution for Cx. has been shown SSun al. (2019) for D. D and M that L. X.s) is lower bound for the log marginal likelihood log p (), i.e., the maximization in Eq. 13 implies the minimization in Eq. 10 Although, as noted by Buurt al.] (2020). if is parametric family, the objective is ill-defined. we did not encounter any problems in practice. Also, we could straightforwardly use the grid functional DKL proposed by Ma & Hermández-Lobato (202I) which fixes some of these theoretical shortcomings. However, note that SFVGD itself does not assume Q to be parametric.
3.2 FUNCTIONAL DERIVATIVE OF THE OBJECTIVE
When using conventional gradient descent methods, we want to apply a map to update the parameters of our model such that our loss is reduced. In SFVGD, we proceed in similar manner but update functions towards a loss minimmingg direction. A map that takes function as an argument and returns another function is called an operator. Hence, we want to express how our objective value Eq. [3chnness when an operator F H > H. f + fr is applied to every Q. This means that the objective value changes with such that
LD ( =
= Bm (D. DKL(Q P
(14)
where =f and QiTy is the distribution of F when f Q. Naturally, we are interested in the functional derivative of Eq. 14 w.r.t to F. since this gives us the direction of the steepest ascent in operator space regarding the functional ELBO. However, in order to make our computations tractable, we must limit the space of feasible operators:
5

---

Published as a conference paper at ICLR 2023
Definition 3.1 Let F -1 fF be continuous operator with the property that for all m E I and each X E xm exists a function Fx :
such that f Fx for any
He call such operator "evaluationonnly dependent".
Thus, F does not depend on derivatives of f (which is not a restriction, since we only assumed f to be continuous); state the functional gradient of the objective functional w.r.t. an evaluation-only dependent operator F, for X = -XD,, x and X XD.
also treat F construction rule of Fx for arbitrary m and X. Now, can
VrLD (Q(T)) = V Vr P Vr D
(15)
-V (D.,, VIDRL(Q(TIx P1
where we assumed that there exists a logelikel function é USEN (X x 21 x + R such that {(D, {(Da; px) for every Dz. If set F O, then T becomes the identity operator, i.e. QIzTI = Q. Since we want to iteratively update our particle functions, we must only consider small perturbations around F O.
Proposition 3.1 For an evaluation-only dependent operator F, the functional derivative of the functional ELBO F evaluated for function! f
Qx Vyl(D..)) 8y (y(1 ()..... (x ()]
x
+ Eynax Vy log log (y) (y, (x, + Ty, (16)
[6xx .)....... (x
where we assume that Hy C. {f yotM Ms+ an RKHS with associated kernels ky
The proof is given in A.1.2 where we also show the following corollary.
Corollary 3.1.1 For an evaluation-only dependent operator F, the functional derivative of the functional ELBO at F = evaluated for funcciinnn f
VFLp,x(Qm))) = Ey~ax (g((De,)) ky (x), [0xx, ( O)....,xx.1 ()] (17)
+ Ey [V log logppix (y)lix(y (y, )xx [x)x(y,
10x, ()),,,,)xx........... 0x. (:)
where we assume that Hy C. {f yot + ys J. J.Hg C {f: y + yd} are RKHSs with associated kernels by, kig. respectively.
We call FLLD, x ((Q F the Stein functional variational gradient operator. inherits its name from SVGD, which internally is used to find the functional derivative of the DKL term. The key idea of SFVGD that by updating every particle function f ~ Q via functional gradient descent in the direction of V VLLD, ((Q ( we carry out gradient step in the distribution space. This increases the current overall functional ELBO value we want to maximize by pulling Q closer to Q" and consequently also closer to the true posterior stochastic process PjIp-
3.3 ALGORITHM
Based on the particle functions fi,... fr.
Sr. we can find an estimator of Eq. 16
FFLP, (=)
V x
((D
12 x |xx, (....., 0x (9)
1=1
A
Σ
V log logpp/ (1i JK px) Vaxti pX]
+
[ox, (+) ())..., 5x. (1)
(18)

---

Published as a conference paper at ICLR 2023
Algorithm 1: Stein Functional Variational Gradient Descent Step sfvgd_step
Hyperparameters: Dataset D, log likelihood prior SP P/, number of measure points M, sampling distribution Cx over X, regularization parameter À
Input: Set of particle functions [[]]=) treated as multi-output function f Output: Input locations to update X. Stein functional variational gradient (of evallated at X) Apx
XM Cx: D. = (X, C
X
XM
for j = 1, 1,...:r do
A = 5 Di= [ (D,( (xx 1 (x.....xx,
A = : Ei=l FEx log pp,x (bx (fx" (6x .fjx) )+ Vyxxr (fix (xxxx(fixff3)) .83}]
10x, ()....,, 6x ())
end
px
(Ar + A. (X)
Algorithm 2: Stein Functional Variational Neural Network
Hyperparameters: Same as for sfvgd_ step
Input: Variational posterior g(-), optimizer opt
Output: Variational posterior g(.), which approximates the target distribution
while db not converged do
g( (X, &, p(E), i= I....,
X. px stepd_
Φ = (0,
end
where we introduce a regularization parameter À E Rz0. Furthermore, if we set À = 1, the estimator becomes an unbiased estimator of Eq. (T)). Since LD. is lower bound of the log marginal likeli- hood P(D), it would be preferable to update the particle functions via V ( E=O" However, the major computation bottleneck in Eq. I8| is the calculation of the score gradient ......... r evaluated at X. For example, if Py is GP, then the costs of computing V log (f:x) are (( M))r). In addition, the computation j = 1, ,,...,, , required in Eq. 18 costs ( How- ever, this is usually small compared to the cost of computing the score gradient for the functional prior. We choose for M a small constant number, since LD.ix,.XM is a lower bound for the log marginal likelihood log PDD) for M > 1, and we set r to a number of particle functions that can represent the posterior SP reasonably well. Thus, we are interested in estimating V x (Qr))) with
VGx log
(fix) for all particle functions fa,
of kernel values ky (f,x,, x2x), i
mini-batches. In principle, unbiased estimate of É(D, fxx²)) is n/s- f;x,, which suggests that À = s/n. Although (in general) LD, x is not a lower bound of P(D), we found in a practice setting that À to n still results in reasonable performance. However, our theoretical framework gives the reassuring guarantee that if we use full-batch training, we would, in fact, maximize a lower bound of ((D) In the following, we present two algorithms, namely Stein functional variational NNs and Stein functional variational gradient boosting (A.3.I), based on the estimated Stein functional variational gradient i.e., they depend the score gradient of the functional prior evaluated at X. If there exists no analytical score gradient, we can use a score gradient estimator, as suggested in Sun al. (2019). This only requires function samples of the prior process evaluated at X, but estimating the score gradient is usually computationally expensive (Zhou et al., 2020). Since our approach builds upon SVGD, there exists an additional approach in our framework based on gradient-fres SVGD (Han & Liu 2018) that only requires the evaluation of the marginal densities of the prior process.
Stein Functional Variational Neural Network (SFVNN) Sun et al. (019)) proposed to train neural networks (NNs) acting as function generators with the negative functional ELBO as loss, which they call Bayesian Functional Variational Neural Networks (BFVNNs). Such a function generator can be modeled via an NN with stochastic weights, which can be represented as a differentiable function g 2 + V, > g(2), where = E 2 consists of the deterministic input x and stochastic
7

---

Published as a conference paper at ICLR 2023
inputs, i.e., we can model as the reparameterization trick (Kingma & Welling 2014) can be used, i.e., there exists a random variable =\t with & p(E) and differentiable function h& X = + 2 parametrized by ø such that he (x, {) ~ p(= x). With this, we can sample function by sampling E ~ p(E) and defining fc x' D/ レ gh (x, {)) In this case, we can write the gradient of Eq. 14 w.r.t.
random variable p(2|x). These NNs applicable long
FxCD..(Q11))) BempiE)
[Vex V
Vofe"" x
-Bem)()
(VExx Pox (fxx))
VE log (fe (fx²))
of x
This is also the result obtained in Sun (2019) where they then use score estimator (namely, the spectral stein gradient estimator (SSGE; Shi al. 2018) to approximate Vy log pQx (fg ). SSGE estimates the score gradient an RKHS, i.e., the entropy gradient ViGx log pQx (Ig Hence, the entropy gradient and the cross entropy gradient Vr,x log PP
are taken in different functional (fg xxxx)
spaces. Our SFVNN is based the parameter gradient
V x ((C,) = Re)p(()
(olo,x(Qm))
Vale
-xxx=xx V log lox.
where we use the general Stein functional variational gradient from Eq. 16 In contrast to Sun et al. (2019), we thereby directly take the functional gradient of the DKL term in an RKHS, and our score gradients the prior process are also subject to the implicit kernel smoothing.
Runtime comparison between SVFNN and FVBNN While FVBNN scales as O(r + M)) (because of SSGE), our approach scales only quadratically in (because of SVGD), allowing for larger number of sample functions (see Appendix B
3.4 ILLUSTRATIVE EXAMPLE
v, = 0.5
-5
125
150
-175
0.5
3.0
2.0
25
1.0
1.5
2.0
(a) Data, sample functions
Samples from prior GP
(c) log p(v), B2, 0.5)
(d) log p(an, 0.5, yt))
3.5
15
3.5
3.5
3.5
1.0
10
3.0
05
0
03
0.5
-0.3
1.0
1.0
03
1.5
1.0
2.0
(e) Step functions fitted to gradients
(1) Updated sample functions
(g) Converged sample funetioon
(h) Samples from posterior GP
Figure/ 1: Illustrative example of SFVGB. The points in (a) represent the given data points, and the dashed lines represent all x values that detine the marginal prior density in (c.d) w.r.t. the prior GP shown in (b). The arrows in (c) show the resulting SFVGD gradients.
Given three sample functions and two data points {00.75, 1.0), (2.0, 0.0)] (a. we want to ap- proximate the posterior GP (Figure [h] w.r.t. the prior GP shown in Figure 16 and Gaussian likelihood via SFVGB. Hence, we also sample necessary measure point R' T.5. The resulting three-dimensional marginal density is defined by the prior GP. SVGD gives us the optimal update direction for the sample function values to fit this marginal density (Figures) Ic ld). We fit a kernel ridge regression these directions after adding the log likelihood gradients at the two data points (Figure Iel. The resulting updated function samples after this SFVGD step can be seen in Figure If and the converged function samples in Figure Ig Qualitatively comparing these converged sample functions in Figure Ig with sample functions from the exact posterior GP in Figure reassures that able to approximate the posterior GP in this toy example reasonably well.
8

---

Published as a conference paper at ICLR 2023
Table I: Comparison of different methods (columns) small benchmark data sets (rows) using the average NLL (smaller is better) and RMSE over train-test data splits with standard deviation in brackets. The best performing method for each data set is highlighted in bold.
Test negative logllikeldhood
Test rost--ncnnn- error
SEVNN
FVBNN
BNN
GP
SFVNN
FVENN
BNN
GP
Airfoil
2,10 (0,17)
2.29 (0.04)
2.62 (0.12)
0.50
1822 (0.20)
(0.19)
3:40 (0.40)
2.77 (0.25)
Concrete
2.99 (0.19)
3.07 (9.05)
1.25 (0.04)
3.06 (0.05)
4,58 (0.34)
4.64 (0.54)
6.18 (0.34)
5.13 (0.40)
Diabetes
5.42 (0.08)
5.49 (0.03)
5.41 (0.04)
6.19 (0.3)
54.57 (3.74)
57.1 (2.48)
(2.88)
57.45 (6.6)
Energy
0.62 (0.10)
0.70 (0.09)
2.26 (0.32)
2.38 (0.05)
0.44 (0.05)
0.43 (0.08)
2.37 (0.65)
2.34 (0.23)
Forestl:
(0.44)
(0.05)
183 (0.05)
(0.31)
1.51 (0.07)
1.51
1.56 (0.08)
Wine
1.96 (1.45)
1.47 (1.07)
-0.33 (0.07)
-0.04 (0.06)
0.11 (0.02)
0.14 (0.02)
0.21 (0.03)
0.16 (0.03)
(0.30)
(0.24)
1.35 (0.19)
(0.15)
(0.26)
(0.25)
0.96 (0.28)
3.95
Mean rank
1.336"
2.45
2.57
3.14
1,36
T.79
33,77
3:25
Table 2: Comparison different methods (columns) on large benchmark data sets (rows) using the average NLL (smaller is better) and RMSE over 10 train-test data splits with standard deviation in brackets. The best performing method for each data set is highlighted in bold.
Test negative log- log-likelthood
Test root-mean--suuar
SPVNN
FVBNN
BNN
SPVNN
FVBNN
BNN
(0.04)
4.50 (0.03)
(0.02)
(1.26)
(0.9)
(0.67)
NavalT
-6.91 (0.06)
-6.85 (0.08)
-5.03 (0.24)
1.70E.4
19.4-4
6.50E-4
(6.5E-5)
NavalC
-6.53 (0.01)
-6.41 (0.05)
-6.44 (0.11)
1.55--(
1.35E-
2.39B-4
(3.5E-5)
2.15E-4 (3.9E-5)
Protein
2.85 (0.01)
2,87 (0.01)
2.96 (0.01)
4.19 (0.04)
4.27 (0.04)
4.65 (0.05)
VideoMem
11.39 (0.39)
11.3 (0.10)
(0.03
21119 (4910)
20800 (2320)
21800 (788)
VideoTTim
(0.05)
2.53 (0.36)
2.51 (0.16)
(2.08)
Mcan ranik
125
2,77"
2.58
C33
2.17
2.30
BENCHMARK STUDY
We further investigate the competitiveness of our approach using neural network variant (SFVNN) with its closest neighbor, the functional variational Bayesian neural network (FVBNN) from Sun jet al. (2019). We also include one well-established BÑN baseline (Blundell et al. 2015) and the standard Gaussian Process for the small data sets (where analytical computation is feasible). For results from SVFGB we refer to Section AA3.2 in the Appendix. For most of the datasets, however, the NN provides better fit to the data. Further details and contextual bandits experiment can be found in Appendix A.5
Data and experimental setup All data sets are standardized prior to model fitting and split into 90% training data and 10% test data. For the comparisons, this splitting process is repeated 10 times based on 10 different splits to also evaluate the variability of each method. Further details on data sets, data set-specific pre- pre-processing, and their references can be found in the Appendix.
Details methods and comparisons In order to provide a fair comparison between methods, reproduce the best results reported by [Snn et al. (2019) for FVBNN and BNN, and also use the same hyperparameters for our method except that while Sun al. (201)) use À 1, we set À to 8/ //n. Details for each procedure are given in the Appendix. We compare methods based on the negative log-licelhhodd (NLL) and the root mean squared error (RMSE) on each test data set and calculate the mean and standard deviation across all 10 data splits.
Results Results are summarized in Tables and indicating that SFVNN is competitive with other existing approaches for both small and large data sets. As the two functional approaches (SFVNN, FVBNN) optimize the same objective, we would expect them to perform similarly, which is confirmed by the results. We further observe that a weight space approach (the BNN) seems to work better than the functional approaches for a few datasets (in particular, for the Wine dataset, where this is expected as the outcome is of discrete nature).
5 CONCLUSION
We introduced novel gradient descent in distribution space that allows us update set of particle functions in a general manner to resemble sample functions from a target process. SFVNN was found to be competitive with to outperform FBVNN while having less computational costs.

---

Published as a conference paper at ICLR 2023
REFERENCES
Rafael Ballester-Ripol Enrique G. Paredes, and Renato Pajarola. Sobol tensor trains for global sensitivity analysis, 2017.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, mlr
pp. 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. URL html
Peter Buehlmann. Boosting for high-dimensional linear models. The Annals of Statistics, 34(2): 559-583, 2006.
David R. Burt, Sebastian W. Ober, Adrià Garriga-Alonsee and Mark van der Wilk. Understanding vari- ational inference in function-space, 2020. URL https:://t.xxit..ee org/ abs 2011. 09421
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16, pp. 785-794 New York, NY, USA, 2016. ACM. ISBN 978 1-4403-423222-22 doi: URL http:: acm. 1045 10t.:1115/229933 2939672 2939777
10.1145/993972..9997777
Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo, 2014.
URL https::/arxiv..om
abs/1402.
Andrea Coraddu, Luca Oneto, Alessandro Ghio, Stefano Savio, Davide Anguita, and Massimo Figari. Machine learning approaches for improving condition?based maintenance of naval propulsion plants. Journal of Engineering for the Maritime Environment, -():-, 2014.
Paulo Cortez and Anibal de Jesus Raimundo Morais. A data mining approach to predict forest fires using meteorological data. 2007.
Francesco D) Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian, 2021, URL
https://axxiv.om 2210 11642
Francesco D' D'Angelo, Vincent Fortuin, and Florian Wenzel. On stein variational neural network /ttp:///aaiivv.v.a wwwpii.or/was//////oop/p.. 2tt.://2.......0
ensembles, 2021. (tttss/::://an
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://acchvve ics www.udu/mmmmmmmmm
Tony Duan, Anand Avati, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Y. Ng, and Alejandro Schuler. Ngboost: Natural gradient boosting for probabilistic prediction, 2019. URL https
7/ warxiv.org/aba.. /1910.11910000 03225
Jerome H. Friedman. Greedy function approximation: gradient boosting machine. The Annals of Statistics, 29(5): 1232, 2001. doi: 00.1214/a0s/001222322 URL httph:ttps:/ttt org/ 10. 10.11114/ao//001225333
Jun Han and Qiang Liu. Stein variational gradient descent without gradient. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1900-1908. PMLR, 10-15 Jul 2018. URL htttpp:://pprccccinsses
mlr wwt.mm.eee..///aaaaaaa.a html
Jose Miguel Hernandez-Lobate and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 1861-1869, Lille, France, 07-09 Jul 2015. PMLR. URL/sttp::/prcceeee mlr. hul=:ppees///3//eerreeeeeeeo
Alexander Immer, Macicj Korzepa, and Matthias Bauer. predicion of bayesian neural 08400
nets via local linearization, 2020. TRL/httss://arae
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1 14, 2014.
10

---

Published as a conference paper at ICLR 2023
John Lamperti. Stochastic processes. a survey of the mathematical theory J. Lamperti. Applied mathematical sciences (Springer-Veree New York Inc.); v. 23. Springer- Verlag, New York, 1977.
ISBN 0387902759.
Alex Leviyev, Joshua Chen, Yifei Wang, Omar Ghattas, and Aaron Zimmerman. A stochastic stein variational newton method, 2022. htLtttts:///axxt org/ abs /2044 09039
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 9781510838819.
Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN
Chao Ma and José Miguel Hernández-Lobato. Functional variational inference based on stochas- tic process generators. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21795-21807. Curran Associates, Inc., 2021. URL /Rtp.htsssss///prcceciisgssss neu.uus....erri...ommmmmmmmmm cc/ pdf
Andrey Malinin, Liudmila Prokhorenkova, and Aleksei Ustimenko. Uncertainty in gradient boosting via ensembles. International Conference on Learning Representations, 2021. URL https
net //tppenreeyeeennne/e/oeomrrmoceo
Alexander G. Matthews, James Hensman, Richard E. Turner, and Zoubin Ghahramani. On sparse variational methods and the kullback-leibler divergence between stochastic processes, 2015.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown. In Interna- tional Conference on Learning Representations, 2018.
Tim G. J. Rudner, Zonghao Chen, Yee Whye Teh, and Yarin Gal. Tractable function-space variational inference in bayesian neural networks. 2021.
Mark J. Schervish. Theory of Statistics. Springer New York, 1995. doi: 11007/9778-1-6612-455-55
URL httpp::://....ogg/////// wttp:///.1.11//11111 161222----------- -112-4250-5555
Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions, 2018.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks, 2019. URL htttes///arxiv..oo////////1/ /tt///993300779
http:/bb/119000
Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and buildings, 49:560-567, 2012.
Dilin Wang, Ziyang Tang, Chandrajit Bajaj, and Qiang Liu. Stein variational gradient descent with matrix-valued kernels, 2019a. URL/https://aae http::////xxxxxxx////aoaa// 119...//1911.1.999
Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for bayesian neural networks, 2019b.
I-C Yeh. Modeling strength of hirg--eeformancce concrete using artificial neural networks. Cement and Concrete research, 18(12):1977-1880 1998.
Yuhao Zhou, Jiaxin Shi, and Jun Zhu. Nonparametric Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11513-11522. PMLR, 13-18 Jul 2020. URL
estimators. In Hal Daumé III and Aarti
mlr www.sppssssse........ (ttess/vv19//hhoor zho .tmll www.:mer....eemmemmmmmmmm
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOFS
A.1.1 FUNCTIONAL DERIVATIVE OF THE RISK FUNCTIONAL
Assuming that p(x, y) (xx.) [f] E L' using Fubini's theorem, we find that
Ri = E ((xx)) 1xx0)111= Sr
xxx p(x, u) Lo [[dx dy
(19)
= S p(x) / ply/)x) dx.
(20)
-(x,)((())
Assuming that L is sufficiently smooth using the Fuler- derivative, we find that
V V)R[f((x) = ① (([(x,((x)))
(21)
0f(x)
= -((x) / plulx)
(l 0f(x)
dy
(22)
0Lixvi]f]
=((x) BayxPll 0f(x)
(23)
A.1.2 FUNCTIONAL DERIVATIVE OF THE FUNCTIONAL ELBO
Let F be operator that depends on evaluation only with associated maps Fx 3 - ) Fx
yetM ye+ad, Further, let F[F] = LD, (Qrx RirQurx ((D..)) DKI (QTT 11xx
=F
=F,[Fx]
In general, and under the assumption that é is sufficiently smooth, we find that
F - F ( +x = By-Qx Vy -y-)) (24)
Under the assumption that Hy {f 1dth ym is RKHS with associated kernels ky. find that
File& + -
= Ex~ex (00-:-----000---00000-0--------- (25)
= E Bg- T
(26)
E BBnnQx (y(0,,55-43()))0((()))))))---1- (27)
= (By~Ox
(4(10,55--2(())4466,)),----0---0 (28)
from which it follows that
Fi P2
Eÿ-Qz [V(((D.. + Fx())) 1y )] Px
(29)
= EBnax (v((DD.))) ky (}.))
(30)
Under the assumption that Hy C {f Ps yot M is an RKHS with associated kernels by, has been shown in Liu & Wang (016) that
V F2 [x Fxx -Ex-ex [vy (V)PP)/x())xx()) +Vyrry))
(31)
Using the chain rule, we obtain
V (0) V V Fx
(32)
F=D
V (Fx] F=D
(33)
(34)
12

---

Published as a conference paper at ICLR 2023
Algorithm 3: Stein Functional Variational Gradient Boosting
Hyperparameters: Same for sÊvgd step
Input: number of iterations fmax, learning rate n/ in the t-th iteration, set of initial partiele functions {S,
( Jim1 treated as multi-output function ,l multi-output base learner Tb
Output: Set of particle functions {S{ [mmalyy Jizi, which approximate the target distribution
for = .,... fmax 1 do
X, [+|] fld n1 Ix 4xx)
x = 8 sfvgd_ste (f
end
Since F is evaluation-only dependent, we observe that VF [Fxx]] (/) = 6x(-) and conclude that
Vrlo,x(9m))==000)
xx=0 |xx, (.),..., ,
5x, ()]
(35)
Fo =x [8x, (-),...,, 8x, x.]]
(36)
If H is assumed to be RKHS with associated kernel kx(x,) .) then this becomes
V VFCD, ( = Fi
Px=U (X1( Ax (X.:
(37)
Vx FFx
K...,,xxiiaa kx(Xo+M,
(38)
xx=0
A.2 GRADIENT BOOSTING
GB (Friedman 2001) is powerful supervised learning algorithm where, iteratively, the residuals are minimized via so-called weak learners. The resulting model consists of a weighted ensemble of these weak learners. In its original form, tree-based learners were used as weak learners, which proved to be effective, especially the presence heterogeneous features. XGBoost (Chen & Guestrin (016) a highly efficient algorithm that builds upon the GB paradigm, which proves to be strong baseline for many structured, supervised regression and classification tasks.
A.3 SFVGB
A.3.1 SFVGB ALGORITHM
We treat the particle functions mapping from X' to ) single function flu mapping from X' to y" - i.e. we identify the i-th sample function f! 1,...,, with the i-th component of f Analogously to standard GB, we choose a base learner I, which defines hypothesis space H C [ always possible to use an ensemble single-output learners. With this, SFVGB vanilla GB of a multi-output function where the loss is the negative functional ELBO. Consequently, we update f(dl in the t-th iteration via
+ }" While this requires the base learner I, to be multi-output learner, it is
f(++1] f( + n II X, PFCD, (Q
VFLD, (=)
using Eq. 18 and (potentially adaptive) learning rate 1/(9)
A.3.2 SFVGB REGRESSION EXPERIMENTS
We also test SFVGB on the small regression datasets and compare 1) the approach proposed in Malinin et al. (2021). i.e., uncertainty quantification approaches by randomly subsampling the data in every iteration
stochastic gradient boosting (SGB) model and 2) boosting generalized linear model (GLMB Buehlmann] 2006) as a baseline approach. As base learners, SGB uses trees and GLMB uses linear models. Our model uses a Nadaraya- Wataoon kernel regression variant, which does not scale the resulting assumed to be an RHKS, as discussed in aection[.1] The hyperparameters of the SFVGD step are the same the ones used for SFVNN. The results after 1000 iterations summarized in Table
of kernel functions. This is the natural learner if the hypothesis space H is
13

---

Published as a conference paper at ICLR 2023
Table Comparison of boosting approaches the small benchmark data sets (columns) using the average NLL (smaller better) over train-test data splits with standard deviation brackets. The best performing method for each data set is highlighted in bold.
Airfoil
Concrete
Diabetes
Energy
ForestF
Wine
Yachi
SFVGB
2.87 (0.18)
3.41 (0.13)
7.04 (0.67)
2.16 (0.10)
6.31 (0.69)
2.37 (2.00)
3.21 (0.52)
GLMB
3.10 (0.03)
3,93 (0.03)
5.50 (0.03)
3.28 (0.01)
1.84 (0.07)
0.72 (0.02)
3.80 (0.04)
SGB
1.98 (0.05)
3.06 (0.10)
5.51 (0.07)
0.79 (0.49)
1.86 (0.08)
0.11 (0.44)
0.36 (0.23)
Table 4: Comparison of boosting approaches on the small henchmark data sets (columns) using the average RMSE (smaller is better) over train-test data splits with standard deviation in brackets. The best performing method for cach data set highlighted in bold.
Airfoil
Concrete
Diabetes
Energy
ForestF
Wine
Yachi
SFVGB
3.31 (0.26)
6.71 (0.52)
57.8 (0.67)
1.93 (0.14)
1.53 (0.07)
0.18 (0.04)
4,83 (1.24)
GLMB
4.85 (0.27)
10.5 (1.03)
53.4
3.07 (0.21)
1.51 (0.09)
0.27 (0.04)
8,51 (1.18)
SGB
2.06 (0.21)
5.06 (0.53)
(2.71)
0.57 (0.09)
1.52 (0.09)
0.29 (0.14)
0.85 (0.45)
Further experimental details Model tuning of SGB is done as explained in Malinin et al. (2021). Here, different tree depths E (3,4, 4,4,6,,) 6), learning rates (0.001, 0.01, 0.1). and numbers of samples {0.25, 0.5, 3.75) of approaches are trained on the first 80% of the training data and evaluated on the latter 20%. The GLMB approach is tuned using 10-fold cross-validation to determine the number of stopping iterations, which is the only hyperparameter of the model.
Results Results show that the SFVGB can often improve over the GLMB baseline but still yields inferior performance on some data sets. This is, in particular, the case if there is a rather discrete outcome space (e.g., Wine which only consists of values O, 1, and 2). The SGB model, in turn, works better than both the SFVGB and GLMB in most cases. This is likely due to the much more flexible base learner structure (as SGB like most of the state-of-the-an boosting approaches uses trees, whereas GLMB uses linear regression and ours uses kernel regression).
A.4 NUMERICAL EXPERIMENTS: FURTHER DETAILS
A.4.1 FURTHER DATA DETAILS
We use the benchmark data setup proposed by Hernandezz-Loato & Adams (2015) and [un al. (2019) for evaluating probabilistic regression approaches. This setup includes selected data sets from the UCI repository namely, the four smaller data sets Concrete, Energy, Wine, Yacht, and the four larger data sets Naval, Protein, Video (Memory and Time), and GPU. In addition to Sun et al. (2019), we also compare our approaches on three additional smaller data sets (Airfoil, Diabetes, Forest Fire) and further investigate the second task on the Naval data set (i.e., examine both the compressor decay and the turbine decay state coefficient, referred to as NavalC and NavalT, respectively). In Table the data characteristics and pre-processing steps are listed.
A.4.2 FURTHER EXPERIMENTAL. DETAILS
BNN approaches fitted using the recommended architecture and tuning parameters by Sun et al.) 2019). We reduced the epochs from 10,000 to 1,000 epochs for the smaller data sets to reduce the computational runtime. This did not negatively impact the BNN's performance. In all our benchmark experiments, we follow the setup for BNNs and FVBNNs described by Sun et al. (2019).
A.5 FURTHER RESULTS
Here, we provide further results using the application of probabilistic methods for contextual bandits.
One important application of uncertainty- aware models is for exploration, as in Bayesian optimization, reinforcement learning, bandits. Following Sun et al.) (2019), evaluate SFVNN using the contextual bandits benchmark by Riquelme et al. (2018) by re-runing the settings investigated in (Sun al. (2019) and report the cumulative regret based on the best expected
Contextual Bandits
14

---

Published as a conference paper at ICLR 2023
Table 5: Data characteristics, additional pre-processing and references.
Dataset
# Obs.
# Feat.
Pre-processing
Reference
Airfoil
1503
5
Dua & Graff (2017)
Concrete
1030
8
Yeh 1998
Diabetes
442
Dua Grall (2017
Energy
768
Isanas & Xitara 201
ForestF
517
12
logpl transformation for area; numerical representation for month and day
Cortez & Morais (007
Wine
178
Dua & Grali
(2017
Yacht
308
Dua & Grall
(001)
GPU
241600
14
outcome drop features with variance
only
Ballester- Ripoll ct al.
(2017)
NavalT
11934
Coraddu cl
2014
NavalC
11934
drop features with zero variance
Coraddu el al.
2014
Protein
45730
9
Dua & Graif
2017
Video
68784
19
drop highly correlated features: drop id and b size; dummy-coding for codec and o_codec
Dua & Grail
(2017)
Table 6: Relative contextual bandits regret (relative the cumulative regret of Uniform sampling) for different data sets (columms) and methods (rows). Numbers brackets of methods indicate the network sizes. Reported numbers are the mean (and standard derivation in brackets) over trials. The best algorithms per data set are highlighted in bold.
Adult
Census
Covertype
Jester
Mushroom
Statlog
Wheel
BNN (50)
95.79 (1.33)
66. (5.10)
61.18(1.9)
83.70 (3.31)
7.81 (9.39)
29.2 (4.05)
82.94 (22.2)
BNN (500)
(7.32)
(11.1)
(13.3)
BonRMS
82.74 (8.16)
53.40
36.95 (10.3)
63.3 (10.74)
7.11 (5.61)
1.81 (2.70)
119.16 (3.7)
Dropout
85.46 (4.74)
37.27 (12.3)
39.55 (5.84)
65.11 (9.62)
4.43 (6.83)
2,89 (4.26)
28.61 (12.1)
FVBNN (50, 50)
72.84 (11.5)
30.46 (28.4)
24.29 (14.7)
50.45 (20.0)
(10.3)
3.46 (3.75)
13.62
FVENN (50)
75.05 (7.66)
40.65 (14.9)
46.57 (3.99)
57.28 (17.8)
2.39 (6.24)
1.58 (2.09)
24.24 (22.0)
Param
89.00
86
48.18
66.52 (13.1)
6.74 (5.89)
7.69 (3:57)
21.93
SEVNN (50, 50, 50)
71.99 (7.29)
36.65 (30.3)
28.47 (16.4)
50.09 (21.2)
7.81 (9.35)
4.52 (4.12)
44.12 (27.7)
SEVNN (50)
79.62 (5.44)
(11.4)
(18.0)
(7.99)
4.01 (2.19)
(33.2)
Uniform Sampling
100.0 (0.00)
100.0 (0.00)
100.0 (0.00)
100.0 (0.00)
100.0 (0.00)
100.0
100.0 (0.00)
reward. Following Riquelme et al. (2018), we use the benchmark data sets Adult, Census, Covertype, Financial, Jester, Mushroom, Statlog, and Wheel. For these, the rewards are deterministic, and the regret is equal to the best realized reward. As in previous works, we report the regret as a relative value, relative to a random uniform sampling procedure that emulates Thompson Sampling (see Riquelme et al. 2018). As comparison methods, we use the BNN (with 50 and 500 units), three spinnoffs of the NeuralLinear algorithm (namely, the Bootstrapped NN trained with RMSprop (BootRMS), Parameter Noise (ParamNoise), and Dropout (see Riquelme et al. 2018, for more details)), well as two variants of the FVBNN (with one and three layers each with units). Experiments are run times with shuffled contexts, for which we report mean and standard deviation of the relative cumulative regret.
Results are given in Table suggesting that both FVBNN and SFVBNN are well- and particularly similar-performing methods in the application of contextual bandits.
B RUNTIME EXPERIMENT
B. SETUP
We trained FVBNN and SVENN times on the Energy data set for each number of particle functions (50, 100, 150} and plotted the resulting runtimes in Figure 2 It becomes apparent that the number particle functions influences the runtime of FVBNN more substantially than SFVNN, as expect from computational complexity analysis.
15

---

Published as a conference paper at ICLR 2023
400
runtime [seconds]
Method
FVBNN
300
SFVNN
200
number of particle functions
Comparision of the runtimes of FVBNN and SVENN on the Energy data set with 5 repetitions for cach number of particle functions
Figure 2:
B.2 COMPUTATIONAL ENVIRONMENT
All experiments and benchmarks carried out internal cluster with Intel(R) Xeon(R) CPU E5-2655 v2 @ 2.60GHz, 32 cores, 64 GB Random-access memory, and operating system Ubuntu 20.04.1 LTS.
16


Reviews:
[{"review_id": "sLJZQgYDIv", "reviewer": "Reviewer_9cu5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study general variational algorithms, which generalize Stein variational gradient descent (SVGD) in functional space. They generalize the functional space in minimizing the KL divergence step. They apply the algorithms in Bayesian neural networks and ensemble gradient boosting. The numerical experiments demonstrate the efficiency of their algorithms.  ", "summary_of_the_review": "This paper provides a method to approximate the Stein variational gradient in general settings. However, there is no analytical or motivation examples to support the proposed method.  ", "strength_and_weaknesses": "Weakness: Please do not use blue and red colors on the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rA8aSoCN4m7", "reviewer": "Reviewer_bTr3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to perform Stein Variational Gradient Descent directly in the function space of neural networks or gradient-boosted models, thus offering two new methods. It derives theoretically how these approaches can be approximated in practice and shows empirically how they perform on simple benchmark regression tasks.", "summary_of_the_review": "Overall, the method seems theoretically well-motivated, but the weak experiments and insufficient discussion of related work currently hinder me from recommending acceptance. If the authors could provide stronger experiments with relevant baselines on realistic data, I would be willing to change my assessment.\n\nUPDATE: I have increased my score thanks to the changes made during the rebuttal.", "strength_and_weaknesses": "Strengths:\n- The problem of function-space inference is well-motivated.\n- The paper is clearly written.\n- The theoretical derivations seem to be correct.\n\nWeaknesses:\n- The related work is incompletely acknowledged.\n- The experiments seem rather inconclusive.\n\nComments:\n- The proposed SFVNN seems (at least superficially) very similar to [1] and to some lesser degree to [2]. Other functional BNN approaches that seem worth mentioning are [3] and [4].\n- The paper relies a lot on the KL divergence estimator from Sun et al. However, [5] has shown that the KL divergence is generally infinite between many different function-space distributions and that the estimator is thus ill-defined. Could the authors comment on that?\n- In the presented regression experiments, the proposed method only outperforms the baselines on a few datasets. Could the authors comment on why it doesn't work well on the others?\n- Could the authors comment on the runtime of the proposed method? I would naively think that it would probably be slower than the baselines.\n- Since NNs are not necessarily much better on these regression tasks than, e.g., GPs, I think an image classification task would be a more suited experiment to motivate the method, similar to e.g. [1,2].\n- Since [2] have shown that their method works better than the functional SVGD [1], it seems like that might also outperform the proposed method and should be a baseline in the experiment.\n- The notation is sometimes hard to parse (up to 4 stacked subscripts), so maybe that could be made lighter.\n\n[1] https://arxiv.org/abs/2106.10760\n\n[2] https://arxiv.org/abs/2106.11642\n\n[3] https://arxiv.org/abs/2008.08400\n\n[4] https://hudsonchen.github.io/papers/Tractable_Function_Space_Variational_Inference_in_Bayesian_Neural_Networks.pdf\n\n[5] https://arxiv.org/abs/2011.09421", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aJc2qX6oiD", "reviewer": "Reviewer_qRMB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies function-space BNN inference, and formulated function-space variants of the Stein variational gradient descent algorithm.  The algorithm is evaluated on synthetic and tabular regression data.", "summary_of_the_review": "While this work tackles an important problem and appears to have interesting contributions, the clarity issues make it difficult to judge its merits.\n\n(My ratings below reflect the uncertainty due to the clarity issues, and I'm willing to update them after clarifications.)", "strength_and_weaknesses": "### Strengths\n\nThis work attempts to provide a rigorous formulation of function-space, SVGD-like algorithms.  This is a sensible problem, as similar particle-based VI approaches have shown promises in the past, including for BNN inference where their function-space analogues have been studied in less rigorous ways.  A rigorous formulation would allow the principled use of such algorithms, and may also identify suboptimal constructions in past algorithms.\n\n### Weaknesses\n\nMy main concerns are the following:\n\n1. The current manuscript did not discuss previous work adequately.  It is not obvious from a first reading what efforts have been made to construct function-space SVGD algorithms: this work only cited Wang et al (2019), and neglected the works of D'Angelo et al (2021a; 2021b) which studied similar issues.  Moreover, there is no comparison between the update rules derived in this work and those in past works; and there is no empirical comparison either.  This leaves it very unclear how new derivations in this work have practical implications.\n\n2. The presentation of the technical contents is quite confusing, which makes it difficult to understand or verify the claims.  Gradients were presented without specifying the inner product structure they are defined with.  The same gradient notation can be used to refer to both L2 and RKHS gradients (e.g., Eq. 3 and the discussion below).  Typos such as in Eq. 8 further complicated reading (Eq. 8 refers to the $\\mathcal{H}$-Wasserstein gradient in a space of distribution over parameters, not \"functions\" which do not have to exist there).  It takes a lot of guesswork -- and familiarity with the background -- to understand what the authors really meant, and it should not have happened in a work that attempts to present a rigorous formulation for ideas that already exist (in part, and in various forms) in the past.\n\nThese two issues make it difficult to evaluate the manuscript in its present form, although I'm willing to go through it again once they are clarified.\n\n3. A less important issue, which should nonetheless be made clear, is that all discussions in this work only applies to the full-batch training setting, where the sampled \"measurement points\" (Sun et al, 2019) always include the entire training set.  This is an inherent limitation (Burt et al, 2020) of the framework of Sun et al (2019), and the readers should be made aware about it.\n\n### References\n\n* Wang, Z., Ren, T., Zhu, J., & Zhang, B. (2018). Function Space Particle Optimization for Bayesian Neural Networks. In International Conference on Learning Representations.\n* D'Angelo, Francesco, and Vincent Fortuin. \"Repulsive deep ensembles are bayesian.\" Advances in Neural Information Processing Systems 34 (2021): 3451-3465.\n* D'Angelo, Francesco, Vincent Fortuin, and Florian Wenzel. \"On stein variational neural network ensembles.\" arXiv preprint arXiv:2106.10760 (2021).\n* Burt, D. R., Ober, S. W., Garriga-Alonso, A., & van der Wilk, M. (2020). Understanding variational inference in function-space. arXiv preprint arXiv:2011.09421.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N20hko5mA1P", "reviewer": "Reviewer_NXDp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to perform approximate Bayesian inference in function spaces with an emphasis on applications to Bayesian neural networks and gradient boosting. The proposed approach extends Stein variational gradient descent (Liu & Wang, 2016) to function spaces by applying it to minimise the KL divergence with respect to a posterior stochastic process. The approach is made tractable by approximations propagating the functional gradients to the parameter space of the models being learnt. Experiments learning BNNs and gradient boosting models on benchmark datasets are presented showing performance improvements with respect to baselines.", "summary_of_the_review": "The paper's contributions seem to be mostly on the formulation of a new method with theoretical support, but with only marginal practical performance gains in experimental evaluations, which cast doubts regarding its potential impact.", "strength_and_weaknesses": "### Strengths\n* Compared to previous work (Sun et al. 2019; Wang et al., 2019), the proposed approach more naturally translates SVGD to function spaces by starting its derivation from functional gradients.\n* Theoretical results are provided on the validity of the functional gradient estimators.\n* Experimental results show performance improvements against previous function-space variational approach FVBNN.\n\n### Weaknesses\n* Related work on SVGD could mention some recent work on other forms of SVGD, such as second-order methods, matrix-valued kernels, and its convergence analysis.\n* Related work on BNNs only mentions variational inference approaches, while Markov chain Monte Carlo methods, such as stochastic gradient Hamiltonian Monte Carlo, have also shown some success on inference for BNNs, though usually in problems of low dimensionality.\n* Background on gradient boosting could be expanded.\n* What is most concerning to me is that practical performance gains seem marginal when considering the results in Table 1.They are mostly within the +/- 1 std. deviation margin.\n* There is no discussion on the results in Table 1. For example, why did the BNN (using Bayes by backprop) baseline perform significantly better than the functional-gradient methods in a few of the benchmarks? And importantly, what would be a few reasons for the performance of the proposed SFVNN and the FVBNN baseline to be so similar in most cases?\n* In the appendix, there were no baselines for the comparisons with SFVGB.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster