Convergence Aspects of Hybrid Kernel SVGD
Authors: Anson MacDonald
Scott A Sisson
Sahani Pathiraja
Venue: Accepted by TMLR ()

Abstract: Stein variational gradient descent (SVGD) is a particle-based approximate inference algorithm. Many variants of SVGD have been proposed in recent years, including the hybrid kernel variant (h-SVGD), which has demonstrated promising results on image classification with deep neural network ensembles. By framing h-SVGD as a kernelised Wasserstein gradient flow on a functional that is not the Kullback-Leibler divergence, we demonstrate that h-SVGD does not converge to the target distribution in the mean field limit. Despite this theoretical result, we provide intuition and experimental support for the ability of h-SVGD to improve variance estimation in high dimensions. Unlike other SVGD variants that also alleviate variance collapse, this is achieved at no additional computational cost and without further assumptions on the posterior.

Published in Transactions on Machine Learning Research (12/2025)
Convergence Aspects of Hybrid Kernel SVGD
Anson MacDonald
School of Mathematics and Statistics Uniuersity of New South Wales
Scott A. Sisson
sisto.sison sisson&unsuu edu.
School of Mathematics and Statistics
University of New South Wales
Sahani Pathiraja
8. Cachiraja@uussvee edu.
School of Mathematics and Statistics
University of New South Wales
Reviewed OpenReview: // openreview. net/ forum? td= J=2bkk55)
https://openrevieu
Abstract
Stein variational gradient descent (SVGD) is particle-based approximate inference algo- rithm. Many variants of SVGD have been proposed in recent years, including the hybrid kernel variant (b-SVGD), which has demonstrated promising results image classification with deep neural network ensembles. By framing h-SVGD as kernelised Wasserstein gra- dient flow on a functional that is not the Kullback-Leibler divergence, we demonstrate that h-SVGD does not converge to the target distribution in the mean field limit. Despite this thcoretical result, we provide intuition and experimental support for the ability of h-SVGD to improve variance estimation in high dimensions. Unlike other SVGD variants that also alleviate variance collapse, this is achieved at no additional computational cost and without further assumptions on the posterior.
1 Introduction
Stein variational gradient descent (SVGD) is a variational inference algorithm that generates samples from a target probability density (Liu & Wang 2016). It has proven useful many lasks Bayesian inference and machine learning. SVGD evolves an interacting particle system until the particles resemble sample from target density. The dynamics of this system include driving term that moves particles to regions of high probability, and repulsive term that repels particles from one another. This repulsive term prevents particles from converging to the same mode. Tt has been shown that within a unit ball of reproducing kernel Hilbert space (RKHS), the SVGD update direction optimally reduces the Kullback-Leibler (KL) divergence between the target density and the approximating density (Liu & Wang, 2016). The reproducing kernel of this RKHS appears in both the driving and repulsive terms, making the choice of kernel a key ingredient for SVGD.
The theoretical properties of vanilla SVGD have been studied extensively. Liu showed that the empirical measure of the particles converges weakly to the target distribution (Li 2017) SVGD in the mean field regime has been described as a gradient flow on the KL divergence (Liu & Wang. 2016 and the chi-squared divergence (Chewi el all, 2020). Furthering this geometric point of view, Duncan el all (2026) developed the Stein geometry along with its associated tangent spaces and geodesics, leading to guidelines for choosing kernels to improve convergence. The existence and uniqueness of the solution to the Stein partial differential
equation (PDE) has been established (La et all 201) along with various descent lemmas bounding the
decrease in KT divergence at each iteration (Li.. 2010: Korba et. all 2020, Salim el all 2022)

---

Published in Transactions on Machine Learning Research (12/2025)
SVGD is known to suffer from the curse of dimensionality through variance collapse (Wang et all 2018: Ba all 20T whereby the marginal variances of the particles underestimate the true marginal variances of the target in high dimensions. Zhno al (2 explained that this phenomenon is due to the size of the repulsive term of the update direction scaling inversely with dimension. This enables the driving term to dominate in high dimensions, thereby forcing particles to converge to the mode(s) of the target. This insight suggests that strengthening the repulsive term in SVGD should lead to better variance estimation, an idea which we explore in Sections @ and B
Many variants of SVGD have also been proposed in recent years, some offering improvements and others providing generalisations. Riemannian SVGD & 2018) generalises SVGD by allowing for target densities on Riemannian manifolds, not just Euclidean spaces. Matrix SVGD (Wang et all 201M) replaces the scalar valued kernel with a matrix valued kernel to incorporate preconditioning information and speed up particle exploration. Message passing SVGD (Zhno ef all, 21TS and graphical SVGD (Wang et all, 2018) focus on target densitics that factorise according
graph structure. This approach reduces variance collapse in high dimensions by converting the problem to collection of low dimensional problems. Projected SVGD (Chen & Ghattas 2IZI), sliced SVGD (Gong all (0z4) and Grassman SVGD (Lan all (IZ)) also mitigate the issue of variance collapse by updating particles within lower dimensional subspaces, which comes at the expense of additional computation.
In this work, we study a variant called hybrid kernel Stein variational gradient descent (h-SVGD). The name comes from its use of two distinct kernels for the driving and repulsive terms with the aim of mitigating variance collapse. This variant was originally proposed by D'Angelo et all (202T) in the context of training deep neural network ensembles by sampling from the distribution of network parameters. In that setting, two particles may parameterise networks with very similar outputs despite being far apart in the weight space. Their insight was to encourage functional diversity between networks in the ensemble by using a standard kernel in the driving term, but a functional kernel in the repulsive term. In this neural network ensemble sctting, h-SVGD demonstrated better performance than other variants on image classification. Anncaled SVGD (D'Angelo & Fortuin, 2021) may also be considered an example of h-SVGD. In this variant, the driving kernel is a scalar multiple 7() E [o, 1] of the repulsive kernel, and this factor 7(E) gradually increases to 1 as the iteration l increases. Numerical experiments show that anncaled SVGD improves the ability of particles to escape local modes. Scaling one of the update terms has also been used computational technique to aid other SVGD variants when training Bayesian neural networks (Gong et al, 20211).
Although preliminary numerical experiments have shown the benefits of h-SVGD (D'Angelo et all 211 the theoretical results for SVGD do not directly apply in the hybrid kernel setting due to the presence of a second kernel. In this paper, we address this theoretical gap and reinforce the practical benefits of h-SVGD through the following contributions.
We establish existence of solution to the hybrid Stein PDE and a kernelised Wasserstein gradient flow interpretation. Through the study of this gradient flow, we demonstrate that h-SVGD does not converge to the target distribution in the mean field limit.
We also quantify the rate of dissipation of the gradient flow functional and develop a discrete time version of this result, otherwise known as descent lemma.
Despite not converging to the target distribution, we demonstrate through numerical experiments that h-SVGD can mitigate variance collapse in the finite particle regime at negligible additional cost, whilst remaining competitive at high dimensional inference tasks.
In Section we clarify notation, recall necessary theory, and outline the vanilla and hybrid SVGD algorithms. Section contains the theoretical contributions, with proofs relegated to Appendix Al Numerical experiments are in Section H with additional experiments and details in Appendix B
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Background
2.1 Notation
Let X' RR. Let т denote the target probability density on X' and let 8 (x = Va log (2). We will often write π for the corresponding measure. Assume that T(x) = e v-((m) for some potential V. Let P(x) be the set of probability measures on A' and let (x)) denote the subset where || := each p 2 1, let Pp(X) denote the subsct satisfying IIel|,, == fx [[=l)) < oo and define the Wasscrstein p-distance between the two measures μ4, / E Pp(() as Wp(n, v) := (inf,er(a.))) / yl v)) 1/P, where T(p, v) is the set of couplings between μ and V. Let (X) denote the set of probability densities bounded almost everywhere with compact support. Let (n) denote the set of functions that are square integrable with respect to the measure μ.. Given μ E P(X) and smooth, invertible transform T X X, let T denote the pushforward measure of through T. The KL divergence between two measures p. V E P(X)) is denoted by KL(μ] || v).
< co. For
2.2 Reproducing Kernel Hilbert Spaces
A function k X' x X + R is positive definite if Eng a;k(a; mz)aa > 0 for any choice of 01,...,. On E R and 01....,.. La E X. Given a Hilbert space H of functions o : X - R, function k : X' x X - R is said to be a reproducing kernel for H if it satisfies the reproducing property, Ø(x) = (6. k(x, )) for all é E H. A positive definite : X' x X' > admits unique Hilbert space H of functions : X - for which the Dirac functionals 8= H + RR فوه o(x) are all continuous and i is a reproducing kernel. This Hilbert space called the reproducing kernel Hilbert space (RKHS) of k and it is equal to the closure of the span of (k(x,.) x R}. Let Hd H. x x H. denote the Hilbert space of functions & X' Rd whose components are all in H. and equip it with the usual inner product (Ø, v)
-i=ids sis.. Given two kernels ki, k2 X' X IR, let Hi, H2 denote their respective RKHS. An important, kernel used throughout this paper is the radial basis function (RBF) kernel KRBF (z, y; h) := exp(- | 10-2 -y112 /0l2//1 (2h)) with bandwidth
h > O. For thorough treatment of RKHS we refer the reader to Aronszajn (1950)) Steinwart & Christmant
(2008) and Berlinet & Thomas- Agnar (201T).
2.3 Stein Variational Gradient Descent
The key result from Liu & Wang (21I1) identifies a transform T X - X that optimally decreases the KL divergence from an arbitrary probability measure to æ. More precisely, let H be an RKHS with kernel k: X x X' + R and consider transforms of the form T(x) = x + cΦ(x:) where e > 0 and & is in the unit ball {o E Hd [| S 1}. The maximum value of
V.KL T)
(1)
occurs al & /l where
(+) == Exony
+ Vak(a,))
(2)
When y is an empirical distribution (i.c. sum of Dirac mcasures), the expectation in (2) can be computed exactly by summing over the particles of each Dirac measure. Using this observation, the SVGD algorithm starts with an initial set of N particles ( and iteratively applies the transform T with (2) as the update direction. At each iteration E, this yields a set of particles (x and corresponding empirical distribution He = N E, This is captured in Algorithm M. The intention is that after sufficiently many iterations, the
set of particles will resemble samples from and expectations of the form (a>nn(()) can be approximated by Eawp, (xx)= = N Σ, h(x)). We also recall the definition of the kernelisod Stein discrepancy (KSD) from Liu el all (2016).
S4 R) = Enggyr (2) k(x,, y) 8 (m))
(3)
where 6,.Y (xx) sT(x))
3

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm Stein Variational Gradient Descent (Liu & Wang. 201
Input: target probability distribution n, a kernel k; an initial set of particles (x)) in X, and a
sequence of step sizes (ce).
Output: set of particles (xr) in whose empirical distribution approximates n. for iteration do
Fit11 f- 2.
(x))
A 1
N
Meet
N
3 ue, (x)
klay x 8n (x
ki kiaza x)
(4)
N
driving force
repulsive force
Algorithm 2 Hybrid Kernel Stein Variational Gradient Descent
Input: A target probability distribution æ; two kernels k1, h2, an initial set of particles (at) in X, and sequence of step sizes (ez).
Output: A set of particles (x:) in &' whose empirical distribution approximates for iteration do
2241 t x + ce pe (z))). Vi = 1, I...,,N N
pr. (x))
ki (a x nn (ax))
(2 x)
(5)
driving force
repulsive force
2.4 Hybrid Kernel Stein Variational Gradient Descent
The SVGD update in ( contains two terms, each using the same kernel. The first lerm, often referred to as the driving term, uses the score function to move particles towards regions of high probability density, and the repulsive term prevents particles from collapsing at the modes. The h-SVGD variant proposed by D'Angelo et all (2022) uses different kernel in each term. Let kı denote the kernel that appears alongside the score function, and let l/ denote the repulsive kernel. For the remain- der of this paper, kı and l2 will both be positive definite. We present h-SvGD in Algorithm A.
3 Theoretical Results
3.1 Definitions and Assumptions
A function X + R is in the Stein class of if it is smooth and satisfies Saex Vn (f(x)=(z)) dz 0. A function f = (f1 i fa) X gd is in the Stein class of æ if each f; belongs the Stein class of n. A kernel k; X' x X' I is in the Stein class of if it has continuous second order partial derivatives and both k(x,.) and k(; 3) are in the Stein class of for all x, y X. The hybrid Stein operator acts on pair of
kernels k1, h2 X' x X + R by
S & (ki, (2) = (x, )8 (x) + Vakg(w.)),
provided k, and h2 both belong to the Stein class of w. This roduces to the Stein operator defined in Lm all (2016) when kj = k2. Motivated by the h-SVGD update in (B)) define the update direction
ki. = Enpa (S, s (k), k2) m))
(6)
and write 6² = ot..k2 1l (-))l H1 for the normalised direction. Let G( A u. T) == Exza [ki 5)sx (x) and R( k22 (1) Eazp (V (2t.) denote the driving (or gradient) term and the repulsive term respec-
4

---

Published in Transactions on Machine Learning Research (12/2025)
tively. We can then write (.) G( : (,, + R (2,
p)) The update transform
Thk (x) = c p (x)
(x)
(7)
and the map da 2 : pu+t + (iit
characterise the h-SVGD dynamics. For each e, define
M241 Ba (p
Pit1
(8)
where n is the empirical measure of the initial particles (at))=1 drawn i.id. from some M
All technical assumptions required in the theorems throughout this section are detailed here for completeness. The first set of assumptions relate to the potential of the target distribution.
(A1) Ve (X), V 2 and lim m0 V(x) +o0.
(A2) There exist constants Cv > 0 and > 1 such that for all æ, y E X,
[V(x)) V(x))
and
sup 8E[0,1]
+ + -
(A3) For any a, B > 0, there exists constant Ca.8 > 0 such that
(1 + - ()VVV (Vv)))))))))))))))>>>)))))) CC.
whenever |y| aa| +
(A4) The Hessian Hy of V is well-defined and satisfies |[y llap S M for some M 0.
Assumptions A
A2 and
A3
identical to those from Lu (2019) Assumption [44 is identical
to Assumption (A2) from Korba el all (2020), and Assumption 2.1 from Salin el all (20222) Assumplions on the kernels are also required.
(B1) There exist symmetric functions K1, K2 : X R such that A, y) K2(w y) for i 1,2, Ki is C with bounded derivatives, and K2 is C'4 with bounded derivatives. We use B > 0 as bound for all derivatives in the proofs.
(B2) There exists a constant (V)))a(; is D-Lipschitz for each That is,
such that both and Vkg are D-Lipschitz, and such that
(1 x') (x) < D (|æ vll, + ||æ' 3 112), ['') Vyk2(y. < D (|ı= yll, + || y 112)
D ([ - 21l2)
for all x. x', y. v E
Assumption B1
is a slight relaxation Assumption 2.1 from Ln al (am)), and contains Assumption 2.6
from Salim all (2022). The first two parts of Assumption [B2] are hybrid kernel versions of Assumption (B2) from Kor
(2020), and the third part of Assumption [B]] relaxes the restrictive Assumption
(B1) from Korba el. all 202)
5

---

Published in Transactions on Machine Learning Research (12/2025)
3.2 Large Particle Limit
We begin our theoretical study with result the devialion of the empirical distributions of h-SVGD to the large particle limit. The single kernel version (KKorba et all 2020) of the following result uses an assumption that is quite restrictive. It requires [V(x)] I Cv for some constant Cy > 0, which rules out even a Gaussian target distribution. We relax this with the third part of Assumption B2) and provide an updated proof in the appendix.
Proposition 3.1. Assume [ATA
A44
[BI] and B2 and let T 0. For any 0 S e S I, there erists
constant L depending on ki, 22 and such that
E ( pF
2/N V var(up)ee IT (E2TT - -1).
3.3 Hybrid Stein PDE
In the contimuous time limit, equation (E)
becomes a coupled system of differential equations,
dæ;
ki
+ V (22 (
(9)
di
for i= 1, 1,..,, N. In the mean field limit, integration by parts and the identity Vp- = pV log yields
dæ dt
/ (k1(2", (l) x)) + Var æ)) plar yda
(10)
ʃf (a .z) Vog < (xx))((())))))) (paa))d''' ka2 as )la")da
T (Vlog (a) Tkz.p (V p) (a:);
(11)
where Te.p : (p) - 2 is the Hilbert-Schmidt operator given by (T = f dd'' for a kernel k. Recalling that V is the potential of and so V log R, this mean field limit can be described by the hybrid Stein partial differential equation
Oep: = V (pr (K1 VV Pt + K2 * Vp.)).
(12)
Definition 3.2. Given a probability measure v on Rd a map (1,, [0, 00) x Rd Rd that C² with respect and satisfies
(1, = (Ki - Pt = 20
(VK2 pr) (x(122))))
(13)
((0,2;,1 2v:) = x
is called a mean field characteristic flow of 18I or of (I2)
We now generalise Theorem 2.4 from ot (2019)) ensuring the existence of a solution to the hybrid Stein
PDE. First, define Y = (u E C(X, X) supeex (u(x) - x < oi) with dy(u, (u, v) = supzex |u(x)) v(x)) and note that dy, is a complete metric space. Proposition 3.3. Assume ((..p) ((O,T))) to ik and the corvesponding Pt. is weak solution to (TB)) satisfying
[A7, [A2]
(A3] and BTJ and let T >>> 0. Then there enists a unique solution
pell, S || exp min 1/KKllll
(14)
for some constant C' > 0 depending on K1, K2 and V.
The second kernel enables a stronger bound than Theorem 2.4 from La mhettaa all (2012) by careful modification of the telescoping section of the proof (sce Appendix [ and Equation (3.8).. In particular, ensuring that ||V/iloo < || //Vll when choosing K₂ yields a stronger bound in (LE)) than if K2 were used for both kernels. We remark that this bound describes regularity of the solution to the PDE, not rate of convergence.

---

Published in Transactions on Machine Learning Research (12/2025)
3.4 Kernelised Wasserstein Gradient Flow and Asymptotic Density: 12 = ck1
Zhno et all (2118) uncovered correlation between dimension and the magnitude of the repulsive force R, defined at the beginning of Section B.L. Under some technical conditions, for any a, (0. 1), they show that with probability at least 1 - 8, SVGD under an RBF kernel yields [ (2)1 - O)d This suggests that simply scaling the repulsive force by d for some a E (0, 1) should offset the decrease in || k2, 1) in high dimensions, thereby alleviating variance collapse at negligible additional computational cost. Scaling the repulsive kernel in this way corresponds to h-SVGD where h is an RBF kernel and l2 = d This motivales our study of the h-SVGD gradient flow under the special case K2 cki.
In the case where k 11 k2, equations (IT)) and describe a kernelised Wasserstein gradient flow of the form depi- = V (p:TT.pp F(P)), where
F(p) (L || T) Ea [log log (x))
is the KL divergence functional 6E Sp (p) satisfying
2017). Recall that a functional derivative of F is a mcasurable function
d Fll+
6F (p)dx
=
de
c=0 óp
for all perturbations X p- with ý (X) Santambrogig AI 57
Definition 7.12). In particular
8F log log + 1 up to constant, and Vw
(p)
V log V Voog æ is the Wasserstein
6p
óp
gradient of F. Using [II) and the linearity of Tk.ps the corresponding Fokker-Planck equation is then
a.p. = V (poTk,pt (V loog Pt V log
(15)
where (p: : 2 0} is curve of probability densities. The following result generalises this gradient flow interpretation to the case where l/2 = ck1. Note that it applies to any positive definite kernel k1.
Proposition 3.4. Given positive definåte kernel hn and constant O, let 12 ck1. Then the mean field dynamics of h-SVGD describe a kernelised Wasserstein gradiend flow on the funclional
F(p) Ezop [clog p- (x))
(16)
The corresponding continuity equation is
dept V (piTir.p
loV log Pr V llggm) r)).
(17)
Even in this simple hybrid kernel sctting, the following result establishes that the limiting distribution pr of the mean field regime is not equal to the target distribution
Corollary 3.5. If l2 = cki for some constant > where ky is a positive definite kernel, then the mean field h-SVGD has fized point p² (xx) x nxx)
Although Corollary Ban applics to any target density N satisfying AI]. insightful to consider a Gaussian target. If π (a:) = N(x:; p. Σ), then (=x)) = N(x; (4, cE). So scaling the repulsive kernel ky adjusts the variance of the target by the same factor. This supports the motivation that scaling the repulsive kernel should offset the variance underestination in the finite particle regime in high dimensions at negligible additional cost. This idea will be revisited in Section H. Note that Corollary 3 extends the 1 case where SVGD converges to the target in the mean field limit (Liu & Wang. 20TT)
We now generalise an existing result (Korba el all 2121) that describes the dissipation of the KL divergence along the SVGD gradient flow. The result below describes the dissipation of the functional in (يل) along the
h-SVGD gradient flow, ensuring that the functional always docreases. It also describes the dissipation of the KL divergence with respect the mean field limiting distribution p", which we emphasise not equal the target distribution n, per Corollary 3.5

---

Published in Transactions on Machine Learning Research (12/2025)
Proposition 3.6. Under the assumptions of Proposition 3.4
d
F(p.) -c² ||Tki Vlog Vlog M1 te / P((z) log po()dd
(18)
dt
d _ di
kLip ll b0) = -c -TI.TR.P. (V Pogg
v log p² M1 + p((x)
log (o)dd
(19)
01.
where (a) x <<)) 1/ is the mean field fized point. Furthermore,
3 Ot f |((xx)
log pi (x)dx s 0,
d. de
F(pe) I 0 and KL(P l1 p²) S for t 2 0, and ds F(p") = 0.
d
The following descent lemma is adapted from Lin (2117) and provides discrete time version of Proposition R.R We use to denole discrele time steps, as in as opposed to Pt for the continuous time analysis. We remark that other descent lemmas for SVGD have been proved (Korha et all 2020: Salim et all 2122).
Proposition 3.7. Under the assumptions of Proposition 3.A. let o(-) denote the spectral radius and set & S (2sup, o (V (a)) + (==(x) Define sup log T Tll1ap ki (x. x) + 2V vV (x, x)} where 1(x))-(()) is the Lipschitz norm and Va.y hi x x) E, Oz ky, M>m==y- Then I = supaty |---
- -FWE)
S - R) S(pu,
3.5 Kernelised Wasserstein Gradient Flow: The General Case
In this section, we present a generalisation of Proposition s and discuss some difficulties in finding kernels that satisfy the required conditions, For ease of presentation, we restrict our attention to = Throughout,
we assume [B1] and that any required Fourier transforms exist. Define the function r X + R by
(((a) ((())
((VD) (x)
rx::p)== : I-1
(20)
where F denotes the Fourier transform (see proof in Appendix A and let R X + R be a function satisfying V RR(x; p) = -r(x;p) p(x).
Proposition 3.8. Assume that both r and R exist. Then the corresponding continuity equation is
S.p: = = (p.Trr.p (VR(: log x))
(21)
If in addition
I De
R(x; +x) = =0
(22)
for any measure X = p- p with ő E L (x) n P()), then the mean field dynamics of h-SVGD describe a kernelised Wasserstein gradient flow on the functional
F(p) Expp [R(x; p) (x))
(23)
Note that i2 = cki implics ((2))/ = and so r(z; = =cp((x) Therefore, R(x; = clog p(z), and so (2II) reduces to (IZ). Furthermore, the left hand side of (22) is equal to c.f dx(x) dĀ(x) -ef dp(x),
which is zero because p, p are both probability measures. So t
is satisfied this special case. We remark that verifying (22) remains a challenge general hybrid kernel scttings. However, under this assumption, the mean field dynamics of h-SVGD converges to a distribution that not the target distribution.
Proposition 3.9. If k1 + k2, both r and R exist, and PE is satisfied, then any fixed point p" of the gradient flow in Proposition (IL8) is not equal

---

Published in Transactions on Machine Learning Research (12/2025)
The example below demonstrates some behaviour of SVGD in the general hybrid kernel setting. In particular, the form of the target and the mean field steady state p" can look quite different.
Example 3.10. Let hı, h2, > O and assume that Ah == h2 hn + O. Let ha ka( = ARar (z, ys hs) for 1,2, and let (xx & exp(-o expl (a((a))) be the target on R where
h2 h
o² o²
0 Ah)
a V
+ Ah
Ah'
Ah
Then o² (x) = (2; 0, 02) is a fixed point of the h-SVGD mean field dynamics.
We briefly remark that the right hand side of Equation 22 exists in this setting of two RBF kernels with bandwidths h2 > h1 provided decays sufficiently fast. In particular,
(K2)(W) (VV))()) = i2Twy (K))
h22 hn
exp (h2 F ((D)(w).
-
We also remark that Section focuses experimental results in the l22 cki case due to its capacity to improve variance estimation. We leave more detailed study of the general hybrid setting for future work.
4 Experiments
Although Corollary M shows that h-SVGD does not converge to the target distribution, we demonstrate in this section that it has the ability to improve variance estimation when compared to SVGD. Furthermore, it does this al no extra computational cost,, and without any assumptions on the structure of the posterior, as is common in other SVGD variants that alleviate variance collapse, We measure variance collapse using dimension averaged marginal variance (DAMV), 4 + Var, ((=,))))) as is standard in the literature (Zhno etlall, 2018 Ba et all 2019 2021: Gong all 20221). Further details on the the experimental details
be found in Appendix D
Before describing the first numerical example, we reiterate the intuition behind the ability of h-SVGD to improve variance estimation. In high dimensions, with finite number of particles, SVGD is well known to suffer from variance collapse (Wang et all 2018: Ba et all 2019). Indeed, for any a, 5 E (0, 1), the repulsive force of SVGD under an RBF kernel decreases as [ k. pill = O(d-a) with probability at least 1-6 8 (Zhuo el. all 2 By Corollary SR., under a Gaussian larget. distribution, a repulsive kernel A = cki with 0 will scale the variance of the limiting distribution by e in the infinite particle limit. So scaling the repulsive kernel by d for some
(0,1) should compensate for the decrease in || : mill and consequence offset the variance collapse in high dimensions in the finite particle regime. The factor a should be tuned and we expect that a suitable choice of will depend on the number of particles N, the target distribution π and the kernel kj. We leave more detailed theoretical analysis guiding the choice of a for future work. In the experiments that follow, we found that a = 0.5 provided a good balance between variance estimation and inference.
4.1 Multivariate Gaussian Mixture
In this first example, we sample from a high -igh-dimensional mixture of Gaussians with 10 equally weighted components. The mean vectors are randomly initialised from (00 Ia), and the covariance matrices are cla, where e >
factor that makes the DAMV equal to 1. We repeat this for dimensions up to d = 1000 at intervals of 100. We choose to sample N = 50 particles in order to demonstrate the performance of h- SVGD when d is much greater than N, as is often the case in high dimensional Bayesian inference. Particles are initialised from (00 Ia) and each SVGD variant is run for 2000 iterations with an initial step size of = 0.01, adapted using AdaGrad. We run SVGD and sliced SVGD (SSVGD) (Gong et all, 202T) with the RBF kernel KRAP and compare against h-SVGD with kernels k = kReF and k = vd KRBF- All algorithms use med? log(N) as the bandwidth, where med is the median pairwise distance between particles (Li
The python code for reproducing these experiments availahle https: http://nonmmmcccaadddees//ssssse
9

---

Published in Transactions on Machine Learning Research (12/2025)
& Wang. 2016). For SSVGD, there is one bandwidth per dimension, the median distances are computed along each projection, as described in Gong et all (202T). We also compute the energy distance wetn the samples generated by each method and the target distribution as described in Székely & Rizzd (2013). All configurations are run 10 times with a different initialisation and results for cach configuration are averaged.
Figure D demonstrates that h-SVGD provides an uplift marginal variance estimation and an improved energy distance when compared to SVGD. Although SSVGD estimates the variance fairly consistently the dimension increases, Figure M shows that this comes at significant increase in runtime, whereas there is noticeable difference between the runtimes of SVGD and h-SVGD, Furthermore, h-SVGD outperforms SSVGD in terms of the energy distance. Figure 2 shows projections of the N = 50 particles onto the first two two dimensions along with contour plot of the marginal density along those two dimensions. This visual representation emphasises the extent to which both h-SVGD and SSVGD offset variance collapse in the finite particle setting. Further figures are included Appendix B
5VVG
5VSD
h-VGGD SSVGD
5SVG0
SSVGD
Emerge Distance
1.3
307
Time
1.1
101
0.8
600
600
800
430
200
400
(a) DAMV
(b) Energy Distance
(c) Time (seconds)
Figure 1: DAMV, energy distance, and runtime of different SVGD variants sampling from high-dimensional Gaussian mixtures.
SWCD
SHGG
5V9DD
Figure 2: Projections onto the first two dimensions of particles sampled from a high high-ddmensional Gaussian mixture using different SVGD variants. The contour plot of the corresponding marginal density is overlaid.
4.2 Bayesian Neural Network
In this section, we sample weights from a Bayesian neural network (BNN). Aside from scaling the repulsive kernel by Vd, as described in the previous experiment, our setup is identical to Liu & Wang (2mmF). For completeness, details are included in Appendix B Using the no-U-turn sampler (NUTS) (Hoffman ct all, 2оι with 10 parallel chains, we generated 10000 ground truth samples for of the 10 datasets. The Protein and Year datasets were large enough to make NUTS prohibitively slow. We then these ground
10

---

Published in Transactions on Machine Learning Research (12/2025)
Scheme
SVGD
h-SvGD
SSVGD
Energy Distance
5
Boston
Concrete
Energy
Kin8nm
Naval
Combined
Wine
Yacht
Figure 3: Energy distance of BNN samples generated by each SVGD variant as compared against ground truth samples generated by NUTS. Ranges indicate one standard deviation above and below the mean over repeated trials.
truth samples to compute the energy distance (Szekely & Rizza, 2UKK) of samples generated by each variant. Table M shows that with no additional computational cost, the problem of variance collapse is mitigated under h-SVGD through an increased DAMV. Table 2 demonstrates that it remains competitive at inference through improved test log-likelihood (LL) for all datascts and improved root mcan squared error (RMSE) for all but one dataset. Appendix B includes comparison of the same metrics between h-SVGD and SSVGD to emphasise that the alleviation of variance collapse under h-SVGD comes lower cost. Figure & shows that across all datasets except Wine, the samples generated by h-SVGD are closer to the ground truth than those generated by SVGD, as measured by the energy distance. The h-SVGD samples are also closer to the ground truth than the SSVGD samples for all but two datasets.
Table 1: DAMV and runtime in seconds of SVGD and h-SVGD. Ranges indicate one standard deviation above and below the mean over repeated trials. NUTS runtime is included for those datasets where was computationally feasible.
DAMV
Runtime (seconds)
Dataset
SVGD
h-SVGD
SVGD
h-SVGD
NUTS
Boston
0.051 + 0.011
0.112 + 0.015
28.9 + 1.4
28.6 + 0.8
133
Concrete
0.084 + 0.010
0.120 + 0.010
28.7 + 1.3
28.8 1.8
189
Energy
0.065 + 0.015
0.154 + 0.023
30.8 + 288
30.9 = 2.5
167
Kin8nm
0.105 0.003
0.102 + 0.003
1.3
36.2 = 1.3
1748
Naval
0.059 0.004
0.091 0.019
33.4 + 0.8
33.1 1.5
51
Combined
0.128 0.008
0.145
36.3 1.5
36.8 2.7
954
Protein
0.089 + 0.001
0.087 0.001
72.7 + 1.0
72.8 1.7
NA
Wine
0.068 = 0.005
0.090 + 0.003
29.5 + 1.4
29.8 1.1
393
Yacht
0.060 + 0.020
0.194 + 0.034
29.3 + 0.4
29.6
0.4
83
Year
0.011 + 0.000
0.012 + 0.000
592 + 24
563 + 23
NA
5
Conclusion
In this paper we have developed the mean field theory of h-SVGD by proving the existence of a solution to the hybrid Stein PDE and identifying it as a gradient flow on functional other than the KI. divergence. We characterised the mean field fixed point for the special case k ck₃ and demonstrated that h-SVGD does not converge to the target in the mean field limit unless 1. This suggests that h-SVGD may
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Average RMSE and LL of SGVD and h-SVGD evaluated on the test dataset. Ranges indicate one standard deviation above and below the mean over repeated trials.
Test RMSE
Test. LL
Dataset
SVGD
h-SVGD
SVGD
h-SVGD
Boston
3.094 + 0.579
3.034 = 0.587
-2.123 = 0.116
-1.959 + 0.148
Concrete
5.857 + 0.468
5.384 + 0.504
-2.616 + 0.099
-2.499 + 0.128
Energy
1.528 + 0.169
1.157 0.138
-1.702 + 0.094
-1.072 + 0.096
Kin8nm
0.124 + 0.005
0.100 0.003
-1.293 = 0.108
-0.314 0.104
Naval
0.006 + 0.000
0.004 0.000
-1.35
= 0.161
-0.404
= 0.140
Combined
4.105 0.220
4.072 0.220
-2.459 + 0.051
-2.367 + 0.050
Protein
4.791 + 0.025
4.679 + 0.025
-2.633 + 0.035
-2.511 + 0.023
Wine
0.637 + 0.044
0.631 + 0.045
-1.63 = 0.120
-0.819 + 0.080
Yacht
1.677 + 0.571
1.886 + 0.664
1.587 + 0.120
-1.045 + 0.160
Year
8.882 + 0.043
8.804 + 0.039
2.908 + 0.031
-2.800 + 0.019
not converge to the target more generally, even in its continuous time, mean-field form. We provided a result on the dissipation of the new functional, as well as a discrete time version, otherwise known as a descent lemma. We also highlighted the complexities of the gradient flow the general hybrid kernel setting. Despite cov-cenveeeence to the target, experimental results demonstrated that h-SVGD can alleviate variance collapse in high dimensions observed in Umile-partichl SVGD much lower cost than other SVGD variants. We also showed that h-SVGD maintains its performance high dimensional inference tasks, whilst improving variance estimation without the additional computational cost required of other SVGD variants. One interesting direction for future research is to find a principled method of scaling the repulsive kernel. Another avenue is to further develop the theory of h-SVGD the general hybrid kernel setting. It would also be interesting to incorporate tempered target densities, as considered in (Chehab all 2022)
Acknowledgments
SP gratefully acknowledges funding from the Eva Mayr-Stihl foundation.
References
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathemalical Sociely, 68(3):333741D 404, 1950.
Jimmy Ba, Murat A Erdogdu, Marzyeh Ghassemi, Taiji Suzuki, Shengyang Sun, Denny Wu, and Tianzong Zhang. Towards characterizing the high-dimenssonal bias kernel-based particle inference algorithms. In Second Symposium on Advances in Approzimate Bayesian Inference, 2019.
Jimmy Ba, Murat Erdogdu, Marzyeh Ghassemi, Shengyang Sun, Taiji Suzuki, Denny Wu, and Tianzong Zhang. Understanding the variance collapse of SVGD high dimensions. In International Conference on Learning Representations, 2021.
Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and stafistics. Springer Science Business Media, 2011.
Omar Chehab, Anna Korba, Austin Stromme, and Adrien Vacher. Provable convergence and limitations of geometric tempering for langevin dynamics. arXiv preprint arXiv:2410.0960 2024.
Peng Chen and Omar Ghattas. Projected Stein variational gradient descent. Advances in Neural Information Processing Systems, 33:1947-1958, 2020.
Sinho Chewi, Thibaut Le Gouic, Chen Lu, Tyler Maunu, and Philippe Rigollet. SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence. Advances in Neural Information Processing Sys- tems, 33:2098 22099, 2020.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Francesco D' D'Angelo and Vincent Fortuin. Annealed Stein variational gradient descent. In Third Symposium on Advances in Approzimate Bayesion Inference, 2021.
Francesco D'Angelo, Vincent. Fortuin, and Florian Wenzel. On Stein variational neural network ensembles. arXiv preprint vrXiw:2106.10760, 2021.
Andrew Duncan, Nikolas Nüsken, and Lukasz Szpruch. On the geometry of Stein variational gradient descent. Journal of Machine Learning Research, 24:1 39, 2023.
Wenbo Gong, Yingzhen Li, and José Miguel Hernández-Lobatte Sliced kernelized Stein discrepancy. In International Conference on Learning Representations, 2021.
Matthew D Hoffman, Andrew Gelman, et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593 1623, 2014.
Anna Korba, Adil Salim, Michael Arbel, Giulia Luise, and Arthur Gretton. A non-asymptotic analysis for Stein variational gradient descent. Advances in Neural Information Processing Systems, 33:4672-4682, 2020.
Chang Liu and Jun Zhu. Riemannian Stein variational gradient descent for Bayesian inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Qiang Liu. Stein variational gradient descent as gradient flow. Advances Neural Information Processing Systems, 30, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algo- rithm. Advances in Newral Information Processing Systems, 29, 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized Stein discrepancy for goodness-of-fit tests. In International Conference on Machine Learning, pp. 276 284. PMLR, 2016.
Xing Liu, Harrison Zhu, Jean-François Ton, George Wynne, and Andrew Duncan. Grassmann Stein varia- tional gradient descent. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151, pp. 2002 2021. PMLR, 2022.
Jianfeng Lu, Yulong Lu, and James Nolen. Scaling limit of the Stein variational gradient descent: The mean field regime. SIAM Journal on Mathematical Analysis, 51(2):648-677 671, 2019.
Adil Salim, Lukang Sun, and Peter Richtarik. A convergence theory for SVGD in the population limit under Talagrands inequality T1. In International Conference on Machine Learning, pp. 19139 19152. PMLR, 2022.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015.
Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
Gábor Székely and Maria L Rizzo. Energy statistics: class of statistics based on distances. Journal of statistical planning and inference, 143(9) 12722 2013.
Dilin Wang, Zhe Zeng, and Qiang Liu. Stein variational message passing for continuous graphical models. In International Conference on Machine Learning, pp. 5219 5227. PMLR, 2018
Dilin Wang, Ziyang Tang, Chandrajit Bajaj, and Qiang Liu. Stein variational gradient descent with matrix- valued kernels. Advances in Neural Information Processing Systems, 32, 2019.
Jingwei Zhuo, Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang. Message passing Stein variational gradient descent. In International Conference Machine Learning, pp. 6018-6027. PMLR, 2018.
13

---

Published in Transactions on Machine Learning Research (12/2025)
A Proofs
Lemma A.1. Under assumptions A1
A4
B2
the map
(2. - E(z,)) = -ki VV + (2
is L-piscchitz. That is,
[2) E(z', 11)12 S L(1 21112 + W2(H,,11)
where L > 0 depends on k1, 22 and V.
Proof. Largely following the proof of Lemma 14 Korba el all (2020), choosing an optimal coupling 7 of p
and p²,
[E(2, 11) - E(E', - E, (x) ki (1))
+ E, [VV((2)) - (V(2))/ 2²)]
+ E, \2 2) V k2 =2]1
< DE,, ||| x 2212 + || 2111
+ BME, ||| x' z'11z]
+ DE, ||| x + ||- 1z==21l 2'112)
- (2D + BM) (| x111 + (2(p, (())
Note that the second term is bounded using the relaxed Assumplion that V is bounded by a constant.
[B2 and there is no need to require
Proof of Proposition E.I. This follows identically to the proof of Proposition 7 in Korba et all (2022) with
Lemma in place of Lemma 14.
Proof of Proposition k The proof largely follows those of Theorems 2.4 and 3.2 La ef al 20 with some minor adjustments. Notably, after fixing and defining
Y { u Y sup lu(x) x < <r]
and the complete metric space
S = C(O,, Tol:To))),))
ds(u,v) = sup dy dy(u(t),v()) (e(0,7))
for some sufficiently small To (to be determined later), the operator 9 u(t, (t)) must be modified to act on E S, via
G(w)(1x)== =x x -e- VK2(u(s, - u(s, xt (dd
- -f. / Ki (i(u(s,))- u(s, x') VV (u(s,, ())) yds.
Note that use G instead of the JF used in Lu et al (201 to avoid confusion between the functional F defined in (II). The same techniques of In et all (201) are sufficient to establish the required bounds to
show that G is a contraction on S, for sufficiently small Ty- Note that Assumptions [B.] A2 and [A3]
are used to establish this. So the unique fixed point X(55u) E S, of G solves (IK) in the interval
[0, Tol
The min (IV /VK2ll term emerges because the telescoping in Equation (3.8) of Lu all
can be performed with either kernel. The remainder the proof follows Theorems 2.4 and 3.2 of (201))
14

---

Published in Transactions on Machine Learning Research (12/2025)
Proof of Proposition 8.4. Following Definition 7.12 in Santambrogio (2115), functional derivative of F is measurable function op (p) satisfying
d
=
6F (p)dx
de F(p+ox)
öp
c=0
65 for all perturbations X p- with PE (2)))(())) it exists, 8p (p) is unique up additive constants.
We first compute
d
F(p Flp+ de
c=0
de ( + +c /
f (logm(z)((E)de= f (log)())((2)da))
- -s ((x) + ex(xx) p())dx / clog(o(a) + +xx()))x(a)d
(f()))) X(x)
/ { x(x)
((x) + ex(x) x(x))dx
(ogT(())(()dd))
c=0
/ clogp(x) log r (x)x( + &e x(xr)dx.
(24)
Since P.PE E P(X), P(X), the final integral is zero. So the functional gradient F
8F (p) = clog log n 5p
Its Wasserstein gradient is then
F() cologp V log
Since K2- = cki, equation (III can be written as
dx
/ (ki( (1) log + x))=((")))))))) p(x)))))))))))))) plx')dx''
dt
(1 x)) logn(=') ca x) V x' plx')dx'
/ ( x V log log (()))())())))))))))))))))))))))) - c (æ', x)V log p( ) yda
Tki,p log π cV log
-Ta,,p (V (())
The continuity equation dip. + V (pr
0 then becomes
0,pt = v . (piTk. (Vw
= (peTr, (cV log Pt - V logm)).
15

---

Published in Transactions on Machine Learning Research (12/2025)
Proof of Corollary (S..). A measure pa satistying cV logp" V Vlog will be fixed point because this would imply F() 0. Solving this for p", we have
cV log (a) = V (og)(())
clogp" (x) T() + A
log(p" (x)²) = log(e x(())
p (a) = =<^T(x))
p (x) (
for some A E R.
Proof of Proposition F.. Using (IT), integration by parts, and the fact. that Thy (2()))) Hq is the adjoint of the inclusion Hi (p))),
d d dt dt F(m)= dt f ( (clog - log T(:)) da
0t pr(()) + (x) 8 / (clog - log T (T))) / (clog log ((:) llg
2
log log (a
Øt
(cV (oVlog((()) V log (x)) dr
+ / cpe(ax) ① 0t
log (ax)
v (clogp( logn(x)) Te lcVloop (o)))
+ (x)(x)
log px))d
(cV log p.
V o, (cV log - V log
Lo(p)) (P(P))"
+ cx)(x)
log (e()da
Øt
pT (cV log
Vlog T) 11 tc pı(
log pe
-e? IiTku.pe V log v log 11 -e e pc(x)
log (x
(25)
Also,
F(pe)
Eawp, [clog (c()r) log [a:)
Earp, [clog pc log (Ap((x)))
CEEupi [log (x) - log p" (x) + (Ag())
cKL(pt || (1) + log(A)
(26)
for some A E R, we have 1 F(t)) = _ kL p*). Therefore,
d
(T |l p²)² | Th Volloge V log πτ) Fgge I p((x))
log Pt (ar dar
dt
0t
2 + / (e(xx)
-c Thy (V log - V log n: 1/))
log pe da
H4
0t
θ log pe (z)
- (V Vlogpt
Voog pr 11 + (c(2z)
at
16

---

Published in Transactions on Machine Learning Research (12/2025)
To simplify notation in the calculations below, set Pi loVlog pi) and Pi (V log TT. Recall the identity
1 (lull l-/-- - |lull)) (u, u -
(27)
Now we apply the multivariate chain rule to the remainder term along with the fact about the adjoint of Th pt the identity in (27)), and the triangle inequality. This gives
/ pi(a) 01
dæ
log Pa (x)dx =
dx
dl
Pc(x)) Vilx)Voogpite log log) Tki pi loV Pt V loo क (x))
(28)
(V log pe, Tk. pi (cV log V Vogg (P(
= (T (V log pr): Tk (cV log P: - V log ))
(u, u - u)
1
(- -|lullzeg + +110118e4 - 1|u - =1li(s)
2c
<
1
||u|ltg 2 + 1|ue vllie
|| Foef 11u =11/20
2c
0.
The final statement that F(0") = 0 follows from equations V log π = 0.
and
AA
by observing that eV log
with p² instead of the target, Proof of Proposition This follows from applying Theorem 3.3 KM 2
then substituting in
Proof of Proposition f (f) defined by
We first define the Fourier transform of a real-valued function f to be the function
{(D(w)= - ( /
given that the integral exists. Similarly, the inverse Fourier transform of real-valued function is the function f (FF defined by
/
F (F)(x)
F(w) exp(12mur)ddu,
where the integral exists. To recover the continuity equation, apply the convolution theorem to (aI
8-1 (((2) 3 (Vp) (2) = r(z:p)
F(K))
(w) 3 (Vp)(w)) ((K)(()) -())()))(((()))))
3 (K2 Vp)(w) = F(KI r))())
(K2 Vp)(x) = (Ki rx)
/ Kg(z - V x pogg)()))( / |K(2z - y) ply) ply)dy
r(y)
Vlggp) (x) = (T)) (x)
17

---

Published in Transactions on Machine Learning Research (12/2025)
Equation ( can now be rewritten as
dæ dt
/ (b1(d',x)V log + k2 .2)d p(a'))))))))) da'
/ (1(x'x)) log (( (2' x) V pd
/ (Mr(a' x) V log
क (x') k2(2', Vx log p(x 1) p(x') yde
(TRi,P V log x (x) (T V log p) (x)
(T V log VR( :P)) (z),
where VR(x;)) =
r(ri)) p()
yielding the continuity equation 21
As in the proof of Proposition
let X p- p with ď (X)) We first compute
d
F(o+oo)
de
== (|Rlm; (all;++<x)e te f R(x:p+ (x)x(()
d
de
-/ /boTr()ddo(s) - 10gg(()dx(()) (1187/000t)))))) L
[d_ [RLESD0 / -(x:++x)dxll + + d ((xxp+))ddx(0))
f (10g(())x((0)) =
R(x:p) log logx(r)
+ (ʃ - = / B(a; ( c=
De
de
/ ((x;p) - log
/
(29)
de
By assumption, the remainder term above equal to zero and the functional derivative of is therefore
8F
(p)
Ri RR( p) log 7 (I),
6p
so its Wasserstein gradient
VwF(p) = v 4JJ
óp (p)
VRi p) lz).
18

---

Published in Transactions on Machine Learning Research (12/2025)
Proof of Proposition B.. To show that p² is not the target, assume for the sake of contradiction that p* So F( = 0, and Proposition B.N gives
r(æ; T)
π(x) = V (z)
r(æ; σ) = VT(X)
(K2)
$ (Vx) = $ (VV))
{KK)
8 (K2) = O(K))
K₂ = K1.
Therefore, it cannot be the case that p² =
Proof of Example Kf A direct computation of 211 along with the definitions of r and R yields
(K)
f12 cxppp2222
(VV))(w)
whh 2 w2
$ (w)
V
h1
h2
2TiW exp( - 2222" Ahi)
h1
h2
x²
r(z;p")
h1
2m
(o2 + Ah)3/2
exp
2( +A))
x²
x²
[R(x;p")
h
(a2 + Ahj3/2
Texp
2( + Ah)
.exp
20
h2
Ahr²
h1
(a²
Ah)3/2
Teep
20 2 + Ah)
Now computing V log using some A E R for the normalising constant, we have
x²
(④x) = A Aexp -axeep
2,3
xx2
log T (x)) = log(A)
exp
2,3
x²
V log Txx)
exp
2,3
h2
0
Ah
hn
(a2 + Ah)3/2
x Texp
20 + Ah)
Since VR(xpp") V (o), equation (2I) implies that p² is a fixed point.
B Additional Experimental Results and Details
B.1 Multivariate Gaussian Mixture
Figure Ø Section shows the projection onto the first two dimensions particles sampled from high- dimensional Gaussian mixtures using different SVGD variants. The marginal density of the corresponding projection of the target density is overlaid. We extend this here with Figure showing additional projection plots of randomly selected pairs of dimensions between 1 and 1000.
19

---

Published in Transactions on Machine Learning Research (12/2025)
SVGD
h-SVG
$ SVGD
Dinensorr 250, 458
C
Colaoa Di
Diresdoos 716
Dinencion 227, 225
Dinencionn 525, 26
Diventioee 227 169
Figure Projections onto pairs of randomly selected dimensions of particles sampled from high- dimensional Gaussian mixture using different SVGD variants. The contour plots of the corresponding marginal density overlaid.
20

---

Published in Transactions on Machine Learning Research (12/2025)
B.2 Bayesian Neural Network
The results presented in Section B.2 follow the settings of (Liu & Wang 2016). In particular, use Gaussian priors for the network weights and Gamma priors for the inverse covariances. There is one hidden layer with 50 units for most datascts, Protein and Year being the exceptions with 100 units cach. The datasots are randomly partitioned into 90% for training and 10% for testing with results averaged over 20 trials, Protein and Year being the exceptions with 5 trials and trials respectively. The number of particles each case is 20, the activation function is (Æ)U(x) = max(0, x), the number of iterations 2000, and the mini-batch size is 100 for all datasets except for Year, which uses mini-batch size of 1000.
We recreate Tables M and 2 below, this time comparing h-SVGD against SSVGD (Gong et all 202T). Table G shows that h-SVGD outperforms SSVGD in mitigation of variance collapse on all but one dataset. We note that h-SVGD achieves this with a significantly faster runtime for all datasets, which is due to SSVGD requiring additional optimistaion the projection matrix. Table E shows that SSVGD and h-SvGD are comparable both RMSE and LL metrics. The h-SVGD algorithm achieves a better LL score on more datasets, but SSVGD achieves better RMSE score on more datasets.
Table 3: DAMV and runtime in seconds of SSVGD and h-SVGD.
DAMV
Runtime (seconds)
Dataset
SSVGD
h-SVGD
SSVGD
h-SVGD
Boston
0.035 + 0.002
0.087 + 0.010
208 = 13
28.5 + 0.9
Concrete
0.070 + 0.004
0.102 + 0.006
148 + 56
28.7 + 1.3
Energy
0.053 + 0.005
0.106 0.011
156 + 28
30.7 + 2.2
Kin8nm
0.083 + 0.002
0.093 + 0.003
141 + 3.9
36.0 + 1.3
Naval
0.070 + 0.021
0.068 0.011
237 11
34.5 + 0.9
Combined
0.118 = 0.005
0.138 0.006
116 18
36.7 + 2.4
Protein
0.057 + 0.006
0.084 + 0.001
390 + 20
72.8 + 1.7
Wine
0.029 + 0.002
0.075 = 0.005
210 + 10
29.8 + 1.1
Yacht
0.066 + 0.009
0.121 + 0.012
97.9 1 1.2
30.0 + 1.3
Year
0.012 + NA
0.012 + NA
12488 + NA
666 + NA
Table 4: Average RMSE and LL of SSGVD and h-SVGD evaluated the test dataset.
Test RMSE
Test LL
Dataset.
SSVGD
h-SVGD
SSVGD
h-SVGD
Boston
3.024 = 0.604
3.001 E 0.584
-2.088 + 0.322
-1.988 + 0.221
Concrete
5.073 + 0.522
5.210 = 0.529
-2.563 + 0.239
-2.535 + 0.179
Energy
0.923 + 0.123
1.040 + 0.128
-0.631 + 0.162
-0.805
0.104
Kin8nm
0.084 0.003
0.090 = 0.003
0.232 + 0.135
0.468
0.090
Naval
0.003 0.000
0.004 0.000
-0.624 = 0.161
-0.090
0.105
Combined
4.028 0.220
4.057 + 0.218
-2.335 + 0.066
-2.354
0.052
Protein
4.581 + 0.026
4.600 = 0.026
-2.526 + 0.045
-2.456 - 0.017
Wine
0.676 + 0.051
0.626 + 0.045
-1.261 + 0.172
-0.750 + 0.097
Yacht
1.664 + 0.607
1.861 = 0.662
-0.788 + 0.511
-0.813 - 0.227
Year
8.922 NA
8.689 + NA
-2.940 + NA
-2.872 NA
21


Reviews:
[{"review_id": "kJ6QkHFGLR", "reviewer": "Reviewer_XMDu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a rigorous theoretical and empirical analysis of hybrid kernel SVGD (h-SVGD). The authors frame its mean-field limit as a kernelized Wasserstein gradient flow on a functional other than the Kullback-Leibler divergence. Theoretically, they demonstrate that h-SVGD does not converge to the target distribution $\\pi$ in this limit. Despite this negative asymptotic result, the paper's key practical contribution is its demonstration that h-SVGD effectively mitigates the variance collapse that plagues standard SVGD in high dimensions. Crucially, this improved variance estimation is achieved at a negligible additional computational cost, making h-SVGD a computationally efficient alternative to other methods that also address variance collapse.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper successfully uses mathematical proof to establish its theoretical (asymptotic) results. It then uses clear, targeted experiments to demonstrate the algorithm's practical utility in the finite-particle, high-dimensional regime—a setting where the asymptotic assumptions are no longer met and finite-particle effects dominate. The evidence for both of these complementary aspects is accurate and clearly presented.", "audience_yn": "Yes", "audience_explanation": "Methods based on SVGD are widely used, and this paper provides a valuable analysis of the h-SVGD variant. The TMLR audience will be interested in the paper's central, and somewhat paradoxical, finding: while the authors prove theoretically that h-SVGD does not converge to the target distribution in the mean-field limit , they also demonstrate empirically that it effectively solves the practical problem of variance collapse in high dimensions. The finding that it provides this benefit at almost no additional computational cost  makes this a result of clear practical interest.", "requested_changes": "1. The paper's main conclusion rests on a disconnect between the mean-field theory and the practical results. The gradient flow framework suggests convergence to a \"smoothed\" distribution $\\rho^{*}(x) \\propto \\pi(x)^{1/c}$ (Corollary 3.5) when $k_2=c k_1$, but the Gaussian mixture experiments show the particle variance converging to the true target variance, not a scaled one. The paper would be far more convincing if the authors could provide a more principled justification, even if not a full proof, for why this heuristic scaling in the finite-particle regime so neatly \"corrects\" the variance, thereby bridging the gap between the mean-field theory and the practical experiments.\n\n2. In (B2), \"such that both $k\\_1$ and $\\nabla k\\_2$ and are $D$-Lipschitz\" contains a typo.\n\n3. The derivation from (10) to (11) appears to be missing a step, such as an integration by parts, that would be necessary to move from $\\nabla_{x_j}k_2$ to $T_{k_2,\\rho}(\\nabla \\log \\rho)$. I suggest the authors complete the computation for better completeness. And the definition of $T_{k,\\rho}$ is not consistently written. \n\n4. The repulsive force $R(\\cdot;k,\\mu)$ is not defined before being used in the first paragraph of Section 3.4; please define it previously.", "broader_impact_concerns": "This paper is a foundational, theoretical work on algorithm properties, not a specific real-world application. As such, it does not present any direct broader impact concerns. Any potential societal impacts would be indirect and depend on future downstream applications, which are outside the scope of this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uwEj4oM8a7", "reviewer": "Reviewer_iHeY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Stein variational gradient descent (SVGD) is a non-parametric variational inference framework, which has gathered significant attention for appealing empirical and theoretical performance. A variant of SVGD, called hybrid kernel Stein variational gradient descent (h-SVGD), uses two different kernels for \"driving\" and \"repulsive\" terms of SVGD respectively. The variant, h-SVGD, can mediate a well-known issue of SVGD in underestimating the variance of a target distribution, by controlling the balance between the driving and repulsive terms. The focus of the paper is on theoretical analysis of h-SVGD, which had not been studied before. In particular, the paper shows that an updating scheme, or a flow, of densities defined by h-SVGD corresponds to minimisation of some functional over the space of densities, similarly to original SVGD. However, due to the use of two different kernels, the density to which this flow converges is no longer a target density, unlike SVGD. The numerical analysis provided in the paper illustrates that h-SVGD estimates the variance of a target distribution better than original SVGD.\n\nThe main part of the theoretical analysis is to reproduce the result of [1] and [2], in the setting that the kernels of the driving and repulsive terms are different from each other, as well as to study the limiting density of the flow. The strength is in confirming that h-SVGD indeed cannot converge to a given target density due to the use of two kernels, while it still defines a flow that minimises some functional. The weakness may be in that some main results themselves could be a relatively straightforward extension of [1] and [2], as we can apply their analysis to each driving and repulsive term separately, with a different kernel used.\n\n[1] Anna Korba, Adil Salim, Michael Arbel, Giulia Luise, and Arthur Gretton. A non-asymptotic analysis for\nStein variational gradient descent. Advances in Neural Information Processing Systems, 33:4672–4682,\n2020.\n\n[2] Jianfeng Lu, Yulong Lu, and James Nolen. Scaling limit of the Stein variational gradient descent: The mean\nfield regime. SIAM Journal on Mathematical Analysis, 51(2):648–671, 2019.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The proof seems correct to me. Proposition 3.1, 3.3, and 3.7 essentially follow from [1] and [2]. The rest follows from standard calculus, e.g., to derive Wasserstein gradients and time derivatives.", "audience_yn": "Yes", "audience_explanation": "The idea of h-SVGD to use two different kernels for the driving and repulsive terms in SVGD is an interesting idea to control the balance between the two terms. Understanding what properties are same as, or different from, original SVGD would be an interesting question for the community working on SVGD.", "requested_changes": "### Major Point\n\n- The terminology, driving and repulsive terms, are fine by readers who are familiar with the context. Though, for those who are not familiar, it may be worth concisely explaining what they are. For example, something like (2) in [3] would suffice.\n- It would be worth containing a definition and concise explanation about the first variation (which is referred to as functional derivative) and the Wasserstein gradient in P.6.\n- It would be nice to define the Fourier transform and the inverse somewhere either in Appendix or main text.\n- Readers may benefit some 2D illustrative figure to see how h-SVGD samples differ from standard SVGD samples. Currently, only the graphs and numbers of the error/variance estimate are provided.\n- It seems to me that one finding in this paper can suggest that the use of tempered density is helpful to mediate the variance-underestimation of SVGD, which makes sense. If this is correct, it may be worth adding some discussion in the paper.\n- In Corollary 3.5, the constant c should be $c \\ne 1$?\n- In Proposition 3.8, since the condition of the kernels are mentioned in the main text, it may be nice to clarify the condition under which the inverse Fourier transform exists in (20) rather than directly assuming the existence.\n\n\n\n### Minor Point\n\n- Abstract: particle based -> particle-based?\n- P.1: perhaps \\cite should be used for Duncan et al. \n- P.1: perhaps \\cite should be used for Zhou et al. \n- P.7: perhaps \\Cref should be used for Proposition (3.4).\n- P.9: such a a -> such a\n\n\n\n[3] Jimmy Ba, Murat A Erdogdu, Marzyeh Ghassemi, Shengyang Sun, Taiji Suzuki, Denny Wu, and Tianzong\nZhang. Understanding the variance collapse of SVGD in high dimensions. In International Conference on\nLearning Representations, 2021.", "broader_impact_concerns": "No concern on the ethical implication.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UjsZS8OEQr", "reviewer": "Reviewer_5Dn5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper studies the theoretical, notably convergence, properties of Hybrid Kernel SVGD (h-SVGD) that was proposed by D’Angelo et al. (2021), where the attraction and repulsion kernels are two distinct kernels. In this setting, the infamous variance collapse issue can be mitigated and has empirically shown to be superior for high dimensional scenarios such as image classification\n- This paper establishes the hybrid Stein operator and subsequently the hybrid Stein PDE, analogous to the Stein PDE of SVGD, under the mean field limit\n- Regularity conditions are then shown and an equivalence with a Wasserstein gradient flow are shown when $k_1 = c * k_2$ for constant $c$\n- The corresponding limiting distribution of the flow is given and it is proven that this is not equal to the target distribution $\\pi$\n- Furthermore, dissipation results are also shown for the proposed Wasserstein gradient flow function\n- A more general theoretical result is derived for any $k_1$ and $k_2$, and the theory shows the difficulties in obtaining feasible kernels.\n- A toy example is given that shows this inequality. \n- Finally, numerical results show that despite not having a mean field limiting distribution of the target distribution, h-SVGD surpasses the performance of SVGD. The theory suggests a scaling factor of $c=d^\\alpha$ for some $\\alpha\\in (0,1)$ to help alleviate variance collapse. Moreover, using DAMV as a measure of spread reviews that h-SVGD particles are more spread out than SVGD and SSVGD", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The theoretical claims are accurate upon checking the proofs. Some of the proofs rely on the theorems and proofs in Lu et al. (2019) and Korba et al. (2020), where the setting is the classic SVGD setting with a single kernel $k$. The authors modify the proof techniques with $k_1$ and $k_2$ and obtain similarly-looking results.\n- Example 3.10 is a nice illustration of what the limiting distribution $\\rho^*$ is and how it is not equal to the target distribution\n- Although DAMV has been used in various works, such as in Sliced SVGD and Message Passing SVGD, it simply measures the spread of the particles, and not whether the inferred distribution matches (or is spread out according to) the target distribution. I would suggest also looking at the energy distance as utilized in Grassmann SVGD (Liu et al. 2022)., or looking at the inferred covariance matrix (for the synthetic experiment) and using HMC samples as a reference ground-truth distribution.", "audience_yn": "Yes", "audience_explanation": "- Variance collapse issue is one of the main obstacles for SVGD that stops it from being widely adopted by the Bayesian machine learning community, especially for high-dimensional tasks such as BNN inference.\n- This paper shows an important theoretically-backed result for h-SVGD, one of the popular methods out there that has shown promise in the high-dimensional setting\n- Past and recent works such as Sliced SVGD, Grassmann SVGD, Projected SVGD etc… have all attempted to tackle the issue of variance collapse as well, so there is ample interest and room for improvement in this topic.", "requested_changes": "I would suggest considering extra measures of “goodness-of-spread” as I suggested above, such as the energy distance, inferred covariance matrix or using HMC samples.", "broader_impact_concerns": "This is mostly theoretical work, no concerns that I am aware of.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)