SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianle Liu
tianleliu@fas.harvard.edu
Promit Ghosal
Waltham, MA 02453
promit@mit.edu
Krishnakumar Balasubramanian
Davis, CA 95616
kbala@ucdavis.edu
Natesh S. Pillai
pillai@fas.harvard.edu
Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. Despite its wide usage, understanding the theoretical properties of SVGD has remained a challenging problem. For sampling from a Gaussian target, the SVGD dynamics with a bilinear kernel will remain Gaussian as long as the initializer is Gaussian. Inspired by this fact, we undertake a detailed theoretical study of the Gaussian-SVGD, i.e., SVGD projected to the family of Gaussian distributions via the bilinear kernel, or equivalently Gaussian variational inference (GVI) with SVGD. We present a complete picture by considering both the mean-field PDE and discrete particle systems. When the target is strongly log-concave, the mean-field Gaussian-SVGD dynamics is proven to converge linearly to the Gaussian distribution closest to the target in KL divergence. In the finite-particle setting, there is both uniform in time convergence to the mean-field limit and linear convergence in time to the equilibrium if the target is Gaussian. In the general case, we propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unifying framework. Interestingly, one of the new particle-based instance from this framework empirically outperforms existing approaches. Our results make concrete contributions towards obtaining a deeper understanding of both SVGD and GVI.
Sampling from a given target density arises frequently in Bayesian statistics, machine learning and applied mathematics. Specifically, given a potential  $V:\mathbb{R}^d\to \mathbb{R}$ , the target density is given by
$\rho (x)\coloneqq Z^{-1}e^{-V(x)}$  where  $Z\coloneqq \int e^{-V(x)}dx$  is the normalizing constant.
Traditionally-used Markov Chain Monte Carlo (MCMC) sampling algorithms are invariably not scalable to large-scale datasets [8, 61]. Variational inference and particle-based methods are two related alternatives proposed in the literature, both motivated by viewing sampling as optimization over the space of densities. We refer to [33, 39] for additional details related to this line of works.
In the literature on variational inference, recent efforts have focused on the Gaussian Variational Inference (GVI) problem. On the theoretical side, this is statistically motivated by the Bernstein-von
Mises theorem, which posits that in the limit of large samples posterior distributions tend to be Gaussian distributed under certain regularity assumptions. We refer to [81, Chapter 10] for details of the classical results, and to [34, 73] for some recent non-asymptotic analysis. On the algorithmic side, efficient algorithms with both statistical and computational guarantees are developed for GVI [14, 20, 1, 35, 43, 19]. From a practical point-of-view, several works [64, 78, 80, 67] have shown superior performance of GVI, especially in the presence of large datasets.
Turning to particle-based methods, [53] proposed the Stein Variational Gradient Descent (SVGD) algorithm, a kernel-based deterministic approach for sampling. It has gained significant attention in the machine learning and applied mathematics communities due to its intriguing theoretical properties and wide applicability [23, 31, 56, 87]. Researchers have also developed variations of SVGD motivated by algorithmic and applied challenges [89, 49, 18, 28, 84, 12, 55, 71]. In its original form, SVGD could be viewed as a nonparametric variational inference method with a kernel-based practical implementation.
The flexibility offered by the nonparametric aspect of SVGD also leads to unintended consequences. On one hand, from a practical perspective, the question of how to pick the right kernel for implementing the SVGD algorithm is unclear. Existing approaches are mostly ad-hoc and do not provide clear instructions on the selection of kernels. On the other hand, developing a deeper theoretical understanding of SVGD dynamics is challenging due to its nonparametric formulation. Notably [58] derived the continuous-time PDE for the evolving density that emerges as the mean-field limit of the finite-particle SVGD systems, and shows the well-posedness of the PDE solutions. In general, the following different types of convergences could be examined regarding SVGD:
(a) Unified convergence of the empirical measure for  $N$  finite particles to the continuous target as time  $t$  and  $N$  jointly grow to infinity;
(b) Convergence of mean-field SVGD to the target distribution over time;
(c) Convergence of the empirical measure for finite particles to the mean-field distribution at any finite given time  $t \in [0, \infty)$ ;
(d) Convergence of finite-particle SVGD to the equilibrium over time;
(e) Convergence of the empirical measure for finite particles to the continuous target at time  $t = \infty$ .
From a practical point of view (a) is the ideal type of result that fully characterizes the algorithmic behavior of SVGD, which could be obtained by combining either (b) and (c) or (d) and (e). Regarding (b), [51] showed the convergence of mean-field SVGD in kernel Stein discrepancy (KSD, [17, 52, 29]), which is known to imply weak convergence under appropriate assumptions. [40, 15, 70, 75, 22] sharpened the results with weaker conditions or explicit rates. [32] extended the above result to the stronger Fisher information metric and Kullback-Leibler divergence based on a regularization technique. [58, 30, 40] obtained time-dependent mean-field convergence (c) of  $N$  particles under various assumptions using techniques from the literature of propagation of chaos. [72] obtained even stronger results for (c) and combined (b) to get the first unified convergence (a) in terms of KSD. However, they have a rather slow rate  $1 / \sqrt{\log\log N}$ , resulting from the fact that their bounds for (c) still depend on the time  $t$  (sum of step sizes) double-exponentially. Moreover, there has not been any work that studies the convergence (d) and (e) for SVGD, which illustrate a new way to characterize the unified convergence (a).
In an attempt to overcome the drawbacks of the nonparametric formulation of SVGD and also taking cue from the GVI literature, in this work we study the dynamics of Gaussian-SVGD, a parametric formulation of SVGD. Our contributions in this work are three-fold:
- Mean-field results: We study the dynamics of Gaussian-SVGD in the mean-field setting and establish linear convergence for both Gaussian and strongly log-concave targets. As an example of the obtained results, Table 1 shows the convergence rates of covariance for centered Gaussian
Table 1: Convergence rates of SVGD with different bilinear kernels for Gaussian families.
K1-SVGD	K2-SVGD	WGF (K3-SVGD)	R-SVGD (K4-SVGD)
Centered Gaussian	O(e-2t) [3.2]	O(e-2t) [F.2]	O(e-2t/λ) [F.1]	O(e-(2t/(1-ν)λ+ν) [3.4]
General Gaussian	Theorem 3.1	O(e-(1/λ)∧2t) [F.2]	O(e-(t/λ) [F.1]	O(e-(1/λ)∧(2/(1-ν)λ+ν)t) [3.4]
families for several revelant algorithms. All of them will be introduced later in Sections 2 and 3. We also establish the well-posedness of the solutions for the mean-field PDE and discrete particle systems that govern SVGD with bilinear kernels (see Appendix C). Prior work [58] requires that the kernel be radial which rules out the important class of bilinear kernels that we consider. [32] relaxed the radial kernel assumption, however, they required boundedness assumptions which we avoid in this work for the case of bilinear kernels.
- Finite-particle results: We study the finite-particle SVGD systems in both continuous and discrete time for Gaussian targets. We show that for SVGD with a bilinear kernel if the target and initializer are both Gaussian, the mean-field convergence can be uniform in time (See Theorem 3.7). To the best of our knowledge, this is the first uniform in time result for SVGD dynamics and should be contrasted with the double exponential dependency on  $t$  for nonparametric SVGD [58, 72]. Our numerical simulations suggest that similar results should hold for certain classes of non-Gaussian target as well for Gaussian-SVGD. We also study the convergence (d) by directly solving the finite-particle systems (See Theorems 3.6 and 3.8). Moreover, in Theorem 3.10, assuming centered Gaussian targets, we obtain a linear rate for covariance convergence in the finite-particles, discrete-time setting, precisely characterizing the step size choice for the practical algorithm.
- Unifying algorithm frameworks: We propose two unifying algorithm frameworks for finite-particle, discrete-time implementations of the Gaussian-SVGD dynamics. The first approach assumes access to samples from Gaussian densities with the mean and covariance depending on the current time instance of the dynamics. The second is a purely particle-based approach, in that, it assumes access only to an initial set of samples from a Gaussian density. In particular, we show that three previously proposed methods from [27, 43] for GVI emerge as special cases of the proposed frameworks, by picking different bilinear kernels, thereby further strengthening the connections between GVI and the kernel choice in SVGD. Furthermore, we conduct experiments for eight algorithms that can be implied from our framework, and observe that the particle-based algorithms are invariably more stable than density-based ones. Notably one of the new particle-based algorithms emerging from our analysis outperforms existing approaches.
Denote the space of probability densities on  $\mathbb{R}^d$  by  $\mathcal{P}(\mathbb{R}^d) \coloneqq \{\rho \in \mathcal{F}(\mathbb{R}^d) : \int \rho \, \mathrm{d}\pmb{x} = 1, \rho \geq 0\}$ , where  $\mathcal{F}(\mathbb{R}^d)$  is the set of smooth functions. As studied in [42],  $\mathcal{P}(\mathbb{R}^d)$  forms a Fréchet manifold called the density manifold. For any "point"  $\rho \in \mathcal{P}(\mathbb{R}^d)$ , we denote the tangent space and cotangent space at  $\rho$  by  $T_{\rho}\mathcal{P}(\mathbb{R}^{d})$  and  $T_{\rho}^{*}\mathcal{P}(\mathbb{R}^{d})$  respectively. A Riemannian metric tensor assigns to each  $\rho \in \mathcal{P}(\mathbb{R}^d)$  a positive definite inner product  $g_{\rho}: T_{\rho}\mathcal{P}(\mathbb{R}^{d}) \times T_{\rho}\mathcal{P}(\mathbb{R}^{d}) \to \mathbb{R}$  and uniquely corresponds to an isomorphism  $G_{\rho}$  (called the canonical isomorphism) between the tangent and cotangent bundles [66], i.e., we have  $G_{\rho}: T_{\rho}\mathcal{P}(\mathbb{R}^{d}) \to T_{\rho}^{*}\mathcal{P}(\mathbb{R}^{d})$ .
Definition 2.1 (Wasserstein metric). The Wasserstein metric is induced by the following canonical isomorphism  $G_{\rho}^{\mathrm{Wass}}: T_{\rho} \mathcal{P}(\mathbb{R}^{d}) \to T_{\rho}^{*} \mathcal{P}(\mathbb{R}^{d})$  such that
$$
(G _ {\rho} ^ {\text {W a s s}}) ^ {- 1} \Phi = - \nabla \cdot (\rho \nabla \Phi), \quad \Phi \in T _ {\rho} ^ {*} \mathcal {P} (\mathbb {R} ^ {d}).
$$
The Wasserstein gradient flow (WGF) can be seen as the natural gradient flow for KL divergence on the density manifold with respect to the Wasserstein metric. In specific, the mean-field PDE is given by the linear Fokker-Planck equation [33]:
$$
\dot {\rho} _ {t} = - \left(G _ {\rho_ {t}} ^ {\text {W a s s}}\right) ^ {- 1} \frac {\delta}{\delta \rho_ {t}} \operatorname {K L} \left(\rho_ {t} \| \rho^ {*}\right) = \nabla \cdot \left(\rho_ {t} \nabla \frac {\delta}{\delta \rho_ {t}} \operatorname {K L} \left(\rho_ {t} \| \rho^ {*}\right)\right) = \nabla \cdot \left(\nabla \rho_ {t} + \rho_ {t} \nabla V\right), \tag {1}
$$
where  $\frac{\delta}{\delta\rho_t}$  denotes the variational derivative with respect to  $\rho_{t}$ ,  $\mathrm{KL}(\rho \| \rho^{*})$  is the so-called energy function, and  $V(\pmb {x})$  is the potential function that satisfies  $\rho^{*}(\pmb {x})\propto \exp (-V(\pmb {x}))$
Interestingly, the mean field flow of SVGD can also be seen as a gradient flow for the KL divergence but with respect to the Stein metric, where we perform kernelization in the cotangent space before taking the divergence [51, 22].
Definition 2.2 (Stein metric). The Stein metric is induced by the following canonical isomorphism  $G_{\rho}^{\mathrm{Stein}}: T_{\rho} \mathcal{P}(\mathbb{R}^{d}) \to T_{\rho}^{*} \mathcal{P}(\mathbb{R}^{d})$  such that
$$
(G _ {\rho} ^ {\mathrm {S t e i n}}) ^ {- 1} \Phi = - \nabla \cdot (\rho (\cdot) \int K (\cdot , \boldsymbol {y}) \rho (\boldsymbol {y}) \nabla \Phi (\boldsymbol {y})   \mathrm {d} \boldsymbol {y}), \quad \Phi \in T _ {\rho} ^ {*} \mathcal {P} (\mathbb {R} ^ {d}),
$$
where  $K:\mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}$  is a positive-definite kernel.
In particular, the mean field PDE of the SVGD algorithm can be written as
$$
\dot {\rho} _ {t} = - \left(G _ {\rho_ {t}} ^ {\text {S t e i n}}\right) ^ {- 1} \frac {\delta}{\delta \rho_ {t}} \operatorname {K L} \left(\rho_ {t} \| \rho^ {*}\right) = \nabla \cdot \left(\rho_ {t} (\cdot) \int K (\cdot , \boldsymbol {y}) \left(\nabla \rho_ {t} (\boldsymbol {y}) + \rho_ {t} (\boldsymbol {y}) \nabla V (\boldsymbol {y})\right) \mathrm {d} \boldsymbol {y}\right). \tag {2}
$$
Gaussian Families as Submanifolds. We consider the family of (multivariate) Gaussian densities  $\rho_{\theta} \in \mathcal{P}(\mathbb{R}^d)$  where  $\theta = (\mu, \Sigma) \in \Theta = \mathbb{R}^d \times \operatorname{Sym}^+(d, \mathbb{R})$ . Note that  $\operatorname{Sym}^+(d, \mathbb{R})$  is the set of (symmetric) positive definite  $d \times d$  matrices. In this way,  $\Theta$  can be identified as a Riemannian submanifold of the density manifold  $\mathcal{P}(\mathbb{R}^d)$  with the induced Riemannian structure. If we further restrict the Gaussian family to have zero mean, it further induces the submanifold  $\Theta_0 = \operatorname{Sym}^+(d, \mathbb{R})$ . Notably the Wasserstein metric on the density manifold induces the Bures-Wasserstein metric for the Gaussian families [6, 82]. In our paper, however, we consider the induced Stein metric on  $\Theta$  or  $\Theta_0$  (we call it the Gaussian-Stein metric; for details see Appendix B).
Different Bilinear Kernels and Induced Metrics. There are several different bilinear kernels that appear in literature. [54] considers the simple bilinear kernel  $K_{1}(\pmb{x},\pmb{y}) = \pmb{x}^{\top}\pmb{y} + 1$  while [27, 13] suggest the use of an affine-invariant bilinear kernel  $K_{2}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}(\pmb{y} - \pmb{\mu}) + 1$ . [13] further points out that with the rescaled affine-invariant kernel  $K_{3}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}\Sigma^{-1}(\pmb{y} - \pmb{\mu}) + 1$ , the Gaussian-Stein metric magically coincides with the Bures-Wasserstein metric on  $\Theta$  (not true on the whole density manifold). Note that here  $\pmb{\mu}$  and  $\boldsymbol{\Sigma}$  are the mean and covariance of the current meanfield Gaussian distribution which could change with time. Moreover, [32] proposed a regularized version of SVGD (R-SVGD) that interpolates the dynamics between WGF and SVGD. Interestingly R-SVGD with the affine-invariant kernel  $K_{2}$  for Gaussian families can be reformulated as Gaussian-SVGD with a new kernel  $K_{4}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}((1 - \nu)\Sigma + \nu I)^{-1}(\pmb{y} - \pmb{\mu})$ , which interpolates between  $K_{2}$  and  $K_{3}$  (see Theorem 3.4). For clarity we present the results for  $K_{1}$  in the main article while leave the analogues for  $K_{2} - K_{4}$  in the appendix. We also point out that  $K_{1}$  and  $K_{2}$  are the same on  $\Theta_0$ .
Gaussian-Stein Variational Gradient Descent. With a bilinear kernel and Gaussian targets, we will prove in the next subsection that the SVGD dynamics would remain Gaussian as long as the initializer is Gaussian. However, this is not true in more general situations especially when the target is non-Gaussian. Fortunately for Gaussian variational inference we can still consider the gradient flow restricted to the Gaussian submanifold. In general, we denote by  $G_{\theta}^{\mathrm{Stein}}$  the canonical isomorphism on  $\Theta$  induced by  $G_{\rho}^{\mathrm{Stein}}$ , and define the Gaussian-Stein variational gradient descent as
$$
\dot {\theta} _ {t} = - \left(G _ {\theta_ {t}} ^ {\text {S t e i n}}\right) ^ {- 1} \nabla_ {\theta_ {t}} \operatorname {K L} \left(\rho_ {\theta_ {t}} \| \rho^ {*}\right),
$$
where  $\rho^{*}$  might not be a Gaussian density. Notably Gaussian-SVGD solves the following optimization problem
$$
\min  _ {\theta \in \Theta} \mathrm {K L} (\rho_ {\theta} \| \rho^ {*}), \quad \text {w h e r e} \rho_ {\theta} \text {i s} \mathcal {N} (\boldsymbol {\mu}, \Sigma),
$$
via gradient descent under the Gaussian-Stein metric.
3 Dynamics of Gaussian-SVGD for Gaussian Targets
3.1 Mean-Field Analysis from WGF to SVGD
The Wasserstein gradient flow (WGF) restricted to the general Gaussian family  $\Theta$  is known as the Bures-Wasserstein gradient flow [43, 19]. For consistency in this subsection we always set
the initializer to be  $\mathcal{N}(\pmb{\mu}_0, \Sigma_0)$  with density  $\rho_0$  and target  $\mathcal{N}(b, Q)$  with density  $\rho^*$  for the general Gaussian family. Then the WGF at any time  $t$  remains Gaussian, and can be fully characterized by the following dynamics of the mean  $\pmb{\mu}_t$  and covariance matrix  $\Sigma_t$ :
$$
\dot {\boldsymbol {\mu}} _ {t} = - Q ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right), \quad \dot {\Sigma} _ {t} = 2 I - \Sigma_ {t} Q ^ {- 1} - Q ^ {- 1} \Sigma_ {t}. \tag {3}
$$
For SVGD with bilinear kernels we have similar results:
Theorem 3.1 (SVGD). For any  $t \geq 0$  the solution  $\rho_t$  of SVGD (2) with the bilinear kernel  $K_1$  remains a Gaussian density with mean  $\pmb{\mu}_t$  and covariance matrix  $\Sigma_t$  given by
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = (I - Q ^ {- 1} \Sigma_ {t}) \boldsymbol {\mu} _ {t} - (1 + \boldsymbol {\mu} _ {t} ^ {\top} \boldsymbol {\mu} _ {t}) Q ^ {- 1} (\boldsymbol {\mu} _ {t} - \boldsymbol {b}) \\ \dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} \left(\Sigma_ {t} + \boldsymbol {\mu} _ {t} (\boldsymbol {\mu} _ {t} - \boldsymbol {b}) ^ {\top}\right) Q ^ {- 1} - Q ^ {- 1} \left(\Sigma_ {t} + (\boldsymbol {\mu} _ {t} - \boldsymbol {b}) \boldsymbol {\mu} _ {t} ^ {\top}\right) \Sigma_ {t}, \end{array} \right. \tag {4}
$$
which has a unique global solution on  $[0,\infty)$  given any  $\pmb{\mu}_0\in \mathbb{R}^d$  and  $\Sigma_0\in \mathrm{Sym}^+ (d,\mathbb{R})$ . And  $\rho_{t}$  converges weakly to  $\rho^{*}$  as  $t\to \infty$  at the following rates
$$
\| \pmb {\mu} _ {t} - \pmb {b} \| = \mathcal {O} (e ^ {- 2 (\gamma - \epsilon) t}), \quad \| \Sigma_ {t} - Q \| = \mathcal {O} (e ^ {- 2 (\gamma - \epsilon) t}), \quad \forall \epsilon > 0,
$$
where  $\gamma$  is the smallest eigenvalue of the matrix
$$
\left[ \begin{array}{c c} I _ {d ^ {2}} & \frac {1}{\sqrt {2}} \boldsymbol {b} \otimes Q ^ {- 1 / 2} \\ \frac {1}{\sqrt {2}} \boldsymbol {b} ^ {\top} \otimes Q ^ {- 1 / 2} & \frac {1}{2} (1 + \boldsymbol {b} ^ {\top} \boldsymbol {b}) Q ^ {- 1} \end{array} \right] w i t h a l o w e r b o u n d \gamma > \frac {1}{1 + \boldsymbol {b} ^ {\top} \boldsymbol {b} + 2 \lambda},
$$
where  $\lambda$  is the largest eigenvalue of  $Q$ .
Note that for any vector  $\pmb{x}$ ,  $\| \pmb{x} \|$  denotes its Euclidean norm and for any matrix  $A$  we use  $\| A \|$  for its spectral norm,  $\| A \|_*$  for the nuclear norm and  $\| A \|_F$  for the Frobenius norm. All matrix convergence are considered under the spectral norm in default for technical simplicity (though all matrix norms are equivalent in finite dimensions).
If we restrict to the centered Gaussian family where both the initializer and target have zero mean (setting  $\mu_0 = b = 0$ ), the dynamics can further be simplified.
Theorem 3.2 (SVGD for centered Gaussian). Let  $\rho_0$  and  $\rho^*$  be two centered Gaussian densities. Then for any  $t\geq 0$  the solution  $\rho_{t}$  of SVGD (2) with the bilinear kernel  $K_{1}$  or  $K_{2}$  remains a centered Gaussian density with the covariance matrix  $\Sigma_t$  given by the following Riccati type equation:
$$
\dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} ^ {2} Q ^ {- 1} - Q ^ {- 1} \Sigma_ {t} ^ {2}, \tag {5}
$$
which has a unique global solution on  $[0,\infty)$  given any  $\Sigma_0, Q \in \mathrm{Sym}^+(d,\mathbb{R})$ . If  $\Sigma_0Q = Q\Sigma_0$ , we have the closed-form solution:
$$
\Sigma_ {t} ^ {- 1} = e ^ {- 2 t} \Sigma_ {0} ^ {- 1} + (1 - e ^ {- 2 t}) Q ^ {- 1}. \tag {6}
$$
In particular, if we let  $\Sigma_0 = I$  and  $Q = I + \eta \pmb{v}\pmb{v}^\top$  for some  $\eta > 0$  and  $\pmb{v} \in \mathbb{R}^d$  such that  $\pmb{v}^\top \pmb{v} = 1$ , then  $\Sigma_t$  can be rewritten as
$$
\Sigma_ {t} = I + \frac {\eta (1 - e ^ {- 2 t})}{1 + \eta e ^ {- 2 t}} \boldsymbol {v} \boldsymbol {v} ^ {\top}. \tag {7}
$$
[13] shows that for WGF if  $\Sigma_0Q = Q\Sigma_0$ , then we have  $\| \pmb{\mu}_t - \pmb{b}\| = \mathcal{O}(e^{-t / \lambda})$  and  $\| \Sigma_t - Q\| = \mathcal{O}(e^{-2t / \lambda})$ . For the centered Gaussian family SVGD converges faster if  $\lambda > 1$ . For the general Gaussian family WGF and SVGD have rather comparable rates (e.g., take  $\lambda \gg \| \pmb{b}\|$  then the lower bound here is roughly  $\mathcal{O}(e^{-t / \lambda})$ ). Another observation is that the WGF rates depend on  $Q$  alone but the SVGD rates here sometimes also depend on  $\pmb{b}$ , which breaks the affine invariance of the system. This is a problem originated from the choice of kernels as addressed in [13], where they propose to use  $K_{2}$  instead of  $K_{1}$ . Such approach has both advantages and disadvantages. The convention in SVGD is that the kernel should not depend on the mean-field density because the density is usually unknown and changes with time. But for GVI the affine-invariant bilinear kernel  $K_{2}$  only requires estimating the means from Gaussian distributions and is not a big issue.
Regularized Stein Variational Gradient Descent. In Section 2 we show that SVGD can be regarded as WGF kernelized in the cotangent space  $T_{\rho}^{*}\mathcal{P}(\mathbb{R}^{d})$ . The regularized Stein variational gradient descent (R-SVGD) [32] interpolates WGF and SVGD by pulling back part of the kernelized gradient of the cotangent vector  $\Phi$ , which is also seen as gradient flows under the regularized Stein metric:
Definition 3.3 (Regularized Stein metric). The regularized Stein metric is induced by the following canonical map
$$
(G _ {\rho} ^ {\mathrm {R S}}) ^ {- 1} \Phi := - \nabla \cdot \left(\rho \left((1 - \nu) \mathcal {T} _ {K, \rho} + \nu I\right) ^ {- 1} \mathcal {T} _ {K, \rho} \nabla \Phi\right),
$$
where  $\mathcal{T}_{K,\rho}$  is the kernelization operator given by  $\mathcal{T}_{K,\rho}f\coloneqq \int K(\cdot ,\pmb {y})f(\pmb {y})\rho (\pmb {y})\mathrm{d}\pmb {y}$ .
The R-SVGD is defined as
$$
\dot {\rho} _ {t} = - \left(G _ {\rho_ {t}} ^ {\mathrm {R S}}\right) ^ {- 1} \frac {\delta}{\delta \rho_ {t}} \operatorname {K L} \left(\rho_ {t} \| \rho^ {*}\right) = \nabla \cdot \left(\rho_ {t} \left((1 - \nu) \mathcal {T} _ {K, \rho_ {t}} + \nu I\right) ^ {- 1} \mathcal {T} _ {K, \rho_ {t}} \nabla \log \frac {\rho_ {t}}{\rho^ {*}}\right). \tag {8}
$$
Theorem 3.4 (R-SVGD). Let  $\rho_0$  and  $\rho^*$  be two Gaussian densities. Then the solution  $\rho_t$  of R-SVGD (8) with the bilinear kernel  $K_{2}$  converges to  $\rho^*$  as  $t\to \infty$ , and  $\rho_t$  is the density of  $\mathcal{N}(\pmb {b},\Sigma_t)$  with
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = - Q ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) \\ \dot {\Sigma} _ {t} = 2 \left((1 - \nu) \Sigma_ {t} + \nu I\right) ^ {- 1} \Sigma_ {t} - \left((1 - \nu) \Sigma_ {t} + \nu I\right) ^ {- 1} \Sigma_ {t} ^ {2} Q ^ {- 1} - Q ^ {- 1} \left((1 - \nu) \Sigma_ {t} + \nu I\right) ^ {- 1} \Sigma_ {t} ^ {2} \end{array} . \right. \tag {9}
$$
If  $\Sigma_0Q = Q\Sigma_0$ , we have  $\| \Sigma_t - Q\| = \mathcal{O}\left(e^{-2t / ((1 - \nu)\lambda +\nu)}\right)$ , where  $\lambda$  is the largest eigenvalue of  $Q$ .