Title: Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stein variational gradient descent

Gaussian variational inference

Rates of Convergence
Abstract: Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. Despite its wide usage, understanding the theoretical properties of SVGD has remained a challenging problem. For sampling from a Gaussian target, the SVGD dynamics with a bilinear kernel will remain Gaussian as long as the initializer is Gaussian. Inspired by this fact, we undertake a detailed theoretical study of the Gaussian-SVGD, i.e., SVGD projected to the family of Gaussian distributions via the bilinear kernel, or equivalently Gaussian variational inference (GVI) with SVGD. We present a complete picture by considering both the mean-field PDE and discrete particle systems. When the target is strongly log-concave, the mean-field Gaussian-SVGD dynamics is proven to converge linearly to the Gaussian distribution closest to the target in KL divergence. In the finite-particle setting, there is both uniform in time convergence to the mean-field limit and linear convergence in time to the equilibrium if the target is Gaussian. In the general case, we propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unified framework. Interestingly, one of the new particle-based instance from this framework empirically outperforms existing approaches. Our results make concrete contributions towards obtaining a deeper understanding of both SVGD and GVI.

Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent

Abstract

Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. Despite its wide usage, understanding the theoretical properties of SVGD has remained a challenging problem. For sampling from a Gaussian target, the SVGD dynamics with a bilinear kernel will remain Gaussian as long as the initializer is Gaussian. Inspired by this fact, we undertake a detailed theoretical study of the Gaussian-SVGD, i.e., SVGD projected to the family of Gaussian distributions via the bilinear kernel, or equivalently Gaussian variational inference (GVI) with SVGD. We present a complete picture by considering both the mean-field PDE and discrete particle systems. When the target is strongly log-concave, the mean-field Gaussian-SVGD dynamics is proven to converge linearly to the Gaussian distribution closest to the target in KL divergence. In the finite-particle setting, there is both uniform in time convergence to the mean-field limit and linear convergence in time to the equilibrium if the target is Gaussian. In the general case, we propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unifying framework. Interestingly, one of the new particle-based instance from this framework empirically outperforms existing approaches. Our results make concrete contributions towards obtaining a deeper understanding of both SVGD and GVI.

1 Introduction

Sampling from a given target density arises frequently in Bayesian statistics, machine learning and applied mathematics. Specifically, given a potential V : R d → R , the target density is given by

Traditionally-used Markov Chain Monte Carlo (MCMC) sampling algorithms are invariably not scalable to large-scale datasets [8, 61]. Variational inference and particle-based methods are two related alternatives proposed in the literature, both motivated by viewing sampling as optimization over the space of densities. We refer to [33, 39] for additional details related to this line of works.

In the literature on variational inference, recent efforts have focused on the Gaussian Variational Inference (GVI) problem. On the theoretical side, this is statistically motivated by the Bernstein-von

Mises theorem, which posits that in the limit of large samples posterior distributions tend to be Gaussian distributed under certain regularity assumptions. We refer to [81, Chapter 10] for details of the classical results, and to [34, 73] for some recent non-asymptotic analysis. On the algorithmic side, efficient algorithms with both statistical and computational guarantees are developed for GVI [14, 20, 1, 35, 43, 19]. From a practical point-of-view, several works [64, 78, 80, 67] have shown superior performance of GVI, especially in the presence of large datasets.

Turning to particle-based methods, [53] proposed the Stein Variational Gradient Descent (SVGD) algorithm, a kernel-based deterministic approach for sampling. It has gained significant attention in the machine learning and applied mathematics communities due to its intriguing theoretical properties and wide applicability [23, 31, 56, 87]. Researchers have also developed variations of SVGD motivated by algorithmic and applied challenges [89, 49, 18, 28, 84, 12, 55, 71]. In its original form, SVGD could be viewed as a nonparametric variational inference method with a kernel-based practical implementation.

The flexibility offered by the nonparametric aspect of SVGD also leads to unintended consequences. On one hand, from a practical perspective, the question of how to pick the right kernel for implementing the SVGD algorithm is unclear. Existing approaches are mostly ad-hoc and do not provide clear instructions on the selection of kernels. On the other hand, developing a deeper theoretical understanding of SVGD dynamics is challenging due to its nonparametric formulation. Notably [58] derived the continuous-time PDE for the evolving density that emerges as the mean-field limit of the finite-particle SVGD systems, and shows the well-posedness of the PDE solutions. In general, the following different types of convergences could be examined regarding SVGD:

Initial particles

ζ

(0)

N

=

1

N

∑

N

j

=1

δ

x

(0)

j

Evolving particles

ζ

(

t

)

N

=

1

N

∑

N

j

=1

δ

x

(

t

)

j

Equilibrium

ζ

(

∞

)

N

=

1

N

∑

N

j

=1

δ

x

(

∞

)

j

Initial density

ρ

0

Evolving density

ρ

t

Target

ρ

∗

(

c

)

(

b

)

(

e

)

(

d

)

(

a

)

(a) Unified convergence of the empirical measure for N finite particles to the continuous target as time t and N jointly grow to infinity;

(b) Convergence of mean-field SVGD to the target distribution over time;

(c) Convergence of the empirical measure for finite particles to the mean-field distribution at any finite given time t ∈ [0 , ∞ ) ;

(d) Convergence of finite-particle SVGD to the equilibrium over time;

(e) Convergence of the empirical measure for finite particles to the continuous target at time t = ∞ .

From a practical point of view (a) is the ideal type of result that fully characterizes the algorithmic behavior of SVGD, which could be obtained by combining either (b) and (c) or (d) and (e). Regarding (b), [51] showed the convergence of mean-field SVGD in kernel Stein discrepancy (KSD, [17, 52, 29]), which is known to imply weak convergence under appropriate assumptions. [40, 15, 70, 75, 22] sharpened the results with weaker conditions or explicit rates. [32] extended the above result to the stronger Fisher information metric and Kullback-Leibler divergence based on a regularization technique. [58, 30, 40] obtained time-dependent mean-field convergence (c) of N particles under various assumptions using techniques from the literature of propagation of chaos . [72] obtained even stronger results for (c) and combined (b) to get the first unified convergence (a) in terms of KSD. However, they have a rather slow rate 1 / √ log log N , resulting from the fact that their bounds for (c) still depends on the time t (sum of step sizes) double-exponentially. Moreover, there has not been any work that studies the convergence (d) and (e) for SVGD, which illustrate a new way to characterize the unified convergence (a).

In an attempt to overcome the drawbacks of the nonparametric formulation of SVGD and also taking cue from the GVI literature, in this work we study the dynamics of Gaussian-SVGD, a parametric formulation of SVGD. Our contributions in this work are three-fold:

Mean-field results : We study the dynamics of Gaussian-SVGD in the mean-field setting and establish linear convergence for both Gaussian and strongly log-concave targets. As an example of the obtained results, Table 1 shows the convergence rates of covariance for centered Gaussian

families for several revelant algorithms. All of them will be introduced later in Sections 2 and 3. We also establish the well-posedness of the solutions for the mean-field PDE and discrete particle systems that govern SVGD with bilinear kernels (see Appendix C). Prior work [58] requires that the kernel be radial which rules out the important class of bilinear kernels that we consider. [32] relaxed the radial kernel assumption, however, they required boundedness assumptions which we avoid in this work for the case of bilinear kernels.

Finite-particle results : We study the finite-particle SVGD systems in both continuous and discrete time for Gaussian targets. We show that for SVGD with a bilinear kernel if the target and initializer are both Gaussian, the mean-field convergence can be uniform in time (See Theorem 3.7). To the best of our knowledge, this is the first uniform in time result for SVGD dynamics and should be contrasted with the double exponential dependency on t for nonparametric SVGD [58, 72]. Our numerical simulations suggest that similar results should hold for certain classes of nonGaussian target as well for Gaussian-SVGD. We also study the convergence (d) by directly solving the finite-particle systems (See Theorems 3.6 and 3.8). Moreover, in Theorem 3.10, assuming centered Gaussian targets, we obtain a linear rate for covariance convergence in the finite-particles, discrete-time setting, precisely characterizing the step size choice for the practical algorithm.

Unifying algorithm frameworks : We propose two unifying algorithm frameworks for finiteparticle, discrete-time implementations of the Gaussian-SVGD dynamics. The first approach assumes access to samples from Gaussian densities with the mean and covariance depending on the current time instance of the dynamics. The second is a purely particle-based approach, in that, it assumes access only to an initial set of samples from a Gaussian density. In particular, we show that three previously proposed methods from [27, 43] for GVI emerge as special cases of the proposed frameworks, by picking different bilinear kernels, thereby further strengthening the connections between GVI and the kernel choice in SVGD. Furthermore, we conduct experiments for eight algorithms that can be implied from our framework, and observe that the particle-based algorithms are invariably more stable than density-based ones. Notably one of the new particlebased algorithms emerging from our analysis outperforms existing approaches.

2 Preliminaries

Denote the space of probability densities on R d by P ( R d ) := { ρ ∈ F ( R d ) : ∫ ρ d x = 1 , ρ ≥ 0 } , where F ( R d ) is the set of smooth functions. As studied in [42], P ( R d ) forms a Fréchet manifold called the density manifold. For any 'point' ρ ∈ P ( R d ) , we denote the tangent space and cotangent space at ρ by T ρ P ( R d ) and T ∗ ρ P ( R d ) respectively. A Riemannian metric tensor assigns to each ρ ∈ P ( R d ) a positive definite inner product g ρ : T ρ P ( R d ) × T ρ P ( R d ) → R and uniquely corresponds to an isomorphism G ρ (called the canonical isomorphism) between the tangent and cotangent bundles [66], i.e., we have G ρ : T ρ P ( R d ) → T ∗ ρ P ( R d ) .

Definition 2.1 (Wasserstein metric) . The Wasserstein metric is induced by the following canonical isomorphism G Wass ρ : T ρ P ( R d ) → T ∗ ρ P ( R d ) such that

The Wasserstein gradient flow (WGF) can be seen as the natural gradient flow for KL divergence on the density manifold with respect to the Wasserstein metric. In specific, the mean-field PDE is given by the linear Fokker-Planck equation [33]:

where δ δρ t denotes the variational derivative with respect to ρ t , KL( ρ ∥ ρ ∗ ) is the so-called energy function, and V ( x ) is the potential function that satisfies ρ ∗ ( x ) ∝ exp( -V ( x )) .

Interestingly, the mean field flow of SVGD can also be seen as a gradient flow for the KL divergence but with respect to the Stein metric, where we perform kernelization in the cotangent space before taking the divergence [51, 22].

Definition 2.2 (Stein metric) . The Stein metric is induced by the following canonical isomorphism G Stein ρ : T ρ P ( R d ) → T ∗ ρ P ( R d ) such that

where K : R d × R d → R is a positive-definite kernel.

In particular, the mean field PDE of the SVGD algorithm can be written as

Gaussian Families as Submanifolds. We consider the family of (multivariate) Gaussian densities ρ θ ∈ P ( R d ) where θ = ( µ , Σ) ∈ Θ = R d × Sym + ( d, R ) . Note that Sym + ( d, R ) is the set of (symmetric) positive definite d × d matrices. In this way, Θ can be identified as a Riemannian submanifold of the density manifold P ( R d ) with the induced Riemannian structure. If we further restrict the Gaussian family to have zero mean, it further induces the submanifold Θ 0 = Sym + ( d, R ) . Notably the Wasserstein metric on the density manifold induces the Bures-Wasserstein metric for the Gaussian families [6, 82]. In our paper, however, we consider the induced Stein metric on Θ or Θ 0 (we call it the Gaussian-Stein metric ; for details see Appendix B).

Different Bilinear Kernels and Induced Metrics. There are several different bilinear kernels that appear in literature. [54] considers the simple bilinear kernel K 1 ( x , y ) = x ⊤ y +1 while [27, 13] suggest the use of an affine-invariant bilinear kernel K 2 ( x , y ) = ( x -µ ) ⊤ ( y -µ ) + 1 . [13] further points out that with the rescaled affine-invariant kernel K 3 ( x , y ) = ( x -µ ) ⊤ Σ -1 ( y -µ ) + 1 , the Gaussian-Stein metric magically coincides with the Bures-Wasserstein metric on Θ (not true on the whole density manifold). Note that here µ and Σ are the mean and covariance of the current meanfield Gaussian distribution which could change with time. Moreover, [32] proposed a regularized version of SVGD (R-SVGD) that interpolates the dynamics between WGF and SVGD. Interestingly R-SVGD with the affine-invariant kernel K 2 for Gaussian families can be reformulated as GaussianSVGD with a new kernel K 4 ( x , y ) = ( x -µ ) ⊤ ((1 -ν )Σ + νI ) -1 ( y -µ ) , which interpolates between K 2 and K 3 (see Theorem 3.4). For clarity we present the results for K 1 in the main article while leave the analogues for K 2 -K 4 in the appendix. We also point out that K 1 and K 2 are the same on Θ 0 .

Gaussian-Stein Variational Gradient Descent. With a bilinear kernel and Gaussian targets, we will prove in the next subsection that the SVGD dynamics would remain Gaussian as long as the initializer is Gaussian. However, this is not true in more general situations especially when the target is non-Gaussian. Fortunately for Gaussian variational inference we can still consider the gradient flow restricted to the Gaussian submanifold. In general, we denote by G Stein θ the canonical isomorphism on Θ induced by G Stein ρ , and define the Gaussian-Stein variational gradient descent as

where ρ ∗ might not be a Gaussian density. Notably Gaussian-SVGD solves the following optimization problem

via gradient descent under the Gaussian-Stein metric.

3 Dynamics of Gaussian-SVGD for Gaussian Targets

3.1 Mean-Field Analysis from WGF to SVGD

The Wasserstein gradient flow (WGF) restricted to the general Gaussian family Θ is known as the Bures-Wasserstein gradient flow [43, 19]. For consistency in this subsection we always set

the initializer to be N ( µ 0 , Σ 0 ) with density ρ 0 and target N ( b , Q ) with density ρ ∗ for the general Gaussian family. Then the WGF at any time t remains Gaussian, and can be fully characterized by the following dynamics of the mean µ t and covariance matrix Σ t :

For SVGD with bilinear kernels we have similar results:

Theorem 3.1 (SVGD) . For any t ≥ 0 the solution ρ t of SVGD (2) with the bilinear kernel K 1 remains a Gaussian density with mean µ t and covariance matrix Σ t given by

which has a unique global solution on [0 , ∞ ) given any µ 0 ∈ R d and Σ 0 ∈ Sym + ( d, R ) . And ρ t converges weakly to ρ ∗ as t →∞ at the following rates

where γ is the smallest eigenvalue of the matrix

where λ is the largest eigenvalue of Q .

Note that for any vector x , ∥ x ∥ denotes its Euclidean norm and for any matrix A we use ∥ A ∥ for its spectral norm, ∥ A ∥ ∗ for the nuclear norm and ∥ A ∥ F for the Frobenius norm. All matrix convergence are considered under the spectral norm in default for technical simplicity (though all matrix norms are equivalent in finite dimensions).

If we restrict to the centered Gaussian family where both the initializer and target have zero mean (setting µ 0 = b = 0 ), the dynamics can further be simplified.

Theorem 3.2 (SVGD for centered Gaussian) . Let ρ 0 and ρ ∗ be two centered Gaussian densities. Then for any t ≥ 0 the solution ρ t of SVGD (2) with the bilinear kernel K 1 or K 2 remains a centered Gaussian density with the covariance matrix Σ t given by the following Riccati type equation:

which has a unique global solution on [0 , ∞ ) given any Σ 0 , Q ∈ Sym + ( d, R ) . If Σ 0 Q = Q Σ 0 , we have the closed-form solution:

In particular, if we let Σ 0 = I and Q = I + η vv ⊤ for some η > 0 and v ∈ R d such that v ⊤ v = 1 , then Σ t can be rewritten as

[13] shows that for WGF if Σ 0 Q = Q Σ 0 , then we have ∥ µ t -b ∥ = O ( e -t/λ ) and ∥ Σ t -Q ∥ = O ( e -2 t/λ ) . For the centered Gaussian family SVGD converges faster if λ > 1 . For the general Gaussian family WGF and SVGD have rather comparable rates (e.g., take λ ≫∥ b ∥ then the lower bound here is roughly O ( e -t/λ ) ). Another observation is that the WGF rates depend on Q alone but the SVGD rates here sometimes also depend on b , which breaks the affine invariance of the system. This is a problem originated from the choice of kernels as addressed in [13], where they propose to use K 2 instead of K 1 . Such approach has both advantages and disadvantages. The convention in SVGD is that the kernel should not depend on the mean-field density because the density is usually unknown and changes with time. But for GVI the affine-invariant bilinear kernel K 2 only requires estimating the means from Gaussian distributions and is not a big issue.

Regularized Stein Variational Gradient Descent. In Section 2 we show that SVGD can be regarded as WGF kernelized in the cotangent space T ∗ ρ P ( R d ) . The regularized Stein variational gradient descent (R-SVGD) [32] interpolates WGF and SVGD by pulling back part of the kernelized gradient of the cotangent vector Φ , which is also seen as gradient flows under the regularized Stein metric:

Definition 3.3 (Regularized Stein metric) . The regularized Stein metric is induced by the following canonical map

where T K,ρ is the kernelization operator given by T K,ρ f := ∫ K ( · , y ) f ( y ) ρ ( y ) d y .

The R-SVGD is defined as

Theorem 3.4 (R-SVGD) . Let ρ 0 and ρ ∗ be two Gaussian densities. Then the solution ρ t of R-SVGD (8) with the bilinear kernel K 2 converges to ρ ∗ as t →∞ , and ρ t is the density of N ( b , Σ t ) with

From this theorem we see that R-SVGD can take the advantage of both regimes by choosing ν wisely. Another interesting connection is that on Θ the induced regularized Stein metric coincides with the Stein metric with a different kernel K 4 ( x , y ) = ( x -µ ) ⊤ ((1 -ν )Σ + νI ) -1 ( y -µ ) + 1 (see Theorem B.7).

Stein AIG Flow. Accelerating methods are widely used in first-order optimization algorithms and have attracted considerable interest in particle-based variational inference [50]. [76, 86] study the accelerated information gradient (AIG) flows as the analogue of Nesterov's accelerated gradient method [63] on the density manifold. Given a probability space P ( R d ) with a metric tensor g ρ ( · , · ) , let G ρ : T ρ P ( R d ) → T ∗ ρ P ( R d ) be the corresponding isomorphism. The Hamiltonian flow in probability space [16] follows from

is called the Hamiltonian function, which consists of a kinetic energy 1 2 ∫ Φ G -1 ρ Φd x and a potential energy KL( ρ ∥ ρ ∗ ) . Following [86] we introduce the accelerated information gradient flow in probability space. Let α t ≥ 0 be a scalar function of time t . We add a damping term α t Φ t to the Hamiltonian flow:

By adopting the Stein metric we obtain the Stein AIG flow (S-AIGF):

Again we characterize the dynamics of S-AIGF with the linear kernel for the Gaussian family:

Theorem 3.5 (S-AIGF) . Let ρ 0 and ρ ∗ be two centered Gaussian densities. Then the solution ρ t of S-AIGF (11) with the bilinear kernel K 1 or K 2 is the density of N ( 0 , Σ t ) where Σ t satisfies

where S t ∈ Sym( d, R ) with initial value S 0 = 0 .

Note that the convergence properties of S-AIGF still remains open in contrast to the Wasserstein AIG flow (W-AIGF) as [86] shows that the W-AIGF for the centered Gaussian family is

and that if α t is well-chosen, the KL divergence in W-AIGF converges at the rate of O ( e -t/ √ λ ) . Thus, when λ is large it converges faster than WGF. It is also interesting to point out that the acceleration effect of Nesterov's scheme also comes from time discretization of the ODE system (see [74]) as it moves roughly √ ϵ rather than ϵ along the gradient path when the step size is ϵ .

3.2 Finite-Particle Systems

In this subsection, we consider the case where N < ∞ particles evolve in time t . We set a Gaussian target N ( b , Q ) (i.e., the potential is V ( x ) = 1 2 ( x -b ) ⊤ Q -1 ( x -b ) ) and run the SVGD algorithm with a bilinear kernel, and obtain the dynamics of x ( t ) 1 , · · · , x ( t ) N . The continuous-time particle-based SVGD corresponds to the following deterministic interactive system in R d :

with initial particles given by x (0) i ( i = 1 , · · · , N ). Now denoting the sample mean and covariance matrix at time t by µ t := 1 N ∑ N j =1 x ( t ) j and C t := 1 N ∑ N j =1 x ( t ) j x ( t ) ⊤ j -µ t µ ⊤ t , we have the following theorem.

Theorem 3.6 (SVGD) . Suppose the initial particles satisfy that C 0 is non-singular. Then SVGD (13) with the bilinear kernel K 1 and Gaussian potential V has a unique solution given by

where A t is the unique (matrix) solution of the linear system

and µ t and C t are the unique solution of the ODE system

The ODE system (16) is exactly the same as that in the density flow (4). Thus, we have the the same convergence rates as in Theorem 3.1. Theorem 3.6 can be interpreted as: At each time t the particle positions are a linear transformation of the initialization. On one hand, if we initialize i.i.d. from Gaussian, there is uniform in time convergence as shown in the theorem below.

On the other hand, if we initialize i.i.d. from a non-Gaussian distribution ρ 0 . At each time t the mean field limit ρ t should be a linear transformation of ρ 0 and cannot converge to the Gaussian target ρ ∗ as t →∞ . Note that in general SVGD with the bilinear kernel might not always converge to the target distribution for nonparametric sampling but for GVI there is no such issue. This will be discussed in detail in Appendix C together with general results of well-posedness and mean-field convergence of SVGD with the bilinear kernel, which has not yet been studied in literature.

Theorem 3.7 (Uniform in time convergence) . Given the same setting as Theorem 3.6, further suppose the initial particles are drawn i.i.d. from N ( µ 0 , Σ 0 ) . Then there exists a constant C d,Q, b , Σ 0 , µ 0 such that for all t ∈ [0 , ∞ ] , for all N ≥ 2 , with the empirical measure ζ ( t ) N = 1 N ∑ N i =1 δ x ( t ) i , the second moment of Wasserstein2 distance between ζ ( t ) and ρ converges:

N t

Similar to Theorem 3.2, we also provide the finite-particle result for a centered Gaussian target.

Theorem 3.8 (SVGD for centered Gaussian) . Suppose the SVGD particle system (13) with the bilinear kernel K 1 or K 2 is targeting a centered Gaussian distribution and initialized by ( x (0) i ) N i =1 such that µ 0 = 0 and C 0 Q = QC 0 . Then we have the following closed-form solution

Analogous Result for R-SVGD. Next we consider the particle dynamics of R-SVGD. As shown in [32], the finite-particle system of R-SVGD is

where X t := ( x ( t ) 1 , · · · , x ( t ) N ) ⊤ , L t f := ( f ( x ( t ) i ) , · · · , f ( x ( t ) N ) ) ⊤ for all f : R d → R d and ( K t ) ij := K ( x ( t ) i , x ( t ) j ) for all 1 ≤ i, j ≤ N . Similar to Theorem 3.6, we have the following result:

Theorem 3.9 (R-SVGD) . Suppose the R-SVGD system (13) with K 1 or K 2 is targeting a centered Gaussian distribution and initialized by ( x (0) i ) N i =1 such that µ 0 = 0 . Then we have x ( t ) i = A t x (0) i where A t is the unique solution of the linear system

and the sample covariance matrix C t is given by

Again we observe that the particles at time t is a time-changing linear transformation of the initializers.

Discrete-time Analysis for Finite Particles. Next we consider the algorithm in discrete time t . The SVGD updates according to the following equation:

For simplicity, we only consider the case where both the target and initializers are centered, i.e., b = µ 0 = 0 and show the convergence:

Theorem 3.10 (Discrete-time convergence) . For a centered Gaussian target, suppose the particle system (21) with K 1 or K 2 is initialized by ( x (0) i ) N i =1 such that µ 0 = 0 and C 0 Q = QC 0 . For 0 < ϵ < 0 . 5 , we have µ t = 0 and ∥ C t -Q ∥ → 0 as long as all the eigenvalues of Q -1 C 0 lie in the interval (0 , 1 + 1 /ϵ ) .

Furthermore, if we set u ϵ to be the smaller root of the equation f ′ ϵ ( u ) = 1 -ϵ (it has 2 distinct roots) where f ϵ ( x ) := (1 + ϵ (1 -x )) 2 x , then we have linear convergence, i.e.,

as long as all the eigenvalues of Q -1 C 0 lie in the interval [ u ϵ , 1 / 3 + 1 / (3 ϵ )] .

The above result illustrates that firstly the step sizes required are restricted by the largest eigenvalue of Q -1 C 0 . In particular if C 0 = I d then we need smaller step size if the smallest eigenvalue of Q is smaller, which corresponds to the β -log-smoothness condition of the target distribution. Secondly we can potentially have faster convergence over iteration given larger step sizes. We believe that the commutativity assumption in Theorem 3.10 can be relaxed and similar results can be obtained for general targets. Detailed examinations are left as future work.

4 Beyond Gaussian Targets

In this section we consider the Gaussian-SVGD with a general target and have the following dynamics.

Theorem 4.1. Let ρ ∗ be the density of the target distribution with the potential function V ( x ) that satsifies Assumption C.1 and ρ 0 be the density of N ( µ 0 , Σ 0 ) . The Gaussian-SVGD with K 1 produces a Gaussian density ρ t with mean µ t and covariance matrix Σ t given by

where Γ t = E x ∼ ρ t [ ∇ 2 V ( x )] and m t = E x ∼ ρ t [ ∇ V ( x )] .

Furthermore, suppose that θ ∗ is the unique solution of the following optimization problem

Then we have ρ t → ρ θ ∗ ∼ N ( µ ∗ , Σ ∗ ) as t →∞ .

In particular, if the target is strongly log-concave, it gives rise to the following linear convergence:

Algorithm 1 Density-based Gaussian-SVGD.

Algorithm 2 Particle-based Gaussian-SVGD.

Theorem 4.2. Assume that the target ρ ∗ is α -strongly log-concave and β -log-smooth, i.e., αI ⪯ ∇ 2 V ( x ) ⪯ βI . Then ρ t of Theorem 4.1 converges to ρ θ ∗ at the following rate

where γ/α is the smallest eigenvalue of the matrix

Typically the β -log-smoothness condition is not required for continuous-time analyses. However, it is required in the above statement, as our proof technique is based on comparing the decay of the energy function of the flow to that for WGF, following [13]. Relaxing this condition is interesting and we leave it as future work.

Unifying Algorithms. For general targets, we propose two unifying algorithm frameworks where we can choose any bilinear kernel (e.g., K 1 -K 4 ) to solve GVI with SVGD. The first framework is density-based where we update µ t and Σ t according to the mean-field dynamics. It requires the closed-form of the ODE system

where m t = E x ∼ ρ t [ ∇ V ( x )] and Γ t = E x ∼ ρ t [ ∇ 2 V ( x )] , and F and G are some closed-form functions. For example F ( µ t , Σ t , m t , Γ t ) = ( I -Γ t Σ t ) µ t -(1+ µ ⊤ t µ t ) m t and G ( µ t , Σ t , m t , Γ t ) = I -Σ t Γ t -µ t m ⊤ t for K 1 as shown in (22). Note that m t and Γ t can be estimated from samples using

or using the first-order estimator

The second framework is particle-based and does not need the closed-form ODE of the mean and covariance, making it more flexible than Algorithm 1. Here we intially draw N particles and keep updating them over time using

where ̂ ∇ V is a (time-dependent) linear approximation of ∇ V defined as ̂ ∇ V ( x ) = ̂ Γ t ( x -µ t )+ ̂ m t . Intuitively this is used instead of ∇ V to ensure the Gaussianity of the particle system.

We now remark on the algorithms proposed in the literature that emerge as instances of the two proposed algorithm frameworks, thereby highlighting the unifying viewpoint offered by our analysis. The use of the kernel K 1 for SVGD in variational inference dates back to [54]. Algorithms 1 and 2 with the kernel K 2 correspond precisely to the GF and GPF algorithms in [27]. Moreover, if K 3 is chosen, Algorithm 1 reproduces the BWGD algorithm in [43] (with N = 1 ) and shares some similarity with the FB-GVI algorithm in [19]. Detailed discussions on these variants, connections and convergence properties are deferred to Appendix D.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

time

5.0

7.5

10.0

12.5

15.0

17.5

20.0

(

)

SBGD (

K

1)

GF (

K

2)

BWGD (

K

3)

RGF (

K

4)

SBPF (

K

1)

GPF (

K

2)

BWPF (

K

3)

RGPF (

K

4)

(a)

̂

F

(

ρ

θ

)

over time.

0

250

500

750

1000

1250

1500

1750

2000

iteration

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

(

)

SBGD (

K

1)

GF (

K

2)

BWGD (

K

3)

RGF (

K

4)

SBPF (

K

1)

GPF (

K

2)

BWPF (

K

3)

RGPF (

K

4)

(b) ̂ F ( ρ θ ) over iteration.

5 Simulations

In this section, we conduct simulations to compare Gaussian-SVGD dynamics with different kernels and the performance of the algorithms mentioned in the previous section. We consider three settings, Bayesian logistic regression , and Gaussian and Gaussian mixture targets . Here we present the results for Bayesian logistic regression as it involves a non-Gaussian but unimodal target and is one of the typical setups such that GVI is preferred in practice. For sake of space we leave the simulations for the other two settings along with further discussions to Appendix E.

Bayesian Logistic Regression. The following generative model is considered: Given a parameter ξ ∈ R d , draw samples { ( X i , Y i ) } n i =1 ∈ ( R d × { 0 , 1 } ) n such that X i i.i.d. ∼ N ( 0 , I d ) and Y i | X i ∼ Bern( σ ( ⟨ ξ , X i ⟩ )) where σ ( · ) is the logistic function. Given the samples { ( X i , Y i ) } n i =1 and a uniform (improper) prior on ξ , the potential function of the posterior ρ ∗ on ξ is given by V ( ξ ) = ∑ n i =1 (log(1 + exp( ⟨ ξ , X i ⟩ ) -Y i ⟨ ξ , X i ⟩ )) . We run both Algorithms 1 and 2 initialized at ρ 0 = N ( 0 , I d ) to find the ρ θ ∗ that minimizes KL( ρ θ ∥ ρ ∗ ) . In Figure 1, SBGD (Simple Bilinear Gradient Descent), GF (Gaussian Flow [27]), BWGD (Bures-Wasserstein Gradient Descent [43]), and RGF (Regularized Gaussian Flow) are density-based algorithms with the bilinear kernels K 1 , K 2 , K 3 , and K 4 ( ν = 0 . 5 ) respectively; SBPF (Simple Bilinear Particle Flow), GPF (Gaussian Particle Flow [27]), BWPF (Bures-Wasserstein Particle Flow), and RGPF (Regularized Gaussian Particle Flow) are particle-based algorithms with the bilinear kernels K 1 , K 2 , K 3 , and K 4 ( ν = 0 . 5 ) respectively. We use the sample estimator of F ( ρ θ ) = ∫ (log ρ θ + V ) d ρ θ = KL( ρ θ ∥ ρ ∗ ) + C ( C is some constant) to evaluate the learned parameters θ = ( µ , Σ) . For a fair comparison in Figure 1b, we draw 2000 particles for particle-based algorithms and run all algorithms for 2000 iterations so that a total of 2000 samples are drawn for the density-based algorithms. The largest safe step sizes are 0 . 02 , 0 . 1 , 2 , 0 . 8 , 0 . 02 , 0 . 2 , 4 , 4 .

Figure 1 shows the decay of ̂ F ( ρ θ ) over time or iterations. For Figure 1a the same step size 0 . 01 is specified for all algorithms while for Figure 1b we choose the largest safe step size for each algorithm. In other words, Figure 1a provides the continuous-time flow of the dynamical system in each algorithm while Figure 1b emphasizes more on the discrete-time algorithmic behaviors. In Figure 1a there are roughly four distinct curves indicating four different kernels. K 1 has the most rapid descent, followed by K 2 , K 4 , and K 3 .

From Figure 1b we observe that BWPF and RGPF are the better choices for practical use. The difference in the largest step sizes shows that in terms of stability K 3 is the best, K 4 is almost as stable, but K 2 and K 1 are much worse. We also observe that particle-based algorithms are consistently more stable than density-based counterparts (which are essentially stochastic gradient based). The superiority of particle-based algorithms are even more evident in the Gaussian mixture experiment where the target is multi-modal. We further remark that another recently proposed density-based algorithm, the FB-GVI [19] shows comparable performance to BWPF and RGPF with large step sizes. We conduct a comparison of these three algorithms in Appendix E but do not include it here for clarity. It would also be really interesting to study the particle-based analogue of FB-GVI as future works.

References

Pierre Alquier and James Ridgway, Concentration of tempered posteriors and of their variational approximations , The Annals of Statistics 48 (2020), no. 3, 1475-1497.

David Alvarez-Melis, Yair Schiff, and Youssef Mroueh, Optimizing functionals on the space of probabilities with input convex neural networks , Transactions on Machine Learning Research (2022).

Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton, Maximum mean discrepancy gradient flow , Advances in Neural Information Processing Systems, vol. 32, 2019.

Jimmy Ba, Murat A Erdogdu, Marzyeh Ghassemi, Shengyang Sun, Taiji Suzuki, Denny Wu, and Tianzong Zhang, Understanding the variance collapse of svgd in high dimensions , International Conference on Learning Representations, 2021.

Julio Backhoff-Veraguas, Joaquin Fontbona, Gonzalo Rios, and Felipe Tobar, Stochastic gradient descent in Wasserstein space , arXiv preprint arXiv:2201.04232 (2022).

Rajendra Bhatia, Tanvi Jain, and Yongdo Lim, On the Bures-Wasserstein distance between positive definite matrices , Expositiones Mathematicae 37 (2019), no. 2, 165-191.

Rajendra Bhatia and Peter Rosenthal, How and why to solve the operator equation AX -XB = Y , Bulletin of the London Mathematical Society 29 (1997), no. 1, 1-21.

David M Blei, Alp Kucukelbir, and Jon D McAuliffe, Variational inference: A review for statisticians , Journal of the American statistical Association 112 (2017), no. 518, 859-877.

S Bobkov and M Ledoux, One-dimensional empirical measures, order statistics, and Kantorovich transport distances , Memoirs Amer. Math. Soc (2019).

José Antonio Carrillo, Katy Craig, and Francesco S Patacchini, A blob method for diffusion , Calculus of Variations and Partial Differential Equations 58 (2019), 1-53.

Anthony Caterini, Rob Cornish, Dino Sejdinovic, and Arnaud Doucet, Variational inference with continuously-indexed normalizing flows , Uncertainty in Artificial Intelligence, PMLR, 2021, pp. 44-53.

Peng Chen and Omar Ghattas, Projected Stein variational gradient descent , Advances in Neural Information Processing Systems 33 (2020), 1947-1958.

Yifan Chen, Daniel Zhengyu Huang, Jiaoyang Huang, Sebastian Reich, and Andrew M Stuart, Gradient flows for sampling: Mean-field models, Gaussian approximations and affine invariance , arXiv preprint arXiv:2302.11024 (2023).

Badr-Eddine Chérief-Abdellatif, Pierre Alquier, and Mohammad Emtiyaz Khan, A generalization bound for online variational inference , Asian Conference on Machine Learning, PMLR, 2019, pp. 662-677.

Sinho Chewi, Thibaut Le Gouic, Chen Lu, Tyler Maunu, and Philippe Rigollet, SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence , Advances in Neural Information Processing Systems 33 (2020), 2098-2109.

Shui-Nee Chow, Wuchen Li, and Haomin Zhou, Wasserstein Hamiltonian flows , Journal of Differential Equations 268 (2020), no. 3, 1205-1219.

Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton, A kernel test of goodness of fit , International conference on machine learning, PMLR, 2016, pp. 2606-2615.

Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, and Robert Scheichl, A Stein variational Newton method , Advances in Neural Information Processing Systems 31 (2018).

Michael Diao, Krishnakumar Balasubramanian, Sinho Chewi, and Adil Salim, Forwardbackward Gaussian variational inference via JKO in the Bures-Wasserstein space , arXiv preprint arXiv:2304.05398 (2023).

Justin Domke, Provable smoothness guarantees for black-box variational inference , International Conference on Machine Learning, PMLR, 2020, pp. 2587-2596.

Justin Domke and Daniel R Sheldon, Importance weighting and variational inference , Advances in neural information processing systems 31 (2018).

Andrew Duncan, Nikolas Nüsken, and Lukasz Szpruch, On the geometry of Stein variational gradient descent , Journal of Machine Learning Research 24 (2023), 1-39.

Yihao Feng, Dilin Wang, and Qiang Liu, Learning to draw samples with amortized Stein variational gradient descent , Conference on Uncertainty in Artificial Intelligence, 2017.

Nicolas Fournier, Convergence of the empirical measure in expected Wasserstein distance: Non asymptotic explicit bounds in R d , arXiv preprint arXiv:2209.00923 (2022).

Nicolas Fournier and Arnaud Guillin, On the rate of convergence in Wasserstein distance of the empirical measure , Probability theory and related fields 162 (2015), no. 3-4, 707-738.

Oded Galor, Discrete dynamical systems , Springer Science & Business Media, 2007.

Théo Galy-Fajou, Valerio Perrone, and Manfred Opper, Flexible and efficient inference with particles for the variational Gaussian approximation , Entropy 23 (2021), no. 8, 990.

Chengyue Gong, Jian Peng, and Qiang Liu, Quantile Stein variational gradient descent for batch Bayesian optimization , International Conference on Machine Learning, PMLR, 2019, pp. 2347-2356.

Jackson Gorham and Lester Mackey, Measuring sample quality with kernels , International Conference on Machine Learning, PMLR, 2017, pp. 1292-1301.

Jackson Gorham, Anant Raj, and Lester Mackey, Stochastic Stein discrepancies , Advances in Neural Information Processing Systems 33 (2020), 17931-17942.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine, Reinforcement learning with deep energy-based policies , International conference on machine learning, PMLR, 2017, pp. 1352-1361.

Ye He, Krishnakumar Balasubramanian, Bharath K Sriperumbudur, and Jianfeng Lu, Regularized Stein variational gradient flow , arXiv preprint arXiv:2211.07861 (2022).

Richard Jordan, David Kinderlehrer, and Felix Otto, The variational formulation of the FokkerPlanck equation , SIAM journal on mathematical analysis 29 (1998), no. 1, 1-17.

Mikołaj J Kasprzak, Ryan Giordano, and Tamara Broderick, How good is your Gaussian approximation of the posterior? Finite-sample computable error bounds for a variety of useful divergences , arXiv preprint arXiv:2209.14992 (2022).

Anya Katsevich and Philippe Rigollet, On the approximation accuracy of Gaussian variational inference , arXiv preprint arXiv:2301.02168 (2023).

Gabriel Khan and Jun Zhang, When optimal transport meets information geometry , Information Geometry 5 (2022), no. 1, 47-78.

Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow , Advances in Neural Information Processing Systems 29 (2016).

Anna Korba, Pierre-Cyril Aubin-Frankowski, Szymon Majewski, and Pierre Ablin, Kernel Stein discrepancy descent , International Conference on Machine Learning, PMLR, 2021, pp. 57195730.

Anna Korba and Adil Salim, Sampling as first-order optimization over a space of probability measures , Tutorial at the International Conference of Machine Learning, 2022, https:// akorba.github.io/resources/Baltimore_July2022_ICMLtutorial.pdf .

Anna Korba, Adil Salim, Michael Arbel, Giulia Luise, and Arthur Gretton, A non-asymptotic analysis for Stein variational gradient descent , Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 4672-4682.

Andreas Kriegl and Peter W Michor, The convenient setting of global analysis , vol. 53, American Mathematical Soc., 1997.

John D Lafferty, The density manifold and configuration space quantization , Transactions of the American Mathematical Society 305 (1988), no. 2, 699-741.

Marc Lambert, Sinho Chewi, Francis Bach, Silvère Bonnabel, and Philippe Rigollet, Variational inference via Wasserstein gradient flows , Advances in Neural Information Processing Systems, vol. 35, 2022.

Thomas Laurent, Local and global existence for an aggregation equation , Communications in Partial Differential Equations 32 (2007), no. 12, 1941-1964.

Michel Ledoux, On optimal matching of Gaussian samples , Journal of Mathematical Sciences 238 (2019), 495-522.

Michel Ledoux and Jie-Xiang Zhu, On optimal matching of Gaussian samples III , Probability and Mathematical Statistics 41 (2021).

Jing Lei, Convergence and concentration of empirical measures under Wasserstein distance in unbounded functional spaces , Bernoulli 26 (2020), no. 1, 767-798.

Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt, Fast and simple natural-gradient variational inference with mixture of exponential-family approximations , International Conference on Machine Learning, PMLR, 2019, pp. 3992-4002.

Chang Liu and Jun Zhu, Riemannian Stein variational gradient descent for Bayesian inference , Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.

Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, and Jun Zhu, Understanding and accelerating particle-based variational inference , International Conference on Machine Learning, PMLR, 2019, pp. 4082-4092.

Qiang Liu, Stein variational gradient descent as gradient flow , Advances in neural information processing systems, vol. 30, 2017.

Qiang Liu, Jason Lee, and Michael Jordan, A kernelized Stein discrepancy for goodness-of-fit tests , International conference on machine learning, PMLR, 2016, pp. 276-284.

Qiang Liu and Dilin Wang, Stein variational gradient descent: A general purpose Bayesian inference algorithm , Advances in neural information processing systems, vol. 29, 2016.

, Stein variational gradient descent as moment matching , Advances in Neural Information Processing Systems, vol. 31, 2018.

Xing Liu, Harrison Zhu, Jean-Francois Ton, George Wynne, and Andrew Duncan, Grassmann Stein variational gradient descent , International Conference on Artificial Intelligence and Statistics, PMLR, 2022, pp. 2002-2021.

Xingchao Liu, Xin Tong, and Qiang Liu, Sampling with trusthworthy constraints: A variational gradient framework , Advances in Neural Information Processing Systems 34 (2021), 2355723568.

John Lott, Some geometric calculations on Wasserstein space , Communications in Mathematical Physics 277 (2008), no. 2, 423-437.

Jianfeng Lu, Yulong Lu, and James Nolen, Scaling limit of the Stein variational gradient descent: The mean field regime , SIAM Journal on Mathematical Analysis 51 (2019), no. 2, 648-671.

Jan R. Magnus and Heinz Neudecker, Matrix differential calculus with applications in statistics and econometrics , Wiley Series in Probability and Statistics, John Wiley, 1988.

Luigi Malagò, Luigi Montrucchio, and Giovanni Pistone, Wasserstein Riemannian geometry of Gaussian densities , Information Geometry 1 (2018), 137-179.

Gael M Martin, David T Frazier, and Christian P Robert, Approximating Bayes in the 21st century , Statistical Science 1 (2023), no. 1, 1-26.

Boris Muzellec and Marco Cuturi, Generalizing point embeddings using the wasserstein space of elliptical distributions , Advances in Neural Information Processing Systems 31 (2018).

Yurii Evgenévich Nesterov, A method of solving a convex programming problem with convergence rate O ( k 2 ) , Doklady Akademii Nauk 269 (1983), no. 3, 543-547.

Manfred Opper and Cédric Archambeau, The variational Gaussian approximation revisited , Neural Computation 21 (2009), no. 3, 786-792.

Felix Otto, The geometry of dissipative evolution equations: the porous medium equation , Comm. Partial Differential Equations 26 (2001), 101-174.

Peter Petersen, Riemannian geometry , vol. 171, Springer, 2006.

Matias Quiroz, David J Nott, and Robert Kohn, Gaussian variational approximations for high-dimensional state space models , Bayesian Analysis 1 (2022), no. 1, 1-28.

Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows , International Conference on Machine Learning, PMLR, 2015, pp. 1530-1538.

Adil Salim, Anna Korba, and Giulia Luise, The Wasserstein proximal gradient algorithm , Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, eds.), vol. 33, Curran Associates, Inc., 2020, pp. 12356-12366.

Adil Salim, Lukang Sun, and Peter Richtarik, A convergence theory for SVGD in the population limit under Talagrand's inequality T1 , International Conference on Machine Learning, PMLR, 2022, pp. 19139-19152.

Jiaxin Shi, Chang Liu, and Lester Mackey, Sampling with mirrored Stein operators , International Conference on Learning Representations, 2022.

Jiaxin Shi and Lester Mackey, A Finite-Particle Convergence Rate for Stein Variational Gradient Descent , arXiv preprint arXiv:2211.09721 (2022).

Vladimir Spokoiny, Dimension free non-asymptotic bounds on the accuracy of high dimensional Laplace approximation , arXiv preprint arXiv:2204.11038 (2022).

Weijie Su, Stephen Boyd, and Emmanuel Candes, A differential equation for modeling Nesterov's accelerated gradient method: theory and insights , Advances in neural information processing systems, vol. 27, 2014.

Lukang Sun, Avetik Karagulyan, and Peter Richtarik, Convergence of Stein variational gradient descent under a weaker smoothness condition , International Conference on Artificial Intelligence and Statistics, PMLR, 2023, pp. 3693-3717.

Amirhossein Taghvaei and Prashant Mehta, Accelerated flow for probability distributions , International Conference on Machine Learning, PMLR, 2019, pp. 6076-6085.

Asuka Takatsu, Wasserstein geometry of Gaussian measures , Osaka Journal of Mathematics 48 (2011), no. 4, 1005-1026.

Linda SL Tan and David J Nott, Gaussian variational approximation with sparse precision matrices , Statistics and Computing 28 (2018), 259-275.

Gerald Teschl, Ordinary differential equations and dynamical systems , Graduate Studies in Mathematics 140 (2000), 08854-8019.

Marcin Tomczak, Siddharth Swaroop, and Richard Turner, Efficient low rank Gaussian variational inference for neural networks , Advances in Neural Information Processing Systems 33 (2020), 4610-4622.

Aad W van der Vaart, Asymptotic statistics , vol. 3, Cambridge University Press, 2000.

Jesse van Oostrum, Bures-Wasserstein geometry for positive-definite Hermitian matrices and their trace-one subset , Information Geometry 5 (2022), no. 2, 405-425.

Cedric Villani, Topics in optimal transportation , Graduate Studies in Mathematics 58, American Mathematical Society, 2003.

Dilin Wang, Ziyang Tang, Chandrajit Bajaj, and Qiang Liu, Stein variational gradient descent with matrix-valued kernels , Advances in neural information processing systems 32 (2019).

Yifei Wang, Peng Chen, and Wuchen Li, Projected Wasserstein gradient descent for highdimensional Bayesian inference , SIAM/ASA Journal on Uncertainty Quantification 10 (2022), no. 4, 1513-1532.

Yifei Wang and Wuchen Li, Accelerated information gradient flow , Journal of Scientific Computing 90 (2022), 1-47.

Lantian Xu, Anna Korba, and Dejan Slepcev, Accurate quantization of measures via interacting particle-based optimization , International Conference on Machine Learning, PMLR, 2022, pp. 24576-24595.

Rentian Yao and Yun Yang, Mean field variational inference via Wasserstein gradient flow , arXiv preprint arXiv:2207.08074 (2022).

Jingwei Zhuo, Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang, Message passing Stein variational gradient descent , International Conference on Machine Learning, PMLR, 2018, pp. 6018-6027.

A Further Discussion

Other Related Works. There exist several other approaches for minimizing KL-divergence over the Wasserstein space (or over appropriately restricted subsets). For example, normalizing flows [68, 37, 11], the blob method [10], variants of gradient descent in Wasserstein spaces [69, 85, 5], naturalgradient based variational inference techniques [48], mean-field methods [88], neural networks based approaches [2] and MCMC methods. This is certainly not a comprehensive list and summarizing the large literature on this topic is impossible in the limited space available. However, we emphasize that a majority of the above methods are nonparametric and do not come with strong theoretical guarantees. Our main focus in this work is on theoretically understanding (Gaussian)-SVGD and its connection to GVI.

Beyond Gaussian VI. While we present a comprehensive study of Gaussian-SVGD, it is currently unclear how such relation between bilinear kernels and Gaussian family could be generalized. For example, it would be interesting to ask if there is a class (submanifold) C of probability densities such that for any initialization-target pair ρ 0 , ρ ∗ ∈ C , the dynamics of SVGD with a radius-based kernel function (RBF-SVGD) would always remain in C . If such C is identified, we could similarly carry out the convergence analysis of RBF-SVGD and perform variational inference with respect to the class C . It remains an open question whether the uniform in time propagation of chaos as shown in Theorem 3.7 would hold for general kernels. Moreover, there has been increasing interest on variational inference with respect to other special density classes. Remarkably [43] considered Gaussian mixtures in additional to GVI, and [62] analyzed the family of elliptical distributions, which has applications for performing variational inference under heavy-tails [21].

SVGD in High Dimensions. It is well-known in literature [89, 4] that RBF-SVGD suffers from severe particle degeneracy in high dimensions and cannot guarantee good covariance estimation. However, such issue has not been observed for Gaussian-SVGD in our simulations. It would be interesting future works to study whether this undesired phenomenon is tied to specific choices of kernels and to carry out theoretical analysis of (Gaussian)-SVGD in high dimensions.

B Details on Gaussian-Stein Metrics

For any region Ω ⊆ R d , denote the set of probability densities on Ω by

where F (Ω) is the set of C ∞ -smooth functions on Ω . As studied in [42], P (Ω) forms a Fréchet manifold called the density manifold. For any 'point' ρ ∈ P (Ω) , the tangent space is given by

And the cotangent space at ρ , T ∗ ρ P (Ω) consists of equivalent classes of F (Ω) , each containing functions that differ by a constant. A Riemannian metric (tensor) assigns to each ρ ∈ P (Ω) a positive definite inner product g ρ : T ρ P (Ω) × T ρ P (Ω) → R . If we define the pairing between T ∗ ρ P (Ω) and T ρ P (Ω) by

where Φ ∈ T ∗ ρ P (Ω) and σ ∈ T ρ P (Ω) . Any Riemannian metric uniquely corresponds to an isomorphism (called the canonical isomorphism ) between the tangent and cotangent bundles [66], i.e., we have G ρ : T ρ P (Ω) → T ∗ ρ P (Ω) through

Definition B.1 (Wasserstein metric) . The Wasserstein metric is induced by the following canonical isomorphism G Wass ρ : T ρ P (Ω) → T ∗ ρ P (Ω) such that

Note that the Wasserstein metric we define here corresponds exactly to the Wasserstein2 distance studied in the literature of optimal transport [83]. For details of such connection, please refer to [36]. Also we need to clarify that the concepts we introduce follow from the convention in Riemannian geometry, where the manifold strictly speaking is required to be finite-dimensional. For

a mathematically formal formulation of infinite-dimensional calculus on the density manifold, please refer to [42, 41, 65, 57].

The Wasserstein gradient flow can be seen as the natural gradient flow on the density manifold with respect to the Wasserstein metric. In specific, consider any energy functional E ( ρ ) , e.g., the KL divergence from ρ to some fixed target ρ ∗ . The Wasserstein gradient flow for E ( ρ ) is provided by

where δE δρ t is the variational derivative of the functional E with respect to ρ t . If E ( ρ ) is the KL divergence mentioned above, (27) gives the linear Fokker-Planck equation [33]

where V ( x ) = -log ρ ∗ ( x ) is called the potential function. If E ( ρ ) is the Wasserstein metric between ρ and ρ ∗ , (1) gives the geodesic flow on the density manifold. If E ( ρ ) is the maximum mean discrepancy (MMD) or kernelized Stein discrepancy (KSD) between ρ and ρ ∗ , it leads to the MMD descent and KSD descent algorithms [3, 38].

Interestingly, the mean field flow of SVGD can also be seen as the gradient flows of the KL divergence on the density manifold but with respect to the Stein metric, where we perform kernelization in the cotangent space before taking the divergence [51, 22].

Definition B.2 (Stein metric) . The Stein metric is induced by the following canonical isomorphism G Wass ρ : T ρ P (Ω) → T ∗ ρ P (Ω) such that

In particular, the mean field PDE of the SVGD algorithm can be written as

where V ( x ) = -log ρ ∗ ( x ) .

We set Ω = R d and consider the multivariate Gaussian densities N ( µ , Σ) in R d :

where θ := ( µ , Σ) ∈ Θ and Θ := R d × Sym + ( d, R ) . Here Sym + ( d, R ) is the set of (symmetric) positive definite d × d matrices, which is an open subset of the d × d symmetric matrix space Sym( d, R ) . In this way, Θ can be identified as a Riemannian submanifold of the density manifold P ( R d ) with the induced Riemannian structure.

We first look into the Stein metric with the bilinear kernel K 1 ( x , y ) = x ⊤ y + 1 , and derive the closed form of the induced metric tensor on Θ , which plays an essential role in characterizing the SVGD dynamics.

Theorem B.3 (Gaussian-Stein metric with the simple bilinear kernel) . Given θ = ( µ , Σ) ∈ Θ , let g θ ( · , · ) denote the Gaussian-Stein metric tensor for the multivariate Gaussian model with the bilinear kernel K 1 ( x , y ) = x ⊤ y +1 , and G θ be the corresponding canonical isomorphism from T θ Θ ≃ R d × Sym( d, R ) to T ∗ θ Θ ≃ R d × Sym( d, R ) .

For any ν ∈ R n , and S ∈ Sym( d, R ) , the inverse map G -1 θ is given by the following automorphism on R d × Sym( d, R ) :

And for any ξ, η ∈ T θ Θ the Stein metric tensor can be written as

Here ξ = ( ˜ µ 1 , ˜ Σ 1 ) and η = ( ˜ µ 2 , ˜ Σ 2 ) , in which ˜ µ 1 , ˜ µ 2 ∈ R d , ˜ Σ 1 , ˜ Σ 2 ∈ Sym( d, R ) . And b i , S i 's

( i = 1 , 2 ) are defined as

Then for the Gaussian-Stein metric with kernel K 2 ( x , y ) = ( x -µ ) ⊤ ( y -µ ) + 1 we have the following result:

Theorem B.4 (Gaussian-Stein metric with the affine-invariant bilinear kernel) . Given θ = ( µ , Σ) ∈ Θ , let g θ ( · , · ) denote the affine-invariant Gaussian-Stein metric tensor for the multivariate Gaussian model with the affine-invariant bilinear kernel K 2 , and G θ be the corresponding canonical isomorphism from T θ Θ ≃ R d × Sym( d, R ) to T ∗ θ Θ ≃ R d × Sym( d, R ) .

For any ν ∈ R n , and S ∈ Sym( d, R ) , the inverse map G -1 θ is given by the following automorphism on R d × Sym( d, R ) :

And for any ξ, η ∈ T θ Θ the Stein metric tensor can be written as

Here ξ = ( ˜ µ 1 , ˜ Σ 1 ) and η = ( ˜ µ 2 , ˜ Σ 2 ) , in which ˜ µ 1 , ˜ µ 2 ∈ R d , ˜ Σ 1 , ˜ Σ 2 ∈ Sym( d, R ) . And S i 's ( i = 1 , 2 ) are defined as the symmetric solution of

Now if we further restrict the Gaussian family to have zero mean, it induces the submanifold Θ 0 = Sym + ( d, R ) . We have the following result for K 1 or K 2 as a direct corollary of Theorem B.3 or Theorem B.4.

Corollary B.5. Given Σ ∈ Θ 0 , let g Σ ( · , · ) denote the Gaussian-Stein metric tensor for the centered Gaussian family with the bilinear kernel K ( x , y ) = x ⊤ y + 1 , and G Σ be the corresponding canonical isomorphism from T Σ Θ 0 ≃ Sym( d, R ) to T ∗ Σ Θ 0 ≃ Sym( d, R ) .

For any S ∈ Sym( d, R ) , the inverse map G -1 Σ is given by the following automorphism on Sym( d, R ) :

And for any ˜ Σ 1 , ˜ Σ 2 ∈ T Σ Θ 0 the Stein metric tensor can be written as

where for i = 1 , 2 , S i is the unique solution in Sym( d, R ) that satisfies the Lyapunov equation

Next we consider the Bures-Wasserstein metric for Gaussian families, which is defined as the Wasserstein metric restricted to the Gaussian family. It has the following elegant expressions from [77, 60]:

Theorem B.6 (Bures-Wasserstein metric) . Given θ = ( µ , Σ) ∈ Θ , let g θ ( · , · ) denote the BuresWasserstein metric tensor for the multivariate Gaussian model (or equivalently Gaussian-Stein metric with the kernel K 3 ), and G θ be the corresponding isomorphism from T θ Θ ≃ R d × Sym( d, R ) to T ∗ θ Θ ≃ R d × Sym( d, R ) . For any ν ∈ R n , and S ∈ Sym( d, R ) , the inverse map G -1 θ is given by the following automorphism on R d × Sym( d, R ) :

And for any ξ, η ∈ T θ Θ the Bures-Wasserstein metric tensor can be written as

Here ξ = ( ˜ µ 1 , ˜ Σ 1 ) and η = ( ˜ µ 2 , ˜ Σ 2 ) , in which ˜ µ 1 , ˜ µ 2 ∈ R d , ˜ Σ 1 , ˜ Σ 2 ∈ Sym( d, R ) . And S i 's ( i = 1 , 2 ) are defined as the symmetric solution of

Notably this metric coincides with Gaussian-Stein metric with the kernel K 3 ( x , y ) = ( x -µ ) ⊤ Σ -1 ( y -µ ) + 1 .

Note that the only difference between (32) and (36) is the power of Σ .

Theorem B.7 (Regularized Gaussian-Stein metric with the affine-invariant bilinear kernel) . Given θ = ( µ , Σ) ∈ Θ , let g θ ( · , · ) denote the affine-invariant regularized Gaussian-Stein metric tensor for Gaussian families, and G θ be the corresponding isomorphism from T θ Θ ≃ R d × Sym( d, R ) to T ∗ θ Θ ≃ R d Sym( d, R ) . For any ν ∈ R d and S ∈ Sym( d, R ) , the inverse map G -1 θ is given by the following automorphism on R d × Sym( d, R ) :

Notably this metric coincides with Gaussian-Stein metric with the kernel K 4 ( x , y ) = ( x -µ ) ⊤ ((1 -ν )Σ + νI ) -1 ( y -µ ) + 1 .

C Properties of SVGD Solutions with the Simple Bilinear Kernel

[58] showed a few nice properties of the mean field PDE (2) for SVGD with radius-based kernels that can be written as K ( x -y ) which is symmetric and positive definite, meaning that

Although their results covered a large class of kernels commonly used in practice e.g., Gaussian kernels, they do not apply to the bilinear kernel K ( x , y ) = x ⊤ y +1 . In this subsection we establish similar results for the bilinear kernel. In fact, some of their results for radius-based kernels still hold here while some do not.

Before showing the properties of the mean field PDE, we need the following assumption on the potential function V .

Assumption C.1. The potential function V : R d → R satisfies the conditions below:

V ∈ C ∞ ( R d ) , V ≥ 0 , and V ( x ) →∞ as ∥ x ∥ → ∞ .

For any α, β > 0 , there exists a constant C α,β > 0 such that if ∥ y ∥ ≤ α ∥ x ∥ + β , then the following inequality always holds that

To make things precise although all vector norms in R d and all matrix norms are equivalent, we always choose the Euclidean norm for vectors and the spectral norm for matrices unless otherwise specified. Note that our assumption here is a little different from Assumption 2.1 in [58]. We do not require their second formula but our second piece of assumption is slightly stricter than their third one. It is straightforward to check that any positive definite quadratic form satisfies all the assumptions above, corresponding to the case where the target is a non-degenerate Gaussian distribution.

We use P V ( R d ) and P p ( R d ) ( p = 1 , 2 , · · · ) to denote the set of probability measure µ on R d satisfying

and respectively.

Theorem C.2 (Well-posedness and regularity of the mean field solution) . Let V satisfy Assumption C.1. For any ν ∈ P V ( R d ) , there is a unique ρ t ∈ C ( [0 , ∞ ) , P V ( R d ) ) which is a weak solution to (2) with initial condition ρ 0 = ν . Moreover, there exists C 1 > 0 depending on V such that

If ν ∈ P p ( R d ) ∩ P V ( R d ) , then for any t ∈ [0 , ∞ ) , we have that ρ t ∈ P p ( R d ) ∩ P V ( R d ) and there exists C 2 > 0 depending on V such that

If ν has a density ρ 0 ( x ) ≥ 0 , then ρ t also has a density. Furthermore, if ρ 0 ∈ H k ( R d ) for some k , then we have ρ t ∈ H k ( R d ) . Here

denotes the Sobolev (Hilbert) space of order k .

Theorem C.3 (Well-posedness of the finite-particle solution) . Let V satisfy Assumption C.1. Then for any initial condition X 0 = ( x (0) 1 , · · · , x (0) N ) ⊤ ∈ R dN , the system (13) has a unique solution

and the measure µ N t = 1 N ∑ N i =1 δ x ( t ) i is a weak solution to the PDE (2) .

Finally, we show that if two initial probability measures are close to each other, the solutions of (2) up to time T are also close. We need to further impose the following assumption on V .

Assumption C.4. There exists a constant C V > 0 and some index q > 1 such that

and that

Remarkably a Gaussian distribution satisfies both Assumptions C.1 and C.4. Secondly an important observation is that (42) implies that there is C 0 such that

where q ′ = q/ ( q -1) . Indeed, we note that

Integrating from t = 0 to t = ∥ x ∥ , we get (44), which shows that P p ⊂ P V for any p ≥ q ′ = q/ ( q -1) .

Theorem C.5. Let V satisfy Assumptions C.1 and C.4. Let R > 0 . Assume that ν 1 , ν 2 are two initial probability measures in P p ( R d ) satisfying ∥ ν i ∥ P p ≤ R ( i = 1 , 2 ). Let µ 1 ,t and µ 2 ,t be the associated weak solutions to (2) . Then given any T > 0 , there exists a constant C T > 0 depending on V, R and T such that

Remark. 1. Theorem C.5 implies the convergence of empirical measure to the mean field limit at time t ∈ [0 , T ] . In fact, if we set ν 1 ,n to be an empirical measure, which converges to ν 2 as n grows to infinity, then µ 1 ,n,t , the solution of (2) at time t with initializaiton ν 1 ,n , will also converge to µ 2 ,t for any t ∈ [0 , T ] .

In general there is no guarantee that the density ρ t will converge to the target density ρ ∗ in (2) with the bilinear kernel. Some counterexamples will be elaborated in the remarks following Theorem 3.6.

We show in Theorem 3.1 that for Gaussian families ρ t always converges to the target density ρ ∗ as t → ∞ and the convergence rate is linear in KL divergence. We also establish a uniform in time convergence result (Theorem 3.7) for the empirical measure in this case.

D Details of the Gaussian-SVGD Algorithms

Different Ways of Estimating Γ t . The first-order estimator of Γ t arises from

Since Γ t is symmetric we also have

Note that using the first-order estimator also comes at a cost as the inverse of Σ is needed. However, for density-based Gaussian-SVGD this Σ -1 might cancel with Σ in computing.

Previous Algorithms Under the Proposed Frameworks. The use of K 1 for SVGD in variational inference dates back to [54]. Our Algorithm 2 is slightly different from [54] in the sense that ∇ V is replaced by a linear function to ensure Gaussianity. Moreover, Algorithms 1 and 2 with the kernel K 2 correspond precisely to the GF and GPF algorithms in [27]. If K 3 (Bures-Wasserstein metric) is chosen, Algorithm 1 reproduces the BW-SGD algorithm in [43] (with N = 1 ). [19] also uses K 3 (Bures-Wasserstein metric) but their energy function for gradient flow is different from others. Instead of directly performing the gradient descent to minimize KL divergence, they separate the KL divergence into two parts and perform the proximal gradient descent.

Variants of Gradient Descent in Density-Based Gaussian-SVGD. For the density-based SVGD, we draw new samples at each step. It is interesting to study the behaviorial difference between drawing one sample (stochastic) and efficiently many samples (almost deterministic). For example, in [43] only one sample is drawn at each time step and they study the stochastic properties arising from this design. [19] considers both settings. In general, they do not differ much in convergence rates but there could be huge gaps in the constants of the bounds and will actually impact practical performance. Another choice is to only draw N samples at time 0 and we use a linear transformation of the same N points to serve as new samples at time t , which becomes similar to the particle-based Gaussian-SVGD. Moreover, the vanilla gradient descent could also be replaced by accelerated ones or gradient descent with adaptive learning rate, e.g., AdaGrad, RMSProp.

Resampling Scheme and Particle-Level Convergence. As presented in the main text the GaussianSVGD for a general target is given by

where ̂ ∇ V ( x ) = ̂ Γ t ( x -µ t ) + ̂ m t .

The updating rules of Gaussian-SVGD given above is totally deterministic, meaning that at each time step x ( t ) i is updated only using deterministic quantities and ( x ( t ) k ) N k =1 without any external randomness. This is computationally more efficient but imposes difficulty in the analysis. On the other hand, we could also consider slightly modifying the updating rules by applying a resampling scheme to get a better estimation of m t and Γ t . In other words, we resample ( y k ) M k =1 i.i.d. from N ( µ t , Σ t ) and use the following estimators

or first-order estimators

In this way, when M is large enough ̂ ∇ V ( x ) will be sufficiently close to Γ t ( x -µ t ) + m t .

0

25

50

75

100

125

150

175

200

time

5

4

3

2

1

0

1

2

3

logKL(

)

SBGD (

K

1)

GF (

K

2)

BWGD (

K

3)

RGF (

K

4)

SBPF (

K

1)

GPF (

K

2)

BWPF (

K

3)

RGPF (

K

4)

Now we replace ̂ m t and ̂ Γ t by m t and Γ t and consider the continuous-time dynamics

where µ t and Σ t are sample mean and variance and

Theorem D.1 (Equivalence of density-based and particle-based algorithms) . The solution of the finite-particle system (48) with K 1 is given by x ( t ) i = A t ( x (0) i -µ 0 ) + µ t where A t is the unique solution of

where µ t and Σ t are the unique solution of the ODE system

This can be proved using exactly the same technique as in the proof of Theorem 3.6. Here (49) is the same as (22), and hence by Theorem 4.1 µ t and Σ t converges to µ ∗ and Σ ∗ that solves the GVI and the convergence rate is given in Theorem 4.2 when the target is strongly log-concave. We also conjecture that there is still uniform in time convergence to the mean-field limit for this particle system and leave it to future works. .

E Details of the Simulations

Gaussian Targets. Following [19], we consider a scenario where the target is Gaussian N ( µ , Σ) where µ ∼ Unif([0 , 1] 10 ) and Σ -1 = U diag { λ 1 , λ 2 , · · · , λ 10 } U ⊤ with U ∈ R 10 × 10 drawn from the Haar measure of the orthogonal matrices O (10) and λ 1 , · · · , λ 10 being a geometric sequence such that λ 1 = 0 . 01 and λ 10 = 1 . We run the eight different algorithms as introduced in Section 5 and show the decay of log KL( ρ θ ∥ ρ ∗ ) over time in Figure 2. Clearly the algorithms with K 1 show a faster rate over time compared to the others while the other algorithms eventually all converge at the same rate. This is actually confirmed from our theoretical analysis as in all the other dynamics except SBGD and SBPF, the mean converges at the rate of O ( e -t/λ ) while the covariance converges at a faster rate, resulting in the fact that the KL divergence converges at O ( e -2 t/λ ) . But for K 1 the rate is different and given in Theorem 3.1.

Gaussian Mixture Targets. Next we consider the 1 -dimensional Gaussian mixture targets given by w 1 N ( µ 1 , σ 2 1 ) + (1 -w 1 ) N ( µ 2 , σ 2 2 ) . We run the aforementioned eight algorithms with initial

0

20

40

60

80

100

time

2.0

1.5

1.0

0.5

0.0

(

)

SBGD (

K

1)

GF (

K

2)

BWGD (

K

3)

RGF (

K

4)

SBPF (

K

1)

GPF (

K

2)

BWPF (

K

3)

RGPF (

K

4)

(a)

̂

F

(

ρ

θ

)

over time.

0

100

200

300

400

500

time

2.0

1.5

1.0

0.5

0.0

(

)

SBGD (

K

1)

GF (

K

2)

BWGD (

K

3)

RGF (

K

4)

SBPF (

K

1)

GPF (

K

2)

BWPF (

K

3)

RGPF (

K

4)

(b)

̂

F

(

ρ

θ

)

over iteration.

µ = 0 and σ = 1 or particles drawn from N (0 , 1) . Here again we plot the decay of ̂ F ( ρ θ ) over time or iteration as shown in Figure 3. In particular, the plots correspond to the specific setting of µ 1 = 5 , µ 2 = 10 , σ 1 = 5 , σ 2 = 2 and ρ ∗ ( x ) ∝ 0 . 3 exp( -( x -5) 2 / 50) + 0 . 7 exp( -( x -10) 2 / 8) . These parameters are arbitrarily chosen. For the decay of ̂ F ( ρ θ ) over time, we fix the step size to be 0 . 1 and run 1000 iterations. For ̂ F ( ρ θ ) over time, we draw 500 particles for particle-based algorithms and run all algorithms for 500 iterations so that a total of 500 samples are drawn for the density-based algorithms. The step sizes are chosen to be the largest ones that still allow convergence. Specifically for these eight algorithms the step sizes are 0 . 02 , 0 . 1 , 1 , 1 , 0 . 2 , 0 . 8 , 8 , 8 . Consistent with the results of Bayesian logistic regression, the particle-based ones are more stable and allow larger step sizes, with BWPF and RGPF clearly outperforming all the others. In fact, the constrast between particle-based and density-based algorithms is particularly significant in this problem probably because of the non-log-concave target.

More on the Bayesian Logistic Regression. We compare three so-far best performed algorithms, BWPF, RGPF, and FB-GVI [19] for the same problem with different step sizes. From Figure 4 we see that BWPF outperforms the other two. FB-GVI is better than RGPF with a larger learning rate but fluctuate a bit more when η = 2 . This is probably attributed to the stochastic gradients. Furthermore, it is interesting to compare to ordinary gradient descent (OGD) on the variational parameters (mean and covariance) and SVGD with a radius-based kernel function (RBF-SVGD) ∥ x -y ∥ 2 . also notice that RBF-SVGD is significantly slower in computation compared to Gaussian-SVGD.

K h ( x , y ) = exp ( -2 h 2 ) . We show this comparison in Figure 5 with the same step size η = 2 Firstly, OGD does not converge as fast as BWPF and it is not as stable. Secondly, RBF-SVGD is quite sensitive to the choice of bandwidth and it does not converge as fast as BWPF in general. We However, as a particle-based algorithm, RBF-SVGD does have the advantage of being stable even when the step size is large.

F Analogous Results for the Affine-Invariant Bilinear Kernels

For the Bures-Wasserstein metric (Gaussian-Stein metric with K 3 ), the dynamics of natural gradient descent has already been studied in literature. See [60, 86, 13] for proofs for the following theorem.

Theorem F.1 (Wasserstein gradient flow for the Gaussian family) . Let ρ 0 ∼ N ( µ 0 , Σ 0 ) and ρ ∗ ∼ N ( b , Q ) be two Gaussian measures. Then the solution of (1) converges to ρ ∗ as t →∞ . In particular, ρ t is the density of N ( µ t , Σ t ) where the mean µ t and covariance matrix Σ t satisfies

If Σ 0 Q = Q Σ 0 , we have ∥ µ t -b ∥ = O ( e -t/λ ) and ∥ Σ t -Q ∥ = O ( e -2 t/λ ) , where λ is the largest eigenvalue of Q .

Now we focus on the results for Gaussian-SVGD with the kernel K 2 . First we remark that for K 2 the previous results on the well-posedness of mean-field PDE and finite-particle solutions in Appendix C still hold and can be proved using similar techniques to Appendix H but for simplicity they are

0

100

200

300

400

500

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

(

)

BWPF (

=8)

BWPF (

=2)

BWPF (

=0.5)

RGPF (

=8)

RGPF (

=2)

RGPF (

=0.5)

FB-GVI (

=8)

FB-GVI (

=2)

FB-GVI (

=0.5)

iteration

0

100

200

300

400

500

12

14

16

18

20

(

)

BWPF

OGD

SVGD (

h

=1)

SVGD (

h

=2)

SVGD (

h

=5)

SVGD (

h

=10)

SVGD (

h

=20)

iteration

omitted. The proofs of the following theorems will also be omitted unless a different proof technique from the analogous results needs to be applied.

Theorem F.2 (Analogue of Theorem 3.1) . For any t ≥ 0 the solution ρ t of SVGD (2) with the bilinear kernel remains a Gaussian density with mean µ t and covariance matrix Σ t given by

which has a unique global solution on [0 , ∞ ) given any µ 0 ∈ R d and Σ 0 ∈ Sym + ( d, R ) . And ρ t converges weakly to ρ ∗ as t →∞ . If Σ 0 Q = Q Σ 0 then we have the following rates

where λ is the largest eigenvalue of Q .

The proof of the dynamics is similar to that of Theorem 3.1. The rate O ( e -t/λ ) is trivially true from the theory of linear ODEs and the rate O ( e -2 t ) is given by Theorem 3.2.

Theorem F.3 (Analogue of Theorem 3.6) . Suppose the initial particles satisfy that C 0 is non-singular. There exists a unique solution of the finite particle system (13) given by

where A t is the unique (matrix) solution of the linear system

and µ t and C t are the unique solution of the ODE system

The proof is similar to that of Theorem 3.6.

Theorem F.4 (Analogue of Theorem 3.7) . Given the same setting as Theorem F.3, further suppose the initial particles are drawn i.i.d. from N ( µ 0 , Σ 0 ) . Then there exists a constant C d,Q, b , Σ 0 , µ 0 such that for all t , for all N ≥ 2 , with the empirical measure ζ ( t ) N = 1 N ∑ N i =1 δ x ( t ) i , the second moment of Wasserstein2 distance between ζ ( t ) N and ρ t converges:

The proof is similar to that of Theorem 3.7. But for sake of completeness we provide more proof details in Appendix L.

Theorem F.5 (Analogue of Theorem 4.1) . Let ρ ∗ be the density of a target distribution with the potential function V ( x ) = -log ρ ∗ ( x ) and ρ 0 be the density of N ( µ 0 , Σ 0 ) . Then for any t ≥ 0 the Gaussian-SVGD produces a Gaussian density ρ t with mean µ t and covariance matrix Σ t given by the following ODE system:

Furthermore, suppose that θ ∗ is the unique solution of the following optimization problem

Then we have ρ t → ρ θ ∗ ∼ N ( µ ∗ , Σ ∗ ) as t →∞ .

The proof is similar to that of Theorem 4.1. In particular, if the target is strongly log-concave, it gives rise to the following linear convergence rate.

Theorem F.6 (Analogue of Theorem 4.2) . Assume that the target ρ ∗ is α -strongly log-concave and β -log-smooth, i.e., αI ⪯ ∇ 2 V ( x ) ⪯ βI . Then ρ t converges to ρ θ ∗ at the following rate:

The proof is similar to that of Theorem 4.2.

Theorem F.7 (Analogue of Theorem D.1) . The solution of the finite-particle system (48) with K 2 is given by x ( t ) i = A t ( x (0) i -µ 0 ) + µ t where A t is the unique solution of

where µ t and Σ t are the unique solution of the ODE system

The proof is similar to that of Theorem 3.6 or Theorem D.1.

G Proofs for Section B

Lemma G.1. Let ρ ( x ) = (2 π ) -d/ 2 ( det(Σ) ) -1 / 2 exp ( -1 2 x ⊤ Σ -1 x ) be the density of a d -dimensional normal random vector where Σ is a positive definite matrix. Then for any d × d real matrix A and d -dimensional real vector b , we have

Proof. Since Σ -1 is a positive definite matrix, we can find its positive definite root Σ -1 / 2 . Let y = Σ -1 / 2 x and ρ 0 ( x ) = (2 π ) -d/ 2 exp ( -1 2 x ⊤ x ) . Then we have

The second equation is given by

Lemma G.2 (Lyapunov equation) . The Lyapunov equation

has a unique solution. If P, Q ∈ Sym( d, R ) , then the solution X ∈ Sym( d, R ) .

Proof. By Sylvester-Rosenblum theorem in control theory [7], PX + XP = Q has a unique solution X . Note that if P, Q ∈ Sym( d, R ) and X is a solution, then X ⊤ is also a solution. Thus, we have X = X ⊤ which implies that X ∈ Sym( d, R ) .

Proof of Theorem B.3. Note that the tangent space at each θ ∈ Θ = R d × Sym + ( d, R ) is T θ Θ ≃ R d × Sym( d, R ) . If we define the inner product (pairing) on T θ Θ by

for any ξ, η ∈ T θ Θ , where ξ = ( ˜ µ 1 , ˜ Σ 1 ) and η = ( ˜ µ 2 , ˜ Σ 2 ) , then the tangent bundle T Θ is trivial. Since

provides an immersion from Θ to P ( R d ) , we consider its pushforward d ϕ θ given by

On the other hand, for any Φ ∈ T ∗ ρ P ( R d ) , the inverse canonical isomorphism of Stein metric maps it to

Thus, we obtain that

for any x ∈ R d .

Now we try to find a suitable function ∇ Φ ξ that satisfies the equation above. We compute that

Letting Ψ( x ) = ∫ K ( x , y ) ρ ( y , θ ) ∇ Φ( y ) d y , we get

We choose ∇ Φ ξ ( x ) = S 1 ( x -µ ) + b 1 , where S 1 ∈ Sym( d, R ) and b 1 ∈ R d will be determined later. Note that S 1 needs to be symmetric because the gradient field is curl-free. We derive that

By comparison of the coefficients, we need

Note that the Lyapunov equation

has a unique solution X = S 1 ∈ Sym( d, R ) . Together with

where

we find that (59) holds with the unique solution described above. Now from the calculations above it is straightforward to check that (59) is equivalent to the following equation

where G -1 θ is the map defined in (29). Thus, the existence and uniqueness of the solution indicates that G θ is an isomorphism.

Similarly, we let where

and

where X = S 2 ∈ Sym( d, R ) is the unique solution of the Lyapunov equation

Now we compute the Riemannian tensor

Finally, we show that G θ is indeed the canonical isomorphism corresponding to g θ ( · , · ) . We check that

Proof of Theorem B.7. Similar to the proof of Theorem B.3, we define the inner product on T θ Θ by

⟨

ξ, η

⟩

:= tr(Σ

1

Σ

2

) +

µ

⊤

1

µ

2

,

for any ξ, η ∈ T θ Θ , where ξ = ( ˜ µ 1 , ˜ Σ 1 ) and η = ( ˜ µ 2 , ˜ Σ 2 ) . The map

where ρ ( · , Σ) denotes the density of N ( µ , Σ) , provides an immersion from Θ to P ( R d ) . We consider its pushforward d ϕ θ given by

On the other hand, for any Φ ∈ T ∗ ρ P ( R d ) , the inverse canonical isomorphism of regularized Stein metric maps it to

Thus, we obtain that

for any x ∈ R d .

Now we try to find a suitable function ∇ Φ ξ that satisfies the equation above. We compute that

Letting Ψ( x ) = ((1 -ν ) T K,ρ + νI ) -1 T K,ρ ( ∇ Φ( x )) , we get that the RHS of (60) is equal to

We choose ∇ Φ ξ ( x ) = S 1 ( x -µ ) + b 1 , where S 1 ∈ Sym( d, R ) and b 1 ∈ R d will be determined later. Note that S 1 needs to be symmetric because the gradient field is curl-free. We derive that

Thus, we know Ψ( x ) = S 1 Σ((1 -ν )Σ + νI ) -1 x + b 1 . By comparison of the coefficients, we need b 1 = ˜ µ 1 and

Note that the first equation is equivalent to the following Lyapunov equation

which has a unique solution X = S 1 ∈ Sym( d, R ) , and the second equation is automatically satisfied once we have the first one. Now from the calculations above it is straightforward to see that

The proofs of Theorems B.4 and B.6 are similar to that of Theorems B.3 and B.7. In particular, other proofs of Theorem B.6 can also be found in literature, for example, see [77, 60]. Finally, Corollary B.5 is the direct corollary of Theorem B.3 or Theorem B.4.

H Proofs for Section C

Given a probability measure µ and a Borel-measurable map f , we denote by f # µ the pushforward of the measure µ under the map f .

Definition H.1 (Mean field characteristic flow) . Given a probability measure ν , we say that the map

is a mean field characteristic flow associated to the particle system (13) or to the mean field PDE (2) if X is C 1 in time and solves the following problem

Note that here X ( t, · , ν ) # ν is the push-forward of ν under the map x ↦→ X ( t, · , ν ) , and { X ( t, · , ν ) } t ≥ 0 ,ν can be regarded as a family of maps from R d to R d , parameterized by t and ν . In the lemma below we show that the mean field characteristic flow (62) is well-defined.

LemmaH.2 (Solution of the mean field characteristic flow) . Assume the conditions of Assumption C.1 hold, and ν ∈ P V ( R d ) . For any T > 0 , there exists a unique solution X ( · , · , ν ) ∈ C 1 ([0 , T ] , Y ) to the problem (62) , where Y is the function space given by

Moreover, the measure µ t = X ( t, · , ν ) # ν satisfies

for some constant C that is independent of ν .

Proof. We prove the lemma in two steps. First we show local well-posedness of the mean field characteristic flow. Second we extend the local solution to t ∈ [0 , ∞ ) .

Fix r > 0 , and we define

We show that there exists T 0 > 0 such that (62) has a unique solution X ( t, x , ν ) on t ∈ [0 , T 0 ] , and the solution is in the following function class

which is a complete metric space equipped with the uniform metric

Now we check the integral formulation of (62) given by

Let us define the operator F : u ( t, · ) ↦→F ( u )( t, · ) by

We aim to show that F is a contraction map in S r , and thus, it has a unique fixed point. For this purpose we first prove that F maps S r into S r . It is straightforward to check that ( t, x ) ↦→F ( u )( t, x ) . We now need to establish a bound on ∥ ∥ F ( u )( t, x ) -x ∥ ∥ . If u ∈ S r , then for any s ∈ [0 , T 0 ] and x ∈ R d

By Assumption C.1, we have that

As a consequence, we have

for some constant ˜ C r , where we used the assumption that ν ∈ P V ( R d ) . Therefore, choosing T 0 ≤ r/ ˜ C r we get

which shows that F maps from S r to S r for sufficiently small T 0 . Next, we prove that F is indeed a contraction map. If u, v ∈ S r , then for any t ∈ [0 , T 0 ] and x ∈ R d

Term I can be upper-bounded by

Similarly we have

For the third term, we apply Assumption C.1 and get

Thus, we have

Finally applying Assumption C.1 once again, we have

where in ( ∗ ) we have used the fact that λu +(1 -λ ) v ∈ S r , and thus, λu +(1 -λ ) v also satisfies the inequality (3.3), which enables us to apply Assumption C.1.

Thus, Term IV is also bounded from above by (the same as the upper bound of Term III)

Now combining (64)-(67), we conclude that F is a contraction on S r when T 0 is small enough. By the contraction mapping theorem, F has a unique fixed point X ( · , · , ν ) ∈ S r , which solves (63). After defining µ t = X ( t, · , ν ) # ν , one sees that X ( t, x, ν ) solves (62) in the small time interval [0 , T 0 ] .

Now we proceed to the second step of extending the local solution. Define

If τ = ∞ , then we have a global solution. Otherwise suppose τ < ∞ . After examining the bounds we have established in the previous step, we can see that supposing the local solution exists at some time T 0 , it may be extended beyond T 0 as long as the quantity

is finite at time T 0 . We therefore establish an upper bound on this quantity.

The last inequality follows from Assumption C.1. Therefore, by Grönwall's inequality we get

holds for all t ∈ [0 , τ ) . Next we show an upper bound on ∥ X ( t, x , ν ) ∥ . We derive that

Again the first inequality follows from Assumption C.1. Thus, by Grönwall's inequality we have

for all t ∈ [0 , τ ) . This also implies that

for all t ∈ [0 , τ ) .

Next we extend the solution to t ∈ [0 , τ ] . For this purpose, we first prove that for any sequence { t i } ∞ i =1 such that 0 < t 1 < t 2 < · · · < τ and lim i →∞ t i = τ , the sequence of functions { X ( t i , · , ν ) } is Cauchy in Y . Then by completeness of Y , there is a limiting function for the sequence, and it is straightforward to see that such function is unique (does not rely on the choice of the sequence { t i } ∞ i =1 ).

In fact, we check that for any i < j

Thus, for any ϵ > 0 there exists N > 0 such that for any j > i > N , we have

In other words, { X ( t i , · , ν ) } is a Cauchy sequence in Y .

Now we know that (62) has a unique solution on [0 , τ ] . Since ∥ µ τ ∥ P V ( R d ) < ∞ , we can further find a unique solution of (62) on [ τ, τ + T 0 ] for some T 0 small enough, which contradicts the definition of τ . Therefore, we conclude that τ = ∞ , and (62) has a unique global solution. Finally, thanks to the integral formulation (63) ˙ X is continuous on [0 , ∞ ) × R d . The proof is complete.

Proof of Theorem C.2. Given ν , let X ( t, x, ν ) be the mean field characteristic flow defined in (62), and let ρ t = X ( t, · , ν ) # ν . Note that (2) can be rewritten as

where U [ ρ ] is the vector field given by

Then ρ t is a weak solution to (2) in the sense that

and holds for all ϕ ∈ C ∞ 0 ( [0 , ∞ ) × R d ) . This is either directly checked or follows immediately from Theorem 5.34 in [83].

By Lemma H.2 there exists some constant C 1 such that

Suppose that ν ∈ P p ( R d ) ∩ P V ( R d ) . As shown in the proof of Lemma H.2, the map X ( t, x , ν ) is an element of the space Y with d Y ( X ( t, x , ν ) , x ) ≤ C 1 e C 1 t . Therefore, since

we have

for some constant C 2 > 0 and ρ t ∈ P p ( R d ) ∩ P V ( R d ) .

Wenowexplain that the uniqueness of the weak solution follows from the uniqueness of the mean field characteristic flow. Suppose q ∈ C ( [0 , T ] , P V ( R d ) ) is another weak solution to (2). By definition of the weak solution, the vector field ( t, x ) ↦→ U [ q t ] ( x ) is bounded over [0 , T ] × R d , continuous in t and Lipschitz continuous in x . Then we can define a continuous family of maps ˜ X ( t, · , ν ) by

And the measure ˜ q t = ˜ X ( t, · , ν ) # ν is a weak solution to the transport equation

with initial condition ˜ q 0 = ν = q 0 . Uniqueness of the solution to this linear equation implies that ˜ q t = q t . Thus, we have ˜ X ( t, · , ν ) # ν = q t . In other words, ˜ X ( t, x , ν ) is the mean field characteristic flow for ν . Uniqueness of the characteristic flow implies that ˜ X = X , and hence q t = ρ t . Thus, we conclude that the weak solution is unique.

Lastly, we show the regularity result: If ν has a density ρ 0 ( x ) ≥ 0 , then ρ t also has a density. Furthermore, if ρ 0 ∈ H k ( R d ) for some k , then we have ρ t ∈ H k ( R d ) .

Note that we have already proven that ρ t ∈ C ([0 , T ] , P V ) and

Thus, U ( t, x ) := U [ ρ t ]( x ) ∈ C ( [0 , T ] , C k +1 b ( R d ) ) where C k +1 b ( R d ) is the space of continuous functions with bounded ( k +1) -th order derivatives. Let Φ t ( x ) = X ( t, x , ν ) denote the characteristic flow. Since Φ t satisfies the ODE system

from the regularity theory of ODE systems (see Chapter 2 of [79]) we know that both the map x ↦→ Φ t and its inverse Φ -1 t are C k . Therefore, if ρ 0 has a density, then ρ t also has a density and it is given by

Moreover, since ρ satisfies

Noting that we have

for any T ≥ 0 .

with the vector field U ( t, x ) ∈ C ( [0 , T ] , C k +1 b ( R d ) ) , it follows from Lemma 2.8 of [44] that

Proof of Theorem C.3. We show that the particle system (13) is well-posed and that the empirical measure is a weak solution to the mean field PDE. We introduce the function

Since V is C 1 hence locally Lipschitz, by Picard-Lindelöf theorem the problem (13) has a unique solution up to some time T 0 > 0 . Intuitively we only need to show that the solution does not blow up at any finite time. We claim that for some constant C ,

To establish this, we first differentiate V ( x i ( t )) with respect to t and sum over i :

Note that here we have used Assumption C.1. By Grönwall's inequality, (68) holds.

Now to be rigorous, once again we define

τ := sup { t ∈ R + ∪ {∞} : (13) has a (unique) solution on [0 , t ) } .

If τ < ∞ , we define and

Then (13) has a unique solution on [0 , τ ] . Again by Picard-Lindelöf theorem, there exists some ϵ > 0 such that (13) has a unique solution on [ τ, τ + ϵ ] , which contradicts the definition of τ . Thus, we conclude that τ = ∞ , which means that there is a global unique solution to (13).

Having established the well-posedness of the finite particle system, it now follows from the definition of the characteristic flow X ( t, x , µ N 0 ) that

Similar to the proof of Theorem C.2, we conclude that µ N t is a weak solution to the mean field PDE (2).

Finally, we show Theorem C.5.

Proof of Theorem C.5. Recall that p = q ′ = q/ ( q -1) . By assumption ∥ ν i ∥ P p ≤ R < ∞ and the fact that P p ( R d ) ⊂ P V ( R d ) , we know that there exists C > 0 depending on R such that

From the proof of Theorem C.2 and Definition H.1, we know that the weak solutions µ i,t take the form

Now we bound W p p ( µ 1 ,t , µ 2 ,t ) using W p p ( ν 1 , ν 2 ) . Let π 0 be a coupling between ν 1 and ν 2 . For δ > 0 define ϕ δ ( x ) := 1 p ( ∥ x ∥ + δ ) p/ 2 to be an approximation to 1 p ∥ x ∥ p . Given any two points x 1 , x 2 ∈ R d , we have that

Below we bound I i individually. First, noticing that

we obtain

Next we bound I 2 :

Here we have applied Hölder's inequality in a and Theorem C.2 in c . The inequality b is due to Assumption C.4 and the definitions of the P V -norm and P p -norm.

Similarly for I 3 we use Hölder's inequality again and get

Finally we proceed to bound I 4 . An application of the intermediate value theorem to the difference of ∇ V yields that

Note that to get a we have applied (43) of Assumption C.4 and b is implied by Theorem C.2. If we define

combining (70), (71), (73) and (74) we obtain that

Now integrating the inequality above with respect to the coupling π 0 (d x 1 , d x 2 ) using the fact that

Thus, we obtain that

Integrating t we get

Finally letting δ → 0 yields

By Grönwall's inequality, we obtain that

Now since π 0 ∈ Γ( ν 1 , ν 2 ) and µ i,t = ( X ( t, · , ν i )) # ν i , the mapping

satisties that (Ξ t ) # π 0 ∈ Γ( µ 1 ,t , µ 2 ,t ) . As a consequence we have that

I Proofs for Section 3.1

Proof of Theorem 3.1. For any θ = ( µ , Σ) ∈ Θ , define ˜ E ( µ , Σ) := E ( ρ ) . Then we have

where ρ is the density of N ( µ , Σ) and ρ ∗ is the density of N ( b , Q ) .

Now we consider the gradient flow on the submanifold Θ

where ∇ Σ ˜ E ( µ , Σ) denotes the standard matrix derivative (not the covariant derivative or affine connection in the contexts of Riemannian geometry). We calculate that

For clarity note that here

Thus, the gradient flow on Θ is equivalent to

Note that it is trivial to check that the functions on the right-hand-side of (3.1) are locally Lipschitz with respect to µ t and Σ t (continuously differentiable hence locally Lipschitz). By Picard-Lindelöf theorem, this ODE system given µ 0 ∈ R d , Σ 0 ∈ Sym + ( d, R ) has a unique solution on t ∈ [0 , ϵ ) for some ϵ > 0 . Let

For convenience we define the curve on Θ = R d × Sym + ( d, R ) by

Next we consider the density flow given by

By Theorem C.2 we know that there is a unique solution ρ t in P ( R d ) for t ∈ [0 , ∞ ) . We claim that

Claim. ρ t is a Gaussian density for t ∈ [0 , s ) .

In fact, if we let ˜ ρ t := ϕ ( γ ( t )) , where ϕ is the immersion

By uniqueness of the solution it suffices to prove that (76) holds for ρ t = ˜ ρ t . This can be checked by direct calculation of course. But there is also a more elegant way to show it. We consider the following commutative diagram.

Here d ϕ θ is the pushforward of the immersion ϕ at point θ and ψ θ is the inverse of the pullback map ϕ ∗ : T ∗ ˜ ρ P ( R d ) → T ∗ θ Θ restricted on Im G ˜ ρ t ◦ d ϕ θ ( ≃ T ∗ θ Θ) . The diagram is commutative due to the fact that G θ is the canonical isomorphism on the submanifold Θ induced from P ( R d ) .

Now we show that δE δ ˜ ρ t ∈ Im G ˜ ρ t ◦ d ϕ θ . In the proof of Theorem B.3, we have shown that ψ θ maps ( b , 1 2 S ) to some Φ such that ∇ Φ( x ) = S ( x -µ ) + b , where µ is obtained from G -1 θ ( b , 1 2 S ) . Thus, Im ψ θ contains all functions (or more precisely the equivalent classes of functions that differ by a constant) with the form

In other words, Im ψ θ contains all quadratic forms on R d . Note that we can derive that

is exactly a quadratic form. Thus,

Next since d ϕ θ t maps the tangent vector ∂ ∂θ t ∈ T θ t Θ to δ δ ˜ ρ t ∈ T ˜ ρ t P ( R d ) , we have that

Combining this with the fact that δE δ ˜ ρ t ∈ Im G ˜ ρ t ◦ d ϕ θ , we get

Thus, we have

And we conclude

The claim is proven.

Now back to the original problem. Suppose s < ∞ . Since we know that the mean field PDE (76) has a unique solution on [0 , ∞ ) , in particular, it exists on [0 , s ] . Note that the weak limit of Gaussian distributions is Gaussian and since ρ s ∈ P ( R d ) it does not degenerate. By definition of ϕ we have that ϕ -1 ( ρ s ) ∈ Θ . By letting ( µ s , Σ s ) = γ ( s ) := ϕ -1 ( ρ s ) , we obtain the solution of (75) on [0 , s ] . Again by Picard-Lindelöf theorem there exists a small neighborhood [ s, s + ϵ ′ ) such that (75) has a unique solution. This together with the solution on [0 , s ] contradicts the definition of s . Therefore, we conclude that s = ∞ . (75) has a unique global solution corresponding to the mean and covariance matrix of ρ t .

Next, we prove that ρ t converges weakly to ρ ∗ as t →∞ . We calculate the quantity ˙ ˜ E ( µ t , Σ t ) . By Jacobi's formula in matrix calculus (Theorem 8.1 in [59]), we have

Thus, we derive that

Noticing that

we obtain that ˙ ˜ E ( µ t , Σ t ) → 0 as t →∞ , which is equivalent to µ t → b and Σ t → Q by checking the expression above. Thus, we have shown that ρ t converges weakly to ρ ∗ which is the density function of N ( b , Q ) .

Finally, we show the convergence rates of µ t and Σ t . Since we have already proven that ρ t → ρ ∗ , it implies that

If we set η = Q -1 ( µ -b ) and S = ( Q -1 -Σ -1 )Σ , then

On the other hand, ˜ E ( µ t , Σ t ) can be written as

Now we prove that ∀ ϵ > 0 there exists T > 0 such that -˙ ˜ E ( µ t , Σ t ) ≥ 4( γ -ϵ ) ˜ E ( µ t , Σ t ) for t ≥ T . It suffices to show that

This is true because by definition γ is the smallest eigenvalue of the matrix.

By Grönwall's inequality, we know ˜ E ( µ t , Σ t ) = O ( e -4( γ -ϵ ) t ) . Thus, we conclude

̸

Finally, we provide a lower bound on γ . Note that for any u > 0 if b = 0 we have

which is equivalent to

Thus,

Since λ max is the largest eigenvalue of Q , we know the smallest eigenvalue of Ω u is given by

We find u such that this quantity is maximized and get

If b = 0 , then the smallest eigenvalue is given by

Proof of Theorem 3.2. Equation (5) is a direct corollary of Theorem 3.1.

By Theorem C.2 there is a unique global solution. Thus, we only need to check that the Σ t given by (6) and (7) satisfy the algebraic Riccati equation (5).

For

we take the derivative with respect to t and get

Substituting (78) into (77), we get

Multiplying by Σ 2 t and using the fact that Σ t and Q commute, we can see that the algebraic Riccati equation (5) holds.

For

we apply the Sherman-Morrison formula:

Thus,

Moreover,

Thus,

On the other hand,

Thus,

Therefore, with

is a solution with the initial condition Σ 0 = I d . The theorem follows by the uniqueness of the solution of the mean field PDE (Theorem C.2).

Proof of Theorem 3.4. Similar to the proof of Theorem 3.1, we have

and

Following the arguments similar to the proof of Theorem 3.1, we can show

(8) has a unique global solution,

ρ t is the density of N ( µ t , Σ t ) given by (8),

˙ ˜ E ( θ t ) ≤ 0 and ˙ ˜ E ( µ t , Σ t ) → 0 as t →∞ ,

ρ t converges weakly to ρ ∗ .

Finally suppose Σ 0 Q = Q Σ 0 , then Σ t also commutes with Q since 0 is a solution of the ODE satisfied by Σ t Q -Q Σ t and the solution is unique. Thus, we can diagonalize Σ t and Q simutaneously. Then there exists orthogonal matrix P such that

And (9) reduces to

Solving this ODE, we get

Thus, we have σ ( t ) i → λ i as t →∞ and

In particular, we conclude ∥ Σ t -Q ∥ = O ( e -2 t/ ((1 -ν ) λ + ν ) ) where λ is the largest eigenvalue of Q .

Proof of Theorem 3.5. First, we define the Hamiltonian on the centered Gaussian submanifold by

By Corollary B.5 we have

Thus, the Hamiltonian is reduced to

Therefore, we have

and thus the Hamiltonian or AIG flow on the Gaussian submanifold is given by (12).

Next we show Σ t is well-defined and remains positive definite. We check that H t := H (Σ t , S t ) is decreasing with respect to t .

Let σ t be the smallest eigenvalue of Σ t . Then

Therefore, we have

which yields that

This means that the smallest eigenvalue of Σ t has a positive lower bound. Thus, Σ t ∈ Sym + ( d, R ) for any t ≥ 0 .

Finally we show that the AIG flow on the centered Gaussian submanifold coincides with the one on the density manifold. Similar to the proof of Theorem 3.1, we have a commutative graph

Here ψ Σ : S → Φ( x ) = x ⊤ S x + C , i.e., it maps a symmetric matrix to the quadratic function Φ( x ) (or more precisely the equivalent classes of quadratic functions that differ by a constant). Now the only things we need to show are that δ H δρ t ∈ Im ψ Σ and that δ H δ Φ t ∈ Imd ϕ Σ .

δ H δ Φ t = G -1 ρ Φ t ∈ Imd ϕ Σ is trivially true from the commutative graph. Now since δE δρ t = log ρ t + V ∈ Im ψ Σ ( ρ t is centered Gaussian density and check the definition of ψ Σ ), it suffices to prove

Note that we have

Thus, the proof is complete.

We remark that the fact the Stein AIG flow remains Gaussian is highly non-trivial. In fact it requires δ H δρ t to lie in the cotangent space of the Gaussian submanifold. One sufficient condition is that the variational derivatives of both the kinetic and potential energies lie in this same space, and the former could be interpreted as: The Gaussian submanifold is totally geodesic under the given metric, meaning that any geodesic flow with an initial position and velocity chosen from the Gaussian submanifold remains Gaussian. Fortunately both the Wasserstein metric and the Stein metric satisfy this property.

J Proofs for Section 3.2

Proof of Theorem 3.6. From the proof of Theorem 3.1 we know that (16) has a unique solution that is continuous in t and bounded for any t ∈ [0 , T ] with T < ∞ . Thus, the linear system (15) also has a unique solution.

Now we check that the sample mean µ t and covariance matrix C t satisfy (16). We simplify the right-han-side of (13) using µ t and C t .

Let X t = ( x ( t ) 1 , · · · , x ( t ) N ) ⊤ . Then we have that

Then (79) can be written in the matrix form as

Multiplying by 1 ⊤ /N on the left, we get

Thus, we have

Note that ˙ C t = ( ˙ X ⊤ t X t + X ⊤ t ˙ X t ) /N -µ t ˙ µ t -˙ µ t µ ⊤ t . Substituting (80) we obtain

Next we show that (14) and (15) satisfies (13).

By Theorem C.3 (13) has a unique solution. Thus, the solution of (13) is given by (14)-(16).

The proof of Theorem 3.7 is quite long and tedious so we defer it to Appendix L.

Proof of Theorem 3.8. If C 0 is non-singular, this is a direct corollary of Theorem 3.6 and Theorem 3.2. To also accommodate for the singular case, we provide a direct proof by calculation. Since C 0 Q = QC 0 , we know that C 0 and Q are simultaneously diagonalizable. There exists some orthogonal matrix P 0 such that we have the spectral decompositions

where D 0 = diag( λ (0) 1 , · · · , λ (0) d ) and Q 0 = diag( q 1 , · · · , q d ) are diagonal matrices. Let D t := P 0 C t P ⊤ 0 . (18) can be rewritten as

Thus, by taking the derivative with respect to t , we obtain

where

On the other hand, we check that

Thus, we conclude that (18) is a solution of (13). By Theorem C.3, the solution for (13) is unique and hence the theorem follows.

Proof of Theorem 3.9. For any particle system of the R-SVGF, we can derive that

By Woodbury matrix identity we derive

Substituting this into (81), we have

Multiplying by X ⊤ t on the left we get

Thus, the derivative of covariance matrix C t is given by

Next we show that (19) and (20) satisfies (81).

Similar to the proof of Theorem 3.4, it could be shown that the R-SVGF also has a unique solution and the proof is complete.

Next we show Theorem 3.10.

Lemma J.1. The covariance matrix C t ( t = 1 , 2 , · · · ) of the discrete-time finite particle system satisfies the following equation

Proof. Let X = ( x 1 , · · · , x N ) ⊤ . Then (21) can be written as

Thus, we have

Proof of Theorem 3.10. First since C 0 Q = QC 0 they are simutaneously diagonalizable. By Theorem 3.8 any C t and Q are simutaneously diagonalizable. Thus, without loss of generality we assume all C t and Q are diagonal matrices. Then by Lemma J.1 we know

If we define f ϵ ( x ) = (1 + ϵ (1 -x )) 2 x then every entry (i.e., eigenvalue) of Q -1 C t follows the f ϵ -iteration trajectory of the corresponding entry of Q -1 C 0 .

The fixed points of f ϵ ( x ) = (1 + ϵ (1 -x )) 2 x are { 0 , 1 , 2 /ϵ +1 } . If 0 < ϵ < 1 then | f ′ (0) | > 1 , | f ′ (2 /ϵ +1) | > 1 , and | f ′ (1) | < 1 . By Proposition 1.9 of [26], 1 is an attracting fixed point while 0 and 2 /ϵ +1 are repelling fixed points. By definition there exists an interval around 1 such that for all initial points in that interval, the trajectory of any eigenvalue of Q -1 C t converges to 1 . We now quantify that result further.

Note that f ′ ϵ ( x ) = (1 + ϵ -ϵx )(1 + ϵ -3 ϵx ) is an upward-opening parabola whose zeros are 1 / 3 + 1 / (3 ϵ ) and 1 + ϵ . Thus, f ′ ϵ is monotone decreasing on [0 , 1 / 3 + 1 / (3 ϵ )] . In particular, both equation f ′ ϵ ( x ) = 1 and f ′ ϵ ( x ) = 1 -ϵ have two distinct roots, the smaller ones lying on (0 , 1 / 3 + 1 / (3 ϵ )) . Define w ϵ to be the smaller root of f ′ ϵ ( x ) = 1 and u ϵ , the smaller root of f ′ ϵ ( x ) = 1 -ϵ . Since f ′ ϵ is monotone descreasing and f ′ ϵ (1) = 1 -2 ϵ , we have 0 < w ϵ < u ϵ < 1 .

Thus, for any x ∈ [ w ϵ , 1 / 3 + 1 / (3 ϵ )] we have 0 ≤ f ′ ϵ ( x ) ≤ 1 and 0 ≤ f ϵ ( x ) ≤ f ϵ (1 / 3 + 1 / (3 ϵ )) < 1 / 3 + 1 / (3 ϵ ) (here we have used the condition ϵ < 0 . 5 ). On the other hand, since f ′ ϵ (0) ≥ 1 and 0 and 1 are two fixed points, it holds that f ϵ ( x ) ≥ x for any x ∈ [0 , 1] , which implies that for any x ∈ [ w ϵ , 1 / 3 + 1 / (3 ϵ )] we have f ϵ ( x ) ≥ f ϵ ( w ϵ ) ≥ w ϵ . Hence we know that f ′ ϵ is a contraction map and f ϵ ([ w ϵ , 1 / 3 + 1 / (3 ϵ )]) ⊂ [ w ϵ , 1 / 3 + 1 / (3 ϵ )] , which implies that there is a unique fixed point (i.e., 1 ) such that the f ϵ -iteration trajectory converges to it.

Next we prove that for any x ∈ (0 , 1+1 /ϵ ) the trajectory falls into the interval [ w ϵ , 1 / 3+1 / (3 ϵ )] after finite iterations. If x ∈ (1 / 3 + 1 / (3 ϵ ) , 1 + 1 /ϵ ) then after one iteration we get f ϵ ( x ) ∈ (0 , f ϵ (1 / 3 +

we have f ( t 0 ) ϵ ( x ) ∈ [ w ϵ , 1] . Firstly f ( t ) ϵ ( x ) ≤ 1 for any t . It suffices to prove f ( t 0 ) ϵ ( x ) ≥ w ϵ . Suppose f ( t 0 ) ϵ ( x ) < w ϵ . Then for any 0 ≤ t ≤ t 0 we have f ( t ) ϵ ( x ) < w ϵ . By definition of t 0 we know

This is a contradiction. Thus, the claim holds and in conclusion for any x ∈ (0 , 1 + 1 /ϵ ) the f ϵ -iteration trajectory converges to the fixed point 1 .

Finally we consider the case when x ∈ [ u ϵ , 1 / 3 + 1 / (3 ϵ )] . Since f ′ ϵ is monotone descreasing here we have f ′ ϵ ( x ) ∈ [0 , 1 -ϵ ] . And by similar argument we know f ϵ ([ u ϵ , 1 / 3+1 / (3 ϵ )]) ⊂ [ u ϵ , 1 / 3+1 / (3 ϵ )] . Thus, we conclude

K Proofs for Section 4

The following result was derived in [43]. We provide the proof below for completeness.

Lemma K.1. Let ρ ∗ be a probability measure and ρ θ be a Gaussian measure with parameters θ = ( µ , Σ) . Then we have the following expressions:

Proof. We compute that

where we have used the fact that ρ θ is a Gaussian density in ∗ . Similarly, we have

Here again in ( a ) and ( b ) we have used the closed-form expression of Gaussian densities. Thus,

Proof of Theorem 4.1. By definition of Gaussian approximate gradient descent, we have that

Applying Theorem B.3, we obtain that ˙ θ t = -G -1 θ t ( ∇ θ t E ( θ t )) is given by

Substituting (82) into (83), we get

Next we prove the convergence.

Noticing that

we obtain that ˙ E ( µ t , Σ t ) → 0 as t → ∞ , which means that there exists µ ∞ , Σ ∞ such that ρ t converges to ρ ∞ , ρ ∞ is the density of N ( µ ∞ , Σ ∞ ) . (Since E ( µ t , Σ t ) is given by the KL divergence between ρ t and ρ ∗ , by Lemma K.1 it will diverge if µ t or Σ t diverges.) In particular, it satisfies that

which is equivalent to

and implies that ρ ∞ = ρ θ ∗ .

Lemma K.2. Given α -strongly convex measure ρ ∗ , we define θ ∗ as the unique minimizer of KL( ρ θ ∥ ρ ∗ ) where ρ θ denotes a Gaussian measure with parameters θ . Then it holds that

This is proven in the Appendix D of [43]. The proof idea is to consider the Gaussian approximate Wasserstein gradient flow from ρ θ with the target ρ ∗ .

Lemma K.3. Given α -strongly convex measure ρ ∗ , we define θ ∗ as the unique minimizer of KL( ρ θ ∥ ρ ∗ ) where ρ θ denotes a Gaussian measure with parameters θ . Then the Wasserstein2 distance between ρ θ and ρ θ ∗ satisfies that

Proof of Theorem 4.2. From Lemma K.1 and the proof of Theorem 4.1 we know that

For convenience we let η t = E ρ θ [ ∇ V ( x )] and S t = ( E ρ θ [ ∇ 2 V ( x )] -Σ -1 )Σ 1 / 2 . Then we get

Noting that M t → M ∞ , for any ϵ > 0 there exists T > 0 such that the smallest eigenvalue γ t /α of M t satisfies γ t ≥ γ -ϵ for any t > T , where γ/α is the smallest eigenvalue of M ∞ .

Moreover, by Lemma K.2 we have

Thus, for t > T the derivative of KL divergence is controlled, i.e.,

By Grönwall's inequality we know

Noting that

we conclude

̸

Finally, we provide a lower bound on γ . Note that for any u > 0 if µ ∗ = 0 we have

Thus,

Since Σ ∗ satisfies that

and ∇ 2 V ( x ) ⪯ βI d , we know the smallest eigenvalue of Σ ∗ is at least 1 /β . Thus, the smallest eigenvalue of Ω u is

We find u such that this quantity is maximized and get

If µ ∗ = 0 , we still have

L Proofs of Uniform in Time Convergence

To show Theorem 3.7 we need a lemma on the convergence of empirical measures in the i.i.d. setting. There are results for general measures on this given by [25, 47, 24] but for our purpose we always have a Gaussian distributions as the limit and there are tight results with faster convergence rates as shown in [9, 45, 46]:

Lemma L.1 (Convergence of empirical measures for Gaussian distributions) . Fix the dimension d ≥ 1 . There exists a constant C d such that for all N ≥ 1 , with µ N = 1 N ∑ N k =1 δ X k where { X k } is i.i.d. sequence drawn from µ ∼ N ( 0 , I d ) , we have

Proof of Theorem 3.7. Suppose the sample mean and covariance at time t is m t and C t , and that the mean and covariance of the mean-field limit is µ t and Σ t . We bound E [ W 2 2 ( ζ ( t ) N , ρ t )] using Theorems 3.1 and 3.6 and Lemma L.1 in six steps.

Step I. We prove that ( x ( t ) i ) has the same distribution as ( ˜ x ( t ) i ) where ˜ x ( t ) i = C 1 / 2 t C -1 / 2 0 ( x (0) i -m 0 ) + m t . Note that according to Theorem 3.6, where we have x ( t ) i = A t ( x (0) i -m 0 ) + m t . Here A t is the unique (matrix) solution of the linear system

and m t and C t are the unique solution of the ODE system

Since C t is the sample covariance, we have

Thus, P t = C -1 / 2 t A t C 1 / 2 0 is an orthogonal matrix, and we have

Since the multivariate Gaussian distribution N ( 0 , I d ) is invariant under orthogonal transformation, the joint distribution of ( C -1 / 2 0 ( x (0) i -m 0 ) ) N i =1 is also invariant under P t . Thus, we have ( x ( t ) i -m t ) has the same distribution as ( ˜ x ( t ) i -m t ) and Step I is proven.

Step II. Establish uniform decay rates for ∥ C t -Q ∥ F and ∥ m t -b ∥ . We begin by checking the energy function

As shown in the proof of Theorem 3.1, we have

Thus, E ( m t , C t ) ≤ E ( m 0 , C 0 ) for any t ≥ 0 . Furthermore, similar to the proof of Theorem 3.1 we check that

which simplies that

where γ t is the smallest eigenvalue of

and as shown in the proof of Theorem 3.1 it has a lower bound

where q max is the largest eigenvalue of Q . Now since

we know ( m t -b ) ⊤ ( m t -b ) ≤ 2 q max E ( m 0 , C 0 ) . Thus, ∥ m t ∥ is upper bounded by some quantity F 1 = F 1; Q, b ,C 0 , m 0 . Hence γ t is uniformly lower bounded by

Thus, by Grönwall's inequality we have E ( m t , C t ) ≤ e -4 γ ∗ t E ( m 0 , C 0 ) . There exists F 2 = F 2; Q, b ,C 0 , m 0 such that ∥ m t -b ∥ ≤ e -2 γ ∗ t F 2 . Now similarly

also renders an upper bound for ∥ C t -Q ∥ F with exponential decay noting that tr( A ) -log det( I + A ) is quadratic in ∥ A ∥ F when ∥ A ∥ F is small, i.e., there exists F 3 = F 3; Q, b ,C 0 , m 0 such that

By Theorem 3.1 we know that µ t , Σ t satisfy the same ODEs as m t , C t :

Thus, similarly we have

We derive that

Step III. Show that ∥ C t -Σ t ∥ F and ∥ m t -µ t ∥ can be controlled after sufficient time. For any ϵ > 0 we define

and consider the relative energy function

We show that for ϵ small enough if ( m t , C t ) , ( µ t , Σ t ) ∈ Θ ϵ , then

Here note that

On the other hand, we have

Here we have

and

Combining all these together we have

Since 0 < q min I ⪯ Q ⪯ q max I , and ∥ m t ∥ and ∥ µ t ∥ are bounded by F 1 as shown in Step II, there exists ϵ 0 = ϵ Q, b ,C 0 , m 0 , Σ 0 , µ 0 ( ϵ 0 can be seen as a continuous function) such that for any ϵ ≤ ϵ 0 we have ˙ E ( m t , C t , µ t , Σ t ) ≤ 0 as long as we have ( m t , C t ) ∈ Θ ϵ and ( µ t , Σ t ) ∈ Θ ϵ .

Given Q, b , C 0 , m t , Σ 0 , µ 0 suppose at time t 0 we have ( m t 0 , C t 0 ) ∈ Θ ϵ 0 and ( µ t 0 , Σ t 0 ) ∈ Θ ϵ 0 . Then for any t > t 0 we know

Note that

Fixing any δ > 0 as long as ϵ 0 is small enough we have

Therefore, we conclude that for any given Q , b , C 0 , m 0 , Σ 0 and µ 0 there exists ϵ 0 as stated above and F 4 = F 4; Q, b ,C 0 , m 0 , Σ 0 , µ 0 > 0 such that as long as ( m t 0 , C t 0 ) ∈ Θ ϵ 0 and ( µ t 0 , Σ t 0 ) ∈ Θ ϵ 0 then for any t > t 0

Note that by definition of the Frobenius norm (or any matrix norm), given ϵ 0 > 0 there exists ϵ 1 ∈ (0 , ϵ 0 ) such that for any S ∈ Sym( d, R ) as long as the norm is small enough, i.e., ∥ S -Q ∥ F ≤ ϵ 1 , then we have (1 -ϵ 0 ) Q ⪯ S ⪯ (1 + ϵ 0 ) Q . Then by Step II we know that if we set F 5 = max { F 2 , F 3 , F ′ 2 , F ′ 3 } and t 0 > -1 2 γ ∗ log ϵ 1 F 5 then the following bounds hold:

Now it is straight forward to check from (85) and (86) and the results in Step II that there exists F 6 = F 6; Q, b ,C 0 , m 0 , Σ 0 , µ t such that for any t ≥ 0

Thus, by Grönwall's inequality we have

Combining this with Step III, we know for any t ≥ 0 , there exists F 7 = e F 6 t 0 F 4 (only depending on Q, b , C 0 , m 0 , Σ 0 , µ 0 ) such that

Step V. Let y ( t ) i := Σ 1 / 2 t Σ -1 / 2 0 ( x (0) i -µ 0 ) + µ t . Define ξ ( t ) N = 1 N ∑ N i =1 δ y ( t ) i and ˜ ζ ( t ) N = 1 N ∑ N i =1 δ ˜ x ( t ) i . We show that

Since ◦ is trivial by the definition of the Wasserstein metric, we only need to check ⋄ . In fact,

Note that t F

By the central limit theorem N ( m 0 -µ 0 ) converges to N ( 0 , Σ 0 ) in distribution. √ N log log N ( m 0 -µ 0 ) converges to 0 in distribution and hence also in probability. (There could even be almost sure results using the law of iterated logarithm but converge in probability is good enough.) Similarly by CLT every entries of √ N ( C 0 -Σ 0 ) converges to a Gaussian distribution. Thus, √ N log log N ( C 0 -Σ 0 ) converges to zero matrix in probability. Therefore, we have N log log N ∥ C t -Σ ∥ 2 → 0 in probability.

By Step II, we have ∥ C t ∥ F ≤ ∥ Q ∥ F + F 3 . All these constants here ( F 3 , F 7 , ∥ C -1 0 ∥ F ) can be seen or chosen as a continuous function of Q, b , C 0 , m 0 , Σ 0 , µ 0 and by continuous mapping theorem they converge to the values of the same function with C 0 = Σ 0 and m 0 = µ 0 . Thus, we conclude that N log log N ( I 6 + I 7 ) → 0 in probability.

Now we derive that

Now we show a lemma: Suppose A,B ∈ Sym + ( d, R ) are two positive definite matrices. Then we have ∥ A 1 / 2 -B 1 / 2 ∥ ≤ 1 2 √ λ ∥ A -B ∥ where λ is the smallest eigenvalue of A and B . Note that we are using the spectral norm here.

In fact, denote the largest eigenvector of A 1 / 2 -B 1 / 2 by η , and let x ∈ R d be the corresponding eigenvector such that x ⊤ x = 1 . We have

Thus, we have that ∥ A 1 / 2 -B 1 / 2 ∥ = η ≤ 1 2 √ λ ∥ A -B ∥ . Moreover, the Frobenius norm is bounded by √ √

Applying this lemma, we know

where λ is the smallest eigenvalue of C 0 and Σ 0 , which converges to the smallest eigenvalue of Σ 0 as N goes to infinity.

Next we need to show that the smallest eigenvalue of C t and Σ t are uniformly (in time) lower bounded by some λ ′ > 0 (depending on Q, b , C 0 , m 0 , Σ 0 , µ 0 ). We revisit Step II, where we show that E ( m t , C t ) ≤ E ( m 0 , C t ) . Then

leads to a uniform lower bound of the smallest eigenvalue of C t since tr( Q -1 C t -I ) -log det( Q -1 C t ) →∞ as the smallest eigenvalue of C t goes down to zero. Thus, we have

Following similar arguments we conclude that N log log N I 5 → 0 in probability as N →∞ . Therefore, in probability

which implies that

Step VI. Apply Lemma L.1 to get the final result. Note that x ( t ) i are i.i.d. from N ( 0 , I ) and y ( t ) i is a linear function of x ( t ) t (unlike m t and C t which are random, µ t and Σ t are deterministic). By Lemma L.1 and Step II ( µ t and Σ t are uniformly bounded) we have

Thus, we derive that

Note that we have used Step I in ( ∗ ) , the triangle inequality in ( ∗∗ ) , and Step V along with (87) in ( ∗ ∗ ∗ ) .

Proof of Theorem F.4. The proof is roughly similar to that of Theorem 3.7. But Step III is a little different (can be strengthened and simplified at the same time).

We show the global contraction of ∥ m t -µ t ∥ and also the contraction of ∥ C t -Σ t ∥ F after sufficient time. Note here ∥·∥ F is the Frobenius norm.

Frist we check the derivative of squared Euclidean norm of m t -µ t .

Thus, ∥ m t -µ t ∥ ≤ e -q max ∥ m 0 -µ 0 ∥ . To bound ∥ C t -Σ t ∥ F we define S ϵ := { S ∈ Sym + ( d, R ) : (1 -ϵ ) Q ⪯ S ⪯ (1 + ϵ ) Q } .

This time we do not need the relative energy but can directly check the derivative of the squared Frobenius norm:

where tr = denotes equal in trace. Now if C t , Σ t ∈ S ϵ then

and

Note that ⋄ is not trivially true. We show it as a lemma: Suppose A is a symmetric matrix and B is a positive definite symmetric matrix. Then tr( ABAB -1 ) ≥ tr( A 2 ) .

In fact, we write B = P Λ P ⊤ where P is an orthogonal matrix and Λ = diag { λ 2 , · · · , λ d } is a diagonal matrix. Then

Denoting P ⊤ AP = ( a ij ) we get

Thus, substituting (90) and (89) into (88) we get

Suppose at time t 0 both C t 0 and Σ t 0 lie in S ϵ . Then for any t ≥ t 0 we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the theoretical understanding and algorithmic contributions of Gaussian-Stein Variational Gradient Descent (Gaussian-SVGD) for Gaussian Variational Inference (GVI). The paper discusses the dynamics of Gaussian-SVGD and provides convergence rates in both mean-field and finite-particle settings when the target is Gaussian. It also shows that Gaussian-SVGD converges to the best Gaussian approximation to the target in KL-divergence for non-Gaussian targets. The authors propose two algorithmic frameworks for density-based and particle-based implementations of Gaussian-SVGD, which encompass various GVI algorithms proposed in the literature as special cases. The Gaussian-SVGD, and other related algorithms, are compared against one another on three examples: Gaussian target, Bayesian logistic regression, and a Gaussian mixture model. The authors highlight future directions for research, including establishing convergence results for particle-based Gaussian-SVGD algorithms under log-concave and more general targets, and exploring acceleration techniques.\n\nSTRENGTHS:\nThe paper is very well written and the mathematical content is clearly articulated. The authors have done a good job of relating this work to other similar works from the literature. The paper is well-structured and the authors' particular contributions to this line of research are clearly outlined. \n\nThe technical detail, as far as I can see, is correct and the contribution of this work to the ML community, in particular the new convergence rates, are of significant interest. The paper is closely linked to other related works but is still highly original. \n\nThe paper provides a nice balance between establishing important theoretical results and providing a usable algorithm for users to implement in practice.\n\nWEAKNESSES:\nThis is a nice piece of work and there are no areas where I can see significant weaknesses. However there are a few things that the authors may wish to consider to improve their paper:\n\n* Starting from a very simple question, \"Why should I use Gaussian-SVGD?\" I think the authors have skipped some of the motivation for this work. If you know that the target is Gaussian, then of course you would just fit a Gaussian to it and would not require a particle-based variational inference approach. If the target is non-Gaussian, then why would you use Gaussian-SVGD over standard SVGD? \n* It is not clear why regularized SVGD is introduced. What is the motivation here? Is it simply to show that you can derive a Gaussian-SVGD algorithm with the regularized Stein metric?\n* The kernel $K(\\cdot,y)$ is introduced in Def 2.2 without being defined. Additionally, I think in eq. (1) you need to replace $\\rho$ with $\\rho_t$.\n* The simulation section is unfortunately a bit weak compared to other similar papers. The authors only consider three simple models and the details are largely glossed over, (e.g. what is $d$? How many particles? etc.). The authors seem to have run out of space and only provide a short simulation study for the logistic regression example. It would be better to include a more detailed simulation study in the main paper and to consider more challenging models.\n* Additionally, on the point of experiments, it is a bit surprising that SVGD is not also included in this list.\n\nQUESTIONS:\n* This sort of relates to my previous question approach motivation. It is well-known that SVGD does not scale well to high-dimensional target distributions. Is it possible to show, either theoretically (i.e. results like Theorem 3.7) or empirically, that Gaussian-SVGD is better than SVGD when $d$ is large?\n* Similarly, what is known about how many particles are needed compared to SVGD?\n* Particle-based methods are preferred over standard variational methods which fit a Gaussian distribution to the target because of their nonparametric nature. In the experiments section, it seems like the Gaussian mixture model would have been the more interesting example to include in the paper, particularly in the context of how good the approximation to the target is compared against SVGD. Could this be included?\n\nLIMITATIONS:\nThe paper could discuss more the theoretical limitation of this work. However, from an ethical perspective, I do not believe that this paper touches on issues that would have a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-20"}

comments: SUMMARY:
This paper delves into an examination of Gaussian-SVGD through the analysis of mean-field PDE and discrete particle systems. It offers evidence for finite particle convergence and supports these findings with empirical validations.

STRENGTHS:
Through clear modeling, the article provides a comprehensive analysis for Gaussian-SVGD.

The paper gives new interpretations for Wasserstein Gradient Flow and SVGD under linear systems, which are very interesting.

WEAKNESSES:
The setup of the paper is relatively artificial, because in actual scenarios, linear kernels are rarely used to implement SVGD.

The paper does not put much effort into justifying the significance of Gaussian variational inference, such as making comparisons with other distribution classes. If there were clear conclusions, it would have a significant impact on the significance of this paper.

QUESTIONS:
Can you elaborate some advantages of Gaussian-SVGD compared with RBF-SVGD?

LIMITATIONS:
The setup of the article is somewhat artificial. It would be more meaningful if the significance of this linear system could be demonstrated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Stein variational gradient descent and its variants with a bilinear kernel on the space of Gaussian measures. The authors prove the rate of convergence of the dynamics, proposed finite particle algorithms and proved a uniform-in-time propagation of chaos,  and finally prove the convergence rate of the finite particle algorithms.\n\nSTRENGTHS:\n1. The study of the dynamics properties of  SVGD and its variants on Gaussian space seems to be interesting, which is a natural analog to the Bures-Wasserstein gradient flow. This paper gives a comprehensive analysis of such algorithms, including the well-posedness of the dynamics and the rate of convergence. \n\n2. The proof and the results seem to be correct. I didn't check all the details in the proof, but both the proof and the results make sense to me.  \n\n\n3. The paper is in general well-written, the theorems are stated properly, and I think it is not hard for readers to understand the main idea.\n\nWEAKNESSES:\nI do not observe an obvious weakness of this paper, but I do have a few questions, please refer to the \"Questions\" part.\n\nQUESTIONS:\n1. Intuitive explanation of the convergence rate: as proved in the paper, the rate of convergence of SVGD for centered Gaussian is $O(e^{- t})$, which do not depend on $\\lambda$. This seems to be interesting, but I'm wondering if there's an intuitive explanation for this result? The dependence of $O(e^{- t/\\lambda})$ in the Wasserstein gradient flow case makes more sense to me intuitively, since larger $\\lambda$ will give a more \"flat\" potential.\n\n2. Regarding the non-Gaussian target:\n    - It is shown in Theorem 4.1 that when $\\rho_{\\theta^*}$ be the unique Gaussian measure that minimize $D_{KL}(\\rho_{\\theta^*}||\\rho^*)$, then the Gaussian-SVGD will converge to $\\rho_{\\theta^*}$. But when there are multiple $\\rho_{\\theta^*}$ that achieve the minimization, will the dynamic converge to one of them?\n\n    - Does the particle dynamics Algorithm 2 yield a similar uniform-in-time propagation of chaos results as in Theorem 3.7 with the same rate for Gaussian-SVGD?\n  \n\n\nMinor points and typos:\n\n1. Some ambiguity in definitions： \n    - The definition of the kernels on page 3 (lines 111-118) is not entirely clear to me, in particular, what is $\\mu$ and $\\Sigma$ here? If I understand the latter part correctly, I think $\\mu$ and $\\Sigma$ will be the mean and covariance of $\\rho_t$, which is actually time variant. I suggest the authors clarify this point in the definition.\n    - The same problem appears in equation (26) when defining $\\widehat{\\nabla V}$, since $\\widehat{\\nabla V}$ depends on $t$, which is a time-dependent vector field.\n\n\n2. Definition 2, line 101: $G_{\\rho}^{Wass}$ should be $G_{\\rho}^{Stein}$\n\nLIMITATIONS:\nNo limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work performs a theoretical investigation of Gaussian SVGD, a special case of SVGD restricted to the submanifold of Gaussian densities by means of a bilinear kernel. The authors characterize the mean-field dynamics of Gaussian SVGD, with a particular focus on Gaussian targets and obtain finite-particle guarantees in certain settings. Moving beyond Gaussian targets, the authors show that the dynamics of mean-field Gaussian SVGD converges to the best Gaussian approximation (In KL divergence) to the target density, and obtain convergence rates for the mean-field dynamics with the target density is log-smooth and log-concave\n\nSTRENGTHS:\nThe work provides a thorough treatment of the mean-field dynamics of Gaussian SVGD. The experimental evaluation is also satisfactory.\n\nWEAKNESSES:\nMy primary concerns are as follows :\n\n1. Most of the results presented in this work consider continuous-time mean-field dynamics under the assumption of a Bilinear kernel and Gaussian target density. In my opinion, these assumptions adversely impact the significance of these results, as they fail to explain one of the major appeals of SVGD, i.e., the fact that it provides a non-parametric approximation to a large class of target densities. (e.g. [1] shows discrete-time mean-field guarantees for subgaussian densities and [2] extends it to the class of densities satisfying a generalized Talagrand’s inequality). While the exponentially fast convergence guarantees for the mean-field dynamics are seemingly appealing, they fall short of providing a satisfactory explanation of the behavior of SVGD in practically relevant settings. \n\n\n2. Despite the simplifying assumption of Gaussian targets and bilinear kernels, the work does not provide any quantitative discrete-time finite-particle convergence rates. To the best of my understanding, Theorem 3.9 is the only result that considers finite-particle discrete-time dynamics, but it does not provide any quantitative rates.\n\n\n3. It is not clear why one should prefer Gaussian SVGD over the Bures-Wasserstein Gradient Flow [3] since the only setting where it seems to outperform Bures-Wasserstein Gradient Flow is that of centered Gaussians, which is a highly restrictive condition. \n\n\n4. The results in Section 4 leave much to be desired. Considering the fact that the best known results for Bures-Wasserstein gradient flow [3] cover both log-concave and strongly log-concave densities (see also [4] for a computable JKO discretization of the Bures-Wasserstein flow with convergence guarantees for log-concave and log-strongly concave densities), it is not at all clear what the benefits of Gaussian SVGD are when the target is not Gaussian. \n\n\n5. The overall presentation of the results requires a lot of work. For instance, it is not clearly stated where exactly in the Appendix each Theorem is proved. \n\n\n6. In addition to the limited applicability of these results in explaining the behavior of SVGD (as the most practically relevant case is that of non-logconcave targets) and the absence of satisfactory finite-particle discrete-time guarantees, I am somewhat concerned about the technical novelty of these results. To the best of my understanding, It seems like the mean-field continuous-time guarantees of this work can be easily obtained from the well-established results on SVGD [5,6] by restricting to the (finite-dimensional) Gaussian submanifold. While limited theoretical contribution in itself is not a major weakness, I find it difficult to recommend acceptance given the unsatisfactory practical usefulness of these results. \n\n\n[1] Salim et. al., “A Convergence Theory for SVGD in the Population Limit under Talagrand's Inequality T1”\n\n[2] Sun et. al., “Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition”\n\n[3] Lambert et. al., “Variational inference via Wasserstein gradient flows”\n\n[4] Diao et. al., “Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space”\n\n[5] Duncan et. al., “On the geometry of Stein variational gradient descent”\n\n[6] Liu, “Stein Variational Gradient Descent as Gradient Flow”\n\nQUESTIONS:\n1. Could you obtain discrete-time finite-particle rates for Gaussian SVGD under the settings considered in your results. If not, could you highlight what would be the technical challenges involved in obtaining such a result?\n\n\n2. Could you clarify why the mean-field continuous-time Gaussian SVGD dynamics is of interest, when (to the best of my knowledge) : 1) It does not satisfactorily explain the behavior of SVGD in practical scenarios, 2) It is outperformed by Bures-Wasserstein Gradient Flows (as well as the practically implementable Bures-JKO scheme) in most settings\n\n\n3. Could you comment on the technical novelty of these results? To the best of my understanding, the mean-field guarantees can be obtained by adapting the existing analysis of the Gradient Flow induced by SVGD [5,6] to the submanifold of Gaussian densities (a technique which is well-established for Wasserstein gradient flows in [3]). \n\n\n4. I would recommend reworking the overall presentation. Explicitly stating where each theorem is proven in the Appendix would be a good starting point.\n\nLIMITATIONS:\nI found the discussion on the limitations of this work to be somewhat unsatisfactory. The authors state that limitations are discussed on Section 6 (Other Related Works) and Section 7 (Conclusion). Section 6 discusses some of the prior literature and Section 7 highlights some future directions. Neither of these sections adequately discuss the limitations of this work (e.g. applicability is primarily restricted to centered Gaussian targets for most of the results, finite-particle discrete-time rates are absent in most settings)", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the behavior of Gaussian-SVGD (SVGD projected to the space of Gaussian distributions, using bilinear kernels) and its variants.  The authors studied the behavior of the mean-field PDEs and established for Gaussian targets finite-particle convergence results which are significantly better than previous results for nonparametric kernels and targets. They have also derived density- and particle-based implementations of Gaussian-SVGD for general targets, which are shown to generalize recent works on Gaussian variational inference (GVI) with algorithmic guarantees.  One of the proposed algorithms outperform recent works on a Bayesian logistic regression experiment.\n\nSTRENGTHS:\n- The theoretical results appear interesting (to me as a non-expert, and note that I didn't check the proofs): the mean-field results sometimes improve over recent works on Gaussian variational inference with algorithmic guarantees, and the finite-particle results apply to standard SVGD, albeit with the other restrictions.\n\n- The discussions on connections between Gaussian SVGD and existing GVI approaches are also interesting, and one of the proposed methods appear promising empirically.\n\nWEAKNESSES:\n- While the restrictions to Gaussian families and affine kernels are similar to some of the recent works, it inevitably limits the scope of this work as much of the interest around SVGD arises from its flexibility.\n\n- While Gaussian variational inference in general has demonstrated competitive empirical performance, it is unclear if the proposed approach maintains this property, as it is not clear if the more complex algorithms interact nicely with common modifications such as minibatching.  This is also relevant because while some of the recent works (e.g., [17, 36]) also studied alternative GVI approaches, they come with full algorithmic guarantees, whereas for the present method there is no qualitative convergence guarantees for the discrete-time, finite-particle case.\n\nQUESTIONS:\nFor the experiments it would be helpful to compare with ordinary gradient descent on the variational parameters, to provide some ideas on the practical utility.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}