SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaxin Shi*
Stanford, CA 94305
jiaxins@stanford.edu
Lester Mackey
Microsoft Research New England
Cambridge, MA 02474
lmackey@microsoft.com
We provide the first finite-particle convergence rate for Stein variational gradient descent (SVGD), a popular algorithm for approximating a probability distribution with a collection of particles. Specifically, whenever the target distribution is sub-Gaussian with a Lipschitz score, SVGD with  $n$  particles and an appropriate step size sequence drives the kernel Stein discrepancy to zero at an order  $1 / \sqrt{\log\log n}$  rate. We suspect that the dependence on  $n$  can be improved, and we hope that our explicit, non-asymptotic proof strategy will serve as a template for future refinements.
Stein variational gradient descent [SVGD, 18] is an algorithm for approximating a target probability distribution  $P$  on  $\mathbb{R}^d$  with a collection of  $n$  particles. Given an initial particle approximation  $\mu_0^n = \frac{1}{n}\sum_{i=1}^{n}\delta_{x_i}$  with locations  $x_i \in \mathbb{R}^d$ , SVGD (Algorithm 1) iteratively evolves the particle locations to provide a more faithful approximation of the target  $P$  by performing optimization in the space of probability measures. SVGD has demonstrated encouraging results for a wide variety of inferential tasks, including approximate inference [18, 28, 27], generative modeling [26, 13], and reinforcement learning [11, 21].
Despite the popularity of SVGD, relatively little is known about its approximation quality. A first analysis by Liu [17, Thm. 3.3] showed that continuous SVGD—that is, Algorithm 2 initialized with a continuous distribution  $\mu_0^\infty$  in place of the discrete particle approximation  $\mu_0^n$ —converges to  $P$  in kernel Stein discrepancy [KSD, 4, 19, 9]. KSD convergence is also known to imply weak convergence [9, 3, 12, 1] and Wasserstein convergence [14] under various conditions on the target  $P$  and the SVGD kernel  $k$ . Follow-up work by Korba et al. [15], Salim et al. [22], Sun et al. [24] sharpened the result of Liu with path-independent constants, weaker smoothness conditions, and explicit rates of convergence. In addition, Duncan et al. [6] analyzed the continuous-time limit of continuous SVGD to provide conditions for exponential convergence. However, each of these analyses applies only to continuous SVGD and not to the finite-particle algorithm used in practice.
To bridge this gap, Liu [17, Thm. 3.2] showed that  $n$ -particle SVGD converges to continuous SVGD in bounded-Lipschitz distance but only under boundedness assumptions violated by most applications of SVGD. To provide a more broadly applicable proof of convergence, Gorham et al. [10, Thm. 7] showed that  $n$ -particle SVGD converges to continuous SVGD in 1-Wasserstein distance under assumptions commonly satisfied in SVGD applications. However, both convergence results are asymptotic, providing neither explicit error bounds nor rates of convergence. Korba et al. [15, Prop. 7] explicitly bounded the expected squared Wasserstein distance between  $n$ -particle and continuous SVGD but only under the assumption of bounded  $\nabla \log p$ , an assumption that rules out all strongly
*Part of this work was done at Microsoft Research New England.
Algorithm 1  $n$ -particle Stein Variational Gradient Descent [18]:  $\mathrm{SVGD}(\mu_0^n,r)$
Input: Target  $P$  with density  $p$ , kernel  $k$ , step sizes  $(\epsilon_s)_{s \geq 0}$ , particle approximation  $\mu_0^n = \frac{1}{n} \sum_{i=1}^{n} \delta_{x_i}$ , rounds  $r$
for  $s = 0,\dots ,r - 1$  do
$$
x _ {i} \leftarrow x _ {i} + \epsilon_ {s} \frac {1}{n} \sum_ {j = 1} ^ {n} k \left(x _ {j}, x _ {i}\right) \nabla \log p \left(x _ {j}\right) + \nabla_ {x} k \left(x _ {j}, x _ {i}\right) \text {f o r} i = 1, \dots , n.
$$
Output: Updated approximation  $\mu_r^n = \frac{1}{n}\sum_{i = 1}^n\delta_{x_i}$  of the target  $P$
Algorithm 2 Generic Stein Variational Gradient Descent [18]:  $\operatorname{SVGD}(\mu_0, r)$
Input: Target  $P$  with density  $p$ , kernel  $k$ , step sizes  $(\epsilon_s)_{s \geq 0}$ , approximating measure  $\mu_0$ , rounds  $r$  for  $s = 0, \dots, r - 1$  do
Let  $\mu_{s + 1}$  be the distribution of  $X^{s} + \epsilon_{s}\int k(x,X^{s})\nabla \log p(x) + \nabla_{x}k(x,X^{s})d\mu_{s}(x)$  for  $X^s\sim \mu_s$
Output: Updated approximation  $\mu_r$  of the target  $P$
log concave or dissipative distributions and all distributions for which the KSD is currently known to control weak convergence [9, 3, 12, 1]. In addition, Korba et al. [15] do not provide a unified bound for the convergence of  $n$ -particle SVGD to  $P$  and ultimately conclude that "the convergence rate for SVGD using  $[\mu_0^n]$  remains an open problem." The same open problem was underscored in the later work of Salim et al. [22], who write "an important and difficult open problem in the analysis of SVGD is to characterize its complexity with a finite number of particles."
In this work, we derive the first unified convergence bound for finite-particle SVGD to its target. To achieve this, we first bound the 1-Wasserstein discretization error between finite-particle and continuous SVGD under assumptions commonly satisfied in SVGD applications and compatible with KSD weak convergence control (see Theorem 1). We next bound KSD in terms of 1-Wasserstein distance and SVGD moment growth to explicitly control KSD discretization error in Theorem 2. Finally, Theorem 3 combines our results with the established KSD analysis of continuous SVGD to arrive at an explicit KSD error bound for  $n$ -particle SVGD.
2 Notation and Assumptions
Throughout, we fix a nonnegative step size sequence  $(\epsilon_s)_{s\geq 0}$ , a target distribution  $P$  in the set  $\mathcal{P}_1$  of probability measures on  $\mathbb{R}^d$  with integrable first moments, and a reproducing kernel  $k$  — a symmetric positive definite function mapping  $\mathbb{R}^d\times \mathbb{R}^d$  to  $\mathbb{R}$  — with reproducing kernel Hilbert space (RKHS)  $\mathcal{H}$  and product RKHS  $\mathcal{H}^d\triangleq \bigotimes_{i = 1}^d\mathcal{H}$  [2]. We will use the terms "kernel" and "reproducing kernel" interchangeably. For all  $\mu ,\nu \in \mathcal{P}_1$ , we let  $\Gamma (\mu ,\nu)$  be the set of all couplings of  $\mu$  and  $\nu$ , i.e., joint probability distributions  $\gamma$  over  $\mathcal{P}_1\times \mathcal{P}_1$  with  $\mu$  and  $\nu$  as marginal distributions of the first and second variable respectively. We further let  $\mu \otimes \nu$  denote the independent coupling, the distribution of  $(X,Z)$  when  $X$  and  $Z$  are drawn independently from  $\mu$  and  $\nu$  respectively. With this notation in place we define the 1-Wasserstein distance between  $\mu ,\nu \in \mathcal{P}_1$  as  $W_{1}(\mu ,\nu)\triangleq$  inf  $\gamma \in \Gamma (\mu ,\nu)\mathbb{E}_{(X,Z)\sim \gamma}[\| Z - X\| _2]$  and introduce the shorthand  $m_{\mu ,x^{\star}}\triangleq \mathbb{E}_{\mu}[\| \cdot -x^{\star}\| ]_2$  for each  $x^{\star}\in \mathbb{R}^{d}$ ,  $m_{\mu ,P}\triangleq \mathbb{E}_{(X,Z)\sim \mu \otimes P}[\| X - Z\| _2]$ , and  $M_{\mu ,P}\triangleq \mathbb{E}_{(X,Z)\sim \mu \otimes P}[\| X - Z\| _2^2 ]$ . We further define the Kullback-Leibler (KL) divergence as  $\mathrm{KL}(\mu \| \nu)\triangleq \mathbb{E}_{\mu}[\log (\frac{d\mu}{d\nu})]$  when  $\mu$  is absolutely continuous with respect to  $\nu$  (denoted by  $\mu \ll \nu$ ) and as  $\infty$  otherwise.
Our analysis will make use of the following assumptions on the SVGD kernel and target distribution.
Assumption 1 (Lipschitz, mean-zero score function). The target distribution  $P \in \mathcal{P}_1$  has a differentiable density  $p$  with an  $L$ -Lipschitz score function  $s_p \triangleq \nabla \log p$ , i.e.,  $\| s_p(x) - s_p(y) \|_2 \leq L \| x - y \|_2$  for all  $x, y \in \mathbb{R}^d$ . Moreover,  $\mathbb{E}_P[s_p] = 0$  and  $s_p(x^\star) = 0$  for some  $x^\star \in \mathbb{R}^d$ .
Assumption 2 (Bounded kernel derivatives). The kernel  $k$  is twice differentiable and  $\sup_{x,y\in \mathbb{R}^d}\max (|k(x,y)|,\| \nabla_xk(x,y)\| _2,\| \nabla_y\nabla_xk(x,y)\|_{\mathrm{op}},\left\| \nabla_x^2 k(x,y)\right\|_{\mathrm{op}})\leq \kappa_1^2$  for  $\kappa_{1} > 0$ . Moreover, for all  $i,j\in \{1,2,\ldots ,d\}$ ,  $\sup_{x\in \mathbb{R}^d}\nabla_{y_i}\nabla_{y_j}\nabla_{x_i}\nabla_{x_j}k(x,y)|_{y = x}\leq \kappa_2^2$  for  $\kappa_{2} > 0$ .
Assumption 3 (Decaying kernel derivatives). The kernel  $k$  is differentiable and admits a  $\gamma \in \mathbb{R}$  such that, for all  $x, y \in \mathbb{R}^d$  satisfying  $\| x - y \|_2 \geq 1$ ,
$$
\left\| \nabla_ {x} k (x, y) \right\| _ {2} \leq \gamma / \| x - y \| _ {2}.
$$
Assumptions 1, 2, and 3 are both commonly invoked and commonly satisfied in the literature. For example, the Lipschitz score assumption is consistent with prior SVGD convergence analyses [17, 15, 22] and, by Gorham and Mackey [8, Prop. 1], the score  $s_p$  is mean-zero under the mild integrability condition  $\mathbb{E}_{X \sim P}[\| s_p(X)\|_2] < \infty$ . The bounded and decaying derivative assumptions have also been made in prior analyses [15, 9] and, as we detail in Appendix A, are satisfied by the kernels most commonly used in SVGD, like the Gaussian and inverse multiquadric (IMQ) kernels. Notably, in these cases, the bounds  $\kappa_1$  and  $\kappa_2$  are independent of the dimension  $d$ .
To leverage the continuous SVGD convergence rates of Salim et al. [22], we additionally assume that the target  $P$  satisfies Talagrand's  $T_{1}$  inequality [25, Def. 22.1]. Remarkably, Villani [25, Thm. 22.10] showed that Assumption 4 is equivalent to  $P$  being a sub-Gaussian distribution. Hence, this mild assumption holds for all strongly log concave  $P$  [23, Def. 2.9], all  $P$  satisfying the log Sobolev inequality [25, Thm. 22.17], and all distantly dissipative  $P$  for which KSD is known to control weak convergence [9, Def. 4].
Assumption 4 (Talagrand's  $T_{1}$  inequality [25, Def. 22.1]). For  $P \in \mathcal{P}_1$ , there exists  $\lambda > 0$  such that, for all  $\mu \in \mathcal{P}_1$ ,
$$
W _ {1} (\mu , P) \leq \sqrt {2 \mathrm {K L} (\mu \| P) / \lambda}.
$$
Finally we make use of the following notation specific to the SVGD algorithm.
Definition 1 (Stein operator). For any differentiable vector-valued function  $g: \mathbb{R}^d \to \mathbb{R}^d$ , the Langevin Stein operator [8] for  $P$  satisfying Assumption 1 is defined by
$$
\left(\mathcal {T} _ {P} g\right) (x) \triangleq \langle s _ {p} (x), g (x) \rangle + \nabla \cdot g (x) \quad f o r a l l \quad x \in \mathbb {R} ^ {d}.
$$
Definition 2 (Vector-valued Stein operator). For any differentiable function  $h: \mathbb{R}^d \to \mathbb{R}$ , the vector-valued Langevin Stein operator [19] for  $P$  satisfying Assumption 1 is defined by
$$
\left(\mathcal {A} _ {P} h\right) (x) \triangleq s _ {p} (x) h (x) + \nabla h (x) \quad f o r a l l \quad x \in \mathbb {R} ^ {d}.
$$
Definition 3 (SVGD transport map and pushforward). The SVGD transport map [18] for a target  $P$  satisfying Assumption 1, a kernel  $k$  satisfying Assumption 2, a step size  $\epsilon \geq 0$ , and an approximating distribution  $\mu \in \mathcal{P}_1$  takes the form
$$
T _ {\mu , \epsilon} (x) \triangleq x + \epsilon \mathbb {E} _ {X \sim \mu} [ (\mathcal {A} _ {P} k (\cdot , x)) (X) ] \quad f o r a l l \quad x \in \mathbb {R} ^ {d}.
$$
Moreover, the SVGD pushforward  $\Phi_{\epsilon}(\mu)$  represents the distribution of  $T_{\mu ,\epsilon}(X)$  when  $X\sim \mu$
Definition 4 (Kernel Stein discrepancy). The Langevin kernel Stein discrepancy [KSD, 4, 19, 9] for  $P$  satisfying Assumption 1,  $k$  satisfying Assumption 2, and measures  $\mu, \nu \in \mathcal{P}_1$  is given by
$$
\operatorname {K S D} _ {P} (\mu , \nu) \triangleq \sup  _ {\| g \| _ {\mathcal {H} ^ {d}} \leq 1} \mathbb {E} _ {\mu} [ \mathcal {T} _ {P} g ] - \mathbb {E} _ {\nu} [ \mathcal {T} _ {P} g ].
$$
Notably, the KSD so-defined is symmetric in its two arguments and satisfies the triangle inequality.
Lemma 1 (KSD symmetry and triangle inequality). Under Definition 4, for all  $\mu, \nu, \pi \in \mathcal{P}_1$ ,
$$
\mathrm {K S D} _ {P} (\mu , \nu) = \mathrm {K S D} _ {P} (\nu , \mu) \quad a n d \quad \mathrm {K S D} _ {P} (\mu , \nu) \leq \mathrm {K S D} _ {P} (\mu , \pi) + \mathrm {K S D} _ {P} (\pi , \nu).
$$
Proof. Fix any  $\mu, \nu, \pi \in \mathcal{P}_1$ . For symmetry, we note that  $g \in \mathcal{H}^d \Leftrightarrow f = -g \in \mathcal{H}^d$ , so
$$
\operatorname {KSD}_{P}(\mu ,\pi) = \sup_{\| g\|_{\mathcal{H}^{d}}\leq 1}\mathbb{E}_{\mu}[\mathcal{T}_{P}g] - \mathbb{E}_{\pi}[\mathcal{T}_{P}g] = \sup_{\| f\|_{\mathcal{H}^{d}}\leq 1}\mathbb{E}_{\pi}[\mathcal{T}_{P}f] - \mathbb{E}_{\mu}[\mathcal{T}_{P}f] = \operatorname {KSD}_{P}(\pi ,\mu).
$$
To establish the triangle inequality, we write
$$
\begin{array}{l} \operatorname {K S D} _ {P} (\mu , \nu) = \sup  _ {\| g \| _ {\mathcal {H} ^ {d}} \leq 1} \mathbb {E} _ {\mu} \left[ \mathcal {T} _ {P} g \right] - \mathbb {E} _ {\pi} \left[ \mathcal {T} _ {P} g \right] + \mathbb {E} _ {\pi} \left[ \mathcal {T} _ {P} g \right] - \mathbb {E} _ {\nu} \left[ \mathcal {T} _ {P} g \right] \\ \leq \sup  _ {\| g \| _ {\mathcal {H} ^ {d}} \leq 1} \left(\mathbb {E} _ {\mu} \left[ \mathcal {T} _ {P} g \right] - \mathbb {E} _ {\pi} \left[ \mathcal {T} _ {P} g \right]\right) + \sup  _ {\| h \| _ {\mathcal {H} ^ {d}} \leq 1} \left(\mathbb {E} _ {\pi} \left[ \mathcal {T} _ {P} h \right] - \mathbb {E} _ {\nu} \left[ \mathcal {T} _ {P} h \right]\right) \\ \leq \mathrm {K S D} _ {P} (\mu , \pi) + \mathrm {K S D} _ {P} (\pi , \nu). \\ \end{array}
$$
3 Wasserstein Discretization Error of SVGD
Our first main result concerns the discretization error of SVGD and shows that  $n$ -particle SVGD remains close to its continuous SVGD limit whenever the step size sum  $b_{r-1} = \sum_{s=0}^{r-1} \epsilon_s$  is sufficiently small.
Theorem 1 (Wasserstein discretization error of SVGD). Suppose Assumptions 1, 2, and 3 hold. For any  $\mu_0^n, \mu_0^\infty \in \mathcal{P}_1$ , the outputs  $\mu_r^n = \mathrm{SVGD}(\mu_0^n, r)$  and  $\mu_r^\infty = \mathrm{SVGD}(\mu_0^\infty, r)$  of Algorithm 2 satisfy
$$
\log \left(\frac {W _ {1} \left(\mu_ {r} ^ {n} , \mu_ {r} ^ {\infty}\right)}{W _ {1} \left(\mu_ {0} ^ {n} , \mu_ {0} ^ {\infty}\right)}\right) \leq b _ {r - 1} (A + B \exp \left(C b _ {r - 1}\right))
$$
for  $b_{r - 1}\triangleq \sum_{s = 0}^{r - 1}\epsilon_s$ ,  $A = (c_{1} + c_{2})(1 + m_{P,x^{\star}})$ ,  $B = c_{1}m_{\mu_{0}^{n},P} + c_{2}m_{\mu_{0}^{\infty},P}$ ,  $C = \kappa_1^2 (3L + 1)$ ,  $c_{1} = \max (\kappa_{1}^{2}L,\kappa_{1}^{2})$ , and  $c_{2} = \kappa_{1}^{2}(L + 1) + L\max (\gamma ,\kappa_{1}^{2})$
We highlight that Theorem 1 applies to any  $\mathcal{P}_1$  initialization of SVGD: the initial particles supporting  $\mu_0^n$  could, for example, be drawn i.i.d. from a convenient auxiliary distribution  $\mu_0^\infty$  or even generated deterministically from some quadrature rule. To marry this result with the continuous SVGD convergence bound of Section 5, we will ultimately require  $\mu_0^\infty$  to be a continuous distribution with finite  $\mathrm{KL}(\mu_0^\infty \| P)$ . Hence, our primary desideratum for SVGD initialization is that  $\mu_0^n$  have small Wasserstein distance to some  $\mu_0^\infty$  with  $\mathrm{KL}(\mu_0^\infty \| P) < \infty$ . Then, by Theorem 1, the SVGD discretization error  $W_{1}(\mu_{r}^{n},\mu_{r}^{\infty})$  will remain small whenever the step size sum is not too large.
The proof of Theorem 1 in Section 6 relies on two lemmas. The first, due to Gorham et al. [10], shows that the one-step SVGD pushforward map  $\Phi_{\epsilon}$  (Definition 3) is pseudo-Lipschitz with respect to the 1-Wasserstein distance whenever the score function  $\nabla \log p$  and kernel  $k$  fulfill a commonly-satisfied pseudo-Lipschitz condition. Here, for any  $g:\mathbb{R}^d\to \mathbb{R}^d$ , we define the Lipschitz constant  $\mathrm{Lip}(g)\triangleq \sup_{x,z\in \mathbb{R}^d}\| g(x) - g(z)\| _2 / \| x - z\| _2$ .
Lemma 2 (Wasserstein pseudo-Lipschitzness of SVGD [10, Lem. 12]). For  $P$  satisfying Assumption 1, suppose that the following pseudo-Lipschitz bounds hold
$$
\operatorname {L i p} \left(s _ {p} (x) k (x, \cdot) + \nabla_ {x} k (x, \cdot)\right) \leq c _ {1} \left(1 + \| x - x ^ {\star} \| _ {2}\right),
$$
$$
\operatorname {L i p} \left(s _ {p} k (\cdot , z) + \nabla_ {x} k (\cdot , z)\right) \leq c _ {2} \left(1 + \| z - x ^ {\star} \| _ {2}\right).
$$
for some constants  $c_{1}, c_{2} \in \mathbb{R}$  and all  $x, z \in \mathbb{R}^{d}$ . Then, for any  $\mu, \nu \in \mathcal{P}_{1}$ ,
$$
W _ {1} \left(\Phi_ {\epsilon} (\mu), \Phi_ {\epsilon} (\nu)\right) \leq W _ {1} (\mu , \nu) (1 + \epsilon c _ {\mu , \nu}),
$$
where  $\Phi_{\epsilon}$  is the one-step SVGD pushforward (Definition 3) and  $c_{\mu,\nu} = c_1(1 + m_{\mu,x^{\star}}) + c_2(1 + m_{\nu,x^{\star}})$ .
In Section 6, we will show that, under Assumptions 1, 2, and 3, the preconditions of Lemma 2 are fulfilled with  $c_{1}$  and  $c_{2}$  exactly as in Theorem 1. The second lemma, proved in Section 7, controls the growth of the first and second absolute moments under SVGD.
Lemma 3 (SVGD moment growth). Suppose Assumptions 1 and 2 hold, and let  $C = \kappa_1^2(3L + 1)$ . Then the SVGD output  $\mu_r$  of Algorithm 2 with  $b_{r-1} \triangleq \sum_{s=0}^{r-1} \epsilon_s$  satisfies
$$
\begin{array}{l} m _ {\mu_ {r}, x ^ {*}} - m _ {P, x ^ {*}} \leq m _ {\mu_ {r}, P} \leq m _ {\mu_ {0}, P} \prod_ {s = 0} ^ {r - 1} (1 + \epsilon_ {s} C) \leq m _ {\mu_ {0}, P} \exp \left(C b _ {r - 1}\right), \\ M _ {\mu_ {r}, P} \leq M _ {\mu_ {0}, P} \prod_ {s = 0} ^ {r - 1} (1 + \epsilon_ {s} C) ^ {2} \leq M _ {\mu_ {0}, P} \exp (2 C b _ {r - 1}). \\ \end{array}
$$
The key to the proof of Lemma 3 is that we show the norm of any SVGD update, i.e.,  $\| T_{\mu ,\epsilon}(x) - x\| _2$  is controlled by  $m_{\mu ,P}$ , the first absolute moment of  $\mu$  measured against  $P$ . This is mainly due to the Lipschitzness of the score function  $s_p$  and our assumptions on the boundedness of the kernel and its derivatives. Then, we can use the result to control the growth of  $m_{\mu_r,P}$  across iterations since  $m_{\mu_{r + 1},P} = \mathbb{E}_{(X,Z)\sim \mu_r\otimes P}[\| T_{\mu_r,\epsilon_r}(X) - Z\| _2]$ . The same strategy applies to the second absolute moment  $M_{\mu ,P}$ . The proof of Theorem 1 then follows directly from Lemma 2 where we plug in the first moment bound of Lemma 3.
2We say a map  $\Phi :\mathcal{P}_1\to \mathcal{P}_1$  is pseudo-Lipschitz with respect to 1-Wasserstein distance if, for a constant  $C_\Phi \in \mathbb{R}$  , some  $x^{\star}\in \mathbb{R}^{d}$  , and all  $\mu ,\nu \in \mathcal{P}_1$ $W_{1}(\Phi (\mu),\Phi (\nu))\leq W_{1}(\mu ,\nu)\left(1 + m_{\mu ,x^{\star}} + m_{\nu ,x^{\star}}\right)C_{\Phi}$
4 KSD Discretization Error of SVGD
Our next result translates the Wasserstein error bounds of Theorem 1 into KSD error bounds.
Theorem 2 (KSD discretization error of SVGD). Suppose Assumptions 1, 2, and 3 hold. For any  $\mu_0^n, \mu_0^\infty \in \mathcal{P}_1$ , the outputs of Algorithm 2,  $\mu_r^n = \mathrm{SVGD}(\mu_0^n, r)$  and  $\mu_r^\infty = \mathrm{SVGD}(\mu_0^\infty, r)$ , satisfy
$$
\begin{array}{l} \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, \mu_ {r} ^ {\infty}\right) \leq \left(\kappa_ {1} L + \kappa_ {2} d\right) w _ {0, n} \exp \left(b _ {r - 1} (A + B \exp \left(C b _ {r - 1}\right))\right) \\ + \kappa_ {1} d ^ {1 / 4} L \sqrt {2 M _ {\mu_ {0} ^ {\infty} , P W _ {0 , n}}} \exp (b _ {r - 1} (2 C + A + B \exp (C b _ {r - 1})) / 2) \\ \end{array}
$$
for  $w_{0,n} \triangleq W_1(\mu_0^n, \mu_0^\infty)$  and  $A, B, C$  defined as in Theorem 1.
Our proof of Theorem 2 relies on the following lemma, proved in Section 8, that shows that the KSD is controlled by the 1-Wasserstein distance.
Lemma 4 (KSD-Wasserstein bound). Suppose Assumptions 1 and 2 hold. For any  $\mu, \nu \in \mathcal{P}_1$ ,
$$
\mathrm {K S D} _ {P} (\mu , \nu) \leq \left(\kappa_ {1} L + \kappa_ {2} d\right) W _ {1} (\mu , \nu) + \kappa_ {1} d ^ {1 / 4} L \sqrt {2 M _ {\nu , P} W _ {1} (\mu , \nu)}.
$$
Lemma 4 is proved in two steps. We first linearize  $(\mathcal{T}_Pg)(x)$  in the KSD definition through the Lipschitzness of  $s_p$  and the boundedness and Lipschitzness of RKHS functions. Then, we assign a 1-Wasserstein optimal coupling of  $(\mu ,\nu)$  to obtain the Wasserstein bound on the right.
Proof of Theorem 2 The result follows directly from Lemma 4, the second moment bound of Lemma 3, and Theorem 1.
5 A Finite-particle Convergence Rate for SVGD
To establish our main SVGD convergence result, we combine Theorems 1 and 2 with the following descent lemma for continuous SVGD error due to Salim et al. [22] which shows that continuous SVGD decreases the KL divergence to  $P$  and drives the KSD to  $P$  to zero.
Lemma 5 (Continuous SVGD descent lemma [22, Thm. 3.2]). Suppose Assumptions 1, 2, and 4 hold, and consider the outputs  $\mu_r^\infty = \mathrm{SVGD}(\mu_0^\infty, r)$  and  $\mu_{r+1}^\infty = \mathrm{SVGD}(\mu_0^\infty, r+1)$  of Algorithm 2 with  $\mu_0^\infty \ll P$ . If  $\max_{0 \leq s \leq r} \epsilon_s \leq R_{\alpha,2}$  for some  $\alpha > 1$  and
$$
R _ {\alpha , p} \triangleq \min  \left(\frac {p}{\kappa_ {1} ^ {2} (L + \alpha^ {2})}, (\alpha - 1) \left(1 + L m _ {\mu_ {0} ^ {\infty}, x ^ {\star}} + 2 L \sqrt {\frac {2}{\lambda} \mathrm {K L} \left(\mu_ {0} ^ {\infty} \| P\right)}\right)\right) f o r p \in \{1, 2 \}, \tag {1}
$$
$$
\mathrm {K L} \left(\mu_ {r + 1} ^ {\infty} \| P\right) - \mathrm {K L} \left(\mu_ {r} ^ {\infty} \| P\right) \leq - \epsilon_ {r} \left(1 - \frac {\kappa_ {1} ^ {2} (L + \alpha^ {2})}{2} \epsilon_ {r}\right) \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {\infty}, P\right) ^ {2}. \tag {2}
$$
By summing the result (2) over  $r \in \{0, \dots, t\}$ , we obtain the following corollary.
Corollary 1. Under the assumptions and notation of Lemma 5, suppose  $\max_{0\leq r\leq t}\epsilon_r\leq R_{\alpha ,1}$  for some  $\alpha >1$  , and let  $\pi_r\triangleq \frac{c(\epsilon_r)}{\sum_{r = 0}^t c(\epsilon_r)}$  for  $c(\epsilon)\triangleq \epsilon \Big(1 - \frac{\kappa_1^2(L + \alpha^2)}{2}\epsilon \Big).$  Since  $\frac{\epsilon}{2}\leq c(\epsilon) <   \epsilon$  we have
$$
\sum_ {r = 0} ^ {t} \pi_ {r} \mathrm {K S D} _ {P} (\mu_ {r} ^ {\infty}, P) ^ {2} \leq \frac {1}{\sum_ {r = 0} ^ {t} c (\epsilon_ {r})} \mathrm {K L} (\mu_ {0} ^ {\infty} \| P) \leq \frac {2}{\sum_ {r = 0} ^ {t} \epsilon_ {r}} \mathrm {K L} (\mu_ {0} ^ {\infty} \| P).
$$
Finally, we arrive at our main result that bounds the approximation error of  $n$ -particle SVGD in terms of the chosen step size sequence and the initial discretization error  $W_{1}(\mu_{0}^{n}, \mu_{0}^{\infty})$ .
Theorem 3 (KSD error of finite-particle SVGD). Suppose Assumptions 1, 2, 3, and 4 hold, fix any  $\mu_0^\infty \ll P$  and  $\mu_0^n\in \mathcal{P}_1$ , and let  $w_{0,n}\triangleq W_1(\mu_0^n,\mu_0^\infty)$ . If  $\max_{0\leq r < t}\epsilon_r\leq \epsilon_t\triangleq R_{\alpha ,1}^3$  for some  $\alpha >1$  and  $R_{\alpha ,1}$  defined in Lemma 5, then the Algorithm 2 outputs  $\mu_r^n = \mathrm{SVGD}(\mu_0^n,r)$  satisfy
$$
\min  _ {0 \leq r \leq t} \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) \leq \sum_ {r = 0} ^ {t} \pi_ {r} \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) \leq a _ {t - 1} + \sqrt {\frac {2}{R _ {\alpha , 1} + b _ {t - 1}} \mathrm {K L} \left(\mu_ {0} ^ {\infty} \| P\right)}, \tag {3}
$$
for  $\pi_r$  as defined in Lemma 5,  $(A,B,C)$  as defined in Theorem 1,  $b_{t - 1}\triangleq \sum_{r = 0}^{t - 1}\epsilon_r$ , and
$$
\begin{array}{l} a _ {t - 1} \triangleq \left(\kappa_ {1} L + \kappa_ {2} d\right) w _ {0, n} \exp \left(b _ {t - 1} (A + B \exp \left(C b _ {t - 1}\right))\right) \tag {4} \\ + \kappa_ {1} d ^ {1 / 4} L \sqrt {2 M _ {\mu_ {0} ^ {\infty} , P} w _ {0 , n}} \exp (b _ {t - 1} (2 C + A + B \exp (C b _ {t - 1})) / 2). \\ \end{array}
$$
Note that the value assigned to  $\epsilon_{t}$  does not have any impact on the algorithm that generates  $\mu_r^n$  when  $r\leq t$
Proof. By the triangle inequality (Lemma 1) and Theorem 2 we have