SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) - \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {\infty}, P\right) \right| \leq \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, \mu_ {r} ^ {\infty}\right) \leq a _ {r - 1}
$$
for each  $r$ . Therefore
$$
\sum_ {r = 0} ^ {t} \pi_ {r} \left(\mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) - a _ {r - 1}\right) ^ {2} \leq \sum_ {r = 0} ^ {t} \pi_ {r} \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {\infty}, P\right) ^ {2} \leq \frac {2}{R _ {\alpha , 1} + b _ {t - 1}} \mathrm {K L} \left(\mu_ {0} ^ {\infty} \| P\right), \tag {5}
$$
where the last inequality follows from Corollary 1. Moreover, by Jensen's inequality,
$$
\sum_ {r = 0} ^ {t} \pi_ {r} \left(\mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) - a _ {r - 1}\right) ^ {2} \geq \left(\sum_ {r = 0} ^ {t} \pi_ {r} \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) - \sum_ {r = 0} ^ {t} \pi_ {r} a _ {r - 1}\right) ^ {2}. \tag {6}
$$
Combining (5) and (6), we have
$$
\sum_ {r = 0} ^ {t} \pi_ {r} \mathrm {K S D} _ {P} \left(\mu_ {r} ^ {n}, P\right) \leq \sum_ {r = 0} ^ {t} \pi_ {r} a _ {r - 1} + \sqrt {\frac {2}{R _ {\alpha , 1} + b _ {t - 1}} \mathrm {K L} \left(\mu_ {0} ^ {\infty} \| P\right)}.
$$
We finish the proof by noticing that  $\sum_{r=0}^{t} \pi_r a_{r-1} \leq \max_{0 \leq r \leq t} a_{r-1} = a_{t-1}$ .
The following corollary, proved in Appendix B, provides an explicit SVGD convergence bound and rate by choosing the step size sum to balance the terms on the right-hand side of (3). In particular, Corollary 2 instantiates an explicit SVGD step size sequence that drives the kernel Stein discrepancy to zero at an order  $1 / \sqrt{\log\log(n)}$  rate.
Corollary 2 (A finite-particle convergence rate for SVGD). Instantiate the notation and assumptions of Theorem 3, let  $(\bar{w}_{0,n},\bar{A},\bar{B},\bar{C})$  be any upper bounds on  $(w_{0,n},A,B,C)$  respectively, and define the growth functions
$$
\phi (w) \triangleq \log \log (e ^ {e} + \frac {1}{w}) \quad a n d \quad \psi_ {\bar {B}, \bar {C}} (x, y, \beta) \triangleq \frac {1}{\bar {C}} \log (\frac {1}{B} \max  (\bar {B}, \frac {1}{\beta} \log \frac {1}{x} - y)).
$$
If the step size sum  $b_{t-1} = \sum_{r=0}^{t-1} \epsilon_r = s_n^\star$  for
$$
s _ {n} ^ {\star} \triangleq \min  \left(\psi_ {\bar {B}, \bar {C}} \big (\bar {w} _ {0, n} \sqrt {\phi (\bar {w} _ {0 , n})}, \bar {A}, \beta_ {1} \big), \psi_ {\bar {B}, \bar {C}} \big (\bar {w} _ {0, n} \phi (\bar {w} _ {0, n}), \bar {A} + 2 \bar {C}, \beta_ {2} \big)\right),
$$
$$
\beta_ {1} \triangleq \max  \left(1, \psi_ {\bar {B}, \bar {C}} \left(\bar {w} _ {0, n} \sqrt {\phi (\bar {w} _ {0 , n})}, \bar {A}, 1\right)\right), \quad a n d
$$
$$
\beta_ {2} \triangleq \max  \left(1, \psi_ {\bar {B}, \bar {C}} (\bar {w} _ {0, n} \phi (\bar {w} _ {0, n}), \bar {A} + 2 \bar {C}, 1)\right)
$$
$$
\begin{array}{l} \min_{0\leq r\leq t}\operatorname{KSD}_{P}(\mu_{r}^{n},P) \\ \leq \left\{\begin{array}{l l}\left(\kappa_ {1} L + \kappa_ {2} d\right) \bar {w} _ {0, n} + \kappa_ {1} d ^ {1 / 4} L \sqrt {2 M _ {\mu_ {0} ^ {\infty} , P} \bar {w} _ {0 , n}} + \sqrt {\frac {2}{R _ {\alpha , 1}} \mathrm {K L} \left(\mu_ {0} ^ {\infty} \| P\right)}&i f s _ {n} ^ {\star} = 0\\\frac {\left(\kappa_ {1} L + \kappa_ {2} d\right) + \kappa d ^ {1 / 4} L \sqrt {2 M _ {\mu_ {0} ^ {\infty} , P}}}{\sqrt {\phi (\bar {w} _ {0 , n})}} + \sqrt {\frac {2 \mathrm {K L} \left(\mu_ {0} ^ {\infty} \| P\right)}{R _ {\alpha , 1} + \frac {1}{C} \log \left( \right.\frac {1}{B} \left(\frac {\log \left(1 / \left(\bar {w} _ {0 , n} \phi \left(\bar {w} _ {0 , n}\right)\right)\right)}{\max \left(1 , \psi_ {\bar {B}} , \bar {C} \left(\bar {w} _ {0 , n} , 0 , 1\right)\right)} - \bar {A} - 2 \bar {C}\right))}}&o t h e r w i s e\end{array}\right. (7) \\ = O \left(\frac {1}{\sqrt {\log \log \left(e ^ {e} + \frac {1}{\bar {w} _ {0 , n}}\right)}}\right). (8) \\ \end{array}
$$
If, in addition,  $\mu_0^n = \frac{1}{n}\sum_{i = 1}^n\delta_{x_i}$  for  $x_{i}\stackrel {i.i.d.}{\sim}\mu_{0}^{\infty}$  with  $M_{\mu_0^\infty}\triangleq \mathbb{E}_{\mu_0^\infty}[\| \cdot \| _2^2 ] < \infty$  then
$$
\bar {w} _ {0, n} \triangleq \frac {M _ {\mu_ {0} ^ {\infty}} \log (n) ^ {\mathbb {I} [ d = 2 ]}}{\delta n ^ {1 / (2 \vee d)}} \geq w _ {0, n} \tag {9}
$$
with probability at least  $1 - c\delta$  for a universal constant  $c > 0$ . Hence, with this choice of  $\bar{w}_{0,n}$
$$
\min _ {0 \leq r \leq t} \mathrm {K S D} _ {P} (\mu_ {r} ^ {n}, P) = O \left(\frac {1}{\sqrt {\log \log (n \delta)}}\right)
$$
with probability at least  $1 - c\delta$
Specifically, given any upper bounds  $(\bar{w}_{0,n},\bar{A},\bar{B},\bar{C})$  on the quantities  $(w_{0,n},A,B,C)$  defined in Theorem 3, Corollary 2 specifies a recommended step size sum  $s_n^\star$  to achieve an order  $1 / \sqrt{\log\log(e^e + \frac{1}{\bar{w}_{0,n}})}$  rate of SVGD convergence in KSD. Several remarks are in order. First,
the target step size sum  $s_n^\star$  is easily computed given  $(\bar{w}_{0,n},\bar{A},\bar{B},\bar{C})$ . Second, we note that the target  $s_n^\star$  can equal 0 when the initial Wasserstein upper bound  $\bar{w}_{0,n}$  is insufficiently small since  $\log \left(\frac{1}{B}\max \left(\bar{B},\frac{1}{\beta}\log \frac{1}{x} -y\right)\right) = \log \left(\frac{\bar{B}}{B}\right) = 0$  for small arguments  $x$ . In this case, the setting  $b_{t - 1} = 0$  amounts to not running SVGD at all or, equivalently, to setting all step sizes to 0.
Third, Corollary 2 also implies a time complexity for achieving an order  $1 / \sqrt{\log\log(e^{e} + \frac{1}{\bar{w}_{0,n}})}$  error rate. Recall that Theorem 3 assumes that  $\max_{0\leq r < t}\epsilon_r\leq R_{\alpha ,1}$  for  $R_{\alpha ,1}$  defined in (1). Hence,  $t^\star \triangleq \lceil s_n^\star /R_{\alpha ,1}\rceil$  rounds of SVGD are necessary to achieve the recommended setting  $\sum_{r = 0}^{t^{\star} - 1}\epsilon_{r} = s_{n}^{\star}$  while also satisfying the constraints of Theorem 3. Moreover,  $t^\star$  rounds are also sufficient if each step size is chosen equal to  $s_n^\star /t^\star$ . In addition, since  $s_n^\star = O(\log \log (e^e +\frac{1}{\bar{w}_{0,n}}))$ , Corollary 2 implies that SVGD can deliver  $\min_{0\leq r\leq t^{\star}}\mathrm{KSD}_P(\mu_r^n,P)\leq \Delta$  in  $t^\star = O(1 / \Delta^2)$  rounds. Since the computational complexity of each SVGD round is dominated by  $\Theta (n^2)$  kernel gradient evaluations (i.e., evaluating  $\nabla_xk(x_i,x_j)$  for each pair of particles  $(x_{i},x_{j})$ ), the overall computational complexity to achieve the order  $1 / \sqrt{\log\log(e^{e} + \frac{1}{\bar{w}_{0,n}})}$  error rate is  $O(n^{2}\lceil s_{n}^{\star} / R_{\alpha ,1}\rceil) = O(n^{2}\log \log (e^{e} + \frac{1}{\bar{w}_{0,n}}))$ .
6 Proof of Theorem 1: Wasserstein discretization error of SVGD
In order to leverage Lemma 2, we first show that the pseudo-Lipschitzness conditions of Lemma 2 hold given our Assumptions 1 to 3. Recall that  $s_p$  is Lipschitz and  $x^{\star}$  satisfies  $s_p(x^{\star}) = 0$  by Assumption 1. Then, by the triangle inequality, the definition of  $\| \cdot \|_{\mathrm{op}}$ ,  $\| k(x,z)\|_2 \leq \kappa_1^2$  and  $\| \nabla_z\nabla_xk(x,z)\|_{\mathrm{op}} \leq \kappa_1^2$  from Assumption 2, and Cauchy-Schwartz,
$$
\begin{array}{l} \operatorname {L i p} \left(s _ {p} (x) k (x, \cdot) + \nabla_ {x} k (x, \cdot)\right) \\ \leq \left\| s _ {p} (x) - s _ {p} \left(x ^ {\star}\right) \right\| _ {2} \left\| \nabla_ {z} k (x, z) \right\| _ {2} + \left\| \nabla_ {z} \nabla_ {x} k (x, z) \right\| _ {\mathrm {o p}} \\ = \sup  _ {\| u \| _ {2} \leq 1} \left(\| s _ {p} (x) - s _ {p} \left(x ^ {\star}\right) \| _ {2} \left| \nabla_ {z} k (x, z) ^ {\top} u \right|\right) + \| \nabla_ {z} \nabla_ {x} k (x, z) \| _ {\mathrm {o p}} \\ \leq L \| x - x ^ {\star} \| _ {2} \| \nabla_ {z} k (x, z) \| _ {2} + \| \nabla_ {z} \nabla_ {x} k (x, z) \| _ {\mathrm {o p}} \\ \leq \max  (\kappa_ {1} ^ {2} L, \kappa_ {1} ^ {2}) (1 + \| x - x ^ {\star} \| _ {2}). \\ \end{array}
$$
Letting  $c_{1} = \max(\kappa_{1}^{2}L, \kappa_{1}^{2})$  and taking supremum over  $z$  proves the first pseudo-Lipschitzness condition. Similarly, we have
$$
\begin{array}{l} \operatorname {L i p} \left(s _ {p} k (\cdot , z) + \nabla_ {x} k (\cdot , z)\right) \\ \leq \sup  _ {x \in \mathbb {R} ^ {d}} \operatorname {L i p} (s _ {p}) | k (x, z) | + \| s _ {p} (x) - s _ {p} (x ^ {\star}) \| _ {2} \| \nabla_ {x} k (x, z) \| _ {2} + \left\| \nabla_ {x} ^ {2} k (x, z) \right\| _ {\mathrm {o p}} \\ \leq \kappa_ {1} ^ {2} L + \sup  _ {x \in \mathbb {R} ^ {d}} L \| x - x ^ {\star} \| _ {2} \| \nabla_ {x} k (x, z) \| _ {2} + \kappa_ {1} ^ {2}, \tag {10} \\ \end{array}
$$
where we used the Lipschitzness of  $s_p$  from Assumption 1 and  $|k(x,z)| \leq \kappa_1^2$ ,  $\left\| \nabla_x^2 k(x,z) \right\|_{\mathrm{op}} \leq \kappa_1^2$  from Assumption 2. Now we consider two cases separately: when  $\| x - z \|_2 \geq 1$  and  $\| x - z \|_2 < 1$ .
- Case 1:  $\| x - z\| _2\geq 1$ . Recall that there exists  $\gamma >0$  such that  $\| \nabla_xk(x,z)\| _2\leq \gamma /\| x - z\| _2$  by Assumption 3. Then, using this together with the triangle inequality, we have
$$
\left\| x - x ^ {\star} \right\| _ {2} \left\| \nabla_ {x} k (x, z) \right\| _ {2} \leq \gamma \frac {\left\| x - z \right\| _ {2} + \left\| z - x ^ {\star} \right\| _ {2}}{\left\| x - z \right\| _ {2}} \leq \gamma \left(1 + \left\| z - x ^ {\star} \right\| _ {2}\right). \tag {11}
$$
- Case 2:  $\| x - z \|_2 < 1$ . Then, using  $\| \nabla_x k(x,z) \|_2 \leq \kappa_1^2$  from Assumption 2 and by the triangle inequality, we have
$$
\left\| x - x ^ {\star} \right\| _ {2} \left\| \nabla_ {x} k (x, z) \right\| _ {2} \leq \kappa_ {1} ^ {2} \left(\left\| x - z \right\| _ {2} + \left\| z - x ^ {\star} \right\| _ {2}\right) <   \kappa_ {1} ^ {2} \left(1 + \left\| z - x ^ {\star} \right\| _ {2}\right). \tag {12}
$$
Combining (11) and (12) and using the triangle inequality, we get
$$
\left\| x - x ^ {\star} \right\| _ {2} \left\| \nabla_ {x} k (x, z) \right\| _ {2} \leq \max  \left(\gamma , \kappa_ {1} ^ {2}\right) \left(1 + \left\| z - x ^ {\star} \right\| _ {2}\right). \tag {13}
$$
Plugging (13) back into (10), we can show the second pseudo-Lipschitzness condition holds for  $c_{2} = \max(\kappa_{1}^{2}(L + 1) + L\max(\gamma, \kappa_{1}^{2}), L\max(\gamma, \kappa_{1}^{2})) = \kappa_{1}^{2}(L + 1) + L\max(\gamma, \kappa_{1}^{2})$ .
Now we have proved that both pseudo-Lipschitzness preconditions of Lemma 2 hold under our Assumptions 1 to 3. By repeated application of Lemma 2 and the inequality  $(1 + x) \leq e^{x}$ , we have
$$
\begin{array}{l} W _ {1} \left(\mu_ {r + 1} ^ {n}, \mu_ {r} ^ {\infty}\right) = W _ {1} \left(\Phi_ {\epsilon_ {r}} \left(\mu_ {r} ^ {n}\right), \Phi_ {\epsilon_ {r}} \left(\mu_ {r} ^ {\infty}\right)\right) \leq \left(1 + \epsilon_ {r} D _ {r}\right) W \left(\mu_ {r} ^ {n}, \mu_ {r} ^ {\infty}\right) \\ \leq W _ {1} \left(\mu_ {0} ^ {n}, \mu_ {0} ^ {\infty}\right) \prod_ {s = 0} ^ {r} \left(1 + \epsilon_ {s} D _ {s}\right) \leq W _ {1} \left(\mu_ {0} ^ {n}, \mu_ {0} ^ {\infty}\right) \exp \left(\sum_ {s = 0} ^ {r} \epsilon_ {s} D _ {s}\right) \tag {14} \\ \end{array}
$$
for  $D_{s} = c_{1}(1 + m_{\mu_{s}^{\infty},x^{\star}}) + c_{2}(1 + m_{\mu_{s}^{\infty},x^{\star}})$ .
Using the result from Lemma 3, we have
$$
D _ {s + 1} \leq A + B \exp \left(C b _ {s}\right)
$$
for  $A = (c_{1} + c_{2})(1 + m_{P,x^{\star}})$ ,  $B = c_{1}m_{\mu_{0}^{n},P} + c_{2}m_{\mu_{0}^{\infty},P}$ , and  $C = \kappa_1^2 (3L + d)$ . Therefore
$$
\sum_ {s = 0} ^ {r} \epsilon_ {s} D _ {s} \leq \max  _ {0 \leq s \leq r} D _ {s} \sum_ {s = 0} ^ {r} \epsilon_ {s} \leq b _ {r} (A + B \exp (C b _ {r - 1})) \leq b _ {r} (A + B \exp (C b _ {r})).
$$
Plugging this back into (14) proves the result.
7 Proof of Lemma 3: SVGD moment growth
From Assumption 2 we know  $|k(y,x)| \leq \kappa_1^2$  and  $\left\| \nabla_y^2 k(y,x) \right\|_{op} \leq \kappa_1^2$ . The latter implies
$$
\left\| \nabla_ {y} k (y, x) - \nabla_ {z} k (z, x) \right\| _ {2} \leq \kappa_ {1} ^ {2} \| y - z \| _ {2}.
$$
Recall that  $s_p$  is Lipschitz and satisfies  $\mathbb{E}_P[s_p(\cdot)] = 0$  by Assumption 1. Let  $\mu$  be any probability measure. Using the above results, Jensen's inequality and the fact that  $\mathbb{E}_{Z\sim P}[(\mathcal{A}_PK(\cdot,x))(Z)] = 0$ , we have
$$
\begin{array}{l} \left\| T _ {\mu , \epsilon} (x) - x \right\| _ {2} \leq \epsilon \| \mathbb {E} _ {X \sim \mu} [ (\mathcal {A} _ {P} k (\cdot , x)) (X) ] \| _ {2} \\ = \epsilon \| \mathbb {E} _ {X \sim \mu} [ (\mathcal {A} _ {P} k (\cdot , x)) (X) ] - \mathbb {E} _ {Z \sim P} [ (\mathcal {A} _ {P} k (\cdot , x)) (Z) ] \| _ {2} \\ = \epsilon \| \mathbb {E} _ {(X, Z) \sim \mu \otimes P} [ k (Z, x) (s _ {p} (X) - s _ {p} (Z)) + (k (X, x) - k (Z, x)) (s _ {p} (X) - \mathbb {E} _ {P} [ s _ {p} (\cdot) ]) \\ \left. + \left(\nabla_ {X} k (X, x) - \nabla_ {Z} k (Z, x)\right) \right] \| _ {2} \\ \leq \epsilon \mathbb {E} _ {(X, Z) \sim \mu \otimes P} [ | k (Z, x) | \| s _ {p} (X) - s _ {p} (Z) \| _ {2} + (| k (X, x) | + | k (Z, x) |) \| s _ {p} (X) - \mathbb {E} _ {Y \sim P} [ s _ {p} (Y) ] \| _ {2} \\ + \| \nabla_ {X} k (X, x) - \nabla_ {Z} k (Z, x) \| _ {2} ] \\ \leq \epsilon \mathbb {E} _ {(X, Z) \sim \mu \otimes P} [ \kappa_ {1} ^ {2} (L + 1) \| X - Z \| _ {2} ] + \epsilon \cdot 2 \kappa_ {1} ^ {2} L \mathbb {E} _ {(X, Y) \sim \mu \otimes P} [ \| X - Y \| _ {2} ] \\ = \epsilon \kappa_ {1} ^ {2} (3 L + 1) \mathbb {E} _ {(X, Z) \sim \mu \otimes P} [ \| X - Z \| _ {2} ] \\ = \epsilon C m _ {\mu , P}. \tag {15} \\ \end{array}
$$
The last step used the definitions  $m_{\mu ,P}\triangleq \mathbb{E}_{(X,Z)\sim \mu \otimes P}[\| X - Z\| _2]$  and  $C = \kappa_1^2 (3L + 1)$ . Then, applying the triangle inequality and (15), we have
$$
\begin{array}{l} m _ {\mu_ {r + 1}, P} = \mathbb {E} _ {(X, Z) \sim \mu_ {r + 1} \otimes P} [ \| X - Z \| _ {2} ] = \mathbb {E} _ {(X, Z) \sim \mu_ {r} \otimes P} [ \| T _ {\mu_ {r}, \epsilon_ {r}} (X) - Z \| _ {2} ] \\ \leq \mathbb {E} _ {(X, Z) \sim \mu_ {r} \otimes P} [ \| T _ {\mu_ {r}, \epsilon_ {r}} (X) - X \| _ {2} + \| X - Z \| _ {2} ] \leq (1 + \epsilon_ {r} C) m _ {\mu_ {r}, P}, \tag {16} \\ \end{array}
$$
$$
\begin{array}{l} M _ {\mu_ {r + 1}, P} = \mathbb {E} _ {(X, Z) \sim \mu_ {r + 1} \otimes P} [ \| X - Z \| _ {2} ^ {2} ] = \mathbb {E} _ {(X, Z) \sim \mu_ {r} \otimes P} [ \| T _ {\mu_ {r}, \epsilon_ {r}} (X) - Z \| _ {2} ^ {2} ] \\ \leq \mathbb {E} _ {(X, Z) \sim \mu_ {r} \otimes P} [ \| T _ {\mu_ {r}, \epsilon_ {r}} (X) - X \| _ {2} ^ {2} + 2 \| T _ {\mu_ {r}, \epsilon_ {r}} (X) - X \| _ {2} \| X - Z \| _ {2} + \| X - Z \| _ {2} ^ {2} ] \\ \leq \left(\epsilon_ {r} ^ {2} C ^ {2} + 2 \epsilon_ {r} C\right) m _ {\mu_ {r}, P} ^ {2} + M _ {\mu_ {r}, P} \leq \left(1 + 2 \epsilon_ {r} C + \epsilon_ {r} ^ {2} C ^ {2}\right) M _ {\mu_ {r}, P} \\ = \left(1 + \epsilon_ {r} C\right) ^ {2} M _ {\mu_ {r}, P}, \tag {17} \\ \end{array}
$$
where the second last step used Jensen's inequality  $m_{\mu_r,P}^2 \leq M_{\mu_r,P}$ . Then, we repeatedly apply (16) and (17) together with the triangle inequality and the bound  $1 + x \leq e^{x}$  to get
$$
M _ {\mu_ {r}, P} \leq M _ {\mu_ {0}, P} \prod_ {s = 0} ^ {r - 1} (1 + \epsilon_ {s} C) ^ {2} \leq M _ {\mu_ {0}, P} \exp (2 C \sum_ {s = 0} ^ {r - 1} \epsilon_ {s}) \leq M _ {\mu_ {0}, P} \exp (2 C b _ {r - 1}) \text {a n d}
$$
$$
m _ {\mu_ {r}, x ^ {\star}} - m _ {P, x ^ {\star}} \leq m _ {\mu_ {r}, P} \leq m _ {\mu_ {0}, P} \prod_ {s = 0} ^ {r - 1} (1 + \epsilon_ {s} C) \leq m _ {\mu_ {0}, P} \exp \left(C b _ {r - 1}\right).
$$
8 Proof of Lemma 4: KSD-Wasserstein bound
Our proof generalizes that of Gorham and Mackey [9, Lem. 18]. Consider any  $g \in \mathcal{H}^d$  satisfying  $\| g\|_{\mathcal{H}^d}^2 \triangleq \sum_{i = 1}^d\| g_i\|_{\mathcal{H}}^2 \leq 1$ . From Assumption 2 we know
$$
\begin{array}{l} \left\| g (x) \right\| _ {2} ^ {2} \leq k (x, x) \sum_ {i = 1} ^ {d} \left\| g _ {i} \right\| _ {\mathcal {H}} ^ {2} \leq \kappa_ {1} ^ {2}, (18) \\ \| \nabla g (x) \| _ {o p} ^ {2} \leq \| \nabla g (x) \| _ {F} ^ {2} = \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} | \nabla_ {x _ {i}} g _ {j} (x) | ^ {2} \leq \| g \| _ {\mathcal {H} ^ {d}} ^ {2} \mathrm {t r} (\nabla_ {y} \nabla_ {x} k (x, y) | _ {y = x}) \\ \leq d \| \nabla_ {y} \nabla_ {x} k (x, y) | _ {y = x} \| _ {\mathrm {o p}} \leq \kappa_ {1} ^ {2} d, \text {a n d} (19) \\ \end{array}
$$
$$
\begin{array}{l} \| \nabla (\nabla \cdot g (x)) \| _ {2} ^ {2} = \sum_ {i = 1} ^ {d} \left(\sum_ {j = 1} ^ {d} \nabla_ {x _ {i}} \nabla_ {x _ {j}} g _ {j} (x)\right) ^ {2} \leq d \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} | \nabla_ {x _ {i}} \nabla_ {x _ {j}} g _ {j} (x) | ^ {2} \\ \leq d \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \| g _ {j} \| _ {\mathcal {H}} ^ {2} \big (\nabla_ {y _ {i}} \nabla_ {y _ {j}} \nabla_ {x _ {i}} \nabla_ {x _ {j}} k (x, y) | _ {y = x} \big) \leq \kappa_ {2} ^ {2} d ^ {2}, \\ \end{array}
$$
Suppose  $X, Y, Z$  are distributed so that  $(X, Y)$  is a 1-Wasserstein optimal coupling of  $(\mu, \nu)$  and  $Z$  is independent of  $(X, Y)$ . Since  $s_p$  is  $L$ -Lipschitz with  $\mathbb{E}_P[s_p] = 0$  (Assumption 1),  $g$  is bounded (18), and  $g$  and  $\nabla \cdot g$  are Lipschitz (19), repeated use of Cauchy-Schwarz gives
$$
\begin{array}{l} \mathbb {E} _ {\mu} \left[ \mathcal {T} _ {P} g \right] - \mathbb {E} _ {\nu} \left[ \mathcal {T} _ {P} g \right] \\ = \mathbb {E} [ \nabla \cdot g (X) - \nabla \cdot g (Y) ] + \mathbb {E} [ \langle s _ {p} (X) - s _ {p} (Y), g (X) \rangle ] + \mathbb {E} [ \langle s _ {p} (Y) - s _ {p} (Z), g (X) - g (Y) \rangle ] \\ \leq \left(\kappa_ {2} d + \kappa_ {1} L\right) W _ {1} (\mu , \nu) + L \mathbb {E} \left[ \| Y - Z \| _ {2} \min  \left(2 \kappa_ {1}, \kappa_ {1} \sqrt {d} \| X - Y \| _ {2}\right) \right]. \\ \end{array}
$$
Since our choice of  $g$  was arbitrary, the first advertised result now follows from the definition of KSD (Definition 4). The second claim then follows from Cauchy-Schwarz and the inequality  $\min(a, b)^2 \leq ab$  for  $a, b \geq 0$ , since
$$
\begin{array}{l} \mathbb {E} \left[ \| Y - Z \| _ {2} \min  \left(2 \kappa_ {1}, \kappa_ {1} \sqrt {d} \| X - Y \| _ {2}\right) \right] \leq M _ {\nu , P} ^ {1 / 2} \mathbb {E} \left[ \min  \left(2 \kappa_ {1}, \kappa_ {1} \sqrt {d} \| X - Y \| _ {2}\right) ^ {2} \right] ^ {1 / 2} \\ \leq \sqrt {2 M _ {\nu , P}} \kappa_ {1} d ^ {1 / 4} \mathbb {E} [ \| X - Y \| _ {2} ] ^ {1 / 2} = \sqrt {2 M _ {\nu , P} W _ {1} (\mu , \nu)} \kappa_ {1} d ^ {1 / 4}. \\ \end{array}
$$
9 Conclusions and Limitations
In summary, we have proved the first unified convergence bound and rate for finite-particle SVGD. In particular, our results show that with a suitably chosen step size sequence, SVGD with  $n$ -particles drives the KSD to zero at an order  $1 / \sqrt{\log\log(n)}$  rate. The assumptions we have made on the target and kernel are mild and strictly weaker than those used in prior work to establish KSD weak convergence control [9, 3, 12, 1]. However, we suspect that, with additional effort, the Lipschitz score assumption (Assumption 1) can be relaxed to accommodate pseudo-Lipschitz scores as in Erdogdu et al. [7] or weakly-smooth scores as in Sun et al. [24]. A second limitation of this work is that the obtained rate of convergence is quite slow. However, we hope that this initial recipe for explicit, non-asymptotic convergence will serve as both a template and a catalyst for the field to develop refined upper and lower bounds for SVGD error. To this end, we leave the reader with several open challenges. First, can one establish a non-trivial minimax lower bound for the convergence of SVGD? Second, can one identify which types of target distributions lead to worst-case convergence behavior for SVGD? Finally, can one identify commonly met assumptions on the target distribution and kernel under which the guaranteed convergence rate of SVGD can be significantly improved? Promising follow-up work has already begun investigating speed-ups obtainable by focusing on the convergence of a finite set of moments [20] or by modifying the SVGD algorithm [5].
[1] Alessandro Barp, Carl-Johann Simon-Gabriel, Mark Girolami, and Lester Mackey. Targeted separation and convergence with kernel discrepancies. arXiv preprint arXiv:2209.12835, 2022. 1, 2, 9
[2] Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011. 2
[3] Wilson Ye Chen, Lester Mackey, Jackson Gorham, François-Xavier Briol, and Chris Oates. Stein points. In International Conference on Machine Learning, pages 844-853, 2018. 1, 2, 9
[25] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009. 3
[26] Dilin Wang and Qiang Liu. Learning to draw samples: With application to amortized MLE for generative adversarial learning. arXiv preprint arXiv:1611.01722, 2016. 1
[27] Dilin Wang, Zhe Zeng, and Qiang Liu. Stein variational message passing for continuous graphical models. In International Conference on Machine Learning, pages 5219-5227, 2018. 1
[28] Jingwei Zhuo, Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang. Message passing Stein variational gradient descent. In International Conference on Machine Learning, pages 6018-6027, 2018. 1
A Kernel Assumptions
To show that Assumptions 2 and 3 are met by the most commonly used SVGD kernels with constants independent of dimension, we begin by bounding the derivatives of any radial kernel of the form  $k(x,y) = \phi (\| x - y\| _2^2 /2)$  with  $\phi :\mathbb{R}\to \mathbb{R}$  four times differentiable. By the reproducing property and Cauchy-Schwarz we have
$$
| k (x, y) | = | \langle k (x, \cdot), k (y, \cdot) \rangle_ {\mathcal {H}} | \leq \| k (x, \cdot) \| _ {\mathcal {H}} \| k (y, \cdot) \| _ {\mathcal {H}} = \sqrt {k (x , x)} \sqrt {k (y , y)} = \phi (0),
$$
$$
\left\| \nabla_ {x} k (x, y) \right\| _ {2} = \left| \phi^ {\prime} \left(\left\| x - y \right\| _ {2} ^ {2} / 2\right) \right| \| x - y \| _ {2},
$$
$$
\begin{array}{l} \| \nabla_ {y} \nabla_ {x} k (x, y) \| _ {\mathrm {o p}} = \left\| - \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (x - y) (x - y) ^ {\top} - \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) I \right\| _ {\mathrm {o p}} \\ \leq | \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) | \| x - y \| _ {2} ^ {2} + | \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) |, \quad a n d \\ \left\| \nabla_ {x} ^ {2} k (x, y) \right\| _ {\mathrm {o p}} = \left\| \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (x - y) (x - y) ^ {\top} + \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) I \right\| _ {\mathrm {o p}} \\ \leq | \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) | \| x - y \| _ {2} ^ {2} + | \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) |. \\ \end{array}
$$
Similarly, the partial derivatives take the form
$$
\nabla_ {x _ {j}} k (x, y) = \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) (x _ {j} - y _ {j})
$$
$$
\nabla_ {y _ {j}} \nabla_ {x _ {j}} k (x, y) = - \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) - \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (y _ {j} - x _ {j}) ^ {2}
$$
$$
\begin{array}{l} \nabla_ {x _ {i}} \nabla_ {y _ {j}} \nabla_ {x _ {j}} k (x, y) = - \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (x _ {i} - y _ {i}) - \phi^ {\prime \prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (y _ {j} - x _ {j}) ^ {2} (x _ {i} - y _ {i}) \\ - \mathbb {I} [ i = j ] 2 \phi^ {\prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (x _ {i} - y _ {i}) \\ = - \phi^ {\prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(x _ {i} - y _ {i}\right) \\ - \mathbb {I} [ i \neq j ] \phi^ {\prime \prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (y _ {j} - x _ {j}) ^ {2} (x _ {i} - y _ {i}) \\ + \mathbb {I} [ i = j ] \left(\phi^ {\prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(y _ {i} - x _ {i}\right) ^ {3} - 2 \phi^ {\prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(x _ {i} - y _ {i}\right)\right) \\ \end{array}
$$