SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \nabla_ {y _ {i}} \nabla_ {x _ {i}} \nabla_ {y _ {j}} \nabla_ {x _ {j}} k (x, y) = \phi^ {\prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(x _ {i} - y _ {i}\right) ^ {2} + \phi^ {\prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \\ + \mathbb {I} [ i \neq j ] \left(\phi^ {\prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(y _ {j} - x _ {j}\right) ^ {2} \left(x _ {i} - y _ {i}\right) ^ {2} \right. \\ + \phi^ {\prime \prime \prime} (\left\| x - y \right\| _ {2} ^ {2} / 2) \left(y _ {j} - x _ {j}\right) ^ {2}) \\ + \mathbb {I} [ i = j ] \left(\phi^ {\prime \prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(y _ {i} - x _ {i}\right) ^ {4} + 5 \phi^ {\prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(y _ {i} - x _ {i}\right) ^ {2} \right. \\ + 2 \phi^ {\prime \prime} (\left\| x - y \right\| _ {2} ^ {2} / 2)) \\ = \phi^ {\prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(\left(x _ {i} - y _ {i}\right) ^ {2} + \left(x _ {j} - y _ {j}\right) ^ {2}\right) + \phi^ {\prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \\ + \phi^ {\prime \prime \prime \prime} (\| x - y \| _ {2} ^ {2} / 2) (y _ {j} - x _ {j}) ^ {2} (x _ {i} - y _ {i}) ^ {2} \\ + \mathbb {I} [ i = j ] \left(4 \phi^ {\prime \prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right) \left(y _ {i} - x _ {i}\right) ^ {2} + 2 \phi^ {\prime \prime} \left(\| x - y \| _ {2} ^ {2} / 2\right)\right) \\ \end{array}
$$
so that both  $|k(x,y)|$  and
$$
\nabla_ {y _ {i}} \nabla_ {x _ {i}} \nabla_ {y _ {j}} \nabla_ {x _ {j}} k (x, y) | _ {y = x} = \phi^ {\prime \prime} (0) + \mathbb {I} [ i = j ] 2 \phi^ {\prime \prime} (0)
$$
are bounded (Assumption 2) by constants independent of dimension.
Gorham and Mackey [9] popularized the use of IMQ kernels for SVGD, by establishing the convergence-determining properties of the associated KSD. The corresponding  $\phi$  satisfies
$$
\phi (t) = (c ^ {2} + 2 t) ^ {\beta} \quad \text {f o r} \quad c > 0 \quad \text {a n d} \quad \beta \in (- 1, 0),
$$
$$
\phi^ {\prime} (t) = 2 \beta (c ^ {2} + 2 t) ^ {\beta - 1}, \quad \text {a n d} \quad \phi^ {\prime \prime} (t) = 4 \beta (\beta - 1) (c ^ {2} + 2 t) ^ {\beta - 2}.
$$
In this case,  $\| \nabla_y\nabla_xk(x,y)\|_{\mathrm{op}}$  and  $\left\| \nabla_x^2 k(x,y)\right\|_{\mathrm{op}}$  are bounded (Assumption 2) by constants independent of dimension as
$$
\begin{array}{l} | \phi^ {\prime} (\| x - y \| _ {2} ^ {2} / 2) | = - 2 \beta (c ^ {2} + \| x - y \| _ {2} ^ {2}) ^ {\beta - 1} \\ \leq - 2 \beta \min  \left(c ^ {2 \beta - 2}, \| x - y \| _ {2} ^ {2 \beta - 2}\right) \leq - 2 \beta c ^ {2 \beta - 2} \quad \text {a n d} \\ \end{array}
$$
$$
\begin{array}{l} | \phi^ {\prime \prime} (\left\| x - y \right\| _ {2} / 2) | \| x - y \| _ {2} ^ {2} = 4 \beta (\beta - 1) (c ^ {2} + \| x - y \| _ {2} ^ {2}) ^ {\beta - 2} \| x - y \| _ {2} ^ {2} \\ \leq 4 \beta (\beta - 1) \left(c ^ {2} + \| x - y \| _ {2} ^ {2}\right) ^ {\beta - 1} \leq 4 \beta (\beta - 1) c ^ {2 \beta - 2}. \\ \end{array}
$$
For  $\| \nabla_x k(x,y)\| _2$  , we consider two cases:
- When  $\| x - y\| _2\geq 1$
$$
\begin{array}{l} \left| \phi^ {\prime} \left(\left\| x - y \right\| _ {2} ^ {2} / 2\right) \right| \| x - y \| _ {2} \leq - 2 \beta \| x - y \| _ {2} ^ {2 \beta - 2} \| x - y \| _ {2} = - 2 \beta \| x - y \| _ {2} ^ {2 \beta - 1} \\ \leq - 2 \beta / \| x - y \| _ {2} \leq - 2 \beta . \\ \end{array}
$$
- When  $\| x - y \|_2 < 1$ ,  $|\phi'(\| x - y \|_2^2 / 2)| \| x - y \|_2 < |\phi'(\| x - y \|_2^2 / 2)| \leq -2\beta c^{2\beta - 2}$ .
Therefore,  $\| \nabla_x k(x,y)\| _2$  is also bounded (Assumption 2) by constants independent of dimension, and Assumption 3 holds with  $\gamma = -2\beta$ .
The original SVGD paper [18] used Gaussian kernels in all experiments, and they remain perhaps the most common choice in the literature. In this case,  $\phi$  satisfies
$$
\phi (t) = e ^ {- 2 \alpha t} \quad \mathrm {f o r} \quad \alpha > 0, \quad \phi^ {\prime} (t) = - 2 \alpha e ^ {- 2 \alpha t} = - 2 \alpha \phi (t), \quad \mathrm {a n d} \quad \phi^ {\prime \prime} (t) = 4 \alpha^ {2} \phi (t).
$$
Using the inequality  $x \leq e^{x - 1}$ , we find that
$$
\left| \phi^ {\prime} \left(\left\| x - y \right\| _ {2} ^ {2} / 2\right) \right| = 2 \alpha e ^ {- \alpha \| x - y \| _ {2} ^ {2}} \leq \min  \left(2 \alpha , 2 / \left(e \| x - y \| _ {2} ^ {2}\right)\right) \quad \text {a n d}
$$
$$
| \phi^ {\prime \prime} (\| x - y \| _ {2} / 2) | \| x - y \| _ {2} ^ {2} = 4 \alpha^ {2} e ^ {- \alpha \| x - y \| _ {2} ^ {2}} \| x - y \| _ {2} ^ {2} \leq 4 \alpha / e
$$
so that  $\| \nabla_y\nabla_xk(x,y)\|_{\mathrm{op}},\left\| \nabla_x^2 k(x,y)\right\|_{\mathrm{op}}$  , and  $\| \nabla_xk(x,y)\| _2$  are bounded (Assumption 2) by constants independent of dimension, and Assumption 3 holds with  $\gamma = 2 / e$
B Proof of Corollary 2: A finite-particle convergence rate for SVGD
We begin by establishing a lower bound on  $b_{t - 1}$ . Let
$$
b _ {t - 1} ^ {(1)} = \psi_ {\bar {B}, \bar {C}} (\bar {w} _ {0, n} \sqrt {\phi (\bar {w} _ {0 , n})}, \bar {A}, \beta_ {1}) \quad \mathrm {a n d} \quad b _ {t - 1} ^ {(2)} = \psi_ {\bar {B}, \bar {C}} (\bar {w} _ {0, n} \phi (\bar {w} _ {0, n}), \bar {A} + 2 \bar {C}, \beta_ {2})
$$
so that  $b_{t-1} = \min(b_{t-1}^{(1)}, b_{t-1}^{(2)})$ . Since  $\beta_1, \beta_2, \phi(\bar{w}_{0,n}) \geq 1$ , we have
$$
\begin{array}{l} \beta_ {1} = \max  (1, \frac {1}{C} \log (\frac {1}{B} (\log \frac {1}{\bar {w} _ {0 , n} \sqrt {\phi (\bar {w} _ {0 , n})}} - \bar {A}))) \\ \leq \max (1, \frac {1}{C} \log (\frac {1}{B} (\log \frac {1}{\bar {w} _ {0 , n} \sqrt {\phi (\bar {w} _ {0 , n})}}))) \\ \leq \max  \left(1, \frac {1}{C} \log \left(\frac {1}{B} \left(\log \frac {1}{\bar {w} _ {0 , n}}\right)\right)\right) \quad \text {a n d} \\ \end{array}
$$
$$
\begin{array}{l} \beta_ {2} = \max  \left(1, \frac {1}{\bar {C}} \log \left(\frac {1}{\bar {B}} \left(\log \frac {1}{\bar {w} _ {0 , n} \phi (\bar {w} _ {0 , n})} - \bar {A} - 2 \bar {C}\right)\right)\right) \\ \leq \max (1, \frac {1}{C} \log (\frac {1}{B} (\log \frac {1}{\bar {w} _ {0 , n} \phi (\bar {w} _ {0 , n})}))) \\ \leq \max  \left(1, \frac {1}{C} \log \left(\frac {1}{B} \left(\log \frac {1}{\bar {w} _ {0 , n}}\right)\right)\right). \\ \end{array}
$$
Hence,  $\phi (\bar{w}_{0,n})\geq 1$  implies that
$$
\begin{array}{l} b _ {t - 1} ^ {(1)} \geq \frac {1}{C} \log \left(\frac {1}{B} \left(\frac {\log \frac {1}{\bar {w} _ {0 , n} \sqrt {\phi (\bar {w} _ {0 , n})}}}{\max \left(1 , \frac {1}{C} \log \left(\frac {1}{B} \left(\log \frac {1}{\bar {w} _ {0 , n}}\right)\right)\right)} - \bar {A}\right)\right) \\ \geq \frac {1}{C} \log \left(\frac {1}{B} \left(\frac {\log \frac {1}{\bar {w} _ {0 , n} \phi (\bar {w} _ {0 , n})}}{\max \left(1 , \frac {1}{C} \log \left(\frac {1}{B} \left(\log \frac {1}{\bar {w} _ {0 , n}}\right)\right)\right)} - \bar {A} - 2 \bar {C}\right)\right) \quad \text {a n d} \tag {20} \\ \end{array}
$$
$$
b _ {t - 1} ^ {(2)} \geq \frac {1}{C} \log \big (\frac {1}{B} \big (\frac {\log \frac {1}{\bar {w} _ {0 , n}} \phi (\bar {w} _ {0 , n})}{\max (1 , \frac {1}{C} \log (\frac {1}{B} (\log \frac {1}{\bar {w} _ {0 , n}})))} - \bar {A} - 2 \bar {C}) \big) \big).
$$
We divide the remainder of our proof into four parts. First we prove each of the two cases in the generic KSD bound (7) in Appendices B.1 and B.2. Next we show in Appendix B.3 that these two cases yield the generic convergence rate (8). Finally, we prove the high probability upper estimate (9) for  $w_{0,n}$  under i.i.d. initialization in Appendix B.4.
B.1 Case  $b_{t-1} = 0$
In this case, the error bound (7) follows directly from Theorem 3.
B.2 Case  $b_{t-1} > 0$
We first state and prove a useful lemma.
Lemma 6. Suppose  $x = f(\beta)$  for a non-increasing function  $f: \mathbb{R} \to \mathbb{R}$  and  $\beta = \max(1, f(1))$ . Then  $x \leq \beta$  and  $x \leq f(x)$ .
Proof. Because  $f$  is non-increasing and  $\beta \geq 1$ ,  $x = f(\beta) \leq f(1) \leq \beta$ . Since  $x \leq \beta$  and  $f$  is non-increasing, we further have  $f(x) \geq f(\beta) = x$  as advertised.
Since  $\psi_{\bar{B},\bar{C}}$  is non-increasing in its third argument, Lemma 6 implies that  $b_{t - 1}^{(1)}\leq \beta_{1}$  and
$$
b _ {t - 1} ^ {(1)} \leq \psi_ {\bar {B}, \bar {C}} (\bar {w} _ {0, n} \sqrt {\phi (\bar {w} _ {0 , n})}, \bar {A}, b _ {t - 1} ^ {(1)}).
$$
Rearranging the terms and noting that
$$
\bar {B} <   \frac {1}{\beta_ {1}} \log \frac {1}{\bar {w} _ {0 , n} \sqrt {\phi (\bar {w} _ {0 , n})}} - \bar {A} \leq \frac {1}{b _ {t - 1} ^ {(1)}} \log \frac {1}{\bar {w} _ {0 , n} \sqrt {\phi (\bar {w} _ {0 , n})}} - \bar {A}
$$
since  $b_{t-1}^{(1)} \geq b_{t-1} > 0$ , we have
$$
\bar {w} _ {0, n} \exp \left(b _ {t - 1} ^ {(1)} (\bar {A} + \bar {B} \exp \left(\bar {C} b _ {t - 1} ^ {(1)}\right)\right) \leq \frac {1}{\sqrt {\phi \left(\bar {w} _ {0 , n}\right)}}. \tag {21}
$$
Similarly, we have  $b_{t - 1}^{(2)} \leq \psi_{\bar{B},\bar{C}}(\bar{w}_{0,n}\log \log \frac{1}{\bar{w}_{0,n}},\bar{A} +2\bar{C},b_{t - 1}^{(2)})$  and
$$
\sqrt {\bar {w} _ {0 , n}} \exp \left(b _ {t - 1} ^ {(2)} \left(2 \bar {C} + \bar {A} + \bar {B} \exp \left(\bar {C} b _ {t - 1} ^ {(2)}\right)\right) / 2\right) \leq \frac {1}{\sqrt {\phi \left(\bar {w} _ {0 , n}\right)}}. \tag {22}
$$
Since  $b_{t-1} = \min(b_{t-1}^{(1)}, b_{t-1}^{(2)})$ , the inequalities (21) and (22) are also satisfied when  $b_t$  is substituted for  $b_{t-1}^{(1)}$  and  $b_{t-1}^{(2)}$ . Since the error term  $a_{t-1}$  (4) is non-decreasing in each of  $(w_{0,n}, A, B, C)$ , we have
$$
a _ {t - 1} \leq \left(\kappa_ {1} L + \kappa_ {2} d + \kappa_ {1} d ^ {1 / 4} L \sqrt {2 M _ {\mu_ {0} ^ {\infty} , P}}\right) / \sqrt {\phi (\bar {w} _ {0 , n})}.
$$
Since  $b_{t-1} = \min(b_{t-1}^{(1)}, b_{t-1}^{(2)})$ , the claim (7) follows from this estimate, the lower bounds (20), and Theorem 3.
B.3 Generic convergence rate
The generic convergence rate (8) holds as, by the lower bounds (20),  $b_{t - 1} = \min (b_{t - 1}^{(1)}, b_{t - 1}^{(2)}) > 0$  whenever
$$
e ^ {- (\bar {B} + \bar {A} + 2 \bar {C})} > \bar {w} _ {0, n} \phi (\bar {w} _ {0, n}) \quad \text {a n d} \quad \bar {B} ^ {(\bar {B} + \bar {A} + 2 \bar {C}) / \bar {C}} \quad > \bar {w} _ {0, n} \phi (\bar {w} _ {0, n}) (\log (1 / \bar {w} _ {0, n})) ^ {(\bar {B} + \bar {A} + 2 \bar {C}) / \bar {C}},
$$
a condition which occurs whenever  $\bar{w}_{0,n}$  is sufficiently small since the right-hand side of each inequality converges to zero as  $\bar{w}_{0,n} \to 0$ .
B.4 Initializing with i.i.d. particles
We begin by restating an expected Wasserstein bound due to Lei [16].
Lemma 7 (Lei [16, Thm. 3.1]). Suppose  $\mu_0^n = \frac{1}{n}\sum_{i=1}^n\delta_{x_i}$  for  $x_i \stackrel{i.i.d.}{\sim} \mu_0^\infty$  with  $M_{\mu_0^\infty} \triangleq \mathbb{E}_{\mu_0^\infty}[\|\cdot\|_2^2] < \infty$ . Then, for a universal constant  $c > 0$ ,
$$
\mathbb {E} \left[ W _ {1} \left(\mu_ {0} ^ {n}, \mu_ {0} ^ {\infty}\right) \right] \leq c M _ {\mu_ {0} ^ {\infty}} \frac {\log (n) ^ {\mathbb {I} [ d = 2 ]}}{n ^ {1 / (2 \vee d)}}.
$$
Together, Lemma 7 and Markov's inequality imply that
$$
W _ {1} (\mu_ {0} ^ {n}, \mu_ {0} ^ {\infty}) \leq \mathbb {E} [ W _ {1} (\mu_ {0} ^ {n}, \mu_ {0} ^ {\infty}) ] / (c \delta) \leq M _ {\mu_ {0} ^ {\infty}} \frac {\log (n) ^ {\mathbb {I} [ d = 2 ]}}{n ^ {1 / (2 \lor d)}} / \delta
$$
with probability at least  $1 - c\delta$ , proving the high probability upper estimate (9).