SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[44] Jiaxin Shi and Lester Mackey. A finite-particle convergence rate for stein variational gradient descent. CoRR, abs/2211.09721, 2022. doi: 10.48550/arXiv.2211.09721. URL https://doi.org/10.48550/arXiv.2211.09721.
[45] Lukang Sun, Avetik Karagulyan, and Peter Richtarik. Convergence of stein variational gradient descent under a weaker smoothness condition. In International Conference on Artificial Intelligence and Statistics, pages 3693-3717. PMLR, 2023.
[46] Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Uniform-in-time propagation of chaos for the mean-field gradient Langevin dynamics. In ICLR, 2023.
[47] Alain-Sol Sznitman. Topics in propagation of chaos. In Donald L. Burkholder, Etienne Pardoux, Alain-Sol Sznitman, and Paul-Louis Hennequin, editors, *Ecole d'Eté de Probabilités de Saint-Flour XIX* — 1989, Lecture Notes in Mathematics, pages 165–251, Berlin, Heidelberg, 1991. Springer. ISBN 978-3-540-46319-1. doi: 10.1007/BFb0085169.
[48] Juan Luis Vázquez. The porous medium equation: mathematical theory. Oxford University Press on Demand, 2007.
[49] Cédric Villani. Topics in optimal transportation. Number 58. American Mathematical Soc., 2003.
[50] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
[51] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635-3673. PMLR, 2020.
[52] Rentian Yao, Xiaohui Chen, and Yun Yang. Mean-field nonparametric estimation of interacting particle systems. In Conference on Learning Theory, pages 2242-2275. PMLR, 2022.
[53] Jianyi Zhang, Ruiyi Zhang, Lawrence Carin, and Changyou Chen. Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 1877–1887. PMLR, June 2020.
[54] Kelvin Shuangjian Zhang, Gabriel Peyré, Jalal Fadili, and Marcelo Pereyra. Wasserstein Control of Mirror Langevin Monte Carlo. arXiv:2002.04363 [cs, math, stat], February 2020.
A Properties of Mean-field Transfer to Particle System
Lemma A.1. The Assumptions 1-3 transfer seamlessly to the aggregated notions of drift and diffusion given in (7):
1. If  $b(\cdot, \cdot)$  and  $\sigma(\cdot, \cdot)$  satisfy Assumption 1, then  $\pmb{b}(\cdot)$  and  $\sigma(\cdot)$  are  $L(\sqrt{N} + 1)$ -Lipschitz.
2. If  $b(\cdot, \cdot)$  satisfies Assumption 2, then  $\pmb{b}(\cdot)$  satisfies the same condition with the constant  $\sqrt{N} C_v$ .
3. If  $\sigma(\cdot, \cdot)$  satisfies Assumption 3, then the same holds for  $\sigma(\cdot)$  with constant  $\sqrt{N}K$ .
Proof. We prove these statements separately:
1. Let  $x, y \in (\mathbb{R}^d)^{\otimes N}$  and define  $a_i = |x^i - y^i|$ . First, notice that  $\mathcal{W}_2^2(\mu_x, \mu_y) \leq \frac{1}{N} \sum a_i^2$ , as the average on the right-hand-side corresponds to the specific coupling of  $x_i \leftrightarrow y_i$ . Now, observe that
$$
\begin{array}{l} \left| \boldsymbol {b} (x) - \boldsymbol {b} (y) \right| ^ {2} = \sum_ {i} \left| b \left(x ^ {i}, \mu_ {x}\right) - b \left(y ^ {i}, \mu_ {y}\right) \right| ^ {2} \\ \leq L ^ {2} \sum_ {i} \left(a _ {i} + \mathcal {W} _ {2} \left(\mu_ {x}, \mu_ {y}\right)\right) ^ {2} \\ \leq L ^ {2} \sum_ {i} \left(a _ {i} + \sqrt {\frac {1}{N} \sum_ {j} a _ {j} ^ {2}}\right) ^ {2}. \\ \end{array}
$$
Let  $\pmb{a} = (a_{1},\dots ,a_{N})$ , and notice that the last quantity above is equal to
$$
L ^ {2} \left| \boldsymbol {a} + \sqrt {\frac {1}{N}} | \boldsymbol {a} | \cdot \mathbf {1} \right| ^ {2} = L ^ {2} | \boldsymbol {a} | ^ {2} \left| \frac {\boldsymbol {a}}{| \boldsymbol {a} |} + \sqrt {\frac {1}{N}} \cdot \mathbf {1} \right| ^ {2} \leq L ^ {2} | \boldsymbol {a} | ^ {2} N \left(1 + \sqrt {\frac {1}{N}}\right) ^ {2}.
$$
$$
\left| \boldsymbol {b} (x) - \boldsymbol {b} (y) \right| \leq L (\sqrt {N} + 1) | x - y |.
$$
For the diffusion, it suffices to notice that
$$
\left\| \sigma (x) - \sigma (y) \right\| _ {\mathrm {F}} ^ {2} = \sum_ {i} \left\| \sigma \left(x ^ {i}, \mu_ {x}\right) - \sigma \left(y ^ {i}, \mu_ {y}\right) \right\| _ {\mathrm {F}} ^ {2}.
$$
The rest of the proof is similar to the one for the drift.
2. We have
$$
\frac {1}{N} \langle x, \boldsymbol {b} (x) \rangle = \frac {1}{N} \sum_ {i = 1} ^ {N} \langle x ^ {i}, b (x ^ {i}, \mu_ {x}) \rangle \leq C _ {v} (\frac {1}{N} \sum | x ^ {i} | + 1) \leq C _ {v} (\frac {1}{\sqrt {N}} \sqrt {\sum | x ^ {i} | ^ {2}} + 1),
$$
where in the last inequality, we used Cauchy-Schwarz. This implies  $\langle x, \pmb{b}(x) \rangle \leq C_v \sqrt{N} (|x| + 1)$ .
3. It is easy to see that
$$
\left\| \boldsymbol {\sigma} (x) \right\| _ {F} ^ {2} = \operatorname {t r} (\boldsymbol {\sigma} (x) ^ {\top} \boldsymbol {\sigma} (x)) = \sum_ {i = 1} ^ {N} \operatorname {t r} (\sigma (x ^ {i}, \mu_ {x}) ^ {\top} \sigma (x ^ {i}, \mu_ {x})) \leq N K ^ {2}.
$$
Lemma A.2. If  $b(\cdot, \cdot)$  is  $(\alpha, \beta)$ -dissipative on average, then  $\pmb{b}(\cdot)$  is  $(\alpha, N\beta)$ -dissipative in the usual sense, that is, for all  $x \in (\mathbb{R}^d)^{\otimes N}$ ,  $\langle x, \pmb{b}(x) \rangle \leq -\alpha |x|^2 + N\beta$ .
Proof. Observe that for  $x \in (\mathbb{R}^d)^{\otimes N}$  we have
$$
\begin{array}{l} \frac {1}{N} \langle x, \boldsymbol {b} (x) \rangle = \frac {1}{N} \sum_ {i = 1} ^ {N} \langle x ^ {i}, b (x ^ {i}, \mu_ {x}) \rangle = \mathbb {E} _ {\mu_ {x}} [ \langle y, b (y, \mu_ {x}) \rangle ] \leq - \alpha \mathbb {E} _ {\mu_ {x}} | y | ^ {2} + \beta \\ = - \alpha \frac {1}{N} \sum_ {i = 1} ^ {N} | x ^ {i} | ^ {2} + \beta = - \alpha \frac {1}{N} | x | ^ {2} + \beta . \\ \end{array}
$$
This means that  $\langle x, \pmb{b}(x) \rangle \leq -\alpha |x|^2 + N\beta$ .
B The Main Theorem
Recall the Picard process (Picard):
$$
\Pi_ {s} ^ {(t)} = X _ {t} + \int_ {0} ^ {s} \boldsymbol {b} (X _ {t + u}) d u + \int_ {0} ^ {s} \boldsymbol {\sigma} (X _ {t + u}) d W _ {u} ^ {(t)}.
$$
We break down the proof into four steps: first, we prove that the Picard process is close to the flow (Flow), and then we bound the distance between the Picard process and the interpolation (Int). We then conclude the proof of the WAPT property. Lastly, we prove that stability is implied by dissipativity.
$\S$  Distance of Picard from Flow.
$$
\begin{array}{l} \mathbb {E} \left| \Pi_ {s} ^ {(t)} - \Phi_ {s} ^ {(t)} \right| ^ {2} \leq 2 \mathbb {E} \left| \int_ {0} ^ {s} \boldsymbol {b} \left(X _ {t + u}\right) - \boldsymbol {b} \left(\Phi_ {u} ^ {(t)}\right) d u \right| ^ {2} + 2 \mathbb {E} \left| \int_ {0} ^ {s} \boldsymbol {\sigma} \left(X _ {t + u}\right) - \boldsymbol {\sigma} \left(\Phi_ {u} ^ {(t)}\right) d W _ {u} ^ {(t)} \right| ^ {2} \\ \leq T \int_ {0} ^ {s} \mathbb {E} | \boldsymbol {b} \left(\Phi_ {u} ^ {(t)}\right) - \boldsymbol {b} \left(X _ {t + u}\right) | ^ {2} d u + 2 \mathbb {E} \int_ {0} ^ {s} \| \sigma \left(X _ {t + u}\right) - \sigma \left(\Phi_ {u} ^ {(t)}\right) \| _ {\mathrm {F}} ^ {2} d u \\ \leq 2 (T + 1) L ^ {2} \int_ {0} ^ {s} \mathbb {E} \left| \Phi_ {u} ^ {(t)} - X _ {t + u} \right| ^ {2} \\ \end{array}
$$
where we used Lipschitzness of  $\pmb{b}$  and  $\sigma$  (implied by Assumption 1 and Lemma A.1), Itô's isometry (see, e.g., [54, Lemma 3.4]), and Lemma A.1.
§ Distance of Picard to Interpolation. We place a bar above a symbol to denotes its piecewise constant interpolation.
$$
\begin{array}{l} \mathbb {E} \left| \Pi_ {s} ^ {(t)} - X _ {t + s} \right| ^ {2} = \mathbb {E} \left| \int_ {t} ^ {t + s} \boldsymbol {b} (X _ {u}) - \boldsymbol {b} (\overline {{X _ {u}}}) d u + \int_ {t} ^ {t + s} \boldsymbol {\sigma} (X _ {u}) - \boldsymbol {\sigma} (\overline {{X _ {u}}}) d W _ {u} ^ {(t)} + \Delta_ {P} (t, s) \right| ^ {2} \\ \leq 3 (T + 1) L ^ {2} \int_ {t} ^ {t + s} \mathbb {E} | X _ {u} - \overline {{X _ {u}}} | ^ {2} d u + 3 \mathbb {E} \big | \Delta_ {P} (t, s) \big | ^ {2}, \\ \end{array}
$$
where  $\Delta_P(t,s)$  is the accumulated noise and bias from time  $t$  to time  $t + s$ , which is equal to
$$
\Delta_ {P} (t, s) := \sum_ {i = k} ^ {n - 1} \gamma_ {i + 1} P _ {i + 1} + (t + s - \tau_ {n}) \mathbb {E} \left[ P _ {n + 1} \mid \mathcal {F} _ {t + s} \right] - (t - \tau_ {k}) \mathbb {E} \left[ P _ {k + 1} \mid \mathcal {F} _ {t} \right], \tag {B.1}
$$
with  $n = m(t + s)$  and  $k = m(t)$ . It is shown in [23] that  $\lim_{t\to \infty}\mathbb{E}\big|\Delta_P(t,s)\big|^2 = 0$ , a.s.
Continuing to bound the inside of the integral, we have
$$
\mathbb {E} \left| X _ {t} - x _ {k} \right| ^ {2} \leq 3 (t - \tau_ {k}) ^ {2} (\mathbb {E} | \boldsymbol {b} (x _ {k}) | ^ {2} + \mathbb {E} | P _ {k + 1} | ^ {2}) + 3 (t - \tau_ {k}) \mathbb {E} \operatorname {t r} \left(\boldsymbol {\sigma} \left(x _ {k}\right) ^ {\top} \boldsymbol {\sigma} \left(x _ {k}\right)\right)
$$
where we used the fact that conditional expectation is a contraction in  $L^2$ , and
$$
\begin{array}{l} \mathbb {E} \left| \boldsymbol {\sigma} \left(x _ {k}\right) \xi_ {k + 1} \right| ^ {2} = \mathbb {E} \operatorname {t r} \left(\xi_ {k + 1} ^ {\top} \boldsymbol {\sigma} \left(x _ {k}\right) ^ {\top} \boldsymbol {\sigma} \left(x _ {k}\right) \xi_ {k + 1}\right) \\ = \mathbb {E} \operatorname {t r} \left(\boldsymbol {\sigma} \left(\boldsymbol {x} _ {k}\right) ^ {\top} \boldsymbol {\sigma} \left(\boldsymbol {x} _ {k}\right) \xi_ {k + 1} \xi_ {k + 1} ^ {\top}\right) \\ = \mathbb {E} \operatorname {t r} \left(\boldsymbol {\sigma} \left(x _ {k}\right) ^ {\top} \boldsymbol {\sigma} \left(x _ {k}\right) \mathbb {E} \left[ \xi_ {k + 1} \xi_ {k + 1} ^ {\top} \mid \mathcal {F} _ {\tau_ {k}} \right]\right) \\ = \mathbb {E} \operatorname {t r} \left(\boldsymbol {\sigma} \left(x _ {k}\right) ^ {\top} \boldsymbol {\sigma} \left(x _ {k}\right)\right). \\ \end{array}
$$
Moreover, by Assumption 3 we have  $\mathbb{E}\operatorname {tr}(\pmb {\sigma}(x_k)^\top \pmb {\sigma}(x_k)) = \mathcal{O}(1)$ . We thus get by Lemma B.1
$$
\mathbb {E} \left| X _ {t} - x _ {k} \right| ^ {2} \leq 3 C \gamma_ {k + 1} ^ {2} (1 / \gamma_ {k + 1} + 1) + 3 C \gamma_ {k + 1} = \mathcal {O} (\gamma_ {k + 1}).
$$
$$
\sup  _ {s \in [ 0, T ]} \mathbb {E} | \Pi_ {s} ^ {(t)} - X _ {t + s} | ^ {2} \leq C T ^ {2} L ^ {2} \sup  _ {t \leq u \leq t + T} \overline {{\gamma_ {u}}} + 3 \mathbb {E} \left| \Delta_ {P} (t, T) \right| ^ {2} =: A _ {t},
$$
with  $A_{t}\to 0$  as  $t\to \infty$  , a.s.
$\S$  Concluding the proof of APT. By Gronwall inequality,
$$
\mathbb {E} | X _ {t + s} - \Phi_ {s} ^ {(t)} | ^ {2} \leq C \int_ {0} ^ {s} \mathbb {E} | X _ {t + u} - \Phi_ {u} ^ {(t)} | ^ {2} + A _ {t} \leq A _ {t} \exp (s C) \leq A _ {t} \exp (T C) \to 0
$$
as  $t\to \infty$  . Since
$$
\mathcal {W} _ {2} ^ {2} \left(\operatorname {l a w} \left(X _ {t + s}\right), \operatorname {l a w} \left(\Phi_ {s} ^ {(t)}\right)\right) \leq \mathbb {E} \left| X _ {t + s} - \Phi_ {s} ^ {(t)} \right| ^ {2},
$$
we get the desired result.
$\S$  Stability. Lemma A.2 implies that under dissipativity on average, the iterates are stable, and as in [23, Theorem 3], we get the desired convergence result.
B.2 On Propagation of Chaos
Theorem 1 shows that the law  $\mu_{k}$  of  $x_{k}\in (\mathbb{R}^{d})^{\otimes N}$  converges in the Wasserstein space to the limit-set (or the internally chain-transitive (ICT) set)  $S\subset \mathcal{P}_2((\mathbb{R}^d)^{\otimes N})$  of the corresponding flow:
$$
\liminf_{k\to \infty}\inf_{\mu \in S}\mathcal{W}_{2}(\mu_{k},\mu) = 0.
$$
By looking only at the first particle of  $x_{k}$ , namely,  $x_{k}^{1}$ , and given that the dynamics is exchangeable, it follows that
$$
\begin{array}{l} \mathcal {W} _ {2} ^ {2} (\mu_ {k}, \mu) = \inf  _ {\pi} \int | x - y | ^ {2} \pi (d x, d y) \\ = \inf  _ {\pi} \left(\int | x ^ {1} - y ^ {1} | ^ {2} \pi^ {1} \left(d x ^ {1}, d y ^ {1}\right) + \dots + \int | x ^ {N} - y ^ {N} | ^ {2} \pi^ {N} \left(d x ^ {N}, d y ^ {N}\right)\right) \\ \geq \inf  _ {\pi^ {1}} \int | x ^ {1} - y ^ {1} | ^ {2} \pi^ {1} (d x ^ {1}, d y ^ {1}) \\ = \mathcal {W} _ {2} ^ {2} \left(\operatorname {l a w} \left(x _ {k} ^ {1}\right), \operatorname {m a r g i n a l} _ {1} (\mu)\right), \\ \end{array}
$$
where  $\pi^1(dx^1, dy^1) = \int \pi(x, y) dx^2 dy^2 \cdots dx^N dy^N$ , and we used the exchangability in deducing that the law of  $y^i$  are the same as the first marginal of  $\mu$ , for all  $i = 1, \ldots, N$ . Hence, as the limit-set  $S'$  of the first component of the SDE  $X_t^1$  is a subset of the marginal of  $S$ ,
$$
\lim_{k\to \infty}\inf_{\nu \in S^{\prime}}\mathcal{W}_{2}(\operatorname {law}(x_{k}^{1}),\nu)\leq \lim_{k\to \infty}\inf_{\nu \in \operatorname{marginal}_{1}(S)}\mathcal{W}_{2}(\operatorname {law}(x_{k}^{1}),\nu) = 0.
$$
This means that the first particle converges in law to the ICT sets of the corresponding SDE. Assuming a uniform propagation of chaos, we also know that the law of  $X_{t}^{1}$  has a distance of  $O(1 / N)$  from the mean-field equation, and hence, we get that the law of the particles following the discrete algorithm have controllable distance from the mean-field dynamics.
B.3 Supporting Lemmas
Lemma B.1. Suppose Assumptions 1-5 hold. One has  $\mathbb{E}|\pmb {b}(x_k)|^2 = \mathcal{O}(1 / \gamma_{k + 1}),\mathbb{E}|\varepsilon_{k + 1}|^2 = \mathcal{O}(\gamma_{k + 1})$  and  $\mathbb{E}|P_{k + 1}|^2 = \mathcal{O}(1)$
Proof. We repeatedly use the fact that  $\mathbb{E}|\pmb {b}(x_k)|^2\leq 2L^2\mathbb{E}|x_k|^2 +\mathbb{E}|\pmb {b}(x_0)|^2 =:2L^2\mathbb{E}|x_k|^2 +C_0$ . By Assumption 4,  $\mathbb{E}|\varepsilon_{k + 1}|^2\leq \mathcal{O}(\gamma_{k + 1}^2)a_k + \mathcal{O}(\gamma_{k + 1})$ , and we have
$$
\mathbb {E} \left| P _ {k + 1} \right| ^ {2} \leq 2 \mathbb {E} \left| \varepsilon_ {k + 1} \right| ^ {2} + 2 \mathbb {E} \left| U _ {k + 1} \right| ^ {2} = \mathcal {O} \left(\gamma_ {k + 1} ^ {2}\right) a _ {k} + \mathcal {O} (1). \tag {B.2}
$$
$$
\sqrt {\mathbb {E} \left| P _ {k + 1} \right| ^ {2}} \leq \mathcal {O} \left(\gamma_ {k + 1}\right) \sqrt {a _ {k}} + \mathcal {O} (1). \tag {B.3}
$$
Assumption 3 also implies that  $\mathbb{E}|\pmb {\sigma}(x_k)\xi_{k + 1}|^2\leq C_\sigma$
Define  $a_{k}\coloneqq \mathbb{E}|x_{k}|^{2}$  . Then,
$$
\begin{array}{l} a _ {k + 1} - a _ {k} = \gamma_ {k + 1} ^ {2} \mathbb {E} | \boldsymbol {b} (x _ {k}) + P _ {k + 1} | ^ {2} + \gamma_ {k + 1} \mathbb {E} | \sigma (x _ {k}) \xi_ {k + 1} | ^ {2} + 2 \gamma_ {k + 1} \mathbb {E} \langle x _ {k}, \boldsymbol {b} (x _ {k}) + P _ {k + 1} \rangle \\ + 2 \gamma_ {k + 1} ^ {1 / 2} \mathbb {E} \langle x _ {k}, \sigma (x _ {k}) \xi_ {k + 1} \rangle + 2 \gamma_ {k + 1} ^ {3 / 2} \mathbb {E} \langle \boldsymbol {b} (x _ {k}) + P _ {k + 1}, \sigma (x _ {k}) \xi_ {k + 1} \rangle \\ \leq 2 L ^ {2} \gamma_ {k + 1} ^ {2} a _ {k} + \gamma_ {k + 1} ^ {2} C _ {0} + 2 \gamma_ {k + 1} ^ {2} \mathbb {E} \left| P _ {k + 1} \right| ^ {2} + \gamma_ {k + 1} C _ {\sigma} + 2 \gamma_ {k + 1} C _ {v} (\sqrt {a _ {k}} + 1) \\ + 2 \gamma_ {k + 1} \sqrt {a _ {k}} \sqrt {\mathbb {E} \left| P _ {k + 1} \right| ^ {2}} + 2 \gamma_ {k + 1} ^ {3 / 2} \sqrt {C _ {\sigma}} \sqrt {\mathbb {E} \left| P _ {k + 1} \right| ^ {2}} \tag {B.4} \\ \end{array}
$$
Plugging the bounds from (B.2) and (B.3) into (B.4) gives
$$
\begin{array}{l} a _ {k + 1} - a _ {k} \leq \mathcal {O} \left(\gamma_ {k + 1} ^ {2}\right) a _ {k} + \mathcal {O} \left(\gamma_ {k + 1}\right) \sqrt {a _ {k}} + \mathcal {O} \left(\gamma_ {k + 1}\right) \\ =: P \gamma_ {k + 1} ^ {2} a _ {k} + Q \gamma_ {k + 1} \sqrt {a _ {k}} + R \gamma_ {k + 1}, \\ \end{array}
$$
for some  $P, Q, R > 0$  that do not depend on  $k$ .
We now prove  $a_k \leq M / \gamma_{k+1}$  for some fixed  $M > 0$  via induction. Suppose this is the case for  $k$ . For  $k + 1$  we have
$$
\begin{array}{l} a _ {k + 1} \leq \left(P \gamma_ {k + 1} ^ {2} + 1\right) a _ {k} + Q \gamma_ {k + 1} \sqrt {a _ {k}} + R \gamma_ {k + 1} \\ \leq M \left(P \gamma_ {k + 1} + 1 / \gamma_ {k + 1}\right) + \sqrt {M} Q \sqrt {\gamma_ {k + 1}} + R \gamma_ {k + 1} \\ \stackrel {!} {\leq} M / \gamma_ {k + 2}. \\ \end{array}
$$
The last inequality is equivalent to the fact that the following quadratic equation (in  $\sqrt{M}$ ) has a bounded largest root (and the bound shall not depend on  $k$ ):
$$
M \left(P \gamma_ {k + 1} + 1 / \gamma_ {k + 1} - 1 / \gamma_ {k + 2}\right) + \sqrt {M} Q \sqrt {\gamma_ {k + 1}} + R \gamma_ {k + 1}
$$
Notice that by Assumption 5, the leading coefficient is negative, and the larger root is computed as
$$
\begin{array}{l} \frac {Q \sqrt {\gamma_ {k + 1}} + Q \sqrt {\gamma_ {k + 1}} + \sqrt {4 R (\gamma_ {k + 1} / \gamma_ {k + 2} - P \gamma_ {k + 1} ^ {2} - 1)}}{2 (1 / \gamma_ {k + 2} - P \gamma_ {k + 1} - 1 / \gamma_ {k + 1})} \\ \leq \frac {2 Q \sqrt {\gamma_ {k + 1}} + 2 \sqrt {R} \sqrt {\gamma_ {k + 1} / \gamma_ {k + 2}}}{2 \gamma_ {k + 1} / \gamma_ {k + 2}} \leq 2 Q \sqrt {\gamma_ {k + 2}} + \sqrt {R} \sqrt {\gamma_ {k + 2} / \gamma_ {k + 1}} <   2 Q + \sqrt {R} =: M. \\ \end{array}
$$
The claim for  $\pmb{b}$  follows by Lipschitzness, from which the claim for the bias and perturbation follows directly.
C Proofs of Results for Applications
C.1 Two-Layer Neural Networks and Mean-field Langevin
Proof of Corollary 1. Smoothness of drift: We start by showing Lipschitzness with respect to the measure parameter of the drift. First, observe that  $\nabla_{\theta}W(\theta ,\cdot)$  is Lipschitz:
$$
\begin{array}{l} \left| \nabla_ {\theta} W \left(\theta , \theta^ {\prime}\right) - \nabla_ {\theta} W \left(\theta , \theta^ {\prime \prime}\right) \right| = \left| \mathbb {E} _ {z \sim \mathcal {D}} \left[ \left(\varphi (z, \theta^ {\prime}) - \varphi (z, \theta^ {\prime \prime})\right) \nabla_ {\theta} \varphi (z, \theta) \right] \right| \\ = \left| \mathbb {E} _ {z \sim \mathcal {D}} \left[ \left(\kappa \left(\langle z, \theta^ {\prime} \rangle\right) - \kappa \left(\langle z, \theta^ {\prime \prime} \rangle\right)\right) \kappa^ {\prime} \left(\langle z, \theta \rangle\right) z \right] \right| \\ \leq C \left| \theta^ {\prime} - \theta^ {\prime \prime} \right|, \\ \end{array}
$$
due to the boundedness of  $\kappa^{\prime}$  and  $z$ , and Lipschitzness of  $\kappa$ .
Now, consider  $\mu, \nu \in \mathcal{P}_2(\mathbb{R}^d)$  and let  $\pi$  be the optimal coupling (in  $\mathcal{W}_2$  sense) between them. Then, for a fixed  $\theta \in \mathbb{R}^d$ ,
$$
\begin{array}{l} | b (\theta , \mu) - b (\theta , \nu) | ^ {2} = \left| \int \nabla_ {\theta} W (\theta , p) - \nabla_ {\theta} W (\theta , q) \pi (d p, d q) \right| ^ {2} \\ \leq \int | \nabla_ {\theta} W (\theta , p) - \nabla_ {\theta} W (\theta , q) | ^ {2} \pi (d p, d q) \\ \leq \int C ^ {2} | p - q | ^ {2} \pi (d p, d q) \\ = C ^ {2} W _ {2} ^ {2} (\mu , \nu). \\ \end{array}
$$
Next, we show for a fixed measure  $\mu \in \mathcal{P}_2(\mathbb{R}^d)$ ,  $b(\cdot ,\mu)$  is Lipschitz in the first input.
$$
| b (\theta , \mu) - b (\theta^ {\prime}, \mu) | \leq \left| \int \nabla_ {\theta} W (\theta , p) - \nabla_ {\theta} W (\theta^ {\prime}, p) \mu (d p) \right| + | \nabla V (\theta) - \nabla V (\theta^ {\prime}) |.
$$
Let us treat each term separately. We have
$$
\begin{array}{l} \left| \nabla_ {\theta} W (\theta , p) - \nabla_ {\theta} W \left(\theta^ {\prime}, p\right) \right| = \left| \mathbb {E} _ {z \sim \mathcal {D}} \left[ \varphi (z, p) \left(\nabla_ {\theta} \varphi (z, \theta) - \nabla_ {\theta} \varphi (z, \theta^ {\prime})\right) \right] \right| \\ = \left| \mathbb {E} _ {z \sim \mathcal {D}} \left[ \kappa (\langle p, z \rangle) \left(\kappa^ {\prime} (\langle z, \theta \rangle) - \kappa^ {\prime} (\langle z, \theta^ {\prime} \rangle)\right) z \right] \right| \\ \leq C \mathbb {E} _ {z \sim \mathcal {D}} [ | z | | \theta - \theta^ {\prime} | z ] \\ \leq C | \theta - \theta^ {\prime} |. \\ \end{array}
$$
$$
\left| \nabla V (\theta) - \nabla V \left(\theta^ {\prime}\right) \right| = \left| \mathbb {E} _ {(y, z) \sim \mathcal {D}} \left[ y z \left(\kappa^ {\prime} \left(\langle \theta , z \rangle\right) - \kappa^ {\prime} \left(\langle \theta^ {\prime}, z \rangle\right)\right) \mathcal {D} (d y, d z) \right] \right| \leq C | \theta - \theta^ {\prime} |.
$$
$$
\left| b (\theta , \mu) - b \left(\theta^ {\prime}, \nu\right) \right| \leq \left| b (\theta , \mu) - b (\theta , \nu) \right| + \left| b (\theta , \nu) - b \left(\theta^ {\prime}, \nu\right) \right| \leq L \left(\left| \theta - \theta^ {\prime} \right| + \mathcal {W} _ {2} (\mu , \nu)\right),
$$
showing  $b$  satisfies Assumption 1.
Growth control: First, let us calculate
$$
\begin{array}{l} \int \langle \theta , b (\theta , \mu) \rangle \mu (d \theta) = \iint \varphi (z, \theta^ {\prime}) \langle \theta , \nabla_ {\theta} \varphi (z, \theta) \rangle \mathcal {D} (d z) \mu (d \theta^ {\prime}) \mu (d \theta) \\ - \iint y \left\langle \theta , \nabla_ {\theta} \varphi (z, \theta) \right\rangle \mathcal {D} (d y, d z) \mu (d \theta) \\ - \lambda \int | \theta | ^ {2} \mu (d \theta) \\ = \iint \varphi (z, \theta^ {\prime}) \kappa^ {\prime} (\langle z, \theta \rangle) \langle \theta , z \rangle \mathcal {D} (d z) \mu (d \theta^ {\prime}) \mu (d \theta) \\ - \iint y \kappa^ {\prime} (\langle z, \theta \rangle) \langle \theta , z \rangle \mathcal {D} (d y, d z) \mu (d \theta) \\ - \lambda \int | \theta | ^ {2} \mu (d \theta). \\ \end{array}
$$
As  $\varphi, \kappa'$ , and  $\operatorname{supp}(\mathcal{D})$  are bounded, we can see that
$$
\left| \int \langle \theta , b (\theta , \mu) \rangle \mu (d \theta) \right| \leq C \iint | \theta | | z | \mathcal {D} (d z) \mu (d \theta) + C ^ {\prime} \int | \theta | | z | \mathcal {D} (d z) \mu (d \theta) \leq C \int | \theta | \mu (d \theta),
$$
thus, satisfying Assumption 2.
Dissipativity on average: Here we use the extra assumption that  $|a\kappa'(a)|$  is bounded. We directly bound the terms  $\kappa'(\langle \theta, z \rangle) \langle \theta, z \rangle$  above and obtain
$$
\int \langle \theta , b (\theta , \mu) \rangle \mu (d \theta) \leq - \lambda \int | \theta | ^ {2} \mu (d \theta) + C.
$$
C.2 Stein Variational Gradient Descent
Proof of Corollary 2. While the first term in the drift is standard to work with (see Section 4.4), it is the second term in the drift that makes it difficult to analyze. Specifically, we prove the dissipativity on average only for empirical measures. While this would be enough for our purposes (and Theorem 1 goes through), it is an interesting future direction to see when does dissipativity hold in a more general setup. Moreover, for simplicity, we only consider the case where the kernel  $K$  is of the form  $K(x,y) = h(x - y)$ , for some function  $h$ .
Below, we first prove that  $b$  is dissipative on average, which implies that the law of the iterates will be in a compact subset of  $\mathcal{P}_2(\mathbb{R}^d)$ . Then, we show that  $b$  is smooth on this compact subset.
Dissipativity on average: Due to  $K$  being symmetric,  $\nabla_2 K(x,y) = -\nabla_2 K(y,x)$ . We thus have
$$
\begin{array}{l} \iint \langle x - y, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y) \\ = \iint \langle x, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y) - \iint \langle y, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y) \\ = \iint \langle x, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y) + \iint \langle y, \nabla_ {2} K (y, x) \rangle \mu (d x) \mu (d y) \\ = 2 \iint \langle x, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y). \\ \end{array}
$$
$$
\iint \langle x, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y) = \frac {1}{2} \iint \langle x - y, \nabla_ {2} K (x, y) \rangle \mu (d x) \mu (d y) \leq \eta
$$
by Cauchy-Schwarz and the assumption that  $|\nabla_2K(x,y)|\leq \eta /|x - y|$ . With similar arguments, and using dissipativity of  $V$ , we have
$$
\begin{array}{l} \iint \langle x, \nabla V (y) \rangle K (x, y) \mu (d x) \mu (d y) \\ = \iint \langle x, \nabla V (x) \rangle K (x, y) \mu (d x) \mu (d y) - \frac {1}{2} \iint \langle x - y, \nabla V (x) - \nabla V (y) \rangle K (x, y) \mu (d x) \mu (d y) \\ \geq \alpha \iint | x | ^ {2} K (x, y) \mu (d x) \mu (d y) - \beta \| K \| _ {\infty} \\ - \frac {1}{2} L \iint | x - y | ^ {2} K (x, y) \mu (d x) \mu (d y) \\ \geq \alpha \iint | x | ^ {2} K (x, y) \mu (d x) \mu (d y) - \beta \| K \| _ {\infty} + \frac {L \eta}{2}. \\ \end{array}
$$