SlowGuess/NeurIPS_2023_OCR huggingface dataset:
coefficient of  $w^2$  is  $\mathrm{KSD}^2(\mathbb{P}, \mathbb{Q}_{1/2}) / 4$ , and is therefore positive. Then, the polynomial is minimized with respect to  $w$ , at  $w^\star$  defined by
$$
w _ {\lambda} ^ {\star} = \frac {\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - \Delta_ {L , R} + \lambda / 2 (\mathbb {E} [ \log (p (\mathbf {Z} _ {L})) ] - \mathbb {E} [ \log (p (\mathbf {Z} _ {R})) ])}{\mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) + \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) - 2 \Delta_ {L , R}}.
$$
Since  $\mathbb{E}[\log (p(\mathbf{Z}_L))] - \mathbb{E}[\log (p(\mathbf{Z}_R))]\neq 0$  by assumption, we get that  $w_{\lambda}^{\star} = w_{p}$  for lambda defined by
$$
\lambda = 2 \frac {w _ {p} \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {L}) - (1 - w _ {p}) \mathrm {K S D} ^ {2} (\mathbb {P} , \mathbb {Q} _ {R}) + (1 - 2 w _ {p}) \Delta_ {L , R}}{\mathbb {E} [ \log (p (\mathbf {Z} _ {L})) ] - \mathbb {E} [ \log (p (\mathbf {Z} _ {R})) ]}.
$$
E Proof of Theorem 3.4
Theorem 3.4. Let  $k_{p}$  be the Stein kernel associated with the IMQ kernel with  $\ell > 0$ ,  $\beta \in (0,1)$ , and  $c = 1$ . For  $m \geq 2$ , let  $\{\mathbf{x}_i\}_{i=1}^m \subset \mathbb{R}^d$  be a set of points concentrated at  $\mathbf{x}_0$ , a local minimum or saddle point of  $p$ , and of empirical measure  $\mathbb{Q}_m$ . Then, we have  $\mathrm{L - KSD}^2(\mathbb{P}, \mathbb{Q}_m) > \mathbb{E}[\mathrm{L - KSD}^2(\mathbb{P}, \mathbb{P}_m)]$ , if the density at  $\mathbf{x}_0$  satisfies  $p(\mathbf{x}_0) < \Delta^+ p(\mathbf{x}_0) / (\mathbb{E}[||s_p(\mathbf{X})||_2^2] + \mathbb{E}[\Delta^+ \log p(\mathbf{X})])$ .
Proof of Theorem 3.4. Following the same approach as in the proof of Theorem 2.4, we have
$$
\mathbb {E} \left[ \mathrm {L - K S D} ^ {2} \left(\mathbb {P}, \mathbb {P} _ {m}\right) \right] = \frac {2 \beta d}{m \ell^ {2}} + \frac {\mathbb {E} \left[ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} \right]}{m} + \frac {\mathbb {E} \left[ \Delta^ {+} \log p (\mathbf {X}) \right]}{m},
$$
where  $\mathbf{X}\sim \mathbb{P}$ , and, with the empirical distribution  $\mathbb{Q}_m = \frac{1}{m}\sum_{i = 1}^{m}\delta (\mathbf{x}_i)$
$$
\mathrm {L - K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) = \frac {2 \beta d}{m \ell^ {2}} + \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \| s _ {p} (\mathbf {x} _ {i}) \| _ {2} ^ {2} + \frac {1}{m ^ {2}} \sum_ {i \neq j} ^ {m} k _ {p} (\mathbf {x} _ {i}, \mathbf {x} _ {j}) + \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \Delta^ {+} \log (p (\mathbf {x} _ {i})).
$$
As  $\{\mathbf{x}_i\}_{i=1}^m$  are concentrated at a local minimum or saddle point  $\mathbf{x}_0$ , the score is null for all particles, as well as the distances between them, and we get
$$
\mathrm {L} - \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) = \frac {2 \beta d}{m \ell^ {2}} + \frac {(m - 1) 2 \beta d}{m \ell^ {2}} + \frac {\Delta^ {+} \log (p (\mathbf {x} _ {0}))}{m}.
$$
Next, the difference  $m\big(\mathbf{L}\text{-KSD}^2 (\mathbb{P},\mathbb{Q}_m) - \mathbb{E}[\mathbf{L}\text{-KSD}^2 (\mathbb{P},\mathbb{P}_m)]\big)$  writes
$$
\begin{array}{l} m \big (\mathbf {L - K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) - \mathbb {E} [ \mathbf {L - K S D} ^ {2} (\mathbb {P}, \mathbb {P} _ {m}) ] \big) = \Delta^ {+} \log (p (\mathbf {x} _ {0})) + (m - 1) \frac {2 \beta d}{\ell^ {2}} \\ - \left(\mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ] + \mathbb {E} [ \Delta^ {+} \log p (\mathbf {X}) ]\right). \\ \end{array}
$$
$$
\Delta^ {+} \log p (\mathbf {x}) \stackrel {\text {d e f}} {=} \sum_ {j = 1} ^ {d} \left(\frac {\partial^ {2} \log p (\mathbf {x})}{\partial x ^ {(j) 2}}\right) ^ {+}, \tag {7}
$$
$$
\frac {\partial^ {2} \log p (\mathbf {x})}{\partial x ^ {(j) 2}} = \frac {1}{p (\mathbf {x})} \frac {\partial^ {2} p (\mathbf {x})}{\partial x ^ {(j) 2}} - \left(\frac {1}{p (\mathbf {x})} \frac {\partial p (\mathbf {x})}{\partial x ^ {(j)}}\right) ^ {2}.
$$
As  $\mathbf{x}_0$  is a stationary point of  $p$ ,  $\partial p(\mathbf{x}_0) / \partial x^{(j)} = 0$ , and we obtain
$$
\frac {\partial^ {2} \log p (\mathbf {x} _ {0})}{\partial x ^ {(j) 2}} = \frac {1}{p (\mathbf {x} _ {0})} \frac {\partial^ {2} p (\mathbf {x} _ {0})}{\partial x ^ {(j) 2}},
$$
leading to
$$
\Delta^ {+} \log p (\mathbf {x} _ {0}) = \sum_ {j = 1} ^ {d} \frac {1}{p (\mathbf {x} _ {0})} \left(\frac {\partial^ {2} p (\mathbf {x} _ {0})}{\partial x ^ {(j) 2}}\right) ^ {+} = \frac {\Delta^ {+} p (\mathbf {x} _ {0})}{p (\mathbf {x} _ {0})}. \tag {8}
$$
Finally, we get
$$
\begin{array}{l} m \big (\mathrm {L} - \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {Q} _ {m}) - \mathbb {E} [ \mathrm {L} - \mathrm {K S D} ^ {2} (\mathbb {P}, \mathbb {P} _ {m}) ] \big) = \frac {\Delta^ {+} p (\mathbf {x} _ {0})}{p (\mathbf {x} _ {0})} + (m - 1) \frac {2 \beta d}{\ell^ {2}} \\ \left. - \left(\mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ] + \mathbb {E} [ \Delta^ {+} \log p (\mathbf {X}) ]\right), \right. \\ \end{array}
$$
which ensures that  $\mathrm{L - KSD}^2 (\mathbb{P},\mathbb{Q}_m) - \mathbb{E}[\mathrm{L - KSD}^2 (\mathbb{P},\mathbb{P}_m)] > 0$  for  $m\geq 2$ , provided that
$$
p (\mathbf {x} _ {0}) <   \frac {\Delta^ {+} p (\mathbf {x} _ {0})}{\mathbb {E} [ \| s _ {p} (\mathbf {X}) \| _ {2} ^ {2} ] + \mathbb {E} [ \Delta^ {+} \log p (\mathbf {X}) ]}.
$$
F Proof of Theorem 3.6
$$
\sup  _ {i \in \mathbb {N}} \mathbb {E} \Big [ e ^ {\gamma \max  (1, \frac {d \mathbb {P}}{d \mathbb {Q}} (\mathbf {Z} _ {i}) ^ {2}) (\frac {2 \beta d}{\ell^ {2}} + \| s _ {p} (\mathbf {Z} _ {i}) \| _ {2} ^ {2})} \Big ] <   \infty , \sup  _ {i \in \mathbb {N}} \mathbb {E} \Big [ \frac {d \mathbb {P}}{d \mathbb {Q}} (\mathbf {Z} _ {i}) \sqrt {\frac {2 \beta d}{\ell^ {2}} + \| s _ {p} (\mathbf {Z} _ {i}) \| _ {2} ^ {2}} V (\mathbf {Z} _ {i}) \Big ] <   \infty .
$$
Assumption 3.5 is close to the assumption made in Riabiz et al. [2022]. For the last two finite expectations of the assumption, notice that we have just plugged the formula  $k_{p}(\mathbf{x},\mathbf{x}) = 2\beta d / \ell^{2} + ||s_{p}(\mathbf{x})||^{2}$  into the integrability conditions, since this formula is quite straightforward. But in addition to Riabiz et al. [2022], we require two integrability assumptions, one for each regularization term. We give additional insights about these quite complex conditions below. Overall, our assumptions are hardly stronger than those of Riabiz et al. [2022].
The condition  $E[e^{\gamma |\log (p(\mathbf{Z}_i))|}] < \infty$  is satisfied if  $\int f(\mathbf{x})p(\mathbf{x})^{-\gamma}dx < \infty$ , where  $f$  is the distribution of  $\mathbf{Z}_i$  (since  $p(\mathbf{x}) > 1$  only on a compact set and  $p$  is continuous). Therefore, there exists  $\gamma >0$  satisfying such condition, provided that the tails of density  $f$  are not too heavy compared to the tails of  $p$ . This has to be verified for all iterations of the Markov Chain, and happens to be a quite mild assumption.
Regarding the condition involving the Laplacian term, we can use the analysis from Riabiz et al. (2022) (Appendix S2.4). It relies on a Lipschitz condition for the score function  $\nabla \log p$ , to transform the first finite expectation of Assumption 3.5 into a more amenable and practical integrability condition which writes  $E[\kappa \| \mathbf{Z}_i\|_2^2] < \infty$  for some  $\kappa \in (0,\infty)$ . In the same spirit, if we add a Lipschitz condition on  $\Delta^{+}\log p$ , our second additional integrability assumption reduces to a similar amenable condition. Finally, even though Riabiz et al (2022) do not elaborate further on this condition, it can be noticed that it is satisfied if all  $\mathbf{Z}_i$  have subgaussian tails, for example. Besides, one of the main example of distantly dissipative distributions are log-concave functions outside of a compact set. In this case, the Laplacian regularization is null outside of a compact set, since the Laplacian is negative for concave functions. Then, the integrability condition is automatically satisfied for this type of distributions.
Theorem 3.6. Let  $\mathbb{P}$  be a distantly dissipative probability measure, that admits the density  $p\in$ $\mathcal{C}^2 (\mathbb{R}^d)$ ,  $k_{p}$  be the Stein kernel associated with the IMQ kernel wh  $\ell ,c > 0,\beta \in (0,1)$ . Let  $\{\mathbf{Z}_i\}_{i\in \mathbb{N}}\subset \mathbb{R}^d$  be a Markov chain satisfying Assumption 3.5,  $\pi$  be the index sequence of length  $m_{n}$  generated by regularized Stein thinning, and  $\mathbb{Q}_{m_n}$  be the empirical measure of  $\{\mathbf{Z}_{\pi_i}\}_{i = 1}^{m_n}$ . If  $\log (n)^{\alpha} < m_n < n$ , with any  $\alpha >1$ , and  $\lambda_{m_n} = o(\log (m_n) / m_n)$ , then we have almost surely  $\mathbb{Q}_{m_n}\underset {n\to \infty}{\Longrightarrow}\mathbb{P}$ .
Theorem 3.6 extends Theorem 3 from Riabiz et al. [2022] to regularized Stein Thinning, using Lemmas 3-4 and assuming the following convergence rate of the regularization parameter  $\lambda_{m_n} = o(\log (m_n) / m_n)$ . Lemma 3 also extends Theorem 1 from Riabiz et al. [2022][Theorem 1], whereas Lemma 4 is a slight modification of Lemma 5 from Riabiz et al. [2022].
Lemma 3. Let  $\mathbb{P}$  be a probability measure on  $\mathbb{R}^d$  that admits density  $p\in \mathcal{C}^2 (\mathbb{R}^d)$ ,  $k_{p}$  be a reproducing Stein kernel, and  $\{\mathbf{x}_i\}_{i = 1}^n\subset \mathbb{R}^d$  a fixed set of points. If  $\pi$  is an index sequence of length  $m$  produced by regularized Stein thinning, then we have for  $\lambda >0$
$$
\begin{array}{l} \mathrm {K S D} ^ {2} \big (\frac {1}{m} \sum_ {j = 1} ^ {m} \delta \left(\mathbf {x} _ {\pi_ {j}}\right) \big) \leq \mathrm {K S D} ^ {2} \big (\sum_ {i = 1} ^ {n} w _ {i} ^ {\star} \delta \left(\mathbf {x} _ {i}\right) \big) + \frac {1 + \log (m)}{m} \max  _ {i = 1, \dots , n} k _ {p} \left(\mathbf {x} _ {i}, \mathbf {x} _ {i}\right) \\ + \frac {1 + \log (m)}{m} \max  _ {i = 1, \dots , n} \Delta^ {+} \log (p (\mathbf {x} _ {i})) + 2 \lambda \max  _ {i = 1, \dots , n} | \log (p (\mathbf {x} _ {i})) |, \\ \end{array}
$$
where the weights  $w^{\star}$  are defined as
$$
w^{\star}\in \arg \min_{\substack{\sum_{i}w_{i} = 1\\ w_{i}\geq 0}}\mathrm{KSD}^{2}\big(\sum_{i = 1}^{n}w_{i}\delta (\mathbf{x}_{i})\big).
$$
Lemma 4. Let  $f$  be a non-negative function on  $\mathbb{R}^d$ . Consider a sequence of random variables  $(\mathbf{X}_i)_{i\in \mathbb{N}}\subset \mathbb{R}^d$  such that, for some  $\gamma >0$ ,
$$
b \stackrel {d e f} {=} \sup  _ {i \in \mathbb {N}} \mathbb {E} \left[ e ^ {\gamma f \left(\mathbf {X} _ {i}\right)} \right] <   \infty .
$$
If  $\log (n)^{\alpha} < m_{n} < n$ , with any  $\alpha > 1$ , then we have almost surely,
$$
\lim _ {n \to \infty} \frac {\log (m _ {n})}{m _ {n}} \max _ {i = 1, \ldots , n} f (\mathbf {X} _ {i}) = 0.
$$
The proofs of Lemmas 3 and 4 are reported at the end of this section. We first proceed with the proof of Theorem 3.6.
Proof of Theorem 3.6. From Lemma 3, we have
$$
\begin{array}{l} \mathrm {K S D} ^ {2} \big (\frac {1}{m _ {n}} \sum_ {j = 1} ^ {m _ {n}} \delta (\mathbf {Z} _ {\pi_ {j}}) \big) \leq \underbrace {\mathrm {K S D} ^ {2} \big (\sum_ {i = 1} ^ {n} w _ {i} ^ {\star} \delta (\mathbf {Z} _ {i}) \big)} _ {(\star)} + \underbrace {\frac {1 + \log (m _ {n})}{m _ {n}} \max _ {i = 1 , \dots , n} k _ {p} (\mathbf {Z} _ {i} , \mathbf {Z} _ {i})} _ {(\star \star)} \\ + \underbrace {\frac {1 + \log (m _ {n})}{m _ {n}} \max  _ {i = 1 , \ldots , n} \Delta^ {+} \log (p (\mathbf {Z} _ {i}))} _ {(\star \star \star)} + \underbrace {2 \lambda_ {m _ {n}} \max  _ {i = 1 , \ldots , n} | \log (p (\mathbf {Z} _ {i})) |} _ {(\diamond)}. \\ \end{array}
$$
Riabiz et al. [2022][Proof of Theorem 3, p.12] showed that the term  $(\star)$  converges towards 0 almost surely as  $n\to \infty$ . For the remaining terms, from Assumption 3.5, we have
$$
\sup  _ {i \in \mathbb {N}} \mathbb {E} \left[ e ^ {\gamma \left(d / \ell^ {2} + \left\| s _ {p} (\mathbf {Z} _ {i}) \right\| _ {2} ^ {2}\right)} \right] <   \infty , \quad \sup  _ {i \in \mathbb {N}} \mathbb {E} \left[ e ^ {\gamma | \log (p (\mathbf {Z} _ {i})) |} \right] <   \infty ,
$$
$$
\sup  _ {i \in \mathbb {N}} \mathbb {E} \left[ e ^ {\gamma \Delta^ {+} \log (p (\mathbf {Z} _ {i}))} \right] <   \infty .
$$
We can use Lemma 4 with  $f(\mathbf{x}) = k_p(\mathbf{x},\mathbf{x})$  and  $f(\mathbf{x}) = \Delta^{+}\log (\mathbf{x})$  to deduce that  $(\star \star)\to 0$  and  $(\star \star \star)\rightarrow 0$ , respectively. The remaining term  $(\diamondsuit)$  can be rewritten as
$$
2 \lambda_ {m _ {n}} \max _ {i = 1, \ldots , n} | \log (p (\mathbf {Z} _ {i})) | = \frac {2 m _ {n} \lambda_ {m _ {n}}}{\log (m _ {n})} \times \frac {\log (m _ {n})}{m _ {n}} \max _ {i = 1, \ldots , n} | \log (p (\mathbf {Z} _ {i})) |.
$$
Using the assumption that  $\lambda_{m_n} = o(\log(m_n) / m_n)$  and Lemma 4 with  $f(\mathbf{x}) = |\log(p(\mathbf{x}))|$ , we conclude that  $(\diamond) \to 0$ . It follows that  $\mathrm{KSD}^2\left(\frac{1}{m_n}\sum_{j=1}^{m_n}\delta(\mathbf{x}_{\pi_j})\right) \to 0$  almost surely as  $n \to \infty$ . Given that  $p$  is assumed to be distantly dissipative, we apply Theorem 4 from Chen et al. [2019] to obtain that  $\mathbb{Q}_{m_n} \Rightarrow \mathbb{P}$  almost surely, as  $n \to \infty$ .
Proof of Lemma 3. We consider an iteration  $t \in \{2, \ldots, m\}$  of regularized Stein thinning, where  $m \geq 2$  is the final length of the thinned sample. We define  $a_{t} = t^{2}\mathrm{KSD}^{2}(\mathbb{P}, \frac{1}{t}\sum_{j=1}^{t}\delta(\mathbf{x}_{\pi_{j}}))$  and  $f_{t} = \sum_{j=1}^{t}k_{p}(\mathbf{x}_{\pi_{j}}, \cdot)$ . We also denote  $S_{1}^{2} = \max_{i=1,\dots,n}k_{p}(\mathbf{x}_{i}, \mathbf{x}_{i}) + \max_{i=1,\dots,n}\Delta^{+}\log(p(\mathbf{x}_{i}))$ , and  $S_{2} = 2\max_{i=1,\dots,n}|\log(p(\mathbf{x}_{i}))|$ . Using the definition of the squared KSD of an empirical measure, we have
$$
a _ {t} = a _ {t - 1} + k _ {p} (\mathbf {x} _ {\pi_ {t}}, \mathbf {x} _ {\pi_ {t}}) + 2 \sum_ {j = 1} ^ {t - 1} k _ {p} (\mathbf {x} _ {\pi_ {j}}, \mathbf {x} _ {\pi_ {t}}).
$$
Let  $\mathbf{x}_t^\star = \arg \min_{\mathbf{y} \in \{\mathbf{x}_i\}_{i=1}^n} f_{t-1}(\mathbf{y})$ . By definition,  $\mathbf{x}_{\pi_t}$  minimizes the cost function of the regularized Stein thinning algorithm at iteration  $t$ , and we have
$$
\begin{array}{l} k_{p}(\mathbf{x}_{\pi_{t}},\mathbf{x}_{\pi_{t}}) + 2\sum_{j = 1}^{t - 1}k_{p}(\mathbf{x}_{\pi_{j}},\mathbf{x}_{\pi_{t}}) + \Delta^{+}\log (p(\mathbf{x}_{\pi_{t}})) - \lambda t\log (p(\mathbf{x}_{\pi_{t}})) \\ \leq k _ {p} (\mathbf {x} _ {t} ^ {\star}, \mathbf {x} _ {t} ^ {\star}) + 2 \sum_ {j = 1} ^ {t - 1} k _ {p} (\mathbf {x} _ {\pi_ {j}}, \mathbf {x} _ {t} ^ {\star}) + \Delta^ {+} \log (p (\mathbf {x} _ {t} ^ {\star})) - \lambda t \log (p (\mathbf {x} _ {t} ^ {\star})). \\ \end{array}
$$
We combine this last inequality with the first equation to obtain
$$
a_{t}\leq a_{t - 1} + k_{p}(\mathbf{x}_{t}^{\star},\mathbf{x}_{t}^{\star}) + 2\sum_{j = 1}^{t - 1}k_{p}(\mathbf{x}_{\pi_{j}},\mathbf{x}_{t}^{\star}) + \Delta^{+}\log (p(\mathbf{x}_{t}^{\star})) - \Delta^{+}\log (p(\mathbf{x}_{\pi_{t}}))\\ -\lambda t(\log (p(\mathbf{x}_{t}^{\star})) - \log (n(\mathbf{x}_{\pi_{t}}))).
$$
$$
\begin{array}{l}a_{t}\leq a_{t - 1} + k_{p}(\mathbf{x}_{t}^{\star},\mathbf{x}_{t}^{\star}) + 2\sum_{j = 1}^{t - 1}k_{p}(\mathbf{x}_{\pi_{j}},\mathbf{x}_{t}^{\star}) + \Delta^{+}\log (p(\mathbf{x}_{t}^{\star})) - \Delta^{+}\log (p(\mathbf{x}_{\pi_{t}}))\\ \hskip 14.226378pt + \lambda t(|\log (p(\mathbf{x}_{t}^{\star}))| + |\log (p(\mathbf{x}_{\pi_{t}}))|). \end{array}
$$
By definition,  $0 \leq |\log(p(\mathbf{x}_i))| \leq S_2$  and  $k_p(\mathbf{x}_t^\star, \mathbf{x}_t^\star) + \Delta^+ \log(p(\mathbf{x}_t^\star)) - \Delta^+ \log(p(\mathbf{x}_{\pi_t})) \leq S_1^2$ , hence, we have
$$
a_{t}\leq a_{t - 1} + S_{1}^{2} + t\lambda S_{2} + 2\min_{\mathbf{y}\in \{\mathbf{x}_{i}\}_{i = 1}^{n}}f_{t - 1}(\mathbf{y}).
$$
As in the proof of Riabiz et al. [2022], we have  $\min_{\mathbf{y} \in \{\mathbf{x}_i\}_{i=1}^n} f_{t-1}(\mathbf{y}) \leq \sqrt{a_{t-1}} \| h^\star \|_{\mathcal{H}(k_p)}$  where  $h^\star$  is the element in the RKHS  $\mathcal{H}(k_p)$  of the form  $h^\star = \sum_{i=1}^n w_i^\star k_p(\mathbf{x}_i, \cdot)$ . As a result,  $a_t$  is bounded as follows:
$$
a _ {t} \leq a _ {t - 1} + S _ {1} ^ {2} + t \lambda S _ {2} + 2 \sqrt {a _ {t - 1}} \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})}.
$$
We then show by induction that
$$
a _ {t} \leq t ^ {2} (\| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + C _ {t} + \lambda S _ {2}),
$$
$$
C _ {t} \stackrel {d e f} {=} \frac {1}{t} \left(S _ {1} ^ {2} - \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2}\right) \sum_ {j = 1} ^ {t} \frac {1}{j}.
$$
With such a result, we will have  $\mathrm{KSD}^2 (\mathbb{P},\frac{1}{t}\sum_{j = 1}^{t}\delta (\mathbf{x}_{\pi_j}))\leq \mathrm{KSD}^2 (\mathbb{P},\sum_{i = 1}^{n}w_i^\star \delta (\mathbf{x}_{\pi_j})) + C_t+$ $\lambda S_2$  and obtain the advertised result in Theorem 3 for the last iteration  $t = m$
For  $t = 1$ , we have  $a_1 = k_p(\mathbf{x}_{\pi_1}, \mathbf{x}_{\pi_1}) \leq S_1^2$  and thus  $a_1 \leq \|h^\star\|_{\mathcal{H}(k_p)}^2 + C_1 + \lambda S_2$ . For a fixed  $t \geq 2$ , assume that  $a_{t-1} \leq (t-1)^2 (\|h^\star\|_{\mathcal{H}(k_p)}^2 + C_{t-1} + \lambda S_2)$  where  $C_{t-1} = \frac{1}{t-1} (S_1^2 - \|h^\star\|_{\mathcal{H}(k_p)}^2) \sum_{j=1}^{t-1} \frac{1}{j}$ . We then have
$$
\begin{array}{l} a _ {t} \leq a _ {t - 1} + S _ {1} ^ {2} + t \lambda S _ {2} + 2 \sqrt {a _ {t - 1}} \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} \\ \begin{array}{r l} & {\leq (t - 1) ^ {2} (\| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + C _ {t - 1} + \lambda S _ {2}) + S _ {1} ^ {2} + t \lambda S _ {2}} \\ & {\quad \quad \quad + 2 (t - 1) \sqrt {\| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + C _ {t - 1} + \lambda S _ {2}} \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})}} \end{array} \tag {9} \\ = t ^ {2} \left(\left\| h ^ {\star} \right\| _ {\mathcal {H} \left(k _ {p}\right)} ^ {2} + C _ {t} + \lambda S _ {2}\right) + R _ {t} \\ \end{array}
$$
$$
\begin{array}{l} R _ {t} = (t - 1) ^ {2} C _ {t - 1} - t ^ {2} C _ {t} + (1 - 2 t) \left(\| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} ^ {2} + \lambda S _ {2}\right) + S _ {1} ^ {2} \\ + t \lambda S _ {2} + 2 (t - 1) \sqrt {\| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} ^ {2} + C _ {t - 1} + \lambda S _ {2}} \| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} \\ = (t - 1) ^ {2} C _ {t - 1} - t ^ {2} C _ {t} + (1 - 2 t) \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + S _ {1} ^ {2} \\ + \lambda S _ {2} (1 - t) + 2 (t - 1) \sqrt {\| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + C _ {t - 1} + \lambda S _ {2}} \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} \\ \end{array}
$$
Using Riabiz et al. [2022][Lemma 1], we have
$$
2 \| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} \sqrt {\| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} ^ {2} + C _ {t - 1} + \lambda S _ {2}} \leq 2 \| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} ^ {2} + C _ {t - 1} + \lambda S _ {2}
$$
It follows from Equation (9) that we need  $R_{t} \leq 0$ , i.e.,
$$
2 \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + C _ {t - 1} + \lambda S _ {2} \leq \frac {t ^ {2} C _ {t} - (t - 1) ^ {2} C _ {t - 1}}{t - 1} - \frac {S _ {1} ^ {2} - \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2}}{t - 1} + \lambda S _ {2} + 2 \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2}.
$$
The above inequality is always satisfied as long as
$$
2 \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2} + C _ {t - 1} + \leq \frac {t ^ {2} C _ {t} - (t - 1) ^ {2} C _ {t - 1}}{t - 1} - \frac {S _ {1} ^ {2} - \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2}}{t - 1} + 2 \| h ^ {\star} \| _ {\mathcal {H} (k _ {p})} ^ {2},
$$
$$
t C _ {t} - (t - 1) C _ {t - 1} \geq \frac {1}{t} \left(S _ {1} ^ {2} - \| h ^ {\star} \| _ {\mathcal {H} \left(k _ {p}\right)} ^ {2}\right),
$$
and always true by definition of  $C_t$ . Hence we have shown that  $a_t \leq t^2 (\| h^\star \|_{\mathcal{H}(k_p)}^2 + C_t + \lambda S_2)$ . Given that  $\| h^\star \|_{\mathcal{H}(k_p)}^2 = \mathrm{KSD}^2(\mathbb{P}, \sum_{i=1}^{n} w_i^\star \delta(\mathbf{x}_i))$ , we have
$$
\mathrm {K S D} ^ {2} (\mathbb {P}, \frac {1}{t} \sum_ {j = 1} ^ {t} \delta (\mathbf {x} _ {\pi_ {j}})) \leq \mathrm {K S D} ^ {2} (\mathbb {P}, \sum_ {i = 1} ^ {n} w _ {i} ^ {\star} \delta (\mathbf {x} _ {i})) + C _ {t} + \lambda S _ {2},
$$
where  $C_t \leq \frac{1 + \log(t)}{t} \left( \max_{i=1,\dots,n} k_p(\mathbf{x}_i, \mathbf{x}_i) + \max_{i=1,\dots,n} \Delta^+ \log(p(\mathbf{x}_i)) \right)$  (see [Riabiz et al., 2022][Lemma 2]).
Proof of Lemma 4. We follow the proof of Lemma 5 from [Riabiz et al., 2022]. In the last step of the proof, we essentially need to show that
$$
\sum_ {m _ {n} = 1} ^ {\infty} c _ {1} \left(m _ {n} ^ {2}\right) <   \infty \quad \text {a n d} \quad \sum_ {m _ {n} = 1} ^ {\infty} c _ {2} \left(m _ {n}\right) <   \infty
$$
$$
c _ {1} \left(m _ {n} ^ {2}\right) \stackrel {{d e f}} {=} \frac {2 \log (m _ {n})}{m _ {n} ^ {2}} \frac {\log (n b)}{\gamma} \quad \text {a n d} \quad c _ {2} \left(m _ {n}\right) \stackrel {{d e f}} {=} 4 \frac {\log (m _ {n})}{m _ {n} ^ {2}} \frac {\log (n ((m _ {n} + 1) ^ {2}) b)}{\gamma}.
$$
With the assumption that  $\log (n)^{\beta} \leq m_n < n$  with  $\beta > 1$ , it is deduced that  $c_{1}(m_{n}^{2}) \to 0$  and  $c_{2}(m_{n}) \to 0$  as  $n \to \infty$ .
G Laplacian Stein Operator
Instead of the standard Langevin operator, we can use  $\mathcal{T}_p(g) = \Delta (pg) / p$ , mentioned in Oates et al. [2017, Appendix A.2]. However, such Stein operator introduces similar problems as Pathology I, since the Stein kernel associated with  $\mathcal{T}_p^\prime (g)$  also has spurious minimum in regions where second derivatives of  $p$  vanish, as illustrated below. Therefore, it is more appropriate to taylor a specific Laplacian correction as proposed in this paper, which cannot directly be derived from a Stein kernel, since it is not differentiable. In this appendix, we study the operator  $\mathcal{T}_p$  defined as  $\mathcal{T}_p g = \Delta (pg) / p$  and such that [Oates et al., 2017, Appendix A.2]
$$
\mathbb {E} [ (\mathcal {T} _ {p} g) (\mathbf {Z}) ] = 0,
$$
for all  $g$  belonging to  $\mathcal{G}$ , and with  $\mathbf{Z} \sim \mathbb{P}$ . For each dimension  $j \in \{1, \dots, d\}$ , let  $\mathcal{T}_p^j$  be the operator defined as
$$
\left(\mathcal {T} _ {p} ^ {j} g\right) (\mathbf {x}) = \frac {1}{p (\mathbf {x})} \left(\nabla_ {x _ {j}} ^ {2} p (\mathbf {x}) g (\mathbf {x}) + 2 \nabla_ {x _ {j}} p (\mathbf {x}) \nabla_ {x _ {j}} g (\mathbf {x}) + p (\mathbf {x}) \nabla_ {x _ {i}} ^ {2} g (\mathbf {x})\right)
$$
for  $g: \mathbb{R}^d \to \mathbb{R}$ , and where  $x_j$  is the  $j$ -th coordinate of  $\mathbf{x}$  to simplify notations. The operator  $\mathcal{T}_p g$  can then be rewritten as  $(\mathcal{T}_p g)(\mathbf{x}) = \sum_{j=1}^{d} (\mathcal{T}_p^j g)(\mathbf{x})$ . Gorham and Mackey [2017][Proposition 2] establish the closed-form expression of the KSD in the case of the multidimensional Langevin operator. We generalize the proof of Gorham and Mackey [2017][Proposition 2] for the Laplacian operator  $(\mathcal{T}_p g)(\mathbf{x}) = \Delta(p(\mathbf{x})g(\mathbf{x})) / p(\mathbf{x})$  and establish a closed-form expression of the Stein kernel  $k_p: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$ . For a given kernel function  $k: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$ ,  $k \in C^{2,2}$ , the Stein kernel  $k_p$  is given by [Gorham and Mackey, 2017]
$$
k _ {p} (\mathbf {x}, \mathbf {y}) = \sum_ {j = 1} ^ {d} k _ {p} ^ {j} (\mathbf {x}, \mathbf {y}), \tag {10}
$$
$$
k _ {p} ^ {j} (\mathbf {x}, \mathbf {y}) = \left\langle \mathcal {T} _ {p} ^ {j} (k (\mathbf {x}, \cdot)), \mathcal {T} _ {p} ^ {j} (k (\cdot , \mathbf {y})) \right\rangle_ {\mathcal {H} _ {k}}. \tag {11}
$$
After a few developments, it is found that