SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial \eta_ {t}}{\partial t} - 2 \nabla \cdot \left(\nabla \left(\frac {\eta_ {t}}{\nu}\right) \eta_ {t}\right) = 0, \quad \mu_ {t} = (\nabla \phi) _ {\#} \eta_ {t}. \tag {66}
$$
Similar to before, we will begin by studying the convergence properties of this MWGF in continuous time.
Proposition 11. The dissipation of  $\mathrm{KL}(\cdot ||\pi)$  along the MWGF in (66) is given by
$$
\frac {\mathrm {d} \operatorname {K L} \left(\mu_ {t} \| \pi\right)}{\mathrm {d} t} = - 2 \left| \left| [ \nabla^ {2} \phi ] ^ {- 1} \nabla \frac {\mu_ {t}}{\pi} \right| \right| _ {L ^ {2} (\pi)} ^ {2}. \tag {67}
$$
Proof. The proof is essentially identical to the proof of Proposition 2, although the roles of the Lyapunov functional and the Wasserstein gradient are now reversed. In particular, we now have
$$
\frac {\mathrm {d} \mathrm {K L} (\eta_ {t} | | \nu)}{\mathrm {d} t} = \left\langle 2 \nabla \frac {\eta_ {t}}{\nu}, - \nabla \log \left(\frac {\eta_ {t}}{\nu}\right) \right\rangle_ {L ^ {2} (\eta_ {t})} = - 2 \left| \left| \nabla \frac {\eta_ {t}}{\nu} \right| \right| _ {L ^ {2} (\nu)} ^ {2}.
$$
By Theorem 2 in [43] and (41) - (43) in the proof of Proposition 2, we have that  $\mathrm{KL}(\eta_t||\nu) = \mathrm{KL}(\mu_t||\pi)$  and  $\left|\left|\nabla \left(\frac{\eta_t}{\nu}\right)\right|\right|_{L^2 (\nu)}^2 = \left|\left|\left[\nabla^2\phi \right]^{-1}\nabla \left(\frac{\mu_t}{\pi}\right)\right|\right|_{L^2 (\pi)}^2$  from which the conclusion follows.
Proposition 12. The dissipation of  $\chi^2 (\cdot ||\pi)$  along the MWGF in (66) is given by
$$
\frac {\mathrm {d} \chi^ {2} \left(\mu_ {t} \right\| \pi)}{\mathrm {d} t} = - 4 \left| \left| [ \nabla^ {2} \phi ] ^ {- 1} \nabla \left(\frac {\mu_ {t}}{\pi}\right) \right| \right| _ {L ^ {2} \left(\mu_ {t}\right)} ^ {2}. \tag {68}
$$
Proof. Similar to the last result, the proof follows very closely the proof of Proposition 1. On this occasion, we have
$$
\frac {\mathrm {d} \chi^ {2} \left(\eta_ {t} \mid \mid \nu\right)}{\mathrm {d} t} = \left\langle w _ {t}, 2 \nabla \left(\frac {\eta_ {t}}{\nu}\right) \right\rangle_ {L ^ {2} \left(\eta_ {t}\right)} = - 4 \left| \left| \nabla \left(\frac {\eta_ {t}}{\nu}\right) \right| \right| _ {L ^ {2} \left(\eta_ {t}\right)} ^ {2}. \tag {69}
$$
Arguing as in (39) - (40) and (41) - (43) in the proof of Proposition 2, we have that  $\chi^2 (\eta ||\nu) = \chi^2 (\mu ||\pi)$  and  $\left|\left|\nabla \left(\frac{\eta_t}{\nu}\right)\right|\right|_{L^2 (\eta_t)}^2 = \left|\left|\left[\nabla^2\phi \right]^{-1}\nabla \left(\frac{\mu_t}{\pi}\right)\right|\right|_{L^2 (\mu_t)}^2$  from which the conclusion follows.
Similar to before, we can also obtain exponential convergence to the target under a mirrored functional inequality.
Proposition 13. Assume that  $\pi$  satisfies a mirrored PI with constant  $\kappa >0$ . In particular, for all locally Lipschitz  $g\in L^{2}(\pi)$ , there exists  $\kappa >0$  such that
$$
\operatorname {V a r} _ {\pi} [ g ] \leq \frac {1}{\kappa} \left| \left| \left[ \nabla^ {2} \phi \right] ^ {- 1} \nabla g \right| \right| _ {L ^ {2} (\pi)} ^ {2} \tag {70}
$$
Then the KL divergence converges exponentially along the MWGF in (66):
$$
\mathrm {K L} \left(\mu_ {t} | | \pi\right) \leq \mathrm {K L} \left(\mu_ {0} | | \pi\right) e ^ {- 2 \lambda t}. \tag {71}
$$
Furthermore, for  $t \geq \frac{1}{2\lambda}$ , the following convergence rate holds:
$$
\mathrm {K L} \left(\mu_ {t} | | \pi\right) \leq \left(\mathrm {K L} \left(\mu_ {0} | | \pi\right) \wedge 2\right) e ^ {- 2 \lambda t}. \tag {72}
$$
Proof. The proof follows closely the proof of [20, Theorem 1]. In particular, substituting the mirrored PI (70) with  $g = \frac{\mu_t}{\pi} - 1$  into (67), we have that
$$
\frac {\mathrm {d} \operatorname {K L} \left(\mu_ {t} \mid \mid \pi\right)}{\mathrm {d} t} = - \left| \left| [ \nabla^ {2} \phi ] ^ {- 1} \nabla \left(\frac {\mu_ {t}}{\pi}\right) \right| \right| _ {L ^ {2} \left(\mu_ {t}\right)} ^ {2} \leq - 2 \lambda \chi^ {2} \left(\mu_ {t} \mid \mid \pi\right) \leq - 2 \lambda \operatorname {K L} \left(\mu_ {t} \mid \mid \pi\right), \tag {73}
$$
where in the final inequality we have used the fact that  $\mathrm{KL}(\cdot ||\pi)\leq \chi^2 (\cdot ||\pi)$  (e.g., [93, Sec. 2.4]). The first bound now follows straightforwardly from Gronwall's inequality.
For the second bound, we will instead use the inequality  $\mathrm{KL}(\cdot ||\pi)\leq \log \left(1 + \chi^2 (\cdot ||\pi)\right)$  (e.g., [93, Sec. 2.4]), which implies that  $e^{\mathrm{KL}(\cdot ||\pi)} - 1\leq \chi^2 (\cdot ||\pi)$ . Using this in (73), it follows that
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left(1 - e ^ {- \mathrm {K L} \left(\mu_ {t} \mid \mid \pi\right)}\right) \leq - 2 \lambda \left(1 - e ^ {- \mathrm {K L} \left(\mu_ {t} \mid \mid \pi\right)}\right). \tag {74}
$$
Applying Gronwall's lemma, we then have that
$$
\left(1 - e ^ {- \mathrm {K L} \left(\mu_ {t} \mid \mid \pi\right)}\right) \leq e ^ {- 2 \lambda t} \left(1 - e ^ {- \mathrm {K L} \left(\mu_ {t} \mid \mid \pi\right)}\right) \leq e ^ {- 2 \lambda t}, \tag {75}
$$
where in the final line we have used the elementary inequality  $g(x) = 1 - e^{-x} \leq 1$ , for  $x \geq 0$ . Finally, observe that if  $t \geq \frac{1}{2\lambda}$ , then  $\left(1 - e^{-\mathrm{KL}(\mu_t||\pi)}\right) \leq e^{-2\lambda t} \leq e^{-1}$ . Combining this with the fact that  $x \leq 2g(x)$  whenever  $g(x) \leq e^{-1}$ , we have that
$$
\mathrm {K L} \left(\mu_ {t} \mid \mid \pi\right) \leq 2 \left(1 - e ^ {- \mathrm {K L} \left(\mu_ {t} \mid \mid \pi\right)}\right) \leq 2 e ^ {- 2 \lambda t} \tag {76}
$$
for  $t \geq \frac{1}{2\lambda}$ . Alongside the bound in (71), this completes the proof of (72).
B.3.2 Kernelised MWGF of the Chi-Squared Divergence
In [20], the authors provide an interpretation of SVGD as a kernelised WGF of the  $\chi^2$  divergence. Similarly, we can interpret MSVGD as a kernelised MWGF of the  $\chi^2$  divergence; see also [82, App. H]. In particular, observe that [20, Sec. 2.3]
$$
\begin{array}{l} \mathcal {P} _ {\eta_ {t}, k _ {\phi}} \nabla \log \left(\frac {\eta_ {t}}{\nu}\right) = \int k (x, \cdot) \nabla \log \left(\frac {\eta_ {t}}{\nu} (x)\right) \mathrm {d} \eta_ {t} (x) (77) \\ = \int k (x, \cdot) \nabla \left(\frac {\eta_ {t}}{\nu} (x)\right) \mathrm {d} \nu (x) = \mathcal {P} _ {\nu , k _ {\phi}} \nabla \left(\frac {\eta_ {t}}{\nu}\right). (78) \\ \end{array}
$$
It follows that the continuous-time MSVGD dynamics in (5) or (54) can equivalently be written as
$$
\frac {\partial \eta_ {t}}{\partial t} - \nabla \cdot \left(P _ {\nu , k _ {\phi}} \nabla \left(\frac {\eta_ {t}}{\nu}\right) \eta_ {t}\right) = 0, \quad \mu_ {t} = \left(\nabla \phi^ {*}\right) _ {\#} \eta_ {t}. \tag {79}
$$
Comparing (79) and (66), it is clear that, up to a factor of two, MSVGD can be interpreted as the MWGF obtained by replacing the gradient of chi-squared divergence,  $\nabla \left(\frac{\eta_t}{\nu}\right)$ , by  $P_{\nu,k_\phi} \nabla \left(\frac{\eta_t}{\nu}\right)$ . In [82, App. H], the authors obtain continuous-time convergence results for this scheme, under a mirrored Stein PI. Here, we proceed differently, based on the approach in [20].
B.3.3 Mirrored Laplacian Adjusted Wasserstein Gradient Descent (MLAWGD)
Following [20], suppose now that we replace  $P_{\nu ,k_{\phi}}\nabla \left(\frac{\eta_t}{\nu}\right)$  by the vector field  $\nabla P_{\nu ,k_{\phi}}\left(\frac{\eta_t}{\nu}\right)$ . The new dynamics, which we refer to as the mirrored Laplacian adjusted Wasserstein gradient flow (MLAWGF), are then given by
$$
\frac {\partial \eta_ {t}}{\partial t} - \nabla \cdot \left(\nabla P _ {\nu , k _ {\phi}} \left(\frac {\eta_ {t}}{\pi}\right) \eta_ {t}\right) = 0, \quad \mu_ {t} = \left(\nabla \phi^ {*}\right) _ {\#} \eta_ {t}. \tag {80}
$$
The dynamics in (80) can essentially be viewed as a change of measure of the standard LAWGD dynamics in [20, Sec. 4], designed to converge to the dual target  $\nu = (\nabla \phi)_{\#}\pi$
Suppose, in addition, that the kernel  $k_{\phi}$  were chosen such that  $P_{\nu ,k_{\phi}} = -\mathcal{L}_{\nu}^{-1}$ , where  $\mathcal{L}_{\nu ,k_{\phi}}$  denotes the infinitesimal generator of the Langevin SDE with stationary distribution  $\nu$ . Thus, in particular,
$$
\mathcal {L} _ {\nu} f _ {\phi} (y) = - \langle \nabla W (y), \nabla f _ {\phi} (y) \rangle + \Delta f _ {\phi} (y). \tag {81}
$$
where  $f_{\phi}:\mathbb{R}^{d}\to \mathbb{R}$  denotes the function  $f_{\phi}(\cdot) = f(\nabla \phi^{*}(\cdot))$  , given some base function  $f:\mathcal{X}\rightarrow \mathbb{R}$  Letting  $y = \nabla \phi (x)$  , using (20), and then the chain rule, we can rewrite this as
$$
\begin{array}{l} \mathcal {L} _ {v} f (x) = - \left\langle \left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \left[ \nabla V (x) + \nabla \log \det  \nabla^ {2} \phi (x) \right], \left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \nabla f (x) \right\rangle (82) \\ + \operatorname {T r} \left[ - [ \nabla^ {2} \phi (x) ] ^ {- 1} \nabla^ {3} \phi (x) [ \nabla^ {2} \phi (x) ] ^ {- 2} \nabla f (x) + [ \nabla^ {2} \phi (x) ] ^ {- 2} \nabla^ {2} f (x) \right]. (83) \\ \end{array}
$$
We will denote this kernel  $k_{\mathcal{L}_{\nu}}$ . In practice, computing  $k_{\mathcal{L}_{\nu}}$  requires computing the spectral decomposition of the operator  $\mathcal{L}_{\nu}$ , which is challenging even in the moderate dimensions. Nonetheless, following [20, Theorem 4], for this choice of kernel we can show that the MLAWGF converges exponentially fast to the target distribution, at a rate independent of the Poincaré constant.
Proposition 14. Assume that  $\pi$  satisfies a mirrored PI with some constant  $\kappa >0$ . In addition, suppose that  $\mathcal{L}_{\nu}$  has a discrete spectrum. Then the KL divergence converges exponentially along the MLAWGF, with a rate independent of  $\kappa$ :
$$
\mathrm {K L} \left(\mu_ {t} | | \pi\right) \leq \left(\mathrm {K L} \left(\mu_ {0} | | \pi\right) \wedge 2\right) e ^ {- t}. \tag {84}
$$
Proof. Using standard rules for calculus on the Wasserstein space, and the integration by parts formula  $\mathbb{E}_{\nu}\langle \nabla f,\nabla g\rangle = \mathbb{E}_{\nu}[f\mathcal{L}_{\nu}g]$  [e.g., 5], we have
$$
\frac {\mathrm {d} \mathrm {K L} (\eta_ {t} \| \nu)}{\mathrm {d} t} = - \left\langle \nabla \left(\frac {\eta_ {t}}{\nu}\right), \nabla P _ {\nu , k _ {\mathcal {L} _ {\nu}}} \left(\frac {\eta_ {t}}{\nu}\right) \right\rangle_ {L ^ {2} (\pi)} = \int \frac {\eta_ {t}}{\nu} \mathcal {L} _ {\nu} P _ {\nu , k _ {\mathcal {L} _ {\nu}}} \frac {\eta_ {t}}{\nu} \mathrm {d} \nu (x) = - \left| \left| \frac {\eta_ {t}}{\nu} \right| \right| _ {L ^ {2} (\nu)} ^ {2}.
$$
By [43, Theorem 2] and (41) - (43) in the proof of Proposition 2, we have that  $\mathrm{KL}(\eta_t||\nu) = \mathrm{KL}(\mu_t||\pi)$  and  $\left|\left|\nabla \left(\frac{\eta_t}{\nu}\right)\right|\right|_{L^2 (\nu)}^2 = \left|\left|\left[\nabla^2\phi \right]^{-1}\left(\frac{\mu_t}{\pi}\right)\right|_{L^2 (\pi)}^2$  which implies that
$$
\frac {\mathrm {d} \operatorname {K L} \left(\mu_ {t} \mid \mid \pi\right)}{\mathrm {d} t} = - \left| \left| \frac {\mu_ {t}}{\pi} \right| \right| _ {L ^ {2} (\pi)} ^ {2} = - \left| \left| \left(\frac {\mu_ {t}}{\pi} - 1\right) \right| \right| _ {L ^ {2} (\pi)} ^ {2} = - \mathcal {X} ^ {2} \left(\mu_ {t} \mid \mid \pi\right) \leq - \operatorname {K L} \left(\mu_ {t} \mid \mid \pi\right), \tag {85}
$$
where in the final line we have once again used the fact that  $\mathrm{KL}(\cdot ||\pi)\leq \mathcal{X}^2 (\cdot ||\pi)$ . The conclusion now follows via Gronwall's inequality.
To obtain an implementable algorithm, observe that the vector field in the continuity equation can be rewritten as [20]
$$
- \nabla P _ {\nu , k _ {\mathcal {L} _ {\nu}}} \frac {\eta_ {t}}{\nu} (\cdot) = - \int \nabla_ {1} k _ {\mathcal {L} _ {\nu}} (\cdot , y) \frac {\eta_ {t}}{\nu} (y) \mathrm {d} \nu (y) = - \int \nabla_ {1} k _ {\mathcal {L} _ {\nu}} (\cdot , y) \mathrm {d} \eta (y). \tag {86}
$$
Next, using a forward Euler discretisation in time, we arrive at
$$
\eta_ {t + 1} = \left(\mathrm {i d} - \gamma \nabla P _ {\nu , k _ {\mathcal {L} _ {\nu}}} \frac {\eta_ {t}}{\nu}\right) _ {\#} \eta_ {t}, \quad \mu_ {t + 1} = (\nabla \phi) _ {\#} \eta_ {t + 1}. \tag {87}
$$
Finally, we approximate the expectations in (87) with samples. In particular, after initialising a collection of particles  $x_0^i \stackrel{\mathrm{i.i.d.}}{\sim} \mu_0$  for  $i \in [N]$ , we set  $y_0^i = \nabla \phi(x_0^i)$  for  $i \in [N]$ , and then update
$$
y _ {t + 1} ^ {i} = y _ {t} ^ {i} - \gamma \frac {1}{N} \sum_ {j = 1} ^ {N} \nabla_ {y ^ {i}} k _ {\mathcal {L} _ {\nu}} \left(y _ {t} ^ {i}, y _ {t} ^ {j}\right), \quad x _ {t + 1} ^ {i} = \nabla \phi^ {*} \left(y _ {t + 1} ^ {i}\right). \tag {88}
$$
This algorithm, which we refer to as the mirrored LAWGD (MLAWGD) algorithm, is summarised in Alg. 4.
Algorithm 4 Mirrored LAWGD
input: target density  $\pi$ , kernel  $k_{\mathcal{L}_{\nu}}$ , mirror function  $\phi$ , particles  $(x_0^i)_{i=1}^N \sim \mu$ , step size  $\gamma$ .
initialise: for  $i\in [N]$ $y_0^i = \nabla \phi (x_0^i)$
For  $i\in [N],y_{t + 1}^{i} = y_{t}^{i} - \gamma \frac{1}{N}\sum_{j = 1}^{N}\nabla_{y^{i}}k_{\mathcal{L}_{\nu}}(y_{t}^{i},y_{t}^{j}).$
B.4 Mirrored Kernel Stein Discrepancy Descent
Finally, we consider the MWGF of the kernel Stein discrepancy (KSD) [21, 38, 62]. In so doing, we will obtain a mirrored version of the kernel Stein discrepancy descent (KSDD) algorithm in [51].
B.4.1 MWGF of the Kernel Stein Discrepancy
We begin by defining  $\mathcal{F}(\eta) = \frac{1}{2}\mathrm{KSD}_{\phi}^{2}(\eta |\nu)$ , where  $\mathrm{KSD}_{\phi}(\eta |\nu)$  is the mirrored KSD, which we define as
$$
\operatorname {K S D} _ {\phi} (\eta | \nu) = \sqrt {\int_ {\mathbb {R} ^ {d}} \int_ {\mathbb {R} ^ {d}} k _ {\nu , \phi} (y , y ^ {\prime}) \mathrm {d} \eta (y) \mathrm {d} \eta \left(y ^ {\prime}\right)}, \tag {89}
$$
where  $k_{\nu,\eta}$  is the mirrored Stein kernel, defined in terms of the score  $s_\nu = \nabla \log \nu$  of the mirrored target  $\nu = (\nabla \phi)_{\#} \pi$ , and the positive semi-definite kernel  $k_{\phi}$  as
$$
\begin{array}{l} k _ {\nu , \phi} (y, y ^ {\prime}) = \nabla_ {y} \log \nu (y) ^ {\top} k _ {\phi} (y, y ^ {\prime}) \nabla_ {y ^ {\prime}} \log \nu (y ^ {\prime}) + \nabla_ {y} \log \nu (y) ^ {\top} \nabla_ {y ^ {\prime}} k _ {\phi} (y, y ^ {\prime}) \\ + \nabla_ {y} k _ {\phi} ^ {\top} \left(y, y ^ {\prime}\right) \nabla \log \nu \left(y ^ {\prime}\right) + \left\langle \nabla_ {y} k _ {\phi} \left(y, \cdot\right), \nabla_ {y ^ {\prime}} k _ {\phi} \left(\cdot , y ^ {\prime}\right) \right\rangle_ {\mathcal {H} _ {k _ {\phi}} ^ {d}}, \tag {90} \\ \end{array}
$$
and where, as before,  $k_{\phi}$  is the kernel defined according to  $k_{\phi}(y,y^{\prime}) = k(\nabla \phi^{*}(y),\nabla \phi^{*}(y^{\prime}))$  , for some base kernel  $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$  . The MKSD is nothing more than the KSD between  $\eta = (\nabla \phi)_{\#}\mu$  and  $\nu = (\nabla \phi)_{\#}\pi$  , with respect to the kernel  $k_{\phi}$
In this case, one can show that  $\nabla_{W_2}\mathcal{F}(\eta) = \mathbb{E}_{y\sim \eta}[\nabla_2k_{\nu ,\phi}(y,\cdot)]$  [51, Proposition 2]. We can thus define the MKSD gradient flow according to
$$
\frac {\partial \eta_ {t}}{\partial t} - \nabla \cdot \left(\mathbb {E} _ {y \sim \eta_ {t}} [ \nabla_ {2} k _ {\nu , \phi} (y, \cdot) ] \eta_ {t}\right) = 0, \quad \mu_ {t} = (\nabla \phi^ {*}) _ {\#} \eta_ {t}. \tag {91}
$$
Similar to the previous sections, we would like to study the properties of this scheme in continuous time.
Proposition 15. The dissipation of  $\frac{1}{2}\mathrm{KSD}_{\phi}^{2}(\eta_{t}|\nu)$  along the MWGF in (91) is given by
$$
\frac {\mathrm {d} ^ {\frac {1}{2}} \mathrm {K S D} _ {\phi} ^ {2} \left(\eta_ {t} \mid \nu\right)}{\mathrm {d} t} = - \mathbb {E} _ {y ^ {\prime} \sim \eta_ {t}} \left[ \left| \left| \mathbb {E} _ {y \sim \eta_ {t}} \left[ \nabla_ {y ^ {\prime}} k _ {\nu , \phi} (y, y ^ {\prime}) \right] \right| \right| ^ {2} \right] \tag {92}
$$
Proof. The result follows straightforwardly using the chain rule. In particular, we have
$$
\begin{array}{l} \frac {\mathrm {d} \frac {1}{2} \mathrm {K S D} _ {\phi} ^ {2} \left(\eta_ {t} \mid \nu\right)}{\mathrm {d} t} = \left\langle w _ {t}, \nabla_ {W _ {2}} \frac {1}{2} \mathrm {K S D} ^ {2} \left(\eta_ {t} \mid \nu\right) \right\rangle_ {L ^ {2} \left(\eta_ {t}\right)} (93) \\ = - \mathbb {E} _ {y ^ {\prime} \sim \eta_ {t}} \left[ \left| \left| \mathbb {E} _ {y \sim \eta_ {t}} \left[ \nabla_ {y ^ {\prime}} k _ {\nu , \phi} \left(y, y ^ {\prime}\right) \right] \right| \right| ^ {2} \right]. (94) \\ \end{array}
$$
Remark 10. Unlike elsewhere (i.e., Propositions 1 - 2, Proposition 8, and Propositions 11 - 12) this dissipation result is given in terms of the mirrored densities  $(\eta_t)_{t\geq 0}$  and the mirrored target  $\nu = (\nabla \phi)_{\#}\pi$ , rather than densities  $(\mu_t)_{t\geq 0}$  and the target  $\pi$ . The crucial difference here is that the objective functional  $\frac{1}{2}\mathrm{KSD}^2 (\eta_t|\nu)$  is not equal to  $\frac{1}{2}\mathrm{KSD}^2 (\mu_t|\pi)$ , whereas previously it was true that, e.g.,  $\mathrm{KL}(\eta_t||\nu) = \mathrm{KL}(\mu_t||\pi)$  or  $\chi^2 (\eta_t||\nu) = \chi^2 (\mu_t||\pi)$ . Nonetheless, it is possible to give a dissipation result in terms of the target  $\pi$ .
Algorithm 5 Mirrored KSDD
input: target density  $\pi$ , kernel  $k$ , mirror function  $\phi$ , particles  $(x_0^i)_{i=1}^N \sim \mu$ , step size  $\gamma$ .
For  $i\in [N],y_{t + 1}^{i} = y_{t}^{i} - \gamma \frac{1}{N^{2}}\sum_{j = 1}^{N}\nabla_{y^{i}}k_{\nu ,\phi}(y_{t}^{j},y_{t}^{i}).$
We begin by rewriting the LHS of (92). Recall that the squared KSD between  $\mu$  and  $\pi$  is identical to the Stein Fisher information between  $\mu$  and  $\pi$  [e.g., 62, Theorem 3.6]. This also holds for  $\eta = (\nabla \phi)_{\#} \mu$  and  $\nu = (\nabla \phi)_{\#} \pi$  in the mirrored space, and thus we have
$$
\mathrm {K S D} _ {\phi} ^ {2} (\eta | \nu) = \left| \left| \mathcal {S} _ {\mu , k _ {\phi}} \nabla \log \left(\frac {\eta}{\nu}\right) \right| \right| _ {\mathcal {H} _ {k _ {\phi}} ^ {d}} ^ {2} = \left| \left| S _ {\mu , k} [ \nabla^ {2} \phi ] ^ {- 1} \nabla \log \left(\frac {\mu}{\pi}\right) \right| \right| _ {\mathcal {H} _ {k} ^ {d}} ^ {2}, \tag {95}
$$
where in the second equality we have used the result obtained in (57) - (60); see the proof of Proposition 8. Thus, in particular,  $\mathcal{F}(\eta)$  is equal to the mirrored Stein Fisher information, up to a factor half.
We now turn our attention to the RHS of (92). We would like to simplify  $\nabla_{y'} k_{\nu, \phi}(y, y')$ , as defined in (90). Let  $y = \nabla \phi(x)$  and  $y' = \nabla \phi(x')$ . Using the definition of  $k_{\phi}$ , we have  $k_{\phi}(y, y') = k(x, x')$ . Using also the chain rule, we have
$$
\nabla_ {y} k _ {\phi} (y, y ^ {\prime}) = \left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \nabla_ {x} k (x, x ^ {\prime}), \quad \nabla_ {y ^ {\prime}} k _ {\phi} (y, y ^ {\prime}) = \left[ \nabla^ {2} \phi (x ^ {\prime}) \right] ^ {- 1} \nabla_ {x ^ {\prime}} k (x, x ^ {\prime}) \tag {96}
$$
Similarly, it holds that
$$
\langle \nabla_ {y} k _ {\phi} (y, \cdot), \nabla_ {y ^ {\prime}} k _ {\phi} (\cdot , y ^ {\prime}) \rangle_ {\mathcal {H} _ {k _ {\phi}} ^ {d}} = \langle [ \nabla^ {2} \phi (x) ] ^ {- 1} \nabla_ {x} k (x, \cdot), [ \nabla^ {2} \phi (x ^ {\prime}) ] ^ {- 1} \nabla_ {x ^ {\prime}} k (\cdot , x ^ {\prime}) \rangle_ {\mathcal {H} _ {k} ^ {d}}. \tag {97}
$$
Finally, due to (20), we have  $\nabla_y\log \nu (y) = [\nabla^2\phi (x)]^{-1}\left[\nabla_x\log \pi (x) - \nabla_x\log \operatorname *{det}\nabla^2\phi (x)\right]$ . Putting everything together, we thus have that
$$
\begin{array}{l} k _ {\nu , \phi} (y, y ^ {\prime}) = \left([ \nabla^ {2} \phi (x) ] ^ {- 1} [ \nabla_ {x} \log \pi (x) - \nabla_ {x} \log \det  \nabla^ {2} \phi (x) ]\right) ^ {\top} k (x, x ^ {\prime}) \\ \cdot \left(\left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \left[ \nabla_ {x} \log \pi (x) - \nabla_ {x} \log \det \nabla^ {2} \phi (x) \right]\right) \\ + \left(\left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \left[ \nabla_ {x} \log \pi (x) - \nabla_ {x} \log \det \nabla^ {2} \phi (x) \right]\right) ^ {\top} \left(\left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \nabla_ {x} k (x, x ^ {\prime})\right) \\ + \left(\left[ \nabla^ {2} \phi (x ^ {\prime}) \right] ^ {- 1} \nabla_ {x ^ {\prime}} k (x, x ^ {\prime})\right) ^ {\top} \left(\left[ \nabla^ {2} \phi (x) \right] ^ {- 1} \left[ \nabla_ {x} \log \pi (x) - \nabla_ {x} \log \det  \nabla^ {2} \phi (x) \right]\right) \\ + \langle [ \nabla^ {2} \phi (x) ] ^ {- 1} \nabla_ {x} k (x, \cdot), [ \nabla^ {2} \phi \left(x ^ {\prime}\right) ] ^ {- 1} \nabla_ {x ^ {\prime}} k (\cdot , x ^ {\prime}) \rangle := \tilde {k} _ {\pi , \phi} (x, x ^ {\prime}). \tag {98} \\ \end{array}
$$
Using (95) and (98) we thus have the following dissipation result for the mirrored Stein Fisher information in terms of the target  $\pi$ :
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left| \left| S _ {\mu_ {t}, k} \left[ \nabla^ {2} \phi \right] ^ {- 1} \nabla \log \left(\frac {\mu_ {t}}{\pi}\right) \right| \right| _ {\mathcal {H} _ {k} ^ {d}} ^ {2} = - 2 \mathbb {E} _ {x ^ {\prime} \sim \mu_ {t}} \left[ \left| \left| \mathbb {E} _ {x \sim \mu_ {t}} \left[ \tilde {k} _ {\pi , \phi} \left(x, x ^ {\prime}\right) \right] \right| \right| ^ {2} \right]. \tag {99}
$$
B.4.2 Mirrored Kernel Stein Discrepancy Descent (MKSSD)
To obtain an implementable MKSDD algorithm, it is, of course, necessary to discretise in time and in space. In this case, following [51], we will first discretise in space, replacing the measure  $\eta$  by a discrete approximation  $\hat{\eta} = \frac{1}{N}\sum_{j=1}^{N}\delta_{y^j}$ . This yields, writing  $\omega^N = \{y^1,\dots,y^n\}$ ,
$$
F \left(\omega^ {N}\right) := \mathcal {F} (\hat {\eta}) = \frac {1}{2 N ^ {2}} \sum_ {i, j = 1} ^ {N} k _ {\nu , \phi} \left(y ^ {i}, y ^ {j}\right), \quad \nabla_ {y ^ {i}} F \left(\omega^ {N}\right) := \mathcal {F} (\hat {\eta}) = \frac {1}{N ^ {2}} \sum_ {j = 1} ^ {N} \nabla_ {2} k _ {\nu , \phi} \left(y ^ {j}, y ^ {i}\right). \tag {100}
$$
The mirrored KSDD algorithm thus takes the following form. Initialise a collection of particles  $x_0^{i\text{i.d.}}$ $\mu_0$  for  $i \in [N]$ , and set  $y_0^i = \nabla \phi (x_0^i)$  for  $i \in [N]$ . Then, update
$$
y _ {t + 1} ^ {i} = y _ {t} - \gamma \frac {1}{N ^ {2}} \sum_ {j = 1} ^ {N} \nabla_ {2} k _ {\nu , \phi} \left(y _ {t} ^ {j}, y _ {t} ^ {i}\right), \quad x _ {t + 1} ^ {i} = \nabla \phi^ {*} \left(y _ {t + 1} ^ {i}\right). \tag {101}
$$
C Convergence of Mirrored Coin Wasserstein Gradient Descent
Proposition 16. Suppose that  $\pi$  is strongly log-concave and that  $||\nabla_{W_2}\mathcal{F}(\eta_t)(y_t)||\leq L$  for all  $t\in [T]$ . In addition, suppose that  $(\eta_t)_{t\in \mathbb{N}_0}$  and  $\nu$  satisfy [81, Assumption B.1]. Then there exists a mirror map  $\phi$  such that
$$
\begin{array}{l} \operatorname {K L} \left(\left. \frac {1}{T} \sum_ {t = 1} ^ {T} \mu_ {t} \right| | \pi\right) \leq \frac {L}{T} \left[ 1 + \int_ {\mathcal {X}} | | \nabla \phi (x) | | \sqrt {T \ln \left(1 + 9 6 K ^ {2} T ^ {2} | | \nabla \phi (x) | | ^ {2}\right)} \pi (\mathrm {d} x) \right. \\ \left. + \int_ {\mathcal {X}} | | \nabla \phi (x) | | \sqrt {T \ln \left(1 + 9 6 T ^ {2} \left| | \nabla \phi (x) \right| ^ {2}\right)} \mu_ {0} (\mathrm {d} x) \right], \tag {102} \\ \end{array}
$$
where  $K > 0$  is the constant defined in [81, Assumption B.1].
Proof. First note, using Jensen's inequality, and [43, Theorem 2], that
$$
\begin{array}{l} \operatorname {K L} \left(\left. \frac {1}{T} \sum_ {t = 1} ^ {T} \mu_ {t} \right| | \pi\right) \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {K L} (\mu_ {t} | | \pi) (103) \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} \left[ \mathrm {K L} \left(\eta_ {t} | | \nu\right) - \mathrm {K L} (\nu | | \nu) \right] (104) \\ := \frac {1}{T} \sum_ {t = 1} ^ {T} \mathcal {F} (\eta_ {t}) - \mathcal {F} (\nu), (105) \\ \end{array}
$$