SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for  $h \in C_c^2(\mathbb{R}^J, \mathbb{R})$  (Barbu and Röckner, 2020, cf. equation (1.1)-(1.4)). Note that (56) corresponds exactly to the Wasserstein gradient flow equation in (47). We can therefore follow the WGF by simulating solutions to (55).
The standard approach to simulate solutions to (55) (Veretennikov, 2006) is to use an ensemble of interacting particles. Formally, we replace  $Q(t)$  by  $\frac{1}{N_E} \sum_{n=1}^{N_E} \delta_{\theta_n(t)}$  and obtain
$$
d \theta_ {n} (t) = - \left(\nabla V \left(\theta_ {n} (t)\right) + \frac {\lambda_ {1}}{N _ {E}} \sum_ {j = 1} ^ {N _ {E}} \left(\nabla_ {1} \kappa\right) \left(\theta_ {n} (t), \theta_ {j} (t)\right)\right) d t + \sqrt {2 \lambda_ {2}} d B _ {n} (t) \tag {58}
$$
for  $n = 1, \ldots, N_E$  where  $N_E \in \mathbb{N}$  denotes the number of particles. The Euler-Maruyama approximation of (58) leads to the final algorithm:
Step 1: Initialise  $N_{E} \in \mathbb{N}$  particles  $\theta_{1,0}, \ldots, \theta_{N_{E},0}$  from a use chosen initial distribution  $Q_{0}$ .
Step 2: Evolve the particles forward in time according to
$$
\theta_ {n, k + 1} = \theta_ {n, k} - \eta \left(\nabla V \left(\theta_ {n, k}\right) + \frac {\lambda_ {1}}{N _ {E}} \sum_ {j = 1} ^ {N _ {E}} \left(\nabla_ {1} \kappa\right) \left(\theta_ {n, k}, \theta_ {j, k}\right)\right) + \sqrt {2 \eta \lambda_ {2}} Z _ {n, k} \tag {59}
$$
for  $n = 1,\ldots ,N_E,k = 0,\ldots ,T - 1$  with  $Z_{n,k}\sim \mathcal{N}(0,I_{J\times J})$
Note that  $\theta_{n,k}$  is thought of as approximation of  $\theta_n(t)$  at position  $t = k\eta$ . Furthermore, as discussed in Section 4, various choices of  $V$ ,  $\lambda_1$  and  $\lambda_2$  allow us to implement the WGF for different regularised optimisation problems in the space of probability measures. This is summarised below:
- Deep ensembles:  $V(\theta) = \ell(\theta)$ ,  $\lambda_1 = 0$ ,  $\lambda_2 = 0$
- Deep Langevin ensembles:  $V(\theta) = \ell(\theta) - \lambda \log p(\theta)$ ,  $\lambda_1 \coloneqq 0$ ,  $\lambda \coloneqq \lambda_2$
- Deep repulsive Langevin ensembles:  $V(\theta) = \ell(\theta) - \lambda_1 \log p(\theta) - \lambda_2 \mu_P(\theta)$
C Asymptotic distribution of particles: unregularised objective
In this section, we investigate the asymptotic distribution of the WGF for the objective
$$
L (Q) := \int \ell (\theta) d Q (\theta) \tag {60}
$$
for  $Q\in \mathcal{P}(\mathbb{R}^J)$ . The associated particle method is:
- Sample  $\theta_{1}(0),\dots,\theta_{N_{E}}(0)$  independently from  $Q_{0}$ .
- Simulate (deterministically)  $\theta_{n}^{\prime}(t) = -\nabla \ell \big(\theta_{n}(t)\big)$  for  $n = 1,\dots ,N_E$
We start by introducing some notation for the deterministic gradient system. Let  $\phi^t (\theta_0)$  denote the solution to the ordinary differential equation (ODE)
$$
\theta (0) = \theta_ {0} \in \mathbb {R} ^ {J} \tag {61}
$$
$$
\theta^ {\prime} (t) = - \nabla \ell (\theta (t)) \tag {62}
$$
at time  $t > 0$ . In a first step, we show the following lemma, which is a simple application of the famous Lojasiewicz theorem (Colding and Minicozzi II, 2014), and the fact that Lebesgue almost every initialisation leads to a local minimum (Lee et al., 2016).
Lemma 3. Assume  $\ell : \mathbb{R}^J \to \mathbb{R}$  is norm-coercive and satisfies the Lojasiewicz inequality, i.e. for every  $\theta \in \mathbb{R}^J$  exists an environment  $U$  of  $\theta$  and constants  $0 < \gamma < 1$  and  $C > 0$  such that
$$
| \ell (\theta) - \ell (\bar {\theta}) | ^ {\gamma} <   C | \nabla \ell (\theta) |. \tag {63}
$$
for all  $\bar{\theta} \in U$ . Then we know that  $\phi^t(\theta_0)$  converges for  $t \to \infty$  to a local minimum of  $\ell$  for Lebesgue almost every  $\theta_0 \in \mathbb{R}^J$ .
Proof. First we show that  $t \mapsto \phi^t(\theta_0)$  is bounded. We proof this by contradiction. Assume that  $\phi^t(\theta_0)$  is unbounded. Then there exists a subsequence  $(t_n)_{n \in \mathbb{N}} \subset [0, \infty)$  with  $t_n \to \infty$  for  $n \to \infty$  such that
$$
\left| \phi^ {t _ {n}} \left(\theta_ {0}\right)\right|\rightarrow \infty \tag {64}
$$
for  $n\to \infty$  . The norm-coercivity immediately implies that
$$
\ell \left(\phi^ {t _ {n}} \left(\theta_ {0}\right)\right)\rightarrow \infty \tag {65}
$$
for  $n\to \infty$  . However, this contradicts
$$
\ell \left(\phi^ {t} \left(\theta_ {0}\right)\right) \leq \ell \left(\phi^ {0} \left(\theta_ {0}\right)\right) = \ell \left(\theta_ {0}\right) <   \infty , \tag {66}
$$
where the first inequality follows from the fact that  $t \mapsto \ell\big(\phi^t(\theta_0)\big)$  is decreasing, which is a consequence of
$$
\begin{array}{l} \frac {d}{d t} \ell \left(\phi^ {t} \left(\theta_ {0}\right)\right) = \nabla \ell \left(\phi^ {t} \left(\theta_ {0}\right)\right) \frac {d}{d t} \phi^ {t} \left(\theta_ {0}\right) (67) \\ = - \left| \nabla \ell \left(\phi^ {t} \left(\theta_ {0}\right)\right) \right| ^ {2} \leq 0. (68) \\ \end{array}
$$
Hence  $t \mapsto \phi^t(\theta_0)$  is bounded. By the Bolzano-Weierstrass theorem we can find a sequence  $(t_n)_{n \in \mathbb{N}} \subset [0, \infty)$  with  $t_n \to \infty$  and a point  $\theta_\infty \in \mathbb{R}^J$  such that
$$
\phi^ {t _ {n}} \left(\theta_ {0}\right)\rightarrow \theta_ {\infty} \tag {69}
$$
for  $n\to \infty$ . Hence  $\left(\phi^t (\theta_0)\right)_{t > 0}$  has the accumulation point  $\theta_{\infty}$ . The Lojasiewicz theorem (Colding and Minicozzi II, 2014) allows us to deduce that
$$
\phi^ {t} \left(\theta_ {0}\right)\rightarrow \theta_ {\infty} \tag {70}
$$
for  $t\to \infty$  , and that  $\theta_{\infty}$  satisfies  $\nabla \ell (\theta_{\infty}) = 0$
It remains to show that  $\theta_{\infty}$  is not a saddle point for Lebesgue almost every initial value  $\theta_0$ . However, this is very similar to the proof in Lee et al. (2016). The only difference is that one would need to use a continuous-time version of the stable manifold theorem, which is readily available, for example in Bressan (2003).
Let  $\{m_i\}_{i\in \mathbb{N}}$  denote the local minima of  $\ell$  which are by assumption countable. Denote further by
$$
\Theta_ {i} := \left\{\theta_ {0} \in \mathbb {R} ^ {J}: \lim  _ {t \rightarrow \infty} \phi^ {t} \left(\theta_ {0}\right)\rightarrow m _ {i} \right\} \tag {71}
$$
the domain of attraction for the minimum  $m_{i}$ . The next theorem is then an easy consequence of Lemma 3.
Theorem 5. Assume that the loss function  $\ell$  only has countably many local minima, is norm coercive, and satisfies the Lojasiewicz inequality. Let further  $\theta_0 \sim Q_0$  for some  $Q_0 \in \mathcal{P}(\mathbb{R}^J)$  such that  $\sum_{i=1}^{\infty} Q_0(\Theta_i) = 1$ . Then,
$$
\phi^ {t} \left(\theta_ {0}\right) \xrightarrow {\mathcal {D}} \sum_ {i = 1} ^ {\infty} Q _ {0} \left(\Theta_ {i}\right) \delta_ {m _ {i}} =: Q _ {\infty} \tag {72}
$$
for  $t\to \infty$  .Here  $\xrightarrow{\mathcal{D}}$  denotes convergence in distribution.
Proof. Let  $\theta_0\in \mathbb{R}^J$  be fixed. Due to Lemma 3, we know that
$$
\phi^ {t} \left(\theta_ {0}\right)\rightarrow \sum_ {i = 1} ^ {\infty} m _ {i} \mathbb {1} \left\{\theta_ {0} \in \Theta_ {i} \right\} \tag {73}
$$
for Lebesgue almost every  $\theta_0$  for  $t\to \infty$ . Here,  $\mathbb{1}\{\cdot \}$  denotes the indicator function. Let  $Y$  now be a random variable with law  $Q_{0}$ . By assumption, we know that  $Y\in \Theta_{i}$  for some  $i\in \mathbb{N}$  with probability 1. Hence,
$$
\phi^ {t} (Y) \rightarrow \sum_ {i = 1} ^ {\infty} m _ {i} \mathbb {I} \{Y \in \Theta_ {i} \} \tag {74}
$$
almost surely for  $t \to \infty$ . Since almost sure convergence implies convergence in distribution, we conclude that
$$
\phi^ {t} (Y) \xrightarrow {\mathcal {D}} \mathcal {L} \left(\sum_ {i = 1} ^ {\infty} m _ {i} \mathbb {1} \{Y \in \Theta_ {i} \}\right), \tag {75}
$$
where  $\mathcal{L}(\cdot)$  denotes the law of a random variable. However, the law of the RHS is easily recognised as
$$
\mathcal {L} \left(\sum_ {i = 1} ^ {\infty} m _ {i} \mathbb {1} \{Y \in \Theta_ {i} \}\right) = \sum_ {i = 1} ^ {\infty} Q _ {0} (\Theta_ {i}) \delta_ {m _ {i}}, \tag {76}
$$
Remark 2. Note that the condition
$$
\sum_ {i = 1} ^ {\infty} Q _ {0} \left(\Theta_ {i}\right) = 1 \tag {77}
$$
in Theorem 5 is easy to satisfy. According to Lemma 3 the set
$$
\mathbb {R} ^ {J} \backslash \bigcup_ {i = 1} ^ {n} \Theta_ {i} \tag {78}
$$
has Lebesgue measure zero. Therefore, any  $Q_0$  which has a density w.r.t. the Lebesgue measure will satisfy (77).
D Asymptotic distribution for deep Langevin ensembles
In this section, we analyse the objective
$$
L (Q) := \int \ell (\theta) d Q (\theta) + \lambda \operatorname {K L} (Q, P) \tag {79}
$$
for  $Q \in \mathcal{P}(\mathbb{R}^J)$ . The corresponding particle method is given as:
- Sample  $\theta_{1}(0),\dots,\theta_{N_{E}}(0)$  independently from  $Q_{0}$ .
- Simulate the SDE  $d\theta_{n}(t) = -\nabla V(\theta_{n}(t))dt + \sqrt{2\lambda} dB_{n}(t)$  for each  $n = 1,\dots ,N_E$
Recall that  $V(\theta) = \ell(\theta) - \lambda \log p(\theta)$ . This case is well-studied in the literature and known as Langevin diffusion. Under mild assumptions (Chiang et al., 1987; Roberts and Tweedie, 1996),
$$
\theta_ {n} (t) \xrightarrow {\mathcal {D}} Q _ {\infty} \tag {80}
$$
for  $t \to \infty$  and each particle  $n = 1, \ldots, N_E$  independently. The probability measure  $Q_{\infty}$  has the density
$$
q _ {\infty} (\theta) = \frac {1}{Z} \exp \left(- \frac {V (\theta)}{\lambda}\right) \tag {81}
$$
$$
= \frac {1}{Z} \exp (- \frac {\ell (\theta)}{\lambda}) p (\theta), \tag {82}
$$
where  $Z > 0$  is the normalising constant. As a consequence, the WGF asymptotically produces samples from  $Q_{\infty}$ . However, it is a priori unclear that  $Q_{\infty}$  is in fact the same as the global minimiser  $Q^{*}$  of  $L$ .
We investigate this question by relating invariant measures to stationary points of the Wasserstein gradient.
Definition 1. (Liggett, 2010, Thm. 3.3.7) A measure  $Q$  is called an invariant measure (for a given Feller-process) if
$$
\int A h (\theta) d Q (\theta) = 0 \tag {83}
$$
for all  $h \in C_c^2(\mathbb{R}^J)$ . Here  $A$  is the infinitesimal generator of the corresponding Feller-process.
Recall that the infinitesimal generator of the Langevin diffusion for  $h \in C_c^2(\mathbb{R}^J)$  is given as
$$
A h = - \nabla V \cdot \nabla h + \lambda \Delta h. \tag {84}
$$
Definition 2. A measure  $Q \in \mathcal{P}_2(\mathbb{R}^J)$  is called a stationary point of the Wasserstein gradient if
$$
\nabla_ {W} L [ Q ] (\theta) = 0 \tag {85}
$$
for  $Q$ -almost every  $\theta \in \mathbb{R}^J$ .
In finite dimensions, it is well-known that a local minimiser is a stationary point of the gradient. This carries over to the infinite-dimensional case, with a similar proof. Since we could not find this result anywhere in the literature we included it for completeness.
Lemma 4. Let  $\widehat{Q}$  be a local minimiser of  $L$ , i.e. there exists and  $\epsilon > 0$  such that
$$
L (\widehat {Q}) \leq L (Q) \tag {86}
$$
for all  $Q$  with  $W_{2}(\widehat{Q}, Q) \leq \epsilon$ . Then  $\widehat{Q}$  is a stationary point of the Wasserstein gradient in the sense of Definition 2.
Proof. Let  $h \in C_c^2(\mathbb{R}^J)$  be arbitrary and  $\widehat{Q} \in \mathcal{P}_2(\mathbb{R}^J)$  be a local minimum of  $L$ . Further, let  $\phi^t(\theta_0)$  be the solution to the initial value problem
$$
\theta (0) = \theta_ {0} \tag {87}
$$
$$
\theta^ {\prime} (t) = \nabla h (\theta (t)) \tag {88}
$$
for  $t \in (-\epsilon, \epsilon)$  for some  $\epsilon > 0$ . We now define  $Q(t) \coloneqq \phi^t \# \widehat{Q}$  for  $t \in (-\epsilon, \epsilon)$  where  $f \# \mu$  denotes the push-forward of the measures  $\mu$  through the function  $f$ . In the Riemannian interpretation of the Wasserstein space,  $\big(Q(t)\big)_{t \in (-\epsilon, \epsilon)}$  is a curve in  $\mathcal{P}_2(\mathbb{R}^J)$  with tangent vector  $h$  at point  $\widehat{Q}$  (Ambrosio et al., 2005, Chapter 8). We, further, define  $f: (-\epsilon, \epsilon) \to \mathbb{R}$  as  $f(t) \coloneqq L(Q(t))$ . Application of the chain-rule (Ambrosio et al., 2005, p. 233) gives
$$
f ^ {\prime} (0) = \left. \frac {d}{d t} L (Q (t)) \right| _ {t = 0} \tag {89}
$$
$$
= \left\langle \nabla_ {W} L [ Q (0) ], \nabla h \right\rangle_ {L ^ {2} (Q (0))} \tag {90}
$$
$$
= \int \nabla_ {W} L [ \widehat {Q} ] (\theta) \cdot \nabla h (\theta) d \widehat {Q} (\theta). \tag {91}
$$
We know that  $f$  has a local minimum at  $t = 0$  and, therefore,  $f'(0) = 0$  which gives
$$
0 = \int \nabla_ {W} L [ \widehat {Q} ] (\theta) \cdot \nabla h (\theta) d \widehat {Q} (\theta). \tag {92}
$$
Since (92) holds for arbitrary test functions  $h \in C_c^2(\mathbb{R}^J)$  and as  $C_c^2(\mathbb{R}^J)$  is dense in  $L^2(\widehat{Q})$ , we obtain that  $\nabla_W L[\widehat{Q}](\theta) = 0$  for  $\widehat{Q}$ -a.e  $\theta \in \mathbb{R}^J$ .
The next lemma relates invariant measures and stationary points of the Wasserstein gradient for infinitesimal generators of the form (84). It will prove extremely useful to translate between the Langevin diffusion literature and our optimisation perspective.
Lemma 5. Let  $Q \in \mathcal{P}_2(\mathbb{R}^J)$  be such that  $Q$  has a density  $q$  with respect to the Lebesgue measure. Then, the following two statements are equivalent:
-  $Q$  is a stationary point of the Wasserstein gradient.
-  $Q$  is an invariant measure.
Proof. Let  $Q$  be a measure with density  $q$ . Recall that the generator of the Langevin diffusion is for  $h \in C_c^2(\mathbb{R}^J)$  given as
$$
A h = - \nabla V \cdot \nabla h + \lambda \Delta h. \tag {93}
$$
By partial integration, it is easy to verify that the  $L^2$ -adjoint (w.r.t the Lebesgue measure) is given as
$$
A ^ {*} h = \nabla \cdot (h \cdot \nabla V) + \lambda \Delta h. \tag {94}
$$
We, therefore, conclude that
$$
\begin{array}{l} \int A h (\theta) d Q (\theta) = \int A h (\theta) q (\theta) d \theta (95) \\ = \int h (\theta) A ^ {*} q (\theta) d \theta (96) \\ = \int h (\theta) \left(\nabla \cdot (q (\theta) \cdot \nabla V (\theta)) + \lambda \Delta q (\theta)\right) d \theta . (97) \\ \end{array}
$$
Furthermore, we have  $\nabla_W L[Q] = \nabla V + \lambda \nabla \log q$ , and therefore
$$
\begin{array}{l} \int \nabla_ {W} L [ Q ] (\theta) \cdot \nabla h (\theta) d Q (\theta) = \int \nabla_ {W} L [ Q ] (\theta) \cdot \nabla h (\theta) q (\theta) d \theta (98) \\ = \int (\nabla V (\theta) q (\theta) + \lambda \nabla q (\theta)) \cdot \nabla h (\theta) d \theta (99) \\ = - \int h (\theta) \left(\nabla \cdot (q (\theta) \nabla V (\theta)) + \lambda \Delta q (\theta)\right) d \theta , (100) \\ \end{array}
$$
where the last line follows from applying partial integration. This allows us to conclude that
$$
\int A h (\theta) d Q (\theta) = - \int \nabla_ {W} L [ Q ] (\theta) \cdot \nabla h (\theta) d Q (\theta) \tag {101}
$$
whenever  $Q$  has a density. As a consequence we have that  $Q$  is invariant if and only if it is a stationary point of the Wasserstein gradient.
Lemma 5 allows us to move between the optimisation and stochastic differential equation perspective. In Appendix A, we discussed the existence and uniqueness of a global minimiser  $Q^{*}$  of  $L$ . We know that  $Q^{*}$  has a density since the Kullback-Leibler divergence would be infinite otherwise (assuming  $P$  has a Lebesgue-density which we assume throughout the paper). Lemma 4 guarantees that  $Q^{*}$  is a stationary point of the Wasserstein gradient. Due to Lemma 5, we can infer that  $Q^{*}$  must be an invariant measure. However, due to the uniqueness of the invariant measure under the previously mentioned mild assumptions (Chiang et al., 1987; Roberts and Tweedie, 1996), we can conclude that  $Q^{*} = Q_{\infty}$ .
E Asymptotic distribution of deep repulsive Langevin ensembles
In this section, we consider
$$
\begin{array}{l} L (Q) = \int \ell (\theta) d Q (\theta) + \frac {\lambda_ {1}}{2} \operatorname {M M D} (Q, P) ^ {2} + \lambda_ {2} \operatorname {K L} (Q, P) (102) \\ = \int V (\theta) d Q (\theta) + \frac {\lambda_ {1}}{2} \int \kappa (\theta , \theta^ {\prime}) d Q (\theta) d Q \left(\theta^ {\prime}\right) - \lambda_ {2} H (Q) + c o n s t, (103) \\ \end{array}
$$
as optimisation objective. Here,  $H(Q) = -\int \log q(\theta)q(\theta)d\theta$  denotes the differential entropy.
Recall that in this case  $V(\theta) = \ell(\theta) - \lambda_1 \mu_P(\theta) - \lambda_2 \log p(\theta)$ . We already discussed in Appendix B that the McKean-Vlasov process of the form
$$
\theta (0) \sim Q _ {0} \tag {104}
$$
$$
d \theta (t) = - \left(\nabla V (\theta (t)) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q _ {t}\right) (\theta (t))\right) d t + \sqrt {2 \lambda_ {2}} d B (t), \tag {105}
$$
with  $\left(B(t)\right)_{t\geq 0}$  being a Brownian motion achieves the desired density evolution. Furthermore, the particle approximation of (104) is given as
$$
d \theta_ {n} (t) = - \left(\nabla V \left(\theta_ {n} (t)\right) + \frac {\lambda_ {1}}{N _ {E}} \sum_ {j = 1} ^ {N _ {E}} \left(\nabla_ {1} \kappa\right) \left(\theta_ {n} (t), \theta_ {j} (t)\right)\right) d t + \sqrt {2 \lambda_ {2}} d B _ {n} (t) \tag {106}
$$
for  $n = 1,\dots ,N_E$  where  $N_{E}\in \mathbb{N}$  denotes the number of particles.
The approach follows the same procedure as in Appendix D. We show the notions of invariant measures and stationary points of the Wasserstein gradient are the same for measures with Lebesgue density. We start by introducing the concept of an invariant measure for a nonlinear Markov process (Ahmed and Ding, 1993, Definition 1).
Definition 3. A measure  $Q$  is called an invariant measure for a nonlinear Markov process with the family of infinitesimal generators  $\{A[Q]\}_{Q\in \mathcal{P}(\mathbb{R}^J)}$  if
$$
\int A [ Q ] h (\theta) d Q (\theta) = 0 \tag {107}
$$
for all  $h\in C_c^2 (\mathbb{R}^J)$
Recall that the family of infinitesimal generators in our case is given as
$$
\left(A [ Q ] h\right) (\theta) := - \left(\nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta)\right) \cdot \nabla h (\theta) + \lambda_ {2} \Delta h. \tag {108}
$$
for  $h \in C_c^2(\mathbb{R}^J, \mathbb{R})$ . In analogy to Lemma 5, we obtain the following result.
Lemma 6. Let  $Q \in \mathcal{P}_2(\mathbb{R}^J)$  be such that  $Q$  has a density  $q$  with respect to the Lebesgue measure. Then, the following two statements are equivalent:
-  $Q$  is a stationary point of the Wasserstein gradient for  $L$  in (102) in the sense of Def. 2.
-  $Q$  is an invariant measure for the McKean-Vlasov process with infinitesimal generator defined in (108)
Proof. First, we notice that
$$
\begin{array}{l} \int A [ Q ] h (\theta) d Q (\theta) = \int A [ Q ] h (\theta) q (\theta) d \theta (109) \\ = \int h (\theta) \left(A ^ {*} [ Q ] q\right) (\theta) d \theta . (110) \\ \end{array}
$$
Recall, that  $A^{*}[Q]$  denotes the  $L^2$ -adjoint of the operator  $A[Q]$  and that it is given as
$$
\left(A ^ {*} [ Q ] h\right) (\theta) = \nabla \cdot \left(h (\theta) \left(\nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta) + \lambda_ {2} \nabla \log (h (\theta))\right)\right) \tag {111}
$$
for  $h \in C^2(\mathbb{R}^J, \mathbb{R})$  with compact support. This implies
$$
\left(A ^ {*} [ Q ] q\right) (\theta) = \nabla \cdot \left(q (\theta) \left(\nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta)\right)\right) + \lambda_ {2} \Delta q (\theta). \tag {112}
$$
We plug this into (110) to obtain
$$
\begin{array}{l} \int A [ Q ] h (\theta) d Q (\theta) (113) \\ = \int h (\theta) \nabla \cdot \left(q (\theta) \left(\nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta)\right)\right) d \theta + \int \lambda_ {2} h (\theta) \Delta q (\theta) d \theta . (114) \\ \end{array}
$$
On the other hand, we have that
$$
\nabla_ {W} L [ Q ] (\theta) = \nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta) + \lambda_ {2} \nabla \log q (\theta), \tag {115}
$$
and therefore
$$
\begin{array}{l} \int \nabla L [ Q ] (\theta) \cdot \nabla h (\theta) d Q (\theta) (116) \\ = \int (\nabla V (\theta) + \lambda_ {1} (\nabla_ {1} \kappa * Q) (\theta) + \lambda_ {2} \nabla \log q (\theta)) \cdot \nabla h (\theta) d Q (\theta) (117) \\ = \int q (\theta) \left(\nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta)\right) \cdot \nabla h (\theta) d \theta + \lambda_ {2} \int \nabla q (\theta) \cdot \nabla h (\theta) d \theta (118) \\ = - \int \nabla \cdot \left(q (\theta) \left(\nabla V (\theta) + \lambda_ {1} \left(\nabla_ {1} \kappa * Q\right) (\theta)\right)\right) h (\theta) d \theta - \lambda_ {2} \int q (\theta) \Delta h (\theta) d \theta , (119) \\ \end{array}
$$
where the last line follows from partial integration. Comparing (114) to (119) gives
$$
\int A [ Q ] h (\theta) d Q (\theta) = - \int \nabla L [ Q ] (\theta) \cdot \nabla h (\theta) d Q (\theta) \tag {120}
$$
for all  $h \in C_c^2(\mathbb{R}^J)$  whenever  $Q$  has a density. This immediately implies that  $Q$  is invariant iff it is a stationary point.
Again, we leverage this correspondence between stationary point and invariant measures. There is a rich literature on ergodicity of nonlinear Markov processes. For example, Theorem 2 of Veretennikov (2006) specifies conditions on  $\kappa$  and  $V$  such that
$$
Q ^ {n, N _ {E}} (t) \xrightarrow {\mathcal {D}} Q _ {\infty} \tag {121}
$$
for  $N_{E}, t \to \infty$ . Here  $Q^{n,N_E}(t)$  denotes the law of a fixed particle  $\theta_{n}(t)$ ,  $n = 1, \dots, N_{E}$ , whose distribution is characterised by the SDE (58). The measure  $Q_{\infty}$  is the unique invariant measure of the nonlinear Markov process. By Lemma 6 every invariant measure is a stationary point of the Wasserstein gradient and vice versa. Hence, existence and uniqueness of the stationary point of the Wasserstein gradient is immediately implied. However, since the global minimiser  $Q^{*}$  is a stationary point of the Wasserstein gradient (cf. Lemma 4), we conclude by uniqueness that  $Q_{\infty} = Q^{*}$ .
F Asymptotic analysis of deep repulsive ensembles
In this section, we consider the objective
$$
L (Q) := \int \ell (\theta) d Q (\theta) + \lambda \operatorname {M M D} (Q, P) \tag {122}
$$
for  $Q \in \mathcal{P}(\mathbb{R}^J)$ . The corresponding McKean-Vlasov process is of the form
$$
d \theta (t) = - \left(\nabla V (\theta (t)) + \lambda \left(\nabla_ {1} \kappa * Q _ {t}\right) (\theta (t))\right) d t, \tag {123}
$$
where  $Q_{t}$  denotes the distribution of  $\theta(t)$  and  $V(\theta) = \ell(\theta) - \mu_{P}(\theta)$  with  $\mu_{P}(\theta) = \int \kappa(\theta, \theta') dP(\theta)$  the kernel mean-embedding of  $P$ . We call the particle method in this case deep repulsive ensembles (DRE).
The existence of the global minimiser  $Q^{*}$  is still guaranteed under the assumptions in Appendix A. Lemma 4 guarantees that  $Q^{*}$  is a stationary point of the Wasserstein gradient, i.e.
$$
\nabla V (\theta) + \lambda \left(\nabla_ {1} \kappa * Q ^ {*}\right) (\theta) = 0 \tag {124}
$$
for  $Q^{*}$ -a.e.  $\theta \in \mathbb{R}^{J}$ . Recall that the infinitesimal generator in this case is given as
$$
\left(A [ Q ] h\right) (\theta) := - \left(\nabla V (\theta) + \lambda \left(\nabla_ {1} \kappa * Q\right) (\theta)\right) \cdot \nabla h (\theta) \tag {125}
$$
for  $Q \in \mathcal{P}(\mathbb{R}^J)$ ,  $h \in C_c^2(\mathbb{R}^J)$ . It immediately follows from the definition that
$$
\left(A [ Q ] h\right) (\theta) = - \nabla_ {W} L [ Q ] (\theta) \cdot \nabla h (\theta) \tag {126}
$$
for all  $h \in C_c^2(\mathbb{R}^J)$ ,  $\theta \in \mathbb{R}^J$ . As in Lemma 5 & 6, this implies that each stationary point of the Wasserstein gradient is an invariant measure of the McKean-Vlasov process and vice versa. In Appendix D & E, we cite relevant literature that guarantees uniqueness of the invariant measure, which is a necessary (but not sufficient) condition for convergence to the invariant measure. The next theorem shows that uniqueness will in general not hold without the presence of the diffusion term.
Theorem 6. The invariant measure for the McKean-Vlasov process with the family of generators  $\left(A[Q]\right)_{Q\in \mathcal{P}(\mathbb{R}^J)}$  defined in (126) is (in general) not unique.