SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Let  $N_E \in \mathbb{N}$  and define  $\widetilde{L} : (\mathbb{R}^J)^{N_E} \to \mathbb{R}$  as
$$
\widetilde {L} \left(\theta_ {1}, \dots , \theta_ {N _ {E}}\right) := \sum_ {i = 1} ^ {N _ {E}} V \left(\theta_ {i}\right) + \frac {\lambda}{2 N _ {E}} \sum_ {i, j = 1} ^ {N _ {E}} \kappa \left(\theta_ {i}, \theta_ {j}\right). \tag {127}
$$
Assume that  $V$  is bounded from below and norm-coercive. Then  $\widetilde{L}$  is bounded from below and norm-coercive and therefore we can find a global minimiser  $\theta^{*} := (\theta_{1}^{*},\dots,\theta_{N_{E}}^{*})\in (\mathbb{R}^{J})^{N_{E}}$  of  $\widetilde{L}$ . Since  $\widetilde{L}$  is differentiable, we know that  $\theta^{*}$  is a stationary point of the gradient which implies
$$
\nabla V \left(\theta_ {i} ^ {*}\right) + \frac {\lambda}{N _ {E}} \sum_ {j = 1} ^ {N _ {E}} \left(\nabla_ {1} \kappa\right) \left(\theta_ {i} ^ {*}, \theta_ {j} ^ {*}\right) = 0 \tag {128}
$$
for all  $i = 1, \dots, N_E$ . Here, we assume that the kernel  $\kappa$  is symmetric, which is standard in the MMD literature. Note that (128) is equivalent to
$$
\nabla V (\theta) + \lambda \left(\nabla_ {1} \kappa * \widehat {Q}\right) (\theta) = 0 \tag {129}
$$
for  $\widehat{Q}$ -a.e.  $\theta \in \mathbb{R}^J$  where
$$
\widehat {Q} (d \theta) := \frac {1}{N _ {E}} \sum_ {j = 1} ^ {N _ {E}} \delta_ {\theta_ {j} ^ {*}} (d \theta). \tag {130}
$$
This means that  $\widehat{Q}$  is a stationary point of the Wasserstein gradient, and therefore an invariant measure for the McKean-Vlasov process. Since  $N_{E} \in \mathbb{N}$  was arbitrary, we have constructed countably many invariant measures and therefore uniqueness can't hold in general.
The reason that non-uniqueness of the invariant measure is an immediate contradiction to convergence is the following: If we initialise with any of the invariant measures constructed in the proof of Theorem 6, then the particle distribution of the McKean-Vlasov process will remain unchanged over time. Convergence to the global minimiser can therefore surely not hold for arbitrary initialisation  $Q_{0}$ . It may be possible to construct conditions on  $Q_{0}$  under which convergence still holds. For example, for Stein variational gradient descent a similar issue occurs. However, in this case one can guarantee convergence (Lu et al., 2019, Theorem 2.8) if  $Q_{0}$  has a Lebesgue-density (and if the kernel satisfies further restrictive assumptions). The existence of conditions that guarantee convergence for DRE remains an open problem.
G Implementation details
In Appendix A, we derived the following algorithm:
Step 1: Simulate  $N_{E}\in \mathbb{N}$  particles  $\theta_{1,0},\dots,\theta_{N_E,0}$  from a use chosen initial distribution  $Q_{0}$ .
$$
\theta_ {n, k + 1} = \theta_ {n, k} - \eta \left(\nabla V \left(\theta_ {n, k}\right) + \frac {\lambda_ {1}}{N _ {E}} \sum_ {j = 1} ^ {N _ {E}} \left(\nabla_ {1} \kappa\right) \left(\theta_ {n, k}, \theta_ {j, k}\right)\right) + \sqrt {2 \eta \lambda_ {2}} Z _ {n, k} \tag {131}
$$
$$
\text {f o r} n = 1, \dots , N _ {E}, k = 0, \dots , K - 1 \text {w i t h} Z _ {n, k} \sim \mathcal {N} (0, I _ {J \times J}).
$$
We can generate samples from DE, DLE and DRLE by setting the potential and regularisation parameters as described below:
- Deep ensembles:  $V(\theta) = \ell(\theta)$ ,  $\lambda_1 = 0$ ,  $\lambda_2 = 0$
- Deep Langevin ensembles:  $V(\theta) = \ell(\theta) - \lambda \log p(\theta)$ ,  $\lambda_1 = 0$ ,  $\lambda = \lambda_2$
- Deep repulsive Langevin ensembles:  $V(\theta) = \ell(\theta) - \lambda_1 \log p(\theta) - \lambda_2 \mu_P(\theta)$
Due to Appendix D & E, we can think of  $\theta_{1,K},\ldots,\theta_{N_E,K}$  as approximately sampled from the global minimiser  $Q^{*}$  for DLE and DRLE if  $K$  is large enough. All experiments use the SE kernel given as
$$
\kappa \left(\theta , \theta^ {\prime}\right) = \exp \left(- \frac {\| \theta - \theta^ {\prime} \| ^ {2}}{2 \sigma_ {\kappa} ^ {2}}\right) \tag {132}
$$
with lengthscale parameter  $\sigma_{\kappa} > 0$ . The kernel mean embedding  $\mu_P$  can easily be approximated as
$$
\mu_ {P} (\theta) = \frac {1}{M} \sum_ {i = 1} ^ {M} \kappa \left(\theta , \theta_ {i}\right), \theta \in \mathbb {R} ^ {J}, \tag {133}
$$
where  $\theta_{1},\ldots ,\theta_{M}\sim P$  independently. We chose  $M = 20$
G.1 Toy example: global minimiser
We describe details regarding the experiments conducted to produce Figure 2 below.
We generate  $N_{E} = 300$  particles and make the following choices:
- Loss:  $\ell(\theta) := \frac{3}{2} \left( \frac{1}{4} \theta^4 + \frac{1}{3} \theta^3 - \theta^2 \right) - \frac{3}{8}$ .
- Prior:  $P \sim  {\mathcal{N}}\left( {0,1}\right)$  and therefore  $\log p\left( \theta \right)  =  - \frac{1}{2}{\theta }^{2}$
- Initialisation:  $Q_{0} = P$
- Reg. parameter:  $\lambda_{DLE} = 1$ ,  $\lambda_{DRLE} = 1$ ,  $\lambda_{DRLE}' = 1$
- Step size:  $\eta = 10^{-4}$ , Iterations:  $K = 100,000$
- Kernel lengthscale,  $\sigma_{\kappa}$ , is chosen according to the median heuristic (Garreau et al., 2017) based on samples from the prior  $P$
The loss is constructed such that we have a global minimum at  $\theta = -2$ , a turning point at  $\theta = 0$  and a local minimum at  $\theta = 1$ .
- Deep ensembles: The optimal  $Q^{*}$  is a Dirac measure located at the global minimiser  $\theta = -2$ . However, as we proved in Theorem 1, the WGF produce samples from
$$
Q _ {\infty} (d \theta) = \frac {1}{2} \delta_ {- 2} (d \theta) + \frac {1}{2} \delta_ {1} (d \theta), \tag {134}
$$
as  $(- \infty, 0)$  is the region of attraction for the global minimum and  $(0, \infty)$  for the local minimum which both have probability 0.5 under  $Q_0 = P = \mathcal{N}(0, 1)$ . In particular,  $Q_{\infty} \neq Q^{*}$  as expected.
- Deep Langevin ensembles: The optimal measure has the pdf
$$
q ^ {*} (\theta) \propto \exp \left(- \frac {\ell (\theta)}{\lambda}\right) p (\theta) \tag {135}
$$
for  $\theta \in \mathbb{R}$ . As expected the WGF produces samples from  $Q^{*}$
- Deep repulsive Langevin ensembles: The optimal  $q^{*}$  for deep repulsive ensembles is harder to determine. From the condition that  $q^{*}$  is a stationary point of the Wasserstein gradient, we can derive that  $u(\theta) \coloneqq \log q^{*}(\theta)$  satisfies the integro-differential equation
$$
u ^ {\prime} (\theta) = - \frac {1}{\lambda_ {2}} V ^ {\prime} (\theta) - \frac {\lambda_ {1}}{\lambda_ {2}} \int \left(\nabla_ {1} \kappa\right) \left(\theta , \theta^ {\prime}\right) \exp \left(u \left(\theta^ {\prime}\right)\right) d \theta^ {\prime} \tag {136}
$$
with some initial value  $u(0) = u_0$ . In principle, we could choose  $u_0$  such that  $q(\theta) \coloneqq \exp(u(\theta))$  integrates to 1. However, since we do not know the appropriate initial condition a priori, we choose an arbitrary  $u_0$  and normalise the pdf afterwards. We use an numerical solver to evaluate  $u(\theta)$  on a fixed grid. As expected, the WGF produces samples from  $Q^*$  in this case.
G.2 Toy example: multimodal loss
The details below correspond to the experimental results presented in Figures 3 and 5. Figure 5 is an alteration of Figure 3 with only 4 particles with the goal of stressing the importance of the number of particles relative to the number of local minima.
DE, DLE, DRLE We generate  $N_{E} = 300$  particles and make the following choices:
- Loss:  $\ell(\theta) = -\log \sum_{i=1}^{4} \frac{1}{4}\mathcal{N}(\theta; \mu_i, I_2)$ ,  $\theta \in \mathbb{R}^2$ ,  $\mu_i = (\pm 3, \pm 3)^T$ ,  $i = 1, \dots, 4$
- Prior:  $P$  flat and therefore  $\log p(\theta) = 0$
- Initialisation:  $Q_0 \sim \mathcal{N}(0, I_2)$
- Reg. parameter:  $\lambda_{DLE} = 0.2$ ,  $\lambda_{DRLE} = 0.2$ ,  $\lambda_{DRLE}^{\prime} = 0.6$
- Step size:  $\eta = 0.1$ , Iterations:  $K = 10,000$
Figure 5: We generate  $N_{E} = 4$  particles from DE, DLE, DRLE and FD-GVI with Gaussian parametrisation. The multimodal loss  $\ell$  is plotted in grey and the particles of the different methods are layered on top. The prior in this example is flat, i.e.  $\log p$  and  $\mu_{P}$  are constant. The initialisation  $Q_{0}$  is standard Gaussian.
- Kernel lengthscale,  $\sigma_{\kappa}$ , is chosen according to the median heuristic (Garreau et al., 2017) based on samples from the prior  $P$
Note that for a translation-invariant kernel such as the SE kernel we obtain for the flat prior  $P$  that
$$
\begin{array}{l} \mu_ {P} (\theta) = \int_ {- \infty} ^ {\infty} \kappa \left(\theta , \theta^ {\prime}\right) d \theta^ {\prime} (137) \\ = \int_ {- \infty} ^ {\infty} \phi \left(\theta - \theta^ {\prime}\right) d \theta^ {\prime} (138) \\ = \int_ {- \infty} ^ {\infty} \phi (\xi) d \xi , (139) \\ \end{array}
$$
where the second line follows from the fact that we can write any translation-invariant kernel as  $\kappa (\theta ,\theta^{\prime}) = \phi (\theta -\theta^{\prime})$  for some function  $\phi :\mathbb{R}^{J}\to \mathbb{R}$  and the second line is simple variable substitution. If (139) is finite, the above expression is well-defined and therefore  $\mu_P$  constant. Note that in particular for the SE kernel, we have  $\phi (\xi) = \exp (-\| \xi \| ^2 /(2\sigma_\kappa^2))$  and therefore (139) is finite. As a consequence, we have that for a flat prior  $P$  the gradient of the potential  $V$  is the same for all three methods. This means that the loss  $\ell$  isn't adjusted and the only difference between the three methods is the presence of repulsion and noise effects.
Remark 3. The astute reader may have noticed that a flat prior  $P$  is in fact not covered by our theory in Appendix A. The problem is that  $\mathrm{KL}(\cdot, \mathcal{L})$ , where  $\mathcal{L}$  denotes the Lebesgue measure, is not positive (and not even bounded from below). To see this, choose  $Q = \mathcal{N}(0, \Sigma)$  with  $\Sigma = \mathrm{diag}(\sigma_1^2, \sigma_2^2)$  and note that
$$
\operatorname {K L} (Q, \mathcal {L}) = \int \log q (\theta) q (\theta) d \theta = - \mathrm {H} (Q), \tag {140}
$$
where  $\mathrm{H}(Q)$  denotes the differential entropy. For a Gaussian, it is known that
$$
\mathrm {H} (Q) = \frac {1}{2} \log \left((2 \pi e) ^ {J} \det (\Sigma)\right) = \frac {1}{2} \left(\log (2 \pi e) ^ {J} + \log \left(\sigma_ {1} ^ {2}\right) + \log \left(\sigma_ {2} ^ {2}\right)\right) \tag {141}
$$
and therefore if either  $\sigma_1^2\to \infty$  or  $\sigma_2^2\rightarrow \infty$  then  $\mathrm{KL}(Q,\mathcal{L})\to -\infty$ . However, note that this difficulty is rather technical in nature and can easily be remedied. Instead of  $\mathcal{L}$ , we could have chosen the uniform prior  $P\sim U(-10^{100},10^{100})$ . In this case, the positivity of  $\mathrm{KL}(\cdot ,P)$  is guaranteed by Jensen's inequality. This choice of  $P$  gives—up to an additive constant—the same objective as a flat prior and up to machine precision the same kernel mean embedding  $\mu_P$ . It is, therefore, algorithmically irrelevant if  $P$  is flat or uniform on a very large set.
FD-GVI We use the same prior and loss as for DE, DLE and DRLE. We parameterise the variational family as independent Gaussian, i.e.
$$
\mathcal {Q} = \left\{\mathcal {N} (\mu , \Sigma) \mid \mu \in \mathbb {R} ^ {2}, \Sigma = \operatorname {d i a g} \left(\exp \left(\beta_ {1}\right), \exp \left(\beta_ {2}\right)\right), \beta := \left(\beta_ {1}, \beta_ {2}\right) ^ {2} \in \mathbb {R} ^ {2} \right\}. \tag {142}
$$
We learn the variational parameters  $\nu \coloneqq (\mu, \beta) \in \mathbb{R}^4$  by minimising
$$
\begin{array}{l} \widetilde {L} (\nu) = \int \ell (\theta) d Q _ {\nu} (\theta) + \lambda K L (Q _ {\nu}, P) (143) \\ = \int \ell (\theta) d Q _ {\nu} (\theta) - \lambda H \left(\mathcal {N} (\mu , \Sigma)\right) (144) \\ \approx \frac {1}{2 0 0} \sum_ {j = 1} ^ {2 0 0} \ell \left(\mu + \Sigma^ {0. 5} Z _ {j}\right) - \frac {\lambda}{2} \log \left((2 \pi e) ^ {2} \exp \left(\beta_ {1}\right) \exp \left(\beta_ {2}\right)\right) (145) \\ = \frac {1}{2 0 0} \sum_ {j = 1} ^ {2 0 0} \ell \left(\mu + \Sigma^ {0. 5} Z _ {j}\right) - \frac {\lambda}{2} \left(\beta_ {1} + \beta_ {2}\right) + c o n s t, (146) \\ \end{array}
$$
where  $Z_{1},\ldots ,Z_{200}\sim \mathcal{N}(0,I_{2})$  and  $H\big(\mathcal{N}(\mu ,\Sigma)\big)$  denotes the differential entropy of the normal distribution. For the regularisation parameter, we chose  $\lambda = 0.5$
G.3 Toy example: more modes than particles
We generate  $N_{E} = 20$  particles and make the following choices:
- Loss:  $\ell(\theta) = -|\sin(\theta)|$ ,  $\theta \in [-M\pi, M\pi]$ , with  $M = 1000$
- Prior:  $P$  flat and therefore  $\log p(\theta) = 0$  and  $\mu_P = \mathrm{const}$ . (cf. Appendix G.2)
- Initialisation:  $Q_0 \sim U(-M\pi, M\pi)$
- Reg. parameter:  $\lambda_{DLE} = 0.001$ ,  $\lambda_{DRLE} = 0.001$ ,  $\lambda_{DRLE}' = 0.6$
- Step size:  $\eta = 0.01$ , Iterations:  $K = 1.000$
- Kernel lengthscale,  $\sigma_{\kappa}$ , is chosen according to the median heuristic (Garreau et al., 2017) based on samples from the prior  $P$
Note that  $\ell$  has  $2M = 2000$  local minima at locations
$$
m _ {i} := \frac {\pi}{2} + i \pi , \quad i \in \{- M, \dots , 0, \dots , (M - 1) \}. \tag {147}
$$
Due to the flat prior  $\nabla V = \nabla \ell$  for all three methods. We observe that it is hard to distinguish the methods since most particles are in their local modes by themselves.
G.4 UCI Regression
The UCI data sets are licensed under Creative Commons Attribution 4.0 International license (CC BY 4.0). Following Lakshminarayanan et al. (2017), we train 5 one-hidden-layer neural networks  $f_{\theta}$  with 50 hidden nodes for 40 epochs. We split each data set into train (81% of samples), validation (9% of samples), and test set (10% of samples). Based on the best hyperparameter runs (according to a Gaussian NLL) found via grid search on a validation data set, we make the following choices:
- Loss:  $\ell(\theta) = \frac{1}{N} \sum_{n=1}^{N} (f_{\theta}(x_n) - y_n)^2$  where  $\{x_n, y_n\}_{n=1}^N$  are paired observations.
- Prior:  $P \sim \mathcal{N}(0,1)$
- Initialisation: Kaiming intilisation, i.e. for each layer  $l \in \{1, \dots, L\}$  that maps features with dimensionality  $n_{l-1}$  into dimensionality  $n_l$ , we sample  $Q_{l,0} \sim \mathcal{N}(0,2/n_l)$
- Reg. parameter:  $\lambda_{DLE} = 10^{-4}$ ,  $\lambda_{DRLE} = 10^{-4}$ ,  $\lambda_{DRLE}' = 10^{-2}$
- Step size:  $\eta = 0.1$ , Iterations:  $K = 10,000$
- Kernel lengthscale,  $\sigma_{\kappa}$ , is chosen according to the median heuristic (Garreau et al., 2017) based on samples from the prior  $P$
G.5 Compute
While the final experimental results can be run within approximately an hour on a single GeForce RTX 3090 GPU, the complete compute needed for the final results, debugging runs, and sweeps amounts to around 9 days.