SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1 shows the decay of  $\widehat{\mathcal{F}} (\rho_{\theta})$  over time or iterations. For Figure 1a the same step size 0.01 is specified for all algorithms while for Figure 1b we choose the largest safe step size for each algorithm. In other words, Figure 1a provides the continuous-time flow of the dynamical system in each algorithm while Figure 1b emphasizes more on the discrete-time algorithmic behaviors. In Figure 1a there are roughly four distinct curves indicating four different kernels.  $K_{1}$  has the most rapid descent, followed by  $K_{2}$ ,  $K_{4}$ , and  $K_{3}$ .
From Figure 1b we observe that BWPF and RGPF are the better choices for practical use. The difference in the largest step sizes shows that in terms of stability  $K_{3}$  is the best,  $K_{4}$  is almost as stable, but  $K_{2}$  and  $K_{1}$  are much worse. We also observe that particle-based algorithms are consistently more stable than density-based counterparts (which are essentially stochastic gradient based). The superiority of particle-based algorithms are even more evident in the Gaussian mixture experiment where the target is multi-modal. We further remark that another recently proposed density-based algorithm, the FB-GVI [19] shows comparable performance to BWPF and RGPF with large step sizes. We conduct a comparison of these three algorithms in Appendix E but do not include it here for clarity. It would also be really interesting to study the particle-based analogue of FB-GVI as future works.
We thank Lester Mackey and Jiaxin Shi for several clarifications about their work, [72], and for helpful discussions regarding the larger literature on SVGD. Promit Ghosal was supported in part by NSF grant DMS-2153661. Krishnakumar Balasubramanian was supported in part by NSF grant DMS-2053918. Natesh S. Pillai was supported by ONR grant N00014-21-1-2664.
[87] Lantian Xu, Anna Korba, and Dejan Slepcev, Accurate quantization of measures via interacting particle-based optimization, International Conference on Machine Learning, PMLR, 2022, pp. 24576-24595.
[88] Rentian Yao and Yun Yang, Mean field variational inference via Wasserstein gradient flow, arXiv preprint arXiv:2207.08074 (2022).
[89] Jingwei Zhuo, Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang, Message passing Stein variational gradient descent, International Conference on Machine Learning, PMLR, 2018, pp. 6018-6027.
A Further Discussion
Other Related Works. There exist several other approaches for minimizing KL-divergence over the Wasserstein space (or over appropriately restricted subsets). For example, normalizing flows [68, 37, 11], the blob method [10], variants of gradient descent in Wasserstein spaces [69, 85, 5], natural-gradient based variational inference techniques [48], mean-field methods [88], neural networks based approaches [2] and MCMC methods. This is certainly not a comprehensive list and summarizing the large literature on this topic is impossible in the limited space available. However, we emphasize that a majority of the above methods are nonparametric and do not come with strong theoretical guarantees. Our main focus in this work is on theoretically understanding (Gaussian)-SVGD and its connection to GVI.
Beyond Gaussian VI. While we present a comprehensive study of Gaussian-SVGD, it is currently unclear how such relation between bilinear kernels and Gaussian family could be generalized. For example, it would be interesting to ask if there is a class (submanifold)  $\mathcal{C}$  of probability densities such that for any initialization-target pair  $\rho_0,\rho^*\in \mathcal{C}$ , the dynamics of SVGD with a radius-based kernel function (RBF-SVGD) would always remain in  $\mathcal{C}$ . If such  $\mathcal{C}$  is identified, we could similarly carry out the convergence analysis of RBF-SVGD and perform variational inference with respect to the class  $\mathcal{C}$ . It remains an open question whether the uniform in time propagation of chaos as shown in Theorem 3.7 would hold for general kernels. Moreover, there has been increasing interest on variational inference with respect to other special density classes. Remarkably [43] considered Gaussian mixtures in additional to GVI, and [62] analyzed the family of elliptical distributions, which has applications for performing variational inference under heavy-tails [21].
SVGD in High Dimensions. It is well-known in literature [89, 4] that RBF-SVGD suffers from severe particle degeneracy in high dimensions and cannot guarantee good covariance estimation. However, such issue has not been observed for Gaussian-SVGD in our simulations. It would be interesting future works to study whether this undesired phenomenon is tied to specific choices of kernels and to carry out theoretical analysis of (Gaussian)-SVGD in high dimensions.
B Details on Gaussian-Stein Metrics
For any region  $\Omega \subseteq \mathbb{R}^d$ , denote the set of probability densities on  $\Omega$  by
$$
\mathcal {P} (\Omega) := \left\{\rho \in \mathcal {F} (\Omega): \int_ {\Omega} \rho \mathrm {d} \boldsymbol {x} = 1, \rho \geq 0 \right\},
$$
where  $\mathcal{F}(\Omega)$  is the set of  $\mathcal{C}^\infty$ -smooth functions on  $\Omega$ . As studied in [42],  $\mathcal{P}(\Omega)$  forms a Fréchet manifold called the density manifold. For any "point"  $\rho \in \mathcal{P}(\Omega)$ , the tangent space is given by
$$
T _ {\rho} \mathcal {P} (\Omega) = \left\{\sigma \in \mathcal {F} (\Omega): \int \sigma \mathrm {d} \boldsymbol {x} = 0 \right\}.
$$
And the cotangent space at  $\rho$ ,  $T_{\rho}^{*}\mathcal{P}(\Omega)$  consists of equivalent classes of  $\mathcal{F}(\Omega)$ , each containing functions that differ by a constant. A Riemannian metric (tensor) assigns to each  $\rho \in \mathcal{P}(\Omega)$  a positive definite inner product  $g_{\rho}: T_{\rho}\mathcal{P}(\Omega) \times T_{\rho}\mathcal{P}(\Omega) \to \mathbb{R}$ . If we define the pairing between  $T_{\rho}^{*}\mathcal{P}(\Omega)$  and  $T_{\rho}\mathcal{P}(\Omega)$  by
$$
\langle \Phi , \sigma \rangle := \int \Phi \cdot \sigma d \mathbf {x},
$$
where  $\Phi \in T_{\rho}^{*}\mathcal{P}(\Omega)$  and  $\sigma \in T_{\rho}\mathcal{P}(\Omega)$ . Any Riemannian metric uniquely corresponds to an isomorphism (called the canonical isomorphism) between the tangent and cotangent bundles [66], i.e., we have  $G_{\rho}:T_{\rho}\mathcal{P}(\Omega)\to T_{\rho}^{*}\mathcal{P}(\Omega)$  through
$$
g _ {\rho} \left(\sigma_ {1}, \sigma_ {2}\right) = \left\langle G _ {\rho} \sigma_ {1}, \sigma_ {2} \right\rangle = \left\langle G _ {\rho} \sigma_ {2}, \sigma_ {1} \right\rangle , \quad \sigma_ {1}, \sigma_ {2} \in T _ {\rho} \mathcal {P} (\Omega).
$$
Definition B.1 (Wasserstein metric). The Wasserstein metric is induced by the following canonical isomorphism  $G_{\rho}^{\mathrm{Wass}}: T_{\rho} \mathcal{P}(\Omega) \to T_{\rho}^{*} \mathcal{P}(\Omega)$  such that
$$
(G _ {\rho} ^ {\mathrm {W a s s}}) ^ {- 1} \Phi = - \nabla \cdot (\rho \nabla \Phi), \quad \Phi \in T _ {\rho} ^ {*} \mathcal {P} (\Omega).
$$
Note that the Wasserstein metric we define here corresponds exactly to the Wasserstein-2 distance studied in the literature of optimal transport [83]. For details of such connection, please refer to [36]. Also we need to clarify that the concepts we introduce follow from the convention in Riemannian geometry, where the manifold strictly speaking is required to be finite-dimensional. For
a mathematically formal formulation of infinite-dimensional calculus on the density manifold, please refer to [42, 41, 65, 57].
The Wasserstein gradient flow can be seen as the natural gradient flow on the density manifold with respect to the Wasserstein metric. In specific, consider any energy functional  $E(\rho)$ , e.g., the KL divergence from  $\rho$  to some fixed target  $\rho^*$ . The Wasserstein gradient flow for  $E(\rho)$  is provided by
$$
\dot {\rho} _ {t} = - \left(G _ {\rho_ {t}} ^ {\text {W a s s}}\right) ^ {- 1} \frac {\delta E}{\delta \rho_ {t}} = \nabla \cdot \left(\rho_ {t} \nabla \frac {\delta E}{\delta \rho_ {t}}\right), \tag {27}
$$
where  $\frac{\delta E}{\delta\rho_t}$  is the variational derivative of the functional  $E$  with respect to  $\rho_{t}$ . If  $E(\rho)$  is the KL divergence mentioned above, (27) gives the linear Fokker-Planck equation [33]
$$
\dot {\rho} _ {t} = \nabla \cdot (\nabla \rho_ {t} + \rho_ {t} \nabla V),
$$
where  $V(\pmb{x}) = -\log \rho^{*}(\pmb{x})$  is called the potential function. If  $E(\rho)$  is the Wasserstein metric between  $\rho$  and  $\rho^{*}$ , (1) gives the geodesic flow on the density manifold. If  $E(\rho)$  is the maximum mean discrepancy (MMD) or kernelized Stein discrepancy (KSD) between  $\rho$  and  $\rho^{*}$ , it leads to the MMD descent and KSD descent algorithms [3, 38].
Interestingly, the mean field flow of SVGD can also be seen as the gradient flows of the KL divergence on the density manifold but with respect to the Stein metric, where we perform kernelization in the cotangent space before taking the divergence [51, 22].
Definition B.2 (Stein metric). The Stein metric is induced by the following canonical isomorphism  $G_{\rho}^{\mathrm{Wass}}: T_{\rho} \mathcal{P}(\Omega) \to T_{\rho}^{*} \mathcal{P}(\Omega)$  such that
$$
(G _ {\rho} ^ {\mathrm {S t e i n}}) ^ {- 1} \Phi = - \nabla \cdot \left(\rho (\cdot) \int K (\cdot , \boldsymbol {y}) \rho (\boldsymbol {y}) \nabla \Phi (\boldsymbol {y}) \mathrm {d} \boldsymbol {y}\right), \quad \Phi \in T _ {\rho} ^ {*} \mathcal {P} (\Omega).
$$
$$
\begin{array}{l} \dot {\rho} _ {t} (\boldsymbol {x}) = - \left(G _ {\rho} ^ {\text {S t e i n}}\right) ^ {- 1} \frac {\delta \operatorname {K L} \left(\rho_ {t} \| \rho^ {*}\right)}{\delta \rho_ {t}} (\boldsymbol {x}) \tag {28} \\ = \nabla \cdot \left(\rho_ {t} (\boldsymbol {x}) \int K (\boldsymbol {x}, \boldsymbol {y}) \rho_ {t} (\boldsymbol {y}) \nabla \frac {\delta \operatorname {K L} \left(\rho_ {t} \| \rho^ {*}\right)}{\delta \rho_ {t}} (\boldsymbol {y}) \mathrm {d} \boldsymbol {y}\right) \\ = \nabla \cdot \left(\rho_ {t} (\boldsymbol {x}) \int K (\boldsymbol {x}, \boldsymbol {y}) \rho_ {t} (\boldsymbol {y}) \nabla \left(\log \rho_ {t} (\boldsymbol {y}) + V + 1\right) \mathrm {d} \boldsymbol {y}\right) \\ = \nabla \cdot \left(\rho_ {t} (\boldsymbol {x}) \int K (\boldsymbol {x}, \boldsymbol {y}) \left(\nabla \rho_ {t} (\boldsymbol {y}) + \rho_ {t} (\boldsymbol {y}) \nabla V (\boldsymbol {y})\right) d \boldsymbol {y}\right), \\ \end{array}
$$
where  $V(\pmb {x}) = -\log \rho^{*}(\pmb {x})$
We set  $\Omega = \mathbb{R}^d$  and consider the multivariate Gaussian densities  $\mathcal{N}(\boldsymbol{\mu},\Sigma)$  in  $\mathbb{R}^d$ :
$$
\rho (\boldsymbol {x}, \theta) = \frac {1}{\sqrt {\det  (2 \pi \Sigma)}} \exp \left(- \frac {1}{2} (\boldsymbol {x} - \boldsymbol {\mu}) ^ {\top} \Sigma^ {- 1} (\boldsymbol {x} - \boldsymbol {\mu})\right),
$$
where  $\theta := (\pmb{\mu}, \Sigma) \in \Theta$  and  $\Theta := \mathbb{R}^d \times \mathrm{Sym}^+(d, \mathbb{R})$ . Here  $\mathrm{Sym}^+(d, \mathbb{R})$  is the set of (symmetric) positive definite  $d \times d$  matrices, which is an open subset of the  $d \times d$  symmetric matrix space  $\mathrm{Sym}(d, \mathbb{R})$ . In this way,  $\Theta$  can be identified as a Riemannian submanifold of the density manifold  $\mathcal{P}(\mathbb{R}^d)$  with the induced Riemannian structure.
We first look into the Stein metric with the bilinear kernel  $K_{1}(\pmb{x},\pmb{y}) = \pmb{x}^{\top}\pmb{y} + 1$ , and derive the closed form of the induced metric tensor on  $\Theta$ , which plays an essential role in characterizing the SVGD dynamics.
Theorem B.3 (Gaussian-Stein metric with the simple bilinear kernel). Given  $\theta = (\pmb{\mu}, \pmb{\Sigma}) \in \Theta$ , let  $g_{\theta}(\cdot, \cdot)$  denote the Gaussian-Stein metric tensor for the multivariate Gaussian model with the bilinear kernel  $K_{1}(\pmb{x}, \pmb{y}) = \pmb{x}^{\top} \pmb{y} + 1$ , and  $G_{\theta}$  be the corresponding canonical isomorphism from  $T_{\theta} \Theta \simeq \mathbb{R}^{d} \times \operatorname{Sym}(d, \mathbb{R})$  to  $T_{\theta}^{*} \Theta \simeq \mathbb{R}^{d} \times \operatorname{Sym}(d, \mathbb{R})$ .
For any  $\nu \in \mathbb{R}^n$ , and  $S \in \operatorname{Sym}(d, \mathbb{R})$ , the inverse map  $G_{\theta}^{-1}$  is given by the following automorphism on  $\mathbb{R}^d \times \operatorname{Sym}(d, \mathbb{R})$ :
$$
G _ {\theta} ^ {- 1} (\boldsymbol {\nu}, S) = \left(2 S \Sigma \boldsymbol {\mu} + \left(1 + \boldsymbol {\mu} ^ {\top} \boldsymbol {\mu}\right) \boldsymbol {\nu}, \left(\Sigma \left(2 \Sigma S + \boldsymbol {\mu} \boldsymbol {\nu} ^ {\top}\right) + \left(2 S \Sigma + \boldsymbol {\nu} \boldsymbol {\mu} ^ {\top}\right) \Sigma\right)\right). \tag {29}
$$
And for any  $\xi, \eta \in T_{\theta} \Theta$  the Stein metric tensor can be written as
$$
g _ {\theta} (\xi , \eta) = \operatorname {t r} \left(S _ {1} S _ {2} \Sigma^ {2}\right) + \left(\boldsymbol {b} _ {1} ^ {\top} S _ {2} + \boldsymbol {b} _ {2} ^ {\top} S _ {1}\right) \Sigma \boldsymbol {\mu} + (1 + \boldsymbol {\mu} ^ {\top} \boldsymbol {\mu}) \boldsymbol {b} _ {1} ^ {\top} \boldsymbol {b} _ {2}. \tag {30}
$$
Here  $\xi = \left(\widetilde{\mu}_1,\widetilde{\Sigma}_1\right)$  and  $\eta = \left(\widetilde{\mu}_2,\widetilde{\Sigma}_2\right)$ , in which  $\widetilde{\mu}_1,\widetilde{\mu}_2\in \mathbb{R}^d,\widetilde{\Sigma}_1,\widetilde{\Sigma}_2\in \mathrm{Sym}(d,\mathbb{R})$ . And  $\pmb {b}_i,S_i$ ’s  $(i = 1,2)$  are defined as
$$
\left(\boldsymbol {b} _ {i}, \frac {1}{2} S _ {i}\right) = G _ {\theta} (\widetilde {\boldsymbol {\mu}} _ {i}, \widetilde {\Sigma} _ {i}).
$$
Then for the Gaussian-Stein metric with kernel  $K_{2}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}(\pmb{y} - \pmb{\mu}) + 1$  we have the following result:
Theorem B.4 (Gaussian-Stein metric with the affine-invariant bilinear kernel). Given  $\theta = (\pmb{\mu},\Sigma) \in \Theta$ , let  $g_{\theta}(\cdot,\cdot)$  denote the affine-invariant Gaussian-Stein metric tensor for the multivariate Gaussian model with the affine-invariant bilinear kernel  $K_{2}$ , and  $G_{\theta}$  be the corresponding canonical isomorphism from  $T_{\theta}\Theta \simeq \mathbb{R}^{d} \times \operatorname{Sym}(d,\mathbb{R})$  to  $T_{\theta}^{*}\Theta \simeq \mathbb{R}^{d} \times \operatorname{Sym}(d,\mathbb{R})$ .
For any  $\pmb{\nu} \in \mathbb{R}^n$ , and  $S \in \operatorname{Sym}(d, \mathbb{R})$ , the inverse map  $G_{\theta}^{-1}$  is given by the following automorphism on  $\mathbb{R}^d \times \operatorname{Sym}(d, \mathbb{R})$ :
$$
G _ {\theta} ^ {- 1} (\nu , S) = \left(\nu , 2 \left(\Sigma^ {2} S + S \Sigma^ {2}\right)\right). \tag {31}
$$
$$
g _ {\theta} (\xi , \eta) = \operatorname {t r} \left(S _ {1} S _ {2} \Sigma^ {2}\right) + \widetilde {\boldsymbol {\mu}} _ {1} ^ {\top} \widetilde {\boldsymbol {\mu}} _ {2}. \tag {32}
$$
Here  $\xi = \left(\widetilde{\mu}_1,\widetilde{\Sigma}_1\right)$  and  $\eta = \left(\widetilde{\mu}_2,\widetilde{\Sigma}_2\right)$ , in which  $\widetilde{\mu}_1,\widetilde{\mu}_2\in \mathbb{R}^d,\widetilde{\Sigma}_1,\widetilde{\Sigma}_2\in \mathrm{Sym}(d,\mathbb{R})$ . And  $S_{i}$ 's  $(i = 1,2)$  are defined as the symmetric solution of
$$
\widetilde {\Sigma} _ {i} = \Sigma^ {2} S _ {i} + S _ {i} \Sigma^ {2}.
$$
Now if we further restrict the Gaussian family to have zero mean, it induces the submanifold  $\Theta_0 = \operatorname{Sym}^+(d, \mathbb{R})$ . We have the following result for  $K_1$  or  $K_2$  as a direct corollary of Theorem B.3 or Theorem B.4.
Corollary B.5. Given  $\Sigma \in \Theta_0$ , let  $g_{\Sigma}(\cdot, \cdot)$  denote the Gaussian-Stein metric tensor for the centered Gaussian family with the bilinear kernel  $K(\boldsymbol{x}, \boldsymbol{y}) = \boldsymbol{x}^{\top} \boldsymbol{y} + 1$ , and  $G_{\Sigma}$  be the corresponding canonical isomorphism from  $T_{\Sigma} \Theta_0 \simeq \operatorname{Sym}(d, \mathbb{R})$  to  $T_{\Sigma}^{*} \Theta_{0} \simeq \operatorname{Sym}(d, \mathbb{R})$ .
For any  $S \in \operatorname{Sym}(d, \mathbb{R})$ , the inverse map  $G_{\Sigma}^{-1}$  is given by the following automorphism on  $\operatorname{Sym}(d, \mathbb{R})$ :
$$
G _ {\Sigma} ^ {- 1} (S) = 2 \left(\Sigma^ {2} S + S \Sigma^ {2}\right). \tag {33}
$$
And for any  $\widetilde{\Sigma}_1, \widetilde{\Sigma}_2 \in T_{\Sigma} \Theta_0$  the Stein metric tensor can be written as
$$
g _ {\Sigma} \left(\widetilde {\Sigma} _ {1}, \widetilde {\Sigma} _ {2}\right) = \operatorname {t r} \left(S _ {1} S _ {2} \Sigma^ {2}\right), \tag {34}
$$
where for  $i = 1,2$ ,  $S_{i}$  is the unique solution in  $\operatorname{Sym}(d,\mathbb{R})$  that satisfies the Lyapunov equation
Next we consider the Bures-Wasserstein metric for Gaussian families, which is defined as the Wasserstein metric restricted to the Gaussian family. It has the following elegant expressions from [77, 60]:
Theorem B.6 (Bures-Wasserstein metric). Given  $\theta = (\mu, \Sigma) \in \Theta$ , let  $g_{\theta}(\cdot, \cdot)$  denote the Bures-Wasserstein metric tensor for the multivariate Gaussian model (or equivalently Gaussian-Stein metric with the kernel  $K_3$ ), and  $G_{\theta}$  be the corresponding isomorphism from  $T_{\theta} \Theta \simeq \mathbb{R}^d \times \operatorname{Sym}(d, \mathbb{R})$  to  $T_{\theta}^{*} \Theta \simeq \mathbb{R}^{d} \times \operatorname{Sym}(d, \mathbb{R})$ . For any  $\pmb{\nu} \in \mathbb{R}^n$ , and  $S \in \operatorname{Sym}(d, \mathbb{R})$ , the inverse map  $G_{\theta}^{-1}$  is given by the following automorphism on  $\mathbb{R}^d \times \operatorname{Sym}(d, \mathbb{R})$ :
$$
\left(G _ {\theta} ^ {\text {W a s s}}\right) ^ {- 1} (\boldsymbol {\nu}, S) = \left(\boldsymbol {\nu}, 2 \left(\Sigma S + S \Sigma\right)\right). \tag {35}
$$
And for any  $\xi, \eta \in T_{\theta} \Theta$  the Bures-Wasserstein metric tensor can be written as
$$
g _ {\theta} (\xi , \eta) = \operatorname {t r} \left(S _ {1} S _ {2} \Sigma\right) + \widetilde {\boldsymbol {\mu}} _ {1} ^ {\top} \widetilde {\boldsymbol {\mu}} _ {2}. \tag {36}
$$
$$
\widetilde {\Sigma} _ {i} = \Sigma S _ {i} + S _ {i} \Sigma .
$$
Notably this metric coincides with Gaussian-Stein metric with the kernel  $K_{3}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}\Sigma^{-1}(\pmb{y} - \pmb{\mu}) + 1$ .
Note that the only difference between (32) and (36) is the power of  $\Sigma$ .
Theorem B.7 (Regularized Gaussian-Stein metric with the affine-invariant bilinear kernel). Given  $\theta = (\pmb {\mu},\Sigma)\in \Theta$  , let  $g_{\theta}(\cdot ,\cdot)$  denote the affine-invariant regularized Gaussian-Stein metric tensor for Gaussian families, and  $G_{\theta}$  be the corresponding isomorphism from  $T_{\theta}\Theta \simeq \mathbb{R}^{d}\times \operatorname {Sym}(d,\mathbb{R})$  to  $T_{\theta}^{*}\Theta \simeq \mathbb{R}^{d}\operatorname {Sym}(d,\mathbb{R})$  . For any  $\nu \in \mathbb{R}^d$  and  $S\in \mathrm{Sym}(d,\mathbb{R})$  , the inverse map  $G_{\theta}^{-1}$  is given by the following automorphism on  $\mathbb{R}^d\times \operatorname {Sym}(d,\mathbb{R})$  ..
$$
\left(G _ {\theta} ^ {R S}\right) ^ {- 1} (\boldsymbol {\nu}, S) = \left(\boldsymbol {\nu}, 2 \left(\left((1 - \nu) \Sigma + \nu I\right) ^ {- 1} \Sigma^ {2} S + S \left((1 - \nu) \Sigma + \nu I\right) ^ {- 1} \Sigma^ {2}\right)\right). \tag {37}
$$
Notably this metric coincides with Gaussian-Stein metric with the kernel  $K_{4}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}((1 - \nu)\Sigma + \nu I)^{-1}(\pmb{y} - \pmb{\mu}) + 1$ .
C Properties of SVGD Solutions with the Simple Bilinear Kernel
[58] showed a few nice properties of the mean field PDE (2) for SVGD with radius-based kernels that can be written as  $K(\pmb{x} - \pmb{y})$  which is symmetric and positive definite, meaning that
$$
\sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {m} K (\boldsymbol {x} _ {i} - \boldsymbol {x} _ {j}) \xi_ {i} \xi_ {j} \geq 0, \quad \forall \boldsymbol {x} _ {i} \in \mathbb {R} ^ {d}, \xi_ {i} \in \mathbb {R}, m \in \mathbb {N}.
$$
Although their results covered a large class of kernels commonly used in practice e.g., Gaussian kernels, they do not apply to the bilinear kernel  $K(\pmb{x}, \pmb{y}) = \pmb{x}^{\top} \pmb{y} + 1$ . In this subsection we establish similar results for the bilinear kernel. In fact, some of their results for radius-based kernels still hold here while some do not.
Before showing the properties of the mean field PDE, we need the following assumption on the potential function  $V$ .
Assumption C.1. The potential function  $V:\mathbb{R}^d\to \mathbb{R}$  satisfies the conditions below:
1.  $V\in \mathcal{C}^{\infty}(\mathbb{R}^{d}),V\geq 0$  , and  $V(\pmb {x})\to \infty$  as  $\| \pmb {x}\| \to \infty$
2. For any  $\alpha, \beta > 0$ , there exists a constant  $C_{\alpha, \beta} > 0$  such that if  $\| \pmb{y} \| \leq \alpha \| \pmb{x} \| + \beta$ , then the following inequality always holds that
$$
(1 + \| \boldsymbol {x} \|) \| \nabla V (\boldsymbol {y}) \| + (1 + \| \boldsymbol {x} \|) ^ {2} \| \nabla^ {2} V (\boldsymbol {y}) \| \leq C _ {\alpha , \beta} (1 + V (\boldsymbol {x})).
$$
To make things precise although all vector norms in  $\mathbb{R}^d$  and all matrix norms are equivalent, we always choose the Euclidean norm for vectors and the spectral norm for matrices unless otherwise specified. Note that our assumption here is a little different from Assumption 2.1 in [58]. We do not require their second formula but our second piece of assumption is slightly stricter than their third one. It is straightforward to check that any positive definite quadratic form satisfies all the assumptions above, corresponding to the case where the target is a non-degenerate Gaussian distribution.
We use  $\mathcal{P}_V(\mathbb{R}^d)$  and  $\mathcal{P}^p (\mathbb{R}^d)$ $(p = 1,2,\dots)$  to denote the set of probability measure  $\mu$  on  $\mathbb{R}^d$  satisfying
$$
\| \mu \| _ {\mathcal {P} _ {V}} := \int_ {\mathbb {R} ^ {d}} (1 + V (\boldsymbol {x})) \mathrm {d} \mu (\boldsymbol {x}) <   \infty \tag {38}
$$
$$
\| \mu \| _ {\mathcal {P} ^ {p}} := \int_ {\mathbb {R} ^ {d}} \| \boldsymbol {x} \| ^ {p} \mathrm {d} \mu (\boldsymbol {x}) <   \infty \tag {39}
$$
Theorem C.2 (Well-posedness and regularity of the mean field solution). Let  $V$  satisfy Assumption C.1. For any  $\nu \in \mathcal{P}_V(\mathbb{R}^d)$ , there is a unique  $\rho_t \in \mathcal{C}\big([0,\infty),\mathcal{P}_V(\mathbb{R}^d)\big)$  which is a weak solution to (2) with initial condition  $\rho_0 = \nu$ . Moreover, there exists  $C_1 > 0$  depending on  $V$  such that
$$
\left\| \rho_ {t} \right\| _ {\mathcal {P} _ {V}} \leq e ^ {C _ {1} t} \| \nu \| _ {\mathcal {P} _ {V}} \quad t \geq 0. \tag {40}
$$
If  $\nu \in \mathcal{P}^p (\mathbb{R}^d)\cap \mathcal{P}_V(\mathbb{R}^d)$ , then for any  $t\in [0,\infty)$ , we have that  $\rho_t\in \mathcal{P}^p (\mathbb{R}^d)\cap \mathcal{P}_V(\mathbb{R}^d)$  and there exists  $C_2 > 0$  depending on  $V$  such that
$$
\left\| \rho_ {t} \right\| _ {\mathcal {P} ^ {p}} \leq e ^ {C _ {2} t} \| \nu \| _ {\mathcal {P} ^ {p}} \quad t \geq 0. \tag {41}
$$
If  $\nu$  has a density  $\rho_0(\pmb{x})\geq 0$ , then  $\rho_{t}$  also has a density. Furthermore, if  $\rho_0\in \mathcal{H}^k (\mathbb{R}^d)$  for some  $k$ , then we have  $\rho_t\in \mathcal{H}^k (\mathbb{R}^d)$ . Here
$$
\mathcal {H} ^ {k} (\mathbb {R} ^ {d}) = \mathcal {W} ^ {k, 2} (\mathbb {R} ^ {d}) := \left\{u \in \mathcal {L} ^ {p} (\mathbb {R} ^ {d}): D ^ {\alpha} u \in \mathcal {L} ^ {p} (\mathbb {R} ^ {d}) \forall | \alpha | \leq k \right\}, \quad k \geq 1
$$
denotes the Sobolev (Hilbert) space of order  $k$ .
Theorem C.3 (Well-posedness of the finite-particle solution). Let  $V$  satisfy Assumption C.1. Then for any initial condition  $X_0 = \left(\boldsymbol{x}_1^{(0)},\dots ,\boldsymbol{x}_N^{(0)}\right)^\top \in \mathbb{R}^{dN}$ , the system (13) has a unique solution
$$
X _ {t} = \left(\boldsymbol {x} _ {1} ^ {(t)}, \dots , \boldsymbol {x} _ {N} ^ {(t)}\right) ^ {\top} \in \mathcal {C} ^ {1} \left([ 0, \infty), \mathbb {R} ^ {d N}\right),
$$
and the measure  $\mu_t^N = \frac{1}{N}\sum_{i=1}^N \delta_{\pmb{x}_i^{(t)}}$  is a weak solution to the PDE (2).
Finally, we show that if two initial probability measures are close to each other, the solutions of (2) up to time  $T$  are also close. We need to further impose the following assumption on  $V$ .