SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption C.4. There exists a constant  $C_V > 0$  and some index  $q > 1$  such that
$$
\| \nabla V (\boldsymbol {x}) \| ^ {q} \leq C _ {V} (1 + V (\boldsymbol {x})) \quad f o r e v e r y \boldsymbol {x} \in \mathbb {R} ^ {d} \tag {42}
$$
$$
\sup  _ {\theta \in [ 0, 1 ]} \| \nabla^ {2} V (\theta \boldsymbol {x} + (1 - \theta) \boldsymbol {y}) \| ^ {q} \leq C _ {V} \left(1 + \frac {V (\boldsymbol {x}) + V (\boldsymbol {y})}{(\| \boldsymbol {x} \| + \| \boldsymbol {y} \|) ^ {q}}\right). \tag {43}
$$
Remarkably a Gaussian distribution satisfies both Assumptions C.1 and C.4. Secondly an important observation is that (42) implies that there is  $C_0$  such that
$$
V (\boldsymbol {x}) \leq C _ {0} \left(1 + \| \boldsymbol {x} \| ^ {q ^ {\prime}}\right) \quad \forall \boldsymbol {x} \in \mathbb {R} ^ {d} \tag {44}
$$
where  $q^{\prime} = q / (q - 1)$  . Indeed, we note that
$$
\partial_ {t} \left(1 + V \left(t \frac {\boldsymbol {x}}{\| \boldsymbol {x} \|}\right)\right) ^ {\frac {q - 1}{q}} = \frac {q - 1}{q} \cdot \frac {\frac {\boldsymbol {x}}{\| \boldsymbol {x} \|} \cdot \nabla V \left(t \frac {\boldsymbol {x}}{\| \boldsymbol {x} \|}\right)}{\left(1 + V \left(t \frac {\boldsymbol {x}}{\| \boldsymbol {x} \|}\right)\right) ^ {1 / q}} \leq \left(1 - \frac {1}{q}\right) C _ {V} ^ {1 / q}.
$$
Integrating from  $t = 0$  to  $t = \| x\|$ , we get (44), which shows that  $\mathcal{P}^p\subset \mathcal{P}_V$  for any  $p\geq q' = q / (q - 1)$ .
Theorem C.5. Let  $V$  satisfy Assumptions C.1 and C.4. Let  $R > 0$ . Assume that  $\nu_{1}, \nu_{2}$  are two initial probability measures in  $\mathcal{P}^p(\mathbb{R}^d)$  satisfying  $\| \nu_i \|_{\mathcal{P}^p} \leq R$  ( $i = 1, 2$ ). Let  $\mu_{1,t}$  and  $\mu_{2,t}$  be the associated weak solutions to (2). Then given any  $T > 0$ , there exists a constant  $C_T > 0$  depending on  $V, R$  and  $T$  such that
$$
\sup  _ {t \in [ 0, T ]} \mathcal {W} _ {p} \left(\mu_ {1, t}, \mu_ {2, t}\right) \leq C _ {T} \mathcal {W} _ {p} \left(\nu_ {1}, \nu_ {2}\right).
$$
Remark. 1. Theorem C.5 implies the convergence of empirical measure to the mean field limit at time  $t \in [0, T]$ . In fact, if we set  $\nu_{1,n}$  to be an empirical measure, which converges to  $\nu_2$  as  $n$  grows to infinity, then  $\mu_{1,n,t}$ , the solution of (2) at time  $t$  with initialization  $\nu_{1,n}$ , will also converge to  $\mu_{2,t}$  for any  $t \in [0, T]$ .
2. In general there is no guarantee that the density  $\rho_{t}$  will converge to the target density  $\rho^{*}$  in (2) with the bilinear kernel. Some counterexamples will be elaborated in the remarks following Theorem 3.6.
3. We show in Theorem 3.1 that for Gaussian families  $\rho_t$  always converges to the target density  $\rho^*$  as  $t\to \infty$  and the convergence rate is linear in KL divergence. We also establish a uniform in time convergence result (Theorem 3.7) for the empirical measure in this case.
D Details of the Gaussian-SVGD Algorithms
Different Ways of Estimating  $\Gamma_{t}$ . The first-order estimator of  $\Gamma_{t}$  arises from
$$
\begin{array}{l} \Gamma_ {t} = \mathbb {E} _ {\rho_ {\theta}} [ \nabla^ {2} V (\boldsymbol {x}) ] = \int \rho_ {\theta} (\boldsymbol {x}) \nabla^ {2} V (\boldsymbol {x}) d \boldsymbol {x} = - \int \nabla \rho_ {\theta} (\boldsymbol {x}) \nabla V (\boldsymbol {x}) ^ {\top} d \boldsymbol {x} \\ = \int \rho_ {\theta} (\boldsymbol {x}) \Sigma^ {- 1} (\boldsymbol {x} - \boldsymbol {\mu}) \nabla V (\boldsymbol {x}) ^ {\top} d \boldsymbol {x} = \Sigma^ {- 1} \mathbb {E} _ {\rho_ {\theta}} [ (\boldsymbol {x} - \boldsymbol {\mu}) \nabla V (\boldsymbol {x}) ^ {\top} ]. \\ \end{array}
$$
Since  $\Gamma_t$  is symmetric we also have
$$
\Gamma_ {t} = \mathbb {E} _ {\rho_ {\theta}} [ \nabla V (\boldsymbol {x}) (\boldsymbol {x} - \boldsymbol {\mu}) ^ {\top} ] \Sigma^ {- 1}.
$$
Note that using the first-order estimator also comes at a cost as the inverse of  $\Sigma$  is needed. However, for density-based Gaussian-SVGD this  $\Sigma^{-1}$  might cancel with  $\Sigma$  in computing.
Previous Algorithms Under the Proposed Frameworks. The use of  $K_{1}$  for SVGD in variational inference dates back to [54]. Our Algorithm 2 is slightly different from [54] in the sense that  $\nabla V$  is replaced by a linear function to ensure Gaussianity. Moreover, Algorithms 1 and 2 with the kernel  $K_{2}$  correspond precisely to the GF and GPF algorithms in [27]. If  $K_{3}$  (Bures-Wasserstein metric) is chosen, Algorithm 1 reproduces the BW-SGD algorithm in [43] (with  $N = 1$ ). [19] also uses  $K_{3}$  (Bures-Wasserstein metric) but their energy function for gradient flow is different from others. Instead of directly performing the gradient descent to minimize KL divergence, they separate the KL divergence into two parts and perform the proximal gradient descent.
Variants of Gradient Descent in Density-Based Gaussian-SVGD. For the density-based SVGD, we draw new samples at each step. It is interesting to study the behavioral difference between drawing one sample (stochastic) and efficiently many samples (almost deterministic). For example, in [43] only one sample is drawn at each time step and they study the stochastic properties arising from this design. [19] considers both settings. In general, they do not differ much in convergence rates but there could be huge gaps in the constants of the bounds and will actually impact practical performance. Another choice is to only draw  $N$  samples at time 0 and we use a linear transformation of the same  $N$  points to serve as new samples at time  $t$ , which becomes similar to the particle-based Gaussian-SVGD. Moreover, the vanilla gradient descent could also be replaced by accelerated ones or gradient descent with adaptive learning rate, e.g., AdaGrad, RMSProp.
Resampling Scheme and Particle-Level Convergence. As presented in the main text the Gaussian-SVGD for a general target is given by
$$
\boldsymbol {x} _ {i} ^ {(t + 1)} = \boldsymbol {x} _ {i} ^ {(t)} + \frac {\epsilon}{N} \left(\sum_ {j = 1} ^ {N} \nabla_ {\boldsymbol {x} _ {j} ^ {(t)}} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) - \sum_ {j = 1} ^ {N} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) \widehat {\nabla V} \left(\boldsymbol {x} _ {j} ^ {(t)}\right)\right). \tag {45}
$$
where  $\widehat{\nabla V}(\pmb{x}) = \widehat{\Gamma}_t(\pmb{x} - \pmb{\mu}_t) + \widehat{\pmb{m}}_t$ .
The updating rules of Gaussian-SVGD given above is totally deterministic, meaning that at each time step  $\boldsymbol{x}_i^{(t)}$  is updated only using deterministic quantities and  $\left(\boldsymbol{x}_k^{(t)}\right)_{k=1}^N$  without any external randomness. This is computationally more efficient but imposes difficulty in the analysis. On the other hand, we could also consider slightly modifying the updating rules by applying a resampling scheme to get a better estimation of  $\boldsymbol{m}_t$  and  $\Gamma_t$ . In other words, we resample  $(\boldsymbol{y}_k)_{k=1}^M$  i.i.d. from  $\mathcal{N}(\boldsymbol{\mu}_t, \Sigma_t)$  and use the following estimators
$$
\widehat {\boldsymbol {m}} _ {t} = \frac {1}{M} \sum_ {k = 1} ^ {M} \nabla V \left(\boldsymbol {y} _ {k} ^ {(t)}\right), \quad \widehat {\Gamma} _ {t} = \frac {1}{M} \sum_ {k = 1} ^ {M} \nabla^ {2} V \left(\boldsymbol {y} _ {k} ^ {(t)}\right), \tag {46}
$$
or first-order estimators
$$
\widehat {\Gamma} _ {t} = \frac {1}{M} \sum_ {k = 1} ^ {M} \nabla V \left(\boldsymbol {y} _ {k} ^ {(t)}\right) \left(\boldsymbol {y} _ {k} ^ {(t)} - \boldsymbol {\mu} _ {t}\right) ^ {\top} \Sigma_ {t} ^ {- 1} = \frac {1}{M} \sum_ {k = 1} ^ {M} \Sigma_ {t} ^ {- 1} \left(\boldsymbol {y} _ {k} ^ {(t)} - \boldsymbol {\mu} _ {t}\right) \nabla V \left(\boldsymbol {y} _ {k} ^ {(t)}\right) ^ {\top}. \tag {47}
$$
In this way, when  $M$  is large enough  $\widehat{\nabla V} (\pmb {x})$  will be sufficiently close to  $\Gamma_t(\pmb {x} - \pmb {\mu}_t) + \pmb {m}_t$
Figure 2: Convergence of Algorithms 1 and 2 with bilinear kernels for a Gaussian target.
Now we replace  $\widehat{\pmb{m}}_t$  and  $\widehat{\Gamma}_t$  by  $\pmb{m}_t$  and  $\Gamma_t$  and consider the continuous-time dynamics
$$
\dot {\boldsymbol {x}} _ {i} ^ {(t)} = \frac {1}{N} \left(\sum_ {j = 1} ^ {N} \nabla_ {\boldsymbol {x} _ {j} ^ {(t)}} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) - \sum_ {j = 1} ^ {N} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) \left(\Gamma_ {t} \left(\boldsymbol {x} _ {j} ^ {(t)} - \boldsymbol {\mu} _ {t}\right) + \boldsymbol {m} _ {t}\right)\right), \tag {48}
$$
where  $\pmb{\mu}_t$  and  $\Sigma_t$  are sample mean and variance and
$$
\boldsymbol {m} _ {t} := \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}\right)} [ \nabla V (\boldsymbol {x}) ], \quad \Gamma_ {t} := \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {N} \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}\right)} [ \nabla^ {2} V (\boldsymbol {x}) ].
$$
Theorem D.1 (Equivalence of density-based and particle-based algorithms). The solution of the finite-particle system (48) with  $K_{1}$  is given by  $\pmb{x}_{i}^{(t)} = A_{t}(\pmb{x}_{i}^{(0)} - \pmb{\mu}_{0}) + \pmb{\mu}_{t}$  where  $A_{t}$  is the unique solution of
$$
\dot {A} _ {t} = \left(I - \Gamma_ {t} C _ {t} - \boldsymbol {m} _ {t} \boldsymbol {\mu} _ {t} ^ {\top}\right) A _ {t}, \quad A _ {0} = I,
$$
where  $\pmb{\mu}_t$  and  $\Sigma_t$  are the unique solution of the ODE system
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = \left(I - \Gamma_ {t} \Sigma_ {t}\right) \boldsymbol {\mu} _ {t} - \left(1 + \boldsymbol {\mu} _ {t} ^ {\top} \boldsymbol {\mu} _ {t}\right) \boldsymbol {m} _ {t} \\ \dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} \left(\Sigma_ {t} \Gamma_ {t} + \boldsymbol {\mu} _ {t} \boldsymbol {m} _ {t} ^ {\top}\right) - \left(\Gamma_ {t} \Sigma_ {t} + \boldsymbol {m} _ {t} \boldsymbol {\mu} _ {t} ^ {\top}\right) \Sigma_ {t} \end{array} . \right. \tag {49}
$$
This can be proved using exactly the same technique as in the proof of Theorem 3.6. Here (49) is the same as (22), and hence by Theorem  $4.1\mu_{t}$  and  $\Sigma_t$  converges to  $\pmb{\mu}^{*}$  and  $\Sigma^{*}$  that solves the GVI and the convergence rate is given in Theorem 4.2 when the target is strongly log-concave. We also conjecture that there is still uniform in time convergence to the mean-field limit for this particle system and leave it to future works.
E Details of the Simulations
Gaussian Targets. Following [19], we consider a scenario where the target is Gaussian  $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$  where  $\boldsymbol{\mu}\sim \mathrm{Unif}([0,1]^{10})$  and  $\Sigma^{-1} = U\mathrm{diag}\{\lambda_1,\lambda_2,\dots ,\lambda_{10}\} U^\top$  with  $U\in \mathbb{R}^{10\times 10}$  drawn from the Haar measure of the orthogonal matrices  $O(10)$  and  $\lambda_1,\dots ,\lambda_{10}$  being a geometric sequence such that  $\lambda_{1} = 0.01$  and  $\lambda_{10} = 1$ . We run the eight different algorithms as introduced in Section 5 and show the decay of  $\log \mathrm{KL}(\rho_{\theta}\parallel \rho^{*})$  over time in Figure 2. Clearly the algorithms with  $K_{1}$  show a faster rate over time compared to the others while the other algorithms eventually all converge at the same rate. This is actually confirmed from our theoretical analysis as in all the other dynamics except SBGD and SBPF, the mean converges at the rate of  $\mathcal{O}(e^{-t / \lambda})$  while the covariance converges at a faster rate, resulting in the fact that the KL divergence converges at  $\mathcal{O}(e^{-2t / \lambda})$ . But for  $K_{1}$  the rate is different and given in Theorem 3.1.
Gaussian Mixture Targets. Next we consider the 1-dimensional Gaussian mixture targets given by  $w_{1}\mathcal{N}(\mu_{1},\sigma_{1}^{2}) + (1 - w_{1})\mathcal{N}(\mu_{2},\sigma_{2}^{2})$ . We run the aforementioned eight algorithms with initial
(b)  $\widehat{\mathcal{F}} (\rho_{\theta})$  over iteration.
Figure 3: Convergence of Algorithms 1 and 2 with bilinear kernels for a Gaussian mixture target.
$\mu = 0$  and  $\sigma = 1$  or particles drawn from  $\mathcal{N}(0,1)$ . Here again we plot the decay of  $\widehat{\mathcal{F}} (\rho_{\theta})$  over time or iteration as shown in Figure 3. In particular, the plots correspond to the specific setting of  $\mu_1 = 5,\mu_2 = 10,\sigma_1 = 5,\sigma_2 = 2$  and  $\rho^{*}(x)\propto 0.3\exp (-(x - 5)^{2} / 50) + 0.7\exp (-(x - 10)^{2} / 8)$ . These parameters are arbitrarily chosen. For the decay of  $\widehat{\mathcal{F}} (\rho_{\theta})$  over time, we fix the step size to be 0.1 and run 1000 iterations. For  $\widehat{\mathcal{F}} (\rho_{\theta})$  over time, we draw 500 particles for particle-based algorithms and run all algorithms for 500 iterations so that a total of 500 samples are drawn for the density-based algorithms. The step sizes are chosen to be the largest ones that still allow convergence. Specifically for these eight algorithms the step sizes are 0.02, 0.1, 1, 1, 0.2, 0.8, 8, 8. Consistent with the results of Bayesian logistic regression, the particle-based ones are more stable and allow larger step sizes, with BWPF and RGPF clearly outperforming all the others. In fact, the constraint between particle-based and density-based algorithms is particularly significant in this problem probably because of the non-log-concave target.
More on the Bayesian Logistic Regression. We compare three so-far best performed algorithms, BWPF, RGPF, and FB-GVI [19] for the same problem with different step sizes. From Figure 4 we see that BWPF outperforms the other two. FB-GVI is better than RGPF with a larger learning rate but fluctuate a bit more when  $\eta = 2$ . This is probably attributed to the stochastic gradients. Furthermore, it is interesting to compare to ordinary gradient descent (OGD) on the variational parameters (mean and covariance) and SVGD with a radius-based kernel function (RBF-SVGD)  $K_{h}(\boldsymbol{x},\boldsymbol{y}) = \exp \left(-\frac{\|\boldsymbol{x} - \boldsymbol{y}\|^{2}}{2h^{2}}\right)$ . We show this comparison in Figure 5 with the same step size  $\eta = 2$ . Firstly, OGD does not converge as fast as BWPF and it is not as stable. Secondly, RBF-SVGD is quite sensitive to the choice of bandwidth and it does not converge as fast as BWPF in general. We also notice that RBF-SVGD is significantly slower in computation compared to Gaussian-SVGD. However, as a particle-based algorithm, RBF-SVGD does have the advantage of being stable even when the step size is large.
F Analogous Results for the Affine-Invariant Bilinear Kernels
For the Bures-Wasserstein metric (Gaussian-Stein metric with  $K_{3}$ ), the dynamics of natural gradient descent has already been studied in literature. See [60, 86, 13] for proofs for the following theorem.
Theorem F.1 (Wasserstein gradient flow for the Gaussian family). Let  $\rho_0\sim \mathcal{N}(\pmb {\mu}_0,\Sigma_0)$  and  $\rho^{*}\sim$ $\mathcal{N}(\pmb {b},Q)$  be two Gaussian measures. Then the solution of (1) converges to  $\rho^*$  as  $t\to \infty$  . In particular,  $\rho_{t}$  is the density of  $\mathcal{N}(\pmb {\mu}_t,\Sigma_t)$  where the mean  $\pmb{\mu}_{t}$  and covariance matrix  $\Sigma_t$  satisfies
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = - Q ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) \\ \dot {\Sigma} _ {t} = 2 I - \Sigma_ {t} Q ^ {- 1} - Q ^ {- 1} \Sigma_ {t} \end{array} . \right. \tag {50}
$$
If  $\Sigma_0Q = Q\Sigma_0$ , we have  $\| \pmb{\mu}_t - \pmb{b}\| = \mathcal{O}(e^{-t / \lambda})$  and  $\| \Sigma_t - Q\| = \mathcal{O}(e^{-2t / \lambda})$ , where  $\lambda$  is the largest eigenvalue of  $Q$ .
Now we focus on the results for Gaussian-SVGD with the kernel  $K_{2}$ . First we remark that for  $K_{2}$  the previous results on the well-posedness of mean-field PDE and finite-particle solutions in Appendix C still hold and can be proved using similar techniques to Appendix H but for simplicity they are
Figure 4: Performance of BWPF, RGPF, and FB-GVI with different step sizes  $\eta$ .
Figure 5: Performance of BWPF, OGD, and RBF-SVGD with bandwidth  $h$ .
omitted. The proofs of the following theorems will also be omitted unless a different proof technique from the analogous results needs to be applied.
Theorem F.2 (Analogue of Theorem 3.1). For any  $t \geq 0$  the solution  $\rho_t$  of SVGD (2) with the bilinear kernel remains a Gaussian density with mean  $\pmb{\mu}_t$  and covariance matrix  $\Sigma_t$  given by
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = - Q ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) \\ \dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} ^ {2} Q ^ {- 1} - Q ^ {- 1} \Sigma_ {t} ^ {2}, \end{array} \right. \tag {51}
$$
which has a unique global solution on  $[0,\infty)$  given any  $\pmb{\mu}_0\in \mathbb{R}^d$  and  $\Sigma_0\in \mathrm{Sym}^+ (d,\mathbb{R})$ . And  $\rho_{t}$  converges weakly to  $\rho^{*}$  as  $t\to \infty$ . If  $\Sigma_0Q = Q\Sigma_0$  then we have the following rates
$$
\left\| \boldsymbol {\mu} _ {t} - \boldsymbol {b} \right\| = \mathcal {O} (e ^ {- t / \lambda}), \quad \left\| \Sigma_ {t} - Q \right\| = \mathcal {O} (e ^ {- 2 t}), \quad \forall \epsilon > 0,
$$
The proof of the dynamics is similar to that of Theorem 3.1. The rate  $\mathcal{O}(e^{-t / \lambda})$  is trivially true from the theory of linear ODEs and the rate  $\mathcal{O}(e^{-2t})$  is given by Theorem 3.2.
Theorem F.3 (Analogue of Theorem 3.6). Suppose the initial particles satisfy that  $C_0$  is non-singular. There exists a unique solution of the finite particle system (13) given by
$$
\boldsymbol {x} _ {i} ^ {(t)} = A _ {t} \left(\boldsymbol {x} _ {i} ^ {(0)} - \boldsymbol {\mu} _ {0}\right) + \boldsymbol {\mu} _ {t}, \tag {52}
$$
$$
\dot {A} _ {t} = \left(I - Q ^ {- 1} C _ {t}\right) A _ {t}, \quad A _ {0} = I, \tag {53}
$$
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = - Q ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) \\ \dot {C} _ {t} = 2 C _ {t} - C _ {t} ^ {2} Q ^ {- 1} - Q ^ {- 1} C _ {t} ^ {2}. \end{array} \right. \tag {54}
$$
The proof is similar to that of Theorem 3.6.
Theorem F.4 (Analogue of Theorem 3.7). Given the same setting as Theorem F.3, further suppose the initial particles are drawn i.i.d. from  $\mathcal{N}(\pmb{\mu}_0,\Sigma_0)$ . Then there exists a constant  $C_{d,Q,\pmb{b},\Sigma_0,\pmb{\mu}_0}$  such that for all  $t$ , for all  $N\geq 2$ , with the empirical measure  $\zeta_N^{(t)} = \frac{1}{N}\sum_{i = 1}^N\delta_{\pmb{x}_i^{(t)}}$ , the second moment of Wasserstein-2 distance between  $\zeta_N^{(t)}$  and  $\rho_t$  converges:
$$
\mathbb {E} \left[ \mathcal {W} _ {2} ^ {2} \left(\zeta_ {N} ^ {(t)}, \rho_ {t}\right) \right] \leq C _ {d, Q, \boldsymbol {b}, \Sigma_ {0}, \boldsymbol {\mu} _ {0}} \times \left\{ \begin{array}{c c} N ^ {- 1} \log \log N & i f d = 1 \\ N ^ {- 1} (\log N) ^ {2} & i f d = 2 \\ N ^ {- 2 / d} & i f d \geq 3 \end{array} \right.. \tag {55}
$$
The proof is similar to that of Theorem 3.7. But for sake of completeness we provide more proof details in Appendix L.
Theorem F.5 (Analogue of Theorem 4.1). Let  $\rho^{*}$  be the density of a target distribution with the potential function  $V(\pmb{x}) = -\log \rho^{*}(\pmb{x})$  and  $\rho_0$  be the density of  $\mathcal{N}(\pmb{\mu}_0, \Sigma_0)$ . Then for any  $t \geq 0$  the Gaussian-SVGD produces a Gaussian density  $\rho_t$  with mean  $\pmb{\mu}_t$  and covariance matrix  $\Sigma_t$  given by the following ODE system:
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = - \mathbb {E} _ {\boldsymbol {x} \sim \rho_ {t}} [ \nabla V (\boldsymbol {x}) ] \\ \dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} ^ {2} \mathbb {E} _ {\boldsymbol {x} \sim \rho_ {t}} [ \nabla^ {2} V (\boldsymbol {x}) ] - \mathbb {E} _ {\boldsymbol {x} \sim \rho_ {t}} [ \nabla^ {2} V (\boldsymbol {x}) ] \Sigma_ {t} ^ {2}. \end{array} \right. \tag {56}
$$
The proof is similar to that of Theorem 4.1. In particular, if the target is strongly log-concave, it gives rise to the following linear convergence rate.
Theorem F.6 (Analogue of Theorem 4.2). Assume that the target  $\rho^{*}$  is  $\alpha$ -strongly log-concave and  $\beta$ -log-smooth, i.e.,  $\alpha I \preceq \nabla^2 V(\pmb{x}) \preceq \beta I$ . Then  $\rho_t$  converges to  $\rho_{\theta^*}$  at the following rate:
$$
\left\| \boldsymbol {\mu} _ {t} - \boldsymbol {\mu} ^ {*} \right\| = \mathcal {O} \left(e ^ {- \alpha t / \max  \{\beta , 1 \}}\right), \quad \left\| \Sigma_ {t} - \Sigma^ {*} \right\| = \mathcal {O} \left(e ^ {- \alpha t / \max  \{\beta , 1 \}}\right).
$$
The proof is similar to that of Theorem 4.2.
Theorem F.7 (Analogue of Theorem D.1). The solution of the finite-particle system (48) with  $K_{2}$  is given by  $\pmb{x}_{i}^{(t)} = A_{t}(\pmb{x}_{i}^{(0)} - \pmb{\mu}_{0}) + \pmb{\mu}_{t}$  where  $A_{t}$  is the unique solution of
$$
\dot {A} _ {t} = \left(I - \Gamma_ {t} C _ {t}\right) A _ {t}, \quad A _ {0} = I,
$$
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = - \boldsymbol {m} _ {t} \\ \dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} ^ {2} \Gamma_ {t} - \Gamma_ {t} \Sigma_ {t} ^ {2} \end{array} . \right.
$$
The proof is similar to that of Theorem 3.6 or Theorem D.1.
G Proofs for Section B
Lemma G.1. Let  $\rho(\boldsymbol{x}) = (2\pi)^{-d/2} \left( \operatorname*{det}(\Sigma) \right)^{-1/2} \exp \left( -\frac{1}{2} \boldsymbol{x}^\top \Sigma^{-1} \boldsymbol{x} \right)$  be the density of a  $d$ -dimensional normal random vector where  $\Sigma$  is a positive definite matrix. Then for any  $d \times d$  real matrix  $A$  and  $d$ -dimensional real vector  $\boldsymbol{b}$ , we have
$$
\int \boldsymbol {x} ^ {\top} A \boldsymbol {x} \rho (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} = \operatorname {t r} (A \Sigma), \quad \int \boldsymbol {b} ^ {\top} \boldsymbol {x} A \boldsymbol {x} \rho (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} = A \Sigma \boldsymbol {b}.
$$
Proof. Since  $\Sigma^{-1}$  is a positive definite matrix, we can find its positive definite root  $\Sigma^{-1/2}$ . Let  $\pmb{y} = \Sigma^{-1/2}\pmb{x}$  and  $\rho_0(\pmb{x}) = (2\pi)^{-d/2} \exp\left(-\frac{1}{2}\pmb{x}^\top \pmb{x}\right)$ . Then we have
$$
\begin{array}{l} \int \boldsymbol {x} ^ {\top} A \boldsymbol {x} \rho (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} \\ = \int \boldsymbol {y} ^ {\top} \Sigma^ {1 / 2} A \Sigma^ {1 / 2} \boldsymbol {y} (\det (\Sigma)) ^ {- 1 / 2} \rho_ {0} (\boldsymbol {y}) (\det (\Sigma)) ^ {1 / 2} d \boldsymbol {y} \\ = \int \left(\sum_ {j = 1} ^ {d} \left(\Sigma^ {1 / 2} A \Sigma^ {1 / 2}\right) _ {j j} y _ {j} ^ {2}\right) \rho_ {0} (\boldsymbol {y}) d \boldsymbol {y} \\ = \sum_ {j = 1} ^ {d} \left(\Sigma^ {1 / 2} A \Sigma^ {1 / 2}\right) _ {j j} \\ = \operatorname {t r} \left(\Sigma^ {1 / 2} A \Sigma^ {1 / 2}\right) = \operatorname {t r} (A \Sigma). \\ \end{array}
$$
The second equation is given by
$$
\begin{array}{l} \int \boldsymbol {b} ^ {\top} \boldsymbol {x} A \boldsymbol {x} \rho (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} = \int A \boldsymbol {x} \boldsymbol {x} ^ {\top} \boldsymbol {b} \rho (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} \\ = A \left(\int \boldsymbol {x} \boldsymbol {x} ^ {\top} \rho (\boldsymbol {x}) \mathrm {d} \boldsymbol {x}\right) \boldsymbol {b} = A \Sigma \boldsymbol {b}. \\ \end{array}
$$
Lemma G.2 (Lyapunov equation). The Lyapunov equation
$$
P X + X P = Q
$$
has a unique solution. If  $P, Q \in \operatorname{Sym}(d, \mathbb{R})$ , then the solution  $X \in \operatorname{Sym}(d, \mathbb{R})$ .
Proof. By Sylvester-Rosenblum theorem in control theory [7],  $PX + XP = Q$  has a unique solution  $X$ . Note that if  $P, Q \in \operatorname{Sym}(d, \mathbb{R})$  and  $X$  is a solution, then  $X^{\top}$  is also a solution. Thus, we have  $X = X^{\top}$  which implies that  $X \in \operatorname{Sym}(d, \mathbb{R})$ .
Proof of Theorem B.3. Note that the tangent space at each  $\theta \in \Theta = \mathbb{R}^d\times \mathrm{Sym}^+ (d,\mathbb{R})$  is  $T_{\theta}\Theta \simeq \mathbb{R}^{d}\times \operatorname {Sym}(d,\mathbb{R})$ . If we define the inner product (pairing) on  $T_{\theta}\Theta$  by