SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From this theorem we see that R-SVGD can take the advantage of both regimes by choosing  $\nu$  wisely. Another interesting connection is that on  $\Theta$  the induced regularized Stein metric coincides with the Stein metric with a different kernel  $K_{4}(\pmb{x},\pmb{y}) = (\pmb{x} - \pmb{\mu})^{\top}((1 - \nu)\Sigma + \nu I)^{-1}(\pmb{y} - \pmb{\mu}) + 1$  (see Theorem B.7).
**Stein AIG Flow.** Accelerating methods are widely used in first-order optimization algorithms and have attracted considerable interest in particle-based variational inference [50]. [76, 86] study the accelerated information gradient (AIG) flows as the analogue of Nesterov's accelerated gradient method [63] on the density manifold. Given a probability space  $\mathcal{P}(\mathbb{R}^d)$  with a metric tensor  $g_{\rho}(\cdot, \cdot)$ , let  $G_{\rho}: T_{\rho}\mathcal{P}(\mathbb{R}^{d}) \to T_{\rho}^{*}\mathcal{P}(\mathbb{R}^{d})$  be the corresponding isomorphism. The Hamiltonian flow in probability space [16] follows from
$$
\partial_ {t} \left[ \begin{array}{c} \rho_ {t} \\ \Phi_ {t} \end{array} \right] = \left[ \begin{array}{c c} 0 & 1 \\ - 1 & 0 \end{array} \right] \left[ \begin{array}{c} \frac {\delta}{\delta \rho_ {t}} \mathcal {H} (\rho_ {t}, \Phi_ {t}) \\ \frac {\delta}{\delta \Phi_ {t}} \mathcal {H} (\rho_ {t}, \Phi_ {t}) \end{array} \right], \text {w h e r e} \mathcal {H} (\rho_ {t}, \Phi_ {t}) := \frac {1}{2} \int \Phi_ {t} G _ {\rho_ {t}} ^ {- 1} \Phi_ {t}   d \boldsymbol {x} + \mathrm {K L} (\rho \| \rho^ {*})
$$
is called the Hamiltonian function, which consists of a kinetic energy  $\frac{1}{2}\int \Phi G_{\rho}^{-1}\Phi \mathrm{d}\pmb{x}$  and a potential energy  $\mathrm{KL}(\rho \parallel \rho^{*})$ . Following [86] we introduce the accelerated information gradient flow in probability space. Let  $\alpha_{t}\geq 0$  be a scalar function of time  $t$ . We add a damping term  $\alpha_{t}\Phi_{t}$  to the Hamiltonian flow:
$$
\partial_ {t} \left[ \begin{array}{l} \rho_ {t} \\ \Phi_ {t} \end{array} \right] = - \left[ \begin{array}{l} 0 \\ \alpha_ {t} \Phi_ {t} \end{array} \right] + \left[ \begin{array}{l l} 0 & 1 \\ - 1 & 0 \end{array} \right] \left[ \begin{array}{l} \frac {\delta}{\delta \rho_ {t}} \mathcal {H} (\rho_ {t}, \Phi_ {t}) \\ \frac {\delta}{\delta \Phi_ {t}} \mathcal {H} (\rho_ {t}, \Phi_ {t}) \end{array} \right], \tag {10}
$$
By adopting the Stein metric we obtain the Stein AIG flow (S-AIGF):
$$
\left\{ \begin{array}{l} \dot {\rho} _ {t} = - \nabla \cdot \left(\rho_ {t} (\cdot) \int K (\cdot , \boldsymbol {y}) \rho_ {t} (\boldsymbol {y}) \nabla \Phi_ {t} (\boldsymbol {y}) d \boldsymbol {y}\right) \\ \dot {\Phi} _ {t} = - \alpha_ {t} \Phi_ {t} - \int \nabla \Phi_ {t} (\cdot) ^ {\top} \nabla \Phi_ {t} (\boldsymbol {y}) K (\cdot , \boldsymbol {y}) \rho_ {t} (\boldsymbol {y}) d \boldsymbol {y} - \frac {\delta}{\delta \rho_ {t}} \mathrm {K L} \left(\rho_ {t} \| \rho^ {*}\right). \end{array} \right. \tag {11}
$$
Again we characterize the dynamics of S-AIGF with the linear kernel for the Gaussian family:
Theorem 3.5 (S-AIGF). Let  $\rho_0$  and  $\rho^*$  be two centered Gaussian densities. Then the solution  $\rho_t$  of S-AIGF (11) with the bilinear kernel  $K_{1}$  or  $K_{2}$  is the density of  $\mathcal{N}(\mathbf{0},\Sigma_t)$  where  $\Sigma_t$  satisfies
$$
\left\{ \begin{array}{l} \dot {\Sigma} _ {t} = 2 \left(S _ {t} \Sigma_ {t} ^ {2} + \Sigma_ {t} ^ {2} S _ {t}\right) \\ \dot {S} _ {t} = - \alpha_ {t} S _ {t} - 2 \left(S _ {t} ^ {2} \Sigma_ {t} + \Sigma_ {t} S _ {t} ^ {2}\right) + \frac {1}{2} \left(\Sigma_ {t} ^ {- 1} - Q ^ {- 1}\right) \end{array} , \right. \tag {12}
$$
where  $S_{t}\in \mathrm{Sym}(d,\mathbb{R})$  with initial value  $S_0 = 0$
Note that the convergence properties of S-AIGF still remains open in contrast to the Wasserstein AIG flow (W-AIGF) as [86] shows that the W-AIGF for the centered Gaussian family is
$$
\dot {\Sigma} _ {t} = 2 (S _ {t} \Sigma_ {t} + \Sigma_ {t} S _ {t}), \qquad \dot {S} _ {t} = - \alpha_ {t} S _ {t} - 2 S _ {t} ^ {2} + \frac {1}{2} (\Sigma_ {t} ^ {- 1} - Q ^ {- 1}),
$$
and that if  $\alpha_{t}$  is well-chosen, the KL divergence in W-AIGF converges at the rate of  $\mathcal{O}(e^{-t / \sqrt{\lambda}})$ . Thus, when  $\lambda$  is large it converges faster than WGF. It is also interesting to point out that the acceleration effect of Nesterov's scheme also comes from time discretization of the ODE system (see [74]) as it moves roughly  $\sqrt{\epsilon}$  rather than  $\epsilon$  along the gradient path when the step size is  $\epsilon$ .
3.2 Finite-Particle Systems
In this subsection, we consider the case where  $N < \infty$  particles evolve in time  $t$ . We set a Gaussian target  $\mathcal{N}(\boldsymbol{b}, Q)$  (i.e., the potential is  $V(\boldsymbol{x}) = \frac{1}{2} (\boldsymbol{x} - \boldsymbol{b})^\top Q^{-1}(\boldsymbol{x} - \boldsymbol{b}))$  and run the SVGD algorithm with a bilinear kernel, and obtain the dynamics of  $x_1^{(t)}, \dots, x_N^{(t)}$ . The continuous-time particle-based SVGD corresponds to the following deterministic interactive system in  $\mathbb{R}^d$ :
$$
\dot {\boldsymbol {x}} _ {i} ^ {(t)} = \frac {1}{N} \sum_ {j = 1} ^ {N} \nabla_ {\boldsymbol {x} _ {j}} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) - \frac {1}{N} \sum_ {j = 1} ^ {N} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) \nabla V \left(\boldsymbol {x} _ {j} ^ {(t)}\right) \tag {13}
$$
with initial particles given by  $\pmb{x}_i^{(0)} (i = 1, \dots, N)$ . Now denoting the sample mean and covariance matrix at time  $t$  by  $\pmb{\mu}_t := \frac{1}{N} \sum_{j=1}^N \pmb{x}_j^{(t)}$  and  $C_t := \frac{1}{N} \sum_{j=1}^N \pmb{x}_j^{(t)} \pmb{x}_j^{(t)^\top} - \pmb{\mu}_t \pmb{\mu}_t^\top$ , we have the following theorem.
Theorem 3.6 (SVGD). Suppose the initial particles satisfy that  $C_0$  is non-singular. Then SVGD (13) with the bilinear kernel  $K_{1}$  and Gaussian potential  $V$  has a unique solution given by
$$
\boldsymbol {x} _ {i} ^ {(t)} = A _ {t} \left(\boldsymbol {x} _ {i} ^ {(0)} - \boldsymbol {\mu} _ {0}\right) + \boldsymbol {\mu} _ {t}, \tag {14}
$$
where  $A_{t}$  is the unique (matrix) solution of the linear system
$$
\dot {A} _ {t} = \left(I - Q ^ {- 1} \left(C _ {t} + \boldsymbol {\mu} _ {t} \boldsymbol {\mu} _ {t} ^ {\top}\right) + Q ^ {- 1} \boldsymbol {b} \boldsymbol {\mu} _ {t} ^ {\top}\right) A _ {t}, \quad A _ {0} = I, \tag {15}
$$
and  $\pmb{\mu}_t$  and  $C_t$  are the unique solution of the ODE system
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = \left(I - Q ^ {- 1} C _ {t}\right) \boldsymbol {\mu} _ {t} - \left(1 + \boldsymbol {\mu} _ {t} ^ {\top} \boldsymbol {\mu} _ {t}\right) Q ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) \\ \dot {C} _ {t} = 2 C _ {t} - C _ {t} \left(C _ {t} + \boldsymbol {\mu} _ {t} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) ^ {\top}\right) Q ^ {- 1} - Q ^ {- 1} \left(C _ {t} + \left(\boldsymbol {\mu} _ {t} - \boldsymbol {b}\right) \boldsymbol {\mu} _ {t} ^ {\top}\right) C _ {t}. \end{array} \right. \tag {16}
$$
The ODE system (16) is exactly the same as that in the density flow (4). Thus, we have the same convergence rates as in Theorem 3.1. Theorem 3.6 can be interpreted as: At each time  $t$  the particle positions are a linear transformation of the initialization. On one hand, if we initialize i.i.d. from Gaussian, there is uniform in time convergence as shown in the theorem below.
On the other hand, if we initialize i.i.d. from a non-Gaussian distribution  $\rho_0$ . At each time  $t$  the mean field limit  $\rho_{t}$  should be a linear transformation of  $\rho_0$  and cannot converge to the Gaussian target  $\rho^{*}$  as  $t\to \infty$ . Note that in general SVGD with the bilinear kernel might not always converge to the target distribution for nonparametric sampling but for GVI there is no such issue. This will be discussed in detail in Appendix C together with general results of well-posedness and mean-field convergence of SVGD with the bilinear kernel, which has not yet been studied in literature.
Theorem 3.7 (Uniform in time convergence). Given the same setting as Theorem 3.6, further suppose the initial particles are drawn i.i.d. from  $\mathcal{N}(\pmb{\mu}_0, \Sigma_0)$ . Then there exists a constant  $C_{d,Q,\pmb{b},\Sigma_0,\pmb{\mu}_0}$  such that for all  $t \in [0,\infty]$ , for all  $N \geq 2$ , with the empirical measure  $\zeta_N^{(t)} = \frac{1}{N} \sum_{i=1}^N \delta_{\pmb{x}_i^{(t)}}$ , the second moment of Wasserstein-2 distance between  $\zeta_N^{(t)}$  and  $\rho_t$  converges:
$$
\mathbb {E} \left[ \mathcal {W} _ {2} ^ {2} \left(\zeta_ {N} ^ {(t)}, \rho_ {t}\right) \right] \leq C _ {d, Q, \boldsymbol {b}, \Sigma_ {0}, \boldsymbol {\mu} _ {0}} \times \left\{ \begin{array}{c c} N ^ {- 1} \log \log N & i f d = 1 \\ N ^ {- 1} (\log N) ^ {2} & i f d = 2 \\ N ^ {- 2 / d} & i f d \geq 3 \end{array} . \right. \tag {17}
$$
Similar to Theorem 3.2, we also provide the finite-particle result for a centered Gaussian target.
Theorem 3.8 (SVGD for centered Gaussian). Suppose the SVGD particle system (13) with the bilinear kernel  $K_{1}$  or  $K_{2}$  is targeting a centered Gaussian distribution and initialized by  $\left(\boldsymbol{x}_i^{(0)}\right)_{i=1}^N$  such that  $\boldsymbol{\mu}_0 = \mathbf{0}$  and  $C_0Q = QC_0$ . Then we have the following closed-form solution
$$
\boldsymbol {x} _ {i} ^ {(t)} = \left(e ^ {- 2 t} I + (1 - e ^ {- 2 t}) Q ^ {- 1} C _ {0}\right) ^ {- 1 / 2} \boldsymbol {x} _ {i} ^ {(0)}. \tag {18}
$$
Analogous Result for R-SVGD. Next we consider the particle dynamics of R-SVGD. As shown in [32], the finite-particle system of R-SVGD is
$$
\dot {X} _ {t} = - \left((1 - \nu) \frac {K _ {t}}{N} + \nu I\right) ^ {- 1} \left(\frac {K _ {t}}{N} \mathcal {L} _ {t} \nabla V - \frac {1}{N} \sum_ {j = 1} ^ {N} \mathcal {L} _ {t} \nabla K (\boldsymbol {x} _ {j} ^ {(t)}, \cdot)\right),
$$
where  $X_{t} := \left(\pmb{x}_{1}^{(t)},\dots ,\pmb{x}_{N}^{(t)}\right)^{\top}$ ,  $\mathcal{L}_t f := \left(f(\pmb{x}_i^{(t)}),\dots ,f(\pmb{x}_N^{(t)})\right)^\top$  for all  $f:\mathbb{R}^d\to \mathbb{R}^d$  and  $(K_{t})_{ij}:= K\left(\pmb{x}_{i}^{(t)},\pmb{x}_{j}^{(t)}\right)$  for all  $1\leq i,j\leq N$ . Similar to Theorem 3.6, we have the following result:
Theorem 3.9 (R-SVGD). Suppose the  $R$ -SVGD system (13) with  $K_{1}$  or  $K_{2}$  is targeting a centered Gaussian distribution and initialized by  $(\pmb{x}_i^{(0)})_{i=1}^N$  such that  $\pmb{\mu}_0 = \mathbf{0}$ . Then we have  $\pmb{x}_i^{(t)} = A_t\pmb{x}_i^{(0)}$  where  $A_t$  is the unique solution of the linear system
$$
\dot {A} _ {t} = \left(I - Q ^ {- 1} C _ {t}\right) \left(\left(1 - \nu\right) C _ {t} + \nu I\right) ^ {- 1} A _ {t}, \quad A _ {0} = I, \tag {19}
$$
and the sample covariance matrix  $C_t$  is given by
$$
\dot {C} _ {t} = 2 \left(\left(1 - \nu\right) C _ {t} + \nu I\right) ^ {- 1} C _ {t} - \left(\left(1 - \nu\right) C _ {t} + \nu I\right) ^ {- 1} C _ {t} ^ {2} Q ^ {- 1} - Q ^ {- 1} \left(\left(1 - \nu\right) C _ {t} + \nu I\right) ^ {- 1} C _ {t} ^ {2}. \tag {20}
$$
Again we observe that the particles at time  $t$  is a time-changing linear transformation of the initializers.
Discrete-time Analysis for Finite Particles. Next we consider the algorithm in discrete time  $t$ . The SVGD updates according to the following equation:
$$
\boldsymbol {x} _ {i} ^ {(t + 1)} = \boldsymbol {x} _ {i} ^ {(t)} + \frac {\epsilon}{N} \left(\sum_ {j = 1} ^ {N} \nabla_ {\boldsymbol {x} _ {j} ^ {(t)}} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) - \sum_ {j = 1} ^ {N} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) \nabla_ {\boldsymbol {x} _ {j} ^ {(t)}} V \left(\boldsymbol {x} _ {j} ^ {(t)}\right)\right). \tag {21}
$$
For simplicity, we only consider the case where both the target and initializers are centered, i.e.,  $\pmb{b} = \pmb{\mu}_0 = \mathbf{0}$  and show the convergence:
Theorem 3.10 (Discrete-time convergence). For a centered Gaussian target, suppose the particle system (21) with  $K_{1}$  or  $K_{2}$  is initialized by  $\left(\boldsymbol{x}_{i}^{(0)}\right)_{i=1}^{N}$  such that  $\boldsymbol{\mu}_{0} = \mathbf{0}$  and  $C_{0}Q = QC_{0}$ . For  $0 < \epsilon < 0.5$ , we have  $\boldsymbol{\mu}_{t} = \mathbf{0}$  and  $\|C_{t}-Q\| \to 0$  as long as all the eigenvalues of  $Q^{-1}C_{0}$  lie in the interval  $(0,1 + 1/\epsilon)$ .
Furthermore, if we set  $u_{\epsilon}$  to be the smaller root of the equation  $f_{\epsilon}'(u) = 1 - \epsilon$  (it has 2 distinct roots) where  $f_{\epsilon}(x) \coloneqq (1 + \epsilon (1 - x))^{2}x$ , then we have linear convergence, i.e.,
$$
\left\| C _ {t} - Q \right\| \leq (1 - \epsilon) ^ {t} \left\| C _ {0} - Q \right\| \leq e ^ {- \epsilon t} \left\| C _ {0} - Q \right\|
$$
as long as all the eigenvalues of  $Q^{-1}C_0$  lie in the interval  $[u_{\epsilon}, 1/3 + 1/(3\epsilon)]$ .
The above result illustrates that firstly the step sizes required are restricted by the largest eigenvalue of  $Q^{-1}C_0$ . In particular if  $C_0 = I_d$  then we need smaller step size if the smallest eigenvalue of  $Q$  is smaller, which corresponds to the  $\beta$ -log-smoothness condition of the target distribution. Secondly we can potentially have faster convergence over iteration given larger step sizes. We believe that the commutativity assumption in Theorem 3.10 can be relaxed and similar results can be obtained for general targets. Detailed examinations are left as future work.
4 Beyond Gaussian Targets
In this section we consider the Gaussian-SVGD with a general target and have the following dynamics.
Theorem 4.1. Let  $\rho^{*}$  be the density of the target distribution with the potential function  $V(\pmb{x})$  that satisfies Assumption C.1 and  $\rho_0$  be the density of  $\mathcal{N}(\pmb{\mu}_0, \Sigma_0)$ . The Gaussian-SVGD with  $K_{1}$  produces a Gaussian density  $\rho_t$  with mean  $\pmb{\mu}_t$  and covariance matrix  $\Sigma_t$  given by
$$
\left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = \left(I - \Gamma_ {t} \Sigma_ {t}\right) \boldsymbol {\mu} _ {t} - \left(1 + \boldsymbol {\mu} _ {t} ^ {\top} \boldsymbol {\mu} _ {t}\right) \boldsymbol {m} _ {t} \\ \dot {\Sigma} _ {t} = 2 \Sigma_ {t} - \Sigma_ {t} \left(\Sigma_ {t} \Gamma_ {t} + \boldsymbol {\mu} _ {t} \boldsymbol {m} _ {t} ^ {\top}\right) - \left(\Gamma_ {t} \Sigma_ {t} + \boldsymbol {m} _ {t} \boldsymbol {\mu} _ {t} ^ {\top}\right) \Sigma_ {t} \end{array} , \right. \tag {22}
$$
where  $\Gamma_t = \mathbb{E}_{\boldsymbol{x} \sim \rho_t}[\nabla^2 V(\boldsymbol{x})]$  and  $\boldsymbol{m}_t = \mathbb{E}_{\boldsymbol{x} \sim \rho_t}[\nabla V(\boldsymbol{x})]$ .
Furthermore, suppose that  $\theta^{*}$  is the unique solution of the following optimization problem
$$
\min  _ {\theta = (\boldsymbol {\mu}, \Sigma)} \operatorname {K L} \left(\rho_ {\theta} \| \rho^ {*}\right), \text {w h e r e} \rho_ {\theta} \text {i s t h e G a u s s i a n m e a s u r e} \mathcal {N} (\boldsymbol {\mu}, \Sigma).
$$
Then we have  $\rho_t \to \rho_{\theta^*} \sim \mathcal{N}(\pmb{\mu}^*, \Sigma^*)$  as  $t \to \infty$ .
In particular, if the target is strongly log-concave, it gives rise to the following linear convergence:
Algorithm 1 Density-based Gaussian-SVGD.
for  $t$  in  $0:T\mathbf{d}\mathbf{o}_{N}$  Draw  $\left(\pmb {x}_i^{(t)}\right)_{i = 1}$  from  $\mathcal{N}(\pmb {\mu}_t,\Sigma_t)$
Update  $\widehat{m}_t$  and  $\widehat{\Gamma}_t$  using (24) or (25)
$$
\begin{array}{l} \boldsymbol {\mu} _ {t + 1} \leftarrow \boldsymbol {\mu} _ {t} + \epsilon_ {1} F \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}, \widehat {\boldsymbol {m}} _ {t}, \widehat {\Gamma} _ {t}\right) \\ M _ {t + 1} \leftarrow I + \epsilon_ {2} G \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}, \widehat {m} _ {t}, \widehat {\Gamma} _ {t}\right) \\ \Sigma_ {t + 1} \leftarrow M _ {t + 1} \Sigma_ {t} M _ {t + 1} ^ {\top} \\ \end{array}
$$
Algorithm 2 Particle-based Gaussian-SVGD.
Draw  $(\pmb{x}_i^{(0)})_{i = 1}^N$  from  $\mathcal{N}(\pmb{\mu}_0,\Sigma_0)$
for  $t$  in  $0: T$  do
$$
\begin{array}{l} \boldsymbol {\mu} _ {t} \leftarrow \frac {1}{N} \sum_ {k = 1} ^ {N} \boldsymbol {x} _ {k} ^ {(t)} \\ \Sigma_ {t} \leftarrow \frac {1}{N} \sum_ {k = 1} ^ {N} \boldsymbol {x} _ {k} ^ {(t)} \boldsymbol {x} _ {k} ^ {(t) \top} - \boldsymbol {\mu} _ {t} \boldsymbol {\mu} _ {t} ^ {\top} \\ \end{array}
$$
Update  $(\pmb{x}_i^{(t + 1)})_{i = 1}^N$  using (26)
Theorem 4.2. Assume that the target  $\rho^{*}$  is  $\alpha$ -strongly log-concave and  $\beta$ -log-smooth, i.e.,  $\alpha I \preceq \nabla^2 V(\pmb{x}) \preceq \beta I$ . Then  $\rho_t$  of Theorem 4.1 converges to  $\rho_{\theta^*}$  at the following rate
$$
\left\| \boldsymbol {\mu} _ {t} - \boldsymbol {\mu} ^ {*} \right\| = \mathcal {O} \left(e ^ {- (\gamma - \epsilon) t}\right), \quad \left\| \Sigma_ {t} - \Sigma^ {*} \right\| = \mathcal {O} \left(e ^ {- (\gamma - \epsilon) t}\right), \quad \forall \epsilon > 0,
$$
where  $\gamma /\alpha$  is the smallest eigenvalue of the matrix
$$
\left[ \begin{array}{c c} I _ {d} \otimes \Sigma^ {*} & \boldsymbol {\mu} ^ {*} \otimes (\Sigma^ {*}) ^ {1 / 2} \\ \boldsymbol {\mu} ^ {* ^ {\top}} \otimes (\Sigma^ {*}) ^ {1 / 2} & (1 + \boldsymbol {\mu} ^ {* ^ {\top}} \boldsymbol {\mu} ^ {*}) I _ {d} \end{array} \right] w i t h a l o w e r b o u n d \gamma > \frac {\alpha}{\beta (1 + \boldsymbol {\mu} ^ {* ^ {\top}} \boldsymbol {\mu} ^ {*}) + 1}.
$$
Typically the  $\beta$ -log-smoothness condition is not required for continuous-time analyses. However, it is required in the above statement, as our proof technique is based on comparing the decay of the energy function of the flow to that for WGF, following [13]. Relaxing this condition is interesting and we leave it as future work.
Unifying Algorithms. For general targets, we propose two unifying algorithm frameworks where we can choose any bilinear kernel (e.g.,  $K_{1} - K_{4}$ ) to solve GVI with SVGD. The first framework is density-based where we update  $\mu_{t}$  and  $\Sigma_{t}$  according to the mean-field dynamics. It requires the closed-form of the ODE system
$$
\begin{array}{l} \left\{ \begin{array}{l} \dot {\boldsymbol {\mu}} _ {t} = F \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}, \boldsymbol {m} _ {t}, \Gamma_ {t}\right) \\ \vdots \end{array} , \right. \tag {23} \\ \left\{\dot {\Sigma} _ {t} = \Sigma_ {t} G \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}, \boldsymbol {m} _ {t}, \Gamma_ {t}\right) + G \left(\boldsymbol {\mu} _ {t}, \Sigma_ {t}, \boldsymbol {m} _ {t}, \Gamma_ {t}\right) ^ {\top} \Sigma_ {t} \right., \\ \end{array}
$$
where  $\pmb{m}_t = \mathbb{E}_{\pmb{x} \sim \rho_t}[\nabla V(\pmb{x})]$  and  $\Gamma_t = \mathbb{E}_{\pmb{x} \sim \rho_t}[\nabla^2 V(\pmb{x})]$ , and  $F$  and  $G$  are some closed-form functions. For example  $F(\pmb{\mu}_t, \Sigma_t, \pmb{m}_t, \Gamma_t) = (I - \Gamma_t \Sigma_t) \pmb{\mu}_t - (1 + \pmb{\mu}_t^\top \pmb{\mu}_t) \pmb{m}_t$  and  $G(\pmb{\mu}_t, \Sigma_t, \pmb{m}_t, \Gamma_t) = I - \Sigma_t \Gamma_t - \pmb{\mu}_t \pmb{m}_t^\top$  for  $K_1$  as shown in (22). Note that  $\pmb{m}_t$  and  $\Gamma_t$  can be estimated from samples using
$$
\widehat {\boldsymbol {m}} _ {t} = \frac {1}{N} \sum_ {k = 1} ^ {N} \nabla V \left(\boldsymbol {x} _ {k} ^ {(t)}\right), \quad \widehat {\Gamma} _ {t} = \frac {1}{N} \sum_ {k = 1} ^ {N} \nabla^ {2} V \left(\boldsymbol {x} _ {k} ^ {(t)}\right), \tag {24}
$$
or using the first-order estimator
$$
\widehat {\Gamma} _ {t} = \frac {1}{N} \sum_ {k = 1} ^ {N} \nabla V \left(\boldsymbol {x} _ {k} ^ {(t)}\right) \left(\boldsymbol {x} _ {k} ^ {(t)} - \boldsymbol {\mu} _ {t}\right) ^ {\top} \Sigma_ {t} ^ {- 1}. \tag {25}
$$
The second framework is particle-based and does not need the closed-form ODE of the mean and covariance, making it more flexible than Algorithm 1. Here we initially draw  $N$  particles and keep updating them over time using
$$
\boldsymbol {x} _ {i} ^ {(t + 1)} = \boldsymbol {x} _ {i} ^ {(t)} + \frac {\epsilon}{N} \left(\sum_ {j = 1} ^ {N} \nabla_ {\boldsymbol {x} _ {j} ^ {(t)}} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) - \sum_ {j = 1} ^ {N} K \left(\boldsymbol {x} _ {i} ^ {(t)}, \boldsymbol {x} _ {j} ^ {(t)}\right) \widehat {\nabla V} \left(\boldsymbol {x} _ {j} ^ {(t)}\right)\right), \tag {26}
$$
where  $\widehat{\nabla V}$  is a (time-dependent) linear approximation of  $\nabla V$  defined as  $\widehat{\nabla V}(\pmb{x}) = \widehat{\Gamma}_t(\pmb{x} - \pmb{\mu}_t) + \widehat{\pmb{m}}_t$ . Intuitively this is used instead of  $\nabla V$  to ensure the Gaussianity of the particle system.
We now remark on the algorithms proposed in the literature that emerge as instances of the two proposed algorithm frameworks, thereby highlighting the unifying viewpoint offered by our analysis. The use of the kernel  $K_{1}$  for SVGD in variational inference dates back to [54]. Algorithms 1 and 2 with the kernel  $K_{2}$  correspond precisely to the GF and GPF algorithms in [27]. Moreover, if  $K_{3}$  is chosen, Algorithm 1 reproduces the BWGD algorithm in [43] (with  $N = 1$ ) and shares some similarity with the FB-GVI algorithm in [19]. Detailed discussions on these variants, connections and convergence properties are deferred to Appendix D.
(a)  $\widehat{\mathcal{F}} (\rho_{\theta})$  over time.
(b)  $\widehat{\mathcal{F}} (\rho_{\theta})$  over iteration.
Figure 1: Convergence of Algorithms 1 and 2 with bilinear kernels in Bayesian logistic regression.
In this section, we conduct simulations to compare Gaussian-SVGD dynamics with different kernels and the performance of the algorithms mentioned in the previous section. We consider three settings, Bayesian logistic regression, and Gaussian and Gaussian mixture targets. Here we present the results for Bayesian logistic regression as it involves a non-Gaussian but unimodal target and is one of the typical setups such that GVI is preferred in practice. For sake of space we leave the simulations for the other two settings along with further discussions to Appendix E.
Bayesian Logistic Regression. The following generative model is considered: Given a parameter  $\pmb{\xi} \in \mathbb{R}^d$ , draw samples  $\{(X_i, Y_i)\}_{i=1}^n \in (\mathbb{R}^d \times \{0,1\})^n$  such that  $X_i \stackrel{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, I_d)$  and  $Y_i \mid X_i \sim \operatorname{Bern}(\sigma(\langle \pmb{\xi}, X_i \rangle))$  where  $\sigma(\cdot)$  is the logistic function. Given the samples  $\{(X_i, Y_i)\}_{i=1}^n$  and a uniform (improper) prior on  $\pmb{\xi}$ , the potential function of the posterior  $\rho^*$  on  $\pmb{\xi}$  is given by  $V(\pmb{\xi}) = \sum_{i=1}^n (\log(1 + \exp(\langle \pmb{\xi}, X_i \rangle) - Y_i \langle \pmb{\xi}, X_i \rangle))$ . We run both Algorithms 1 and 2 initialized at  $\rho_0 = \mathcal{N}(\mathbf{0}, I_d)$  to find the  $\rho_{\theta^*}$  that minimizes  $\mathrm{KL}(\rho_\theta \parallel \rho^*)$ . In Figure 1, SBGD (Simple Bilinear Gradient Descent), GF (Gaussian Flow [27]), BWGD (Bures-Wasserstein Gradient Descent [43]), and RGF (Regularized Gaussian Flow) are density-based algorithms with the bilinear kernels  $K_1$ ,  $K_2$ ,  $K_3$ , and  $K_4$  ( $\nu = 0.5$ ) respectively; SBPF (Simple Bilinear Particle Flow), GPF (Gaussian Particle Flow [27]), BWPF (Bures-Wasserstein Particle Flow), and RGPF (Regularized Gaussian Particle Flow) are particle-based algorithms with the bilinear kernels  $K_1$ ,  $K_2$ ,  $K_3$ , and  $K_4$  ( $\nu = 0.5$ ) respectively. We use the sample estimator of  $\mathcal{F}(\rho_\theta) = \int (\log \rho_\theta + V) \, \mathrm{d}\rho_\theta = \mathrm{KL}(\rho_\theta \parallel \rho^*) + C$  ( $C$  is some constant) to evaluate the learned parameters  $\theta = (\mu, \Sigma)$ . For a fair comparison in Figure 1b, we draw 2000 particles for particle-based algorithms and run all algorithms for 2000 iterations so that a total of 2000 samples are drawn for the density-based algorithms. The largest safe step sizes are 0.02, 0.1, 2, 0.8, 0.02, 0.2, 4, 4.