SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, for each time-step  $t$ , we focus on the time evolution of the particles  $(\mathbf{x}_t^{(l)})_{l\geq t}$  (called the lower triangular evolution). From (3), we observe that for any  $t\in \mathbb{N}$  and  $l\geq t$ ,  $\mathbf{x}_t^{(l)}$  depends only on  $\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t - 1)},\mathbf{x}_0^{(l)}$ . Hence, there exists a deterministic, measurable function  $H_{t}$  such that:
$$
\mathbf {x} _ {t} ^ {(l)} = H _ {t} \left(\mathbf {x} _ {0} ^ {(0)}, \dots , \mathbf {x} _ {0} ^ {(t - 1)}, \mathbf {x} _ {0} ^ {(l)}\right); \quad \text {f o r e v e r y} l \geq t \tag {4}
$$
Since  $\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t - 1)},\mathbf{x}_0^{(l)}\stackrel {i.i.d.}{\sim}\mu_0$ , we conclude from (4) that  $(\mathbf{x}_t^{(l)})_{l\geq t}$  are i.i.d when conditioned on  $\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t - 1)}$ . To this end, we define the random measure  $\mu_t|\mathcal{F}_t$  as the law of  $\mathbf{x}_t^{(t)}$  conditioned on  $\mathcal{F}_t$ , i.e.,  $\mu_t|F_t$  is a probability kernel  $\mu_t(\cdot ;\mathbf{x}_0^{(0)},\dots ,\mathbf{x}_0^{(t - 1)})$ , where  $\mu_0|\mathcal{F}_0\coloneqq \mu_0$ . By the strong law of large numbers,  $\mu_t|F_t$  is equal to the empirical measure of  $(\mathbf{x}_t^{(l)})_{l\geq t}$  conditioned on  $\mathcal{F}_t$ . We will use  $\mu_t|F_t$  and  $\mu_t(\cdot ;\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t - 1)})$  interchangeably.
Define the random function  $g_{t}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  as  $g_{t}(\mathbf{x})\coloneqq h(\mathbf{x},\mathbf{x}_{t}^{(t)})$ . From (4), we note that  $g_{t}$  is  $\mathcal{F}_{t + 1}$  measurable. From (3), we infer that the particles satisfy the following relation:
$$
\mathbf {x} _ {t + 1} ^ {(s)} = (I - \gamma g _ {t}) (\mathbf {x} _ {t} ^ {(s)}), \quad s \geq t + 1
$$
Recall that  $\mathbf{x}_{t + 1}^{(s)}|\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t)}\sim \mu_{t + 1}|\mathcal{F}_{t + 1}$  for any  $s\geq t + 1$ . Furthermore, from Equation (4), we note that for  $s\geq t + 1$ ,  $\mathbf{x}_t^{(s)}$  depends only on  $\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t - 1)}$  and  $\mathbf{x}_0^{(s)}$ . Hence, we conclude that  $\mathrm{Law}(\mathbf{x}_t^{(s)}|\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t)}) = \mathrm{Law}(\mathbf{x}_t^{(s)}|\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(t - 1)}) = \mu_t|\mathcal{F}_t$ . With this insight, the dynamics of the lower-triangular evolution in  $\mathcal{P}_2(\mathbb{R}^d)$  that the following holds almost surely:
$$
\mu_ {t + 1} \mid \mathcal {F} _ {t + 1} = (I - \gamma g _ {t}) _ {\#} \mu_ {t} \mid \mathcal {F} _ {t} \tag {5}
$$
$\mathbf{x}_t^{(t)}|\mathcal{F}_t\sim \mu_t|\mathcal{F}_t$  implies  $\mathbb{E}[g_t(\mathbf{x})|\mathcal{F}_t] = h_{\mu_t|\mathcal{F}_t}(\mathbf{x})$ . Thus lower triangular dynamics (5) is a stochastic approximation in  $\mathcal{P}_2(\mathbb{R}^d)$  to the population limit of SVGD (2). Setting the batch size to general  $K$  and tracking the evolution of the first  $KT + n$  particles, we obtain VP-SVGD (Algorithm 1).
Algorithm 1 Virtual Particle SVGD (VP-SVGD)
Input: Number of steps  $T$ , number of output particles  $n$ , batch size  $K$ , Initial positions  $\mathbf{x}_0^{(0)}, \ldots, \mathbf{x}_0^{(n+KT-1)}$  i.i.d.  $\mu_0$ , Kernel  $k$ , step size  $\gamma$ .
1: for  $t \in \{0, \dots, T - 1\}$  do
2: for  $s \in \{0, \dots, KT + n - 1\}$  do
3:  $\mathbf{x}_{t + 1}^{(s)} = \mathbf{x}_t^{(s)} - \frac{\gamma}{K}\sum_{l = 0}^{K - 1}[k(\mathbf{x}_t^{(s)},\mathbf{x}_t^{(tK + l)})\nabla F(\mathbf{x}_t^{(tK + l)}) - \nabla_2k(\mathbf{x}_t^{(s)},\mathbf{x}_t^{(tK + l)})]$
4: end for
6: Draw  $S$  uniformly at random from  $\{0,\dots ,T - 1\}$
7: Output  $(\mathbf{y}^{(0)},\dots ,\mathbf{y}^{(n - 1)}) = (\mathbf{x}_S^{(TK)},\dots ,\mathbf{x}_S^{(TK + n - 1)})$
Virtual Particles In Algorithm 1,  $(\mathbf{x}_t^{(l)})_{KT\leq l\leq KT + n - 1}$  are the real particles which constitute the output.  $(\mathbf{x}_t^{(l)})_{l <   KT}$  are virtual particles which propagate information about the probability measure  $\mu_t|F_t$  to enable computation of  $g_{t}$ , an unbiased estimate of the projected Wasserstein gradient  $h_{\mu_t|F_t}$ .
VP-SVGD as SVGD Without Replacement VP-SVGD is a without-replacement random-batch approximation of SVGD (1), where a different batch is used across timesteps, but the same batch is across particles given a fixed timestep. With i.i.d. initialization, picking the 'virtual particles' in a fixed order or from a random permutation does not change the evolution of the real particles. With this insight, we design GB-SVGD (Algorithm 2) where we consider  $n$  particles and output  $n$  particles (instead of wasting  $KT$  particles as 'virtual particles') via a random-batch approximation of SVGD.
Algorithm 2 Global Batch SVGD (GB-SVGD)
Input: # of time steps  $T$ , # of particles  $n$ ,  $\mathbf{x}_0^{(0)}, \ldots, \mathbf{x}_0^{(n-1)}$  i.i.d.  $\mu_0$ , Kernel  $k$ , step size  $\gamma$ , Batch size  $K$ , Sampling method  $\in \{ \text{with replacement, without replacement} \}$
1: for  $t \in \{0, \dots, T - 1\}$  do
2:  $\mathcal{K}_t\gets$  random subset of  $[n]$  of size  $K$  (via. sampling method)
3: for  $s \in \{0, \dots, n-1\}$  do
4:  $\mathbf{x}_{t + 1}^{(s)} = \mathbf{x}_t^{(s)} - \frac{\gamma}{K}\sum_{r\in \mathcal{K}_t}[k(\mathbf{x}_t^{(s)},\mathbf{x}_t^{(r)})\nabla F(\mathbf{x}_t^{(r)}) - \nabla_2k(\mathbf{x}_t^{(s)},\mathbf{x}_t^{(r)})]$
5: end for
6: end for
7: Draw  $S$  uniformly at random from  $\{0,1,\dots,T - 1\}$
8: Output  $(\bar{\mathbf{y}}^{(0)},\dots ,\bar{\mathbf{y}}^{(n - 1)}) = (\mathbf{x}_S^{(0)},\dots ,\mathbf{x}_S^{(n - 1)})$
In Algorithm 2, with replacement sampling means selecting a batch of  $K$  particles i.i.d. from the uniform distribution over  $[n]$ . Without replacement sampling means fixing a random permutation  $\sigma$  over  $\{0, \dots, n - 1\}$  and selecting the batches in the order specified by the permutation.
5 Assumptions
In this section, we discuss the key assumptions required for our analysis of VP-SVGD and GB-SVGD. Our first assumption is smoothness of  $F$ , which is standard in optimization and sampling.
Assumption 1 (L-Smoothness).  $\nabla F$  exists and is  $L$  Lipschitz. Moreover  $\| \nabla F(0)\| \leq \sqrt{L}$
It is easy to find a point such that  $\| \nabla F(\mathbf{x}^{*})\| \leq \sqrt{L}$  (e.g., using  $\Theta(1)$  gradient descent steps [37]) and center the initialization at  $\mu_0$  at  $\mathbf{x}^*$ . For clarity, we take  $\mathbf{x}^* = 0$  without loss of generality. We now impose the following growth condition on  $F$ .
Assumption 2 (Growth Condition). There exist  $\alpha, d_1, d_2 > 0$  such that
$$
F (\mathbf {x}) \geq d _ {1} \| \mathbf {x} \| ^ {\alpha} - d _ {2} \quad \forall \mathbf {x} \in \mathbb {R} ^ {d}
$$
Note that Assumption 1 ensures  $\alpha \leq 2$ . Assumption 2 is essentially a tail decay assumption on the target density  $\pi^{\star}(\mathbf{x}) \propto e^{-F(\mathbf{x})}$ . In fact, as we shall show in Appendix B, Assumption 2 ensures that the tails of  $\pi^{\star}$  decay as  $\propto e^{-\|\mathbf{x}\|^{\alpha}}$ . Consequently, Assumption 2 holds with  $\alpha = 2$  when  $\pi^{\star}$  is subgaussian and with  $\alpha = 1$  when  $\pi^{\star}$  is subexponential. Subgaussianity is equivalent to  $\pi^{\star}$  satisfying the  $\mathsf{T}_1$  inequality [5, 49], commonly assumed in prior works on SVGD [42, 43]. We also note that subexponentiality is implied when  $\pi^{\star}$  satisfies the Poincaré Inequality [4, Section 4], which is considered a mild condition in the sampling literature [47, 8, 2, 12, 7]. This makes Assumption 1 significantly weaker than the isoperimetric or information-transport assumptions considered in prior works.
Next, we impose a mild assumption on the RKHS of the kernel  $k$ , which has been used by several prior works [42, 26, 45, 43].
Assumption 3 (Bounded RKHS Norm). For any  $\mathbf{y} \in \mathbb{R}^d$ ,  $k(\cdot, \mathbf{y})$  satisfies  $\| k(\cdot, \mathbf{y}) \|_{\mathcal{H}_0} \leq B$ . Furthermore,  $\nabla_2 k(\cdot, \mathbf{y}) \in \mathcal{H}$  and  $\| \nabla_2 k(\cdot, \mathbf{y}) \|_{\mathcal{H}} \leq B$
Assumption 3 ensures that the adjoint operator  $P_{\mu}$ , used in Sections 2 and 3, is well-defined. We also make the following assumptions on the kernel  $k$ , which is satisfied by a large class of standard kernels such as Radial Basis Function kernels and Matérn kernels of order  $\geq 3/2$ .
Assumption 4 (Kernel Decay). The kernel  $k$  satisfies the following for constants  $A_{1}, A_{2}, A_{3} > 0$ .
$$
0 \leq k (\mathbf {x}, \mathbf {y}) \leq \frac {A _ {1}}{1 + \| \mathbf {x} - \mathbf {y} \| ^ {2}}, \quad \| \nabla_ {2} k (\mathbf {x}, \mathbf {y}) \| \leq A _ {2}, \quad \| \nabla_ {2} k (\mathbf {x}, \mathbf {y}) \| ^ {2} \leq A _ {3} k (\mathbf {x}, \mathbf {y})
$$
Finally, we make the following mild assumption on the initialization.
Assumption 5 (Initialization). The initial distribution  $\mu_0$  is such that  $\mathrm{KL}(\mu_0\| \pi^{\star}) < \infty$ . Furthermore,  $\mu_0$  is supported in  $\mathcal{B}(R)$ , the  $\ell_2$  ball of radius  $R$
Since prior works usually assume Gaussian initialization [42, 47], Assumption 5 may seem slightly non-standard. However, this is not a drawback. In fact, whenever  $R = \Theta(\sqrt{d} + \mathrm{polylog}(n / \delta))$ , Gaussian initialization can be made indistinguishable from  $\mathrm{Uniform}(\mathcal{B}(R))$  initialization, with probability at least  $1 - \delta$ , via a coupling argument. To this end, we impose Assumption 5 for ease of exposition and our results can be extended to consider Gaussian initialization. In Appendix B we show that taking  $R = \sqrt{d / L}$  and  $\mu_0 = \mathrm{Uniform}(\mathcal{B}(R))$  suffices to ensure  $\mathrm{KL}(\mu_0 \| \pi^\star) = O(d)$ .
6.1 VP-SVGD
Our first result, proved in Appendix C, shows that the law of the real particles of VP-SVGD, when conditioned on the virtual particles, is close to  $\pi^{\star}$  in KSD. As a consequence, it shows that the particles output by VP-SVGD are i.i.d. samples from a random probability measure  $\bar{\mu} (\cdot ;\mathbf{x}_0^{(0)},\dots ,\mathbf{x}_0^{(KT - 1)},S)$  which is close to  $\pi^{\star}$  in KSD. We also present a high-probability version of this result in Appendix C.
Theorem 1 (Convergence of VP-SVGD). Let  $\mu_t$  be as defined in Section 4. Let Assumptions 1, 2, 3, 4, and 5 be satisfied and let  $\gamma \leq \min \{1/2A_1L, 1/(4 + L)B\}$ . There exist  $(\zeta_i)_{0 \leq i \leq 3}$  depending polynomially
on  $A_{1}, A_{2}, A_{3}, B, L, d_{1}, d_{2}$  for any fixed  $\alpha \in (0, 2]$ , such that whenever  $\gamma \xi \leq \frac{1}{2B}$ , with  $\xi = \zeta_{0} + \zeta_{1} (\gamma T)^{1 / \alpha} + \zeta_{2} (\gamma^{2}T)^{1 / \alpha} + \zeta_{3} R^{2 / \alpha}$ , the following holds:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \mathrm {K S D} _ {\pi^ {\star}} ^ {2} \left(\mu_ {t} | \mathcal {F} _ {t} | | \pi^ {\star}\right) \right] \leq \frac {2 \mathrm {K L} \left(\mu_ {0} \| \pi^ {\star}\right)}{\gamma T} + \frac {\gamma B (4 + L) \xi^ {2}}{K}
$$
Define the probability kernel  $\bar{\mu} (\cdot ;\cdot)$  as follows: For any  $x_{\tau}\in \mathbb{R}^{d}$ ,  $\tau \in (KT)$  and  $s\in (T)$ ,  $\bar{\mu} (\cdot ;x_0,\ldots ,x_{KT - 1},s)\coloneqq \mu_s(\cdot ;x_0,\ldots ,x_{Ks - 1})$  and  $\bar{\mu} (\cdot ;x_0,\ldots ,x_{KT - 1},s = 0)\coloneqq \mu_0(\cdot)$ . Conditioned on  $\mathbf{x}_{\tau}^{(0)} = x_{\tau}$ ,  $S = s$  for every  $\tau \in (KT)$ , the outputs  $\mathbf{y}^{(0)},\dots ,\mathbf{y}^{(n - 1)}$  of VP-SVGD are i.i.d samples from  $\bar{\mu} (\cdot ;x_0,\ldots ,x_{KT - 1},s)$ . Furthermore,
$$
\mathbb {E} [ \mathrm {K S D} _ {\pi^ {\star}} ^ {2} (\bar {\mu} (\cdot ; \mathbf {x} _ {0} ^ {(0)}, \dots , \mathbf {x} _ {0} ^ {(K T - 1)}, S) | | \pi^ {\star}) ] \leq \frac {2 \mathrm {K L} (\mu_ {0} \| \pi^ {\star})}{\gamma T} + \frac {\gamma B (4 + L) \xi^ {2}}{K}
$$
Convergence Rates Taking  $\mu_0 = \mathrm{Uniform}(\mathcal{B}(R))$  with  $R = \sqrt{d / L}$  ensures KL  $(\mu_0|\mathcal{F}_0||\pi^\star) = O(d)$  (see Appendix B). Under this setting, choosing  $\gamma = O\left(\frac{(Kd)^{\eta}}{T^{1 - \eta}}\right)$  ensures that  $\mathbb{E}[\mathrm{KSD}_{\pi^{\star}}^{2}(\bar{\mu} ||\pi^{\star})] = O\left(\frac{d^{1 - \eta}}{(KT)^{\eta}}\right)$  where  $\eta = \frac{\alpha}{2(1 + \alpha)}$ . Thus, for  $\alpha = 2$ , (i.e., sub-Gaussian  $\pi^{\star}$ ),  $\mathrm{KSD}^2 = O\left(\frac{d^{2/3}}{(KT)^{1/3}}\right)$ . For  $\alpha = 1$  (i.e., sub-Exponential  $\pi^{\star}$ ), the rate (in squared KSD) becomes  $O\left(\frac{d^{3/4}}{(KT)^{1/4}}\right)$ . To the best of our knowledge, our convergence guarantee for sub-exponential  $\pi^{\star}$  is the first of its kind.
Comparison with Prior Works Salim et al. [42] analyzes population-limit SVGD for subgaussian  $\pi^{\star}$ , obtaining  $\mathrm{KSD}^2 = O(d^{3/2}/T)$  rate. We note that population-limit SVGD (which requires infinite particles) is not implementable whereas VP-SVGD is an implementable algorithm whose outputs are conditionally i.i.d samples from a distribution with guaranteed convergence to  $\pi^{\star}$ .
6.2 GB-SVGD
We now use VP-SVGD as the basis to analyze GB-SVGD. Assume  $n > KT$ . Then, with probability at least  $1 - K^2 T^2 / n$  (for with-replacement sampling) and 1 (for without-replacement sampling), the random batches  $\mathcal{K}_t$  in GB-SVGD (Algorithm 2) are disjoint and contain distinct elements. When conditioned on this event  $\mathcal{E}$ , we note that the  $n - KT$  particles that were not included in any random batch  $\mathcal{K}_t$  evolve exactly like the  $n$  real particles of VP-SVGD. With this insight, we show that, conditioned on  $\mathcal{E}$ , the outputs of VP-SVGD and GB-SVGD can be coupled such that the first  $n - KT$  particles of both the algorithms are exactly equal. This allows us to derive the following squared KSD bound between the empirical measures of the outputs of VP-SVGD and GB-SVGD. The proof of this result is presented in Appendix D.
Theorem 2 (KSD Bounds for GB-SVGD). Let  $n > KT$  and let  $\mathbf{Y} = (\mathbf{y}^{(0)},\dots,\mathbf{y}^{(n - 1)})$  and  $\bar{\mathbf{Y}} = (\bar{\mathbf{y}}^{(0)},\ldots ,\bar{\mathbf{y}}^{(n - 1)})$  denote the outputs of VP-SVGD and GB-SVGD respectively. Moreover, let  $\hat{\mu}^{(n)} = \frac{1}{n}\sum_{i = 0}^{n - 1}\delta_{\mathbf{y}^{(i)}}$  and  $\hat{\nu}^{(n)} = \frac{1}{n}\sum_{i = 0}^{n - 1}\delta_{\bar{\mathbf{y}}^{(i)}}$  denote their respective empirical measures. Under the assumptions and parameter settings of Theorem 1, there exists a coupling of  $\mathbf{Y}$  and  $\bar{\mathbf{Y}}$  such that the following holds:
$$
\mathbb {E} \left[ K S D _ {\pi^ {*}} ^ {2} \left(\hat {\nu} ^ {(n)} \mid \mid \hat {\mu} ^ {(n)}\right) \right] \leq \left\{ \begin{array}{l l} \frac {2 K ^ {2} T ^ {2} \xi^ {2}}{n ^ {2}} & \text {(w i t h o u t r e p l a c e m e n t s a m p l i n g)} \\ \frac {2 K ^ {2} T ^ {2} \xi^ {2}}{n ^ {2}} \left(1 - \frac {K ^ {2} T ^ {2}}{n}\right) + \frac {2 K ^ {2} T ^ {2} \xi^ {2}}{n} & \text {(w i t h r e p l a c e m e n t s a m p l i n g)} \end{array} \right. \tag {6}
$$
6.3 Convergence of the Empirical Measure to the Target
As a corollary of Theorem 1 and Theorem 2, we show that the empirical measure of the output of VP-SVGD and GB-SVGD rapidly converges to  $\pi^{\star}$  in KSD. We refer to Appendix E for the full statement and proof.
Corollary 1 (VP-SVGD and GB-SVGD: Fast Finite Particle Rates). Let the assumptions and parameter settings of Theorem 1 be satisfied. Let  $\hat{\mu}^{(n)}$  be the empirical measure of the  $n$  particles output by VP-SVGD, run with run with  $KT = d^{\frac{\alpha}{2 + \alpha}}$ ,  $R = \sqrt{d / L}$  and appropriately chosen  $\gamma$ . Then:
$$
\mathbb {E} \left[ \mathrm {K S D} _ {\pi^ {\star}} ^ {2} \left(\hat {\mu} ^ {(n)} | | \pi^ {\star}\right) \right] \leq O \left(\frac {d ^ {\frac {2}{2 + \alpha}}}{n ^ {\frac {\alpha}{2 + \alpha}}} + \frac {d ^ {2 / \alpha}}{n}\right)
$$
Similarly, let  $\hat{\nu}^{(n)}$  be the empirical measure of the output of GB-SVGD without replacement, run with  $KT = \sqrt{n}$ ,  $R = \sqrt{\frac{d}{L}}$  and appropriately chosen  $\gamma$ . Then, the following holds:
$$
\mathbb {E} [ \mathsf {K S D} _ {\pi^ {\star}} ^ {2} (\hat {\nu} ^ {(n)} | | \pi^ {\star}) ] \leq O (\frac {d ^ {2 / \alpha}}{n} + \frac {d ^ {\frac {1}{1 + \alpha}}}{n ^ {\frac {1 + 2 \alpha}{2 (1 + \alpha)}}} + \frac {d ^ {\frac {2 + \alpha}{2 (1 + \alpha)}}}{n ^ {\frac {\alpha}{4 (1 + \alpha)}}})
$$
Comparison to Prior Work For subgaussian  $\pi^{\star}$  (i.e.  $\alpha = 2$ ), VP-SVGD has a finite-particle rate of  $\mathbb{E}[\mathrm{KSD}_{\pi^{\star}}(\hat{\mu}^{(n)}||\pi^{\star})] = O((d / n)^{1 / 4} + (d / n)^{1 / 2})$  while that of GB-SVGD is  $\mathbb{E}[\mathrm{KSD}_{\pi^{\star}}(\hat{\nu}^{(n)}||\pi^{\star})] = O(d^{1 / 3} / n^{1 / 12} + (d / n)^{1 / 2})$ . Both these rates are a double exponential improvement over the  $\tilde{O} (\frac{\mathrm{poly}(d)}{\sqrt{\log\log n^{\Theta(1 / d)}}})$  KSD rate obtained by Shi and Mackey [43] for SVGD with subgaussian  $\pi^{\star}$ . For subexponential  $\pi^{\star}$  (i.e.  $\alpha = 1$ ) the KSD rate of VP-SVGD is  $O\left(\frac{d^{1 / 3}}{n^{1 / 6}} +\frac{d}{n^{1 / 2}}\right)$  while that of GB-SVGD is  $O\left(\frac{d^{3 / 8}}{n^{1 / 16}} +\frac{d}{n^{1 / 2}}\right)$ . To our knowledge, both these results are the first of their kind.
Oracle Complexity As illustrated in Section E.3, for subgaussian  $\pi^{\star}$ , the oracle complexity of VP-SVGD to achieve  $\epsilon$ -convergence in KSD is  $O\left(d^{4} / \epsilon^{12}\right)$  and that of GB-SVGD is  $O\left(d^{6} / \epsilon^{18}\right)$ . To our knowledge, these results are the first known oracle complexities for this problem with polynomial dimension dependence, and significantly improve upon the  $O\left(\frac{\text{poly}(d)}{\epsilon^{2}} e^{\Theta(de^{\text{poly}(d) / \epsilon^{2}})\right)$  oracle complexity of SVGD as implied by Shi and Mackey [43]. For subexponential  $\pi^{\star}$ , the oracle complexity of VP-SVGD is  $O\left(d^{6} / \epsilon^{16}\right)$  and that of GB-SVGD is  $O\left(d^{9} / \epsilon^{24}\right)$ .
We now present a sketch of our analysis. As shown in Section 4, the particles  $(\mathbf{x}_t^{(l)})_{l\geq Kt}$  are i.i.d conditioned on the filtration  $\mathcal{F}_t$ , and the random measure  $\mu_t|\mathcal{F}_t$  is the law of  $(\mathbf{x}_t^{(Kt)})$  conditioned on  $\mathbf{x}_0^{(0)},\ldots ,\mathbf{x}_0^{(Kt - 1)}$ . Moreover, from equation (5), we know that  $\mu_t|\mathcal{F}_t$  is a stochastic approximation of population limit SVGD dynamics, i.e.,  $\mu_{t + 1}|\mathcal{F}_{t + 1} = (I - \gamma g_t)_{\#}\mu_t|\mathcal{F}_t$ . Lemma 1 (similar to Salim et al. [42, Proposition 3.1] and Korba et al. [26, Proposition 5]) shows that under appropriate conditions, the KL divergence between  $\mu_t|\mathcal{F}_t$  and  $\pi^{\star}$  satisfies a (stochastic) descent lemma. Hence  $\mu_t|\mathcal{F}_t$  admits a density and KL  $(\mu_t|\mathcal{F}_t||\pi^\star)$  is almost surely finite.
Lemma 1 (Descent Lemma for  $\mu_t|F_t$ ). Let Assumptions 1, 3 and 5 be satisfied and let  $\beta > 1$  be an arbitrary constant. On the event  $\gamma \| g_t\|_{\mathcal{H}} \leq \frac{\beta - 1}{\beta B}$ , the following holds almost surely
$$
\mathsf {K L} \left(\mu_ {t + 1} | \mathcal {F} _ {t + 1} | | \pi^ {\star}\right) \leq \mathsf {K L} \left(\mu_ {t} | \mathcal {F} _ {t} | | \pi^ {\star}\right) - \gamma \left\langle h _ {\mu_ {t} | \mathcal {F} _ {t}}, g _ {t} \right\rangle_ {\mathcal {H}} + \frac {\gamma^ {2} (\beta^ {2} + L) B}{2} \| g _ {t} \| _ {\mathcal {H}} ^ {2}
$$
Lemma 1 is analogous to the noisy descent lemma which is used in the analysis of SGD for smooth functions. Notice that  $\mathbb{E}[g_t|\mathcal{F}_t] = h_{\mu_t|\mathcal{F}_t}$  (when interpreted as a Gelfand-Pettis integral [46], as discussed in Appendix B and Appendix C) and hence in expectation, the KL divergence decreases in time. In order to apply Lemma 1, we establish an almost-sure bound on  $\| g_t\|_{\mathcal{H}}$  below.
Lemma 2. Let Assumptions 1, 2, 3, 4 and 5 hold. For  $\gamma \leq 1 / 2A_{1}L$ , the following holds almost surely,
$$
\left\| g _ {t} \right\| _ {\mathcal {H}} \leq \xi = \zeta_ {0} + \zeta_ {1} (\gamma T) ^ {1 / \alpha} + \zeta_ {2} (\gamma^ {2} T) ^ {1 / \alpha} + \zeta_ {3} R ^ {2 / \alpha}
$$
where  $\zeta_0,\zeta_1,\zeta_2$  and  $\zeta_{3}$  which depend polynomially on  $A_{1},A_{2},A_{3},B,d_{1},d_{2},L$  for any fixed  $\alpha$ .
Let  $K = 1$  for clarity. To prove Lemma 2, we first note via smoothness of  $F(\cdot)$  and Assumption 3 that  $\| g_t\|_{\mathcal{H}}\leq C_0\| \mathbf{x}_t^{(t)}\| +C_1$ , and then bound  $\| \mathbf{x}_t^{(t)}\|$ . Now,  $g_{s}(\mathbf{x}) = k(\mathbf{x},\mathbf{x}_{s}^{(s)})\nabla F(\mathbf{x}_{s}^{(s)}) - \nabla_{2}k(\mathbf{x},\mathbf{x}_{s}^{(s)})$ . When  $\| \mathbf{x}_s^{(s)} - \mathbf{x}\|$  is large,  $\| g_s(\mathbf{x})\|$  is small due to decay assumptions on the kernel (Assumption 4) implying that the particle does not move much. When  $\mathbf{x}_s^{(s)}\approx \mathbf{x}$ , we have  $g_{s}(\mathbf{x})\approx k(\mathbf{x},\mathbf{x}_{s}^{(s)})\nabla F(\mathbf{x})-\nabla_{2}k(\mathbf{x},\mathbf{x}_{s}^{(s)})$  and  $k(\mathbf{x},\mathbf{x}_s^{(s)})\geq 0$ . This is approximately a gradient descent update on  $F(\cdot)$  along with a bounded term  $\nabla_{2}k(\mathbf{x},\mathbf{x}_{s}^{(s)})$ . Thus, the value of  $F(\mathbf{x}_t^{(l)})$  cannot grow too large after  $T$  iterations. By Assumption 2,  $F(\mathbf{x}_t^{(l)})$  being small implies that  $\| \mathbf{x}_t^{(l)}\|$  is small.
Equipped with Lemma 2, we set the step-size  $\gamma$  to ensure that the descent lemma (Lemma 1) always holds. The remainder of the proof involves unrolling through Lemma 1 by taking iterated expectations on both sides. To this end we control  $\left\langle h_{\mu_t|\mathcal{F}_t},g_t\right\rangle_{\mathcal{H}}$  and  $\| g_t\|_{\mathcal{H}}^2$  in expectation, in Lemma 3.
Lemma 3. Let Assumptions 1,2,3,4,5 hold and  $\xi$  be as defined in Lemma 2. Then, for  $\gamma \leq 1 / 2A_{1}L$ ,
$$
\mathbb {E} \left[ \left\langle h _ {\mu_ {t} | \mathcal {F} _ {t}}, g _ {t} \right\rangle_ {\mathcal {H}} | \mathcal {F} _ {t} \right] = \| h _ {\mu_ {t} | \mathcal {F} _ {t}} \| _ {\mathcal {H}} ^ {2} \quad a n d \quad \mathbb {E} [ \| g _ {t} \| _ {\mathcal {H}} ^ {2} ] \leq \xi^ {2} / K + \| h _ {\mu_ {t} | \mathcal {F} _ {t}} \| _ {\mathcal {H}} ^ {2}
$$
We compare the performance of GB-SVGD and SVGD. We take  $n = 100$  and use the Laplace kernel with  $h = 1$  for both. We pick the stepsize  $\gamma$  by a grid search for each algorithm. Additional details are presented in Appendix G. We observe that SVGD takes fewer iterations to converge, but the compute time for GB-SVGD is lower. This is similar to the typical behavior of stochastic optimization algorithms like SGD.
(a) MMD vs Compute Time
(b) MMD vs Iterations
Figure 1: Gaussian Experiment Comparing SVGD and GB-SVGD averaged over 10 experiments.
(a) Accuracy vs Compute Time
(b) Accuracy vs Iterations
Figure 2: Covertype Experiment, averaged over 50 runs. The error bars represent  $95\%$  CI.
Sampling from Isotropic Gaussian (Figure 1): As a sanity check, we set  $\pi^{\star} = \mathcal{N}(0,\mathbf{I})$  with  $d = 5$ . We pick  $K = 10$  for GB-SVGD. The metric of convergence is MMD with respect to the empirical measure of 1000 i.i.d. sampled Gaussians.
Bayesian Logistic Regression (Figure 2) We consider the Covertype dataset which contains  $\sim$  580,000 data points with  $d = 54$ . We consider the same priors suggested in Gershman et al. [16] and implemented in Liu and Wang [31]. We take  $K = 40$  for GB-SVGD. For both VP-SVGD and GB-SVGD, we use AdaGrad with momentum to set the step-sizes as per Liu and Wang [31]