SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.1. The functions  $A_{11}(t)$  and  $A_{12}(t)$  in Lemma 4.1 satisfy
$$
\left| A _ {1 2} (t) \right| \lesssim t, \quad \left| A _ {1 1} (t) - 1 \right| \lesssim t
$$
for  $t\in [0,h]$
Proof. In the proof, we will repeatedly use the inequality  $e^t - 1 = \mathcal{O}(t)$ . For notational simplicity, denote  $\sqrt{1 - um}$  by  $k$ . For the first claim,
$$
A _ {1 2} (t) = - \frac {1}{2 k} e ^ {- t - k t} + \frac {1}{2 k} e ^ {- t + k t} = e ^ {- t} \left(\frac {1 - e ^ {- k t}}{2 k} + \frac {e ^ {k t} - 1}{2 k}\right) = \mathcal {O} (t).
$$
As for the second claim, we have
$$
\begin{array}{l} A _ {1 2} (t) - 1 = \frac {k - 1}{2 k} e ^ {- t - k t} + \frac {k + 1}{2 k} e ^ {- t + k t} - 1 \\ = e ^ {- t} \left(\frac {e ^ {- k t} + e ^ {k t} - 2}{2} + \frac {1 - e ^ {- k t}}{2 k} + \frac {e ^ {k t} - 1}{2 k}\right) = \mathcal {O} (k t) + \mathcal {O} (t). \\ \end{array}
$$
Note that  $k = \sqrt{1 - um} \leq 1$  and  $h \leq \frac{1}{\kappa} \leq 1$ . When  $t \leq h$  our claims hold true.
A.3 Simulation of the Brownian motion
Lemma A.2. Let  $\mathbf{B}_t$  be a standard Brownian Motion. In Algorithm 1, we have  $G = \int_0^t A_{12}(s - t)\mathrm{d}\mathbf{B}_s$ ,  $H = \int_0^\alpha A_{12}(s - \alpha)\mathrm{d}\mathbf{B}_s$  and  $W = \int_0^t A_{22}(s - t)\mathrm{d}\mathbf{B}_s$ . Then conditioned on the randomness of  $\alpha$  and  $\beta$ ,  $(G,H,W)$  follows a mean-zero Gaussian distribution with covariance
$$
\mathbb {E} [ (G - \mathbb {E} G) (H - \mathbb {E} H) ^ {T} ] = \left(\int_ {0} ^ {\min  (t, \alpha)} A _ {1 2} (s - t) A _ {1 2} (s - \alpha) \mathrm {d} s\right) I,
$$
$$
\mathbb {E} \left[ (G - \mathbb {E} G) (G - \mathbb {E} G) ^ {T} \right] = \left(\int_ {0} ^ {t} A _ {1 2} (s - t) ^ {2} \mathrm {d} s\right) I,
$$
$$
\mathbb {E} [ (H - \mathbb {E} H) (H - \mathbb {E} H) ^ {T} ] = \left(\int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) ^ {2} \mathrm {d} s\right) I, \tag {A.3}
$$
$$
\mathbb {E} [ (W - \mathbb {E} W) (W - \mathbb {E} W) ^ {T} ] = \left(\int_ {0} ^ {t} A _ {2 2} (s - t) ^ {2} \mathrm {d} s\right) I,
$$
$$
\mathbb {E} \left[ (W - \mathbb {E} W) (H - \mathbb {E} H) ^ {T} \right] = \left(\int_ {0} ^ {\min  (\alpha , t)} A _ {1 2} (s - \alpha) A _ {2 2} (s - t) \mathrm {d} s\right) I,
$$
$$
\mathbb {E} \left[ (W - \mathbb {E} W) (G - \mathbb {E} G) ^ {T} \right] = \left(\int_ {0} ^ {t} A _ {1 2} (s - t) A _ {2 2} (s - t) \mathrm {d} s\right) I.
$$
Proof. By the properties of Brownian motion,  $(G,H,W)$  is a zero-mean Gaussian variable. Its variance is given by
$$
\begin{array}{l} \mathbb {E} \left[ (G - \mathbb {E} G) (H - \mathbb {E} H) ^ {T} \right] = \mathbb {E} \left[ \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} \right] \\ = \left(\int_ {0} ^ {\min  (t, \alpha)} A _ {1 2} (s - t) A _ {1 2} (s - \alpha) \mathrm {d} s\right) I. \\ \end{array}
$$
The rest of the claims can be proved similarly.
In this section, we present the proof of Theorem 4.2.
B.1 Contractive lemma
We begin by defining the n-step exact process  $(\mathbf{x}_n^*,\mathbf{v}_n^*)$  and  $(\mathbf{x}_n^*(t),\mathbf{v}_n^*(t))$ . Denote  $\mathrm{d}p^{*}\propto e^{-U(\mathbf{x}) - \frac{1}{2u}\| \mathbf{v}\|^2}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{v}$ . Let  $(\mathbf{x}_0^*,\mathbf{v}_0^*)\sim p^*$  and
$$
\begin{array}{l} \mathrm {d} \mathbf {x} _ {n} ^ {*} (t) = \mathbf {v} _ {n} ^ {*} (t) \mathrm {d} t, \quad \mathrm {d} \mathbf {v} _ {n} ^ {*} (t) = - u U (\mathbf {x} _ {n} ^ {*} (t)) \mathrm {d} t - 2 \mathbf {v} _ {n} ^ {*} (t) \mathrm {d} t + \sqrt {2 u} \mathrm {d} \mathbf {B} _ {t}, \tag {B.1} \\ (\mathbf {x} _ {n} ^ {*} (0), \mathbf {v} _ {n} ^ {*} (0)) = (\mathbf {x} _ {n - 1} ^ {*} (\beta_ {n - 1}), \mathbf {x} _ {n - 1} ^ {*} (\beta_ {n - 1})), \\ \end{array}
$$
For the simplicity of notation, we denote  $(\mathbf{x}_n^*,\mathbf{v}_n^*) = (\mathbf{x}_n^* (0),\mathbf{v}_n^* (0))$ . And we have for any  $n$  and  $t$ ,  $(\mathbf{x}_n^* (t),\mathbf{v}_n^* (t))\sim p^*$ . With slight abuse of notation, let  $\widehat{\mathbf{x}}_n(t)$  and  $\mathbf{x}_n(t)$  denote inaccurate starting point discretization process and midpoint process in Algorithm 1, respectively. And recall that in Algorithm 1 we have
$$
\widehat {\mathbf {x}} _ {n} (t) = A _ {1 1} (t) \mathbf {x} _ {n} + A _ {1 2} (t) \mathbf {v} _ {n} + u \int_ {0} ^ {t} A _ {1 2} (s - t) \nabla f (\mathbf {x} _ {n}) \mathrm {d} s + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} \quad \tag {B.2}
$$
$$
\mathbf {x} _ {n} (t) = A _ {1 1} (t) \mathbf {x} _ {n} + A _ {1 2} (t) \mathbf {v} _ {n} + u \int_ {0} ^ {t} A _ {1 2} (s - t) \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \mathrm {d} s + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s}. \tag {B.3}
$$
And we assume  $\widehat{\mathbf{x}}_n(t)$  is driven by the same Brownian Motion as  $\mathbf{x}_n(t)$  and  $\mathbf{x}_n^*(t)$  at the  $n$ -th step.
Define  $\Omega_{n}(t) = \| \mathbf{x}_{n}(t) - \mathbf{x}_{n}^{*}(t)\|^{2} + \| \mathbf{x}_{n}(t) - \mathbf{x}_{n}^{*}(t) + \mathbf{v}_{n}(t) - \mathbf{v}_{n}^{*}(t)\|^{2}$  and  $\Omega_{n} = \| \mathbf{x}_{n} - \mathbf{x}_{n}^{*}\|^{2} +$ $\| \mathbf{x}_n - \mathbf{x}_n^* +\mathbf{v}_n - \mathbf{v}_n^*\| ^2$  . The proof concentrates on the dynamics of the  $\Omega_{n}$  . Note that Wasserstein distance minimizes all the couplings and  $\Omega_{n}(t)\geq \| \mathbf{x}_{n}(t) - \mathbf{x}_{n}^{*}(t)\|^{2}$  . Hence  $\mathbb{E}\Omega_n$  upper bounds of the Wasserstein distance  $W_{2}(\mathrm{Law}(\mathbf{x}_{n}),p)$  . Given a synchronously coupled assumption, the dynamics of  $\Omega_{n}(t)$  can be characterized as follow.
B.1.1 Proof of Lemma 5.1
Proof. Let the random process  $\mathbf{B}_t^\alpha$  given  $\mathbf{B}_{\alpha}$  be
$$
\mathrm {d} \mathbf {B} _ {t} ^ {\alpha} = \left\{ \begin{array}{c c} \frac {\mathbf {B} _ {\alpha} - \mathbf {B} _ {t} ^ {\alpha}}{\alpha - t} + \mathrm {d} \mathbf {B} _ {t} ^ {\prime}, & 0 \leq t \leq \alpha \\ \mathrm {d} \mathbf {B} _ {t}, & t \geq \alpha \end{array} \right.
$$
where  $\mathbf{B}_t^\prime$  is an independent Brownian motion. Note that  $\mathbf{B}_t^\alpha$  is a Brownian bridge and  $\mathbf{B}_{\alpha}^{\alpha} = \mathbf{B}_{\alpha}$ . Then when  $\widehat{\mathbf{x}}_n(\alpha)$  is given and  $\mathbf{x}_n^* (t),\mathbf{x}_n(t)$  are coupled synchronously, given  $\mathbf{B}_{\alpha}$ , we have
$$
\begin{array}{l} \mathrm {d} \mathbf {x} _ {n} ^ {*} (t) = \mathbf {v} _ {n} ^ {*} (t) \mathrm {d} t, \\ \mathrm {d} \mathbf {v} _ {n} ^ {*} (t) = - u \nabla g (\mathbf {x} _ {n} ^ {*} (t)) \mathrm {d} t - u \nabla f (\mathbf {x} _ {n} ^ {*} (t)) \mathrm {d} t - 2 \mathbf {v} _ {n} ^ {*} (t) \mathrm {d} t + 2 \sqrt {u} \mathrm {d} \mathbf {B} _ {t} ^ {\alpha}, \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {d} \mathbf {x} _ {n} (t) = \mathbf {v} _ {n} (t) \mathrm {d} t, \\ \mathrm {d} \mathbf {v} _ {n} (t) = - u \nabla g (\mathbf {x} _ {n} (t)) \mathrm {d} t - u \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \mathrm {d} t - 2 \mathbf {v} _ {n} (t) \mathrm {d} t + 2 \sqrt {u} \mathrm {d} \mathbf {B} _ {t} ^ {\alpha}. \\ \end{array}
$$
For simplicity, denote  $\mathbf{z}_n(t) = \mathbf{x}_n(t) - \mathbf{x}_n^* (t)$  and  $\psi_n(t) = \mathbf{v}_n(t) - \mathbf{v}_n^* (t)$ . Then by Taylor's Theorem
$$
\nabla U \left(\mathbf {x} _ {n} (t)\right) - \nabla U \left(\mathbf {x} _ {n} ^ {*} (t)\right) = \int_ {0} ^ {1} \nabla^ {2} U \left(s \mathbf {x} _ {n} (t) + (1 - s) \mathbf {x} _ {n} ^ {*} (t)\right) d s \mathbf {z} _ {n} (t). \tag {B.4}
$$
Define  $\mathcal{H}_t = \int_0^1\nabla^2 U(s\mathbf{x}_n(t) + (1 - s)\mathbf{x}_n^* (t))\mathrm{d}s,$  then
$$
\begin{array}{l} \mathrm {d} \Omega_ {n} (t) / \mathrm {d} t = 2 \left<   \mathbf {z} _ {n} (t) + \psi_ {n} (t), \psi_ {n} (t) - u (g (\mathbf {x} _ {n} (t)) - g (\mathbf {x} _ {n} ^ {*} (t))) - u \Big (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n} ^ {*} (t)) \Big) - \gamma \psi_ {n} (t) \right> \\ + 2 \left\langle \mathbf {z} _ {n} (t), \psi_ {n} (t) \right\rangle \\ = - 2 \Big (\underbrace {\langle \mathbf {z} _ {n} (t) + \psi_ {n} (t) , (\gamma - 1) \psi_ {n} (t) + u \mathcal {H} _ {t} \mathbf {z} _ {n} (t) \rangle - \langle \mathbf {z} _ {n} (t) , \psi_ {n} (t) \rangle} _ {A} \Big) \\ + 2 u \langle \mathbf {z} _ {n} (t) + \psi_ {n} (t), \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))) \rangle , \\ \end{array}
$$
where in the second equality we use (B.4) and divide  $\nabla f(\widehat{\mathbf{x}}_n(\alpha))$  into  $\nabla f(\mathbf{x}_n(t))$  and the difference  $\nabla f(\widehat{\mathbf{x}}_n(\alpha)) - \nabla f(\mathbf{x}_n(t))$ . Note that  $A$  can be converted to a quadratic form
$$
A = [ \mathbf {z} _ {n} (t) ^ {T} + \psi_ {n} (t) ^ {T}, \mathbf {z} _ {n} (t) ^ {T} ] \left[ \begin{array}{c c} I & \frac {1}{2} u \mathcal {H} _ {t} - I \\ \frac {1}{2} u \mathcal {H} _ {t} - I & I \end{array} \right] \left[ \begin{array}{c} \mathbf {z} _ {n} (t) + \psi_ {n} (t) \\ \mathbf {z} _ {n} (t) \end{array} \right]
$$
whose eigenvalues  $\lambda_{i}$  are given by characteristic function  $(1 - \lambda)^{2} - \frac{1}{4} (u\lambda_{i}(\mathcal{H}_{t}) + um - 2)^{2} = 0$  Given the strong-convexity and Lipschitz smoothness of the potential  $U(x)$ , when  $u$  is set to be  $\frac{1}{L + m}$  the eigenvalue of  $A$  is greater than  $\frac{1}{2\kappa}$ . Therefore,
$$
\mathrm {d} \Omega_ {n} (t) / \mathrm {d} t \leq - \frac {1}{\kappa} \Omega_ {n} (t) + 2 u \langle \mathbf {z} _ {n} (t) + \psi_ {n} (t), \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))) \rangle .
$$
Then by multiplying  $e^{\frac{t}{\kappa}}$  and taking the integral form 0 to  $\beta$ , we obtain
$$
\Omega_ {n} (\beta) \leq e ^ {- \frac {\beta}{\kappa}} \Omega_ {n} (0) + 2 u \int_ {0} ^ {\beta} e ^ {\frac {s - \beta}{\kappa}} \langle \mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s) + \mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s), \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \mathrm {d} s.
$$
In Algorithm 1 the random step size  $\beta \sim \rho$  and thus we reach that
$$
\mathbb {E} \Omega_ {n + 1} \leq \mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} \mathbb {E} \Omega_ {n} + 2 u \mathbb {E} \mathbb {E} _ {\beta \sim \rho} \int_ {0} ^ {\beta} e ^ {\frac {s - \beta}{\kappa}} \langle \mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s) + \mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s), \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \mathrm {d} s.
$$
B.2 Proof of Theorem 4.2
We devote the rest of this section to the proof of the main theorem. Lemma 5.1 shows that the dynamics of the  $\mathbb{E}\Omega_{n}$  is driven by a shrinkage term and a local discretization error. To achieve the claimed convergence rate, the proof aims to establish an upper bound of local error which is: (1) dimensional-independent, (2) optimal in the sense of max step size  $h$ .
Proof. Given that  $\mathbb{E}\Omega_{n}$  is an upper bound of the squared 2-Wasserstein distance, it suffices to prove  $\mathbb{E}\Omega_{n} \leq \epsilon^{2}$ . By Lemma 5.1 and Lemma 5.2, we have
$$
\begin{array}{l} \mathbb {E} \Omega_ {n} (\beta) \leq \mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} \mathbb {E} \Omega_ {n} \\ + 2 u \mathbb {E} \mathbb {E} _ {\beta \sim \rho} \int_ {0} ^ {\beta} e ^ {\frac {s - \beta}{\kappa}} \langle \mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s) + \mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s), \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \mathrm {d} s \\ = \mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} \mathbb {E} \Omega_ {n} \\ + \underbrace {2 u h C _ {1} \mathbb {E} \mathbb {E} _ {\beta \sim \rho^ {\prime}} \left\langle \mathbf {x} _ {n} (\beta) - \mathbf {x} _ {n} ^ {*} (\beta) + \mathbf {v} _ {n} (\beta) - \mathbf {v} _ {n} ^ {*} (\beta) , \nabla f \left(\mathbf {x} _ {n} (\beta)\right) - \nabla f \left(\widehat {\mathbf {x}} _ {n} (\alpha)\right) \right\rangle} _ {A}. \tag {B.5} \\ \end{array}
$$
Next, we bound the error term  $A$  by
$$
\begin{array}{l} A = 2 C _ {1} u h \mathbb {E} \mathbb {E} _ {\beta \sim \rho^ {\prime}} \langle \mathbf {x} _ {n} (\beta) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (\beta) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (\beta) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (\beta) + \mathbf {v} _ {n} ^ {*}, \nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \\ + 2 C _ {1} u h \mathbb {E} \left\langle \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*}, \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {\beta \sim \rho^ {\prime}} \left(\nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))\right) \right\rangle \\ \lesssim \frac {u h}{u h} \mathbb {E} \| \mathbf {x} _ {n} (\beta) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (\beta) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (\beta) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (\beta) + \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ + u ^ {2} h ^ {2} \mathbb {E} \| \nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} + \frac {h}{u} u h \mathbb {E} \| \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ + \frac {u}{h} u h \mathbb {E} \| \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {\beta \sim \rho^ {\prime}} (\nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))) \| ^ {2} \\ \stackrel {a} {\lesssim} \left(h ^ {2} \mathbb {E} _ {\beta \sim_ {\rho}} \mathbb {E} \Omega_ {n} (\beta) + h ^ {4} \mathbb {E} \Omega_ {n} + \frac {u ^ {2} h ^ {4} L}{m} \operatorname {t r} (H) + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2}\right) \\ + u ^ {2} h ^ {2} \left(h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L}{m} \operatorname {t r} (H) + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2}\right) + h ^ {2} \mathbb {E} \Omega_ {n} \\ + u ^ {2} \left(h ^ {4} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {4} L}{m} \operatorname {t r} (H) + h ^ {4} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2}\right) \\ \stackrel {b} {\lesssim} h ^ {2} \mathbb {E} \mathbb {E} _ {\beta \sim \rho} \Omega_ {n} (\beta) + h ^ {2} \mathbb {E} \Omega_ {n} + \frac {u ^ {2} h ^ {4} L}{m} \mathrm {t r} (H) + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2}, \\ \end{array}
$$
where in inequality  $\stackrel{a}{\lesssim}$ , we apply Lemma D.1, Lemma 5.3 and Lemma D.5; inequality  $\stackrel{b}{\lesssim}$  follows by the observation  $uL = \frac{L}{L + m} \leq 1$ . Then by plugging the upper bound of  $A$  into (B.5), there exists constant  $C_2$  such that
$$
\mathbb {E} \Omega_ {n} (\beta) \leq \mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} \mathbb {E} \Omega_ {n} + C _ {2} \left(h ^ {2} \mathbb {E} \Omega_ {n} (\beta) + h ^ {2} \mathbb {E} \Omega_ {n} + \frac {u ^ {2} h ^ {4} L}{m} \operatorname {t r} (H) + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2}\right). \tag {B.6}
$$
(B.6) can be simplified as
$$
\mathbb {E} \Omega_ {n} (\beta) \leq \frac {\mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} + C _ {2} h ^ {2}}{1 - C _ {2} h ^ {2}} \mathbb {E} \Omega_ {n} + \frac {C _ {2} h ^ {4}}{1 - C _ {2} h ^ {2}} \left(\frac {u ^ {2} L}{m} \operatorname {t r} (H) + \| \mathbf {x} _ {*} \| ^ {2}\right). \tag {B.7}
$$
Note that when  $\beta \sim \rho$ ,  $\mathbb{E}\Omega_n(\beta) = \mathbb{E}\Omega_{n + 1}$ . And  $\mathbb{E}\Omega_{n}$  is an upper bound of Wasserstein distance. Equation (B.7) characterizes the one-step discretization of the Langevin process, where the first term indicates a linear convergence if without discretization error, and the second term is the one-step discretization error.
Without loss of generality, let  $C_2 \geq 1$ . Since  $h \leq 1 \leq \kappa$ , we have  $\mathbb{E}_{\beta \sim \rho} e^{-\frac{\beta}{\kappa}} = \frac{\kappa}{h}\left(1 - e^{\frac{h}{\kappa}}\right) \leq 1 - \frac{h}{3\kappa}$ . Since  $h \leq \frac{1}{12C_2\kappa}$ , we have  $h \leq \frac{1}{2C_2\kappa}$  and  $h \leq \frac{1}{\sqrt{C_2}}$  and therefore  $1 - C_2 h^2 \geq 0$ ,  $2C_2 h^2 \leq \frac{h}{6\kappa}$ . Hence the first term of (B.7) can be bounded by
$$
\begin{array}{l} \frac {\mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} + C _ {2} h ^ {2}}{1 - C _ {2} h ^ {2}} \mathbb {E} \Omega_ {n} \leq \frac {1 - \frac {h}{3 \kappa} + C _ {2} h ^ {2}}{1 - C _ {2} h ^ {2}} \mathbb {E} \Omega_ {n} \\ = \left(\frac {- \frac {h}{3 \kappa} + 2 C _ {2} h ^ {2}}{1 - C _ {2} h ^ {2}} + 1\right) \mathbb {E} \Omega_ {n} \tag {B.8} \\ \leq \left(\frac {- \frac {h}{3 \kappa} + \frac {h}{6 \kappa}}{1 - C _ {2} h ^ {2}} + 1\right) \mathbb {E} \Omega_ {n} \\ \leq \left(1 - \frac {h}{6 \kappa}\right) \mathbb {E} \Omega_ {n}, \\ \end{array}
$$
As for error term in (B.7), since  $h \leq \frac{1}{12C_2\kappa} \leq \frac{1}{\sqrt{2C_2}}$ , we have  $\frac{C_2h^4}{1 - C_2h^2} \leq 2C_2h^4$ . Denote  $r = 1 - \frac{h}{6\kappa}$  and  $\mathcal{E} = 2C_2h^4\left(\frac{u^2L}{m}\mathrm{tr}(H) + \| \mathbf{x}_*\|^2\right)$  and therefore
$$
\mathbb {E} \Omega_ {N} \leq r \mathbb {E} \Omega_ {N - 1} + \mathcal {E} \leq r ^ {N} \mathbb {E} \Omega_ {0} + \mathcal {E} (1 + r + \dots + r ^ {N - 1}) \leq r ^ {N} \mathbb {E} \Omega_ {0} + \frac {\mathcal {E}}{1 - r}. \tag {B.9}
$$
When  $N\geq \frac{6}{\kappa}\log \left(\frac{2\mathbb{E}\Omega_0}{\epsilon^2}\right)$ , we have
$$
r ^ {N} \mathbb {E} \Omega_ {0} \leq e ^ {\frac {N h}{6 \kappa}} \mathbb {E} \Omega_ {0} \leq \frac {\epsilon^ {2}}{2}. \tag {B.10}
$$
And when  $h \leq \frac{\epsilon^{2/3}}{\left(24C_2\kappa\left(\frac{u^2L}{m}\operatorname{tr}(H) + \|\mathbf{x}_*\|^2\right)\right)^{1/3}}$ , we get
$$
\frac {\mathcal {E}}{1 - r} \leq \frac {\epsilon^ {2}}{2}. \tag {B.11}
$$
Plugging (B.10) and (B.11) into (B.9) yields
$$
\mathbb {E} \Omega_ {N} \leq \epsilon^ {2},
$$
and therefore completes the proof.
C Supporting lemmas
We assume that Assumption 3.1 holds in this section and Section D.
C.1 Proof of Lemma 5.2
We would like to point out that how to match the random step size  $\alpha$  and  $\rho$  is critical for obtaining a low-bias estimator. And the control for local discretization error in Lemma 5.1 involves a challenging multiple integral. These considerations inspire our choice of  $\rho$  and  $\rho'$ . And in the following lemma, we summarize the main properties of  $\rho$  and  $\rho'$ .
Lemma C.1. Let  $\rho$  be probability measure defined on  $[0,h]$  satisfying  $\rho (\mathrm{d}t)\propto \left(\frac{1}{h} -\frac{t}{h\kappa}\right)\mathrm{d}t$ , and the probability measure  $\rho^{\prime}$  defined on  $[0,h]$  satisfies  $\rho^{\prime}(\mathrm{d}t)\propto \left(e^{\frac{t - h}{\kappa}} - \frac{t}{h}\right)\mathrm{d}t$  on  $[0,h]$ . Then for measurable function  $F(t)$ ,  $\rho$  and  $\rho^{\prime}$  satisfy that
(A) There exists positive constant  $C_1$  such that  $\mathbb{E}_{t\sim \rho}\int_0^t e^{\frac{s - t}{\kappa}}F(t) = C_1h\mathbb{E}_{t\sim \rho '}F(t)$ .
(B) There exists positive constant  $C_2$  such that  $\mathbb{E}_{t\sim \rho '}|F(t)|\leq C\mathbb{E}_{t\sim \rho}|F(t)|$
Proof. We begin with the first claim. By the definition of  $\rho$  and integration by parts, we obtain
$$
\begin{array}{l} \mathbb {E} _ {t \sim \text {u n i f} [ 0, 1 ]} \int_ {0} ^ {t} e ^ {\frac {s - t}{\kappa}} F (s) d s = \int_ {0} ^ {h} \frac {1}{h} \int_ {0} ^ {t} e ^ {\frac {s - t}{\kappa}} F (s) d s d t \\ = \left(\frac {t}{h} \int_ {0} ^ {t} e ^ {\frac {s - t}{\kappa}} F (s) d s\right) \Bigg | _ {0} ^ {h} - \int_ {0} ^ {h} \frac {t}{h} \left(F (t) - \frac {1}{\kappa} \int_ {0} ^ {t} e ^ {\frac {s - t}{\kappa}} F (s) d s\right) d t \\ = \int_ {0} ^ {h} e ^ {\frac {t - h}{\kappa}} F (t) \mathrm {d} t - \int_ {0} ^ {h} \frac {t}{h} F (t) \mathrm {d} t + \int_ {0} ^ {h} \frac {t}{h \kappa} \int_ {0} ^ {t} e ^ {\frac {s - t}{\kappa}} F (s) \mathrm {d} s \mathrm {d} t, \tag {C.1} \\ \end{array}
$$
$$
\int_ {0} ^ {h} \left(\frac {1}{h} - \frac {t}{h \kappa}\right) \int_ {0} ^ {t} e ^ {\frac {s - t}{\kappa}} F (s) d s d t = \int_ {0} ^ {h} \left(e ^ {t - h} \kappa - \frac {t}{h}\right) F (t) d t.
$$
$\rho \propto \frac{1}{h} - \frac{t}{h \kappa}$  has a normalizing constant which is of order  $\Theta(1)$ . Denote the normalizing constant of  $\int_0^h \left(e^{\frac{t - h}{\kappa}} - \frac{t}{h}\right) \, \mathrm{d}s$  by a scalar function  $Z'(h)$ . To obtain the first claim, it suffices to show  $Z'(h)$  is of order  $\mathcal{O}(h)$ . Then given that  $e^{-\frac{h}{\kappa}} \geq 1 - \frac{h}{\kappa}$ ,  $Z'(h)$  satisfies
$$
Z ^ {\prime} (h) = \int_ {0} ^ {h} \left(e ^ {\frac {t - h}{\kappa}} - \frac {t}{h}\right) d t = \kappa \left(1 - e ^ {- \frac {h}{\kappa}}\right) - \frac {h}{2} \leq h - \frac {h}{2} = \frac {h}{2},
$$
which proves the first claim.
Then we prove the second claim of the proposition. Note that when  $h \leq \kappa$
$$
Z ^ {\prime} (h) = \kappa \left(1 - e ^ {- \frac {h}{\kappa}}\right) - \frac {h}{2} \geq \left(1 - \frac {1}{e}\right) h - \frac {h}{2} = \left(\frac {1}{2} - \frac {1}{e}\right) h.
$$
$$
\frac {1}{Z ^ {\prime} (h)} \left(e ^ {\frac {t - h}{\kappa}} - \frac {t}{h}\right) \leq \frac {1}{Z ^ {\prime} (h)} \lesssim \frac {1}{h}. \tag {C.2}
$$
As for  $\rho$ , denote the normalizing constant of  $\rho$  by  $Z(h) = \int_0^h \frac{1}{h} - \frac{t}{h \kappa}$ . Since  $h = o(1)$  and  $\kappa > 1$ ,
$$
Z (h) = 1 - \frac {h}{2 \kappa} = \Theta (1).
$$
$$
\frac {1}{Z (h)} \left(\frac {1}{h} - \frac {t}{j \kappa}\right) \gtrsim \frac {1}{h} \left(1 - \frac {t}{\kappa}\right) \gtrsim \frac {1}{h}, \tag {C.3}
$$
where the last inequality follows by that  $t \leq h = 0(1)$  and  $\kappa > 1$ . Combining (C.2) and (C.3) yields Claim (B).
C.2 Convergence lemma
Lemma C.2. Let  $\mathbf{x}_{*}$  be the minimizer of the potential function, that is,  $\mathbf{x}_{*} = \arg \min \{g(\mathbf{x}) + f(\mathbf{x})\}$ , and  $A$  is a positive definite matrix that  $mI \preceq A \preceq LI$ . Let  $\mathbf{x}_n(t)$  and  $\mathbf{x}_n^*(t)$  be defined as (B.3) and (B.1). Assume that  $u = \frac{1}{L + m}$ . Then we have the following bound
$$
\mathbb {E} \| A \mathbf {x} _ {n} (t) \| ^ {2} \lesssim L ^ {2} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) \| ^ {2} + \frac {1}{m} \mathrm {t r} (A ^ {2}) + L ^ {2} \| \mathbf {x} _ {*} \| ^ {2},
$$
$$
\mathbb {E} \| A (\mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t)) \| ^ {2} \lesssim L ^ {2} \mathbb {E} \| \mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) - \mathbf {v} _ {n} ^ {*} (t) \| ^ {2} + \frac {1}{m} \mathrm {t r} (A ^ {2}) + L ^ {2} \| \mathbf {x} _ {*} \| ^ {2},
$$
$$
\mathbb {E} \| A \nabla f (\mathbf {x} _ {n} (t)) \| ^ {2} \lesssim L ^ {4} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) \| ^ {2} + \frac {L ^ {2}}{m} \mathrm {t r} (A ^ {2}) + L ^ {2} m ^ {2} \| \mathbf {x} _ {*} \| ^ {2}.
$$
Proof. We begin with the first bound. By Young's inequality and Lipschitz smoothness of  $f$ ,
$$
\begin{array}{l} \mathbb {E} \| A \mathbf {x} _ {n} (t) \| ^ {2} \lesssim \mathbb {E} \| A \left(\mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t)\right) \| ^ {2} + \mathbb {E} \| A \left(\mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) \| ^ {2} + \| A \mathbf {x} _ {*} \| ^ {2} \tag {C.4} \\ \leq L ^ {2} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) \| ^ {2} + \mathbb {E} \| A \left(\mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) \| ^ {2} + L ^ {2} \| \mathbf {x} _ {*} \| ^ {2}. \\ \end{array}
$$
Since  $U(\cdot)$  is m strongly convex and  $\| H\mathbf{x}\|$  is a convex function with respect to  $\mathbf{x}$ , Theorem 1.1 of Hargé [2004] implies that
$$
\mathbb {E} \| A \left(\mathbf {x} _ {n} ^ {*} (t) - \mathbb {E} \mathbf {x} _ {n} ^ {*}\right) \| ^ {2} \leq \mathbb {E} _ {\mathbf {x} \sim \mathcal {N} \left(0, \frac {1}{m}\right)} \| A \mathbf {x} \| ^ {2} = \frac {1}{m} \operatorname {t r} \left(A ^ {2}\right). \tag {C.5}
$$
On the other hand, by Theorem 7 of Basu and DasGupta [1997],