SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The algorithm introduces a random step size  $\beta \sim \rho \propto \left(\frac{1}{h} -\frac{t}{h\kappa}\right)$  defined on  $[0,h]$  other than deterministic  $h$ . And let  $\alpha$  follows the distribution  $\rho^{\prime}\propto \left(e^{\frac{t - h}{\kappa}} - \frac{t}{h}\right)$  on  $[0,h]$ .
The analysis focuses on a Gaussian prior setting. Under a Gaussian prior, the following Lemma points out that  $\mathcal{I}$  is linear in starting point  $(\mathbf{x}_n,\mathbf{v}_n)$  and has a decoupled Brownian motion, and thus the update (4.2) can be easily implemented.
Lemma 4.1. Assume  $g(\mathbf{x}) = \frac{m}{2} \| \mathbf{x} \|^2$ . If  $\mathbf{x}_n(t)$  follows the discretized Langevin diffusion process (4.1) with starting point  $(\mathbf{x}_n, \mathbf{v}_n)$  and a given  $\widetilde{\mathbf{x}}_n$ , for any  $t > 0$ , it satisfies the integral equation
$$
\begin{array}{l} \mathbf {x} _ {n} (t) = \mathcal {J} \left(t, \widetilde {\mathbf {x}} _ {n}; \left\{\mathbf {B} _ {s} \right\} _ {0 \leq s \leq t}, \left(\mathbf {x} _ {n}, \mathbf {v} _ {n}\right)\right) = A _ {1 1} (t) \mathbf {x} _ {n} + A _ {1 2} (t) \mathbf {v} _ {n} \\ + u \int_ {0} ^ {t} A _ {1 2} (s - t) \nabla f (\widetilde {\mathbf {x}} _ {n}) \mathrm {d} s + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s}, \tag {4.3} \\ \end{array}
$$
$$
A _ {1 1} (t) = \frac {\sqrt {1 - u m} - 1}{2 \sqrt {1 - u m}} e ^ {(- 1 - \sqrt {1 - u m}) t} + \frac {\sqrt {1 - u m} + 1}{2 \sqrt {1 - u m}} e ^ {(- 1 + \sqrt {1 - u m}) t},
$$
$$
A _ {1 2} (t) = - \frac {1}{2 \sqrt {1 - u m}} e ^ {(- 1 - \sqrt {1 - u m}) t} + \frac {1}{2 \sqrt {1 - u m}} e ^ {(- 1 + \sqrt {1 - u m}) t}
$$
are deterministic functions of  $t$ . Moreover, if  $\mathbf{x}_n^* (t)$  follows the exact Langevin process (3.4) with starting point  $(\mathbf{x}_n,\mathbf{v}_n)$ , then for any  $t > 0$
$$
\begin{array}{l} \mathbf {x} _ {n} ^ {*} (t) = A _ {1 1} (t) \mathbf {x} _ {n} + A _ {1 2} (t) \mathbf {v} _ {n} + u \int_ {0} ^ {t} A _ {1 2} (s - t) \nabla f \left(\mathbf {x} _ {n} ^ {*} (s)\right) \mathrm {d} s \tag {4.4} \\ + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s}. \\ \end{array}
$$
Given the integral formula (4.3), the algorithm can be summarized as Algorithm 1. In Algorithm 1,  $A_{21}$ ,  $A_{22}$  is deterministic scalar functions, and  $\Sigma : \mathbb{R}^2 \to \mathbb{R}^{3d \times 3d}$  is also deterministic. For the explicit formula, please refer to Appendix A. The distribution of  $(H,G,W)$  is induced by the coupling between  $\{\mathbf{B}_t\}_{0 \leq t \leq \alpha}$  and  $\{\mathbf{B}_t\}_{0 \leq t \leq \beta}$  in (4.2).
4.1 Main theorem
The convergence guarantee of DRUL can be stated as the following theorem.
Theorem 4.2 (Main theorem, convergence of DRUL). For any tolerance  $\epsilon \in (0,1)$ , denote the minimizer of  $U(\mathbf{x})$  by  $\mathbf{x}_{*}$  and set the step size
$$
h \leq \min \left\{\frac {1}{1 2 C _ {2} \kappa}, \frac {\epsilon^ {2 / 3}}{\left(2 4 C _ {2} \kappa (\frac {1}{m (L + m)} \mathrm {t r} (H) + \| \mathbf {x} _ {*} \| ^ {2})\right) ^ {1 / 3}} \right\},
$$
where  $C_2 \geq 1$  is a universal constant. With initial point  $(\mathbf{x}_0, \mathbf{v}_0)$ , define  $\Omega_0 = \mathbb{E}_{\mathbf{x} \sim p, \mathbf{v} \sim \mathcal{N}(0, u)} \left( \| \mathbf{x}_0 - \mathbf{x} \|^2 + \| \mathbf{x}_0 + \mathbf{v}_0 - \mathbf{v} - \mathbf{x} \|^2 \right)$ . Then under Assumptions 3.1, when
$$
n \geq \frac {8 e \kappa}{h} \log \left(\frac {2 \mathbb {E} \Omega_ {0}}{\epsilon^ {2}}\right),
$$
Algorithm 1 outputs  $\mathbf{x}_n$  such that  $W_{2}(\mathrm{Law}(\mathbf{x}_{n}),p)\leq \epsilon$ .
Theorem 4.2 shows an overall  $\widetilde{\mathcal{O}}\left(\frac{(\mathrm{tr}(H) + \|x_{*}\|^{2})^{1 / 3}}{\epsilon^{2 / 3}}\right)$  iteration and gradient complexity to find an  $\epsilon$ -approximate sample in 2-Wasserstein distance. By pre-finding  $x_{*}$  using a convex optimization algorithm and linearly drifting the coordinates, we can assume  $x_{*} = 0$  without loss of generality. DRUL admits the stepsize reaching  $\mathcal{O}\left(\frac{\epsilon^{2 / 3}}{(\mathrm{tr}(H))^{1 / 3}}\right)$  and the overall complexity is  $\widetilde{\mathcal{O}}\left(\frac{(\mathrm{tr}(H))^{1 / 3}}{\epsilon^{2 / 3}}\right)$ .
Now we consider a realizable case where our result achieves better complexity on  $d$ . The concrete example is that  $f$  admits a separable structure as discussed in Section 3.3. Corollary 4.3 below shows that when  $f$  admits a ridge separable structure, Algorithm 1 enjoys a dimension-free iteration and gradient complexity.
Corollary 4.3. Follow the notations in Theorem 4.2. Further, if  $f(\mathbf{x})$  admits a ridge-separable form and satisfies Assumptions 3.4 and 3.5. Then if we set the step size
$$
h \leq \min \left\{\frac {1}{1 2 C _ {2} \kappa}, \frac {\epsilon^ {2 / 3}}{\left(2 4 C _ {2} \kappa (\frac {1}{m} + \| \mathbf {x} _ {*} \| ^ {2})\right) ^ {1 / 3}} \right\},
$$
With initial point  $(\mathbf{x}_0,\mathbf{v}_0)$  and  $\Omega_0 = \mathbb{E}_{\mathbf{x}\sim p,\mathbf{v}\sim \mathcal{N}(0,u)}\left(\| \mathbf{x}_0 - \mathbf{x}\|^2 +\| \mathbf{x}_0 + \mathbf{v}_0 - \mathbf{v} - \mathbf{x}\|^2\right)$ . Then when
Discussion. Theorem 4.2 establishes the convergence rate of DRUL with a weak dimension dependence. We shall note that for lots of high-dimensional sampling problems, the trace of the Hessian matrices for the potential function is much smaller than  $d$  times the largest eigenvalue because the eigenvalues often drop rapidly. In practice, this situation has been considered in lots of topics. For
(b)
Figure 1: A demonstration of the eigenvalues of the Hessian matrix. (a) The eigenvalues of the Gram matrix of MNIST data. (b) Eigenvalues of a three-layer neural network from Sagun et al. [2016].
example, one defines the effective rank (e.g. Hsu et al. [2012]) to represent the acting dimension of the data on linear models. In distributed machine learning, one can show fewer bits are needed to be transmitted between machines when the eigenvalues decrease fast (e.g. Hanzely et al. [2018]). As shown in Figure 1(a), the Gram matrix of the MNIST dataset, which is also the Hessian of Bayesian ridge linear regression, has rapidly decreasing eigenvalues. Similar empirical results are observed on deep neural network models by Sagun et al. [2016], as in Figure 1(b). In fact, beyond the ridge separable case, there are many problems admitting  $\mathrm{tr}(H) = o(d)$ , such as the concentrated posterior distributions with bounded gradients and neural networks with regularization. However, one may notice that the result of Theorem 4.2 is based on a uniform upper bound of the Hessian matrices. We think this is the basic case to show that our algorithm achieves convergence with an intrinsically low-dimension dependence. It is possible to relax the condition to the upper bound of the traces for local Hessian matrices with an additional Hessian smoothness assumption. Please see more discussions in Appendix E.
5 Intuition
In this section, we provide the intuitions of the algorithm design. For the simplicity of the notation, let  $\widehat{\mathbf{x}}_n(s) = \mathcal{J}(s,\mathbf{x}_n;\{\mathbf{B}_t\}_{0\leq t\leq s},(\mathbf{x}_n,\mathbf{v}_n))$  and  $\mathbf{x}_n(s) = \mathcal{J}(s,\mathcal{J}(\alpha ,\mathbf{x}_n;\{\mathbf{B}_t\}_{0\leq t\leq \alpha},(\mathbf{x}_n,\mathbf{v}_n));\{\mathbf{B}_t\}_{0\leq t\leq s},(\mathbf{x}_n,\mathbf{v}_n))$ . Then given  $\alpha$  and  $\beta$  in Algorithm 1,  $\mathbf{x}_n(s) = \mathcal{J}(s,\widehat{\mathbf{x}}_n(\alpha);\{\overline{\mathbf{B}}_t\}_{0\leq t\leq \beta},(\mathbf{x}_n,\mathbf{v}_n))$  and  $\mathbf{x}_n(\beta) = \mathbf{x}_{n + 1}$ .
Our analysis tracks the dynamics of the distance to the stationary distribution. Specifically, we analysis the dynamics of  $\mathbb{E}\Omega_n(t) = \mathbb{E}\| \mathbf{x}_n(t) - \mathbf{x}_n^* (t) + \mathbf{v}_n(t) - \mathbf{v}_n^* (t)\| ^2 +\mathbb{E}\| \mathbf{x}_n(t) - \mathbf{x}_n^* (t)\| ^2$  and  $\mathbb{E}\Omega_{n} = \mathbb{E}\Omega_{n}(0)$ , which upper bound the squared 2-Wasserstein distance  $W_{2}(\mathrm{Law}(\mathbf{x}_{n}(t)),p)^{2}$  and  $W_{2}(\mathrm{Law}(\mathbf{x}_{n}),p)^{2}$ , respectively. Here,  $(\mathbf{x}_n^* (t),\mathbf{v}_n^* (t))$  evolving along the exact Langevin diffusion (3.4) follows the stationary distribution and is synchronously coupled with  $(\mathbf{x}_n(t),\mathbf{v}_n(t))$ . One can find its formal definition in Appendix B. Via tracking the flow and using the contraction property of the process, the following Lemma characterizes the one-step discretization.
Lemma 5.1. Let  $\mathbf{x}_n$  be the  $n$ -step output of Algorithm 1 and  $\mathbf{x}_n^*(t)$  be defined as above. Set  $u = \frac{1}{L + m}$ . Under Assumptions 3.1, given that  $\mathbf{x}_n$  and  $\mathbf{x}_n^*$  are coupled synchronously, we have for any  $h > 0$
$$
\mathbb {E} \Omega_ {n + 1} = \mathbb {E} \mathbb {E} _ {\beta \sim \rho} \Omega_ {n} (\beta) \leq \mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} \mathbb {E} \Omega_ {n} + \mathcal {E} \tag {5.1}
$$
where  $\mathcal{E}$  is
$$
\mathcal {E} = 2 u \mathbb {E} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {\beta \sim \rho} \int_ {0} ^ {\beta} e ^ {\frac {s - \beta}{\kappa}} \langle \mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s) + \mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s), \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \mathrm {d} s.
$$
Lemma 5.1 indicates at each step,  $\mathbb{E}\Omega_{n}$  contracts with a local discretization error. Telescoping (5.1) and upper bounding the term  $\mathcal{E}$  yields the Theorem 4.2. To obtain the convergence rate in the
main theorem, we will show that Algorithm 1 guarantees an upper bound of  $\mathcal{E}$  which (1) achieves the state-of-the-art dependence on stepsize and (2) satisfies that the dimension dependence can be controlled by the contraction.
Improved discretization error. We accomplish the first goal by matching the expectation, which is detailed by Lemma 5.2.
Lemma 5.2. Let  $\rho$  be probability measure defined on  $[0,h]$  satisfying  $\rho (\mathrm{d}t)\propto \left(\frac{1}{h} -\frac{t}{h\kappa}\right)\mathrm{d}t$ , and the probability measure  $\rho^{\prime}$  defined on  $[0,h]$  satisfies  $\rho^{\prime}(\mathrm{d}t)\propto \left(e^{\frac{t - h}{\kappa}} - \frac{t}{h}\right)\mathrm{d}t$  on  $[0,h]$ . Then for measurable function  $F(t)$ ,  $\rho$  and  $\rho^{\prime}$  satisfy that
(A) There exists positive constant  $C_1$  such that  $\mathbb{E}_{t\sim \rho}\int_0^t e^{\frac{s - t}{\kappa}}F(t)\mathrm{d}s = C_1h\mathbb{E}_{t\sim \rho '}F(t)$ .
(B) There exists positive constant  $C_2$  such that  $\mathbb{E}_{t\sim \rho '}|F(t)|\leq C\mathbb{E}_{t\sim \rho}|F(t)|$
Claim (A) in Lemma 5.2 states that by choosing a random step size  $\beta \sim \rho'$ , we can leverage the low discretization error of the randomized midpoint method. Denote the random weight  $\mathbf{w}_n(s, \alpha) = \mathbf{x}_n(s) - \mathbf{x}_n^*(s) + \mathbf{v}_n(s) - \mathbf{v}_n^*(s)$ . One can split  $\mathbf{w}_n(s, \alpha)$  into  $(\mathbf{w}_n(s, \alpha) - \mathbf{w}_n(0, 0)) + \mathbf{w}_n(0, 0)$ . The former term can be bounded using the one-step move, and by claim (A), the dominating latter one is
$$
\begin{array}{l} 2 u \mathbb {E} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {\beta \sim \rho} \int_ {0} ^ {\beta} e ^ {\frac {s - \beta}{\kappa}} \langle \mathbf {w} (0, 0), \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \mathrm {d} s \\ = 2 u C _ {1} h \langle \mathbf {w} (0, 0), \mathbb {E} \mathbb {E} _ {s \sim \rho^ {\prime}} \big (\nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (s)) \big) \rangle , \\ \end{array}
$$
which attains a low discretization error given that  $\mathbb{E}_{s\sim \rho^{\prime}}\nabla f(\widehat{\mathbf{x}}_n(s))$  is a low biased approximation to  $\mathbb{E}_{s\sim \rho^{\prime}}\nabla f(\mathbf{x}_n(s))$ .
Averaged contraction can control the weight variance. Then we consider the variation of the weight  $w_{n}(s,\alpha)$ . The time difference of  $\mathbf{w}_n(s,\alpha)$  writes
$$
\begin{array}{l} \mathbf {w} (s, \alpha) - \left(\mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*}\right) = \int_ {0} ^ {s} \left(\mathbf {v} _ {n} (r) - \mathbf {v} _ {n} ^ {*} (r) - u m \left(\mathbf {x} _ {n} (r) - \mathbf {x} _ {n} ^ {*} (r)\right) \right. \tag {5.2} \\ - 2 \left(\mathbf {v} _ {n} (r) - \mathbf {v} _ {n} ^ {*} (r)\right) - u \nabla f \left(\widehat {\mathbf {x}} _ {n} (\alpha)\right) + u \nabla f \left(\mathbf {x} _ {n} ^ {*} (r)\right)) \mathrm {d} r. \\ \end{array}
$$
(5.2) indicates the variance  $\mathrm{Var}_{s\sim \mu}(\mathbf{w}(s,\alpha))$  is dimension dependent for nondegenerated distributions  $\mu$  on  $[0,h]$ , since it will introduce the  $\mathbf{v}_n(r)$  whose difference to  $\mathbf{v}_n(\beta)$  or  $\mathbf{v}_n(0)$  is dimension dependent. We control the dimension dependence via the averaged contraction. And the randomized step size makes it possible to consider the averaged effect. We have the following Lemma to bound the variation of the weight.
Lemma 5.3. Let  $\mathbf{x}_n(t),\mathbf{v}_n(t),\mathbf{v}_n^* (t)$  and  $\mathbf{v}_n^* (t)$  be defined as above and  $u = \frac{1}{L + m}$ . Under Assumption 3.1, for any  $t,\alpha \leq h$ , we have
$$
\begin{array}{l} \mathbb {E} \left\| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \right\| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} \Omega_ {n} (t) + h ^ {4} \mathbb {E} \Omega_ {n} + u ^ {2} h ^ {4} \frac {L}{m} \operatorname {t r} (H) + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2}. \tag {5.3} \\ \end{array}
$$
Now the dimension dependence of  $\mathrm{Var}(\mathbf{w}(s,\alpha))$  is contained in  $\mathbb{E}_{t\sim \rho}\Omega_n(t)$ , which can be controlled using the averaged contraction under the stochastic step size.
This paper proposes a double-randomized technique and designs the DRUL algorithm. We prove that with strongly convex and Lipschitz smooth assumptions potentials, the algorithm converges to the target distribution in Wasserstein distance in  $\widetilde{\mathcal{O}}\left(\frac{(\mathrm{tr}(H) + \|x_*\|^2)^{1/3}}{\epsilon^{2/3}}\right)$  iterations. The result illustrates that many sampling tasks in machine learning can achieve a dimension-independent complexity. The proposed DRUL algorithm can be potentially much faster than existing algorithms for high-dimensional problems. As a concrete example, when the negative log-likelihood function admits a ridge-separable structure, under mild conditions, a dimension-free  $\widetilde{\mathcal{O}}\left(\frac{1}{\epsilon^{2/3}}\right)$  iteration complexities can be obtained by DRUL. We hope our technique brings new insights for designing dimension-independent algorithms for high-dimensional sampling.
C. Fang was supported by National Key R&D Program of China (2022ZD0114902), the NSF China (No. 62376008) and Wudao Foundation. T. Zhang was supported by the General Research Fund (GRF) of Hong Kong (No. 16310222).
Jun S Liu and Jun S Liu. Monte Carlo strategies in scientific computing, volume 75. Springer, 2001.
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of markov chain monte carlo. CRC press, 2011.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Arnak S Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(3): 651-676, 2017.
Alain Durmus, Szymon Majewski, and Błajej Miasojedow. Analysis of Langevin monte carlo via convex optimization. The Journal of Machine Learning Research, 20(1):2666-2711, 2019.
Xiang Cheng and Peter Bartlett. Convergence of Langevin mcmc in kl-divergence. In Algorithmic Learning Theory, pages 186-211. PMLR, 2018.
Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted Langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems, 32, 2019.
Sinho Chewi, Murat A Erdogdu, Mufan Bill Li, Ruoqi Shen, and Matthew Zhang. Analysis of Langevin monte carlo from poincar'e to log-sobolev. arXiv preprint arXiv:2112.12662, 2021.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker-planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Alain Durmus and Eric Moulines. Sampling from a strongly log-concave distribution with the unadjusted Langevin algorithm. 2016.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Yi-An Ma, Niladri S Chatterji, Xiang Cheng, Nicolas Flammarion, Peter L Bartlett, and Michael I Jordan. Is there an analog of nesterov acceleration for gradient-based mcmc? Bernoulli, 27(3): 1942-1992, 2021.
Xiang Cheng, Niladri S Chatterji, Peter L Bartlett, and Michael I Jordan. Underdamped Langevin mcmc: A non-asymptotic analysis. In Conference on learning theory, pages 300-323. PMLR, 2018.
Matthew Zhang, Sinho Chewi, Mufan Bill Li, Krishnakumar Balasubramanian, and Murat A Erdogdu. Improved discretization analysis for underdamped Langevin monte carlo. arXiv preprint arXiv:2302.08049, 2023.
Ruoqi Shen and Yin Tat Lee. The randomized midpoint method for log-concave sampling. Advances in Neural Information Processing Systems, 32, 2019.
Andre Wibisono. Proximal Langevin algorithm: Rapid convergence under isoperimetry. arXiv preprint arXiv:1911.01469, 2019.
Ye He, Krishnakumar Balasubramanian, and Murat A Erdogdu. On the ergodicity, bias and asymptotic normality of randomized midpoint sampling method. Advances in Neural Information Processing Systems, 33:7366-7376, 2020.
Xuechen Li, Yi Wu, Lester Mackey, and Murat A Erdogdu. Stochastic runge-kutta accelerates Langevin monte carlo and beyond. Advances in neural information processing systems, 32, 2019.
Ruilin Li, Hongyuan Zha, and Molei Tao. Sqrt (d) dimension dependence of langevin monte carlo. arXiv preprint arXiv:2109.03839, 2021.
Maxime Vono, Daniel Paulin, and Arnaud Doucet. Efficient mcmc sampling with dimension-free convergence rate using admm-type splitting. The Journal of Machine Learning Research, 23(1): 1100-1168, 2022.
Yoav Freund, Yi-An Ma, and Tong Zhang. When is the convergence time of Langevin algorithms dimension independent? a composite optimization viewpoint. Journal of Machine Learning Research, 23(214):1-32, 2022.
Gareth O Roberts and Richard L Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, pages 341-363, 1996.
Jonathan C Mattingly, Andrew M Stuart, and Desmond J Higham. Ergodicity for sdes and approximations: locally lipschitz vector fields and degenerate noise. Stochastic processes and their applications, 101(2):185-232, 2002.
Arnak S Dalalyan and Lionel Riou-Durand. On sampling from a log-concave density using kinetic Langevin diffusions. Bernoulli, 26(3):1956-1988, 2020.
Alain Durmus and Eric Moulines. High-dimensional bayesian inference via the unadjusted Langevin algorithm. Bernoulli, 25(4A):2854-2882, 2019.
Andrew Gelman and Jennifer Hill. Data analysis using regression and multilevel/hierarchical models. Cambridge university press, 2006.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Jean-Yves Audibert. Fast learning rates in statistical inference through aggregation. 2009.
Wenlong Mou, Yi-An Ma, Martin J Wainwright, Peter L Bartlett, and Michael I Jordan. High-order Langevin diffusion yields an accelerated mcmc algorithm. The Journal of Machine Learning Research, 22(1):1919-1959, 2021.
Yin Tat Lee, Zhao Song, and Santosh S Vempala. Algorithmic theory of odes and sampling from well-conditioned logconcave densities. arXiv preprint arXiv:1812.06243, 2018.
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016.
Daniel Hsu, Sham M Kakade, and Tong Zhang. Random design analysis of ridge regression. In Conference on learning theory, pages 9-1. JMLR Workshop and Conference Proceedings, 2012.
Filip Hanzely, Konstantin Mishchenko, Peter Richtárik, and A.Sega: Variance reduction via gradient sketching. Advances in Neural Information Processing Systems, 31, 2018.
Gilles Hargé. A convex/log-concave correlation inequality for gaussian measure and an application to abstract wiener spaces. *Probability theory and related fields*, 130(3):415-440, 2004.
Sanjib Basu and Anirban DasGupta. The mean, median, and mode of unimodal distributions: a characterization. Theory of Probability & Its Applications, 41(2):210-223, 1997.
A Simulation of the discretization process
A.1 Integral form of the discretized process
Proof of Lemma 4.1
Proof. Define the aligned vector  $\mathbf{z}_n(s) = (\mathbf{x}_n(s),\mathbf{v}_n(s))$  and write the diffusion process (4.1) as
$$
\mathrm {d} \mathbf {z} _ {n} (s) = A \mathbf {z} _ {n} (s) \mathrm {d} s + u a \left(\widehat {\mathbf {x}} _ {n}\right) \mathrm {d} s + 2 \sqrt {u} B \mathrm {d} \widetilde {\mathbf {B}} _ {s}, \tag {A.1}
$$
$$
A = \left[ \begin{array}{c c} 0 & I \\ - u m I & - 2 I \end{array} \right], a = \left[ \begin{array}{c} 0 \\ - \nabla f (\widehat {\mathbf {x}}) \end{array} \right], B = \left[ \begin{array}{c c} 0 & 0 \\ 0 & I \end{array} \right].
$$
And  $\widetilde{\mathbf{B}}_s$  is a Brownian Motion in  $\mathbb{R}^{2d}$ . By a variation of constant method, the solution of (A.1) is
$$
\mathbf {z} _ {n} (t) = e ^ {A t} \mathbf {z} _ {n} (0) + u \left(\int_ {0} ^ {t} e ^ {- A (s - t)} \mathrm {d} s\right) a + 2 \sqrt {u} \int_ {0} ^ {t} e ^ {- A (s - t)} B \mathrm {d} \widetilde {\mathbf {B}} _ {s}
$$
To obtain the explicit expression of  $\mathbf{x}_n(t)$  and  $\mathbf{v}_n(t)$ , respectively, it suffices to present a block-wise form
$$
e ^ {A t} = \left[ \begin{array}{l l} A _ {1 1} (t) I & A _ {1 2} (t) I \\ A _ {2 1} (t) I & A _ {2 2} (t) I \end{array} \right]
$$
of the exponential  $e^{At}$ . And a direct decomposition yields that
$$
A _ {1 1} (t) = \frac {\sqrt {2 ^ {2} - 4 u m} - 2}{2 \sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 - \sqrt {2 ^ {2} - 4 u m}) t / 2} + \frac {\sqrt {2 ^ {2} - 4 u m} + 2}{2 \sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 + \sqrt {2 ^ {2} - 4 u m}) t / 2},
$$
$$
A _ {1 2} (t) = - \frac {1}{\sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 - \sqrt {2 ^ {2} - 4 u m}) t / 2} + \frac {1}{\sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 + \sqrt {2 ^ {2} - 4 u m}) t / 2}, \tag {A.2}
$$
$$
A _ {2 1} (t) = \frac {u m}{\sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 - \sqrt {2 ^ {2} - 4 u m}) t / 2} + \frac {- u m}{\sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 + \sqrt {2 ^ {2} - 4 u m}) t / 2},
$$
$$
A _ {2 2} (t) = \frac {2 + \sqrt {2 ^ {2} - 4 u m}}{2 \sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 - \sqrt {2 ^ {2} - 4 u m}) t / 2} + \frac {\sqrt {2 ^ {2} - 4 u m} - 2}{2 \sqrt {2 ^ {2} - 4 u m}} e ^ {(- 2 + \sqrt {2 ^ {2} - 4 u m}) t / 2}.
$$
Since  $\mathbf{z}_n(t) = (\mathbf{x}_n(t),\mathbf{v}_n(t))$ , we obtain the close-form solution of the discretized Langevin process of  $\mathbf{x}_n(t)$
$$
\mathbf {x} _ {n} (t) = A _ {1 1} (t) \mathbf {x} _ {n} + A _ {1 2} (t) \mathbf {v} _ {n} + u \int_ {0} ^ {h} A _ {1 2} (s - t) \nabla f (\widehat {\mathbf {x}} _ {n}) \mathrm {d} s + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s}.
$$
A similar proof can be applied to the exact diffusion process and thus  $x_{n}^{*}(t)$  satisfies
$$
\mathbf {x} _ {n} ^ {*} (t) = A _ {1 1} (t) \mathbf {x} _ {n} + A _ {1 2} (t) \mathbf {v} _ {n} + u \int_ {0} ^ {t} A _ {1 2} (s - t) \nabla f (\mathbf {x} ^ {*} (s)) \mathrm {d} s + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s}.
$$
Thus we can obtain the result in Lemma 4.1.
A.2 Controls of the integral forms