SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanshi Liu*, Cong Fang*, Tong Zhang†
* School of Intelligence Science and Technology, Peking University
† Department of Computer Science & Engineering, the Hong Kong University of Science and Technology {liu_yuanshi, fangcong} @pku.edu.cn, tongzhang@tongzhang-ml.org
This paper focuses on the high-dimensional sampling of log-concave distributions with composite structures:  $p^*(\mathrm{d}\mathbf{x}) \propto \exp(-g(\mathbf{x}) - f(\mathbf{x}))\mathrm{d}\mathbf{x}$ . We develop a double randomization technique, which leads to a fast underdamped Langevin algorithm with a dimension-independent convergence guarantee. We prove that the algorithm enjoys an overall  $\widetilde{\mathcal{O}}\left(\frac{(\mathrm{tr}(H))^{1/3}}{\epsilon^{2/3}}\right)$  iteration complexity to reach an  $\epsilon$ -tolerated sample whose distribution  $p$  admits  $W_2(p, p^*) \leq \epsilon$ . Here,  $H$  is an upper bound of the Hessian matrices for  $f$  and does not explicitly depend on dimension  $d$ . For the posterior sampling over linear models with normalized data, we show a clear superiority of convergence rate which is dimension-free and outperforms the previous best-known results by a  $d^{1/3}$  factor. The analysis to achieve a faster convergence rate brings new insights into high-dimensional sampling.
Sampling from a high-dimensional distribution serves as one of the key components in statistics, machine learning, and scientific computing, and constitutes the foundation of the fields including Bayesian statistics and generative models [Liu and Liu, 2001, Brooks et al., 2011, Song et al., 2020]. Recently, there is an emerging trend in designing provably faster Markov Chain Monte Carlo (MCMC) algorithms using techniques from first-order optimization [Dalalyan, 2017, Durmus et al., 2019, Cheng and Bartlett, 2018, Vempala and Wibisono, 2019, Chewi et al., 2021]. One typical MCMC algorithm that allows following the idea is the Langevin-type algorithms, which are of the central interest of this paper.
Langevin-type algorithms originated in statistical physics discretize a stochastic differential equation with stationary distribution corresponding to the target. For Gibbs distribution  $p(\mathbf{x}) \propto e^{-U(\mathbf{x})}$ , the standard overdamped Langevin algorithm uses Euler Maruyama scheme to discretize the diffusion process  $\mathrm{d}\mathbf{x}_t = -\nabla U(\mathbf{x}_t)\mathrm{d}t + \sqrt{2}\mathrm{d}\mathbf{B}_t$ . The algorithm iteratively performs the updates as for  $n = 0,1,2,\ldots$
$$
\mathbf {x} _ {n + 1} = \mathbf {x} _ {n} - h \nabla U (\mathbf {x} _ {n}) + \sqrt {2 h} \boldsymbol {\epsilon} _ {n}, \tag {1.1}
$$
where  $\epsilon_{n} \sim \mathcal{N}(0, I)$  follows the normal distribution and  $h > 0$  is the step size.
This paper focuses on the dimension dependence of the convergence behavior of Langevin-type algorithms. Specifically, we consider sampling problems over potentially high-dimensional and strongly log-concave distributions with a composite structure:  $p \propto e^{-U(\mathbf{x})} = e^{-g(\mathbf{x}) - f(\mathbf{x})}$  where  $g(\mathbf{x}) = \frac{m}{2}\| \mathbf{x}\|^2$ . Regarding  $p$  as a posterior distribution,  $e^{-g(\mathbf{x})}$  corresponds to a Gaussian prior and  $e^{-f(\mathbf{x})}$  corresponds to the likelihood. This structure also includes general  $m$ -strongly convex
potential functions  $U(\mathbf{x})$ , which can be split into a strongly convex term  $g(\mathbf{x}) = \frac{m}{2} \| \mathbf{x} \|^2$  and a weakly convex one  $f(\mathbf{x}) = U(\mathbf{x}) - \frac{m}{2} \| \mathbf{x} \|^2$ .
The analysis of Langevin sampling from an optimization viewpoint may date back to Jordan et al. [1998]. The viewpoint has led to a surge of works that establish quantitative convergence guarantees for overdamped Langevin algorithms to sample log-concave distributions [Dalalyan, 2017, Durmus and Moulines, 2016]. It should be noted that these convergence guarantees always involve an extra  $d$  dimension dependence due to the injection of non-negligible Gaussian noise, whereas the convergence rates of first-order optimization are often dimension-free [Nesterov, 2003]. Various accelerated methods have been proposed that can mitigate the dimension dependence and also achieve faster convergence. First, as the momentum acceleration of Langevin algorithms [Ma et al., 2021], underdamped Langevin Monte Carlo has a more stable trajectory and is known to exhibit a faster convergence rate [Cheng et al., 2018, Ma et al., 2021, Zhang et al., 2023]. Apart from using a different trajectory, more stable discretization schemes of the same diffusion process also lead to better convergence [Shen and Lee, 2019, Wibisono, 2019, He et al., 2020, Li et al., 2019]. However, the dimension dependence remains. To the best of our knowledge, the fastest randomized midpoint method for overdamped Langevin under Wasserstein distance achieves a  $\mathcal{O}\left(\frac{d^{1/3}}{\epsilon^{2/3}}\right)$  convergence rate and still has a  $d^{1/3}$  dimension dependence [Shen and Lee, 2019].
Recently another thread of works studies the convergence with weaker dimension dependence. Some researchers explore further general assumptions on the target distribution. The idea is by observing that some smoothness conditions can average out the dimension-dependent errors brought by the noise using Ito's formula. For example, Li et al. [2021] achieves a  $\sqrt{d}$  dimension dependence with a 3-rd order growth condition. Another approach investigates the curvature of the target distribution and illustrates that dimension often does not determine the complexity of the sampling problem. Vono et al. [2022] propose an ADMM-type splitting algorithm with a dimension-free convergence rate when the likelihood is separable. Along the same thread, Freund et al. [2022] study the convergence rate of Langevin algorithms with no explicit dependence on dimension and propose to characterize the convergence rate by the upper bound of Hessian matrices of  $f$ . Specifically, by letting  $H$  be the upper bound of the Hessian matrices of  $f$ , they show a variant of the overdamped Langevin algorithm achieves a convergence rate of  $\mathcal{O}\left(\frac{\mathrm{tr}(H)}{\epsilon}\right)$  in KL divergence. This result improves the rate under wide conditions because many high-dimensional sampling problems are intrinsically low-dimensional in the sense that  $\mathrm{tr}(H) = o(d)$ , which frequently appears in machine learning. For example, when the potential function has a ridge separable structure with mild conditions,  $\mathrm{tr}(H)$  can be dimension-free (see Section 3.3 for more details).
Though these works succeed in obtaining a convergence rate of the Langevin algorithm with a weak dependence on the dimension, some important questions remain:
"How to design provably faster algorithms with weak dependencies on dimension even for the log-concave sampling?"
Such a question is significant for understanding high-dimensional sampling and already includes lots of applications in real practice.
In this paper, we follow the regime of Freund et al. [2022] to design provably faster Langevin algorithms, which answers the above question affirmatively. We propose a double-randomized algorithm showing that variants of underdamped algorithms inherit the lower dimensional dependence in overdamped Langevin and a special design of random stepsize can still keep the property when using more stable discretization.
Specifically, we develop a double randomization technique and obtain the Double-Randomized Underdamped Langevin (DRUL) algorithm. We consider an averaged contraction effect to avoid dimension dependence. We design two distributions to achieve the mid-point acceleration with a randomized stepsize. DRUL is proven to enjoy an overall  $\widetilde{\mathcal{O}}\left(\frac{(\mathrm{tr}(H))^{1/3}}{\epsilon^{2/3}}\right)$  iteration and gradient complexity. For the posterior sampling over linear models, we show a clear superiority of DRUL, which achieves a dimension-free  $\widetilde{\mathcal{O}}\left(\frac{1}{\epsilon^{2/3}}\right)$  convergence rate. The novel perspective of DRUL is introducing the random step size at each update. Such a random step size combined with a random midpoint point proposed by Shen and Lee [2019] reduces the discretization error and provides new insights into designing provably faster sampling algorithms.
Table 1: Comparison of the convergence rates for most related works. We consider the convergence in Wasserstein distance to a strongly log-concave and log-Lipschitz smooth target distribution (we summarize below the methods if additional assumptions are required). EU stands for Euler-Maruyama discretization and RMM stands for Randomized Midpoint discretization. 'Ridge-Separable Case' refers to the convergence rate when  $f$  admits a ridge-separable structure (see (3.2)). Note that some results are established in the KL divergence. We convert these convergence rates into the ones in Wasserstein distance using Talagrand's inequality.
Method	Convergence Rate	Ridge-Separable Case
Overdamped as Composite OptimizCRation [Durmus et al., 2019] (Proved in KL divergence)	O(d/ε2)	O(d/ε2)
Overdamped with EU [Li et al., 2021] (With an additional linear growth condition)	O(√d/ε)	O(√d/ε)
Underdamped with RMM [Shen and Lee, 2019]	O(d1/3/ε2/3)	O(d1/3/ε2/3)
Overdamped as Composite Optimization [Freund et al., 2022] (Proved in KL divergence)	O(tr(H)+||x*||2/ε2)	O(1/ε2)
DRUL (Ours)	O((tr(H)+||x*||2)1/3/ε2/3)	O(1/ε2/3)
In summary, the contributions of the paper are listed below:
(A) We propose the Double-Randomized technique and design the DRUL algorithm.
(B) We show the DRUL converges to the target distribution in Wasserstein distance in  $\widetilde{\mathcal{O}}\left(\frac{(\mathrm{tr}(H) + \|x_*\|^2)^{1/3}}{\epsilon^{2/3}}\right)$  iterations. For posterior sampling over generalized linear models, a dimension-free  $\widetilde{\mathcal{O}}\left(\frac{1}{\epsilon^{2/3}}\right)$  complexity can be achieved.
There has been a surge of works investigating the asymptotic guarantees for the Langevin-type algorithms [Roberts and Tweedie, 1996, Mattingly et al., 2002]. And a series of recent works establish the non-asymptotic quantitative analysis framework of the Langevin-type algorithms. For performance on strongly log-concave distributions, early works [Dalalyan, 2017, Durmus and Moulines, 2016] establish the non-asymptotic convergence rate in TV distance with Lipschitz gradients assumption for overdamped dynamics. Along the same setting, similar convergence rates are achieved in KL divergence using the Wasserstein gradient flow [Cheng and Bartlett, 2018, Durmus et al., 2019] or Wasserstein distance using the contractive property for overdamped Langevin. Underdamped Langevin accelerates the vanilla Langevin algorithms [Ma et al., 2021]. With the same setting mentioned above, a faster convergence rate can be established [Cheng et al., 2018, Dalalyan and Riou-Durand, 2020, Shen and Lee, 2019, Zhang et al., 2023] for underdamped Langevin algorithms. Beyond the log Lipschitz-smooth setting, other examples also show better results can be achieved, such as Durmus and Moulines [2019], Li et al. [2019] for overdamped Langevin algorithms. Previous works also investigate the convergence rate without strongly log-convex assumptions, such as the log-Sobolev inequality (LSI) condition which is similar to the Polyak-Lojasiewicz condition in optimization. With target distributions satisfying LSI and log-Lipschitz-smooth, the results can be extended for both overdamped Langevin [Vempala and Wibisono, 2019] and underdamped Langevin [Ma et al., 2021, Zhang et al., 2023].
Another thread of works explores the dimension dependence of Langevin-type algorithms. Despite the great similarity between the analysis in Langevin algorithms and optimization, the convergence of Langevin-type algorithms depends on dimension in the log-concave setting whereas convex optimization algorithms can often achieve dimension-independent results. With general strongly log-
concave (or LSI) and log-Lipschitz-smooth condition, previous works establish a  $\mathcal{O}\left(\frac{d}{\epsilon^2}\right)$  convergence rate for overdamped algorithms [Durmus et al., 2019] and a  $\widetilde{\mathcal{O}}\left(\frac{\sqrt{d}}{\epsilon}\right)$  convergence rate for underdamped algorithms [Cheng et al., 2018] to ensure finding a solution  $x_{n}$  with  $W_{2}(\mathrm{Law}(\mathbf{x}_{n}),p)\leq \epsilon$  or  $\sqrt{KL(\mathrm{Law}(\mathbf{x}_n),p)}\leq \epsilon$ . With an additional linear growth condition on third-order derivative, Li et al. [2021] achieve a  $\widetilde{\mathcal{O}}\left(\frac{\sqrt{d}}{\epsilon}\right)$  convergence rate for overdamped Langevin. And for overdamped Langevin algorithms, Shen and Lee [2019] improve the dependence of dimension and obtains a  $\widetilde{\mathcal{O}}\left(\frac{d^{1/3}}{\epsilon^{2/3}}\right)$  convergence rate. Recent work of Freund et al. [2022] characterizes the dimensional dependence of convergence rate by the upper bound of the Hessian matrix of  $f$ . As discussed in Section 1, Freund et al. [2022] establish a  $\mathcal{O}\left(\frac{\mathrm{tr}(H)+\|\mathbf{x}_{*}\|^{2}}{\epsilon}\right)$  convergence rate in KL divergence, which implies a  $\mathcal{O}\left(\frac{\mathrm{tr}(H)+\|\mathbf{x}_{*}\|^{2}}{\epsilon^{2}}\right)$  convergence rate in Wasserstein distance. And when  $f$  admits a ridge-separable formula, Freund et al. [2022] obtain a dimension-free  $\mathcal{O}\left(\frac{1}{\epsilon^{2}}\right)$  convergence rate with some mild assumptions. We summarize the comparison of these most related works in Table 2.
3 Preliminary and problem setup
We use the convention  $\mathcal{O}(\cdot)$  and  $\Omega (\cdot)$  to denote lower and upper bounds with a universal constant.  $\widetilde{\mathcal{O}} (\cdot)$  ignores the polylogarithmic dependence. And use  $f\lesssim g$  to denote  $f = \mathcal{O}(g)$ . Use  $W_{2}(\mu ,\nu)$  to denote the 2-Wasserstein distance of distribution  $\mu$  and  $\nu$ . Use  $\operatorname {Law}(X)$  to denote the distribution of the random variable  $X$ . The Frobenius norm is denoted by  $\| \cdot \| _F$  while  $\| \cdot \| _2$  stands for operator 2-norm for matrices.
3.2 Sampling problem
We consider the distributions with a composite structure:
$$
\mathrm {d} p (\mathbf {x}) \propto \exp \{- U (\mathbf {x}) \} \mathrm {d} \mathbf {x} = \exp \left\{- g (\mathbf {x}) - f (\mathbf {x}) \right\} \mathrm {d} \mathbf {x}, \tag {3.1}
$$
where  $g(\mathbf{x}) = \frac{m}{2}\| \mathbf{x}\| ^2$  is a quadratic function. In the context of posterior sampling, the associated task is sampling from a distribution with a Gaussian prior. The composite structure also includes the general  $m$ -strongly convex function, which can be divided into  $g(\mathbf{x}) = \frac{m}{2}\| \mathbf{x}\| ^2$  and the weakly convex function  $f(\mathbf{x}) = U(\mathbf{x}) - \frac{m}{2}\| \mathbf{x}\| ^2$ . We make the following assumption on  $f$ .
Assumption 3.1.  $f\in \mathcal{C}^2$  is convex and has  $L$  -Lipschitz continuous gradients, i.e.  $\mathbf{0}\preceq \nabla^{2}f\preceq LI$
It corresponds to making  $m$ -strongly convex and  $L + m$ -Lipschitz smooth assumptions on the potential function  $U$ , which is a basic setting and widely studied in the Langevin sampling literature (see e.g. Dalalyan [2017], Cheng and Bartlett [2018], Cheng et al. [2018], Shen and Lee [2019]).
In addition to the previous assumption, we follow Freund et al. [2022] to characterize the convergence rate by a new factor  $H$  to avoid explicit dimension dependence.
Definition 3.2. Let  $H$  be an upper bound of the Hessian matrices of  $f$ , i.e.  $H\succeq \nabla^{2}f(\mathbf{x})$ .
Note that the Lipschitz smooth condition in Assumption 3.1 provides a loose bound for  $H$  since we always have  $H \preceq LI$ . This implies that  $\operatorname{tr}(H)$  will reach  $dL$  in the worst case, whereas this quantity can be much smaller than  $dL$  under wide conditions. One typical example is when  $f$  admits a so-called ridge separable structure shown below.
3.3 Example: ridge separable functions
Definition 3.3.  $f$  is said to admit the ridge separable form if
$$
f (\mathbf {x}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \left(\mathbf {a} _ {i} ^ {T} \mathbf {x}\right), \tag {3.2}
$$
where  $\sigma_{i}$  are all univariate functions, and  $\mathbf{a}_i$  are given vectors in  $\mathbb{R}^d$ .
Ridge separable functions contain many applications in machine learning, such as regression or classification over generalized linear models as well as (deep) neural networks in the neural tangent kernel regime [Gelman and Hill, 2006, Jacot et al., 2018]. We follow the argument of Freund et al. [2022], showing realizable conditions that ensure  $\operatorname{tr}(H)$  to be dimension-free.
Assumption 3.4. The function  $\sigma_{i}\in \mathcal{C}^{2}$  has a bounded second derivative, i.e.  $\sigma_i^{\prime \prime}\leq L_0$  for all  $i\in [n]$ .
Assumption 3.5. For all  $i\in [n]$ , then norm of  $\mathbf{a}_i$  is bounded by  $R$ , i.e.  $\| \mathbf{a}_i\|^2\leq R^2$
When Assumptions 3.4 and 3.5 hold, the Hessian has the upper bound  $\nabla^2 f(\mathbf{x}) = \sum_{i=1}^{n} \frac{1}{n} \sigma''(\mathbf{a}_i^T \mathbf{x}) \mathbf{a}_i \mathbf{a}_i^T \preceq \frac{L_0}{n} \sum_{i=1}^{n} \mathbf{a}_i \mathbf{a}_i^T$ . Let  $H = \frac{L_0}{n} \sum_{i=1}^{n} \mathbf{a}_i \mathbf{a}_i^T$  and
$$
\operatorname {t r} (H) = \operatorname {t r} \left(\sum_ {i = 1} ^ {n} \frac {L _ {0}}{n} \mathbf {a} _ {i} \mathbf {a} _ {i} ^ {T}\right) = \frac {L _ {0}}{n} \sum_ {i = 1} ^ {n} \| \mathbf {a} _ {i} \| ^ {2} \leq L _ {0} R ^ {2},
$$
thus illustrates that  $\operatorname{tr}(H)$  has a dimension-free  $L_0R^2$  upper bound. Meanwhile the Lipschitz constant of  $\nabla f$  can be bounded by
$$
\operatorname {L i p} (\nabla f) \leq \| H \| _ {2} = \left\| \frac {L _ {0}}{n} \sum_ {i = 1} ^ {n} \mathbf {a} _ {i} \mathbf {a} _ {i} ^ {T} \right\| \leq L _ {0} R ^ {2}.
$$
It indicates that worst-case upper bounds of  $\operatorname{tr}(H)$  and  $\operatorname{Lip}(\nabla f)$  are the same.
Note that Assumptions 3.4 and 3.5 are easy to achieve in practice. For example, consider using posterior sampling to compute the Bayes estimator over a linear model, whose advantages against maximum a posteriori estimator have been discussed broadly (see e.g. Audibert [2009]). Here,  $\mathbf{a}_i$  is associated with the data. So Assumption 3.5 can be realized by simply normalizing the data. Moreover,  $\sigma_{i}$  corresponds to the loss function and is only required to have a bounded second derivative by Assumption 3.4. Note that sampling from a ridge separable potential functions are extensively studied in related literature (see [Mou et al., 2021, Vono et al., 2022, Lee et al., 2018] as examples).
3.4 Preliminary
Overdamped Langevin. The overdamped Langevin dynamics for target distribution (3.1) is a diffusion process that evolves along the following SDE
$$
\mathrm {d} \mathbf {x} (t) = - \nabla g (\mathbf {x} (t)) \mathrm {d} t - \nabla f (\mathbf {x} (t)) \mathrm {d} t + \sqrt {2} \mathrm {d} \mathbf {B} _ {t} \tag {3.3}
$$
where  $\mathbf{B}_t$  is the standard Brownian motion. The overdamped Langevin algorithms simulate and discretize the SDE (3.3). Different algorithms vary mainly by different discretization methods.
Underdamped Langevin Underdamped Langevin algorithms accelerate the convergence rate of overdamped Langevin algorithms and instead discretize the following dynamics
$$
\mathrm {d} \mathbf {x} (t) = \mathbf {v} (t) \mathrm {d} t,
$$
$$
\mathrm {d} \mathbf {v} (t) = - u \nabla g (\mathbf {x} (t)) \mathrm {d} t - u \nabla f (\mathbf {x} (t)) \mathrm {d} t - 2 \mathbf {v} (t) \mathrm {d} t + 2 \sqrt {u} \mathrm {d} \mathbf {B} _ {t}. \tag {3.4}
$$
The diffusion process has the stationary distribution  $p_{(\mathbf{x},\mathbf{v})}\propto \exp \left\{-\frac{1}{2u}\| \mathbf{v}\| ^2 -U(\mathbf{x})\right\} .$
Contractive Property One notable property that yields the convergence of Langevin dynamics is the contractive property [Cheng and Bartlett, 2018, Cheng et al., 2018], given as follows.
Definition 3.6. A stochastic differential equation has contractive property if there exists a positive constant  $m$  and
$$
\mathbb {E} \| \mathbf {x} (t) - \mathbf {y} (t) \| ^ {2} \leq \mathbb {E} \| \mathbf {x} (0) - \mathbf {y} (0) \| ^ {2} \exp (- m t), \tag {3.5}
$$
for any pair of solutions  $\mathbf{x}(t)$  and  $\mathbf{y}(t)$  driven by the same Brownian motion.
Remark 3.7.  $\mathbb{E}\| \mathbf{x}(t) - \mathbf{y}(t)\| ^2$  is an upper bound of squared Wasserstein distance of  $\mathrm{Law}(\mathbf{x}(t))$  and  $\mathrm{Law}(\mathbf{y}(t))$ , and thus (3.5) implies a geometric convergence of the Wasserstein distance.
Contractive property can be established for both underdamped and overdamped Langevin dynamics under suitable conditions. The convergence analysis for our proposed algorithms follows a similar argument as the contractive property.
Algorithm 1 Double-Randomized Underdamped Langevin (DRUL)
Require: Iteration  $N$ , target function  $U = \frac{m}{2}\| \mathbf{x}\|^2 + f(\mathbf{x})$ , initial point  $(\mathbf{x}_0, \mathbf{v}_0)$ , max step size  $h$ ,  $u = \frac{1}{m + L}$  and  $\kappa = \frac{L + m}{m}$ .  
set  $\rho(\mathrm{d}t) \propto \left(\frac{1}{h} - \frac{t}{h\kappa}\right) \mathrm{d}t$  with support  $[0, h]$ .  
set  $\rho'(\mathrm{d}t) \propto \left(e^{\frac{t - h}{\kappa}} - \frac{t}{h}\right) \mathrm{d}t$  with support  $[0, h]$ .  
for  $n = 1, 2, \dots, N$  do  
Sample  $\alpha_n \sim \rho'$  and  $\beta_n \sim \rho$ .  
Obtain the covariance matrix  $\Sigma(\alpha_n, \beta_n)$ .  
Sample random vector  $(G, H, W)$  from distribution  $\mathcal{N}(0, \Sigma)$ .  
 $\hat{\mathbf{x}}_n \gets A_{11}(\alpha_n)\mathbf{x}_n + A_{12}(\alpha_n)\mathbf{v}_n + u\int_0^{\alpha_n}A_{12}(s - \alpha_n)\nabla f(\mathbf{x}_n) \mathrm{d}s + 2\sqrt{u}H$ .  
Update  $\mathbf{x}_{n+1} \gets A_{11}(\beta_n)\mathbf{x}_n + A_{12}(\beta_n)\mathbf{v}_n + u\int_0^{\beta_n}A_{12}(s - \beta_n)\nabla f(\hat{\mathbf{x}}_n) \mathrm{d}s + 2\sqrt{u}G$ .  
Update  $\mathbf{v}_{n+1} \gets A_{21}(\beta_n)\mathbf{x}_n + A_{22}(\beta_n)\mathbf{v}_n + u\int_0^{\beta_n}A_{22}(s - \beta_n)\nabla f(\hat{\mathbf{x}}_n) \mathrm{d}s + 2\sqrt{u}W$ .  
end for
4 Double-randomized underdamped Langevin algorithm
In this section, we introduce our double-randomized sampling method and present our main result. We will illustrate our intuition in Section 5.
The proposed algorithm is built upon the following discretization scheme given a fixed point  $\widetilde{x}_n$
$$
\begin{array}{l} \mathrm {d} \mathbf {x} _ {n} (t) = \mathbf {v} _ {n} (t) \mathrm {d} t, \\ \mathrm {d} \mathbf {v} _ {n} (t) = - u \nabla g (\mathbf {x} _ {n} (t)) \mathrm {d} t - u \nabla f (\widetilde {\mathbf {x}} _ {n}) \mathrm {d} t - 2 \mathbf {v} _ {n} (t) \mathrm {d} t + 2 \sqrt {u} \mathrm {d} \mathbf {B} _ {t}. \tag {4.1} \\ \end{array}
$$
The purpose of splitting the strongly convex part of  $U(x)$  is to avoid the md dimension dependence of  $\operatorname{tr}(H)$ . Although  $m$  is reasonably small in practice, one cannot obtain a fully dimensional-free convergence rate if the term with respect to  $g$  is discretized.
Denote the solution of process (4.1) at time  $t$  given starting point  $(\mathbf{x}_n,\mathbf{v}_n)$ , Brownian motion  $\{\mathbf{B}_t\}_{0\leq t\leq \beta}$ , step size  $\beta$  and point  $\widetilde{x}_n$  by  $\mathcal{J}(\beta ,\widetilde{\mathbf{x}}_n;\{\mathbf{B}_t\}_{0\leq t\leq \beta},(\mathbf{x}_n,\mathbf{v}_n))$ . The double-randomized algorithm performs the following one-step update:
$$
\mathbf {x} _ {k + 1} = \mathcal {J} (\beta , \mathcal {J} (\alpha , \mathbf {x} _ {n}; \left\{\mathbf {B} _ {t} \right\} _ {0 \leq t \leq \alpha}, \left(\mathbf {x} _ {n}, \mathbf {v} _ {n}\right)); \left\{\mathbf {B} _ {t} \right\} _ {0 \leq t \leq \beta}, \left(\mathbf {x} _ {n}, \mathbf {v} _ {n}\right)). \tag {4.2}
$$