SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Plugging (D.11) and (D.9) in to control (D.8) yields
$$
\begin{array}{l} \mathbb {E} \left. \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s}\right) \right\| ^ {2} \\ \lesssim h ^ {3} \mathbb {E} \operatorname {t r} \left(\left(\mathbb {E} [ \bar {\mathcal {H}} | \mathcal {F} _ {n} ]\right) ^ {2}\right) + L _ {2} ^ {2} u h ^ {6} d ^ {2} \\ \leq h ^ {3} L M + L _ {2} ^ {2} u h ^ {6} d ^ {2}, \\ \end{array}
$$
where the last inequality follows by Assumption E.2 and the observation that  $\mathbb{E}\mathrm{tr}\left(\left(\mathbb{E}[\bar{\mathcal{H}} |\mathcal{F}_n]\right)^2\right)\leq$ $L\mathbb{E}\mathbb{E}[\mathrm{tr}\left(\bar{\mathcal{H}}\right)|\mathcal{F}_n]\leq LM.$
Lemma D.3. Let  $\bar{\mathcal{H}}$  and  $\bar{\mathcal{H}}'$  be defined as above. When Assumptions E.2 and E.1 hold, we have
$$
u L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \lesssim u L ^ {3} h ^ {3} M + L _ {2} ^ {2} u ^ {2} L ^ {2} h ^ {6} d ^ {2}.
$$
Proof. Similar to the proof of Lemma D.2, we have
$$
\begin{array}{l} u L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \\ = u L ^ {2} \mathbb {E} \left\| \bar {\mathcal {H}} ^ {\prime} \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \\ \lesssim u L ^ {2} \mathbb {E} \left\| \left(\bar {\mathcal {H}} ^ {\prime} - \mathbb {E} \left[ \bar {\mathcal {H}} ^ {\prime} | \mathcal {F} _ {n} \right]\right) \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} + u L ^ {2} \mathbb {E} \left\| \mathbb {E} \left[ \bar {\mathcal {H}} ^ {\prime} | \mathcal {F} _ {n} \right] \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \\ \lesssim u L ^ {2} \mathbb {E} \left(\left\| \bar {\mathcal {H}} ^ {\prime} - \mathbb {E} \left[ \bar {\mathcal {H}} ^ {\prime} \mid \mathcal {F} _ {n} \right] \right\| ^ {2} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2}\right) + u L ^ {2} h ^ {3} \mathbb {E} \operatorname {t r} \left(\left(\mathbb {E} \left[ \bar {\mathcal {H}} ^ {\prime} \mid \mathcal {F} _ {n} \right]\right) ^ {2}\right). \tag {D.12} \\ \end{array}
$$
We follow the notation of  $\mathbf{B}_s^{\prime}$  and  $\hat{\mathbf{x}}_n^\prime (\alpha)$  in (D.10). Then
$$
\begin{array}{l} \mathbb {E} \left(\left\| \bar {\mathcal {H}} ^ {\prime} - \mathbb {E} \left[ \bar {\mathcal {H}} ^ {\prime} | \mathcal {F} _ {n} \right] \right\| ^ {2} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2}\right) \\ \lesssim \mathbb {E} \left(L _ {2} ^ {2} \int_ {0} ^ {1} (1 - k) ^ {2} \| \widehat {\mathbf {x}} _ {n} (\alpha) - \widehat {\mathbf {x}} _ {n} ^ {\prime} (\alpha) \| ^ {2} d k \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) d \mathbf {B} _ {t} \right\| ^ {2}\right) \\ \lesssim L _ {2} ^ {2} \mathbb {E} \left(u \left\| \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} ^ {\prime} \right\| ^ {2} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2}\right) \tag {D.13} \\ \lesssim L _ {2} ^ {2} u \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} \right\| ^ {4} \\ \lesssim L _ {2} ^ {2} u h ^ {6} d ^ {2}. \\ \end{array}
$$
Combining (D.12) and (D.13) and using  $\operatorname{tr}\left(\mathbb{E}[\widetilde{\mathcal{H}}'|\mathcal{F}_n]\right) \leq LM$  yields
D.2 Bound of  $\mathbb{E}\| \mathbf{x}_n(t) - \mathbf{x}_n - \mathbf{x}_n^* (t) + \mathbf{x}_n^* +\mathbf{v}_n(t) - \mathbf{v}_n - \mathbf{v}_n^* (t) + \mathbf{v}_n^*\|$
To prove Lemma 5.3, we consider the following Lemma which also includes the Hessian smooth case.
Lemma D.4. Let  $\mathbf{x}_n(t),\mathbf{v}_n(t),\mathbf{v}_n^* (t)$  and  $\mathbf{v}_n^* (t)$  be defined as above and  $u = \frac{1}{L + m}$ .
(i) Under Assumption 3.1, for any  $t,\alpha \leq h$  , we have
$$
\begin{array}{l} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} \Omega_ {n} (t) + h ^ {4} \mathbb {E} \Omega_ {n} + u ^ {2} h ^ {4} \frac {L}{m} \operatorname {t r} (H) + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2}. \tag {D.14} \\ \end{array}
$$
(ii) Under Assumptions 3.1, E.1 and E.2, for any  $t,\alpha \leq h$  , we have
$$
\begin{array}{l} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} \mathbb {E} _ {t \sim \rho} \Omega_ {n} (t) + h ^ {4} \mathbb {E} \Omega_ {n} + u ^ {2} \frac {h ^ {4} L M}{m} + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {4} h ^ {8} d ^ {2}. \\ \end{array}
$$
Proof. Since the discretized process and the continuous process are synchronously coupled,
$$
\begin{array}{l} \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \\ = \int_ {0} ^ {t} \left(\mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s) - u m \left(\mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s)\right) - 2 \left(\mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s)\right) - u \nabla f \left(\widehat {\mathbf {x}} _ {n} (\alpha)\right) + u \nabla f \left(\mathbf {x} _ {n} ^ {*} (s)\right)\right) d s. \tag {D.15} \\ \end{array}
$$
Hence the square norm satisfies
$$
\begin{array}{l} \left\| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \right\| ^ {2} \\ \stackrel {\mathrm {a}} {\leq} \left(3 t \int_ {0} ^ {t} \| \mathbf {v} _ {n} (s) - \mathbf {v} _ {n} ^ {*} (s) + \mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s) \| ^ {2} \mathrm {d} s + 3 (1 - u m) ^ {2} t \int_ {0} ^ {t} \| \mathbf {x} _ {n} (s) - \mathbf {x} _ {n} ^ {*} (s) \| ^ {2} \mathrm {d} s \right. \\ + 3 u ^ {2} t \int_ {0} ^ {t} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n} ^ {*} (s)) \| ^ {2} \mathrm {d} s) \\ \leq 3 t \int_ {0} ^ {t} \Omega_ {n} (s) \mathrm {d} s + 3 u ^ {2} t \int_ {0} ^ {t} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f \left(\mathbf {x} _ {n} ^ {*} (s)\right) \| ^ {2} \mathrm {d} s, \tag {D.16} \\ \end{array}
$$
where  $\stackrel{\mathrm{a}}{\leq}$  follows by equation [D.15] and Jensen's inequality; in  $\stackrel{\mathrm{b}}{\leq}$  we use  $|1 - um| \leq 1$  and substitute the square norm by  $\Omega_{n}(s)$ . To bound the second term in the last inequality, we have
$$
\begin{array}{l} \int_ {0} ^ {t} \left\| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n} ^ {*} (s)) \right\| ^ {2} \mathrm {d} s \\ \lesssim \int_ {0} ^ {t} \| \nabla f (\mathbf {x} _ {n} (s)) - \nabla f \left(\mathbf {x} _ {n} ^ {*} (s)\right) \| ^ {2} d s + \int_ {0} ^ {t} \| \nabla f \left(\widehat {\mathbf {x}} _ {n} (\alpha)\right) - \nabla f \left(\mathbf {x} _ {n} (s)\right) \| ^ {2} d s \tag {D.17} \\ \leq L ^ {2} \int_ {0} ^ {t} \Omega_ {n} (s) \mathrm {d} s + \int_ {0} ^ {t} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n} (s)) \| ^ {2} \mathrm {d} s, \\ \end{array}
$$
where in the second inequality, we use the Lipschitz smoothness of  $f$  and  $\| \mathbf{x}_n(t) - \mathbf{x}_n^* (t)\| \leq \Omega_n(t)$ . By combining bound (D.16) and (D.17), we arrive at that
$$
\begin{array}{l} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ \lesssim t \int_ {0} ^ {t} \mathbb {E} \Omega_ {n} (s) \mathrm {d} s + u ^ {2} t \int_ {0} ^ {t} \mathbb {E} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n} (s)) \| ^ {2} \mathrm {d} s \tag {D.18} \\ \leq h ^ {2} \int_ {0} ^ {h} \frac {1}{h} \mathbb {E} \Omega_ {n} (t) \mathrm {d} t + u ^ {2} h \int_ {0} ^ {h} \mathbb {E} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n} (s)) \| ^ {2} \mathrm {d} s. \\ \end{array}
$$
By (C.3), we have  $h^2 \int_0^h \frac{1}{h} \mathbb{E} \Omega_n(t) \, \mathrm{d}t = h^2 \mathbb{E}_{t \sim \mathrm{unif}[0,h]} \mathbb{E} \Omega_n(t) \lesssim h^2 \mathbb{E}_{t \sim \rho} \mathbb{E} \Omega_n(t)$ .
When the Hessian of  $f$  is upper bounded by  $H$ , by (D.6), we have
$$
\begin{array}{l} \mathbb {E} \left\| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \right\| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} _ {t \sim \rho} \mathbb {E} \Omega_ {n} (t) + u ^ {2} h ^ {2} \left(h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L}{m} \operatorname {t r} (H) + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2}\right) \\ \lesssim h ^ {2} \mathbb {E} \mathbb {E} _ {t \sim \rho} \Omega_ {n} (t) + h ^ {4} \mathbb {E} \Omega_ {n} + u ^ {2} \frac {h ^ {4} L}{m} \mathrm {t r} (H) + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2}, \\ \end{array}
$$
which completes the first claim.
When the  $f$  is  $L_{2}$ -Hessian smooth, by (D.7), we have
$$
\begin{array}{l} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (t) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (t) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} _ {t \sim \rho} \mathbb {E} \Omega_ {n} (t) + u ^ {2} h ^ {2} \left(h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L M}{m} + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {6} d ^ {2}\right) \\ \lesssim h ^ {2} \mathbb {E} \mathbb {E} _ {t \sim \rho} \Omega_ {n} (t) + h ^ {4} \mathbb {E} \Omega_ {n} + u ^ {2} \frac {h ^ {4} L M}{m} + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {4} h ^ {8} d ^ {2}, \\ \end{array}
$$
which establishes the second claim.
D.3 Bound of  $\mathbb{E}\| \mathbb{E}_{\beta \sim \rho '}\mathbb{E}_{\alpha \sim \rho '}\big(\nabla f(\mathbf{x}_n(\beta)) - \nabla f(\widehat{\mathbf{x}}_n(\alpha))\big)\| ^2$
Lemma D.5. Let  $\mathbf{x}_n(t)$  and  $\widehat{\mathbf{x}}_n(t)$  be defined as above and  $u = \frac{1}{L + m}$ . Then,
(i) When Assumption 3.1 holds, and  $H$  be defined as Definition 3.2. Then
$$
\mathbb {E} \| \mathbb {E} _ {\beta \sim \rho^ {\prime}} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \big (\nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \big) \| ^ {2} \lesssim h ^ {6} L ^ {2} \Omega_ {n} + \frac {h ^ {6} L}{m} \mathrm {t r} (H) + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2}.
$$
$$
\begin{array}{l} \mathbb {E} \left\| \mathbb {E} _ {\beta \sim \rho^ {\prime}} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \nabla \big (f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \big) \right\| ^ {2} \\ \lesssim h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6} L}{m} M + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {1 0} d ^ {2}. \\ \end{array}
$$
Proof. Note that  $\mathbf{x}_n(t)$  depends on  $\alpha$ . First, we have
$$
\begin{array}{l} \mathbb {E} \| \mathbb {E} _ {\beta \sim \rho^ {\prime}} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \nabla (f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))) \| ^ {2} \\ = \mathbb {E} \| \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \left(\mathbb {E} _ {\beta \sim \rho^ {\prime}} \nabla f (\mathbf {x} _ {n} (\beta)) - \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))\right) \| ^ {2} \\ = \mathbb {E} \| \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {s \sim \rho^ {\prime}} \left(\nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (s))\right) \| ^ {2} \\ \leq \mathbb {E} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {s \sim \rho^ {\prime}} \| \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (s)) \| ^ {2} \\ = \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {s \sim \rho^ {\prime}} \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (s)) \| ^ {2} | \alpha , s \right]. \\ \end{array}
$$
Define  $\bar{\mathcal{H}}'' = \int_0^1 \nabla^2 f(\mathbf{x}_n(s) + k(\widehat{\mathbf{x}}_n(s) - \mathbf{x}_n(s)))$  and recall that  $\bar{\mathcal{H}}' = \int_0^1 \nabla^2 f(\widehat{\mathbf{x}}_n(\alpha) + k(\mathbf{x}_n - \widehat{\mathbf{x}}_n(\alpha))) \, \mathrm{d}k$ . And then  $\mathbb{E}\left[\|\nabla f(\mathbf{x}_n(s)) - \nabla f(\widehat{\mathbf{x}}_n(s))\|^2 | \alpha, s\right]$  can be bounded by
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {n} (s)) - \nabla f (\widehat {\mathbf {x}} _ {n} (s)) \| ^ {2} | \alpha , s \right] \\ = \mathbb {E} \left[ \| \bar {\mathcal {H}} ^ {\prime \prime} \left(\mathbf {x} _ {n} (s) - \widehat {\mathbf {x}} _ {n} (s)\right) \| ^ {2} | \alpha , s \right] \\ \stackrel {\mathrm {a}} {=} \mathbb {E} \left[ \left. \left\| u \bar {\mathcal {H}} ^ {\prime \prime} \int_ {0} ^ {s} A _ {1 2} (t - s) (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n})) \mathrm {d} t \right\| ^ {2} \right\rvert   \alpha , s \right] \\ \stackrel {{b}} {\leq} u ^ {4} h ^ {4} L ^ {2} \mathbb {E} \left[ \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n}) \| ^ {2} | \alpha , s \right], \\ \end{array}
$$
where  $\stackrel{\mathrm{a}}{=}^{\mathrm{b}}$  is by plugging in the close-form solution of  $\mathbf{x}_n(s)$  and  $\widehat{\mathbf{x}}_n(s)$ ;  $\leq$  follows by  $A_{12}(t - s) \lesssim h$  and the Lipschitz continuous of  $\nabla f$ .
$$
\mathbb {E} \| \mathbb {E} _ {\beta \sim \rho^ {\prime}} \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \nabla (f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))) \| ^ {2}
$$
$$
\lesssim u ^ {2} h ^ {4} L ^ {2} \mathbb {E} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n}) \| ^ {2}
$$
$$
\stackrel {\mathrm {a}} {\lesssim} u ^ {2} h ^ {4} \left(h ^ {2} L ^ {4} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L ^ {2}}{m} \mathbb {E} \operatorname {t r} \left(\bar {\mathcal {H}} ^ {\prime 2}\right) + h ^ {2} L ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + u L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2}\right)
$$
$$
\lesssim h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6}}{m} \mathbb {E} \mathrm {t r} \left(\bar {\mathcal {H}} ^ {\prime 2}\right) + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + u ^ {3} h ^ {4} L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2},
$$
where  $\lesssim$  follows by (D.4). When  $\nabla^2 f$  has a uniform upper bound  $H$ , we have
$$
\lesssim h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6}}{m} \mathbb {E} \operatorname {t r} \left(\bar {\mathcal {H}} ^ {\prime 2}\right) + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + u h ^ {7} \operatorname {t r} \left(H ^ {2}\right)
$$
$$
\lesssim h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6} L}{m} \operatorname {t r} (H) + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2},
$$
where in the last inequality, we use that  $\mathbb{E}\mathrm{tr}(\bar{\mathcal{H}}^{\prime 2})\leq L\mathbb{E}\mathrm{tr}(\bar{\mathcal{H}}^{\prime})\leq L\mathrm{tr}(H)$  and  $\mathrm{tr}(H^2)\leq L\mathrm{tr}(H)$ .
And when Assumptions E.1 and E.2 hold, by Lemma D.3,
$$
\lesssim h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6}}{m} \mathbb {E} \mathrm {t r} \left(\bar {\mathcal {H}} ^ {\prime 2}\right) + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + u h ^ {7} L M + L _ {2} ^ {2} u ^ {2} h ^ {1 0} d ^ {2}
$$
$$
\lesssim h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6} L}{m} M + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {1 0} d ^ {2}.
$$
E Local Hessian bound results
As discussed in Section 4.1, here we show that the uniform Hessian upper bound can be relaxed to local trace control with an additional Hessian smooth assumption. Specifically, we make the following assumptions on  $f$ .
Assumption E.1.  $f$  has a  $L_{2}$ -Lipschitz Hessian.
Assumption E.1 is a standard assumption that frequently appears in the literature of optimization and sampling [Ma et al., 2021, Dalalyan and Riou-Durand, 2020, Durmus and Moulines, 2019].
Assumption E.2. There exist  $M > 0$  such that for all  $\mathbf{x} \in \mathbb{R}$ ,  $\mathrm{tr}(\nabla^2 f(\mathbf{x})) \leq M$ .
Assumption E.2 relax the uniform Hessian bound  $\nabla^2 f(\mathbf{x})\preceq H$  for  $\mathbf{x}\in \mathbb{R}^d$  under which we obtain a convergence rate with  $\mathrm{tr}(H)$ . Assumption E.2 only requires that the trace of Hessian can be controlled locally. Under Assumptions E.1 and E.2, the convergence of DRUL only has a weak dimension dependency, as stated in Theorem E.3.
Theorem E.3. For any tolerance  $\epsilon \in (0,1)$ , denote the minimizer of  $U(\mathbf{x})$  by  $\mathbf{x}_{*}$ . Assume that Assumptions 3.1, E.1 and E.2 hold. Set the step size
$$
h \leq \min \left\{\frac {1}{1 2 C _ {2} \kappa}, \frac {\epsilon^ {2 / 3}}{\left(4 8 C _ {2} \kappa \left(\frac {u ^ {2} L}{m} M + \| \mathbf {x} _ {*} \| ^ {2}\right)\right) ^ {1 / 3}}, \frac {\epsilon^ {2 / 7}}{\left(4 8 C _ {2} L _ {2} ^ {2} u ^ {4} d ^ {2}\right) ^ {1 / 7}} \right\},
$$
Without loss of generality, assume  $\mathbf{x}_{*} = 0$ . Then Theorem E.3 indicates a convergence rate of  $\widetilde{\mathcal{O}}\left(M^{1/3}\epsilon^{-2/3} + d^{2/7}\epsilon^{-2/7}\right)$ , which has a lower dimension dependency both in the sense of the order of  $d$  and the relevant  $\epsilon$  coefficient. When the Hessian of any  $\mathbf{x} \in \mathbb{R}^d$  has a dimension-free trace, we arrive at a convergence rate of  $\mathcal{O}\left(\epsilon^{-2/3} + d^{2/7}\epsilon^{-2/7}\right)$ .
E.1 Proof of Theorem E.3
In this section, we prove the convergence guarantee using the upper bound of the local Hessian trace.
Proof. We follow the notations in the proof of Theorem 4.2 in Appendix B.2. Similarly, denote the error in (B.5) by
$$
A = 2 u h C _ {1} \mathbb {E} \mathbb {E} _ {\beta \sim \rho^ {\prime}} \langle \mathbf {x} _ {n} (\beta) - \mathbf {x} _ {n} ^ {*} (\beta) + \mathbf {v} _ {n} (\beta) - \mathbf {v} _ {n} ^ {*} (\beta), \nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle .
$$
$$
\begin{array}{l} A \leq 2 C _ {1} u h \mathbb {E} \langle \mathbf {x} _ {n} (\beta) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (\beta) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (\beta) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (\beta) + \mathbf {v} _ {n} ^ {*}, \nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \rangle \\ + 2 C _ {1} u h \mathbb {E} \left\langle \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*}, \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {\beta \sim \rho^ {\prime}} \left(\nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))\right) \right\rangle \\ \lesssim \frac {u h}{u h} \mathbb {E} \| \mathbf {x} _ {n} (\beta) - \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} (\beta) + \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} (\beta) - \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} (\beta) + \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ + u ^ {2} h ^ {2} \mathbb {E} \| \nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} + \frac {h}{u} u h \mathbb {E} \| \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} + \mathbf {v} _ {n} - \mathbf {v} _ {n} ^ {*} \| ^ {2} \\ + \frac {u}{h} u h \mathbb {E} \| \mathbb {E} _ {\alpha \sim \rho^ {\prime}} \mathbb {E} _ {\beta \sim \rho^ {\prime}} (\nabla f (\mathbf {x} _ {n} (\beta)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha))) \| ^ {2} \\ \stackrel {a} {\lesssim} \left(h ^ {2} \mathbb {E} \mathbb {E} _ {t \sim \rho} \Omega_ {n} (t) + h ^ {4} \mathbb {E} \Omega_ {n} + u ^ {2} \frac {h ^ {4} L M}{m} + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {4} h ^ {8} d ^ {2}\right) \\ + u ^ {2} h ^ {2} \left(h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L}{m} M + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {6} d ^ {2}\right) + h ^ {2} \mathbb {E} \Omega_ {n} \\ + u ^ {2} \left(h ^ {6} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {6} L}{m} M + h ^ {6} L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {1 0} d ^ {2}\right) \\ \lesssim h ^ {2} \mathbb {E} \mathbb {E} _ {\beta \sim \rho} \Omega_ {n} (\beta) + h ^ {2} \mathbb {E} \Omega_ {n} + \frac {u ^ {2} h ^ {4} L}{m} M + h ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {4} h ^ {8} d ^ {2}, \\ \end{array}
$$
where  $\lesssim$  follows by Lemmas D.1, D.4 and D.5. We shall note that  $\mathbb{E}\Omega_{n + 1} = \mathbb{E}_{\beta \sim \rho}\Omega_n(\beta)$ . Combining the above control with (B.5), we have
$$
\begin{array}{l} \mathbb {E} \Omega_ {n + 1} \leq \frac {\mathbb {E} _ {\beta \sim \rho} e ^ {- \frac {\beta}{\kappa}} + C _ {2} h ^ {2}}{1 - C _ {2} h ^ {2}} \mathbb {E} \Omega_ {n} + \frac {C _ {2} h ^ {4}}{1 - C _ {2} h ^ {2}} \left(\frac {u ^ {2} L}{m} M + \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {4} h ^ {4} d ^ {2}\right) \\ \leq \left(1 - \frac {h}{6 \kappa}\right) \mathbb {E} \Omega_ {n} + 2 C _ {2} h ^ {4} \left(\frac {u ^ {2} L}{m} M + \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {4} h ^ {4} d ^ {2}\right), \\ \end{array}
$$
where the last inequality follows by the control (B.8) and the choice of  $h$  which satisfies  $1 - C_2h^2 \geq \frac{1}{2}$ . Denote  $r = 1 - \frac{h}{6\kappa}$  and  $\mathcal{E}' = 2C_2h^4\left(\frac{u^2L}{m}M + \| \mathbf{x}_*\|^2 + L_2^2u^4h^4d^2\right)$  and therefore
$$
\mathbb {E} \Omega_ {N} \leq r \mathbb {E} \Omega_ {N - 1} + \mathcal {E} \leq r ^ {N} \mathbb {E} \Omega_ {0} + \mathcal {E} (1 + r + \dots + r ^ {N - 1}) \leq r ^ {N} \mathbb {E} \Omega_ {0} + \frac {\mathcal {E}}{1 - r}.
$$
When  $N\geq \frac{6}{\kappa}\log \left(\frac{2\mathbb{E}\Omega_0}{\epsilon^2}\right)$  , we have
$$
r ^ {N} \mathbb {E} \Omega_ {0} \leq e ^ {\frac {N h}{6 \kappa}} \mathbb {E} \Omega_ {0} \leq \frac {\epsilon^ {2}}{2}. \tag {E.1}
$$
$$
h \leq \min  \left\{\frac {\epsilon^ {2 / 3}}{\left(4 8 C _ {2} \kappa \left(\frac {u ^ {2} L}{m} M + \| \mathbf {x} _ {*} \| ^ {2}\right)\right) ^ {1 / 3}}, \frac {\epsilon^ {2 / 7}}{\left(4 8 C _ {2} L _ {2} ^ {2} u ^ {4} d ^ {2}\right) ^ {1 / 7}} \right\},
$$
we get  $\frac{\mathcal{E}}{1 - r} \leq \frac{\epsilon^2}{2}$ . Together with the (E.1), we obtain the convergence rate in Theorem E.3.
Finally, we would say that the trace of Hessian should be a more realistic factor to characterize the complexity of Langevin algorithms. Beyond the ridge separable functions described in Section 3.3, we illustrate the Bayesian sampling over the two-layer neural network model and show that the neural network also has a bounded trace under suitable conditions. Note that our convergence analysis cannot be applied to this task since neural networks are non-convex models, and we leave the extensions of analysis to the general non-log-concave case as future work.