SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\mathbb {E} \mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) ^ {T} \boldsymbol {\Sigma} ^ {- 1} \left(\mathbb {E} \mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) \leq 3,
$$
where  $\Sigma$  is the covariance of  $p$  and therefore bounded by  $\frac{1}{m}$ . Hence
$$
\left\| A \left(\mathbb {E} \mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) \right\| ^ {2} \leq \frac {\| A \| ^ {2}}{m} \leq \frac {1}{m} \operatorname {t r} \left(A ^ {2}\right). \tag {C.6}
$$
Combining equation (C.5) and (C.6), we obtain that
$$
\mathbb {E} \| A \left(\mathbf {x} _ {n} ^ {*} - \mathbf {x} _ {*}\right) \| ^ {2} \lesssim \mathbb {E} \| A \left(\mathbf {x} _ {n} ^ {*} - \mathbb {E} \mathbf {x} _ {n} ^ {*}\right) \| ^ {2} + \| A \left(\mathbb {E} \mathbf {x} _ {n} ^ {*} - \mathbf {x} _ {*}\right) \| ^ {2} \lesssim \frac {1}{m} \operatorname {t r} \left(A ^ {2}\right). \tag {C.7}
$$
Combining (C.4) and (C.7) and then we achieve the first claim.
To bound  $\mathbb{E}\| A(\mathbf{x}_n(t) + \mathbf{v}_n(t))\|^2$  , we have
$$
\mathbb {E} \| A \left(\mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t)\right) \| ^ {2} \lesssim \mathbb {E} \| A \left(\mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) - \mathbf {v} _ {n} ^ {*} (t)\right) \| ^ {2}
$$
$$
\begin{array}{l} + \mathbb {E} \| A \left(\mathbf {x} _ {n} ^ {*} (t) + \mathbf {v} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) \| ^ {2} + \| A \mathbf {x} _ {*} \| ^ {2} \\ \lesssim L ^ {2} \mathbb {E} \| \mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) - \mathbf {v} _ {n} ^ {*} (t) \| ^ {2} + \mathbb {E} \| A (\mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}) \| ^ {2} \\ + \mathbb {E} \| A \mathbf {v} _ {n} ^ {*} (t) \| ^ {2} + L ^ {2} \| \mathbf {x} _ {*} \| ^ {2} \\ \end{array}
$$
$$
\lesssim L ^ {2} \mathbb {E} \| \mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) - \mathbf {v} _ {n} ^ {*} (t) \| ^ {2} + \frac {1}{m} \operatorname {t r} \left(A ^ {2}\right) + \frac {1}{m} \operatorname {t r} \left(A ^ {2}\right) + L ^ {2} \| \mathbf {x} _ {*} ^ {2} \|
$$
$$
\lesssim L ^ {2} \mathbb {E} \| \mathbf {x} _ {n} (t) + \mathbf {v} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) - \mathbf {v} _ {n} ^ {*} (t) \| ^ {2} + \frac {1}{m} \mathrm {t r} (A ^ {2}) + L ^ {2} \| \mathbf {x} _ {*} \| ^ {2},
$$
where the second inequality follows by the Lipschitz smoothness of  $f(\mathbf{x})$  and Young's inequality; in the third inequality we use (C.7) and  $v \sim \mathcal{N}(0, \frac{1}{m})$ .
To prove the third claim, we have
$$
\begin{array}{l} \mathbb {E} \| A \nabla f (\mathbf {x} _ {n} (t)) \| ^ {2} \lesssim \mathbb {E} \| A \left(\nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\mathbf {x} _ {n} ^ {*} (t))\right) \| ^ {2} + \mathbb {E} \| A \left(\nabla f (\mathbf {x} _ {n} ^ {*} (t)) - \nabla f (\mathbf {x} _ {*})\right) \| ^ {2} + \| A \nabla f (\mathbf {x} _ {*}) \| ^ {2} \\ \lesssim L ^ {4} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) \| ^ {2} + \mathbb {E} L ^ {2} \| A \left(\mathbf {x} _ {n} ^ {*} (t) - \mathbf {x} _ {*}\right) \| ^ {2} + L ^ {2} \| \nabla f (\mathbf {x} _ {*}) \| ^ {2} \\ \lesssim L ^ {4} \mathbb {E} \| \mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {*} (t) \| ^ {2} + \frac {L ^ {2}}{m} \mathrm {t r} (A ^ {2}) + L ^ {2} m ^ {2} \| \mathbf {x} _ {*} \| ^ {2}, \\ \end{array}
$$
where in last inequality, we apply (C.7) and  $\nabla f(\mathbf{x}_*) + m\mathbf{x}_* = 0$
D Technical bounds
In this section, we present some controls that are useful in the proof of Theorem 4.2.
D.1 Bound of  $\mathbb{E}\| \nabla f(\mathbf{x}_n(t)) - \nabla f(\hat{\mathbf{x}}_n(\alpha))\|^2$
Lemma D.1. Let  $\mathbf{x}_n(t),\widehat{\mathbf{x}}_n(t)$  be defined as above and  $u = \frac{1}{L + m}$ . For any fixed  $t,\alpha \leq h$
(i) When Assumption 3.1 hold and  $H$  is a uniform upper bound of  $\nabla^2 f$ , i.e.  $\nabla^2 f(\mathbf{x}) \preceq H$  for all  $\mathbf{x} \in \mathbb{R}^d$ ,  $\mathbf{x}_n^*(t)$  and  $\widehat{\mathbf{x}}_n(\alpha)$  satisfies
$$
\mathbb {E} \| \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2}}{m} \mathrm {t r} (H ^ {2}) + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2}.
$$
(ii) When Assumptions 3.1, E.1 and E.2 hold, we have
$$
\mathbb {E} \| \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L}{m} M + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {6} d ^ {2}.
$$
Proof. For any  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$  , we have
$$
\left\| \nabla f (\mathbf {x}) - \nabla f (\mathbf {y}) \right\| ^ {2} = \left\| \int_ {0} ^ {1} \nabla^ {2} f ((1 - k) \mathbf {x} + k \mathbf {y}) (\mathbf {x} - \mathbf {y}) d k \right\| ^ {2}. \tag {D.1}
$$
Denote  $\bar{\mathcal{H}} = \int_0^1\nabla^2 f(\mathbf{x} + k(\mathbf{y} - \mathbf{x}))\mathrm{d}k$  and  $\bar{\mathcal{H}}$  depends on  $\mathbf{x}$  and  $\mathbf{y}$ . Substitute  $\mathbf{x},\mathbf{y}$  by  $\mathbf{x}_n(t),\widehat{\mathbf{x}}_n(\alpha)$  and then we obtain  $\| \nabla f(\mathbf{x}_n(t)) - \nabla f(\widehat{\mathbf{x}}_n(\alpha))\| ^2 = \| \bar{\mathcal{H}} (\mathbf{x}_n(t) - \widehat{\mathbf{x}}_n(\alpha))\| ^2$ . Then by plugging in the close-form solution into  $\| \mathcal{H}(\mathbf{x}_n(t) - \widehat{\mathbf{x}}_n(\alpha))\|^2$ ,
$$
\begin{array}{l} \| \mathcal {\widehat {H}} (\mathbf {x} _ {n} (t) - \widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} \\ \stackrel {a} {=} \left\| \bar {\mathcal {H}} \Big ((A _ {1 1} (t) - A _ {1 1} (\alpha) - A _ {1 2} (t) + A _ {1 2} (\alpha)) \mathbf {x} _ {n} + (A _ {1 2} (t) - A _ {1 2} (\alpha)) (\mathbf {x} _ {n} + \mathbf {v} _ {n}) + \right. \\ u \int_ {0} ^ {t} A _ {1 2} (s - t) \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \mathrm {d} s - u \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \nabla f (\mathbf {x} _ {n}) \mathrm {d} s + 2 \sqrt {u} \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} \\ \left. - 2 \sqrt {u} \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s}\right) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {b} {\lesssim} (A _ {1 1} (t) - A _ {1 1} (\alpha) - A _ {1 2} (t) + A _ {1 2} (\alpha)) ^ {2} \| \bar {\mathcal {H}} \mathbf {x} _ {n} \| ^ {2} + (A _ {1 2} (t) - A _ {1 2} (\alpha)) ^ {2} \| \bar {\mathcal {H}} (\mathbf {x} _ {n} + \mathbf {v} _ {n}) \| ^ {2} \\ + u ^ {2} \underbrace {\left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \mathrm {d} s - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \mathrm {d} s \right\| ^ {2}} _ {①} \\ + u \underbrace {\left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} \right\| ^ {2}} _ {②}, \tag {D.2} \\ \end{array}
$$
where in  $\stackrel{a}{=}$  we use Lemma 4.1;  $\stackrel{b}{\lesssim}$  follows by the Jensen's inequality. Besides Lemma A.1 implies that  $(A_{11}(t) - A_{11}(\alpha) - A_{12}(t) + A_{12}(\alpha))^2 \| H\mathbf{x}_n\|^2 \lesssim h^2 \| H\mathbf{x}_n\|^2$  and  $(A_{12}(t) - A_{12}(\alpha))^2 \| H(\mathbf{x}_n + \mathbf{v}_n)\|^2 \lesssim h^2 \| H(\mathbf{x}_n + \mathbf{v}_n)\|^2$ .
For the expectation of term ①, we have
$$
\begin{array}{l} \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \mathrm {d} s - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \mathrm {d} s \right\| ^ {2} \\ \lesssim \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \big (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n}) \big) \mathrm {d} s \right\| ^ {2} + \\ \mathbb {E} \left\| \int_ {\alpha} ^ {t} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \mathrm {d} s \right\| ^ {2} + \mathbb {E} \left\| \int_ {0} ^ {t} \left(A _ {1 2} (s - t) - A _ {1 2} (s - \alpha)\right) \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \mathrm {d} s \right\| ^ {2} \tag {D.3} \\ \leq \mathbb {E} t \int_ {0} ^ {t} A _ {1 2} (s - t) ^ {2} \mathrm {d} s \| \bar {\mathcal {H}} \left(\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n})\right) \| ^ {2} \\ + \mathbb {E} \left| (\alpha - t) \int_ {\alpha} ^ {t} A _ {1 2} (s - \alpha) ^ {2} \mathrm {d} s \right| \| \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \| ^ {2} \\ + \mathbb {E} t \int_ {0} ^ {t} (A _ {1 2} (s - t) - A _ {1 2} (s - \alpha)) ^ {2} d s \| \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \| ^ {2} \\ \lesssim h ^ {4} \mathbb {E} \| \bar {\mathcal {H}} (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n})) \| ^ {2} + h ^ {4} \mathbb {E} \| \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \| ^ {2}, \\ \end{array}
$$
where in the last inequality we use that  $t, \alpha < h$  and  $A_{12}(t) = \mathcal{O}(t)$ . Let  $\bar{\mathcal{H}}' = \int_0^1 \nabla^2 f(\widehat{\mathbf{x}}_n(\alpha) + k(\widehat{\mathbf{x}}_n(\alpha) - \mathbf{x}_n)) \mathrm{d}k$ . By the definition of  $\widehat{\mathbf{x}}_n(\alpha)$ ,  $\mathbb{E}\|\bar{\mathcal{H}}(\nabla f(\widehat{\mathbf{x}}_n(\alpha)) - \nabla f(\mathbf{x}_n))\|^2$  can be upper bounded by
$$
\begin{array}{l} \mathbb {E} \| \bar {\mathcal {H}} (\nabla f (\hat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n})) \| ^ {2} \\ \leq L ^ {2} \mathbb {E} \| (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n})) \| ^ {2} \\ = L ^ {2} \mathbb {E} \left\| \bar {\mathcal {H}} ^ {\prime} \big ((A _ {1 1} (\alpha) - 1) \mathbf {x} _ {n} + A _ {1 2} (\alpha) \mathbf {v} _ {n} + u \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \nabla f (\mathbf {x} _ {n}) d t + 2 \sqrt {u} \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) d \mathbf {B} _ {t} \big) \right\| ^ {2} \\ \lesssim L ^ {2} \mathbb {E} A _ {1 2} (\alpha) ^ {2} \| \bar {\mathcal {H}} ^ {\prime} (\mathbf {x} _ {n} + \mathbf {v} _ {n}) \| ^ {2} + \mathbb {E} (A _ {1 1} (x) - A _ {1 2} (x) - 1) ^ {2} L ^ {2} \| \bar {\mathcal {H}} ^ {\prime} \mathbf {x} _ {n} \| ^ {2} \\ + u ^ {2} \mathbb {E} \alpha \int_ {0} ^ {\alpha} L ^ {2} \| A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \nabla f (\mathbf {x} _ {n}) \| ^ {2} d t + u \mathbb {E} L ^ {2} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} d \mathbf {B} _ {t} \right\| ^ {2} \\ \lesssim h ^ {2} L ^ {2} \mathbb {E} \| \bar {\mathcal {H}} ^ {\prime} (\mathbf {x} _ {n} + \mathbf {v} _ {n}) \| ^ {2} + h ^ {2} L ^ {2} \mathbb {E} \| \bar {\mathcal {H}} ^ {\prime} \mathbf {x} _ {n} \| ^ {2} + u ^ {2} h ^ {4} L ^ {2} \mathbb {E} \| \bar {\mathcal {H}} ^ {\prime} \nabla f (\mathbf {x} _ {n}) \| ^ {2} \\ + u L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2}, \\ \end{array}
$$
where in the last inequality we apply Lemma A.1. Note that  $\mathbf{x}_n(0) = \mathbf{x}_n$  and  $\mathbf{v}_n(0) = \mathbf{v}_n$ . Then by Lemma C.2,
$$
\begin{array}{l} \mathbb {E} \| \bar {\mathcal {H}} (\nabla f (\hat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n})) \| ^ {2} \\ \leq L ^ {2} \mathbb {E} \| \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\mathbf {x} _ {n}) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \lesssim h ^ {2} \big (L ^ {4} \mathbb {E} \| \mathbf {x} _ {n} + \mathbf {v} _ {n} - \mathbf {x} _ {n} ^ {*} - \mathbf {v} _ {n} ^ {*} \| ^ {2} + \frac {L ^ {2}}{m} \mathbb {E} \mathrm {t r} (\bar {\mathcal {H}} ^ {\prime 2}) + L ^ {4} \| \mathbf {x} _ {*} \| ^ {2} \big) + h ^ {2} (L ^ {4} \mathbb {E} \| \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} \| ^ {2} \\ + \frac {L ^ {2}}{m} \mathbb {E} \operatorname {t r} \left(\bar {\mathcal {H}} ^ {\prime 2}\right) + L ^ {2} \left\| \mathbf {x} _ {*} \right\| ^ {2}) + u ^ {2} h ^ {2} \left(L ^ {4} \mathbb {E} \| \mathbf {x} _ {n} - \mathbf {x} _ {n} ^ {*} \| ^ {2} + \frac {L ^ {2}}{m} \mathbb {E} \operatorname {t r} \left(\bar {\mathcal {H}} ^ {\prime 2}\right) + L ^ {4} m ^ {2} \| \mathbf {x} _ {*} \| ^ {2}\right) \tag {D.4} \\ + u L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \\ \end{array}
$$
$$
\lesssim h ^ {2} L ^ {4} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L ^ {2}}{m} \mathbb {E} \mathrm {t r} (\bar {\mathcal {H}} ^ {\prime 2}) + h ^ {2} L ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + u L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2},
$$
where in the last inequality we use  $u = \frac{1}{L + m} \leq \min \left\{\frac{1}{L}, \frac{1}{m}\right\}$ . Plugging (D.4) into (D.3) and then we have an upper bound of the expectation of term ①. Plugging the upper bound of term ① into (D.2),
and thus we have the following inequality
$$
\begin{array}{l} \mathbb {E} \| \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} \| \bar {\mathcal {H}} \mathbf {x} _ {n} \| ^ {2} + h ^ {2} \mathbb {E} \| \bar {\mathcal {H}} (\mathbf {x} _ {n} + \mathbf {v} _ {n}) \| ^ {2} + u ^ {2} h ^ {4} \mathbb {E} \| \bar {\mathcal {H}} \nabla f (\mathbf {x} _ {n}) \| ^ {2} \\ + u ^ {2} h ^ {4} \left(h ^ {2} L ^ {4} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L ^ {2}}{m} \mathbb {E} \operatorname {t r} \left(\left(\bar {\mathcal {H}} ^ {\prime}\right) ^ {2}\right) + h ^ {2} L ^ {4} \| \mathbf {x} _ {*} \| ^ {2} + u \mathbb {E} L ^ {2} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2}\right) \\ + u \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2}}{m} \mathbb {E} \left(\operatorname {t r} \left(\left(\bar {\mathcal {H}} ^ {\prime}\right) ^ {2}\right) + \operatorname {t r} \left(\bar {\mathcal {H}} ^ {2}\right)\right) + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + u ^ {3} h ^ {4} L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \\ + u \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} \right\| ^ {2}, \\ \end{array}
$$
where in the last inequality we use Lemma C.2 and the definition of  $\Omega_{n}$
When we assume  $\nabla^2 f(\mathbf{x})$  has a uniform upper bound  $H$ , we have
$$
\begin{array}{l} u \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s}\right) \| ^ {2} \\ \lesssim u \mathbb {E} \left\| \int_ {0} ^ {h} h H \mathrm {d} \mathbf {B} _ {s} \right\| ^ {2} \lesssim \frac {h ^ {3}}{m} \operatorname {t r} \left(H ^ {2}\right) \leq \frac {h ^ {3} L}{m} \operatorname {t r} (H) \tag {D.5} \\ \end{array}
$$
$$
u ^ {3} h ^ {4} L ^ {2} \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (t - \alpha) \bar {\mathcal {H}} ^ {\prime} \mathrm {d} \mathbf {B} _ {t} \right\| ^ {2} \leq u ^ {3} h ^ {4} L ^ {2} h ^ {3} \operatorname {t r} (H ^ {2}) \lesssim \frac {h ^ {4}}{m} \operatorname {t r} (H ^ {2}) \leq \frac {h ^ {4} L}{m} \operatorname {t r} (H).
$$
Besides we have  $\operatorname{tr}\left(\left(\bar{\mathcal{H}}^{\prime}\right)^{2}\right) + \operatorname{tr}\left(\bar{\mathcal{H}}^{2}\right) \leq L\operatorname{tr}\left(\bar{\mathcal{H}}^{\prime}\right) + L\operatorname{tr}\left(\bar{\mathcal{H}}\right) \lesssim L\operatorname{tr}(H)$ . These bounds yield the control
$$
\mathbb {E} \| \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2} L}{m} \operatorname {t r} (H) + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2}. \tag {D.6}
$$
And when  $f$  is  $L_{2}$ -Hessian smooth, by Lemmas D.2 and D.3, we have
$$
\begin{array}{l} \mathbb {E} \| \nabla f (\mathbf {x} _ {n} (t)) - \nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) \| ^ {2} \\ \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {h ^ {2}}{m} \mathbb {E} \left(\operatorname {t r} \left(\left(\bar {\mathcal {H}} ^ {\prime}\right) ^ {2}\right) + \operatorname {t r} \left(\bar {\mathcal {H}} ^ {2}\right)\right) + u h ^ {3} L M + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} \\ + L _ {2} ^ {2} u ^ {2} h ^ {6} d ^ {2} + L _ {2} ^ {2} u ^ {4} L ^ {2} h ^ {1 0} d ^ {2} \tag {D.7} \\ \end{array}
$$
$$
\begin{array}{l} \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + L \frac {h ^ {2}}{m} \mathbb {E} \left(\operatorname {t r} \left(\bar {\mathcal {H}} ^ {\prime}\right) + \operatorname {t r} \left(\bar {\mathcal {H}}\right)\right) + \frac {h ^ {2} L}{m} M + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {6} d ^ {2} \\ \lesssim h ^ {2} L ^ {2} \mathbb {E} \Omega_ {n} + \frac {L h ^ {2} M}{m} + L ^ {2} h ^ {2} \| \mathbf {x} _ {*} \| ^ {2} + L _ {2} ^ {2} u ^ {2} h ^ {6} d ^ {2}. \\ \end{array}
$$
Lemma D.2. Let  $\tilde{\mathcal{H}}$  be defined as above. When Assumptions E.1 and E.2 hold, we have
$$
\mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s}\right) \| ^ {2} \lesssim h ^ {3} L M + L _ {2} ^ {2} u h ^ {6} d ^ {2}.
$$
$$
\begin{array}{l} \mathbb {E} \left. \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \bar {\mathcal {H}} \mathrm {d} \mathbf {B} _ {s}\right) \right\| ^ {2} \\ = \mathbb {E} \left\| \bar {\mathcal {H}} \left(\int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s}\right) \right\| ^ {2} \\ \stackrel {\mathrm {a}} {\lesssim} \mathbb {E} \left\| \left(\bar {\mathcal {H}} - \mathbb {E} [ \bar {\mathcal {H}} | \mathcal {F} _ {n} ]\right) \left(\int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s}\right) \right\| ^ {2} \tag {D.8} \\ + \mathbb {E} \left\| \mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right] \left(\int_ {0} ^ {t} A _ {1 2} (s - t) d \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) d \mathbf {B} _ {s}\right) \right\| ^ {2} \\ \lesssim \mathbb {E} \left(\left\| \bar {\mathcal {H}} - \mathbb {E} [ \bar {\mathcal {H}} | \mathcal {F} _ {n} ] \right\| ^ {2} \left\| \left(\int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s}\right) \right\| ^ {2}\right) \\ + \mathbb {E} \left\| \mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right] \left(\int_ {0} ^ {t} A _ {1 2} (s - t) d \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) d \mathbf {B} _ {s}\right) \right\| ^ {2}, \\ \end{array}
$$
where  $\stackrel{\mathrm{a}}{\lesssim}$  follows by dividing  $\bar{\mathcal{H}}$  into  $\mathbb{E}\left[\bar{\mathcal{H}}|\mathcal{F}_n\right]$  and  $\bar{\mathcal{H}} - \mathbb{E}\left[\bar{\mathcal{H}}|\mathcal{F}_n\right]$ . In the last inequality,  $\mathbb{E}\left[\bar{\mathcal{H}}|\mathcal{F}_n\right]$  is independent of  $\mathbf{B}_s$ , and thus
$$
\begin{array}{l} \mathbb {E} \left\| \mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right] \left(\int_ {0} ^ {t} A _ {1 2} (s - t) d \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) d \mathbf {B} _ {s} ^ {\prime}\right) \right\| ^ {2} \\ \lesssim h ^ {2} \mathbb {E} \left\| \mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right] \int_ {0} ^ {h} d \mathbf {B} _ {s} \right\| ^ {2} \tag {D.9} \\ \leq h ^ {3} \mathbb {E} \operatorname {t r} \left(\left(\mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right]\right) ^ {2}\right). \\ \end{array}
$$
As for the first term, let  $\{\mathbf{B}_s^{\prime}\}_{0\leq s\leq h}$  be a standard Brownian motion which is independent of  $\{\mathbf{B}_s\}_{0\leq s\leq h}$ . Define  $\widehat{\mathbf{x}}_n^\prime (\alpha) = \mathcal{J}(\alpha ,\mathbf{x}_n;\{\mathbf{B}_s^\prime \}_{0\leq s\leq \alpha},(\mathbf{x}_n,\mathbf{v}_n))$  and  $\mathbf{x}_n^\prime (t) = \mathcal{J}(\alpha ,\widehat{\mathbf{x}}_n^\prime (\alpha);\{\mathbf{B}_s^\prime \}_{0\leq s\leq t},(\mathbf{x}_n,\mathbf{v}_n))$ . Then we have
$$
\begin{array}{l} \mathbb {E} \left\| \bar {\mathcal {H}} - \mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right] \right\| ^ {2} \stackrel {{\mathrm {a}}} {{\leq}} \mathbb {E} \int_ {0} ^ {1} \left\| \nabla^ {2} f ((1 - k) \mathbf {x} _ {n} (t) + k \widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla^ {2} f ((1 - k) \mathbf {x} _ {n} ^ {\prime} (t) + k \widehat {\mathbf {x}} _ {n} ^ {\prime} (\alpha)) \right\| ^ {2} \mathrm {d} k \\ \stackrel {\mathrm {b}} {\leq} L _ {2} ^ {2} \mathbb {E} \int_ {0} ^ {1} \| (1 - k) (\mathbf {x} _ {n} (t) - \mathbf {x} _ {n} ^ {\prime} (t)) + k (\widehat {\mathbf {x}} _ {n} (\alpha) - \widehat {\mathbf {x}} _ {n} ^ {\prime} (\alpha)) \| ^ {2} \mathrm {d} k \\ \stackrel {c} {\lesssim} L _ {2} ^ {2} u ^ {2} \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\widehat {\mathbf {x}} _ {n} ^ {\prime} (\alpha))) \mathrm {d} s \right\| ^ {2} \\ + L _ {2} ^ {2} u \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} ^ {\prime} \right\| ^ {2} \\ + L _ {2} ^ {2} u \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} ^ {\prime} \right\| ^ {2}, \tag {D.10} \\ \end{array}
$$
where  $\stackrel{\mathrm{a}}{\leq}$  follows by the definition of  $\bar{\mathcal{H}}$  and Jensen's inequality; in  $\stackrel{\mathrm{b}}{\leq}$  we use the Hessian smoothness of  $f$ ;  $\lesssim$  follows by the definition of  $\mathbf{x}_n(t), \mathbf{x}_n'(t), \widehat{\mathbf{x}}_n(\alpha)$  and  $\widehat{\mathbf{x}}_n'(\alpha)$ . For the first term of the last inequality, by the definition of  $\widehat{\mathbf{x}}_n(\alpha)$  and  $\widehat{\mathbf{x}}_n'(\alpha)$ , we have
$$
\begin{array}{l} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) (\nabla f (\widehat {\mathbf {x}} _ {n} (\alpha)) - \nabla f (\widehat {\mathbf {x}} _ {n} ^ {\prime} (\alpha))) \mathrm {d} s \right\| ^ {2} \\ \lesssim L ^ {2} h ^ {4} u \left\| \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} ^ {\prime} \right\| ^ {2}. \\ \end{array}
$$
And thus we have
$$
\begin{array}{l} \mathbb {E} \left(\left\| \bar {\mathcal {H}} - \mathbb {E} \left[ \bar {\mathcal {H}} | \mathcal {F} _ {n} \right] \right\| ^ {2} \left\| \left(\int_ {0} ^ {t} A _ {1 2} (s - t) d \mathbf {B} _ {s} - \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) d \mathbf {B} _ {s}\right) \right\| ^ {2}\right) \\ \lesssim L _ {2} ^ {2} u \mathbb {E} \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} \right\| ^ {4} + \mathbb {E} L _ {2} ^ {2} u \left\| \int_ {0} ^ {t} A _ {1 2} (s - t) \mathrm {d} \mathbf {B} _ {s} ^ {\prime} \right\| ^ {4} \tag {D.11} \\ + L _ {2} ^ {2} u \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} \right\| ^ {4} + L _ {2} ^ {2} u \mathbb {E} \left\| \int_ {0} ^ {\alpha} A _ {1 2} (s - \alpha) \mathrm {d} \mathbf {B} _ {s} ^ {\prime} \right\| ^ {4} \\ \lesssim L _ {2} ^ {2} u h ^ {6} d ^ {2}. \\ \end{array}
$$