SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for some constants  $C_1, C_2, C_3$  depending on  $L, C, \sigma, \alpha, \beta$ , and  $d$ . Since  $\gamma_k \to 0$ , there exist  $\tilde{\alpha}, \tilde{\beta} > 0$  and  $k_0$  such that, for all  $k \geq k_0$ ,
$$
\mathbb {E} \| x _ {k + 1} \| ^ {2} \leq \mathbb {E} \| x _ {k} \| ^ {2} (1 - \tilde {\alpha} \gamma_ {k + 1}) + \tilde {\beta} \gamma_ {k + 1}, \quad 1 - \tilde {\alpha} \gamma_ {k + 1} > 0.
$$
A simple induction yields
$$
\sup  _ {k} \mathbb {E} \| x _ {k} \| ^ {2} \leq \max  \left\{\frac {\tilde {\beta}}{\tilde {\alpha}}, \mathbb {E} \| x _ {k _ {0}} \| ^ {2} \right\}
$$
D.2 Proof of Theorem 4 for Constant Diffusion
Before proceeding, we need a lemma which can be distilled from [20, Proposition 8]:
Lemma 5. Suppose  $\nabla f$  is  $L$ -Lipschitz. Fix  $x \in \mathbb{R}^d$  and  $\gamma > 0$ , let  $\tilde{x}^{+} = x - \gamma \nabla f(x) + \sqrt{2\gamma}\xi$ . Then
$$
\mathbb {E} \left[ \exp \left(\frac {1}{2} \langle \nabla f (x), \tilde {x} ^ {+} - x \rangle + \frac {L}{4} \| \tilde {x} ^ {+} - x \| ^ {2}\right) \right] \leq (1 - \gamma L) ^ {- d / 2} e ^ {- \frac {\gamma}{4} \| \nabla f (x) \| ^ {2}}. \tag {23}
$$
Let  $\tilde{x}_{k + 1} := x_k - \gamma_{k + 1}\nabla f(x_k) + \sqrt{2\gamma_{k + 1}}\xi_{k + 1}$  so that  $x_{k + 1} - x_k = \tilde{x}_{k + 1} - x_k - \gamma_{k + 1}(U_{k + 1} + b_{k + 1})$ . Conditioned on  $x_k, U_{k + 1}, U_{k + 1}', \xi_{k + 1}'$ , and using the  $L$ -Lipschitzness of  $\nabla f$ , we get
$$
\begin{array}{l} e ^ {- \frac {1}{2} f (x _ {k})} \mathbb {E} e ^ {\frac {1}{2} f (x _ {k + 1})} \\ \leq \mathbb {E} \exp \left(\frac {1}{2} \langle \nabla f (x _ {k}), x _ {k + 1} - x _ {k} \rangle + \frac {L}{4} \| x _ {k + 1} - x _ {k} \| ^ {2}\right) (24) \\ \leq \mathbb {E} \exp \left\{\frac {1}{2} \langle \nabla f (x _ {k}), \tilde {x} _ {k + 1} - x _ {k} \rangle - \frac {1}{2} \langle \nabla f (x _ {k}), \gamma_ {k + 1} U _ {k + 1} \rangle \right. (25) \\ \left. - \frac {1}{2} \langle \nabla f (x _ {k}), \gamma_ {k + 1} b _ {k + 1} \rangle + \frac {L}{2} \| \tilde {x} _ {k + 1} - x _ {k} \| ^ {2} + L \gamma_ {k + 1} ^ {2} \| U _ {k + 1} \| ^ {2} + L \gamma_ {k + 1} ^ {2} \| b _ {k + 1} \| ^ {2} \right\}. (26) \\ \end{array}
$$
Let  $\delta \in (0,1)$ . Since
$$
\begin{array}{l} - \frac {1}{2} \langle \nabla f (x _ {k}), \gamma_ {k + 1} U _ {k + 1} \rangle \leq \gamma_ {k + 1} ^ {2 - \delta} \| \nabla f (x _ {k}) \| ^ {2} + \gamma_ {k + 1} ^ {\delta} \| U _ {k + 1} \| ^ {2}, \\ - \frac {1}{2} \langle \nabla f (x _ {k}), \gamma_ {k + 1} b _ {k + 1} \rangle \leq \gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + \| b _ {k + 1} \| ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} e ^ {- \frac {1}{2} f (x _ {k})} \mathbb {E} e ^ {\frac {1}{2} f (x _ {k + 1})} (27) \\ \leq \mathbb {E} \exp \left\{\frac {1}{2} \langle \nabla f (x _ {k}), \tilde {x} _ {k + 1} - x _ {k} \rangle + \frac {L}{2} \| \tilde {x} _ {k + 1} - x _ {k} \| ^ {2} \right. (28) \\ \left. + \left(\gamma_ {k + 1} ^ {2 - \delta} + \gamma_ {k + 1} ^ {2}\right) \| \nabla f (x _ {k}) \| ^ {2} + \left(L \gamma_ {k + 1} ^ {2} + \gamma_ {k + 1} ^ {\delta}\right) \| U _ {k + 1} \| ^ {2} + \left(L \gamma_ {k + 1} ^ {2} + 1\right) \| b _ {k + 1} \| ^ {2} \right\}. (29) \\ \end{array}
$$
Invoking (11) an denoting  $c' \triangleq (L\gamma_{k+1}^2 + 1) \cdot c$ , we get
$$
e ^ {- \frac {1}{2} f (x _ {k})} \mathbb {E} e ^ {\frac {1}{2} f (x _ {k + 1})} \leq e ^ {A _ {k}} \cdot \mathbb {E} \exp \left\{\frac {1}{2} \langle \nabla f (x _ {k}), \tilde {x} _ {k + 1} - x _ {k} \rangle + \frac {L}{2} \| \tilde {x} _ {k + 1} - x _ {k} \| ^ {2} + c ^ {\prime} \cdot \gamma_ {k + 1} \| \xi_ {k + 1} \| ^ {2} \right\}, \tag {30}
$$
$$
\begin{array}{l} A _ {k} \triangleq \left(\gamma_ {k + 1} ^ {2 - \delta} + \gamma_ {k + 1} ^ {2} + c ^ {\prime} \gamma_ {k + 1} ^ {2}\right) \| \nabla f (x _ {k}) \| ^ {2} \\ + \left(L \gamma_ {k + 1} ^ {2} + \gamma_ {k + 1} ^ {\delta}\right) \| U _ {k + 1} \| ^ {2} \tag {31} \\ + c ^ {\prime} \left(\gamma_ {k + 1} ^ {2} \| U _ {k + 1} ^ {\prime} \| ^ {2} + \gamma_ {k + 1} \| \xi_ {k + 1} ^ {\prime} \| ^ {2}\right). \\ \end{array}
$$
Recalling that  $\sqrt{2\gamma_{k + 1}}\xi_{k + 1} = \tilde{x}_{k + 1} - x_k + \gamma_{k + 1}\nabla f(x_k)$ , we have  $\gamma_{k + 1}\| \xi_{k + 1}\| ^2\leq \| \tilde{x}_{k + 1} - x_k\| ^2 +$ $\gamma_{k + 1}^2\| \nabla f(x_k)\| ^2$ , and thus
$$
e ^ {- \frac {1}{2} f (x _ {k})} \mathbb {E} e ^ {\frac {1}{2} f (x _ {k + 1})} \leq e ^ {A _ {k} ^ {\prime}} \cdot \mathbb {E} \exp \left\{\frac {1}{2} \langle \nabla f (x _ {k}), \tilde {x} _ {k + 1} - x _ {k} \rangle + \left(\frac {L}{2} + c ^ {\prime}\right) \| \tilde {x} _ {k + 1} - x _ {k} \| ^ {2} \right\}, \tag {32}
$$
where  $A_k' = A_k + c'\gamma_{k+1}^2 \|\nabla f(x_k)\|^2$ . Lemma 5 then implies
$$
e ^ {- \frac {1}{2} f (x _ {k})} \mathbb {E} e ^ {\frac {1}{2} f (x _ {k + 1})} \leq e ^ {A _ {k} ^ {\prime \prime}} \cdot \left(1 - \gamma_ {k + 1} L ^ {\prime}\right) ^ {- \frac {d}{2}} \tag {33}
$$
where  $A_k^{\prime \prime} = A_k^{\prime} - \frac{\gamma_{k + 1}}{4}\| \nabla f(x_k)\|^2$
We now take the expectation over  $x_{k}, U_{k+1}, U_{k+1}', \xi_{k+1}'$  (in other words, we are now only conditioning on  $x_{k}$ ). Set  $\epsilon \triangleq (1 - \gamma_{k+1}L')^{-\frac{1}{2}} - 1 > 0$ . Since  $U_{k+1}, U_{k+1}', \xi_{k+1}'$  are sub-Gaussian and since  $\gamma_{k} \to 0$ , for  $k$  sufficiently large we have
$$
\begin{array}{l} \mathbb {E} A _ {k} ^ {\prime \prime} \leq (1 + \epsilon) \cdot \exp \left[ \left(- \frac {\gamma_ {k + 1}}{4} + \gamma_ {k + 1} ^ {2 - \delta} + \gamma_ {k + 1} ^ {2} + c ^ {\prime} \gamma_ {k + 1} ^ {2} + c ^ {\prime} \gamma_ {k + 1} ^ {2}\right) \| \nabla f (x _ {k}) \| ^ {2} \right] (34) \\ \leq (1 + \epsilon) \cdot e ^ {- \frac {\gamma_ {k + 1}}{8} \| \nabla f (x _ {k}) \| ^ {2}}. (35) \\ \end{array}
$$
To summarize, we have shown that, conditioned on  $x_{k}$
$$
e ^ {- \frac {1}{2} f (x _ {k})} \mathbb {E} e ^ {\frac {1}{2} f (x _ {k + 1})} \leq \left(1 - \gamma_ {k + 1} L ^ {\prime}\right) ^ {- \frac {d + 1}{2}} e ^ {- \frac {\gamma_ {k + 1}}{8} \| \nabla f (x _ {k}) \| ^ {2}}. \tag {36}
$$
A simple induction à la [20, Lemma 1 & Proposition 8] then concludes the proof.
D.3 Proof of Theorem 4 for Mirror Langevin
Here, we bring the proof of Theorem 4 for the case of Example 4 and without noise. The proof for the noisy case is the same as in Appendix D.2.
$$
x ^ {+} = x - \gamma \nabla f \circ \nabla \phi^ {*} (x) + \sqrt {2 \gamma} (\nabla^ {2} \phi^ {*} (x) ^ {- 1}) ^ {1 / 2} \xi ,
$$
where  $\xi$  is a standard Gaussian random variable. Let  $U(x) = f(\nabla \phi^{*}(x))$ . For a fixed  $x$ , we have
$$
\mathbb {E} e ^ {\frac {1}{2} U (x ^ {+}) - \frac {1}{2} U (x)} = \frac {1}{(2 \pi) ^ {d / 2}} \int \exp \left(\frac {1}{2} U (x ^ {+}) - \frac {1}{2} U (x) - \frac {\| \xi \| ^ {2}}{2}\right) d \xi
$$
Notice that we have
$$
\xi = \frac {1}{\sqrt {2 \gamma}} \left(\nabla^ {2} \phi^ {*} (x)\right) ^ {1 / 2} \left(x ^ {+} - x + \gamma \nabla f \circ \nabla \phi^ {*} (x)\right)
$$
$$
d \xi = (\sqrt {2 \gamma}) ^ {- d} \sqrt {\det \nabla^ {2} \phi^ {*} (x)} d x ^ {+}
$$
Thus, the integral, after the change of variable from  $\xi$  to  $x^{+}$  becomes
$$
\frac {1}{C} \int \exp \left(\frac {1}{2} U (x ^ {+}) - \frac {1}{2} U (x) - \frac {1}{4 \gamma} \| (\nabla^ {2} \phi^ {*} (x)) ^ {1 / 2} (x ^ {+} - x + \gamma \nabla f \circ \nabla \phi^ {*} (x)) \| ^ {2}\right) d x ^ {+} \tag {37}
$$
with  $C = (4\pi \gamma)^{d / 2}\sqrt{\operatorname*{det}\nabla^2\phi^*(x)^{-1}}$ . Now we use the smoothness of  $f$ :
$$
\begin{array}{l} U \left(x ^ {+}\right) - U (x) = f \left(\nabla \phi^ {*} \left(x ^ {+}\right)\right) - f \left(\nabla \phi^ {*} (x)\right) \\ \leq \langle \nabla^ {2} \phi^ {*} (x) \nabla f (\nabla \phi^ {*} (x)), x ^ {+} - x \rangle + \frac {L}{2} \| x ^ {+} - x \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \left\| \left(\nabla^ {2} \phi^ {*} (x)\right) ^ {1 / 2} \left(x ^ {+} - x + \gamma \nabla f \circ \nabla \phi^ {*} (x)\right) \right\| ^ {2} \\ = \| \left(\nabla^ {2} \phi^ {*} (x)\right) ^ {1 / 2} (x ^ {+} - x) \| ^ {2} + \gamma^ {2} \| \left(\nabla^ {2} \phi^ {*} (x)\right) ^ {1 / 2} \nabla f (\nabla \phi^ {*} (x)) \| ^ {2} \\ + 2 \gamma \langle \nabla^ {2} \phi^ {*} (x) \nabla f \nabla \phi^ {*} (x), x ^ {+} - x \rangle \\ \end{array}
$$
Notice that in (37), the colored terms cancel out, and what we are left with is
$$
\begin{array}{l} \mathbb {E} e ^ {\frac {1}{2} U (x ^ {+}) - \frac {1}{2} U (x)} \\ \leq \frac {1}{C} \int \exp \left(\frac {L}{4} \| x ^ {+} - x \| ^ {2} - \frac {1}{4 \gamma} \| (\nabla^ {2} \phi^ {*} (x)) ^ {1 / 2} (x ^ {+} - x) \| ^ {2} - \frac {\gamma}{4} \| (\nabla^ {2} \phi^ {*} (x)) ^ {1 / 2} \nabla f (\nabla \phi^ {*} (x)) \| ^ {2}\right) d x ^ {+} \\ \end{array}
$$
As, by our assumption,  $\nabla^2\phi^*$  is bounded from above and below, we get the exact form as in Lemma 5. The rest of the proof is the same as in Appendix D.2.
D.4 Proof of Proposition 1
In this section, we prove that Examples 1-6 satisfy our bias conditions, which, as we have seen in Section 5, implies Proposition 1. For brevity, we write  $\mathcal{F}_k$  for  $\mathcal{F}_{\tau_k}$ .
$\S$  Proof for Example 1. For randomized mid-point method, by replacing  $\widetilde{\nabla} f(x_k)$  and  $\widetilde{\nabla} f(x_{k + 1 / 2})$  with  $\nabla f(x_{k}) + U_{k + 1}^{\prime}$  and  $\nabla f(x_{k + 1 / 2}) + U_{k + 1}$  respectively, we have
$$
\begin{array}{l} x _ {k + 1 / 2} = x _ {k} - \gamma_ {k + 1} \alpha_ {k + 1} \left\{\nabla f (x _ {k}) + U _ {k + 1} ^ {\prime} \right\} + \sqrt {2 \gamma_ {k + 1} \alpha_ {k + 1}} \xi_ {k + 1} ^ {\prime}, \\ x _ {k + 1} = x _ {k} - \gamma_ {k + 1} \left\{\nabla f \left(x _ {k + 1 / 2}\right) + U _ {k + 1} \right\} + \sqrt {2 \gamma_ {k + 1}} \xi_ {k + 1}, \\ \end{array}
$$
where  $\{\alpha_{k}\}$  are i.i.d. and uniformly distributed in [0, 1],  $\{U_k\}$  and  $\{U_k^{\prime}\}$  are noises in evaluating  $\nabla f$  at the corresponding points, and  $\xi_{k},\xi_{k}^{\prime}$  are independent standard Gaussians.
Notice that the Lipschitzness of  $\nabla f$ , and the fact that  $\alpha_{k} \leq 1$  implies that the bias term  $b_{k+1} := \nabla f(x_{k+1/2}) - \nabla f(x_{k})$  satisfies
$$
\begin{array}{l} \mathbb {E} \big [ \| b _ {k + 1} \| ^ {2} | \mathcal {F} _ {k} \big ] \leq L ^ {2} \mathbb {E} \big [ \| x _ {k + 1 / 2} - x _ {k} \| ^ {2} | \mathcal {F} _ {k} \big ] \\ \leq L ^ {2} \left(\gamma_ {k + 1} ^ {2} \mathbb {E} \left[ \| \nabla f (x _ {k}) + U _ {k + 1} ^ {\prime} \| ^ {2} \mid \mathscr {F} _ {k} \right] + 2 \gamma_ {k + 1} d\right) \\ \leq 2 L ^ {2} \gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + 2 L ^ {2} \gamma_ {k + 1} ^ {2} \sigma^ {2} + 2 L ^ {2} d \gamma_ {k + 1} \\ = O \left(\gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + \gamma_ {k + 1}\right). \\ \end{array}
$$
$\S$  Proof for Example 2. Recall that the new algorithm Optimistic Randomized Mid-Point Method has the iterates
$$
x _ {k + 1 / 2} = x _ {k} - \gamma_ {k + 1} \alpha_ {k + 1} \widetilde {\nabla} f \left(x _ {k - \frac {1}{2}}\right) + \sqrt {2 \gamma_ {k + 1} \alpha_ {k + 1}} \xi_ {k + 1} ^ {\prime},
$$
$$
x _ {k + 1} = x _ {k} - \gamma_ {k + 1} \widetilde {\nabla} f (x _ {k + 1 / 2}) + \sqrt {2 \gamma_ {k + 1}} \xi_ {k + 1},
$$
where  $\{\alpha_k\}$ ,  $\xi_k, \xi_k'$ , and  $\widetilde{\nabla} f$  are the same as in (RMM), and the noise and bias are  $U_{k+1} := \widetilde{\nabla} f(x_{k+1/2}) - \nabla f(x_{k+1/2})$  and  $b_{k+1} := \nabla f(x_{k+1/2}) - \nabla f(x_k)$ . We have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| b _ {k + 1} \right\| ^ {2} \mid \mathcal {F} _ {k} \right] = \mathbb {E} \left[ \left\| \nabla f \left(x _ {k + 1 / 2}\right) - \nabla f \left(x _ {k}\right) \right\| ^ {2} \mid \mathcal {F} _ {k} \right] \\ \leq L ^ {2} \mathbb {E} \left[ \left\| x _ {k + 1 / 2} - x _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k} \right] \\ = L ^ {2} \mathbb {E} \big [ \| - \gamma_ {k + 1} \alpha_ {k + 1} \widetilde {\nabla} f (x _ {k - \frac {1}{2}}) + \sqrt {2 \gamma_ {k + 1} \alpha_ {k + 1}} \xi_ {k + 1} ^ {\prime} \| ^ {2} \mid \mathcal {F} _ {k} \big ] \\ \leq 2 L ^ {2} \gamma_ {k + 1} ^ {2} \mathbb {E} [ \| \nabla f (x _ {k - \frac {1}{2}}) \| | \mathcal {F} _ {k} ] + 2 L ^ {2} \gamma_ {k + 1} ^ {2} \sigma^ {2} + 4 L ^ {2} d \gamma_ {k + 1}. \\ \end{array}
$$
Similar to the proof for Example 6, notice that  $\| \nabla f(x_{k - \frac{1}{2}})\| ^2\leq 2\| \nabla f(x_{k - \frac{1}{2}}) - \nabla f(x_k)\| ^2 +$ $2\| \nabla f(x_{k})\|^{2}$ . As  $\gamma_{k}\to 0$ , one can assume that  $2L^{2}\gamma_{k + 1}^{2} < \frac{1}{2}$ , and we get
$$
\mathbb {E} \big [ \| b _ {k + 1} \| ^ {2} | \mathcal {F} _ {k} \big ] \leq 4 L ^ {2} \gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + 4 L ^ {2} \gamma_ {k + 1} ^ {2} \sigma^ {2} + 8 L ^ {2} d \gamma_ {k + 1} = O (\gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + \gamma_ {k + 1}),
$$
$\S$  Proof for Example 3. The iterates of stochastic Runge-Kutta Langevin algorithm is as follows:
$$
h _ {1} = x _ {k} + \sqrt {2 \gamma_ {k + 1}} \left[ (1 / 2 + 1 / \sqrt {6}) \xi_ {k + 1} + \xi_ {k + 1} ^ {\prime} / \sqrt {1 2} \right]
$$
$$
h _ {2} = x _ {k} - \gamma_ {k + 1} \left\{\nabla f (x _ {k}) + U _ {k + 1} ^ {\prime} \right\} + \sqrt {2 \gamma_ {k + 1}} \left[ (1 / 2 - 1 / \sqrt {6}) \xi_ {k + 1} + \xi_ {k + 1} ^ {\prime} / \sqrt {1 2} \right]
$$
$$
x _ {k + 1} = x _ {k} - \frac {\gamma_ {k + 1}}{2} (\nabla f (h _ {1}) + \nabla f (h _ {2})) + \gamma_ {k + 1} U _ {k + 1} + \sqrt {2 \gamma_ {k + 1}} \xi_ {k + 1},
$$
where  $\xi_{k + 1}$  and  $\xi_{k + 1}^{\prime}$  are independent standard Gaussian random variables independent of  $x_{k}$ , and  $U_{k + 1}$  and  $U_{k + 1}^{\prime}$  are noise in the evaluation of  $f$ .
$$
b _ {k + 1} = \frac {1}{2} (\nabla f (h _ {1}) - \nabla f (x _ {k})) + \frac {1}{2} (\nabla f (h _ {2}) - \nabla f (x _ {k})).
$$
$$
\mathbb {E} \big [ \| \nabla f (h _ {1}) - \nabla f (x _ {k}) \| ^ {2} | \mathcal {F} _ {k} \big ] \leq 2 L ^ {2} d (1 / 4 + 1 / 6 + 1 / 1 2) \gamma_ {k + 1} = O (\gamma_ {k + 1}),
$$
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla f (h _ {2}) - \nabla f (x _ {k}) \| ^ {2} \mid \mathcal {F} _ {k} \right] \leq 2 L ^ {2} \left(\gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + 2 \gamma_ {k + 1} ^ {2} \sigma^ {2} + 2 d (1 / 4 - 1 / 6 + 1 / 1 2) \gamma_ {k + 1}\right) \\ = \mathcal {O} (\gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + \gamma_ {k + 1}). \\ \end{array}
$$
We thus have
$$
\begin{array}{l} \mathbb {E} \left[ \| b _ {k + 1} \| ^ {2} \mid \mathcal {F} _ {t} \right] \leq \frac {1}{2} \mathbb {E} \left[ \| \nabla f (h _ {1}) - \nabla f (x _ {k}) \| ^ {2} \mid \mathcal {F} _ {k} \right] + \frac {1}{2} \mathbb {E} \left[ \| \nabla f (h _ {2}) - \nabla f (x _ {k}) \| ^ {2} \mid \mathcal {F} _ {k} \right] \\ = O (\gamma_ {k + 1} ^ {2} \left\| \nabla f (x _ {k}) \right\| ^ {2} + \gamma_ {k + 1}), \\ \end{array}
$$
$\S$  Proof for Example 4. Suppose  $\phi$  is a Legendre function [52] for  $\mathbb{R}^d$ , and consider the iterates
$$
x _ {k + 1} = x _ {k} - \gamma_ {k + 1} \nabla f \left(\nabla \phi^ {*} \left(x _ {k}\right)\right) + \sqrt {2 \gamma_ {k + 1}} \left(\nabla^ {2} \phi^ {*} \left(x _ {k}\right) ^ {- 1}\right) ^ {1 / 2} \xi_ {k + 1},
$$
where  $\phi^{*}$  is the Fenchel dual of  $\phi$ , that is,  $\phi^{*}(x) = \sup_{y\in \mathbb{R}^{d}}(\langle x,y\rangle -\phi (y))$ . Also recall that [52]
$$
\nabla \phi (\nabla \phi^ {*} (x)) = x, \quad \nabla^ {2} \phi^ {*} (\nabla \phi (x)) ^ {- 1} = \nabla^ {2} \phi (x), \quad \forall x \in \mathbb {R} ^ {d}.
$$
Let  $v = -\nabla f \circ \nabla \phi^{*}$  and  $\sigma = (\nabla^2\phi^*)^{-1 / 2}$ . First, we mention what our assumptions imply on  $f$ :
- The Lipschitzness of  $v$  corresponds to a similar condition in [31, A2]:
$$
\| \nabla f (x) - \nabla f (y) \| \leq L \| \nabla \phi (x) - \nabla \phi (y) \|
$$
- The Lipschitzness of  $\sigma$  in Frobenius norm corresponds to modified self-concordance in [31, A1]:
$$
\| \nabla^ {2} \phi (x) ^ {1 / 2} - \nabla^ {2} \phi (y) ^ {1 / 2} \| _ {F} \leq L \| \nabla \phi (x) - \nabla \phi (y) \|.
$$
- Boundedness of  $\sigma$  in Hilbert-Schmidt norm implies
$$
\left\| \nabla^ {2} \phi (x) ^ {- 1 / 2} \right\| _ {F} \leq C _ {\sigma}.
$$
- Dissipativity and weak-dissipativity of  $v$  corresponds to the conditions below, respectively:
$$
\langle \nabla \phi (x), \nabla f (x) \rangle \geq \alpha \| \nabla \phi (x) \| ^ {2} - \beta , \quad \langle \nabla \phi (x), \nabla f (x) \rangle \geq \alpha \| \nabla \phi (x) \| ^ {1 + \kappa} - \beta .
$$
If  $f$  and  $\phi$  satisfy the conditions above, then the mirror Langevin algorithm Example 4 fits into the (LRM) scheme.
Remark. Note that this version of Mirror Langevin cannot handle the case where  $e^{-f}$  is supported on a compact domain; in that case, the Hessian of  $\phi$  has to blow up near the boundary, and will not satisfy our boundedness assumption. The version of mirror Langevin we consider in this paper, though, can be thought as an adaptive conditioning method for densities supported on  $\mathbb{R}^d$ . This setting has also been studied in the literature, see [55].
§ Proof for Example 6. The iterates of (PLA) follow
We mentioned that the bias term is  $b_{k + 1} = \nabla f(x_{k + 1}) - \nabla f(x_k)$ . Now it remains to prove that it satisfies the conditions (5) and (11). We have
$$
\begin{array}{l} \mathbb {E} \left[ \| b _ {k + 1} \| ^ {2} \mid \mathcal {F} _ {k} \right] = \mathbb {E} \left[ \| \nabla f (x _ {k + 1}) - \nabla f (x _ {k}) \| ^ {2} \mid \mathcal {F} _ {k} \right] \\ \leq L ^ {2} \mathbb {E} \left[ \left\| x _ {k + 1} - x _ {k} \right\| ^ {2} \mid \mathcal {F} _ {k} \right] \\ = L ^ {2} \mathbb {E} [ \| - \gamma_ {k + 1} \nabla f (x _ {k + 1}) + \sqrt {2 \gamma_ {k + 1}} \xi_ {k + 1} \| ^ {2} \mid \mathcal {F} _ {k} ] \\ \leq 2 L ^ {2} \gamma_ {k + 1} ^ {2} \mathbb {E} \left[ \| \nabla f (x _ {k + 1}) \| ^ {2} \mid \mathcal {F} _ {k} \right] + 4 L ^ {2} d \gamma_ {k + 1}. \\ \end{array}
$$
Now, notice that  $\| \nabla f(x_{k + 1})\| ^2\leq 2\| \nabla f(x_{k + 1}) - \nabla f(x_k)\| ^2 +2\| \nabla f(x_k)\| ^2$  . As  $\gamma_{k}\to 0$  , one can assume that  $2L^{2}\gamma_{k + 1}^{2} <   \frac{1}{2}$  , and we get
$$
\mathbb {E} \big [ \| b _ {k + 1} \| ^ {2} \mid \mathcal {F} _ {k} \big ] \leq \frac {1}{2} \mathbb {E} \big [ \| b _ {k + 1} \| ^ {2} \mid \mathcal {F} _ {k} \big ] + \| \nabla f (x _ {k}) \| ^ {2} + 4 L ^ {2} d \gamma_ {k + 1},
$$
$$
\mathbb {E} \big [ \| b _ {k + 1} \| ^ {2} | \mathcal {F} _ {k} \big ] \leq 4 L ^ {2} \gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + 8 L ^ {2} d \gamma_ {k + 1} = O (\gamma_ {k + 1} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + \gamma_ {k + 1}),
$$