SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \stackrel {(b)} {\leq} \Delta_ {1} + 2 \sqrt {\Delta_ {1}} \sqrt {A} C _ {1} + 2 C _ {1} \left(\sqrt {\Delta_ {1}} + 4 \sqrt {A} C _ {1}\right) \log \frac {1}{\delta} + 4 8 \sigma^ {2 p} C _ {2} C _ {3} \\ + 8 0 \sigma^ {p} \left(\frac {3 (\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1})}{2 C _ {1}} + \frac {7}{4}\right) C _ {3} \\ \leq \Delta_ {1} + 2 \sqrt {\Delta_ {1}} \sqrt {A} C _ {1} + 2 C _ {1} \left(\sqrt {\Delta_ {1}} + 4 \sqrt {A} C _ {1}\right) \left(\log \frac {1}{\delta} + \frac {6 0 \sigma^ {p} C _ {3}}{C _ {1} ^ {2}}\right) \\ + 4 8 \sigma^ {2 p} C _ {2} C _ {3} + 1 4 0 \sigma^ {p} C _ {3} \\ \stackrel {(c)} {\leq} \Delta_ {1} + 2 \sqrt {\Delta_ {1}} \sqrt {A} C _ {1} + 2 C _ {1} \left(\sqrt {\Delta_ {1}} + 4 \sqrt {A} C _ {1}\right) \frac {\sqrt {A}}{8} + A C _ {1} ^ {2} \\ \leq \left(\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}\right) ^ {2}. \\ \end{array}
$$
For  $(a)$ , we use  $\left(\frac{1}{\lambda_t}\right)^p \leq C_2 L \eta_t$  and the induction hypothesis. For  $(b)$ , we use  $\sum_{t=1}^{T} L\left(\frac{1}{\lambda_t}\right)^p \lambda_t^2 \eta_t^2 \leq C_3$  and  $Q_t \geq 1$ . For  $(c)$ , we have
$$
\log {\frac {1}{\delta}} + \frac {6 0 \sigma^ {p} C _ {3}}{C _ {1} ^ {2}} \leq \frac {\sqrt {A}}{8}
$$
$$
4 8 \sigma^ {2 p} C _ {2} C _ {3} + 1 4 0 \sigma^ {p} C _ {3} \leq A C _ {1} ^ {2},
$$
$$
A \geq 6 4 \left(\log \frac {1}{\delta} + \frac {6 0 \sigma^ {p} C _ {3}}{C _ {1} ^ {2}}\right) ^ {2} + \frac {4 8 \sigma^ {2 p} C _ {2} C _ {3} + 1 4 0 \sigma^ {p} C _ {3}}{C _ {1} ^ {2}}.
$$
Lemma B.1. The choices of  $\eta_t$  and  $\lambda_t$  in Theorem 3.1 satisfy the condition (1)-(3) of Proposition 3.7 for
$$
C _ {1} = \frac {\sqrt {\Delta_ {1}}}{4 \sqrt {2} \gamma},
$$
$$
C _ {2} = \frac {1}{\sigma^ {p}},
$$
$$
C _ {3} = \frac {\Delta_ {1}}{2 0 4 8 \sigma^ {p} \gamma}.
$$
Proof. We verify for the first case. The second follows exactly the same. First, we have  $p > 1$  hence
$$
\eta_ {t} \lambda_ {t} \sqrt {2 L} = \frac {\sqrt {\Delta_ {1}} T ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma} \sqrt {2 L} \leq \frac {\sqrt {\Delta_ {1}}}{4 \sqrt {2} \gamma} = C _ {1}.
$$
Since  $\eta_t = \frac{\sqrt{\Delta_1} T^{\frac{1 - p}{3p - 2}}}{8\lambda_t \sqrt{L\gamma}}$ ,  $p > 1$  and  $\lambda_t \geq \left(\frac{8\gamma}{\sqrt{L\Delta_1}}\right)^{\frac{1}{p - 1}} T^{\frac{1}{3p - 2}} \sigma^{\frac{p}{p - 1}}$
$$
\begin{array}{l} \eta_ {t} \lambda_ {t} ^ {p} = \frac {\sqrt {\Delta_ {1}} T ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma} \lambda_ {t} ^ {p - 1} \\ \geq \frac {\sqrt {\Delta_ {1}} T ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma} \frac {8 \gamma}{\sqrt {L \Delta_ {1}}} T ^ {\frac {p - 1}{3 p - 2}} \sigma^ {p} \\ = \frac {\sigma^ {p}}{L} \\ \end{array}
$$
$$
\frac {1}{L \eta_ {t}} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \leq \frac {1}{\sigma^ {p}} = C _ {2}.
$$
Finally, we have  $\lambda_t \geq 32^{1/p} \sigma T^{\frac{1}{3p-2}}$  hence
$$
\left(\frac {1}{\lambda_ {t}}\right) ^ {p} T ^ {\frac {p}{3 p - 2}} \leq \frac {1}{3 2 \sigma^ {p}}.
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} L \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \lambda_ {t} ^ {2} \eta_ {t} ^ {2} = \sum_ {t = 1} ^ {T} L \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \left(\frac {\sqrt {\Delta_ {1}} T ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma}\right) ^ {2} \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} L \left(\frac {1}{\lambda_ {t}}\right) ^ {p} T \cdot T ^ {\frac {2 - 2 p}{3 p - 2}} \frac {\Delta_ {1}}{6 4 L \gamma} \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} T ^ {\frac {p}{3 p - 2}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {1}{3 2 \sigma^ {p}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \\ = \frac {1}{3 2 \sigma^ {p}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \leq \frac {\Delta_ {1}}{2 0 4 8 \sigma^ {p} \gamma}. \\ \end{array}
$$
Proof of Theorem 3.1. Note that  $\eta \leq \frac{T^{\frac{1 - p}{3p - 2}}}{16\sqrt{90}L\gamma} \leq \frac{1}{L}$ . We have that with probability at least  $1 - \delta$ , event  $E(\delta)$  happens. Conditioning on this event, we verify the conditions of Proposition 3.7. We select the following constants
$$
C _ {1} = \frac {\sqrt {\Delta_ {1}}}{4 \sqrt {2} \gamma}; \quad C _ {2} = \frac {1}{\sigma^ {p}}; \quad C _ {3} = \frac {\Delta_ {1}}{2 0 4 8 \sigma^ {p} \gamma}; \quad A = 2 5 6 \gamma^ {2}.
$$
We verify in Lemma B.1 that for these choice of constants, conditions (1)-(3) of Proposition 3.7 are satisfied. Furthermore, we have
$$
\begin{array}{l} 6 4 \left(\log \frac {1}{\delta} + \frac {6 0 \sigma^ {p} C _ {3}}{C _ {1} ^ {2}}\right) ^ {2} + \frac {4 8 \sigma^ {2 p} C _ {2} C _ {3} + 1 4 0 \sigma^ {p} C _ {3}}{C _ {1} ^ {2}} \\ = 6 4 \left(\log \frac {1}{\delta} + 6 0 \log \frac {1}{\delta} \frac {3 2}{\Delta_ {1}} \frac {\Delta_ {1}}{2 0 4 8}\right) ^ {2} + \left(4 8 \frac {\Delta_ {1}}{2 0 4 8} + 1 4 0 \frac {\Delta_ {1}}{2 0 4 8}\right) \frac {3 2}{\Delta_ {1}} \\ \leq 2 5 6 \gamma^ {2} = A. \\ \end{array}
$$
We only need to show that, for all  $t$ ,  $\| \nabla f(x_{t})\| \leq \frac{\lambda_{t}}{2}$ . We will show this by induction. Indeed, for the base case we have  $\| \nabla f(x_{1})\| \leq \sqrt{2L\Delta_{1}} \leq \frac{\lambda_{1}}{2}$ . Suppose that it is true for all  $t \leq k$ . We will prove that  $\| \nabla f(x_{k + 1})\| \leq \frac{\lambda_{k + 1}}{2}$ . By Proposition 3.7 and the induction hypothesis
$$
\Delta_ {k + 1} \leq \left(\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}\right) ^ {2} \leq \left(\sqrt {\Delta_ {1}} + \frac {\sqrt {\Delta_ {1}}}{2 \sqrt {2} \gamma} \times 1 6 \gamma\right) ^ {2} \leq 4 5 \Delta_ {1}.
$$
$$
\| \nabla f (x _ {k + 1}) \| \leq \sqrt {2 L \Delta_ {k + 1}} \leq \sqrt {9 0 L \Delta_ {1}} \leq \frac {\lambda_ {k + 1}}{2}
$$
as needed. From Proposition 3.7, we have
$$
\frac {\eta}{2} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| ^ {2} + \Delta_ {k + 1} \leq 4 5 \Delta_ {1}.
$$
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {9 0 \Delta_ {1}}{\eta T} = 7 2 0 \sqrt {\Delta_ {1} L} \gamma \max  \left\{\left(\frac {8 \gamma}{\sqrt {L \Delta_ {1}}}\right) ^ {\frac {1}{p - 1}} T ^ {\frac {2 - 2 p}{3 p - 2}} \sigma^ {\frac {p}{p - 1}}; 2 \sqrt {9 0 L \Delta_ {1}} T ^ {\frac {1 - 2 p}{3 p - 2}}; 3 2 ^ {\frac {1}{p}} \sigma T ^ {\frac {2 - 2 p}{3 p - 2}} \right\}.
$$
Theorem B.2. Assume that  $f$  satisfies Assumption (1'), (2), (3), (4). Let  $\gamma = \max \left\{\log \frac{1}{\delta}; 1\right\}$  and  $\Delta_1 = f(x_1) - f^*$ . For unknown  $T$ , we choose  $\lambda_t$  and  $\eta_t$  such that
$$
\begin{array}{l} \lambda_ {t} = \max \left\{\left(\frac {8 \gamma}{\sqrt {L \Delta_ {1}}}\right) ^ {\frac {1}{p - 1}} \left(2 t (1 + \log t) ^ {2}\right) ^ {\frac {1}{3 p - 2}} \sigma^ {\frac {p}{p - 1}}; 2 \sqrt {9 0 L \Delta_ {1}}; 3 2 ^ {\frac {1}{p}} \sigma (2 t (1 + \log t) ^ {2}) ^ {\frac {1}{3 p - 2}} \right\}, \\ \eta_ {t} = \frac {\sqrt {\Delta_ {1}} \left(2 t \left(1 + \log t\right) ^ {2}\right) ^ {\frac {1 - p}{3 p - 2}}}{8 \lambda_ {t} \sqrt {L} \gamma}. \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| ^ {2} \leq 7 2 0 \sqrt {\Delta_ {1} L} \gamma \max  \left\{\left(\frac {8 \gamma}{\sqrt {L} \Delta_ {1}}\right) ^ {\frac {1}{p - 1}} \left(2 (1 + \log T) ^ {2}\right) ^ {\frac {p}{3 p - 2}} \sigma^ {\frac {p}{p - 1}} T ^ {\frac {2 - 2 p}{3 p - 2}}; \right. \\ \left. 2 \sqrt {9 0 L \Delta_ {1}} \left(2 (1 + \log T) ^ {2}\right) ^ {\frac {p - 1}{3 p - 2}} T ^ {\frac {1 - 2 p}{3 p - 2}}; 3 2 ^ {\frac {1}{p}} \sigma \left(2 (1 + \log T) ^ {2}\right) ^ {\frac {p}{3 p - 2}} T ^ {\frac {2 - 2 p}{3 p - 2}} \right\}. \\ \end{array}
$$
We again verify the conditions of Proposition 3.7 for the choices of  $\eta_t$  and  $\lambda_t$  in Theorem B.2.
Lemma B.3. The choices of  $\eta_t$  and  $\lambda_t$  in Theorem B.2 satisfy the condition (1)-(3) of Proposition 3.7 for
The proof utilizes the following fact:
Fact B.4. We have  $\sum_{t=1}^{\infty} \frac{1}{2t(1 + \log t)^2} < 1$ .
Proof. First, we have  $p > 1$  hence
$$
\begin{array}{l} \eta_ {t} \lambda_ {t} \sqrt {2 L} = \frac {\sqrt {\Delta_ {1}} \left(2 t \left(1 + \log t\right) ^ {2}\right) ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma} \sqrt {2 L} \\ \leq \frac {\sqrt {\Delta_ {1}}}{4 \sqrt {2} \gamma} = C _ {1}. \\ \end{array}
$$
Since  $\eta_t = \frac{\sqrt{\Delta_1} T^{\frac{1 - p}{3p - 2}}}{8\lambda_t \sqrt{L}\gamma}$ ,  $p > 1$  and  $\lambda_t \geq \left(\frac{8\gamma}{\sqrt{L\Delta_1}}\right)^{\frac{1}{p - 1}} \left(2t(1 + \log t)^2\right)^{\frac{1 - p}{3p - 2}} \sigma^{\frac{p}{p - 1}}$
$$
\begin{array}{l} \eta_ {t} \lambda_ {t} ^ {p} = \frac {\sqrt {\Delta_ {1}} \left(2 t \left(1 + \log t\right) ^ {2}\right) ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma} \lambda_ {t} ^ {p - 1} \\ \geq \frac {\sqrt {\Delta_ {1}} \left(2 t \left(1 + \log t\right) ^ {2}\right) ^ {\frac {1 - p}{3 p - 2}}}{8 \sqrt {L} \gamma} \frac {8 \gamma}{\sqrt {L \Delta_ {1}}} \left(2 t \left(1 + \log t\right) ^ {2}\right) ^ {\frac {p - 1}{3 p - 2}} \sigma^ {p} \\ = \frac {\sigma^ {p}}{L}, \\ \end{array}
$$
Finally, we have  $\lambda_t \geq 32^{\frac{1}{p}}\sigma \left(2t(1 + \log t)^2\right)^{\frac{1}{3p - 2}}$ , hence
$$
\left(\frac {1}{\lambda_ {t}}\right) ^ {p} \left(2 t (1 + \log t) ^ {2}\right) ^ {\frac {p}{3 p - 2}} \leq \frac {1}{3 2 \sigma^ {p}}. \tag {13}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} L \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \lambda_ {t} ^ {2} \eta_ {t} ^ {2} = \sum_ {t = 1} ^ {T} L \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \left(2 t (1 + \log t) ^ {2}\right) ^ {\frac {2 - 2 p}{3 p - 2}} \left(\frac {\sqrt {\Delta_ {1}}}{8 \sqrt {L} \gamma}\right) ^ {2} \\ = \sum_ {t = 1} ^ {T} L \frac {1}{2 t (1 + \log t) ^ {2}} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \left(2 t (1 + \log t) ^ {2}\right) ^ {\frac {p}{3 p - 2}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \\ \leq \sum_ {t = 1} ^ {T} L \frac {1}{2 t (1 + \log t) ^ {2}} \frac {1}{3 2 \sigma^ {p}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \tag {by(13)} \\ = \frac {1}{3 2 \sigma^ {p}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \sum_ {t = 1} ^ {T} \frac {1}{2 t (1 + \log t) ^ {2}} \\ \leq \frac {1}{3 2 \sigma^ {p}} \frac {\Delta_ {1}}{6 4 \gamma^ {2}} \leq \frac {\Delta_ {1}}{2 0 4 8 \sigma^ {p} \gamma}. \quad (\text {b y}) \\ \end{array}
$$
Proof of Theorem B.2. Note that
$$
\begin{array}{l} \eta_ {t} = \frac {\sqrt {\Delta_ {1}} \left(2 t (1 + \log t) ^ {2}\right) ^ {\frac {1 - p}{3 p - 2}}}{8 \lambda_ {t} \sqrt {L} \gamma} \\ \leq \frac {\left(2 t (1 + \log t) ^ {2}\right) ^ {\frac {1 - p}{3 p - 2}}}{1 6 L \gamma \sqrt {9 0}} \\ \leq \frac {1}{L}. \\ \end{array}
$$
Note that with Lemma B.3, verifying the conditions of Proposition 3.7 is identical to the proof of theorem 3.1. We have that with probability at least  $1 - \delta$ , event  $E(\delta)$  from 3.7 happens. We have with probability at least  $1 - \delta$ :
$$
\frac {1}{2} \sum_ {t = 1} ^ {T} \eta_ {t} \left\| \nabla f (x _ {t}) \right\| ^ {2} + \Delta_ {k + 1} \leq 4 5 \Delta_ {1}.
$$
Since  $\eta_t$  is decreasing, we have
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {9 0 \Delta_ {1}}{T \eta_ {T}}.
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| ^ {2} \leq 7 2 0 \sqrt {\Delta_ {1} L} \gamma \max  \left\{\left(\frac {8 \gamma}{\sqrt {L \Delta_ {1}}}\right) ^ {\frac {1}{p - 1}} \left(2 (1 + \log T) ^ {2}\right) ^ {\frac {p}{3 p - 2}} \sigma^ {\frac {p}{p - 1}} T ^ {\frac {2 - 2 p}{3 p - 2}}; \right. \\ \left. 2 \sqrt {9 0 L \Delta_ {1}} \left(2 (1 + \log T) ^ {2}\right) ^ {\frac {p - 1}{3 p - 2}} T ^ {\frac {1 - 2 p}{3 p - 2}}; 3 2 ^ {\frac {1}{p}} \sigma \left(2 (1 + \log T) ^ {2}\right) ^ {\frac {p}{3 p - 2}} T ^ {\frac {2 - 2 p}{3 p - 2}} \right\}. \\ \end{array}
$$
C Missing Proofs from Section 4
Lemma C.1. Suppose that  $\eta_t \leq \frac{1}{4L}$  and assume  $f$  satisfies Assumption (1), (2), (3) as well as the following condition
$$
f (y) - f (x) \leq \left\langle \nabla f (x), y - x \right\rangle + G \| y - x \| + \frac {L}{2} \| y - x \| ^ {2}, \quad \forall y, x \in \mathcal {X}. \tag {14}
$$
Then the iterate sequence  $(x_{t})_{t\geq 1}$  output by Algorithm 2 satisfies the following:
$$
\begin{array}{l} \eta_ {t} \Delta_ {t + 1} \leq {\bf D} _ {\psi} (x ^ {*}, x _ {t}) - {\bf D} _ {\psi} (x ^ {*}, x _ {t + 1}) + \eta_ {t} \left<   x ^ {*} - x _ {t}, \theta_ {t} ^ {u} \right> + \eta_ {t} \left<   x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right> \\ + 2 \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) + 2 \eta_ {t} ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] + 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} + 2 G ^ {2} \eta_ {t} ^ {2}. \\ \end{array}
$$
Proof. By condition (14) and convexity,
$$
\begin{array}{l} f \left(x _ {t + 1}\right) - f \left(x ^ {*}\right) \leq \underbrace {f \left(x _ {t + 1}\right) - f \left(x _ {t}\right)} _ {\text {c o n d i t i o n (1 4)}} + \underbrace {f \left(x _ {t}\right) - f \left(x ^ {*}\right)} _ {\text {c o n v e x i t y}} \\ \leq \left\langle \nabla f \left(x _ {t}\right), x _ {t + 1} - x _ {t} \right\rangle + \frac {L}{2} \left\| x _ {t} - x _ {t + 1} \right\| ^ {2} + G \left\| x _ {t} - x _ {t + 1} \right\| + \left\langle \nabla f \left(x _ {t}\right), x _ {t} - x ^ {*} \right\rangle \\ = \left\langle \nabla f \left(x _ {t}\right), x _ {t + 1} - x ^ {*} \right\rangle + \frac {L}{2} \left\| x _ {t} - x _ {t + 1} \right\| ^ {2} + G \left\| x _ {t} - x _ {t + 1} \right\| \\ = \left\langle \theta_ {t}, x ^ {*} - x _ {t + 1} \right\rangle + \left\langle \widetilde {\nabla} f (x _ {t}), x _ {t + 1} - x ^ {*} \right\rangle + \frac {L}{2} \left\| x _ {t} - x _ {t + 1} \right\| ^ {2} + G \left\| x _ {t} - x _ {t + 1} \right\|. \\ \end{array}
$$
By the optimality condition, we have
$$
\left\langle \eta_ {t} \widetilde {\nabla} f (x _ {t}) + \nabla_ {x} \mathbf {D} _ {\psi} \left(x _ {t + 1}, x _ {t}\right), x ^ {*} - x _ {t + 1} \right\rangle \geq 0
$$
$$
\left\langle \eta_ {t} \widetilde {\nabla} f (x _ {t}), x _ {t + 1} - x ^ {*} \right\rangle \leq \left\langle \nabla_ {x} \mathbf {D} _ {\psi} \left(x _ {t + 1}, x _ {t}\right), x ^ {*} - x _ {t + 1} \right\rangle .
$$
$$
\begin{array}{l} \left\langle \nabla_ {x} \mathbf {D} _ {\psi} \left(x _ {t + 1}, x _ {t}\right), x ^ {*} - x _ {t + 1} \right\rangle = \left\langle \nabla \psi \left(x _ {t + 1}\right) - \nabla \psi \left(x _ {t}\right), x ^ {*} - x _ {t + 1} \right\rangle \\ = \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {t}\right) - \mathbf {D} _ {\psi} \left(x _ {t + 1}, x _ {t}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {t + 1}\right). \\ \end{array}
$$
$$
\begin{array}{l} \eta_ {t} \left\langle \widetilde {\nabla} f (x _ {t}), x _ {t + 1} - x ^ {*} \right\rangle \leq \mathbf {D} _ {\psi} (x ^ {*}, x _ {t}) - \mathbf {D} _ {\psi} (x ^ {*}, x _ {t + 1}) - \mathbf {D} _ {\psi} (x _ {t + 1}, x _ {t}) \\ \leq \mathbf {D} _ {\psi} (x ^ {*}, x _ {t}) - \mathbf {D} _ {\psi} (x ^ {*}, x _ {t + 1}) - \frac {1}{2} \| x _ {t + 1} - x _ {t} \| ^ {2}, \\ \end{array}
$$
where we have used that  $\mathbf{D}_{\psi}(x_{t + 1},x_t)\geq \frac{1}{2}\| x_{t + 1} - x_t\|^2$  by the strong convexity of  $\psi$
Combining the two inequalities, and using the assumption that  $L\eta_{t} \leq \frac{1}{4}$ , we obtain
$$
\begin{array}{l} \eta_ {t} \Delta_ {t + 1} + \mathbf {D} _ {\psi} (x ^ {*}, x _ {t + 1}) - \mathbf {D} _ {\psi} (x ^ {*}, x _ {t}) \\ \leq \eta_ {t} \left\langle \theta_ {t}, x ^ {*} - x _ {t + 1} \right\rangle + \frac {L \eta_ {t}}{2} \left\| x _ {t} - x _ {t + 1} \right\| ^ {2} + G \eta_ {t} \left\| x _ {t} - x _ {t + 1} \right\| - \frac {1}{2} \left\| x _ {t + 1} - x _ {t} \right\| ^ {2} \\ \leq \eta_ {t} \left\langle \theta_ {t}, x ^ {*} - x _ {t} \right\rangle + \eta_ {t} \left\langle \theta_ {t}, x _ {t} - x _ {t + 1} \right\rangle - \frac {3}{8} \left\| x _ {t + 1} - x _ {t} \right\| ^ {2} + G \eta_ {t} \left\| x _ {t} - x _ {t + 1} \right\| \\ \leq \eta_ {t} \left\langle \theta_ {t}, x ^ {*} - x _ {t} \right\rangle + \eta_ {t} ^ {2} \left\| \theta_ {t} \right\| _ {*} ^ {2} + 2 G ^ {2} \eta_ {t} ^ {2} \\ \leq \eta_ {t} \left\langle \theta_ {t} ^ {u} + \theta_ {t} ^ {b}, x ^ {*} - x _ {t} \right\rangle + 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {u} \right\| _ {*} ^ {2} + 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} + 2 G ^ {2} \eta_ {t} ^ {2}. \\ \end{array}
$$
This is what we want to show.
Proof of Lemma 4.7. We have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \times \exp \left(\left(\frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \stackrel {(a)} {\leq} \mathbb {E} \left[ \exp \left(z _ {t} \left(\eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {u} \right\rangle + 2 \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right)\right)\right) \mid \mathcal {F} _ {t - 1} \right] \\ \stackrel {(b)} {\leq} \exp \left(\mathbb {E} \left[ \frac {3}{4} \left(z _ {t} \left(\eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {u} \right\rangle + 2 \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right)\right)\right) ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \stackrel {(c)} {\leq} \exp \left(\left(\frac {3}{2} z _ {t} ^ {2} \eta_ {t} ^ {2} \| x ^ {*} - x _ {t} \| ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] + 6 z _ {t} ^ {2} \eta_ {t} ^ {4} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {4} \mid \mathcal {F} _ {t - 1} \right]\right)\right) \\ \stackrel {(d)} {\leq} \exp \left(\left(\frac {3}{2} z _ {t} ^ {2} \eta_ {t} ^ {2} \| x ^ {*} - x _ {t} \| ^ {2} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \stackrel {(e)} {\leq} \exp \left(\left(\frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right). \\ \end{array}
$$
For  $(a)$ , we use Lemma 4.5. For  $(b)$ , we use Lemma 2.2. Notice that
$$
\mathbb {E} \left[ \langle x ^ {*} - x _ {t}, \theta_ {t} ^ {u} \rangle \right] = \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] \right] = 0,
$$
and since  $\| \theta_t^u\|_*\leq 2\lambda_t$  , we have
$$
\begin{array}{l} \left| \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {u} \right\rangle + 2 \eta_ {t} ^ {2} \left(\left\| \theta_ {t} ^ {u} \right\| _ {*} ^ {2} - \mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \right| \\ \leq \eta_ {t} \| x ^ {*} - x _ {t} \| \| \theta_ {t} ^ {u} \| _ {*} + 2 \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| _ {*} ^ {2} + \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ \leq 2 \eta_ {t} \lambda_ {t} \| x ^ {*} - x _ {t} \| + 1 6 \eta_ {t} ^ {2} \lambda_ {t} ^ {2} \\ \leq 2 \eta_ {t} \lambda_ {t} \sqrt {2 \mathbf {D} _ {\psi} (x ^ {*} , x _ {t})} + 1 6 \eta_ {t} ^ {2} \lambda_ {t} ^ {2}. \\ \end{array}
$$
Thus,  $z_{t} \leq \frac{1}{2\eta_{t}\lambda_{t}\sqrt{2\mathbf{D}_{\psi}(x^{*},x_{t})} + 16\eta_{t}^{2}\lambda_{t}^{2}}$ . For (c), we use the inequalities  $(a + b)^{2} \leq 2a^{2} + 2b^{2}$  and  $\mathbb{E}\left[(X - \mathbb{E}[X])^{2}\right] \leq \mathbb{E}\left[X^{2}\right]$ . For (d), we use the fact  $\| \theta_t^u\| _*^2 \leq 4\lambda_t^2$  to get  $\mathbb{E}\left[\| \theta_t^u\| _*^4\mid \mathcal{F}_{t - 1}\right] \leq 4\lambda_t^2\mathbb{E}\left[\| \theta_t^u\| _*^2\mid \mathcal{F}_{t - 1}\right]$ . For (e), we use the fact that  $\| \theta_t^u\| _*\leq 2\lambda_t$  and
$$
z _ {t} \eta_ {t} \left\| x ^ {*} - x _ {t} \right\| \leq \frac {\eta_ {t} \left\| x ^ {*} - x _ {t} \right\|}{2 \eta_ {t} \lambda_ {t} \sqrt {2 \mathbf {D} _ {\psi} (x ^ {*} , x _ {t})}} \leq \frac {1}{2 \lambda_ {t}}.
$$
We obtain  $\mathbb{E}\left[\exp \left(Z_t\right)\mid \mathcal{F}_{t - 1}\right]\leq 1$  . Therefore
$$
\mathbb {E} \left[ \exp \left(S _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] = \exp \left(S _ {t - 1}\right) \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \leq \exp \left(S _ {t - 1}\right).
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \Delta_ {t + 1} + \sum_ {t = 1} ^ {k} \left(z _ {t} \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {t + 1}\right) - z _ {t} \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {t}\right)\right) \\ \leq \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right\rangle + 2 \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} \\ \end{array}
$$
$$
+ \sum_ {t = 1} ^ {k} \left(\left(2 z _ {t} \eta_ {t} ^ {2} + \frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right).
$$
Note that we have  $z_{t}$  is a decreasing sequence, hence the LHS of the above inequality can be bounded by
$$
\begin{array}{l} \mathrm {L H S} = \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \Delta_ {t + 1} + z _ {k} \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) - z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}) + \sum_ {t = 2} ^ {k} (z _ {k - 1} - z _ {k}) \mathbf {D} _ {\psi} (x ^ {*}, x _ {k}) \\ \geq \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \Delta_ {t + 1} + z _ {k} \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) - z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}). \\ \end{array}
$$
We obtain from here the desired inequality.
Proof of Proposition 4.8. We will prove by induction that on  $k$
$$
\sum_ {i = 1} ^ {k} \eta_ {i} \Delta_ {i + 1} + \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) \leq \frac {1}{2} (R _ {1} + 8 A C _ {1}) ^ {2}.
$$
The base case  $k = 0$  is trivial. We have  $\mathbf{D}_{\psi}(x^{*}, x_{1}) = \frac{R_{1}^{2}}{2}$ . Suppose the statement is true for all  $t \leq k \leq \ell$ . Now, we show for  $k + 1$ . Recall that
$$
z _ {t} = \frac {1}{2 \eta_ {t} \lambda_ {t} \max _ {i \leq t} \sqrt {2 \mathbf {D} _ {\psi} (x ^ {*} , x _ {i})} + 1 6 Q \eta_ {t} ^ {2} \lambda_ {t} ^ {2}}.
$$
Let us choose  $Q = A > 1$ . By the induction hypothesis, we have  $\max_{i\leq t}\sqrt{2\mathbf{D}_{\psi}(x^{*},x_{i})} \leq R_{1} + 8AC_{1}$ , which implies
$$
z _ {k} \geq \frac {1}{2 \eta_ {k} \lambda_ {k} (R _ {1} + 8 A C _ {1}) + 1 6 A \eta_ {k} ^ {2} \lambda_ {k} ^ {2}} = \frac {1}{2 C _ {1} (R _ {1} + 1 6 A C _ {1})}.
$$
For an upperbound, since  $\sqrt{2\mathbf{D}_{\psi}(x^{*},x_{1})} = R_{1}$ , we have:
$$
z _ {t} \leq \frac {1}{2 C _ {1} \left(R _ {1} + 8 A C _ {1}\right)}.
$$
Since  $z_{k}$  is a decreasing sequence, we have
$$
\begin{array}{l} z _ {k} \sum_ {t = 1} ^ {k} \eta_ {t} \Delta_ {t + 1} + z _ {k} \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) \leq z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}) + \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right\rangle + 2 \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} \\ + \sum_ {t = 1} ^ {k} \left(\left(2 z _ {t} \eta_ {t} ^ {2} + \frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right). \\ \end{array}
$$
By the choice of  $\lambda_t$ , for all  $t \leq k$ ,  $\|\nabla f(x_t)\|_* \leq \frac{\lambda_t}{2}$ , we can apply Lemma 2.1 and have
$$
\left\| \theta_ {t} ^ {b} \right\| _ {*} \leq 4 \sigma^ {p} \lambda_ {t} ^ {1 - p};
$$
$$
\mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] \leq 4 0 \sigma^ {p} \lambda_ {t} ^ {2 - p}.
$$