SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} z _ {k} \sum_ {t = 1} ^ {k} \eta_ {t} \Delta_ {t + 1} + z _ {k} \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) \\ \leq z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}) + \log \frac {1}{\delta} + 4 \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \sigma^ {p} \lambda_ {t} ^ {1 - p} \sqrt {2 \mathbf {D} _ {\psi} (x ^ {*} , x _ {t})} + 3 2 \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} ^ {2} \sigma^ {2 p} \lambda_ {t} ^ {2 - 2 p} \\ + 4 0 \sum_ {t = 1} ^ {k} \left(\left(2 z _ {t} \eta_ {t} ^ {2} + \frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \sigma^ {p} \lambda_ {t} ^ {2 - p}\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}) + \log \frac {1}{\delta} + \frac {2 C _ {1} (R _ {1} + 8 A C _ {1}) \sigma^ {p}}{C _ {1} (R _ {1} + 8 A C _ {1})} \sum_ {t = 1} ^ {k} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} + \frac {1 6 C _ {1} ^ {2} \sigma^ {2 p}}{C _ {1} (R _ {1} + 8 A C _ {1})} \sum_ {t = 1} ^ {k} \left(\frac {1}{\lambda_ {t}}\right) ^ {2 p} \\ + 4 0 \left(\frac {C _ {1} ^ {2}}{C _ {1} \left(R _ {1} + 8 A C _ {1}\right)} + \frac {3}{8} + \frac {6 C _ {1} ^ {4}}{C _ {1} ^ {2} \left(R _ {1} + 8 A C _ {1}\right) ^ {2}}\right) \sigma^ {p} \sum_ {t = 1} ^ {k} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \\ \leq \frac {R _ {1} ^ {2}}{4 \left(C _ {1} R _ {1} + 8 A C _ {1} ^ {2}\right)} + \log \frac {1}{\delta} + 2 \sigma^ {p} C _ {2} + \frac {2 \sigma^ {2 p} C _ {2} C _ {3}}{A} + 2 4 \sigma^ {p} C _ {2} \\ \leq \frac {R _ {1} ^ {2}}{4 \left(C _ {1} R _ {1} + 8 A C _ {1} ^ {2}\right)} + A, \\ \end{array}
$$
where for the last inequality we use  $\sum_{t=1}^{k}\left(\frac{1}{\lambda_t}\right)^p \leq C_2$  and  $\left(\frac{1}{\lambda_t}\right)^{2p} \leq C_3\left(\frac{1}{\lambda_t}\right)^p$ . We obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {k} \eta_ {t} \Delta_ {t + 1} + \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) \leq 2 C _ {1} (R _ {1} + 1 6 A C _ {1}) \left(\frac {R _ {1} ^ {2}}{4 (C _ {1} R _ {1} + 8 A C _ {1} ^ {2})} + A\right) \\ = \frac {1}{2} R _ {1} ^ {2} + \frac {4 A C _ {1} ^ {2} R _ {1} ^ {2}}{C _ {1} R _ {1} + 8 A C _ {1} ^ {2}} + 2 A \left(C _ {1} R _ {1} + 1 6 A C _ {1} ^ {2}\right) \\ \leq \frac {1}{2} R _ {1} ^ {2} + 6 A C _ {1} R _ {1} + 3 2 A ^ {2} C _ {1} ^ {2} \\ \leq \frac {1}{2} \left(R _ {1} + 8 A C _ {1}\right) ^ {2}. \\ \end{array}
$$
Proof of Theorem 4.1. Note that our choice of  $\eta$  ensures  $\eta \leq \frac{R_1}{16} \frac{1}{4LR_1} \leq \frac{1}{4L}$ . We have that with probability at least  $1 - \delta$ , event  $E(\delta)$  happens. Conditioning on this event, in 4.8 we choose
$$
C _ {1} = \frac {R _ {1}}{2 4 \gamma}; \quad C _ {2} = \frac {\gamma}{2 6 \sigma^ {p}}; \quad C _ {3} = \frac {\gamma}{2 6 T \sigma^ {p}}; \quad A = 3 \gamma .
$$
$$
\lambda_ {t} \eta_ {t} = C _ {1}
$$
$$
\sum_ {t = 1} ^ {T} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \leq \sum_ {t = 1} ^ {T} \left(\frac {\gamma}{2 6 T}\right) \frac {1}{\sigma^ {p}} = C _ {2}
$$
$$
\left(\frac {1}{\lambda_ {t}}\right) ^ {2 p} \leq \frac {1}{\sigma^ {p}} \left(\frac {\gamma}{2 6 T}\right) \left(\frac {1}{\lambda_ {t}}\right) ^ {p} = C _ {3} \left(\frac {1}{\lambda_ {t}}\right) ^ {p}
$$
$$
\max  \left\{\log \frac {1}{\delta} + 2 6 \sigma^ {p} C _ {2} + \frac {2 \sigma^ {2 p} C _ {2} C _ {3}}{A}; 1 \right\} \leq 3 \gamma = A.
$$
We only need to show that for all  $t$
$$
\left\| \nabla f (x _ {t}) \right\| _ {*} \leq \frac {\lambda_ {t}}{2}.
$$
We will show this by induction. Indeed, we have
$$
\left\| \nabla f (x _ {1}) \right\| _ {*} \leq \nabla_ {1} \leq \frac {\lambda_ {1}}{2}.
$$
Suppose that it is true for all  $t \leq k$ . We prove that
$$
\left\| \nabla f (x _ {k + 1}) \right\| _ {*} \leq \frac {\lambda_ {k + 1}}{2}.
$$
By 4.8 we have
$$
\left\| x _ {k + 1} - x ^ {*} \right\| \leq \sqrt {2 \mathbf {D} _ {\psi} \left(x ^ {*} , x _ {k + 1}\right)} \leq R _ {1} + 8 A C _ {1} = 2 R _ {1}.
$$
$$
\begin{array}{l} \left\| \nabla f (x _ {k + 1}) \right\| _ {*} \leq \left\| \nabla f (x _ {k + 1}) - \nabla f (x ^ {*}) \right\| _ {*} + \left\| \nabla f (x _ {1}) - \nabla f (x ^ {*}) \right\| _ {*} + \left\| \nabla f (x _ {1}) \right\| _ {*} \\ \leq L \left\| x _ {k + 1} - x ^ {*} \right\| + L \left\| x _ {1} - x ^ {*} \right\| + \nabla_ {1} \\ \leq 3 L R _ {1} + \nabla_ {1} \leq \frac {\lambda_ {k + 1}}{2} \\ \end{array}
$$
as needed. Therefore from Lemma 4.7 we have
$$
\eta \sum_ {t = 1} ^ {T} \Delta_ {t + 1} + \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {T + 1}\right) \leq 2 R _ {1} ^ {2},
$$
$$
\frac {1}{T} \sum_ {t = 2} ^ {T + 1} \Delta_ {t} \leq \frac {2 R _ {1} ^ {2}}{\eta} = 4 8 R _ {1} \max  \left\{2 6 ^ {\frac {1}{p}} T ^ {\frac {1 - p}{p}} \sigma \gamma^ {\frac {p - 1}{p}}; 2 (3 L R _ {1} + \nabla_ {1}) T ^ {- 1} \gamma \right\}.
$$
Theorem C.2. Assume that  $f$  satisfies Assumption (1), (2), (3), (4) and (5). Let  $\gamma = \max \left\{\log \frac{1}{\delta}; 1\right\}$ ;  $R_{1} = \sqrt{2\mathbf{D}_{\psi}(x^{*}, x_{1})}$  assume that  $\nabla_{1}$  is an upper bound of  $\| \nabla f(x_{1}) \|_{*}$ . For unknown  $T$ , we choose
$$
\begin{array}{l} \lambda_ {t} = \max  \left\{\left(\frac {5 2 t (1 + \log t) ^ {2}}{\gamma}\right) ^ {1 / p} \sigma ; 2 \left(3 L R _ {1} + \nabla_ {1}\right) \right\}, a n d \\ \eta_ {t} = \frac {R _ {1}}{2 4 \lambda_ {t} \gamma} = \frac {R _ {1}}{2 4 \gamma} \min \left\{\left(\frac {5 2 t (1 + \log t) ^ {2}}{\gamma}\right) ^ {- 1 / p} \sigma^ {- 1}; \frac {1}{2} (3 L R _ {1} + \nabla_ {1}) ^ {- 1} \right\}. \\ \end{array}
$$
$$
\frac {1}{T} \sum_ {t = 2} ^ {T + 1} \Delta_ {t} \leq 4 8 R _ {1} \max  \left\{5 2 ^ {\frac {1}{p}} T ^ {\frac {1 - p}{p}} (1 + \log T) ^ {\frac {2}{p}} \sigma \gamma^ {\frac {p - 1}{p}}; 2 (3 L R _ {1} + \nabla_ {1}) T ^ {- 1} \gamma \right\} = \widetilde {O} \left(T ^ {\frac {1 - p}{p}}\right).
$$
Proof. We can follow the similar steps. Notice that  $(\eta_t)$  is a decreasing sequence. We also use Fact B.4 to verify the second condition of Proposition 4.8. The proof is omitted.
Proof of Theorem 4.4. Note that  $\eta_t \leq \frac{1}{4L}$ . We have that with probability at least  $1 - \delta$ , event  $E(\delta)$  happens. Conditioning on this event, in 4.8. We choose
$$
C _ {1} = \frac {c _ {1}}{2 4}; \quad C _ {2} = \frac {1}{2 6 c _ {2}}; \quad C _ {3} = \frac {1}{5 2 c _ {2}}; \quad A = \gamma + \frac {2 \sigma^ {p}}{c _ {2}}.
$$
We verify the conditions of Proposition 4.8
$$
\sum_ {t = 1} ^ {T} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \leq \sum_ {t = 1} ^ {T} \frac {1}{5 2 t (1 + \log t) ^ {2} c _ {2}} \leq \frac {1}{2 6 c _ {2}} = C _ {2}
$$
$$
\left(\frac {1}{\lambda_ {t}}\right) ^ {2 p} \leq \frac {1}{5 2 t c _ {2}} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \leq C _ {3} \left(\frac {1}{\lambda_ {t}}\right) ^ {p}
$$
$$
\max \left\{\log \frac {1}{\delta} + 2 6 \sigma^ {p} C _ {2} + \frac {2 \sigma^ {2 p} C _ {2} C _ {3}}{A}; 1 \right\} = \max \left\{\log \frac {1}{\delta} + \frac {\sigma^ {p}}{c _ {2}} + \frac {\sigma^ {p}}{c _ {2}}; 1 \right\} \leq A,
$$
where we have  $\frac{2\sigma^{2p}C_2C_3}{A}\leq 2\sigma^{2p}C_2C_3\times \frac{c_2}{2\sigma^p}\leq \frac{\sigma^p}{c_2}$ . Also, note that
$$
\begin{array}{l} \left\| \nabla f (x _ {t}) \right\| _ {*} \leq \left\| \nabla f (x _ {t}) - \nabla f (x _ {1}) \right\| _ {*} + \left\| \nabla f (x _ {1}) \right\| _ {*} \\ \leq L \left\| x _ {t} - x _ {1} \right\| _ {*} + \left\| \nabla f (x _ {1}) \right\| _ {*} \leq \frac {\lambda_ {t}}{2}. \\ \end{array}
$$
Therefore, from Lemma 4.7, we have
$$
\begin{array}{l} \eta_ {T} \sum_ {t = 1} ^ {T} \Delta_ {t + 1} + \mathbf {D} _ {\psi} (x ^ {*}, x _ {T + 1}) \leq \frac {1}{2} (R _ {1} + 8 A C _ {1}) ^ {2} \\ = \frac {1}{2} \left(R _ {1} + \frac {c _ {1}}{3} \left(\gamma + \frac {2 \sigma^ {p}}{c _ {2}}\right)\right) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 2} ^ {T + 1} \Delta_ {t} \leq \frac {1}{2 T \eta_ {T}} \left(R _ {1} + \frac {c _ {1}}{3} \left(\gamma + \frac {2 \sigma^ {p}}{c _ {2}}\right)\right) ^ {2} \\ = \frac {8}{T c _ {1}} \left(R _ {1} + \frac {c _ {1}}{3} \left(\gamma + \frac {2 \sigma^ {p}}{c _ {2}}\right)\right) ^ {2} \max  \left\{\left(5 2 T (1 + \log T) ^ {2} c _ {2}\right) ^ {1 / p}; 2 \left(L \max  _ {i \leq T} \| x _ {i} - x _ {1} \| + \nabla_ {1}\right); \frac {L}{8} \right\}. \\ \end{array}
$$
$$
\begin{array}{l} \left\| x _ {i} - x _ {1} \right\| \leq \left\| x _ {i} - x ^ {*} \right\| + \left\| x _ {1} - x ^ {*} \right\| \\ \leq 2 R _ {1} + \frac {c _ {1}}{3} \left(\gamma + \frac {2 \sigma^ {p}}{c _ {2}}\right) \\ \end{array}
$$
which gives us the final convergence rate.
D Clipped Accelerated Stochastic Mirror Descent
In this section, we extend the analysis of Clipped-SMD to the case of Clipped Accelerated Stochastic Mirror Descent (Algorithm 3). We will see that the analysis is basically the same with little modification. We present in Algorithm 3 the clipped version of accelerated stochastic mirror descent (see [15]), where the clipped gradient  $\widehat{\nabla} f(x_{t})$  is used to update the iterates in place of the stochastic gradient  $\widehat{\nabla} f(x_{t})$ .
We use the following additional assumption:
Theorem D.1. Assume that  $f$  satisfies Assumption (1), (2), (3), (4) and (5'). Let  $\gamma = \max \left\{\log \frac{1}{\delta}; 1\right\}$ ; and  $R_1 = \sqrt{2\mathbf{D}_{\psi}(x^*, x_1)}$ .
1. For known  $T$ , we choose a constant  $c$  and  $\lambda_{t}$  and  $\eta_{t}$  such that
$$
c = \max  \left\{1 0 ^ {4}; \frac {4 (T + 1) \left(\frac {2 6 T}{\gamma}\right) ^ {\frac {1}{p}} \sigma}{\gamma L R _ {1}} \right\},
$$
$$
\lambda_ {t} = \frac {c R _ {1} \gamma L \alpha_ {t}}{8} = \max  \left\{\frac {1 0 ^ {4} R _ {1} \gamma L}{6 (t + 1)}; \frac {T + 1}{t + 1} \left(\frac {2 6 T}{\gamma}\right) ^ {1 / p} \sigma \right\},
$$
$$
\eta_ {t} = \frac {1}{3 c \gamma^ {2} L \alpha_ {t}} = \frac {R _ {1}}{2 4 \gamma} \min  \left\{\frac {4 (t + 1)}{1 0 ^ {4} R _ {1} \gamma L}; \frac {t + 1}{T + 1} \left(\frac {2 6 T}{\gamma}\right) ^ {- 1 / p} \sigma^ {- 1} \right\}.
$$
$$
f \left(y _ {T + 1}\right) - f \left(x ^ {*}\right) \leq 6 \max  \left\{1 0 ^ {4} L \gamma^ {2} R _ {1} ^ {2} (T + 1) ^ {- 2}; 4 R _ {1} \left(T + 1\right) ^ {- 1} (2 6 T) ^ {\frac {1}{p}} \gamma^ {\frac {p - 1}{p}} \sigma \right\}.
$$
2. For unknown  $T$ , we choose  $c_{t}$ ,  $\lambda_{t}$  and  $\eta_t$  such that
$$
c _ {t} = \max \left\{1 0 ^ {4}; \frac {4 (t + 1) (\frac {5 2 t (1 + \log t) ^ {2}}{\gamma}) ^ {\frac {1}{p}} \sigma}{\gamma L R _ {1}} \right\},
$$
$$
\lambda_ {t} = \frac {c _ {t} R _ {1} \gamma L \alpha_ {t}}{8} = \max  \left\{\frac {1 0 ^ {4} R _ {1} \gamma L}{4 (t + 1)}; \left(\frac {5 2 t (1 + \log t) ^ {2}}{\gamma}\right) ^ {1 / p} \sigma \right\},
$$
$$
\eta_ {t} = \frac {1}{3 c _ {t} \gamma^ {2} L \alpha_ {t}} = \frac {R _ {1}}{2 4 \gamma} \min  \left\{\frac {4 (t + 1)}{1 0 ^ {4} R _ {1} \gamma L}; \left(\frac {5 2 t (1 + \log t) ^ {2}}{\gamma}\right) ^ {- 1 / p} \sigma^ {- 1} \right\}.
$$
$$
f \left(y _ {T + 1}\right) - f \left(x ^ {*}\right) \leq 6 \max  \left\{1 0 ^ {4} L \gamma^ {2} R _ {1} ^ {2} (T + 1) ^ {- 2}; 4 R _ {1} \left(T + 1\right) ^ {- 1} \left(5 2 T \left(1 + \log T\right) ^ {2}\right) ^ {\frac {1}{p}} \gamma^ {\frac {p - 1}{p}} \sigma \right\}.
$$
Remark D.2. One feature of the accelerated algorithm is the interpolation between the two regimes: When  $\sigma$  is large, the algorithm achieves the  $O\left(T^{\frac{1 - p}{p}}\right)$  convergence rate, which is the same as unaccelerated algorithms; however, when  $\sigma$  is sufficiently small, the algorithm achieves the accelerated  $O\left(T^{-2}\right)$  rate.
We also start the analysis of accelerated stochastic mirror descent with the following lemma.
Lemma D.3. Assume that  $f$  satisfies Assumption (1), (2), (3), (4) and  $\eta_t \leq \frac{1}{2L\alpha_t}$ , the iterate sequence  $(x_{t})_{t \geq 1}$  output by Algorithm 2 satisfies the following
$$
\begin{array}{l} \frac {\eta_ {t}}{\alpha_ {t}} \left(f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right)\right) - \frac {\eta_ {t} \left(1 - \alpha_ {t}\right)}{\alpha_ {t}} \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t}\right) \\ \leq \eta_ {t} \left\langle \theta_ {t} ^ {u}, x ^ {*} - z _ {t} \right\rangle + \eta_ {t} \left\langle \theta_ {t} ^ {b}, x ^ {*} - z _ {t} \right\rangle + 2 \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) + 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} + 2 \eta_ {t} ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]. \\ \end{array}
$$
Proof of Lemma D.3. We have
$$
\begin{array}{l} f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right) = \underbrace {f \left(y _ {t + 1}\right) - f \left(x _ {t}\right)} _ {\text {s m o o t h n e s s}} + \underbrace {f \left(x _ {t}\right) - f \left(x ^ {*}\right)} _ {\text {c o n v e x i t y}} \\ \leq \left\langle \nabla f (x _ {t}), y _ {t + 1} - x _ {t} \right\rangle + \frac {L}{2} \| y _ {t + 1} - x _ {t} \| ^ {2} \\ + \alpha_ {t} \left\langle \nabla f (x _ {t}), x _ {t} - x ^ {*} \right\rangle + \left(1 - \alpha_ {t}\right) \left(f (x _ {t}) - f (x ^ {*})\right) \\ = \underbrace {(1 - \alpha_ {t}) \left\langle \nabla f \left(x _ {t}\right) , y _ {t} - x _ {t} \right\rangle} _ {\text {c o n v e x i t y}} + \alpha_ {t} \left\langle \nabla f \left(x _ {t}\right) , z _ {t + 1} - x ^ {*} \right\rangle \\ + \frac {L \alpha_ {t} ^ {2}}{2} \left\| z _ {t + 1} - z _ {t} \right\| ^ {2} + (1 - \alpha_ {t}) \left(f \left(x _ {t}\right) - f \left(x ^ {*}\right)\right) \\ \leq \left(1 - \alpha_ {t}\right) \left(f \left(y _ {t}\right) - f \left(x _ {t}\right)\right) + \left(1 - \alpha_ {t}\right) \left(f \left(x _ {t}\right) - f \left(x ^ {*}\right)\right) \\ + \alpha_ {t} \left\langle \theta_ {t}, x ^ {*} - z _ {t + 1} \right\rangle + \alpha_ {t} \left\langle \widetilde {\nabla} f (x _ {t}), z _ {t + 1} - x ^ {*} \right\rangle + \frac {L \alpha_ {t} ^ {2}}{2} \| z _ {t + 1} - z _ {t} \| ^ {2} \\ \leq \left(1 - \alpha_ {t}\right) \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \alpha_ {t} \langle \theta_ {t}, x ^ {*} - z _ {t + 1} \rangle \\ + \alpha_ {t} \left\langle \widetilde {\nabla} f (x _ {t}), z _ {t + 1} - x ^ {*} \right\rangle + \frac {L \alpha_ {t} ^ {2}}{2} \| z _ {t + 1} - z _ {t} \| ^ {2}. \\ \end{array}
$$
$$
\left\langle \eta_ {t} \widetilde {\nabla} f (x _ {t}) + \nabla_ {x} \mathbf {D} _ {\psi} \left(z _ {t + 1}, z _ {t}\right), x ^ {*} - z _ {t + 1} \right\rangle \geq 0
$$
$$
\left\langle \eta_ {t} \widetilde {\nabla} f (x _ {t}), z _ {t + 1} - x ^ {*} \right\rangle \leq \left\langle \nabla_ {x} \mathbf {D} _ {\psi} \left(z _ {t + 1}, z _ {t}\right), x ^ {*} - z _ {t + 1} \right\rangle .
$$
$$
\begin{array}{l} \left\langle \nabla_ {x} \mathbf {D} _ {\psi} \left(z _ {t + 1}, z _ {t}\right), x ^ {*} - z _ {t + 1} \right\rangle = \left\langle \nabla \psi \left(z _ {t + 1}\right) - \nabla \psi \left(z _ {t}\right), x ^ {*} - z _ {t + 1} \right\rangle \\ = \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t}\right) - \mathbf {D} _ {\psi} \left(z _ {t + 1}, z _ {t}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right). \\ \end{array}
$$
$$
\begin{array}{l} \eta_ {t} \left\langle \widetilde {\nabla} f (x _ {t}), z _ {t + 1} - x ^ {*} \right\rangle \leq \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(z _ {t + 1}, z _ {t}\right) \\ \leq \mathbf {D} _ {\psi} (x ^ {*}, z _ {t}) - \mathbf {D} _ {\psi} (x ^ {*}, z _ {t + 1}) - \frac {1}{2} \| z _ {t + 1} - z _ {t} \| ^ {2} \\ \end{array}
$$
where we have used that  $\mathbf{D}_{\psi}(z_{t + 1},z_t)\geq \frac{1}{2}\| z_{t + 1} - z_t\|^2$  by the strong convexity of  $\psi$ . We have
$$
\begin{array}{l} f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right) \leq \left(1 - \alpha_ {t}\right) \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \alpha_ {t} \left\langle \theta_ {t}, x ^ {*} - z _ {t + 1} \right\rangle \\ + \frac {\alpha_ {t}}{\eta_ {t}} \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t}\right) - \frac {\alpha_ {t}}{\eta_ {t}} \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right) + \left(\frac {L \alpha_ {t} ^ {2}}{2} - \frac {\alpha_ {t}}{2 \eta_ {t}}\right) \left\| z _ {t + 1} - z _ {t} \right\| ^ {2}. \\ \end{array}
$$
Dividing both sides by  $\frac{\alpha_t}{\eta_t}$  and using the condition  $L\eta_t\alpha_t \leq \frac{1}{2}$ , we have
$$
\begin{array}{l} \frac {\eta_ {t}}{\alpha_ {t}} \left(f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right)\right) + \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t}\right) \\ \leq \frac {\eta_ {t} (1 - \alpha_ {t})}{\alpha_ {t}} \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \eta_ {t} \langle \theta_ {t}, x ^ {*} - z _ {t} \rangle \\ + \eta_ {t} \left\langle \theta_ {t}, z _ {t} - z _ {t + 1} \right\rangle - \frac {1 - L \eta_ {t} \alpha_ {t}}{2} \left\| z _ {t + 1} - z _ {t} \right\| ^ {2} \\ \leq \frac {\eta_ {t} (1 - \alpha_ {t})}{\alpha_ {t}} \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \eta_ {t} \left\langle \theta_ {t}, x ^ {*} - z _ {t} \right\rangle \\ + \frac {\eta_ {t} ^ {2} \| \theta_ {t} \| _ {*} ^ {2}}{2 (1 - L \eta_ {t} \alpha_ {t})} \\ \leq \frac {\eta_ {t} (1 - \alpha_ {t})}{\alpha_ {t}} \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \eta_ {t} \left\langle \theta_ {t} ^ {u} + \theta_ {t} ^ {b}, x ^ {*} - z _ {t} \right\rangle \\ + 2 \eta_ {t} ^ {2} \| \theta_ {t} ^ {u} \| _ {*} ^ {2} + 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} \\ \end{array}
$$
Similarly to the previous section, we define the following variables
$$
\begin{array}{l} Z _ {t} = z _ {t} \left(\frac {\eta_ {t}}{\alpha_ {t}} \left(f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right)\right) - \frac {\eta_ {t} \left(1 - \alpha_ {t}\right)}{\alpha_ {t}} \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right) + \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t}\right) \right. \\ \left. - \eta_ {t} \left\langle \theta_ {t} ^ {b}, x ^ {*} - z _ {t} \right\rangle - 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} - 2 \eta_ {t} ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) - \left(\frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right], \\ \end{array}
$$
where  $z_{t} = \frac{1}{2\eta_{t}\lambda_{t}\max_{i\leq t}\sqrt{2\mathbf{D}_{\psi}(x^{*},x_{i})} + 16Q\eta_{t}^{2}\lambda_{t}^{2}}$
for a constant  $Q \geq 1$ . We also let  $S_{t} = \sum_{i=1}^{t} Z_{i}$ . Following the same analysis as in previous sections, we can obtain Lemma D.4 and Proposition D.5, for which we will omit the proofs here. The only step we need to pay attention to when showing Lemma D.4 is when we bound the sum
$$
\sum_ {t = 1} ^ {k} \frac {z _ {t} \eta_ {t}}{\alpha_ {t}} \left(f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right)\right) - \frac {z _ {t} \eta_ {t} \left(1 - \alpha_ {t}\right)}{\alpha_ {t}} \left(f \left(y _ {t}\right) - f \left(x ^ {*}\right)\right).
$$
If we assume  $\frac{\eta_{t-1}}{\alpha_{t-1}} \geq \frac{\eta_t(1 - \alpha_t)}{\alpha_t}$ , since  $z_t$  is a decreasing sequence and  $\alpha_1 = 0$ , we can lower bound the above sum by the last term  $\frac{z_k \eta_k}{\alpha_k} (f(y_{k+1}) - f(x^*))$ , which gives us the desired inequality.
Lemma D.4. Assume that for all  $t \geq 1$ ,  $\eta_t$  satisfies  $\frac{\eta_{t-1}}{\alpha_{t-1}} \geq \frac{\eta_t(1 - \alpha_t)}{\alpha_t}$ . For any  $\delta > 0$ , let  $E(\delta)$  be the event that for all  $1 \leq k \leq T$
$$
\frac {z _ {k} \eta_ {k}}{\alpha_ {k}} (f (y _ {k + 1}) - f (x ^ {*})) + z _ {k} \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1})
$$
$$
\begin{array}{l} \leq z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}) + \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right\rangle + 2 \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} \\ + \sum_ {t = 1} ^ {k} \left(\left(2 z _ {t} \eta_ {t} ^ {2} + \frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right). \\ \end{array}
$$
Finally, we state a general condition for the choice of  $\eta_t$  and  $\lambda_t$ , which follows exactly the same as in Proposition 4.8. The proof for Theorem D.1 is a direct consequence of this.
Proposition D.5. We assume that the event  $E(\delta)$  from Lemma D.4 happens. Suppose that for some  $\ell \leq T$ , there are constants  $C_1$  and  $C_2$  such that for all  $t \leq \ell$
$$
\frac {\eta_ {t}}{\alpha_ {t}} \left(f \left(y _ {t + 1}\right) - f \left(x ^ {*}\right)\right) + \mathbf {D} _ {\psi} \left(x ^ {*}, z _ {t + 1}\right) \leq \frac {1}{2} \left(R _ {1} + 8 A C _ {1}\right) ^ {2}
$$
for  $A \geq \max \left\{\log \frac{1}{\delta} + 26\sigma^p C_2 + \frac{2\sigma^{2p}C_2C_3}{A}; 1\right\}$ .
Proof of Theorem D.1. 1. Note that  $\eta_t \leq \frac{1}{2c\gamma^2L\alpha_t} \leq \frac{1}{2L\alpha_t}$  and
$$
\frac {\eta_ {t - 1}}{\alpha_ {t - 1}} = \frac {t ^ {2}}{8 c \gamma^ {2} L}
$$
$$
\frac {\eta_ {t} (1 - \alpha_ {t})}{\alpha_ {t}} = \frac {(t + 1) (t - 1)}{8 c \gamma^ {2} L}
$$
thus  $\frac{\eta_{t - 1}}{\alpha_{t - 1}}\geq \frac{\eta_t(1 - \alpha_t)}{\alpha_t}$ . We have that with probability at least  $1 - \delta$ , event  $E(\delta)$  happens. Conditioning on this event, in 4.8 We choose
We can verify the conditions of Proposition D.5 similarly as in previous sections for these choices of  $C_1, C_2$ , and  $C_3$ .
We will show by induction that for all  $t \geq 1$ ,  $\| \nabla f(x_{t}) \|_{*} \leq \frac{\lambda_{t}}{2}$  and  $\max \left\{\| x_{t} - x^{*} \|, \| y_{t} - x^{*} \|, \| z_{t} - x^{*} \| \right\} \leq 2R_{1}$ .
For  $t = 1$ , notice that  $x_{1} = y_{1} = z_{1}$ . Thus, we have
$$
\left\| \nabla f \left(x _ {1}\right) \right\| _ {*} = \left\| \nabla f \left(x _ {1}\right) - \nabla f \left(x ^ {*}\right) \right\| _ {*} \leq L R _ {1} \leq \frac {\lambda_ {1}}{2}.
$$
Now assume that the claim holds for  $1 \leq t \leq k$ . By Proposition D.5, we know that
$$
\frac {2 \eta_ {k}}{\alpha_ {k}} f (y _ {k + 1}) - f (x ^ {*}) + \| z _ {k + 1} - x ^ {*} \| ^ {2} \leq 4 R _ {1} ^ {2}.
$$
$$
\left\| y _ {k + 1} - x ^ {*} \right\| \leq \left(1 - \alpha_ {k}\right) \left\| y _ {k} - x ^ {*} \right\| + \alpha_ {k} \left\| z _ {k + 1} - x ^ {*} \right\| \leq 2 R _ {1}
$$
$$
\left\| x _ {k + 1} - x ^ {*} \right\| \leq \left(1 - \alpha_ {k}\right) \left\| y _ {k + 1} - x ^ {*} \right\| + \alpha_ {k} \left\| z _ {k + 1} - x ^ {*} \right\| \leq 2 R _ {1}
$$
For  $k \geq 1$  we have  $\alpha_{k+1} = \frac{2}{k+2} < 1$ ;  $\frac{\alpha_{k+1}}{1 - \alpha_{k+1}} = \frac{2}{k} \leq \frac{4}{k+2} \leq 2\alpha_{t+1}$  and  $\alpha_t \leq \frac{3}{2}\alpha_{t+1}$ . Hence,
$$
\begin{array}{l} \left\| \nabla f (x _ {k + 1}) \right\| _ {*} \leq \left\| \nabla f (x _ {k + 1}) - \nabla f (y _ {k + 1}) \right\| _ {*} + \left\| \nabla f (y _ {k + 1}) - \nabla f (x ^ {*}) \right\| _ {*} \\ \leq L \left\| x _ {k + 1} - y _ {k + 1} \right\| + \sqrt {2 L \left(f \left(y _ {k + 1}\right) - f \left(x ^ {*}\right)\right)} \\ \leq \frac {L \alpha_ {k + 1} \| x _ {k + 1} - z _ {k + 1} \|}{1 - \alpha_ {k + 1}} + 2 R _ {1} \sqrt {\frac {L \alpha_ {t}}{2 \eta_ {t}}} \\ \end{array}
$$
$$
\begin{array}{l} \leq 4 L R _ {1} \frac {\alpha_ {k + 1}}{1 - \alpha_ {k + 1}} + 2 \sqrt {\frac {3}{2}} c \gamma R _ {1} L \alpha_ {t} \\ \leq 8 \gamma L R _ {1} \alpha_ {t + 1} + 3 \sqrt {\frac {3}{2}} c \gamma L R _ {1} \alpha_ {t + 1} \\ \leq \left(8 + 3 \sqrt {\frac {3}{2} c}\right) R _ {1} \gamma L \alpha_ {t + 1} \\ = \frac {1 6 (8 + 3 \sqrt {\frac {3}{2} c}) \lambda_ {t + 1}}{2 c} \leq \frac {\lambda_ {t + 1}}{2} \\ \end{array}
$$
as needed. Therefore, we have