SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\langle \theta_ {t} - \theta_ {*}, \nabla f \left(\theta_ {t}\right) \right\rangle \geq \frac {1}{L} \| \nabla f \left(\theta_ {t}\right) \| ^ {2} \tag {9}
$$
$$
\left\langle \theta_ {t} - \theta_ {*}, \nabla f \left(\theta_ {t}\right) \right\rangle \geq f \left(\theta_ {t}\right) - f _ {*} + \frac {1}{2 L} \| \nabla f \left(\theta_ {t}\right) \| ^ {2} \tag {10}
$$
$$
\left\langle \theta_ {t} - \theta_ {t - 1}, \nabla f \left(\theta_ {t}\right) \right\rangle \geq f \left(\theta_ {t}\right) - f \left(\theta_ {t - 1}\right) \tag {11}
$$
Substitute them and get
$$
\begin{array}{l} \mathbb {E} \left[ \| z _ {t + 1} - \theta_ {*} \| ^ {2} \right] \leq \| z _ {t} - \theta_ {*} \| ^ {2} - \frac {2 \alpha (1 - \rho)}{L (1 - \beta)} \| \nabla f (\theta_ {t}) \| ^ {2} - \frac {2 \alpha \rho}{1 - \beta} (f (\theta_ {t}) - f _ {*}) \\ - \frac {\alpha \rho}{L (1 - \beta)} \| \nabla f (\theta_ {t}) \| ^ {2} - \frac {2 \alpha \beta}{(1 - \beta) ^ {2}} \left(f (\theta_ {t}) - f \left(\theta_ {t - 1}\right)\right) + \mathbb {E} [ \| z _ {t + 1} - z _ {t} \| ^ {2} ] \\ \end{array}
$$
where  $\rho \in (0,1]$  is a parameter used to balance the first two inequalities. Denote  $M = 2\left(\frac{\alpha}{1 - \beta}\right)^{2}\left(\sigma^{2} + \sigma_{m}^{2}\right)$ . Substitute Eq. 8 into this inequality and collect the terms, we get
$$
\begin{array}{l} \left(\frac {2 \alpha \rho}{1 - \beta} + \frac {2 \alpha \beta}{(1 - \beta) ^ {2}}\right) \left(f \left(\theta_ {t}\right) - f _ {*}\right) + \mathbb {E} [ \| z _ {t + 1} - \theta_ {*} \| ^ {2} ] \\ \leq \frac {2 \alpha \beta}{(1 - \beta) ^ {2}} \left(f \left(\theta_ {t - 1}\right) - f _ {*}\right) + \left\| z _ {t} - \theta_ {*} \right\| ^ {2} + \left(\frac {2 \alpha^ {2}}{(1 - \beta) ^ {2}} - \frac {\alpha (2 - \rho)}{L (1 - \beta)}\right) \| \nabla f (\theta_ {t}) \| ^ {2} + M \\ \end{array}
$$
When  $\alpha$  satisfies the condition  $\frac{2\alpha^2}{(1 - \beta)^2} - \frac{\alpha(2 - \rho)}{L(1 - \beta)} \leq 0$ , i.e.  $0 \leq \alpha \leq \frac{(1 - \beta)(2 - \rho)}{2L}$ , the term about  $\|\nabla f(\theta_t)\|^2$  is non-positive, thus we have
$$
\begin{array}{l} \left(\frac {2 \alpha \rho}{1 - \beta} + \frac {2 \alpha \beta}{(1 - \beta) ^ {2}}\right) \left(f \left(\theta_ {t}\right) - f _ {*}\right) + \mathbb {E} [ \| z _ {t + 1} - \theta_ {*} \| ^ {2} ] \\ \leq \frac {2 \alpha \beta}{(1 - \beta) ^ {2}} \left(f \left(\theta_ {t - 1}\right) - f _ {*}\right) + \left\| z _ {t} - \theta_ {*} \right\| ^ {2} + M \\ \end{array}
$$
Summing this inequality from 0 to  $T - 1$  and taking full expectation gives
$$
\begin{array}{l} \frac {2 \alpha \rho}{1 - \beta} \sum_ {i = 0} ^ {T - 1} \mathbb {E} [ f (\theta_ {i}) - f _ {*} ] + \sum_ {i = 0} ^ {T - 1} \left(\frac {2 \alpha \beta}{(1 - \beta) ^ {2}} \mathbb {E} [ f (\theta_ {i}) - f _ {*} ] + \mathbb {E} [ \| z _ {i + 1} - \theta_ {*} \| ^ {2} ]\right) \\ \leq \sum_ {i = 0} ^ {T - 1} \left(\frac {2 \alpha \beta}{(1 - \beta) ^ {2}} \mathbb {E} \left[ f \left(\theta_ {i - 1}\right) - f _ {*} \right] + \mathbb {E} \left[ \| z _ {i} - \theta_ {*} \| ^ {2} \right]\right) + T \cdot M \\ \end{array}
$$
$$
\frac {2 \alpha \rho}{1 - \beta} \sum_ {i = 0} ^ {T - 1} \mathbb {E} [ f (\theta_ {i}) - f _ {*} ] \leq \frac {2 \alpha \beta}{(1 - \beta) ^ {2}} (f (\theta_ {0}) - f _ {*}) + \| \theta_ {0} - \theta_ {*} \| ^ {2} + T \cdot M
$$
Since  $f$  is convex, we have  $Tf(\bar{\theta}_T) \leq \frac{1}{T} \sum_{i=0}^{T-1} f(\theta_i)$ . Subsequently we have
$$
\begin{array}{l} \mathbb {E} \left[ f \left(\bar {\theta} _ {T}\right) - f _ {*} \right] \leq \frac {1}{T} \left(\frac {\beta}{\rho (1 - \beta)} \left(f \left(\theta_ {0}\right) - f _ {*}\right) + \frac {1 - \beta}{2 \alpha \rho} \left\| \theta_ {0} - \theta_ {*} \right\| ^ {2}\right) \\ + \frac {1 - \beta}{2 \alpha \rho} M \\ \end{array}
$$
Finally, when  $\alpha \in (0,\frac{1 - \beta}{L} ]$ , we can take  $\rho = 1$ , use L-smooth condition again and substitute  $M$  which gives
$$
\begin{array}{l} \mathbb {E} \left[ f \left(\bar {\theta} _ {T}\right) - f _ {*} \right] \leq \frac {1}{2 T} \left(\frac {L \beta}{1 - \beta} + \frac {1 - \beta}{\alpha}\right) \left\| \theta_ {0} - \theta_ {*} \right\| ^ {2} \\ + \frac {\alpha \sigma^ {2}}{(1 - \beta)} + \frac {\alpha \sigma_ {m} ^ {2}}{(1 - \beta)} \\ \end{array}
$$