SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {B} = \mathbb {E} \left[ \left\{\nabla f _ {N} \left(w ^ {t}\right), \nabla f _ {A} \left(w ^ {t}\right) \right\} \cdot \left\{g _ {N} \left(w ^ {t + 1 / 2}\right), g _ {A} \left(w ^ {t + 1 / 2}\right) \right\} \right] (19) \\ = \mathbb {E} \left[ \nabla f _ {A} \left(w ^ {t}\right) \cdot \left(g _ {A} \left(w ^ {t + 1 / 2}\right) - g _ {A} \left(w ^ {t}\right) + g _ {A} \left(w ^ {t}\right)\right) \right] \\ + \mathbb {E} \left[ \nabla f _ {N} \left(w ^ {t}\right) \cdot \left(g _ {N} \left(w ^ {t + 1 / 2}\right) - g _ {N} \left(w ^ {t}\right) + g _ {N} \left(w ^ {t}\right)\right) \right] (20) \\ = \mathbb {E} \left[ \| \nabla f (w ^ {t}) \| ^ {2} \right] \\ + \underbrace {\mathbb {E} \left[ \nabla f _ {A} \left(w ^ {t}\right) \cdot \left(g _ {A} \left(w ^ {t + 1 / 2}\right) - g _ {A} \left(w ^ {t}\right)\right) \right]} _ {\mathcal {C}} + \underbrace {\mathbb {E} \left[ \nabla f _ {N} \left(w ^ {t}\right) \cdot \left(g _ {N} \left(w ^ {t + 1 / 2}\right) - g _ {N} \left(w ^ {t}\right)\right) \right]} _ {\mathcal {C}}. (21) \\ \end{array}
$$
Using  $xy \leq \frac{1}{2} \| x\| _2^2 +\frac{1}{2}\| y\| _2^2$  and Assumption C.1 we get for  $\mathcal{C}$ :
$$
| \mathcal {C} | \leq \frac {1}{2} \mathbb {E} \left[ \| \nabla f (w ^ {t}) \| ^ {2} \right] + \frac {L ^ {2}}{2} \| w ^ {t + 1 / 2} - w ^ {t} \| ^ {2} = \frac {1}{2} \mathbb {E} \left[ \| \nabla f (w ^ {t}) \| ^ {2} \right] + \frac {L ^ {2} \rho^ {2}}{2}. \tag {22}
$$
Plugging this into (17) gives:
$$
\begin{array}{l} \mathbb {E} [ f (w ^ {t + 1}) ] \leq \mathbb {E} [ f (w ^ {t}) ] - \frac {L h ^ {2}}{2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} + L h ^ {2} M + L ^ {3} h ^ {2} \rho^ {2} - (1 - L h) h \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} \\ + (1 - L h) h \left(\frac {1}{2} \mathbb {E} \| \nabla f \left(w ^ {t}\right) \| ^ {2} + \frac {L ^ {2} \rho^ {2}}{2}\right) (23) \\ \leq \mathbb {E} [ f (w ^ {t}) ] - \frac {h}{2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} + L h ^ {2} M + \frac {1}{2} h L ^ {2} \rho^ {2} (1 + L h). (24) \\ \end{array}
$$
In  $T$  iterations we obtain using a telescoping sum:
$$
\begin{array}{l} f (w ^ {*}) - f (w ^ {0}) \leq \mathbb {E} [ f (w ^ {T}) ] - f (w ^ {0}) \\ \leq - \frac {h}{2} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla f (w ^ {t}) \| ^ {2} \right] + L h ^ {2} M T + \frac {1}{2} h L ^ {2} \rho^ {2} (1 + L h) T. \tag {25} \\ \end{array}
$$
This gives Theorem C.5.