SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + (1 - \gamma \mu) \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
In the view of Lemma 11 with
$$
\begin{array}{l} \Psi^ {t} = \frac {2 (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {4 ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \quad \frac {2}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(\frac {4 0 b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right], \\ \end{array}
$$
we can conclude the proof of the theorem.
F.4.4 Proof for DASHA-PP-PAGE under PL-condition
Theorem 9. Suppose that Assumption 1, 2, 3, 7, 4, 8, and 9 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ , probability
$$
p _ {p a g e} = \frac {B}{m + B}, b = \frac {p _ {p a g e} p _ {\mathrm {a}}}{2 - p _ {\mathrm {a}}},
$$
$$
\gamma \leq \min  \left\{\left(L + \sqrt {\frac {2 0 0 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\max} ^ {2}}{B}\right) + \frac {4 8}{n p _ {\mathrm {a}} ^ {2} p _ {p a g e}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\max} ^ {2}}{B}\right)\right) ^ {- 1}, \frac {a}{2 \mu}, \frac {b}{2 \mu} \right\},
$$
Proof. Let us fix constants  $\nu, \rho \in [0, \infty)$  that we will define later. Considering Lemma 12, Lemma 8, and the law of total expectation, we obtain
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \nu \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {1 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(2 \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max} ^ {2}}{B}\right) \right. \\ \left. - \nu \left(\frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\max} ^ {2}}{n p _ {\mathrm {a}} B}\right) - \rho \left(\frac {2 \left(1 - p _ {\mathrm {a}}\right) \widehat {L} ^ {2}}{p _ {\mathrm {a}}} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\max} ^ {2}}{p _ {\mathrm {a}} B}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ \left. + \left(\gamma + \nu \left(p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + (1 - p _ {\text {p a g e}})\right)\right) E \left[ \left\| h ^ {t} - \nabla f \left(x ^ {t}\right) \right\| ^ {2} \right] \right. \\ + \left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {2 \nu \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b ^ {2}}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} \right. \\ \left. + \rho \left(\frac {2 \left(1 - p _ {\mathrm {a}}\right) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right| \right| ^ {2} \right]. \\ \end{array}
$$
Due to  $b = \frac{p_{\mathrm{page}} p_{\mathrm{a}}}{2 - p_{\mathrm{a}}} \leq p_{\mathrm{page}}$ , one can show that  $\left(p_{\mathrm{page}} \left(1 - \frac{b}{p_{\mathrm{page}}}\right)^2 + (1 - p_{\mathrm{page}})\right) \leq 1 - b$ . Thus, if we take  $\nu = \frac{2\gamma}{b}$ , then
$$
\left. \right.\left(\gamma + \nu \left(p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + (1 - p _ {\text {p a g e}})\right)\right) \leq \gamma + \nu (1 - b) = \left(1 - \frac {b}{2}\right) \nu ,
$$
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {1 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(2 \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max} ^ {2}}{B}\right) \right. \\ \left. - \frac {2 \gamma}{b n p _ {\mathrm {a}}} \left(2 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{B}\right) - \rho \left(\frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{p _ {\mathrm {a}} B}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} + \frac {4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) b}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \\ \left. + \rho \left(\frac {2 \left(1 - p _ {\mathrm {a}}\right) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\left(\frac {2 \left(1 - p _ {\mathrm {a}}\right) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right) \leq 1 - b.
$$
If we take  $\rho = \frac{40b\gamma\omega(2\omega + 1)}{np_{\mathrm{a}}^2p_{\mathrm{page}}} +\frac{8\gamma(p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2p_{\mathrm{page}}}$ , then
$$
\left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {4 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \rho \left(\frac {2 \left(1 - p _ {\mathrm {a}}\right) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right)\right) \leq \left(1 - \frac {b}{2}\right) \rho ,
$$
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {1 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(2 \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max} ^ {2}}{B}\right) \right. \\ - \frac {2 \gamma}{b n p _ {\mathrm {a}}} \left(2 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{B}\right) \\ \left. - \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {3} p _ {\text {p a g e}}} + \frac {8 \gamma \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \left(2 \left(1 - p _ {\mathrm {a}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{B}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right) \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\frac {2 \gamma}{b n p _ {\mathrm {a}}} \left(2 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\mathrm {m a x}} ^ {2}}{B}\right) \leq \frac {8 \gamma}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\mathrm {m a x}} ^ {2}}{B}\right).
$$
$$
\begin{array}{l} \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {3} p _ {\text {p a g e}}} + \frac {8 \gamma \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \left(2 \left(1 - p _ {\mathrm {a}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{B}\right) \\ \leq \left(\frac {4 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}}\right) \left(2 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\max } ^ {2}}{B}\right) \\ \leq \frac {8 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\mathrm {m a x}} ^ {2}}{B}\right) \\ + \frac {1 6 \gamma \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\max } ^ {2}}{B}\right) \\ \leq \frac {8 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{B}\right) \\ + \frac {1 6 \gamma}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max} ^ {2}}{B}\right). \\ \end{array}
$$
Combining all bounds together, we obtain the following inequality:
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {1 0 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max} ^ {2}}{B}\right) \right. \\ \left. - \frac {2 4 \gamma}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{B}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right) \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \left| \left| \nabla f (x ^ {t}) \right| \right| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right) \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
Note that  $\gamma \leq \frac{b}{2\mu}$ , thus  $1 - \frac{b}{2} \leq 1 - \gamma \mu$  and
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + (1 - \gamma \mu) \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
It is left to apply Lemma 11 with
$$
\begin{array}{l} \Psi^ {t} = \frac {2 (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {4 ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {2}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(\frac {4 0 b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} + \frac {8 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ \end{array}
$$
F.4.5 Proof for DASHA-PP-MVR under PL-condition
Theorem 10. Suppose that Assumption 1, 2, 3, 7, 5, 6, 8 and 9 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,
$$
b \in \left(0, \frac {p _ {\mathrm {a}}}{2 - p _ {\mathrm {a}}} \right],
$$
$$
\gamma \leq \min \left\{\left(L + \sqrt {\frac {2 0 0 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) + \frac {4 0}{n p _ {\mathrm {a}} b} \left(\frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B} + \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2}\right)}\right) ^ {- 1}, \frac {a}{2 \mu}, \frac {b}{2 \mu} \right\},
$$
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {T}) - f ^ {*} \right] \\ \leq \left(1 - \gamma \mu\right) ^ {T} \left(\Delta_ {0} + \frac {2 \gamma}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \left(\frac {4 0 \gamma b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) \\ + \frac {1}{\mu} \left(\frac {1 0 0 b ^ {2} \omega (2 \omega + 1)}{p _ {\mathrm {a}} ^ {2}} + \frac {2 0 b}{p _ {\mathrm {a}}}\right) \frac {\sigma^ {2}}{n B}. \\ \end{array}
$$