SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, we prove the bound for  $\mathrm{E}_B\left[\mathrm{E}_{p_{\mathrm{a}}}\left[\left\| h_2^{t + 1} - \mathrm{E}_B\left[\mathrm{E}_{p_{\mathrm{a}}}\left[h_2^{t + 1}\right]\right]\right\| ^2\right]$ . Considering that mini-batches in the algorithm are independent, we can use Lemma 1 with  $r_i = h_i^t$  and  $s_i = k_{i,2}^{t + 1}$  to obtain
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[\left\| h _ {2} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {2} ^ {t + 1} \right]\right]\right\| ^ {2} \right]\right] \\ \leq \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {B} \left[ \left\| k _ {i, 2} ^ {t + 1} - \mathrm {E} _ {B} \left[ k _ {i, 2} ^ {t + 1} \right] \right\| ^ {2} \right] + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \mathrm {E} _ {B} \left[ k _ {i, 2} ^ {t + 1} \right] \right\| ^ {2} \\ = \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {B} \left[ \left\| \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ = \frac {1}{n ^ {2} p _ {\mathrm {a}} B ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {B} \left[ \sum_ {j \in I _ {i} ^ {t}} \left\| \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ = \frac {1}{n ^ {2} p _ {\mathrm {a}} B m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \left\| \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \\ + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ \leq \frac {1}{n ^ {2} p _ {\mathrm {a}} B m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \left\| \nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right) \right\| ^ {2} + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2}. \\ \end{array}
$$
Next, we use Assumptions 3 and 4 to get
$$
\left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[\left\| h _ {2} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {2} ^ {t + 1} \right]\right]\right\| ^ {2} \right]\right] \leq \left(\frac {L _ {\max } ^ {2}}{n p _ {\mathrm {a}} B} + \frac {(p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2}. \tag {26}
$$
Applying (25) and (26) into (24), we get
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \mathrm {E} _ {p _ {\mathrm {p a g e}}} \left[\left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right]\right]\right] \\ \leq p _ {\text {p a g e}} \left(\frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b ^ {2}}{n ^ {2} p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2}\right) + \\ + \left(1 - p _ {\text {p a g e}}\right) \left(\frac {L _ {\text {m a x}} ^ {2}}{n p _ {\mathrm {a}} B} + \frac {\left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \\ \left. + \left(p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + \left(1 - p _ {\text {p a g e}}\right)\right) \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right. \\ \leq \left(\frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{n p _ {\mathrm {a}} B}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} \\ + \frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b ^ {2}}{n ^ {2} p _ {\mathrm {a}} ^ {2} p _ {\text {p a g e}}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} + \left(p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + \left(1 - p _ {\text {p a g e}}\right)\right) \left\| h ^ {t} - \nabla f \left(x ^ {t}\right) \right\| ^ {2}. \\ \end{array}
$$
The proof of the second inequality almost repeats the previous one:
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \right. \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \right. \mathrm {E} _ {p _ {\text {p a g e}}} \left[ \right.\left| \right.\left| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right)\right| ^ {2} \left. \right]\left. \right]\left. \right]\left. \right] \\ = p _ {\text {p a g e}} \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 1} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] \right] + (1 - p _ {\text {p a g e}}) \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 2} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] \right] \\ \stackrel {(1 6)} {=} p _ {\text {p a g e}} \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 1} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 1} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] + (1 - p _ {\text {p a g e}}) \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 2} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 2} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] \\ + p _ {\text {p a g e}} \left\| \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 1} ^ {t + 1} \right] \right] - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} + \left(1 - p _ {\text {p a g e}}\right) \left\| \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 2} ^ {t + 1} \right] \right] - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \\ = p _ {\text {p a g e}} \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 1} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 1} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] + (1 - p _ {\text {p a g e}}) \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 2} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 2} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] \\ + \left(p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + (1 - p _ {\text {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2}. \tag {27} \\ \end{array}
$$
Let us consider  $\mathrm{E}_B\left[\mathrm{E}_{p_{\mathrm{a}}}\left[\left\| h_{i,1}^{t + 1} - \mathrm{E}_B\left[\mathrm{E}_{p_{\mathrm{a}}}\left[h_{i,1}^{t + 1}\right]\right]\right\| ^2\right]\right]$
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {\mathrm {p} _ {\mathrm {a}}} \left[\left\| h _ {i, 1} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {\mathrm {p} _ {\mathrm {a}}} \left[ h _ {i, 1} ^ {t + 1} \right]\right]\right\| ^ {2} \right]\right] \\ = \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 1} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 1} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \\ = p _ {\mathrm {a}} \left\| h _ {i} ^ {t} + \frac {1}{p _ {\mathrm {a}}} k _ {i, 1} ^ {t + 1} - \left(h _ {i} ^ {t} + \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right)\right) \right\| ^ {2} \\ \left. \right. + \left(1 - p _ {\mathrm {a}}\right)\left\| h _ {i} ^ {t} - \left(h _ {i} ^ {t} + \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right)\right)\right\| ^ {2} \\ = \frac {\left(1 - p _ {\mathrm {a}}\right) ^ {2}}{p _ {\mathrm {a}}} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \\ + \left(1 - p _ {\mathrm {a}}\right) \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \\ = \frac {1 - p _ {\mathrm {a}}}{p _ {\mathrm {a}}} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2}. \\ \end{array}
$$
Considering (15) and Assumption 3, we obtain
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[\left\| h _ {i, 1} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 1} ^ {t + 1} \right]\right]\right\| ^ {2} \right]\right] \\ \leq \frac {2 (1 - p _ {\mathrm {a}}) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\text {p a g e}} ^ {2}} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2}. \tag {28} \\ \end{array}
$$
Next, we obtain the bound for  $\operatorname{E}_B\left[\operatorname{E}_{p_{\mathrm{a}}}\left[\left\|h_{i,2}^{t+1}-\operatorname{E}_B\left[\operatorname{E}_{p_{\mathrm{a}}}\left[h_{i,2}^{t+1}\right]\right]\right\|^2\right]\right]$ :
$$
\begin{array}{l} \left. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i, 2} ^ {t + 1} - \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i, 2} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] \right] \\ = p _ {\mathrm {a}} \mathrm {E} _ {B} \left[ \left\| h _ {i} ^ {t} + \frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \left(h _ {i} ^ {t} + \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + \left(1 - p _ {\mathrm {a}}\right) \mathbb {E} _ {B} \left[ \left| \left| h _ {i} ^ {t} - \left(h _ {i} ^ {t} + \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})\right) \right| \right| ^ {2} \right] \\ = p _ {\mathrm {a}} \mathrm {E} _ {B} \left[ \left\| \frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \left(\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})\right) \right\| ^ {2} \right] \\ + (1 - p _ {a}) \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ \stackrel {(1 6)} {=} \frac {1}{p _ {\mathrm {a}}} \mathrm {E} _ {B} \left[ \left\| k _ {i, 2} ^ {t + 1} - \left(\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})\right) \right\| ^ {2} \right] + \frac {(1 - p _ {\mathrm {a}}) ^ {2}}{p _ {\mathrm {a}}} \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ + \left(1 - p _ {\mathrm {a}}\right) \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ = \frac {1}{p _ {\mathrm {a}}} \mathrm {E} _ {B} \left[ \left\| k _ {i, 2} ^ {t + 1} - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] + \frac {1 - p _ {\mathrm {a}}}{p _ {\mathrm {a}}} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ \leq \frac {1}{p _ {\mathrm {a}}} \mathrm {E} _ {B} \left[ \left\| k _ {i, 2} ^ {t + 1} - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] + \frac {\left(1 - p _ {\mathrm {a}}\right) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2}, \tag {29} \\ \end{array}
$$
where we used Assumption 3. By plugging (28) and (29) into (27), we get
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \mathrm {E} _ {p _ {\mathrm {p a g e}}} \left[\left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right]\right]\right] \\ \leq p _ {\text {p a g e}} \left(\frac {2 (1 - p _ {\mathrm {a}}) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\text {p a g e}} ^ {2}} \| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \| ^ {2}\right) \\ \left. + \left(1 - p _ {\text {p a g e}}\right) \left(\frac {1}{p _ {\mathrm {a}}} \mathrm {E} _ {B} \left[ \left\| k _ {i, 2} ^ {t + 1} - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] + \frac {\left(1 - p _ {\mathrm {a}}\right) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2}\right) \right. \\ + \left(p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ \leq \frac {2 (1 - p _ {\mathrm {a}}) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {1 - p _ {\mathrm {p a g e}}}{p _ {\mathrm {a}}} \mathrm {E} _ {B} \left[ \left\| k _ {i, 2} ^ {t + 1} - (\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})) \right\| ^ {2} \right] \\ + \left(\frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
From the independence of elements in the mini-batch, we obtain
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \mathrm {E} _ {p _ {\text {p a g e}}} \left[\left\| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right)\right\| ^ {2} \right]\right]\right] \\ \leq \frac {2 (1 - p _ {\mathrm {a}}) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {1 - p _ {\mathrm {p a g e}}}{p _ {\mathrm {a}}} \mathrm {E} _ {B} \left[ \left\| \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} (\nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t})) - (\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})) \right\| ^ {2} \right] \\ + \left(\frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} = \frac {2 \left(1 - p _ {\mathrm {a}}\right) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {1 - p _ {\mathrm {p a g e}}}{p _ {\mathrm {a}} B ^ {2}} \mathrm {E} _ {B} \left[ \sum_ {j \in I _ {i} ^ {t}} \left\| \left(\nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t})\right) - \left(\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})\right) \right\| ^ {2} \right] \\ + \left(\frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ = \frac {2 \left(1 - p _ {\mathrm {a}}\right) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {1 - p _ {\text {p a g e}}}{m p _ {\mathrm {a}} B} \sum_ {j = 1} ^ {m} \left\| \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \\ \left. + \left(\frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\text {p a g e}}} + p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + (1 - p _ {\text {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right. \\ \leq \frac {2 (1 - p _ {\mathrm {a}}) L _ {i} ^ {2}}{p _ {\mathrm {a}}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {1 - p _ {\text {p a g e}}}{m p _ {\mathrm {a}} B} \sum_ {j = 1} ^ {m} \left\| \nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right) \right\| ^ {2} \\ \left. + \left(\frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\text {p a g e}}} + p _ {\text {p a g e}} \left(1 - \frac {b}{p _ {\text {p a g e}}}\right) ^ {2} + (1 - p _ {\text {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right. \\ \leq \left(\frac {2 (1 - p _ {\mathrm {a}}) L _ {i} ^ {2}}{p _ {\mathrm {a}}} + \frac {(1 - p _ {\text {p a g e}}) L _ {\max } ^ {2}}{p _ {\mathrm {a}} B}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} \\ + \left(\frac {2 (1 - p _ {\mathrm {a}}) b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {p a g e}}} + p _ {\mathrm {p a g e}} \left(1 - \frac {b}{p _ {\mathrm {p a g e}}}\right) ^ {2} + (1 - p _ {\mathrm {p a g e}})\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2}, \\ \end{array}
$$
where we used Assumption 4. Finally, we prove the last inequality:
$$
\begin{array}{l} \mathrm {E} _ {B} \left[ \right. \mathrm {E} _ {p _ {\text {p a g e}}} \left[ \right.\left| \right.\left| k _ {i} ^ {t + 1} \right| ^ {2} \left. \right]\left. \right] \\ = p _ {\text {p a g e}} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \\ + \left(1 - p _ {\text {p a g e}}\right) \mathrm {E} _ {B} \left[ \left\| \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ \stackrel {(1 6)} {=} p _ {\text {p a g e}} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - \frac {b}{p _ {\text {p a g e}}} \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \\ + \left(1 - p _ {\text {p a g e}}\right) \mathrm {E} _ {B} \left[ \left\| \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + \left(1 - p _ {\text {p a g e}}\right) \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ \stackrel {(1 5)} {\leq} 2 p _ {\text {p a g e}} \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} + \frac {2 b ^ {2}}{p _ {\text {p a g e}}} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ + \left(1 - p _ {\text {p a g e}}\right) \mathrm {E} _ {B} \left[ \left\| \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + \left(1 - p _ {\text {p a g e}}\right) \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ \leq 2 \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} + \frac {2 b ^ {2}}{p _ {\text {p a g e}}} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \\ \end{array}
$$
$$
+ (1 - p _ {\mathrm {p a g e}}) \mathrm {E} _ {B} \left[ \left\| \frac {1}{B} \sum_ {j \in I _ {i} ^ {t}} \left(\nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t})\right) - \left(\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})\right) \right\| ^ {2} \right].
$$
Using the independence of elements in the mini-batch, we have
$$
\begin{array}{l} \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {p a g e}}} \left[ \left| \left| k _ {i} ^ {t + 1} \right| \right| ^ {2} \right] \right] \\ \leq 2 \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + \frac {2 b ^ {2}}{p _ {\mathrm {p a g e}}} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ + \frac {1 - p _ {\text {p a g e}}}{B ^ {2}} \mathrm {E} _ {B} \left[ \sum_ {j \in I _ {i} ^ {t}} \left\| \left(\nabla f _ {i j} \left(x ^ {t + 1}\right) - \nabla f _ {i j} \left(x ^ {t}\right)\right) - \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ = 2 \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + \frac {2 b ^ {2}}{p _ {\mathrm {p a g e}}} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ + \frac {1 - p _ {\mathrm {p a g e}}}{B m} \sum_ {j = 1} ^ {m} \left\| \left(\nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t})\right) - \left(\nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t})\right) \right\| ^ {2} \\ \leq 2 \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + \frac {2 b ^ {2}}{p _ {\mathrm {p a g e}}} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ + \frac {1 - p _ {\text {p a g e}}}{B m} \sum_ {j = 1} ^ {m} \left\| \nabla f _ {i j} (x ^ {t + 1}) - \nabla f _ {i j} (x ^ {t}) \right\| ^ {2} \\ \end{array}
$$
It it left to consider Assumptions 3 and 4 to get
$$
\begin{array}{l} \mathrm {E} _ {B} \left[ \mathrm {E} _ {p _ {\mathrm {p a g e}}} \left[ \left| \left| k _ {i} ^ {t + 1} \right| \right| ^ {2} \right] \right] \\ \leq \left(2 L _ {i} ^ {2} + \frac {(1 - p _ {\mathrm {p a g e}}) L _ {\mathrm {m a x}} ^ {2}}{B}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {2 b ^ {2}}{p _ {\mathrm {p a g e}}} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
Theorem 3. Suppose that Assumptions 1, 2, 3, 4, 7, and 8 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{page} p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$ , probability  $p_{page} \in (0, 1]$
$$
\gamma \leq \left(L + \left[ \frac {4 8 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\mathrm {m a x}} ^ {2}}{B}\right) + \frac {1 6}{n p _ {\mathrm {a}} ^ {2} p _ {p a g e}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\mathrm {m a x}} ^ {2}}{B}\right) \right] ^ {1 / 2}\right) ^ {- 1}
$$
and  $g_{i}^{0} = h_{i}^{0} = \nabla f_{i}(x^{0})$  for all  $i\in [n]$  in Algorithm 1 (DASHA-PP-PAGE) then  $\operatorname{E}\left[\left\| \nabla f(\widehat{x}^T)\right\|^2\right]\leq \frac{2\Delta_0}{\gamma T}$ .
Proof. Let us fix constants  $\nu, \rho \in [0, \infty)$  that we will define later. Considering Lemma 6, Lemma 8, and the law of total expectation, we obtain