SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {E} _ {\text {v a r , i n}} ^ {t + 1} = p _ {\text {o u t}} \mathbb {E} \left[ \left\| \frac {1}{B _ {1}} \sum_ {i} \left(\mathbb {E} _ {\tilde {\eta} | i} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t}) ]\right) ^ {\top} \left(\nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {t + 1}\right) - \mathbb {E} _ {\eta | i} [ \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {t + 1}\right) ]\right) \right\| _ {2} ^ {2} \right] \\ + \left(1 - p _ {\text {o u t}}\right) \mathbb {E} \left[ \left\| \frac {1}{B _ {2}} \sum_ {i} \left(G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t}\right) - \mathbb {E} _ {\eta | i} \left[ G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right] \right\| _ {2} ^ {2} \right] \tag {27} \\ \leq \frac {p _ {\text {o u t}}}{B _ {1}} C _ {g} ^ {2} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {t + 1}\right) - \mathbb {E} _ {\eta | i} \left[ \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {t + 1}\right) \right] \right\| _ {2} ^ {2} \right] + \frac {1 - p _ {\text {o u t}}}{B _ {2}} \mathbb {E} _ {i} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \\ \leq \frac {p _ {\text {o u t}}}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \mathcal {E} _ {y} ^ {t + 1} + \frac {1 - p _ {\text {o u t}}}{B _ {2}} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \right]. \\ \end{array}
$$
We bound the outer variance as
$$
\begin{array}{l} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \right] = \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} \pm \left(\mathbb {E} _ {\tilde {\eta} | i} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t - 1}) ]\right) ^ {\top} \nabla f _ {i} (\boldsymbol {y} _ {i} ^ {t + 1}) - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \right] \\ \leq 2 \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} - \left(\mathbb {E} _ {\tilde {\eta} | i} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t - 1}) ]\right) ^ {\top} \nabla f _ {i} (\boldsymbol {y} _ {i} ^ {t + 1}) \right\| _ {2} ^ {2} \right] \right] \right] \\ \left. \right. + 2 \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[\left\|\left(\mathbb {E} _ {\tilde {\eta} | i} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t - 1}) ]\right) ^ {\top} \nabla f _ {i} (\boldsymbol {y} _ {i} ^ {t + 1}) - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right]\right]\right] \\ \leq 2 C _ {f} ^ {2} L _ {g} ^ {2} \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {t - 1} \right\| _ {2} ^ {2} + 2 C _ {g} ^ {2} L _ {f} ^ {2} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} [ \| \boldsymbol {y} _ {i} ^ {t + 1} - \tilde {\boldsymbol {y}} _ {i} ^ {t} \| _ {2} ^ {2} ] ] \right] \right]. \tag {28} \\ \end{array}
$$
For  $t = 0$ , as we only use large and small batch in the
$$
\begin{array}{l} \mathcal {E} _ {\text {v a r , i n}} ^ {1} = \mathbb {E} \left[ \left\| \frac {1}{B _ {1}} \sum_ {i} \left(\mathbb {E} _ {\tilde {\eta} | i} \left[ \nabla g _ {\tilde {\eta}} \left(\boldsymbol {x} ^ {0}\right) \right]\right) ^ {\top} \left(\nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {1}\right) - \mathbb {E} _ {\eta | i} \left[ \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {1}\right) \right]\right) \right\| _ {2} ^ {2} \right] \\ \leq \frac {1}{B _ {1}} C _ {g} ^ {2} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {1}\right) - \mathbb {E} _ {\eta | i} [ \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {1}\right) ] \right\| _ {2} ^ {2} \right] \\ \leq \frac {1}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \mathcal {E} _ {y} ^ {1} \tag {29} \\ \leq \frac {1}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \frac {\sigma_ {g} ^ {2}}{S _ {1}} \\ \leq \frac {4 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
Therefore, average over time  $t = 0,\ldots T - 1$  gives
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r , i n}} ^ {t + 1} \leq \frac {p _ {\text {o u t}}}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 1} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {1 - p _ {\text {o u t}}}{B _ {2}} \frac {1}{T} \sum_ {t = 1} ^ {T - 1} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} [ \| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \| _ {2} ^ {2} ] ] \right] \right] + \frac {\mathcal {E} _ {\text {v a r , i n}} ^ {1}}{T} \\ = \frac {p _ {\mathrm {o u t}}}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {1 - p _ {\mathrm {o u t}}}{B _ {2}} \frac {1}{T} \sum_ {t = 1} ^ {T - 1} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\mathrm {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \right] + \frac {(1 - p _ {\mathrm {o u t}}) \mathcal {E} _ {\mathrm {v a r , i n}} ^ {1}}{T} \\ \leq \frac {p _ {\text {o u t}}}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {(1 - p _ {\text {o u t}}) \mathcal {E} _ {\text {v a r , i n}} ^ {1}}{T} \\ + \frac {2 (1 - p _ {\text {o u t}})}{B _ {2}} \left(\frac {C _ {f} ^ {2} L _ {g} ^ {2}}{T} \sum_ {t = 1} ^ {T - 1} \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {t - 1} \right\| _ {2} ^ {2} + C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 1} ^ {T - 1} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| \boldsymbol {y} _ {i} ^ {t + 1} - \tilde {\boldsymbol {y}} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \right]\right) \\ \leq \frac {p _ {\text {o u t}}}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {(1 - p _ {\text {o u t}}) \mathcal {E} _ {\text {v a r , i n}} ^ {1}}{T} \\ + \frac {2 (1 - p _ {\mathrm {o u t}})}{B _ {2}} \frac {C _ {f} ^ {2} L _ {g} ^ {2} \gamma^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \\ + \frac {2 (1 - p _ {\text {o u t}})}{B _ {2}} C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 1} ^ {T - 1} \mathbb {E} _ {i} \left[ \mathbb {E} _ {p _ {\text {i n}}} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| \boldsymbol {y} _ {i} ^ {t + 1} - \tilde {\boldsymbol {y}} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right] \right]. \\ \end{array}
$$
Let us first apply Lemma 12
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r , i n}} ^ {t + 1} \leq \frac {p _ {\text {o u t}}}{B _ {1}} 4 C _ {g} ^ {2} L _ {f} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {(1 - p _ {\text {o u t}}) \mathcal {E} _ {\text {v a r , i n}} ^ {1}}{T} + \frac {2 (1 - p _ {\text {o u t}})}{B _ {2}} \frac {C _ {f} ^ {2} L _ {g} ^ {2} \gamma^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \\ + \frac {2 \left(1 - p _ {\text {o u t}}\right) C _ {g} ^ {2} L _ {f} ^ {2}}{B _ {2}} \left(\frac {4 C _ {g} ^ {2} \gamma^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| G ^ {t + 1} \right\| _ {2} ^ {2} \right] + \frac {4 (1 - p _ {\text {i n}}) C _ {g} ^ {2}}{S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} + \frac {6 (1 - p _ {\text {i n}})}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1}\right) \\ \leq \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {1 2 C _ {g} ^ {2} L _ {f} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {8 (1 - p _ {\text {o u t}}) L _ {F} ^ {2} \gamma^ {2}}{B _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \\ + \frac {8 (1 - p _ {\mathrm {o u t}}) (1 - p _ {\mathrm {i n}}) C _ {g} ^ {4} L _ {f} ^ {2}}{B _ {2} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} + \frac {(1 - p _ {\mathrm {o u t}}) \mathcal {E} _ {\mathrm {v a r , i n}} ^ {1}}{T} \\ \end{array}
$$
Then we apply Lemma 11 on the bound of  $\frac{1}{T}\sum_{t=0}^{T-1}\mathcal{E}_y^{t+1}$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}, \text {i n}} ^ {t + 1} \leq 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \left(\frac {(1 - p _ {\text {i n}}) L _ {F} ^ {2}}{p _ {\text {i n}} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} + \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}}\right) \\ + \frac {8 \left(1 - p _ {\text {o u t}}\right) L _ {F} ^ {2} \gamma^ {2}}{B _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \\ + \frac {8 (1 - p _ {\mathrm {o u t}}) (1 - p _ {\mathrm {i n}}) C _ {g} ^ {4} L _ {f} ^ {2}}{B _ {2} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} + \frac {(1 - p _ {\mathrm {o u t}}) \mathcal {E} _ {\text {v a r , i n}} ^ {1}}{T} \\ \leq 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}} + \frac {p _ {\text {i n}} (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}}) L _ {F} ^ {2}}{p _ {\text {i n}} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} \\ + \frac {8 \left(1 - p _ {\text {o u t}}\right) L _ {F} ^ {2} \gamma^ {2}}{B _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| G ^ {t + 1} \right\| _ {2} ^ {2} \right] \\ + 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}}) \mathcal {E} _ {\text {v a r , i n}} ^ {1}}{T} \\ \leq 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}}) L _ {F} ^ {2}}{p _ {\text {i n}} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} \\ + \frac {8 \left(1 - p _ {\text {o u t}}\right) L _ {F} ^ {2} \gamma^ {2}}{B _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| G ^ {t + 1} \right\| _ {2} ^ {2} \right] \\ + 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \mathcal {E} _ {\text {v a r , i n}} ^ {1}. \\ \end{array}
$$
From Lemma 10, we plug in the upper bound of  $\frac{1}{T}\sum_{t = 0}^{T - 1}\Xi^t$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}, \text {i n}} ^ {t + 1} \leq 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}}) L _ {F} ^ {2}}{p _ {\text {i n}} S _ {2}} \left(\frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ]\right) \\ + \frac {8 \left(1 - p _ {\text {o u t}}\right) L _ {F} ^ {2} \gamma^ {2}}{B _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \\ + 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \mathcal {E} _ {\text {v a r , i n}} ^ {1} \\ \leq 8 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \\ + 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \mathcal {E} _ {\text {v a r , i n}} ^ {1}. \\ \end{array}
$$
Finally, we add the upper bound on with  $\mathcal{E}_{\mathrm{var,in}}^1$  with (29)
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r , i n}} ^ {t + 1} \leq 8 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ] \tag {30} \\ + 2 4 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \frac {4 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
Outer Variance. Now we consider the outer variance for  $t \geq 1$
$$
\begin{array}{l} \mathcal {E} _ {\text {v a r}, \text {o u t}} ^ {t + 1} \leq \frac {(1 - p _ {\text {o u t}}) ^ {2}}{B _ {2}} \mathbb {E} _ {i} \left[ \left\| \mathbb {E} _ {\eta | i} [ G _ {i} ^ {t + 1} ] - \mathbb {E} _ {\eta | i} [ \tilde {G} _ {i} ^ {t} ] \right\| _ {2} ^ {2} \right] \\ \leq \frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{B _ {2}} \mathbb {E} _ {i} \left[ \mathbb {E} _ {\eta | i} \left[ \left\| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right] \right]. \\ \end{array}
$$
Compared to (27) we know that the upper bound of is smaller than that of  $\mathcal{E}_{\mathrm{var,in}}^{t + 1}$ . Besides, whereas  $\mathcal{E}_{\mathrm{var,out}}^1 = 0$  as we use large batch at  $t = 0$ . Therefore, the upper bound of  $\mathcal{E}_{\mathrm{var}}^{t + 1}$  is upper bounded by  $2^{*}(30)$ .
Variance of  $\nabla g_{\tilde{n}}$ . From Lemma 9, we know that
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \mathcal {E} _ {\nabla g} ^ {t + 1} \right] \leq \frac {1}{B _ {1}} \frac {C _ {f} ^ {2} \sigma_ {g} ^ {2}}{m} + \frac {4 (1 - p _ {\text {o u t}})}{B _ {2} m p _ {\text {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \left(\mathbb {E} \left[ \left\| G _ {i} ^ {t + 1} - \tilde {G} _ {i} ^ {t} \right\| _ {2} ^ {2} \right]\right) \\ \leq \frac {1}{B _ {1}} \frac {C _ {f} ^ {2} \sigma_ {g} ^ {2}}{m} + \frac {1}{m} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1} \\ \end{array}
$$
Finally, we use  $\mathbb{E}[\| G^{t + 1}\| _2^2 ] = \big\| \mathbb{E}[G^{t + 1}]\big\| _2^2 +\mathcal{E}_{\mathrm{var}}^{t + 1}$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \leq 1 6 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \| \mathbb {E} [ G ^ {t + 1} ] \| _ {2} ^ {2} \\ + 1 6 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \\ + 4 8 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
By taking step size  $\gamma$  to satisfy
$$
\gamma^ {2} L _ {F} ^ {2} \max \left\{\frac {p _ {\mathrm {o u t}}}{B _ {1}} \frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}}, \frac {1 - p _ {\mathrm {o u t}}}{B _ {2}} \frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}}, \frac {(1 - p _ {\mathrm {o u t}})}{B _ {2}} \right\} \leq \frac {1}{1 6} \cdot \frac {1}{6}
$$
which can be simplified to
$$
\gamma^ {2} L _ {F} ^ {2} \max \left\{\frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}}, \frac {1 8}{B _ {2}}, \frac {1 - p _ {\mathrm {o u t}}}{B _ {2}} \frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}}, \frac {(1 - p _ {\mathrm {o u t}})}{B _ {2}} \right\} \leq \frac {1}{1 6} \cdot \frac {1}{6}.
$$
Then the coefficient of  $\frac{1}{T}\sum_{t = 0}^{T - 1}\mathcal{E}_{\mathrm{var}}^{t + 1}$  is bounded by  $\frac{1}{2}$
$$
1 6 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \gamma^ {2} L _ {F} ^ {2} \leq \frac {1}{2}.
$$
The the variance has the following bound
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \leq 3 2 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \| \mathbb {E} [ G ^ {t + 1} ] \| _ {2} ^ {2} \\ + 9 6 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
Theorem 10 Consider the (FCCO) problem. Suppose Assumptions 3, 4, 5 hold true. Let step size  $\gamma = \mathcal{O}\left(\frac{1}{\sqrt{n}L_F}\right)$ . Then for NestedVR,  $\pmb{x}^s$  picked uniformly at random among  $\{\pmb{x}^t\}_{t=0}^{T-1}$  satisfies:  $\mathbb{E}[\|\nabla F(\pmb{x}^s)\|_2^2] \leq \varepsilon^2$ , for nonconvex  $F$ , if the hyperparameters of the inner loop  $S_1 = \mathcal{O}(\tilde{L}_F^2\varepsilon^{-2})$ ,  $S_2 = \mathcal{O}(\tilde{L}_F\varepsilon^{-1})$ ,  $p_{in} = \mathcal{O}(1 / S_2)$ , the hyperparameters of the outer loop  $B_1 = n$ ,  $B_2 = \sqrt{n}$ ,  $p_{out} = 1 / B_2$ , and the number of iterations
$$
T = \Omega \left(\frac {\sqrt {n} L _ {F} (F (\pmb {x} ^ {0}) - F ^ {\star})}{\varepsilon^ {2}}\right).
$$
The resulting sample complexity is
$$
\mathcal {O} \left(\frac {n L _ {F} \tilde {L} _ {F} (F (\boldsymbol {x} ^ {0}) - F ^ {\star})}{\varepsilon^ {3}}\right).
$$
In fact, it reaches this sample complexity for all  $\frac{p_{in}p_{out}}{\sqrt{1 - p_{in}}}\lesssim \varepsilon$
Proof: Using descent lemma (Lemma 4) and bias-variance bounds of NestedVR (Lemma 13)
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} + \frac {1}{2 T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ \leq \frac {2 (F (\boldsymbol {x} ^ {0}) - F ^ {\star})}{\gamma T} + \frac {L _ {F} \gamma}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} \\ \leq \underbrace {\frac {2 (F (\boldsymbol {x} ^ {0}) - F ^ {\star})}{\gamma T}} _ {\mathcal {T} _ {0}} + \underbrace {\frac {4 \bar {L} _ {F} ^ {2}}{S _ {1}}} _ {\mathcal {T} _ {1}} + \underbrace {\frac {1 2 (1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {n ^ {2}}{B _ {2} ^ {2}} \frac {L _ {F} ^ {2} \gamma^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2}} _ {\mathcal {T} _ {2}} \\ + \underbrace {\left(\frac {1 2 (1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {n ^ {2}}{B _ {2} ^ {2}} L _ {F} ^ {2} \gamma^ {2} + \frac {2 (1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}} + \gamma L _ {F}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1}} _ {T _ {3}}. \\ \end{array}
$$
Compute  $\mathcal{T}_0$ . In order to let  $\mathcal{T}_0 \leq \varepsilon^2$ , we require that
$$
\gamma T \geq \varepsilon^ {- 2}. \tag {31}
$$
Compute  $\mathcal{T}_1$ . In order to let  $\mathcal{T}_1$  to be smaller than  $\varepsilon^2$ , we need
$$
S _ {1} = \frac {4 \tilde {L} _ {F} ^ {2}}{\varepsilon^ {2}}.
$$
Compute  $\mathcal{T}_2$ . In order to let the coefficient of  $\frac{1}{T}\sum_{t=0}^{T-1}\left\|\mathbb{E}[G^{t+1}]\right\|_2^2$  in  $\mathcal{T}_2$  to be less than  $\frac{1}{4}$ , i.e.
$$
\frac {1 2 \left(1 - p _ {\text {i n}}\right)}{p _ {\text {i n}} S _ {2}} \frac {n ^ {2}}{B _ {2} ^ {2}} L _ {F} ^ {2} \gamma^ {2} \leq \frac {1}{4}, \tag {32}
$$
which requires  $\gamma$
$$
\gamma \leq \frac {B _ {2} \sqrt {p _ {\text {i n}} S _ {2}}}{7 L _ {F} n \sqrt {1 - p _ {\text {i n}}}} = \frac {p _ {\text {o u t}} p _ {\text {i n}} \tilde {L} _ {F}}{7 \varepsilon L _ {F} \sqrt {1 - p _ {\text {i n}}}}. \tag {33}
$$
Compute  $\mathcal{T}_3$ . Let us now focus on  $\mathcal{T}_3$  and notice that the middle term  $\frac{2(1 - p_{\mathrm{out}})^2}{p_{\mathrm{out}}}$
$$
\frac {2 \left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1}.
$$
Using Lemma 13 we have that
$$
\begin{array}{l} \frac {2 \left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \\ \leq \underbrace {3 2 \frac {2 (1 - p _ {\text {o u t}}) ^ {2}}{p _ {\text {o u t}}} \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \gamma^ {2} L _ {F} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \mathbb {E} [ G ^ {t + 1} ] \| _ {2} ^ {2}} _ {\mathcal {T} _ {3, 1}} \\ + \underbrace {9 6 \frac {2 (1 - p _ {\text {o u t}}) ^ {2}}{p _ {\text {o u t}}} \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}}} _ {\mathcal {T} _ {3, 2}} + \underbrace {\frac {2 (1 - p _ {\text {o u t}}) ^ {2}}{p _ {\text {o u t}}} \frac {(1 - p _ {\text {o u t}})}{T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}} _ {\mathcal {T} _ {3, 3}}. \\ \end{array}
$$
- Compute  $\mathcal{T}_{3,3}$ : As we already know that  $S_{1} = \mathcal{O}(\varepsilon^{-2})$  and  $T \geq 1$  and  $B_{1}p_{\mathrm{out}} \geq 1$ . This imposes no more constraints, i.e.
$$
S _ {1} = \mathcal {O} \left(\frac {\tilde {L} _ {F} ^ {2}}{\varepsilon^ {2}}\right).
$$
- Compute  $\mathcal{T}_{3,2}$ : As  $S_{1} = \mathcal{O}(\varepsilon^{-2})$  and  $B_{1} = n$  and  $B_{2} = B_{1}p_{\mathrm{out}}$ , then it requires
$$
\frac {(1 - p _ {\mathrm {i n}}) (1 - p _ {\mathrm {o u t}}) ^ {3}}{p _ {\mathrm {o u t}} ^ {2}} \leq n.
$$
- Compute  $\mathcal{T}_{3,1}$ : In order to satisfy the following
$$
3 2 \frac {2 (1 - p _ {\text {o u t}}) ^ {2}}{p _ {\text {o u t}}} \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \gamma^ {2} L _ {F} ^ {2} \leq \frac {1}{1 2}
$$
we need to enforce
$$
\gamma \leq \frac {p _ {\text {i n}} p _ {\text {o u t}} \tilde {L} _ {F}}{\varepsilon L _ {F} \left(1 - p _ {\text {i n}}\right) ^ {1 / 2} \left(1 - p _ {\text {o u t}}\right) ^ {3 / 2}}. \tag {34}
$$
Now we go back to  $\mathcal{T}_3$  and compare the other two coefficients
$$
\frac {1 2 (1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {n ^ {2}}{B _ {2} ^ {2}} L _ {F} ^ {2} \gamma^ {2} + \frac {2 (1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}} + \gamma L _ {F}.
$$
As  $\gamma L_F \leq \frac{1}{2} \lesssim \frac{2(1 - p_{\mathrm{out}})^2}{p_{\mathrm{out}}}$  we can safely ignore  $\gamma L_F$ . On the other hand, from (32) we know that the first term is also have