SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1 2 (1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {n ^ {2}}{B _ {2} ^ {2}} L _ {F} ^ {2} \gamma^ {2} \leq \frac {1}{4} \lesssim \frac {2 (1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}}.
$$
Constraints from the Bias-Variance Lemma (Lemma 13). By setting  $B_{1} = n$  and  $S_{1} = \mathcal{O}\left(\frac{\tilde{L}_{F}^{2}}{\varepsilon^{2}}\right)$ , this constraint translates to
$$
\gamma^ {2} L _ {F} ^ {2} \max \left\{\frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} ^ {2}} \frac {\varepsilon^ {2}}{B _ {2}}, \frac {1 - p _ {\mathrm {o u t}}}{B _ {2}} \frac {(1 - p _ {\mathrm {i n}}) \varepsilon^ {2}}{p _ {\mathrm {i n}} ^ {2}} \frac {1}{p _ {\mathrm {o u t}} ^ {2}}, \frac {(1 - p _ {\mathrm {o u t}})}{B _ {2}} \right\} \lesssim 1
$$
which is weaker than (33).
Summary on the Limit on  $\gamma$ . Combine (33) and (34) and  $\gamma \leq \frac{1}{2L_F}$ , we have a final limit on step size  $\gamma$
$$
\gamma \lesssim \min  \left\{\frac {p _ {\text {o u t}} p _ {\text {i n}} \tilde {L} _ {F}}{\varepsilon L _ {F} \sqrt {1 - p _ {\text {i n}}}}, \frac {1}{L _ {F}} \right\} \tag {35}
$$
Then the total sample complexity of NestedVR can be computed as
(# of iter  $T$ ) × (Avg. outer batch size  $B_{2} = B_{1}p_{\mathrm{out}}$ ) × (Avg. inner batch size  $S_{2} = S_{1}p_{\mathrm{in}}$ ).
This sample complexity has the following requirement
$$
B _ {2} S _ {2} T = \frac {B _ {2} S _ {2} (T \gamma)}{\gamma} \stackrel {(3 1)} {\geq} \frac {B _ {2} S _ {2}}{\varepsilon^ {2} \gamma} = \frac {n \varepsilon^ {- 2}}{\varepsilon^ {2}} \frac {p _ {\mathrm {i n}} p _ {\mathrm {o u t}}}{\gamma} \stackrel {(3 5)} {\gtrsim} n \varepsilon^ {- 3}.
$$
The lower bound  $n \varepsilon^{-3}$  is reached when in (35) we have
$$
\frac {p _ {\mathrm {o u t}} p _ {\mathrm {i n}} \tilde {L} _ {F}}{\varepsilon L _ {F} \sqrt {1 - p _ {\mathrm {i n}}}} \stackrel {<  } {_ \sim} \frac {1}{L _ {F}}.
$$
That is,  $\frac{p_{\mathrm{out}}p_{\mathrm{in}}}{\sqrt{1 - p_{\mathrm{in}}}}\lesssim \varepsilon$
In particular, we can choose the following hyperparameters to reach  $\mathcal{O}(n\varepsilon^{-3})$  sample complexity
$$
B _ {1} = n, \quad B _ {2} = \sqrt {n}, \quad p _ {\mathrm {o u t}} = \frac {1}{\sqrt {n}}, \quad S _ {1} = \mathcal {O} (\tilde {L} _ {F} ^ {2} \varepsilon^ {- 2}), \quad S _ {2} = \mathcal {O} (\tilde {L} _ {F} \varepsilon^ {- 1}), \quad p _ {\mathrm {i n}} = \mathcal {O} (\tilde {L} _ {F} ^ {- 1} \varepsilon)
$$
The step size  $\gamma$  can be chosen as
$$
\gamma \lesssim \frac {1}{\sqrt {n} L _ {F}}.
$$
and the iteration complexity
Putting these together gives the claimed sample complexity bound. By picking  $\boldsymbol{x}^s$  uniformly at random among  $\{\boldsymbol{x}^t\}_{t=0}^{T-1}$ , we get the desired guarantee.
E.3 Convergence of E-NestedVR
In this section, we analyze the sample complexity of Algorithm 1 (E-NestedVR) for the FCCO problem with
$$
G _ {\mathrm {E} - \mathrm {N V R}} ^ {t + 1} = \left\{ \begin{array}{l l} \frac {1}{B _ {1}} \sum_ {i} \left(\boldsymbol {z} _ {i} ^ {t + 1}\right) ^ {\top} \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t + 1}} ^ {(2)} \nabla f _ {i} (0) & \text {w i t h p r o b .} p _ {\text {o u t}} \\ G _ {\mathrm {E} - \mathrm {N V R}} ^ {t} + \frac {1}{B _ {2}} \sum_ {i} \left(\left(\boldsymbol {z} _ {i} ^ {t + 1}\right) ^ {\top} \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t + 1}} ^ {(2)} \nabla f _ {i} (0) - \left(\boldsymbol {z} _ {i} ^ {t}\right) ^ {\top} \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t}} ^ {(2)} \nabla f _ {i} (0)\right) & \text {w i t h p r o b .} 1 - p _ {\text {o u t}}. \end{array} \right. \tag {36}
$$
Lemma 14 (Bias and Variance of E-NestedVR) If the step size  $\gamma$  satisfies
$$
\gamma^ {2} L _ {F} ^ {2} \max \left\{\frac {(1 - p _ {i n})}{p _ {i n} S _ {2}} \frac {1 8}{B _ {2}}, \frac {1 - p _ {o u t}}{B _ {2}} \frac {(1 - p _ {i n})}{p _ {i n} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}}, \frac {(1 - p _ {o u t})}{B _ {2}} \right\} \leq \frac {1}{1 6} \cdot \frac {1}{6}
$$
then the variance and bias of  $E$ -NestedVR are
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \leq 1 4 \cdot 3 2 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ + 1 4 \cdot 9 6 \left(\frac {p _ {o u t}}{B _ {1}} + \frac {(1 - p _ {i n}) (1 - p _ {o u t})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {o u t})}{T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {b i a s} ^ {t + 1} \leq \frac {(1 - p _ {i n}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {i n} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {2 (1 - p _ {i n}) ^ {2} \tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} \\ + \left(\frac {(1 - p _ {i n}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {i n} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} + \frac {(1 - p _ {o u t}) ^ {2}}{p _ {o u t}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {v a r} ^ {t + 1}. \\ \end{array}
$$
Proof: Note that this proof is very similar to NestedVR so we highlight the differences. Let  $G^{t + 1} = G_{\mathrm{E - NVR}}^{t + 1}$  (36) be the E-NestedVR update and define
$$
G _ {i} ^ {t + 1} := \left(\boldsymbol {z} _ {i} ^ {t + 1}\right) ^ {\top} \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t + 1}} ^ {(2)} \nabla f _ {i} (0)
$$
We expand the bias by inserting  $\mathbb{E}_{i,p_{\mathrm{in}},\eta |i}[G_i^{t + 1}]$
$$
\begin{array}{l} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} = \left\| \nabla F (\boldsymbol {x} ^ {t + 1}) - \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ \leq 2 \underbrace {\left\| \nabla F (\boldsymbol {x} ^ {t + 1}) - \mathbb {E} _ {i , p _ {\mathrm {i n}} , \eta , \tilde {\eta} | i} [ G _ {i} ^ {t + 1} ] \right\| _ {2} ^ {2}} _ {A _ {1} ^ {t + 1}} + 2 \underbrace {\left\| \mathbb {E} _ {i , p _ {\mathrm {i n}} , \eta , \tilde {\eta} | i} [ G _ {i} ^ {t + 1} ] - \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2}} _ {A _ {2} ^ {t + 1}}. \\ \end{array}
$$
Consider  $A_1^{t+1}$ . The term  $A_1^{t+1}$  captures the difference between full gradient and extrapolated gradient
$$
\begin{array}{l} A _ {1} ^ {t + 1} = \left\| \mathbb {E} _ {i} \left[ (\mathbb {E} _ {\tilde {\eta} | i} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t}) ]) ^ {\top} \nabla f _ {i} (\mathbb {E} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ]) - \mathbb {E} _ {p _ {\mathrm {i n}}, \eta | i} \left[ (\mathbb {E} _ {\tilde {\eta} | i} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t}) ]) ^ {\top} \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t + 1}} ^ {(2)} \nabla f _ {i} (0) \right] \right] \right\| _ {2} ^ {2} \\ \leq C _ {g} ^ {2} \mathbb {E} _ {i} \left[ \left\| \nabla f _ {i} \left(\mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right]\right) - \mathbb {E} _ {p _ {\text {i n}}, \eta | i} \left[ \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t + 1}} ^ {(2)} \nabla f _ {i} (0) \right] \right\| _ {2} ^ {2} \right] \\ \leq 2C_{g}^{2}\underbrace{\mathbb{E}_{i}\left[ \right.\left| \right.\left|\nabla f_{i}(\mathbb{E}_{\eta |i}[g_{\eta}(\boldsymbol{x}^{t})]) - \mathbb{E}_{p_{\mathrm{in}}}\left[\nabla f_{i}(\mathbb{E}_{\eta |i}[ \boldsymbol{y}_{i}^{t + 1}])\right]\right|_{2}^{2}\left. \right]}_{=: A_{1,1}^{t + 1}} \\ + 2 C _ {g} ^ {2} \underbrace {\mathbb {E} _ {i} \left[ \left\| \mathbb {E} _ {p _ {\mathrm {i n}}} [ \nabla f _ {i} (\mathbb {E} _ {\eta | i} [ \boldsymbol {y} _ {i} ^ {t + 1} ]) ] - \mathbb {E} _ {p _ {\mathrm {i n}}, \eta | i} \left[ \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, i} ^ {t + 1}} ^ {(2)} \nabla f _ {i} (0) \right] \right\| _ {2} ^ {2} \right]} _ {=: A _ {1, 2} ^ {t + 1}}. \\ \end{array}
$$
The first term  $A_{1,1}^{t + 1}$  can be upper bounded through smoothness of  $f_{\xi}$ , for  $t \geq 1$
$$
\begin{array}{l} A _ {1, 1} ^ {t + 1} = \mathbb {E} _ {i} \left[ \left\| \nabla f _ {i} (\mathbb {E} _ {\eta | i} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ]) - p _ {\mathrm {i n}} \nabla f _ {i} (\mathbb {E} _ {\eta | i} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ]) - (1 - p _ {\mathrm {i n}}) \nabla f _ {i} (\boldsymbol {y} _ {i} ^ {t} + \mathbb {E} _ {\eta | i} [ g _ {\eta} (\boldsymbol {x} ^ {t}) - g _ {\eta} (\boldsymbol {\phi} _ {i} ^ {t}) ]) \right\| _ {2} ^ {2} \right] \\ = \left(1 - p _ {\mathrm {i n}}\right) ^ {2} \mathbb {E} _ {i} \left[ \left\| \nabla f _ {i} \left(\mathbb {E} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right]\right) - \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {t} + \mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) - g _ {\eta} \left(\phi_ {i} ^ {t}\right) \right]\right) \right\| _ {2} ^ {2} \right] \\ \leq \left(1 - p _ {\mathrm {i n}}\right) ^ {2} L _ {f} ^ {2} \mathbb {E} _ {i} \left[ \left\| \mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right] - \left(\boldsymbol {y} _ {i} ^ {t} + \mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) - g _ {\eta} \left(\phi_ {i} ^ {t}\right) \right]\right) \right\| _ {2} ^ {2} \right] \\ = \left(1 - p _ {\mathrm {i n}}\right) ^ {2} L _ {f} ^ {2} \mathbb {E} _ {i} \left[ \left\| \boldsymbol {y} _ {i} ^ {t} - \mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\phi_ {i} ^ {t}\right) \right] \right\| _ {2} ^ {2} \right] \\ = \left(1 - p _ {\mathrm {i n}}\right) ^ {2} L _ {f} ^ {2} \mathcal {E} _ {y} ^ {t}. \\ \end{array}
$$
For  $t = 0$ ,  $A_{1,1}^{1} = 0$ , then
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} A _ {1, 1} ^ {t + 1} \leq (1 - p _ {\mathrm {i n}}) ^ {2} L _ {f} ^ {2} C _ {g} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1}. \tag {37}
$$
On the other hand, with Lemma 6
$$
\begin{array}{l} A _ {1, 2} ^ {t + 1} \leq p _ {\text {i n}} \mathbb {E} _ {i} \left[ \left\| \nabla f _ {i} \left(\mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right]\right) - \mathbb {E} _ {\eta | i} \left[ \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, S _ {1}, i}} ^ {(2)} \nabla f _ {i} (0) \right] \right\| _ {2} ^ {2} \right] \\ + \left(1 - p _ {\mathrm {i n}}\right) \mathbb {E} _ {i} \left[ \left\| \nabla f _ {i} \left(\boldsymbol {y} _ {i} ^ {t} + \mathbb {E} _ {\eta | i} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) - g _ {\eta} \left(\boldsymbol {\phi} _ {i} ^ {t}\right) \right]\right) - \mathbb {E} _ {\eta | i} \left[ \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {y}, S _ {2}, i}} ^ {(2)} \nabla f _ {i} (0) \right] \right\| _ {2} ^ {2} \right] \\ \leq \frac {p _ {\mathrm {i n}} C _ {e} ^ {2}}{S _ {1} ^ {4}} + \frac {(1 - p _ {\mathrm {i n}}) C _ {e} ^ {2}}{S _ {2} ^ {4}} \\ \leq \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} \\ \end{array}
$$
where  $\mathcal{D}_{\pmb{y},S_1,i}^{t + 1}$  is the distribution of  $\frac{1}{S_1}\sum_{\eta \in S_1}\pmb {g}_\eta (\pmb {x}^t)$  and  $\mathcal{D}_{\pmb {y},S_2,i}^{t + 1}$  is the distribution of
$$
\boldsymbol {y} _ {i} ^ {t} + \frac {1}{S _ {2}} \sum_ {\eta \in S _ {2}} \left(g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) - \mathbb {E} \left[ g _ {\eta} \left(\phi_ {i} ^ {t}\right) \right]\right).
$$
Thus the  $A_1^{t + 1}$  has the following upper bound
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} A _ {1} ^ {t + 1} \leq (1 - p _ {\mathrm {i n}}) ^ {2} L _ {f} ^ {2} C _ {g} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}}. \tag {38}
$$
Consider  $A_2^{t + 1}$ . Let us expand  $A_2^{t + 1}$  through recursion
$$
\begin{array}{l} A _ {2} ^ {t + 1} = \left\| \mathbb {E} _ {i, p _ {\mathrm {i n}}, \eta , \tilde {\eta} | i} \left[ G _ {i} ^ {t + 1} \right] - \mathbb {E} \left[ G ^ {t + 1} \right] \right\| _ {2} ^ {2} \\ = \left(1 - p _ {\text {o u t}}\right) ^ {2} \left\| G ^ {t} - \mathbb {E} _ {i} \left[ \mathbb {E} _ {\eta , \tilde {\eta} | i} \left[ \tilde {G} _ {i} ^ {t} \right] \right] \right\| _ {2} ^ {2} \\ = \left(1 - p _ {\text {o u t}}\right) ^ {2} \left(\left\| \mathbb {E} [ G ^ {t} ] - \mathbb {E} _ {i} \left[ \mathbb {E} _ {\eta , \tilde {\eta} | i} \left[ \tilde {G} _ {i} ^ {t} \right] \right] \right\| _ {2} ^ {2} + \mathcal {E} _ {\text {v a r}} ^ {t}\right) \\ = \left(1 - p _ {\text {o u t}}\right) ^ {2} \left(A _ {2} ^ {t} + \mathcal {E} _ {\text {v a r}} ^ {t}\right). \\ \end{array}
$$
For  $t = 0$ , we have that  $A_2^1 = 0$ , then average over time gives
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} A _ {2} ^ {t + 1} \leq \frac {\left(1 - p _ {\mathrm {o u t}}\right) ^ {2}}{p _ {\mathrm {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1}.
$$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {b i a s}} ^ {t + 1} \leq (1 - p _ {\mathrm {i n}}) ^ {2} L _ {f} ^ {2} C _ {g} ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {y} ^ {t + 1} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} + \frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1}.
$$
Using Lemma 11
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} \leq (1 - p _ {\text {i n}}) ^ {2} L _ {f} ^ {2} C _ {g} ^ {2} \left(\frac {(1 - p _ {\text {i n}}) C _ {g} ^ {2}}{p _ {\text {i n}} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} + \frac {2 \sigma_ {g} ^ {2}}{S _ {1}}\right) \\ + \frac {C _ {e}}{S _ {2} ^ {4}} + \frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1} \\ \leq \frac {(1 - p _ {\mathrm {i n}}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\mathrm {i n}} S _ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Xi^ {t} + \frac {2 (1 - p _ {\mathrm {i n}}) ^ {2} \tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} + \frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1}. \\ \end{array}
$$
Using Lemma 10 we have that
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} \leq \frac {(1 - p _ {\text {i n}}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\text {i n}} S _ {2}} \left(\frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G ^ {t + 1} ] \| _ {2} ^ {2}\right) + \frac {2 (1 - p _ {\text {i n}}) ^ {2} \tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} + \frac {(1 - p _ {\text {o u t}}) ^ {2}}{p _ {\text {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \\ \leq \frac {\left(1 - p _ {\mathrm {i n}}\right) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\mathrm {i n}} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {2 (1 - p _ {\mathrm {i n}}) ^ {2} \tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} \\ + \left(\frac {(1 - p _ {\mathrm {i n}}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\mathrm {i n}} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} + \frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1}. \\ \end{array}
$$
Variance. Combine the variance of NestedVR in Lemma 13 and Lemma 2 gives
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \leq 1 4 \cdot 3 2 \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ + 1 4 \cdot 9 6 \left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {(1 - p _ {\text {i n}}) (1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\text {o u t}})}{T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
$$
T = \Omega \left(L _ {F} (F (\boldsymbol {x} ^ {0}) - F ^ {\star}) \varepsilon^ {- 2}\right).
$$
Proof: Denote.  $G^{t + 1} = G_{\mathrm{E - NVR}}^{t + 1}$  (36). Using descent lemma (Lemma 3) and bias-variance of E-NestedVR (Lemma 14)
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} + \frac {1}{2 T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ \leq \frac {2 (F (\boldsymbol {x} ^ {0}) - F ^ {\star})}{\gamma T} + \frac {L _ {F} \gamma}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} \\ \leq \frac {2 (F (\boldsymbol {x} ^ {0}) - F ^ {\star})}{\gamma T} + \frac {(1 - p _ {\mathrm {i n}}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\mathrm {i n}} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {2 (1 - p _ {\mathrm {i n}}) ^ {2} \tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {C _ {e} ^ {2}}{S _ {2} ^ {4}} \\ + \left(\frac {\left(1 - p _ {\text {i n}}\right) ^ {3} \bar {L} _ {F} ^ {2}}{p _ {\text {i n}} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} + \frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} + L _ {F} \gamma\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1}. \\ \end{array}
$$
As we would like the right-hand side to be bounded by either  $\frac{1}{T}\sum_{t=0}^{T-1}\left\|\mathbb{E}[G^{t+1}]\right\|_2^2$  or  $\varepsilon^2$ .
- Bound on  $\frac{2(F(\boldsymbol{x}^0) - F^\star)}{\gamma T}$  with  $\varepsilon^2$ , i.e.
$$
\gamma T \gtrsim (F (\boldsymbol {x} ^ {0}) - F ^ {\star}) \varepsilon^ {- 2} \tag {39}
$$
- Coefficient of  $\frac{1}{T}\sum_{t=0}^{T-1}\left\|\mathbb{E}[G^{t+1}]\right\|_2^2$  is bounded by  $\frac{1}{4}$ , i.e.
$$
\frac {(1 - p _ {\mathrm {i n}}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\mathrm {i n}} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \leq \frac {1}{4}
$$
which can be achieved by choosing the following step size
$$
\gamma \leq \frac {p _ {\text {o u t}} p _ {\text {i n}} \sqrt {S _ {1}}}{5 \tilde {L} _ {F} (1 - p _ {\text {i n}}) ^ {3 / 2}}. \tag {40}
$$
- Bound on  $\frac{2(1 - p_{\mathrm{in}})^2\tilde{L}_F^2}{S_1}$  with  $\varepsilon^2$
$$
\frac {2 \left(1 - p _ {\mathrm {i n}}\right) ^ {2} \tilde {L} _ {F} ^ {2}}{S _ {1}} \leq \varepsilon^ {2}. \tag {41}
$$
- Bound  $\frac{C_{\varepsilon}^{2}}{S_{2}^{4}}$  with  $\varepsilon^2$ . This leads to
$$
S _ {2} \geq \sqrt {\frac {C _ {e}}{\varepsilon}}. \tag {42}
$$
- Bound on the variance. First notice from (40) and  $\gamma \leq \frac{1}{2L_F}$ ,
$$
\frac {(1 - p _ {\mathrm {i n}}) ^ {3} \tilde {L} _ {F} ^ {2}}{p _ {\mathrm {i n}} S _ {2}} \frac {6 n ^ {2}}{B _ {2} ^ {2}} \gamma^ {2} \leq \frac {1}{4} \lesssim \frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}}
$$
$$
L _ {F} \gamma \leq \frac {1}{2} \lesssim \frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}}.
$$
Therefore, we only need to consider the upper bound on
$$
\begin{array}{l} \frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathcal {E} _ {\text {v a r}} ^ {t + 1} \\ \leq 1 4 \cdot 3 2 \frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \left(\left(\frac {p _ {\text {o u t}}}{B _ {1}} + \frac {1 - p _ {\text {o u t}}}{B _ {2}}\right) \frac {(1 - p _ {\text {i n}})}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} + \frac {(1 - p _ {\text {o u t}})}{B _ {2}}\right) \frac {\gamma^ {2} L _ {F} ^ {2}}{T} \sum_ {t = 0} ^ {T - 1} \| \mathbb {E} [ G ^ {t + 1} ] \| _ {2} ^ {2} \\ + 1 4 \cdot 9 6 \frac {\left(1 - p _ {\mathrm {o u t}}\right) ^ {2}}{p _ {\mathrm {o u t}}} \left(\frac {p _ {\mathrm {o u t}}}{B _ {1}} + \frac {(1 - p _ {\mathrm {i n}}) (1 - p _ {\mathrm {o u t}})}{B _ {2}}\right) \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} + \frac {(1 - p _ {\mathrm {o u t}}) ^ {3}}{p _ {\mathrm {o u t}} T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}}. \\ \end{array}
$$
We impose the constraints for each term
$$
\frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \frac {p _ {\text {o u t}}}{B _ {1}} \frac {\left(1 - p _ {\text {i n}}\right)}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} L _ {F} ^ {2} \gamma^ {2} \lesssim 1
$$
$$
\frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \frac {1 - p _ {\text {o u t}}}{B _ {2}} \frac {\left(1 - p _ {\text {i n}}\right)}{p _ {\text {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}} L _ {F} ^ {2} \gamma^ {2} \lesssim 1
$$
$$
\frac {\left(1 - p _ {\text {o u t}}\right) ^ {2}}{p _ {\text {o u t}}} \frac {1 - p _ {\text {o u t}}}{B _ {2}} L _ {F} ^ {2} \gamma^ {2} \lesssim 1
$$
$$
\frac {\left(1 - p _ {\mathrm {o u t}}\right) ^ {2}}{p _ {\mathrm {o u t}}} \frac {p _ {\mathrm {o u t}}}{B _ {1}} \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} \lesssim \varepsilon^ {2}
$$
$$
\frac {(1 - p _ {\mathrm {o u t}}) ^ {2}}{p _ {\mathrm {o u t}}} \frac {(1 - p _ {\mathrm {i n}}) (1 - p _ {\mathrm {o u t}})}{B _ {2}} \frac {\tilde {L} _ {F} ^ {2}}{S _ {1}} \lesssim \varepsilon^ {2}
$$
$$
\frac {\left(1 - p _ {\mathrm {o u t}}\right) ^ {3}}{p _ {\mathrm {o u t}} T} \frac {8 \tilde {L} _ {F} ^ {2}}{B _ {1} S _ {1}} \lesssim \varepsilon^ {2}.
$$
These can be simplified as
$$
\gamma \lesssim \frac {p _ {\text {i n}} p _ {\text {o u t}} \sqrt {B _ {1}} \sqrt {S _ {1}}}{(1 - p _ {\text {o u t}}) \sqrt {1 - p _ {\text {i n}}}} \frac {1}{L _ {F}} \tag {43}
$$
$$
\gamma \lesssim \frac {p _ {\text {i n}} p _ {\text {o u t}} ^ {2} \sqrt {B _ {1}} \sqrt {S _ {1}}}{(1 - p _ {\text {o u t}}) ^ {3 / 2} \sqrt {1 - p _ {\text {i n}}}} \frac {1}{L _ {F}} \tag {44}
$$
$$
\gamma \lesssim \frac {\sqrt {B _ {1}}}{(1 - p _ {\text {o u t}}) ^ {3 / 2}} \frac {1}{L _ {F}} \tag {45}
$$
$$
B _ {1} S _ {1} \gtrsim \frac {\left(1 - p _ {\text {o u t}}\right) ^ {2} \tilde {L} _ {F} ^ {2}}{\varepsilon^ {2}} \tag {46}
$$
$$
B _ {1} S _ {1} \gtrsim \frac {\left(1 - p _ {\text {o u t}}\right) ^ {3} \left(1 - p _ {\text {i n}}\right) \tilde {L} _ {F} ^ {2}}{\varepsilon^ {2} p _ {\text {o u t}} ^ {2}} \tag {47}
$$
$$
B _ {1} S _ {1} \gtrsim \frac {\left(1 - p _ {\text {o u t}}\right) ^ {3} \tilde {L} _ {F} ^ {2}}{T \varepsilon^ {2} p _ {\text {o u t}}}. \tag {48}
$$
Constraints from Lemma 14
$$
\gamma^ {2} L _ {F} ^ {2} \max  \left\{\frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {1 8}{B _ {2}}, \frac {1 - p _ {\mathrm {o u t}}}{B _ {2}} \frac {(1 - p _ {\mathrm {i n}})}{p _ {\mathrm {i n}} S _ {2}} \frac {1 8 n ^ {2}}{B _ {2} ^ {2}}, \frac {(1 - p _ {\mathrm {o u t}})}{B _ {2}} \right\} \leq \frac {1}{1 6} \cdot \frac {1}{6}
$$
which can be translated to
$$
\gamma \lesssim \frac {p _ {\text {i n}} \sqrt {S _ {1}} \sqrt {B _ {2}}}{L _ {F} \sqrt {1 - p _ {\text {i n}}}} \tag {49}
$$
$$
\gamma \lesssim \frac {p _ {\text {i n}} p _ {\text {o u t}} \sqrt {S _ {1}} \sqrt {B _ {2}}}{L _ {F} \sqrt {1 - p _ {\text {i n}}} \sqrt {1 - p _ {\text {o u t}}}} \tag {50}
$$
$$
\gamma \lesssim \frac {\sqrt {B _ {2}}}{L _ {F} \sqrt {1 - p _ {\text {o u t}}}} \tag {51}
$$
- Constraint from sufficient decrease lemma:
$$
\gamma \leq \frac {1}{2 L _ {F}}. \tag {52}
$$
We simplify the conditions noticing that 1) (48) is weaker than (46); 2) (45) and (51) are weaker than (52). Combine all the constraints on  $\gamma$ , i.e. (43), (44), (49), (50), (52)