SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We determine  $\alpha_{1}^{\prime}$ ,  $\alpha_{2}^{\prime}$  and  $\alpha_{3}^{\prime}$  by satisfying the following two conditions
- (Affine). Taylor expansion of  $\mathcal{L}_{\mathcal{D}_m}^{(3)}q(s)$  at  $s + h$  should have zero order term  $q(x + h)$ , i.e.
$$
\left(\alpha_ {1} ^ {\prime} + \alpha_ {2} ^ {\prime} + \alpha_ {3} ^ {\prime}\right) q (x + h) = q (x + h).
$$
- Taylor expansion of  $\mathcal{L}_{\mathcal{D}_m}^{(3)}q(s)$  at  $s + h$  should have 4th order term  $\mathbb{E}[(\delta -h)^4 ]$  . That is
$$
\alpha_ {1} ^ {\prime} \mathbb {E} [ (\delta_ {1} - h) ^ {4} ] + \alpha_ {2} ^ {\prime} \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {4} \right] + \alpha_ {3} ^ {\prime} \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2} + \delta_ {3}}{3} - h\right) ^ {4} \right] = 0.
$$
$$
\left(\alpha_ {1} ^ {\prime} + \frac {\alpha_ {2} ^ {\prime}}{8} + \frac {\alpha_ {3} ^ {\prime}}{2 7}\right) \mathbb {E} [ (\delta_ {1} - h) ^ {4} ] = 0
$$
$$
\left(\frac {3}{8} \alpha_ {2} ^ {\prime} + \frac {2}{9} \alpha_ {2} ^ {\prime}\right) \left(\mathbb {E} [ (\delta_ {1} - h) ^ {2} ]\right) ^ {2} = 0.
$$
Therefore,  $\alpha_{1}^{\prime},\alpha_{2}^{\prime}$  and  $\alpha_3^\prime$  can be determined through the following linear system
$$
\alpha_ {1} ^ {\prime} + \alpha_ {2} ^ {\prime} + \alpha_ {3} ^ {\prime} = 1
$$
$$
\alpha_ {1} ^ {\prime} + \frac {1}{8} \alpha_ {2} ^ {\prime} + \frac {1}{2 7} \alpha_ {3} ^ {\prime} = 0
$$
$$
\alpha_ {1} ^ {\prime} + \frac {3}{8} \alpha_ {2} ^ {\prime} + \frac {2}{9} \alpha_ {3} ^ {\prime} = 0.
$$
The solution is  $\alpha_{1}^{\prime} = \frac{1}{12}$ ,  $\alpha_{2}^{\prime} = -\frac{4}{3}$  and  $\alpha_{3}^{\prime} = \frac{9}{4}$ . Then consider the Taylor expansion of  $\mathcal{L}_{\mathcal{D}_m}^{(3)}q(s)$  at  $s + h$  with (2), we can
$$
\left. \right.\left| \mathbb {E} \left[ \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(3)} q (s) \right] - q (s + h) \right| \lesssim \left| q ^ {(5)} (s + h) \mathbb {E} [ (\delta - h) ^ {5} ] \right| + \left| \mathbb {E} \left[ q ^ {(6)} \left(\phi_ {\delta}\right) (\delta - h) ^ {6} \right]\right| \lesssim \mathcal {O} \left(\left(a _ {5} + a _ {6}\right) m ^ {- 3}\right)
$$
where the first inequality uses the fact that  $\mathcal{L}_{\mathcal{D}_m}^{(3)}$  is an affine mapping and the last inequality uses Lemma 1. Therefore,  $\mathcal{L}_{\mathcal{D}_m}^{(3)}$  is a 3rd-order extrapolation operator. We can expand it into
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(3)} q: s \mapsto \frac {1}{1 2} \left(- \frac {1}{3} \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (s) + \frac {4}{3} \mathcal {L} _ {\mathcal {D} _ {2 m}} ^ {(2)} q (s)\right) - \frac {4}{3} \left(- \frac {1}{3} \mathcal {L} _ {\mathcal {D} _ {2 m}} ^ {(2)} q (s) + \frac {4}{3} \mathcal {L} _ {\mathcal {D} _ {4 m}} ^ {(2)} q (s)\right) \\ + \frac {9}{4} \left(- \frac {1}{3} \mathcal {L} _ {\mathcal {D} _ {3 m}} ^ {(2)} q (s) + \frac {4}{3} \mathcal {L} _ {\mathcal {D} _ {6 m}} ^ {(2)} q (s)\right) \\ = \left(- \frac {1}{3 6} \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} + \frac {5}{9} \mathcal {L} _ {\mathcal {D} _ {2 m}} ^ {(2)} - \frac {3}{4} \mathcal {L} _ {\mathcal {D} _ {3 m}} ^ {(2)} - \frac {1 6}{9} \mathcal {L} _ {\mathcal {D} _ {4 m}} ^ {(2)} + 3 \mathcal {L} _ {\mathcal {D} _ {6 m}} ^ {(2)}\right) q (s). \\ \end{array}
$$
Lemma 2 (Variance Bound) Assume that  $q: \mathbb{R}^p \to \mathbb{R}^\ell$  is in  $\mathcal{C}^4$  and  $\mathcal{D}_m$  is the distribution in Assumption 1. Suppose that the variance of  $q(\boldsymbol{s} + \boldsymbol{\delta})$  is bounded as
$$
\mathbb {E} [ \| q (\boldsymbol {s} + \boldsymbol {\delta}) - \mathbb {E} [ q (\boldsymbol {s} + \boldsymbol {\delta}) ] \| _ {2} ^ {2} ] \leq \frac {V ^ {2}}{m} + C.
$$
Then the variance of extrapolation  $\mathcal{L}_{\mathcal{D}_m}^{(2)}q(s)$  is upper bounded by
$$
\mathbb {E} \left[ \left\| \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (\boldsymbol {s}) - \mathbb {E} [ \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (\boldsymbol {s}) ] \right\| _ {2} ^ {2} \right] \leq 1 4 \left(\frac {V ^ {2}}{m} + C\right).
$$
Proof: Let us use the definition of  $\mathcal{L}_{\mathcal{D}_m}^{(2)}q(s)$ :
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (\boldsymbol {s}) - \mathbb {E} [ \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (\boldsymbol {s}) ] \right\| _ {2} ^ {2} \right] \\ \leq \mathbb {E} \left[ \left\| 2 q \left(\boldsymbol {s} + \frac {\delta_ {1} + \delta_ {2}}{2}\right) - \frac {q \left(\boldsymbol {s} + \delta_ {1}\right) + q \left(\boldsymbol {s} + \delta_ {2}\right)}{2} - \mathbb {E} \left[ 2 q \left(\boldsymbol {s} + \frac {\delta_ {1} + \delta_ {2}}{2}\right) - \frac {q \left(\boldsymbol {s} + \delta_ {1}\right) + q \left(\boldsymbol {s} + \delta_ {2}\right)}{2} \right] \right\| _ {2} ^ {2} \right] \\ \leq 3 \mathbb {E} \left[ \left\| 2 q \left(\boldsymbol {s} + \frac {\delta_ {1} + \delta_ {2}}{2}\right) - \mathbb {E} \left[ 2 q \left(\boldsymbol {s} + \frac {\delta_ {1} + \delta_ {2}}{2}\right) \right] \right\| _ {2} ^ {2} \right] + 3 \mathbb {E} \left[ \left\| \frac {q (\boldsymbol {s} + \delta_ {1})}{2} - \mathbb {E} \left[ \frac {q (\boldsymbol {s} + \delta_ {1})}{2} \right] \right\| _ {2} ^ {2} \right] \\ + 3 \mathbb {E} [ \left\| \frac {q (s + \delta_ {2})}{2} - \mathbb {E} \left[ \frac {q (s + \delta_ {2})}{2} \right] \right\| _ {2} ^ {2} ] \\ \leq 1 2 \left(\frac {V ^ {2}}{2 m} + C\right) + \frac {3}{4} \left(\frac {V ^ {2}}{m} + C\right) + \frac {3}{4} \left(\frac {V ^ {2}}{m} + C\right) \\ = \frac {1 5 V ^ {2}}{2 m} + \frac {2 7 C}{2}. \\ \end{array}
$$
D Stationary Point Convergence Proofs from Section 3 (CSO)
In this section, we provide the convergence proofs for the CSO problem. We start by establishing some helpful lemmas in Appendix D.1. In Appendix D.2, we reanalyze the BSGD algorithm to obtain explicit bias and variance bounds, which are then useful when we analyze E-BSGD in Appendix D.3. Similarly, we reanalyze BSpiderBoost in Appendix D.4 and use the resulting bias and variance bounds for the analysis of E-BSpiderBoost in Appendix D.5.
Note that throughout our analyses, we define  $\mathbb{E}^{t + 1}[\cdot |t]$  as the expectation of randomness at time  $t + 1$  conditioning on the randomness until time  $t$ . When there is no ambiguity, we use  $\mathbb{E}[\cdot ]$  instead of  $\mathbb{E}^{t + 1}[\cdot |t]$ .
D.1 Helpful Lemmas
Lemma 3 (Sufficient Decrease) Suppose Assumption 5 holds true and  $\gamma \leq \frac{1}{2L_F}$  then
$$
\left\| \nabla F (\boldsymbol {x} ^ {t}) \right\| _ {2} ^ {2} \leq \frac {2 (\mathbb {E} [ F (\boldsymbol {x} ^ {t + 1}) ] - F (\boldsymbol {x} ^ {t}))}{\gamma} + L _ {F} \gamma \mathcal {E} _ {\text {v a r}} ^ {t + 1} + \mathcal {E} _ {\text {b i a s}} ^ {t + 1},
$$
where  $\mathbb{E}[\cdot ]$  denote conditional expectation over the randomness at time  $t$  conditioned on all of the past randomness until time  $t$ .
Proof: In this proof, we use  $\mathbb{E}[\cdot ]$  to denote conditional expectation over the randomness at time  $t$  conditioned on all the past randomness until time  $t$ .
Let us expand  $F(\pmb{x}^{t + 1})$  and apply the  $L_{F}$ -smoothness of  $F$
$$
\mathbb {E} [ F (\boldsymbol {x} ^ {t + 1}) ] \leq F (\boldsymbol {x} ^ {t}) - \gamma \mathbb {E} [ \langle \nabla F (\boldsymbol {x} ^ {t}), G ^ {t + 1} \rangle ] + \frac {L _ {F} \gamma^ {2}}{2} \mathbb {E} [ \| G ^ {t + 1} \| _ {2} ^ {2} ].
$$
Since  $\mathbb{E}[\| G^{t + 1}\| _2^2 ] = \mathbb{E}[\| G^{t + 1} - \mathbb{E}[G^{t + 1}]\| _2^2 ] + \| \mathbb{E}[G^{t + 1}]\| _2^2 = \mathcal{E}_{\mathrm{var}}^{t + 1} + \| \mathbb{E}[G^{t + 1}]\| _2^2$  , then
$$
\mathbb {E} \left[ F \left(\boldsymbol {x} ^ {t + 1}\right) \right] \leq F \left(\boldsymbol {x} ^ {t}\right) - \gamma \mathbb {E} \left[ \langle \nabla F \left(\boldsymbol {x} ^ {t}\right), G ^ {t + 1} \rangle \right] + \frac {L _ {F} \gamma^ {2}}{2} \left(\mathcal {E} _ {\text {v a r}} ^ {t + 1} + \left\| \mathbb {E} \left[ G ^ {t + 1} \right] \right\| _ {2} ^ {2}\right).
$$
Expand the middle term with
$$
\begin{array}{l} - \gamma \mathbb {E} [ \langle \nabla F (\boldsymbol {x} ^ {t}), G ^ {t + 1} \rangle ] = - \frac {\gamma}{2} \left\| \nabla F (\boldsymbol {x} ^ {t}) \right\| _ {2} ^ {2} - \frac {\gamma}{2} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {\gamma}{2} \left\| \nabla F (\boldsymbol {x} ^ {t}) - \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ = - \frac {\gamma}{2} \left\| \nabla F (\boldsymbol {x} ^ {t}) \right\| _ {2} ^ {2} - \frac {\gamma}{2} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {\gamma}{2} \mathcal {E} _ {\text {b i a s}} ^ {t + 1}. \\ \end{array}
$$
Combine with the inequality
$$
\mathbb {E} [ F (\boldsymbol {x} ^ {t + 1}) ] \leq F (\boldsymbol {x} ^ {t}) - \frac {\gamma}{2} \left\| \nabla F (\boldsymbol {x} ^ {t}) \right\| _ {2} ^ {2} - \frac {\gamma}{2} (1 - L _ {F} \gamma) \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {\gamma}{2} \mathcal {E} _ {\mathrm {b i a s}} ^ {t + 1} + \frac {L _ {F} \gamma^ {2}}{2} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1}.
$$
By taking  $\gamma \leq \frac{1}{2L_F}$ , we have that
$$
\mathbb {E} [ F (\boldsymbol {x} ^ {t + 1}) ] \leq F (\boldsymbol {x} ^ {t}) - \frac {\gamma}{2} \left\| \nabla F (\boldsymbol {x} ^ {t}) \right\| _ {2} ^ {2} - \frac {\gamma}{4} \left\| \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} + \frac {\gamma}{2} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} + \frac {L _ {F} \gamma^ {2}}{2} \mathcal {E} _ {\text {v a r}} ^ {t + 1}.
$$
Re-arranging the terms we get the desired inequality.
A consequence of Lemma 3 is the following result.
Lemma 4 (Descent Lemma) Suppose Assumption 5 holds true. By taking  $\gamma \leq \frac{1}{2L_F}$ , we have,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} \right] + \frac {1}{2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \mathbb {E} ^ {t} [ G ^ {t + 1} | t ] \| _ {2} ^ {2} \right] \\ \leq \frac {2 (F (\pmb {x} ^ {0}) - F ^ {\star})}{\gamma T} + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \mathcal {E} _ {b i a s} ^ {t + 1} ] + \frac {L _ {F} \gamma}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \mathcal {E} _ {v a r} ^ {t + 1} ] \\ \end{array}
$$
where the expectation is taken over all randomness from  $t = 0$  to  $T$ .
Proof: We denote the conditional expectation at time  $t$  in the descent lemma (Lemma 3) as  $\mathbb{E}^{t + 1}[\cdot |t]$  which conditions on all past randomness until time  $t$ . Then the descent lemma can be written as
$$
\mathbb {E} ^ {t + 1} [ F (\boldsymbol {x} ^ {t + 1}) | t ] \leq F (\boldsymbol {x} ^ {t}) - \frac {\gamma}{2} \left\| \nabla F (\boldsymbol {x} ^ {t}) \right\| _ {2} ^ {2} - \frac {\gamma}{4} \left\| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \right\| _ {2} ^ {2} + \frac {\gamma}{2} \mathbb {E} ^ {t + 1} [ \mathcal {E} _ {\mathrm {b i a s}} ^ {t + 1} | t ] + \frac {L _ {F} \gamma^ {2}}{2} \mathbb {E} ^ {t + 1} [ \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1} | t ].
$$
If we additionally consider the randomness at time  $t - 1$ , and apply  $\mathbb{E}^t [\cdot |t - 1]$  to both sides
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \mathbb {E} ^ {t + 1} [ F (\boldsymbol {x} ^ {t + 1}) | t ] | t - 1 \right] \leq \mathbb {E} ^ {t} [ F (\boldsymbol {x} ^ {t}) | t - 1 ] - \frac {\gamma}{2} \mathbb {E} [ \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} | t - 1 ] \\ - \mathbb {E} ^ {t} \left[ \frac {\gamma}{4} \left\| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \right\| _ {2} ^ {2} | t - 1 \right] + \frac {\gamma}{2} \mathbb {E} ^ {t} \left[ \mathbb {E} ^ {t + 1} [ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} | t ] | t - 1 \right] \\ + \frac {L _ {F} \gamma^ {2}}{2} \mathbb {E} ^ {t - 1 ]} \left[ \mathbb {E} ^ {t + 1} \left[ \mathcal {E} _ {\text {v a r}} ^ {t + 1} | t \right] | t - 1 \right]. \\ \end{array}
$$
By the law of iterative expectations, we have  $\mathbb{E}^t\left[\mathbb{E}^{t + 1}[\cdot |t]|t - 1\right] = \mathbb{E}^t\mathbb{E}^{t + 1}[\cdot |t - 1]$
$$
\begin{array}{l} \mathbb {E} ^ {t} \left[ \mathbb {E} ^ {t + 1} \left[ F (\boldsymbol {x} ^ {t + 1}) | t - 1 \right] \right] \leq \mathbb {E} ^ {t} \left[ F (\boldsymbol {x} ^ {t}) | t - 1 \right] - \frac {\gamma}{2} \mathbb {E} \left[ \left| \left| \nabla F (\boldsymbol {x} ^ {t}) \right| \right| _ {2} ^ {2} | t - 1 \right] \\ - \mathbb {E} ^ {t} \left[ \frac {\gamma}{4} \left\| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \right\| _ {2} ^ {2} | t - 1 \right] + \frac {\gamma}{2} \mathbb {E} ^ {t} \left[ \mathbb {E} ^ {t + 1} \left[ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} | t - 1 \right] \right] \\ + \frac {L _ {F} \gamma^ {2}}{2} \mathbb {E} ^ {t} \left[ \mathbb {E} ^ {t} \left[ \mathcal {E} _ {\text {v a r}} ^ {t + 1} | t - 1 \right] \right]. \\ \end{array}
$$
Similarly, we can apply  $\mathbb{E}^{t - 1}[\cdot |t - 2],\mathbb{E}^{t - 2}[\cdot |t - 3],\ldots ,\mathbb{E}^2 [\cdot |1]$  and finally  $\mathbb{E}^1 [\cdot ]$
$$
\mathbb {E} ^ {1} \dots [ \mathbb {E} ^ {t + 1} [ F (\boldsymbol {x} ^ {t + 1}) ] ] \leq \mathbb {E} ^ {1} \dots [ \mathbb {E} ^ {t} [ F (\boldsymbol {x} ^ {t}) ] ] - \frac {\gamma}{2} \mathbb {E} ^ {1} \dots [ \mathbb {E} ^ {t} [ \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} ] ]
$$
$$
\begin{array}{l} - \mathbb {E} ^ {1} \dots \left[ \mathbb {E} ^ {t} \left[ \frac {\gamma}{4} \left\| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \right\| _ {2} ^ {2} \right] \right] + \frac {\gamma}{2} \mathbb {E} ^ {1} \dots \left[ \mathbb {E} ^ {t + 1} \left[ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} \right] \right] \\ + \frac {L _ {F} \gamma^ {2}}{2} \mathbb {E} ^ {1} \dots \left[ \mathbb {E} ^ {t} \left[ \mathcal {E} _ {\text {v a r}} ^ {t + 1} \right] \right]. \\ \end{array}
$$
Now that both sides of the inequality have no randomness, we can simplify the notation by applying  $\mathbb{E}^{t + 1}\ldots [\mathbb{E}^t [\cdot ]]^\prime$  to both sides and by denoting
$$
\mathbb {E} [ \cdot ] = \mathbb {E} ^ {1} \dots [ \mathbb {E} ^ {+ 1} [ \cdot ]).
$$
Then the descent lemma becomes
$$
\mathbb {E} [ F (\boldsymbol {x} ^ {t + 1}) ] \leq \mathbb {E} [ F (\boldsymbol {x} ^ {t}) ] - \frac {\gamma}{2} \mathbb {E} [ \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} ] - \frac {\gamma}{4} \mathbb {E} [ \| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \| _ {2} ^ {2} ] + \frac {\gamma}{2} \mathbb {E} [ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} ] + \frac {L _ {F} \gamma^ {2}}{2} \mathbb {E} [ \mathcal {E} _ {\text {v a r}} ^ {t + 1} ].
$$
Now we can sum the descent lemmas from  $t = 0$  to  $T - 1$
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ F (\boldsymbol {x} ^ {t + 1}) ] \leq \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ F (\boldsymbol {x} ^ {t}) ] - \frac {\gamma}{2} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} \right] \\ - \frac {\gamma}{4} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \right\| _ {2} ^ {2} \right] + \frac {\gamma}{2} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} \right] + \frac {L _ {F} \gamma^ {2}}{2} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \mathcal {E} _ {\text {v a r}} ^ {t + 1} \right]. \\ \end{array}
$$
After simplification and division by  $T$ , we get
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla F (\boldsymbol {x} ^ {t}) \| _ {2} ^ {2} \right] + \frac {1}{2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \mathbb {E} ^ {t + 1} [ G ^ {t + 1} | t ] \| _ {2} ^ {2} \right] \\ \leq \frac {2 (\mathbb {E} [ F (\boldsymbol {x} ^ {T}) ] - \mathbb {E} [ F (\boldsymbol {x} ^ {0}) ])}{\gamma T} + \frac {\gamma}{2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} ] + \frac {L _ {F} \gamma^ {2}}{2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \mathcal {E} _ {\text {v a r}} ^ {t + 1} ] \\ \leq \frac {2 (\mathbb {E} [ F (\boldsymbol {x} ^ {T}) ] - F ^ {\star})}{\gamma T} + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \mathcal {E} _ {\text {b i a s}} ^ {t + 1} ] + \frac {L _ {F} \gamma}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \mathcal {E} _ {\text {v a r}} ^ {t + 1} ]. \\ \end{array}
$$
The following corollary is a consequence of Proposition 1.
Corollary 1 Assume  $\nabla f_{\xi}$  in CSO satisfies
$$
a_{l}:= \sup_{\boldsymbol {x}}\sup_{\xi}\left\| \nabla^{l + 1}f_{\xi}(\boldsymbol {x})\right\|_{2} <   \infty ,\qquad l = 1,2,3,4.
$$
Let's further assume that the higher order moments of  $g_{\eta}(\cdot)$  are bounded,
$$
\sigma_ {k} = \sup  _ {\boldsymbol {x}} \sup  _ {\xi} \mathbb {E} _ {\eta | \xi} \left[ \sum_ {i = 1} ^ {p} \left[ g _ {\eta} (\boldsymbol {x}) - \mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\boldsymbol {x}) ] \right] _ {i} ^ {k} \right] <   \infty , \qquad k = 1, 2, 3, 4
$$
where  $[\cdot ]_i$  refers to the  $i$ -th coordinate of a vector. Consider the  $\mathcal{L}_{\mathcal{D}_{g,\xi}^{t + 1}}^{(2)}\nabla f_{\xi}(0)$  defined in (6), then
$$
\left\| \mathbb {E} \left[ \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {g}, \xi} ^ {t + 1}} ^ {(2)} \nabla f _ {\xi} (0) \right] - \nabla f _ {\xi} \left(\mathbb {E} _ {\eta | \xi} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right]\right) \right\| _ {2} ^ {2} \leq \frac {C _ {e} ^ {2}}{m ^ {4}} \quad \forall \xi ,
$$
where  $C_e^2 (f;g) \coloneqq \left(\frac{8a_3\sigma_3 + 18a_4\sigma_2^2 + 5a_4\sigma_4}{96}\right)^2.$
Proof: The Proposition 1 gives the following upper bound
$$
\left\| \mathbb {E} \left[ \mathcal {L} _ {\mathcal {D} _ {g, \xi} ^ {t + 1}} ^ {(2)} \nabla f _ {\xi} (0) \right] - \nabla f _ {\xi} (\mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ]) \right\| _ {2} ^ {2} \leq \left(\frac {4 a _ {3} \sigma_ {3} + 9 a _ {4} \sigma_ {2} ^ {2}}{4 8 m ^ {2}} + \frac {5 a _ {4}}{9 6} \frac {\sigma_ {4} - 3 \sigma_ {2} ^ {2}}{m ^ {3}}\right) ^ {2}.
$$
For simplicity, we can relax the upper bound to
$$
\left\| \mathbb {E} \left[ \mathcal {L} _ {\mathcal {D} _ {\boldsymbol {g}, \xi} ^ {t + 1}} ^ {(2)} \nabla f _ {\xi} (0) \right] - \nabla f _ {\xi} (\mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ]) \right\| _ {2} ^ {2} \leq \frac {1}{m ^ {4}} \left(\frac {8 a _ {3} \sigma_ {3} + 1 8 a _ {4} \sigma_ {2} ^ {2} + 5 a _ {4} \sigma_ {4}}{9 6}\right) ^ {2}.
$$
D.2 Convergence of BSGD
In this section, we reanalyze the BSGD algorithm of [16] to obtain bounds on bias and variance of its gradient estimates. Theorem 6 shows that BSGD achieves an  $\mathcal{O}(\varepsilon^{-6})$  sample complexity.
Lemma 5 (Bias and Variance of BSGD) The bias and variance of BSGD are
$$
\mathcal {E} _ {b i a s} ^ {t + 1} \leq \frac {\sigma_ {b i a s} ^ {2}}{m}, \quad \mathcal {E} _ {v a r} ^ {t + 1} \leq \frac {\sigma_ {i n} ^ {2}}{m} + \sigma_ {o u t} ^ {2}
$$
where  $\sigma_{in}^{2} = \zeta_{g}^{2}C_{f}^{2} + \sigma_{g}^{2}C_{g}^{2}L_{f}^{2}$ ,  $\sigma_{out}^{2} = C_{F}^{2}$ , and  $\sigma_{bias}^{2} = \sigma_{g}^{2}C_{g}^{2}L_{f}^{2}$ .
Proof: Denote  $G^{t + 1} = G_{\mathrm{BSGD}}^{t + 1}(5)$  and denote  $\mathbb{E}[\cdot ]$  as the conditional expectation  $\mathbb{E}^{t + 1}[\cdot |t]$  which conditions on all past randomness until time  $t$ . Note that the  $\nabla g_{\tilde{\eta}}$  can be estimated without bias, i.e.
$$
\mathbb {E} _ {\tilde {\eta} | \xi} \left[ \frac {1}{m} \sum_ {\tilde {\eta} \in \tilde {H} _ {\xi}} \nabla g _ {\tilde {\eta}} (\pmb {x}) \right] = \mathbb {E} _ {\tilde {\eta} | \xi} \left[ \nabla g _ {\tilde {\eta}} (\pmb {x}) \right],
$$
Then let's first look at the bias of BSGD
$$
\begin{array}{l} \mathcal {E} _ {\text {b i a s}} ^ {t + 1} = \left\| \nabla F (\boldsymbol {x} ^ {t + 1}) - \mathbb {E} [ G ^ {t + 1} ] \right\| _ {2} ^ {2} \\ = \left\| \mathbb {E} _ {\xi} \left[ \left(\mathbb {E} _ {\tilde {\eta} | \xi} [ \nabla g _ {\tilde {\eta}} (\boldsymbol {x} ^ {t}) ]\right) ^ {\top} \left(\nabla f _ {\xi} \left(\mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ]\right) - \mathbb {E} _ {\eta | \xi} [ \nabla f _ {\xi} \left(\frac {1}{m} \sum_ {\eta \in H _ {\xi}} g _ {\eta} (\boldsymbol {x} ^ {t})\right) ]\right) \right] \right\| _ {2} ^ {2} \\ \leq C _ {g} ^ {2} \mathbb {E} _ {\xi} \left[ \left\| \nabla f _ {\xi} \big (\mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\boldsymbol {x} ^ {t}) ] \big) - \mathbb {E} _ {\eta | \xi} [ \nabla f _ {\xi} \big (\frac {1}{m} \sum_ {\eta \in H _ {\xi}} g _ {\eta} (\boldsymbol {x} ^ {t}) \big) ] \right\| _ {2} ^ {2} \right] \\ \leq C _ {g} ^ {2} L _ {f} ^ {2} \mathbb {E} _ {\xi} \left[ \mathbb {E} _ {\eta | \xi} \left[ \left\| \mathbb {E} _ {\eta | \xi} \left[ g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right] - \frac {1}{m} \sum_ {\eta \in H _ {\xi}} g _ {\eta} \left(\boldsymbol {x} ^ {t}\right) \right\| _ {2} ^ {2} \right] \right] \\ \leq \frac {C _ {g} ^ {2} L _ {f} ^ {2}}{m} \mathbb {E} _ {\xi} \left[ \mathbb {E} _ {\eta | \xi} \left[ \left| \left| \mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\pmb {x} ^ {t}) ] - g _ {\eta} (\pmb {x} ^ {t}) \right| \right| _ {2} ^ {2} \right] \right] \\ = \frac {\sigma_ {g} ^ {2} C _ {g} ^ {2} L _ {f} ^ {2}}{m} = \frac {\sigma_ {\mathrm {b i a s}} ^ {2}}{m}. \\ \end{array}
$$
For the first inequality, we take the expectation outside the norm and bound  $\nabla g_{\bar{\eta}}$  with  $C_g$ .
On the other hand, the variance of BSGD can be decomposed into inner variance and outer variance
$$
\begin{array}{l} \mathcal {E} _ {\mathrm {v a r}} ^ {t + 1} = \mathbb {E} _ {\xi} [ \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ \| G ^ {t + 1} - \mathbb {E} _ {\xi} [ \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ] ] \| _ {2} ^ {2} ] ] \\ = \mathbb {E} _ {\xi} \left[ \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ \| (G ^ {t + 1} - \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ]) + (\mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ] - \mathbb {E} _ {\xi} [ \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ]) ]) \| _ {2} ^ {2} ] \right] \\ = \underbrace{\mathbb{E}_{\xi}\bigl{\lbrack}\mathbb{E}_{\eta|\xi,\tilde{\eta} |\xi}\bigl{\lbrack}\bigl\|G^{t + 1} - \mathbb{E}_{\eta|\xi,\tilde{\eta} |\xi}\bigl[G^{t + 1}\bigr]\bigl\|_{2}^{2}\bigr{\rbrack}}_{\text{Inner variance}} + \underbrace{\mathbb{E}_{\xi}\bigl{\lbrack}\bigl\|\mathbb{E}_{\eta|\xi,\tilde{\eta} |\xi}\bigl[G^{t + 1}\bigr] - \mathbb{E}_{\xi}\bigl{\lfloor}\mathbb{E}_{\eta|\xi,\tilde{\eta} |\xi}\bigl[G^{t + 1}\bigr]\bigr{\rfloor}\bigr\|_{2}^{2}\bigr{\rbrack}}_{\text{Outer variance}}. \\ \end{array}
$$
The inner variance is bounded as follows
The outer variance is independent of the inner batch size and can be bounded by
$$
\mathbb {E} _ {\xi} [ \big \| \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ] - \mathbb {E} _ {\xi} [ \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ] ] \big \| _ {2} ^ {2} ] \leq \mathbb {E} _ {\xi} [ \big \| \mathbb {E} _ {\eta | \xi , \tilde {\eta} | \xi} [ G ^ {t + 1} ] \big \| _ {2} ^ {2} ] \leq C _ {f} ^ {2} C _ {g} ^ {2} = C _ {F} ^ {2} = \sigma_ {\mathrm {o u t}} ^ {2}.
$$
Therefore, the variance is bounded as follows
$$
\mathcal {E} _ {\text {v a r}} ^ {t + 1} \leq \frac {\sigma_ {\text {i n}} ^ {2}}{m} + \sigma_ {\text {o u t}} ^ {2}.
$$
Theorem 6 (BSGD Convergence) Consider the (CSO) problem. Suppose Assumptions 3, 4, 5 hold true. Let step size  $\gamma \leq 1 / (2L_F)$ . Then for BSGD,  $\pmb{x}^s$  picked uniformly at random among  $\{\pmb{x}^t\}_{t=0}^{T-1}$  satisfies:  $\mathbb{E}[\|\nabla F(\pmb{x}^s)\|_2^2] \leq \varepsilon^2$ , for nonconvex  $F$ , if the inner batch size  $m = \Omega(\sigma_{bias}^2\varepsilon^{-2})$  and the number of iterations  $T = \Omega((F(\pmb{x}^0) - F^\star)L_F(\sigma_{in}/m + \sigma_{out}^2)\varepsilon^{-4})$ , where  $\sigma_{in}^2 = \zeta_g^2C_f^2 + \sigma_g^2C_g^2L_f^2$ ,  $\sigma_{out}^2 = C_F^2$ , and  $\sigma_{bias}^2 = \sigma_g^2C_g^2L_f^2$ .
Proof: Denote  $G^{t + 1} = G_{\mathrm{BSGD}}^{t + 1}$  (1). Using descent lemma (Lemma 4) and bias-variance bounds of BSGD (Lemma 5)