SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \hat {g} _ {t} - g \left(\mathbf {x} _ {t}\right) \right| \leq 4 L _ {l} D \gamma \sqrt {\log \left(4 / \delta^ {\prime}\right)}, \tag {37}
$$
and with probability  $1 - \delta'$ ,
$$
\left\| \widehat {\nabla f} _ {t} - \nabla f \left(\mathbf {x} _ {t}\right) \right\| \leq 4 L _ {f} D \gamma \sqrt {\log \left(4 / \delta^ {\prime}\right)}. \tag {38}
$$
Then by union bound and  $\delta = 3\delta'$ , we show these three equalities hold with probability  $1 - \delta$ .
Proof. Let  $\mathbf{x}_g^*$  be any point in  $\mathcal{X}_g^*$ , i.e., any optimal solution of the lower-level problem. By definition, we have  $g\left(\mathbf{x}_g^*\right) = g^*$ . Since  $g$  is convex and  $g^* \leq g\left(\mathbf{x}_0\right)$ , we have
$$
g \left(\mathbf {x} _ {0}\right) - g \left(\mathbf {x} _ {t}\right) \geq g ^ {*} - g \left(\mathbf {x} _ {t}\right) = g \left(\mathbf {x} _ {g} ^ {*}\right) - g \left(\mathbf {x} _ {t}\right) \geq \left\langle \nabla g \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {g} ^ {*} - \mathbf {x} _ {t} \right\rangle \tag {39}
$$
Add and subtract terms in inequality above, we have,
$$
\langle \widehat {\nabla} g _ {t}, \mathbf {x} _ {g} ^ {*} - \mathbf {x} _ {t} \rangle + \hat {g} _ {t} - g (\mathbf {x} _ {0}) \leq | \langle \widehat {\nabla} g _ {t} - \nabla g (\mathbf {x} _ {t}), \mathbf {x} _ {g} ^ {*} - \mathbf {x} _ {t} \rangle | + | \hat {g} _ {t} - g (\mathbf {x} _ {t}) | \tag {40}
$$
Considering the random hyperplane we used in (9), we want to prove the following inequality holds with high probability,
$$
\langle \widehat {\nabla} g _ {t}, \mathbf {x} _ {g} ^ {*} - \mathbf {x} _ {t} \rangle + \hat {g} _ {t} - g \left(\mathbf {x} _ {0}\right) \leq K _ {t} \tag {41}
$$
Recall  $K_{t} = K_{0,t} + DK_{1,t}$ . And  $K_{0,t}$  and  $K_{1,t}$  were set as the high probability bounds of  $\| \widehat{\nabla g}_{t} - \nabla g(\mathbf{x}_{t})\|$  and  $|\hat{g}_t - g(\mathbf{x}_t)|$  in Lemma 4.1 for Algorithm 1 or Lemma 4.2 for Algorithm 2. Then compare the two inequalities above and use Jensen's inequality,  $|\langle \widehat{\nabla g}_t,\mathbf{x}_g^* -\mathbf{x}_t\rangle | + |\hat{g}_t - g(\mathbf{x}_0)|\leq K_t$  holds with high probability  $1 - \delta$  for all  $t\geq 0$ . Hence, Lemma 4.3 holds with probability  $1 - \delta$  for all  $t\geq 0$ .
B.4 Improvement in one step
The following lemma characterizes the improvement of both the upper-level and lower-level objective values after one step of the algorithms.
Lemma B.2. If Assumptions 2.1, 2.2, 2.3 are satisfied,
(i) For all  $t \geq 0$ , assume that  $\mathcal{X}_g^* \subset \mathcal{X}_t$ . Then we have
$$
\gamma_ {t + 1} \mathcal {G} (\mathbf {x} _ {t}) \leq f (\mathbf {x} _ {t}) - f (\mathbf {x} _ {t + 1}) + \gamma_ {t + 1} D \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t} \| + \frac {L _ {f} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {42}
$$
As a corollary, if  $f$  is convex, we further have
$$
f \left(\mathbf {x} _ {t + 1}\right) - f ^ {*} \leq \left(1 - \gamma_ {t + 1}\right) \left(f \left(\mathbf {x} _ {t}\right) - f ^ {*}\right)) + \gamma_ {t + 1} D \| \nabla f \left(\mathbf {x} _ {t}\right) - \widehat {\nabla} f _ {t} \| + \frac {L _ {f} D ^ {2} \gamma_ {t + 1} ^ {2}}{2}. \tag {43}
$$
(ii) We have
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq \left(1 - \gamma_ {t + 1}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) + D \gamma_ {t + 1} \left(\| \nabla g \left(\mathbf {x} _ {t}\right) - \widehat {\nabla g} _ {t} \| + K _ {1, t}\right) \\ + \gamma_ {t + 1} \left(\left\| g \left(\mathbf {x} _ {t}\right) - \hat {g} _ {t} \right\| + K _ {0, t}\right) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2}. \tag {44} \\ \end{array}
$$
Proof. (i) Based on the  $L_{f}$ -smoothness of the expected function  $f$  we show that  $f(\mathbf{x}_{t + 1})$  is bounded by
$$
f \left(\mathbf {x} _ {t + 1}\right) \leq f \left(\mathbf {x} _ {t}\right) + \nabla f \left(\mathbf {x} _ {t}\right) ^ {\top} \left(\mathbf {x} _ {t + 1} - \mathbf {x} _ {t}\right) + \frac {L _ {f}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \tag {45}
$$
Replace the terms  $\mathbf{x}_{t + 1} - \mathbf{x}_t$  by  $\gamma_{t + 1}(\mathbf{s}_t - \mathbf{x}_t)$  and add and subtract the term  $\gamma_{t + 1}\widehat{\nabla f}_t^T (\mathbf{s}_t - \mathbf{x}_t)$  to the right hand side to obtain,
$$
f \left(\mathbf {x} _ {t + 1}\right) \leq f \left(\mathbf {x} _ {t}\right) + \gamma_ {t + 1} \left(\nabla f \left(\mathbf {x} _ {t}\right) - \widehat {\nabla f} _ {t}\right) ^ {\top} \left(\mathbf {s} _ {t} - \mathbf {x} _ {t}\right) + \gamma_ {t + 1} \widehat {\nabla f} _ {t} ^ {\top} \left(\mathbf {s} _ {t} - \mathbf {x} _ {t}\right) + \frac {L _ {f}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \tag {46}
$$
By Lemma 4.3,  $\mathcal{X}_g^* \subseteq \mathcal{X}_t$  with high probability  $1 - \delta$ , for all  $t = 1, \dots, T$ . Note that if we define  $\mathbf{s}_t' = \arg \max_{\mathbf{s} \in \mathcal{X}_t} \{\langle \nabla f(\mathbf{x}_t), \mathbf{x}_t - \mathbf{s} \rangle\}$ . Recall that FW gap is  $\mathcal{G}(\hat{\mathbf{x}}) = \max_{s \in \mathcal{X}_g^*} \{\langle \nabla f(\hat{\mathbf{x}}), \hat{\mathbf{x}} - \mathbf{s} \rangle\}$ . We can replace the inner product  $\langle \widehat{\nabla f}_t, \mathbf{s}_t \rangle$  by its upper bound  $\langle \widehat{\nabla f}_t, \mathbf{s}_t' \rangle$ . Applying this substitution leads to
$$
\begin{array}{l} f (\mathbf {x} _ {t + 1}) \leq f (\mathbf {x} _ {t}) + \gamma_ {t + 1} (\nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f _ {t}}) ^ {\top} (\mathbf {s} _ {t} - \mathbf {x} _ {t}) + \gamma_ {t + 1} \widehat {\nabla f _ {t}} ^ {\top} (\mathbf {s} _ {t} ^ {\prime} - \mathbf {x} _ {t}) + \frac {L _ {f}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \\ = f \left(\mathbf {x} _ {t}\right) + \gamma_ {t + 1} \left(\nabla f \left(\mathbf {x} _ {t}\right) - \widehat {\nabla f} _ {t}\right) ^ {\top} \left(\mathbf {s} _ {t} - \mathbf {x} _ {t}\right) + \gamma_ {t + 1} \left(\widehat {\nabla f} _ {t} - \nabla f \left(\mathbf {x} _ {t}\right)\right) ^ {\top} \left(\mathbf {s} _ {t} ^ {\prime} - \mathbf {x} _ {t}\right) \\ - \gamma_ {t + 1} \nabla f (\mathbf {x} _ {t}) ^ {\top} (\mathbf {x} _ {t} - \mathbf {s} _ {t} ^ {\prime}) + \frac {L _ {f}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \\ \leq f (\mathbf {x} _ {t}) + \gamma_ {t + 1} \left(\nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t}\right) ^ {\top} \left(\mathbf {s} _ {t} - \mathbf {s} _ {t} ^ {\prime}\right) - \gamma_ {t + 1} \mathcal {G} (\mathbf {x} _ {t}) + \frac {L _ {f}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \\ \leq f \left(\mathbf {x} _ {t}\right) + \gamma_ {t + 1} D \| \nabla f \left(\mathbf {x} _ {t}\right) - \widehat {\nabla} f _ {t} \| - \gamma_ {t + 1} \mathcal {G} \left(\mathbf {x} _ {t}\right) + \frac {L _ {f} \gamma_ {t + 1} ^ {2} D ^ {2}}{2} \tag {47} \\ \end{array}
$$
Rearrange the terms for the inequality above, we can obtain,
$$
\gamma_ {t + 1} \mathcal {G} (\mathbf {x} _ {t}) \leq f (\mathbf {x} _ {t}) - f (\mathbf {x} _ {t + 1}) + \gamma_ {t + 1} D \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t}) \| + \frac {L _ {f} \gamma_ {t + 1} ^ {2} D ^ {2}}{2} \tag {48}
$$
As a simple corollary, since  $\mathcal{G}(\mathbf{x}_t) \geq f(\mathbf{x}_t) - f^*$  when  $f$  is convex, we have,
$$
f \left(\mathbf {x} _ {t + 1}\right) - f ^ {*} \leq \left(1 - \gamma_ {t + 1}\right) \left(f \left(\mathbf {x} _ {t}\right) - f ^ {*}\right)) + \gamma_ {t + 1} D \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t} \| + \frac {L _ {f} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {49}
$$
(ii) Based on the  $L_{g}$ -smoothness of the expected function  $g$  we show that  $g(\mathbf{x}_{t + 1})$  is bounded by
$$
g \left(\mathbf {x} _ {t + 1}\right) \leq g \left(\mathbf {x} _ {t}\right) + \nabla g \left(\mathbf {x} _ {t}\right) ^ {\top} \left(\mathbf {x} _ {t + 1} - \mathbf {x} _ {t}\right) + \frac {L _ {g}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \tag {50}
$$
Replace the terms  $\mathbf{x}_{t + 1} - \mathbf{x}_t$  by  $\gamma_{t + 1}(\mathbf{s}_t - \mathbf{x}_t)$  and add and subtract the term  $\gamma_{t + 1}\widehat{\nabla g_t}^\top (\mathbf{s}_t - \mathbf{x}_t)$  to the right-hand side to obtain,
$$
g \left(\mathbf {x} _ {t + 1}\right) \leq g \left(\mathbf {x} _ {t}\right) + \gamma_ {t + 1} \left(\nabla g \left(\mathbf {x} _ {t}\right) - \widehat {\nabla g} _ {t}\right) ^ {\top} \left(\mathbf {s} _ {t} - \mathbf {x} _ {t}\right) + \gamma_ {t + 1} \widehat {\nabla g} _ {t} ^ {\top} \left(\mathbf {s} _ {t} - \mathbf {x} _ {t}\right) + \frac {L _ {g}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \tag {51}
$$
Now by definition of the set  $\mathcal{X}_t$ , using  $\langle \widehat{\nabla g}_t, \mathbf{s}_t - \mathbf{x}_t \rangle \leq g(\mathbf{x}_0) - \hat{g}_t + K_{0,t} + DK_{1,t}$ . In addition, we could use Cauchy-Schwarz inequality to upper bound the second term. Then add and subtract  $\gamma_{t+1} g(\mathbf{x}_0)$  on the right hand side to obtain,
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) \leq g \left(\mathbf {x} _ {t}\right) + \gamma_ {t + 1} \left(g \left(\mathbf {x} _ {0}\right) - g \left(\mathbf {x} _ {t}\right)\right) + \gamma_ {t + 1} D \| \nabla g \left(\mathbf {x} _ {t}\right) - \widehat {\nabla g} _ {t} \| \tag {52} \\ + \gamma_ {t + 1} (g (\mathbf {x} _ {t}) - \hat {g} _ {t}) + \gamma_ {t + 1} \left(K _ {0 _ {t}} + D K _ {1 _ {t}}\right) + \frac {L _ {g}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \\ \end{array}
$$
Then subtract  $g(\mathbf{x}_0)$  on both sides,
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq \left(1 - \gamma_ {t + 1}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) \\ + \gamma_ {t + 1} (D \| \nabla g (\mathbf {x} _ {t}) - \widehat {\nabla g} _ {t} \| + \| g (\mathbf {x} _ {t}) - \hat {g} _ {t} \| + K _ {0, t} + D K _ {1, t}) + \frac {L _ {g}}{2} \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \| ^ {2} \tag {53} \\ \end{array}
$$
and the claim in the lemma follows.
C Proof of Theorem for Algorithm 1
Proof. For lower-level, by Lemma B.2, we have
$$
\begin{array}{l} g (\mathbf {x} _ {t + 1}) - g (\mathbf {x} _ {0}) \leq (1 - \gamma_ {t + 1}) (g (\mathbf {x} _ {t}) - g (\mathbf {x} _ {0})) + D \gamma_ {t + 1} (\| \nabla g (\mathbf {x} _ {t}) - \widehat {\nabla g} _ {t} \| + K _ {1, t}) \\ + \gamma_ {t + 1} \left(\left\| g \left(\mathbf {x} _ {t}\right) - \hat {g} _ {t} \right\| + K _ {0, t}\right) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {54} \\ \end{array}
$$
By Lemma 4.1, we have  $\| \nabla g(\mathbf{x}_t) - \widehat{\nabla g}_t\| \leq K_{1,t}$  and  $\| g(\mathbf{x}_t) - \hat{g}_t\| \leq K_{0,t}$  with probability  $1 - \delta'$ . Plug them in the inequality above to obtain,
$$
\begin{array}{l} g (\mathbf {x} _ {t + 1}) - g (\mathbf {x} _ {0}) \leq (1 - \gamma_ {t + 1}) (g (\mathbf {x} _ {t}) - g (\mathbf {x} _ {0})) + 2 \gamma_ {t + 1} (D K _ {1, t} + K _ {0, t}) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \\ \leq \left(1 - \frac {1}{t + 1}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) \tag {55} \\ + \frac {2 (D A _ {1} ^ {1} \sqrt {\log (6 d / \delta^ {\prime})} + A _ {0} ^ {1} \sqrt {\log (6 / \delta^ {\prime})})}{(t + 1) ^ {3 / 2}} + \frac {L _ {g} D ^ {2}}{2 (t + 1) ^ {2}} \\ \end{array}
$$
with probability  $1 - \delta'$  for all  $t$ . Let  $C_1 = 4(DA_1^1 + A_0^1)$  and  $\delta = T\delta'$ . Then, by applying the inequality recursively for  $t = 1, \ldots, T-1$ , we obtain that
$$
\begin{array}{l} g (\mathbf {x} _ {T}) - g (\mathbf {x} _ {0}) \leq \left(1 - \frac {1}{T}\right) \left(g (\mathbf {x} _ {T - 1}) - g (\mathbf {x} _ {0})\right) + \frac {C _ {1} / 2 \sqrt {\log \left(6 d / \delta^ {\prime}\right)}}{T ^ {3 / 2}} + \frac {L _ {g} D ^ {2}}{2 T ^ {2}} \\ = \prod_ {t = 1} ^ {T - 1} \left(1 - \frac {1}{t + 1}\right) (g (\mathbf {x} _ {0}) - g (\mathbf {x} _ {0})) + \sum_ {t = 1} ^ {T - 1} \frac {C _ {1} / 2 \sqrt {\log \left(6 d / \delta^ {\prime}\right)}}{(t + 1) ^ {3 / 2}} \prod_ {i = t + 1} ^ {T - 1} (1 - \frac {1}{i + 1}) \\ + \sum_ {t = 1} ^ {T - 1} \frac {L _ {g} D ^ {2}}{2 (t + 1) ^ {2}} \prod_ {i = t + 1} ^ {T - 1} \left(1 - \frac {1}{i + 1}\right) \\ \leq 0 + \frac {C _ {1} / 2 \sqrt {\log \left(6 d / \delta^ {\prime}\right)}}{T} \sum_ {t = 1} ^ {T - 1} \frac {1}{\sqrt {t + 1}} + \frac {L _ {g} D ^ {2}}{2 T} \sum_ {t = 1} ^ {T - 1} \frac {1}{t + 1} \\ \leq \frac {C _ {1} \sqrt {\log \left(6 d / \delta^ {\prime}\right)}}{\sqrt {T}} + \frac {L _ {g} D ^ {2}}{2 T} (1 + \log T) \\ \leq \frac {C _ {1} \sqrt {\log (6 t d / \delta)}}{\sqrt {T}} + \frac {L _ {g} D ^ {2} \log T}{T} \tag {56} \\ \end{array}
$$
For upper-level, by Lemma B.2, we have
$$
f \left(\mathbf {x} _ {t + 1}\right) - f ^ {*} \leq \left(1 - \gamma_ {t + 1}\right) \left(f \left(\mathbf {x} _ {t}\right) - f ^ {*}\right) + D \gamma_ {t + 1} \left(\| \nabla f \left(\mathbf {x} _ {t}\right) - \widehat {\nabla f} _ {t} \|\right) + \frac {L _ {f} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {57}
$$
By Lemma 4.1, we have  $\| \nabla f(\mathbf{x}_t) - \widehat{\nabla f}_t\| \leq \frac{A_2^1\sqrt{\log(6d / \delta')}}{(t + 1)^{1 / 2}}$  with probability  $1 - \delta'$ . Plug it in the inequality above to obtain,
$$
f \left(\mathbf {x} _ {t + 1}\right) - f ^ {*} \leq \left(1 - \frac {1}{t + 1}\right) \left(f \left(\mathbf {x} _ {t}\right) - f ^ {*}\right) + \frac {D A _ {2} ^ {1} \sqrt {\log \left(6 d / \delta^ {\prime}\right)}}{\left(t + 1\right) ^ {3 / 2}} + \frac {L _ {f} D ^ {2}}{2 \left(t + 1\right) ^ {2}} \tag {58}
$$
with probability  $1 - \delta'$  for all  $t$ . Let  $C_2 = 2DA_2^1$  and  $\delta = T\delta'$ . Then, by applying the inequality recursively for  $t = 1,\ldots,T-1$ , we obtain that
$$
\begin{array}{l} f (\mathbf {x} _ {T}) - f ^ {*} \leq \left(1 - \frac {1}{T}\right) \left(f (\mathbf {x} _ {T - 1}) - f ^ {*}\right) + \frac {D A _ {2} ^ {1} \sqrt {\log (6 d / \delta^ {\prime})}}{T ^ {3 / 2}} + \frac {L _ {f} D ^ {2}}{2 T ^ {2}} \\ = \prod_ {t = 1} ^ {T - 1} \left(1 - \frac {1}{t + 1}\right) \left(f \left(\mathbf {x} _ {0}\right) - f ^ {*}\right) + \sum_ {t = 1} ^ {T - 1} \frac {D A _ {2} ^ {1} \sqrt {\log (6 d / \delta^ {\prime})}}{(t + 1) ^ {3 / 2}} \prod_ {i = t + 1} ^ {T - 1} \left(1 - \frac {1}{i + 1}\right) \\ + \sum_ {t = 1} ^ {T - 1} \frac {L _ {f} D ^ {2}}{2 (t + 1) ^ {2}} \prod_ {i = t + 1} ^ {T - 1} \left(1 - \frac {1}{i + 1}\right) \\ \leq \frac {f (\mathbf {x} _ {0}) - f ^ {*}}{T} + \frac {D A _ {2} ^ {1} \sqrt {\log (6 d / \delta^ {\prime})}}{T} \sum_ {t = 1} ^ {T - 1} \frac {1}{\sqrt {t + 1}} + \frac {L _ {f} D ^ {2}}{2 T} \sum_ {t = 1} ^ {T - 1} \frac {1}{t + 1} \\ \leq \frac {f (\mathbf {x} _ {0}) - f ^ {*}}{T} + \frac {2 D A _ {2} ^ {1} \sqrt {\log (6 d / \delta^ {\prime})}}{\sqrt {T}} + \frac {L _ {f} D ^ {2}}{2 T} (1 + \log T) \\ \leq \frac {f \left(\mathbf {x} _ {0}\right) - f ^ {*}}{T} + \frac {2 D A _ {2} ^ {1} \sqrt {\log (6 t d / \delta)}}{\sqrt {T}} + \frac {L _ {f} D ^ {2} \log T}{T} \tag {59} \\ \end{array}
$$
with probability  $1 - \delta$ . The theorem is obtained.
C.2 Proof of Theorem 4.5
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq \left(1 - \gamma_ {t + 1}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) + D \gamma_ {t + 1} \left(\| \nabla g \left(\mathbf {x} _ {t}\right) - \widehat {\nabla g} _ {t} \| + K _ {1, t}\right) \\ + \gamma_ {t + 1} \left(\left\| g \left(\mathbf {x} _ {t}\right) - \hat {g} _ {t} \right\| + K _ {0, t}\right) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {60} \\ \end{array}
$$
By Lemma 4.1, we have  $\| \nabla g(\mathbf{x}_t) - \widehat{\nabla g}_t\| \leq K_{1,t}$  and  $\| g(\mathbf{x}_t) - \hat{g}_t\| \leq K_{0,t}$  with probability  $1 - \delta^{\prime}$ . Plug them in the inequality above to obtain,
$$
\begin{array}{l} g (\mathbf {x} _ {t + 1}) - g (\mathbf {x} _ {0}) \leq (1 - \gamma_ {t + 1}) (g (\mathbf {x} _ {t}) - g (\mathbf {x} _ {0})) + 2 \gamma_ {T + 1} (D K _ {1, t} + K _ {0, t}) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \\ \leq \left(1 - \frac {1}{(T + 1) ^ {2 / 3}}\right) g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right) \tag {61} \\ + \frac {2 D (A _ {1} ^ {2 / 3} \sqrt {\log (6 d / \delta^ {\prime})} + A _ {0} ^ {2 / 3} \sqrt {\log (6 d / \delta^ {\prime})})}{(t + 1) ^ {1 / 3} (T + 1) ^ {2 / 3}} + \frac {L _ {g} D ^ {2}}{2 (T + 1) ^ {4 / 3}} \\ \end{array}
$$
with probability  $1 - \delta'$  for all  $t$ . Let  $C_3 = 2(DA_1^{2/3} + A_0^{2/3})$ . Then we can sum all the inequality up for all  $t$  to obtain,
$$
\begin{array}{l} g (\mathbf {x} _ {t + 1}) - g (\mathbf {x} _ {0}) \leq (1 - \frac {1}{(T + 1) ^ {2 / 3}}) (g (\mathbf {x} _ {t}) - g (\mathbf {x} _ {0})) + \frac {C _ {3} \sqrt {\log (6 d / \delta^ {\prime})}}{(t + 1) ^ {1 / 3} (T + 1) ^ {2 / 3}} + \frac {L _ {g} D ^ {2}}{2 (T + 1) ^ {4 / 3}} \\ \leq \left(1 - \frac {1}{(T + 1) ^ {2 / 3}}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) + \frac {C _ {3} \sqrt {\log (6 T d / \delta)} + L _ {g} D ^ {2} / 2}{(t + 1) ^ {1 / 3} (T + 1) ^ {2 / 3}} \tag {62} \\ \end{array}
$$
By induction, we have for all  $t \geq 1$ ,
$$
g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq \frac {C _ {3} \sqrt {\log (6 T d / \delta)} + L _ {g} D ^ {2} / 2}{(T + 1) ^ {1 / 3}} \tag {63}
$$
with probability  $1 - \delta$ , where  $\delta = T\delta'$ .
$$
\gamma_ {t + 1} \mathcal {G} (\mathbf {x} _ {t}) \leq f (\mathbf {x} _ {t}) - f (\mathbf {x} _ {t + 1}) + \gamma_ {t + 1} D \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t} \| + \frac {L _ {f} \gamma_ {t + 1} ^ {2} D ^ {2}}{2} \tag {64}
$$
By Lemma 4.1, we have  $\| \nabla f(\mathbf{x}_t) - \widehat{\nabla f}_t \| \leq \frac{A_2^{2/3} \sqrt{\log(6d / \delta')}}{(t + 1)^{1/3}}$  with probability  $1 - \delta'$ . Plug it and  $\gamma_{t+1} = 1 / (T + 1)^{2/3}$  in inequality above to obtain,
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1} \mathcal {G} (\mathbf {x} _ {t}) \leq f (\mathbf {x} _ {0}) - f (\mathbf {x} _ {T}) + D \sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1} \| \nabla f (\mathbf {x} _ {t}) - \widehat {\nabla f} _ {t} \| + \frac {L _ {f} D ^ {2}}{2} \sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1} ^ {2} \\ \leq f \left(\mathbf {x} _ {0}\right) - f \left(\mathbf {x} _ {T}\right) + D \sum_ {t = 0} ^ {T - 1} \frac {A _ {2} ^ {2 / 3} \sqrt {\log (6 d / \delta^ {\prime})}}{(t + 1) ^ {1 / 3} (T + 1) ^ {2 / 3}} + \frac {L _ {f} D ^ {2}}{2} \sum_ {t = 0} ^ {T - 1} \frac {1}{(T + 1) ^ {4 / 3}} \tag {65} \\ \leq f (\mathbf {x} _ {0}) - f (\mathbf {x} _ {T}) + \frac {3}{2} D A _ {2} ^ {2 / 3} \sqrt {\log (6 d / \delta^ {\prime})} + \frac {L _ {f} D ^ {2}}{2} \frac {1}{(T + 1) ^ {1 / 3}} \\ \end{array}
$$
Let  $\mathbf{x}_{t^*} = \arg \min_{1\leq t\leq T}\mathcal{G}(\mathbf{x}_t)$ , then
$$
\begin{array}{l} \mathcal {G} \left(\mathbf {x} _ {t ^ {*}}\right) \leq \frac {1}{\sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1}} \sum_ {t = 0} ^ {T - 1} \gamma_ {t + 1} \mathcal {G} \left(\mathbf {x} _ {t}\right) \\ \leq \frac {1}{(T + 1) ^ {1 / 3}} \left(f \left(\mathbf {x} _ {0}\right) - f \left(\mathbf {x} _ {T}\right) + \frac {3}{2} D A _ {2} ^ {2 / 3} \sqrt {\log (6 T d / \delta)} + \frac {L _ {f} D ^ {2}}{2} \frac {1}{(T + 1) ^ {1 / 3}}\right) \tag {66} \\ \leq \frac {1}{(T + 1) ^ {1 / 3}} (f (\mathbf {x} _ {0}) - \underline {{f}} + \frac {3}{2} D A _ {2} ^ {2 / 3} \sqrt {\log (6 T d / \delta)} + \frac {L _ {f} D ^ {2}}{2} \frac {1}{(T + 1) ^ {1 / 3}}) \\ \end{array}
$$
with probability  $1 - \delta$ , where  $\delta = T\delta'$ . By letting  $C_4 = \frac{3}{2} DA_2^{2/3}$ , the theorem is obtained.
D Proof of Theorem for Algorithm 2
D.1 Proof of Theorem 4.6
Proof. For lower-level By Lemma B.2, we have
$$
\begin{array}{l} g \left(\mathbf {x} _ {t + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq \left(1 - \gamma_ {t + 1}\right) \left(g \left(\mathbf {x} _ {t}\right) - g \left(\mathbf {x} _ {0}\right)\right) + D \gamma_ {t + 1} \left(\| \nabla g \left(\mathbf {x} _ {t}\right) - \widehat {\nabla g} _ {t} \| + K _ {1, t}\right) \\ + \gamma_ {t + 1} \left(\left\| g \left(\mathbf {x} _ {t}\right) - \hat {g} _ {t} \right\| + K _ {0, t}\right) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {67} \\ \end{array}
$$
By Lemma 4.2, we have  $\| \nabla g(\mathbf{x}_t) - \widehat{\nabla} g_t\| \leq 4L_gD\gamma \sqrt{\log(12 / \delta')}$  and  $\| g(\mathbf{x}_t) - \hat{g}_t\| \leq$ $4L_{l}D\gamma \sqrt{\log(12 / \delta^{\prime})}$  with probability  $1 - \delta '$ . Let  $C_5 = 8D(DL_g + L_l)$  and  $\delta = T\delta '$ . Plug them in inequality above and let  $\gamma_{t} = \gamma = \log T / T$  to obtain,
$$
g \left(\mathbf {x} _ {T + 1}\right) - g \left(\mathbf {x} _ {0}\right) \leq (1 - \gamma) \left(g \left(\mathbf {x} _ {T}\right) - g \left(\mathbf {x} _ {0}\right)\right) + \left(C _ {5} \sqrt {\log \left(1 2 / \delta^ {\prime}\right)} + L _ {g} D ^ {2} / 2\right) \gamma^ {2} \tag {68}
$$
with probability  $1 - \delta / T$ . Sum up the inequalities for all  $1 \leq t \leq T$  to get,
$$
\begin{array}{l} g (\mathbf {x} _ {T + 1}) - g (\mathbf {x} _ {0}) = (1 - \gamma) ^ {T} (g (\mathbf {x} _ {0}) - g (\mathbf {x} _ {0})) + (C _ {5} \sqrt {\log (1 2 / \delta^ {\prime})} + L _ {g} D ^ {2} / 2) \gamma^ {2} \sum_ {k = 1} ^ {T} (1 - \gamma) ^ {k} \\ \leq 0 + \left(C _ {5} \sqrt {\log \left(1 2 / \delta^ {\prime}\right)} + L _ {g} D ^ {2} / 2\right) \gamma \leq \frac {\left(C _ {5} \sqrt {\log \left(1 2 T / \delta\right)} + L _ {g} D ^ {2} / 2\right) \log T}{T} \tag {69} \\ \end{array}
$$
For upper-level, by Lemma B.2, we have,
$$
f (\mathbf {x} _ {T}) - f ^ {*} \leq (1 - \gamma_ {T}) f (\mathbf {x} _ {T - 1}) - f ^ {*} + D \gamma_ {T} \| \nabla f (\mathbf {x} _ {T - 1}) - \widehat {\nabla f} _ {t - 1} \| + \frac {L _ {f} D ^ {2} \gamma_ {T} ^ {2}}{2} \tag {70}
$$
Now we proceed by replacing the terms  $\| \nabla f(\mathbf{x}_t) - \widehat{\nabla f}_t\|$  by its upper bounds from Lemma 4.2, i.e.  $\| \nabla f(\mathbf{x}_t) - \widehat{\nabla f}_t\| \leq 4L_fD\gamma \sqrt{\log(12 / \delta')}$ ,
$$
f \left(\mathbf {x} _ {T}\right) - f ^ {*} \leq (1 - \gamma) \left(f \left(\mathbf {x} _ {T - 1}\right) - f ^ {*}\right) + L _ {f} D ^ {2} \gamma^ {2} \left(4 \sqrt {\log \left(1 2 / \delta^ {\prime}\right)} + 1 / 2\right) \tag {71}
$$
with probability  $(1 - \delta')$ . And we can choose  $\delta = 3T\delta'$  Then by telescope, with  $\gamma = \frac{\log T}{T}$ , we can obtain,
$$
\begin{array}{l} f (\mathbf {x} _ {T}) - f ^ {*} \leq (1 - \gamma) ^ {T} (f (\mathbf {x} _ {0}) - f ^ {*}) + (4 \sqrt {\log {(1 2 / \delta^ {\prime})}} + 1 / 2) L _ {f} D ^ {2} \gamma^ {2} \sum_ {i = 1} ^ {T} (1 - \gamma) ^ {i} \\ \leq (1 - \gamma) ^ {T} \left(f \left(\mathbf {x} _ {0}\right) - f ^ {*}\right) + \left(4 \sqrt {\log \left(1 2 / \delta^ {\prime}\right)} + 1 / 2\right) L _ {f} D ^ {2} \gamma \tag {72} \\ \leq \exp {(- \gamma T)} (f (\mathbf {x} _ {0}) - f ^ {*}) + (4 \sqrt {\log {(1 2 / \delta^ {\prime})}} + 1 / 2) L _ {f} D ^ {2} \gamma \\ \leq \left(f \left(\mathbf {x} _ {0}\right) - f ^ {*}\right) / T + \left(4 \sqrt {\log (1 2 T / \delta)} + 1 / 2\right) L _ {f} D ^ {2} \log T / T \\ \end{array}
$$
with probability  $1 - \delta$ . Note that without loss of generality, we can assume  $f(\mathbf{x}_0) - f^* \geq 0$ . If it is less than 0, we can bound it by 0. By letting  $C_6 = 5L_fD^2$ , the theorem is obtained.
D.2 Proof of Theorem 4.7
$$
\begin{array}{l} g (\mathbf {x} _ {t + 1}) - g (\mathbf {x} _ {0}) \leq (1 - \gamma_ {t + 1}) (g (\mathbf {x} _ {t}) - g (\mathbf {x} _ {0})) + D \gamma_ {t + 1} (\| \nabla g (\mathbf {x} _ {t}) - \widehat {\nabla g} _ {t} \| + K _ {1, t}) \\ + \gamma_ {t + 1} \left(\left\| g \left(\mathbf {x} _ {t}\right) - \hat {g} _ {t} \right\| + K _ {0, t}\right) + \frac {L _ {g} D ^ {2} \gamma_ {t + 1} ^ {2}}{2} \tag {73} \\ \end{array}
$$