SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
2 \gamma_ {l, k} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] - E [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} ] + \gamma_ {b, k} ^ {2} G ^ {2}
$$
Using the fact that  $\gamma_{l,K - 1}\leq \gamma_{l,k}\quad \forall k\in [K - 1]$  and set  $\gamma_{b,k} = \frac{\gamma_0}{\sqrt{k + 1}}$  , we obtain
$$
2 \gamma_ {l, K - 1} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] - E [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} ] + \frac {\gamma_ {0} ^ {2}}{k + 1} G ^ {2}
$$
Summing over  $k = 0$  to  $k = K - 1$  we obtain
$$
\begin{array}{l} 2 \gamma_ {l, K - 1} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \frac {\| x ^ {0} - x ^ {*} \| ^ {2} - \mathbb {E} [ \| x ^ {K} - x ^ {*} \| ^ {2} ]}{K} + \gamma_ {0} ^ {2} G ^ {2} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \frac {1}{k + 1} \\ \leq \frac {\| x ^ {0} - x ^ {*} \| ^ {2}}{K} + \frac {\gamma_ {0} ^ {2} G ^ {2} \log (K)}{K} \\ \end{array}
$$
Define  $\bar{x} = \frac{1}{K}\sum_{k=0}^{K-1}x^k$ , apply Jensen's inequality and rearrange
$$
\mathbb {E} [ f (\bar {x} ^ {K}) - f (x ^ {*}) ] \leq \frac {\| x ^ {0} - x ^ {*} \| ^ {2}}{2 \gamma_ {l , K - 1} K} + \frac {\gamma_ {0} ^ {2} G ^ {2} \log (K)}{2 \gamma_ {l , K - 1} K}.
$$
The approach of Theorem 2 is similar to [16, Theorem 3.2]. The crucial difference is that the step size in [16, Theorem 3.2] is constant  $(\gamma_{k} = \frac{1}{L_{\mathrm{max}}})$  for  $k \leq 4\lceil \kappa \rceil$ , whereas for envelope-type step size it is of the form:
$$
\gamma_ {k} = \min  \left\{\max  \left\{\gamma_ {l, k}, \tilde {\gamma} _ {k} \right\}, \gamma_ {b, k} \right\}, \quad \gamma_ {l, k} = \min  \left\{w, \gamma_ {b, k} \right\},
$$
where  $\bar{\gamma}_k$  can be (e.g. in the case of SPSB):
$$
\bar {\gamma} _ {k} = \min \{\frac {f _ {i _ {k}} (x ^ {k}) - l _ {i _ {k}} ^ {*}}{c _ {k} \| \nabla f _ {i _ {k}} (x ^ {k}) \| ^ {2}}, \frac {\gamma_ {b , 0}}{k + 1} \}, w = \frac {1}{2 c _ {k} L _ {\max}}.
$$
The proof of Theorem 2 suggests that the step size can be either  $\frac{f_{i_k}(x^k) - l_{i_k}^*}{c_k\|\nabla f_{i_k}(x^k)\|^2}$  or  $\frac{\gamma_{b,0}}{k + 1}$  depending on their magnitudes for  $k \leq k_0 - 1$  ( $k_0 = \max \{1, \lceil \gamma_0 / \omega \rceil - 1 \}$ ). After  $k_0$  iterations, the step size is  $\gamma_k = \frac{\gamma_{b,0}}{k + 1}$ . This finding is numerically confirmed by the experimental results in Section B.
To proceed with the proof, we have:
$$
\begin{array}{l} \left\| x ^ {k + 1} - x ^ {*} \right\| ^ {2} = \left\| x ^ {k} - \gamma_ {k} \nabla f _ {i} \left(x ^ {k}\right) - x ^ {*} \right\| ^ {2} \\ = \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \gamma_ {k} \left\langle \nabla f _ {i} \left(x ^ {k}\right), x ^ {k} - x ^ {*} \right\rangle + \gamma_ {k} ^ {2} \| \nabla f _ {i} \left(x ^ {k}\right) \| ^ {2} \\ \stackrel {(a)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \gamma_ {k} \langle \nabla f _ {i} (x ^ {k}), x ^ {k} - x ^ {*} \rangle + \gamma_ {b, k} ^ {2} \| \nabla f _ {i} (x ^ {k}) \| ^ {2}, \\ \end{array}
$$
where (a) is because  $\gamma_{k}\leq \gamma_{b,k},\forall k$  . Take conditional expectations
$$
\begin{array}{l} \mathbb {E} [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} | x ^ {k} ] \leq \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \mathbb {E} [ \gamma_ {k} | x ^ {k} ] \langle \nabla f (x ^ {k}), x ^ {k} - x ^ {*} \rangle + \gamma_ {b, k} ^ {2} \mathbb {E} [ \| \nabla f _ {i} (x ^ {k}) \| ^ {2} | x ^ {k} ] \\ \stackrel {(b)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - \mu \mathbb {E} [ \gamma_ {k} | x ^ {k} ] \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \mathbb {E} [ \gamma_ {k} | x ^ {k} ] [ f (x ^ {k}) - f (x ^ {*}) ] + \gamma_ {b, k} ^ {2} G ^ {2} \\ \stackrel {(c)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - \mu \gamma_ {l, k} \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \gamma_ {l, k} [ f (x ^ {k}) - f (x ^ {*}) ] + \gamma_ {b, k} ^ {2} G ^ {2} \\ \end{array}
$$
where (b) is by bounded gradients assumption and strong convexity of  $f$ . Take total expectation and rearrange
$$
2 \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \frac {(1 - \mu \gamma_ {l , k}) \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] - \mathbb {E} [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} ]}{\gamma_ {l , k}} + \frac {\gamma_ {b , k} ^ {2} G ^ {2}}{\gamma_ {l , k}}
$$
Choose  $k_0 = \max \{1, \lceil \gamma_0 / \omega \rceil - 1\}$ , then for  $\forall k$  s.t.  $k \geq k_0$ , we have  $\gamma_{l,k} = \min \{\omega, \gamma_{b,k}\} = \gamma_{b,k} = \frac{\gamma_0}{k + 1}$ , which means  $\gamma_k = \frac{\gamma_0}{k + 1}$  after  $k_0$  steps. Within the first  $k_0$  steps, the step size is  $\gamma_k = \min \{\omega, \gamma_{b,k}\}$ . Hence, for  $k \geq k_0$  we have
$$
2 \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \frac {(1 - \mu \gamma_ {l , k}) \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] - \mathbb {E} [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} ]}{\gamma_ {l , k}} + \frac {\gamma_ {0} G ^ {2}}{k + 1}
$$
Now, sum from  $k = k_{0}$  to  $K - 1$
$$
2 \sum_ {k = k _ {0}} ^ {K - 1} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \sum_ {k = k _ {0}} ^ {K - 1} \frac {(1 - \mu \gamma_ {l , k}) \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] - \mathbb {E} [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} ]}{\gamma_ {l , k}} + \sum_ {k = k _ {0}} ^ {K - 1} \frac {\gamma_ {0} G ^ {2}}{k + 1} \tag {29}
$$
For the first term in (29), call it  $A$ , we expand it as
$$
\begin{array}{l} A = \sum_ {k = k _ {0} + 1} ^ {K - 1} \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] (\frac {1}{\gamma_ {l , k}} - \frac {1}{\gamma_ {l , k - 1}} - \mu) + \mathbb {E} [ \| x _ {k _ {0}} - x ^ {*} \| ^ {2} ] (\frac {1}{\gamma_ {l , k _ {0}}} - \mu) - \frac {\mathbb {E} [ \| x _ {K} - x ^ {*} \| ^ {2} ]}{\gamma_ {l , K - 1}} \\ \leq \sum_ {k = k _ {0} + 1} ^ {K - 1} \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] (\frac {k + 1}{\gamma_ {0}} - \frac {k}{\gamma_ {0}} - \mu) + \mathbb {E} [ \| x _ {k _ {0}} - x ^ {*} \| ^ {2} ] (\frac {k _ {0} + 1}{\gamma_ {0}} - \mu) \\ \stackrel {(d)} {\leq} \mathbb {E} [ \| x _ {k _ {0}} - x ^ {*} \| ^ {2} ] \mu k _ {0}. \\ \end{array}
$$
where (d) is because  $\gamma_0\geq \frac{1}{\mu}$ , we have  $\frac{k + 1}{\gamma_0} -\frac{k}{\gamma_0} -\mu \leq 0,\forall k$  and  $\frac{k_0 + 1}{\gamma_0} -\mu \leq \mu k_0$ . For the second term in (29), call it B, we have
$$
B = \sum_ {k = k _ {0}} ^ {K - 1} \frac {\gamma_ {0} G ^ {2}}{k + 1} \leq \gamma_ {0} G ^ {2} \int_ {k _ {0}} ^ {K - 1} \frac {1}{x + 1} d x = \gamma_ {0} G ^ {2} [ \log (K) - \log (k _ {0} + 1) ] \leq \gamma_ {0} G ^ {2} \log K
$$
Putting  $A$  and  $B$  together, we obtain
$$
\frac {1}{K - k _ {0}} \sum_ {k = k _ {0}} ^ {K - 1} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \frac {\mathbb {E} [ \| x ^ {k _ {0}} - x ^ {*} \| ^ {2} ] \mu k _ {0}}{2 (K - k _ {0})} + \frac {\gamma_ {0} G ^ {2} \log (K)}{2 (K - k _ {0})} \tag {30}
$$
Within the first  $k_{0} - 1$  iterations, similarly to the above, we have
$$
\begin{array}{l} \mathbb {E} [ \| x ^ {k + 1} - x ^ {*} \| ] \leq (1 - \mu \gamma_ {l, k}) \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] - 2 \gamma_ {l, k} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] + \gamma_ {b, k} ^ {2} G ^ {2} \\ \leq (1 - \mu \gamma_ {l, k}) \mathbb {E} [ \| x ^ {k} - x ^ {*} \| ^ {2} ] + \gamma_ {b, k} ^ {2} G ^ {2}. \\ \end{array}
$$
For the first  $k \leq k_0 - 1$  iterations,  $\gamma_{l,k} = \omega$  where  $\omega \mu < 1$ ; thus, we obtain the following
$$
\mathbb {E} \left[ \left\| x ^ {k + 1} - x ^ {*} \right\| \right] \leq (1 - \mu \omega) E \left[ \left\| x ^ {k} - x ^ {*} \right\| ^ {2} \right] + \gamma_ {b, k} ^ {2} G ^ {2}.
$$
Solve this recursively,
$$
\begin{array}{l} \mathbb {E} [ \| x ^ {k _ {0}} - x ^ {*} \| ^ {2} ] \leq (1 - \mu \omega) ^ {k _ {0}} \| x ^ {0} - x ^ {*} \| ^ {2} + \sum_ {k = 0} ^ {k _ {0} - 1} (1 - \mu \omega) ^ {k _ {0} - k - 1} \frac {\gamma_ {0} ^ {2} G ^ {2}}{(k + 1) ^ {2}} \\ \leq (1 - \mu \omega) ^ {k _ {0}} \| x ^ {0} - x ^ {*} \| ^ {2} + \sum_ {k = 0} ^ {k _ {0} - 1} \frac {\gamma_ {0} ^ {2} G ^ {2}}{(k + 1) ^ {2}} \\ \leq \left(1 - \mu \omega\right) ^ {k _ {0}} \| x ^ {0} - x ^ {*} \| ^ {2} + \gamma_ {0} ^ {2} G ^ {2} \int_ {0} ^ {k _ {0} - 1} \frac {1}{(1 + x) ^ {2}} d x \\ = (1 - \mu \omega) ^ {k _ {0}} \| x ^ {0} - x ^ {*} \| ^ {2} + \gamma_ {0} ^ {2} G ^ {2} \left(1 - \frac {1}{k _ {0}}\right) \\ \leq \left(1 - \mu \omega\right) ^ {k _ {0}} \| x ^ {0} - x ^ {*} \| ^ {2} + \gamma_ {0} ^ {2} G ^ {2}. \tag {31} \\ \end{array}
$$
Putting this into (30), we obtain
$$
\frac {1}{K - k _ {0}} \sum_ {k = k _ {0}} ^ {K - 1} \mathbb {E} [ f (x ^ {k}) - f (x ^ {*}) ] \leq \frac {\mu k _ {0}}{2 (K - k _ {0})} \left\{\exp (- k _ {0} \mu \omega) \| x _ {0} - x ^ {*} \| ^ {2} + \gamma_ {0} ^ {2} G ^ {2} \right\} + \frac {\gamma_ {0} G ^ {2} \log K}{2 (K - k _ {0})}.
$$
Define  $\bar{x}_K = \frac{1}{K - k_0}\sum_{k = k_0}^{K - 1}x^k$ , then by Jensen's inequality we have
$$
\mathbb {E} \left[ f \left(\bar {x} _ {K}\right) - f \left(x ^ {*}\right) \right] \leq \frac {\mu k _ {0}}{2 \left(K - k _ {0}\right)} \left\{\exp \left(- k _ {0} \mu \omega\right) \| x _ {0} - x ^ {*} \| ^ {2} + \gamma_ {0} ^ {2} G ^ {2} \right\} + \frac {\gamma_ {0} G ^ {2} \log K}{2 \left(K - k _ {0}\right)},
$$
where  $k_{0} = \max \{1, \lceil \gamma_{0} / \omega \rceil - 1\}$  and  $\gamma_{0} \geq \frac{1}{\mu}$ .
A.4 Bi-level Proofs
A.4.1 Proof of Theorem 3
Start with Lemma 4:
$$
\begin{array}{l} E \left[ V ^ {k + 1} \right] \leq \mathbb {E} \left[ V ^ {k} \right] - \frac {\alpha_ {l , k}}{2} \mathbb {E} \left[ \| \nabla F \left(x ^ {k}\right) \| ^ {2} \right] + \left(\alpha_ {b, k} L _ {f} ^ {2} + 2\right) \mathbb {E} \left[ \| y ^ {k + 1} - y ^ {*} \left(x ^ {k}\right) \| ^ {2} \right] + \\ \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2}) \tilde {\sigma} _ {f} ^ {2} - \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ]. \\ \end{array}
$$
We substitute the result of Lemma 5 for the expression  $\mathbb{E}[\| y^{k + 1} - y^* (x^k)\| ^2 ]$
$$
\begin{array}{l} E [ V ^ {k + 1} ] \leq \mathbb {E} [ V ^ {k} ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] + [ (\alpha_ {b, k} L _ {f} ^ {2} + 2) (1 - \mu_ {g} C _ {k}) ^ {T} - 1 ] \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \\ 2 \alpha_ {b, k} \beta_ {b, k} T L _ {f} ^ {2} \sigma_ {g} ^ {2} + 4 \beta_ {b, k} T \sigma_ {g} ^ {2} + \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} + \frac {L _ {F}}{2}) \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2} \\ \stackrel {(a)} {\leq} \mathbb {E} \left[ V ^ {k} \right] - \frac {\alpha_ {l , k}}{2} \mathbb {E} \left[ \| \nabla F (x ^ {k}) \| ^ {2} \right] + \left[ \left(\alpha_ {b, 0} L _ {f} ^ {2} + 2\right) \left(1 - \mu_ {g} C _ {K - 1}\right) ^ {T} - 1 \right] \mathbb {E} \left[ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} \right] + \\ 2 \alpha_ {b, k} \beta_ {b, k} T L _ {f} ^ {2} \sigma_ {g} ^ {2} + 4 \beta_ {b, k} T \sigma_ {g} ^ {2} + \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} + \frac {L _ {F}}{2}) \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2} \\ \stackrel {(b)} {\leq} E [ V ^ {k} ] - \frac {\alpha_ {l , k}}{2} E [ \| \nabla F (x ^ {k}) \| ^ {2} ] + 2 \alpha_ {b, k} \beta_ {b, k} T L _ {f} ^ {2} \sigma_ {g} ^ {2} + 4 \beta_ {b, k} T \sigma_ {g} ^ {2} + \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} + \frac {L _ {F}}{2}) \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2}, \\ \end{array}
$$
where (a) is by  $\alpha_{b,0} \geq \alpha_{b,k}, C_{K-1} \leq C_k, \forall k$ , hence  $(\alpha_{b,0}L_f^2)(1 - \mu_gC_{K-1})^T \geq (\alpha_{b,k}L_f^2)(1 -$
$\mu_g C_k)^T$  (recall that  $C_k = \min \{\frac{1}{2pL_g}, \beta_{b,k}\}$  in Lemma 1); (b) is by  $T \geq \frac{\log[\alpha_{b,0} L_f^2 + 2]}{-\log(1 - \mu_{C_{K-1}})}$ , which implies that  $(\alpha_{b,0} L_f^2 + 2)(1 - \mu_g C_{K-1})^T \leq 1$ . Now, rearrange and use the fact that  $\alpha_{l,K-1} \leq \alpha_{l,k}$ ,
$$
\alpha_ {l, K - 1} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \leq 2 \mathbb {E} [ V ^ {k} ] - 2 \mathbb {E} [ V ^ {k + 1} ] + 4 \alpha_ {b, k} \beta_ {b, k} T L _ {f} ^ {2} \sigma_ {g} ^ {2} + 8 \beta_ {b, k} T \sigma_ {g} ^ {2} + 2 \alpha_ {b, k} B ^ {2} + (4 L _ {y} ^ {2} + L _ {F}) \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2}.
$$
Then sum over  $k = 0$  to  $K - 1$ :
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \leq \frac {2 V ^ {0}}{\alpha_ {l , K - 1} K} + \frac {4 T L _ {f} ^ {2} \sigma_ {g} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \alpha_ {b, k} \beta_ {b, k} + \frac {8 T \sigma_ {g} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \beta_ {b, k} + \frac {2 B ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \alpha_ {b, k} + \\ \frac {\left(4 L _ {y} ^ {2} + L _ {F}\right) \tilde {\sigma} _ {f} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \alpha_ {b, k} ^ {2} \\ \stackrel {(c)} {=} \frac {2 V ^ {0}}{\alpha_ {l , K - 1} K} + \frac {4 T L _ {f} ^ {2} \sigma_ {g} ^ {2} \alpha_ {b , 0} \beta_ {b , 0}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{(k + 1) ^ {1 . 5}} + \frac {8 T \sigma_ {g} ^ {2} \beta_ {b , 0}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{k + 1} + \\ + \frac {2 B ^ {2} \alpha_ {b , 0}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{\sqrt {k + 1}} + \frac {\left(4 L _ {y} ^ {2} + L _ {F}\right) \tilde {\sigma} _ {f} ^ {2} \alpha_ {b , 0} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{k + 1} \\ \stackrel {(d)} {\leq} \frac {2 V ^ {0}}{\alpha_ {l , 0} \sqrt {K}} + \frac {8 T L _ {f} ^ {2} \sigma_ {g} ^ {2} \alpha_ {b , 0} \beta_ {b , 0}}{\alpha_ {l , 0} \sqrt {K}} + \frac {8 T \sigma_ {g} ^ {2} \beta_ {b , 0} \log (K)}{\alpha_ {l , 0} \sqrt {K}} + \\ \frac {2 B ^ {2} \alpha_ {b , 0}}{\alpha_ {l , 0}} + \frac {\left(4 L _ {y} ^ {2} + L _ {F}\right) \tilde {\sigma} _ {f} ^ {2} \alpha_ {b , 0} ^ {2} \log (K)}{\alpha_ {l , 0} \sqrt {K}}, \\ \end{array}
$$
where we substituted  $\alpha_{b,k} = \frac{\alpha_{b,0}}{\sqrt{k + 1}}$  and  $\beta_{b,k} = \frac{\beta_{b,0}}{k + 1}$  in (c); (d) is based on  $\sum_{k=0}^{K-1} \frac{1}{(k+1)^{1.5}} \leq 2$ ,  $\sum_{k=0}^{K-1} \frac{1}{k+1} \leq \log(K)$ , and  $\sum_{k=0}^{K-1} \frac{1}{\sqrt{k+1}} \leq \sqrt{K}$ . Recall that in Lemma 2, we have  $L_f \sim \mathcal{O}(\kappa^2)$ ,  $L_y \sim \mathcal{O}(\kappa)$ , and  $L_F \sim \mathcal{O}(\kappa^3)$ . Also recall that  $\alpha_{b,k} = \frac{\alpha_{b,0}}{\sqrt{k+1}}$  and  $\alpha_{l,k} = \frac{\alpha_{l,0}}{\sqrt{k+1}}$ , hence we choose  $\alpha_{l,0} \sim \alpha_{b,0} \sim \mathcal{O}(\kappa^{-3})$ ,  $T \sim \mathcal{O}(\kappa)$ , and  $\beta_{b,0} \sim \mathcal{O}(\kappa^{-2})$ . Then we can obtain  $\frac{1}{K} \sum_{k=0}^{K-1} \mathbb{E}[\|\nabla F(x^k)\|^2] \leq \mathcal{O}\left(\frac{\kappa^3}{\sqrt{K}} + \frac{\kappa^2 \log(K)}{\sqrt{K}}\right)$ .
A.4.2 Theorem 4 and its proof
Theorem 4. Suppose  $f$  and  $g$  satisfy Assumptions 3, 4, and 5, and, learning-rate upper bounds  $\alpha_{b,k} = \frac{\alpha_{b,0}}{\sqrt{k + 1}}$  and  $\alpha_{l,k} = \frac{\alpha_{l,0}}{\sqrt{k + 1}}$  with  $\alpha_{b,0}$  and  $\alpha_{l,0}$  satisfying  $\frac{1}{L_F + 4L_g^2} \geq \frac{\alpha_{b,0}^2}{\alpha_{l,0}}$  and  $\alpha_{l,0} \leq \alpha_{b,0}$ . Further assume that  $\alpha_k$  is independent of the stochastic hypergradient  $h_f^k$ . Then, under the Bounded-variance assumption in (16) with  $p \geq \max \left\{ \frac{\mu_g}{\mu_g + L_g}, \frac{\mu_g + L_g}{4L_g} \right\}$ ,  $\beta_{b,0} \leq \min \left\{ \frac{2}{\mu_g + L_g}, \frac{\mu_g + L_g}{2\mu_gL_g}, \frac{1}{2pLg - \frac{2\mu_gL_g}{\mu_g + L_g}} \right\}$ , and  $T \geq \frac{\log(\frac{3}{2}\alpha_{b,0}L_f^2 + 2)}{\min\{-\log(1 - \frac{2\mu_gL_g}{\mu_g + L_g}\beta_{b,K-1}), -\log((2pL_g - \frac{2\mu_gL_g}{\mu_g + L_g})\beta_{b,0})\}}$ , BiSPS achieves the following rate:
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \leq \tilde {\mathcal {O}} (\frac {\kappa^ {3}}{\sqrt {K}} + \frac {\kappa^ {2} \log K}{\sqrt {K}}).
$$
Proof. Start with Lemma 4:
$$
E \left[ V ^ {k + 1} \right] \leq \mathbb {E} \left[ V ^ {k} \right] - \frac {\alpha_ {l , k}}{2} \mathbb {E} \left[ \| \nabla F \left(x ^ {k}\right) \| ^ {2} \right] + \left(\alpha_ {b, k} L _ {f} ^ {2} + 2\right) \mathbb {E} \left[ \| y ^ {k + 1} - y ^ {*} \left(x ^ {k}\right) \| ^ {2} \right] +
$$
$$
\alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2}) \tilde {\sigma} _ {f} ^ {2} - \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ].
$$
We substitute the result of Lemma 6 for the expression  $\mathbb{E}[\| y^{k + 1} - y^* (x^k)\| ^2 ]$
$$
\begin{array}{l} E \left[ V ^ {k + 1} \right] \leq \mathbb {E} \left[ V ^ {k} \right] - \frac {\alpha_ {l , k}}{2} \mathbb {E} \left[ \| \nabla F (x ^ {k}) \| ^ {2} \right] + \left[ \left(\alpha_ {b, k} L _ {f} ^ {2} + 2\right) \left(\frac {\gamma_ {b , k}}{C _ {k}} - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}} \beta_ {b, k}\right) ^ {T} - 1 \right] \mathbb {E} \left[ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} \right] + \\ + L _ {f} ^ {2} T \sigma_ {g} ^ {2} \alpha_ {b, k} \beta_ {b, k} ^ {2} + 2 T \sigma_ {g} ^ {2} \beta_ {b, k} ^ {2} + \alpha_ {b, k} B ^ {2} + \left[ 2 L _ {y} ^ {2} + \frac {L _ {F}}{2} \right] \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2} \\ \stackrel {(a)} {\leq} E [ V ^ {k} ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] + L _ {f} ^ {2} T \sigma_ {g} ^ {2} \alpha_ {b, k} \beta_ {b, k} ^ {2} + 2 T \sigma_ {g} ^ {2} \beta_ {b, k} ^ {2} + \alpha_ {b, k} B ^ {2} + [ 2 L _ {y} ^ {2} + \frac {L _ {F}}{2} ] \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2}, \\ \end{array}
$$
where in (a) we have chosen  $T \geq \max \left\{\frac{\log(\alpha_{b,0}L_f^2 + 2)}{-\log(1 - \frac{2\mu_gL_g}{\mu_g + L_g}\beta_{b,K - 1})},\frac{\log(\alpha_{b,0}^2L_f^2 + 2)}{-\log(2pL_g - \frac{2\mu_gL_g}{\mu_g + L_g})\beta_{b,0}}\right\}$ . This ensures
that  $T\geq \frac{\log(\alpha_{b,0}L_f^2 + 2)}{-\log(\frac{\beta_{b,k}}{C_k} - \frac{2\mu_gL_g}{\mu_a + L_a}\beta_{b,k})},\forall k$  , which guarantees that  $(\alpha_{b,k}L_f^2 +2)(\frac{\beta_{b,k}}{C_k} -\frac{2\mu_gL_g}{\mu_g + L_g}\beta_{b,k})^T -1\leq 0.$
Now, rearrange and use the fact that  $\alpha_{l,K - 1}\leq \alpha_{l,k}$
$$
\alpha_ {l, K - 1} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \leq 2 \mathbb {E} [ V ^ {k} ] - 2 \mathbb {E} [ V ^ {k + 1} ] + 2 \alpha_ {b, k} \beta_ {b, k} ^ {2} T L _ {f} ^ {2} \sigma_ {g} ^ {2} + 4 \beta_ {b, k} ^ {2} T \sigma_ {g} ^ {2} + 2 \alpha_ {b, k} B ^ {2} + (4 L _ {y} ^ {2} + L _ {F}) \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2}.
$$
Sum over  $k = 0$  to  $k = K - 1$ :
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \leq \frac {2 V ^ {0}}{\alpha_ {l , K - 1} K} + \frac {2 T L _ {f} ^ {2} \sigma_ {g} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \alpha_ {b, k} \beta_ {b, k} ^ {2} + \frac {4 T \sigma_ {g} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \beta_ {b, k} ^ {2} + \\ \frac {2 B ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \alpha_ {b, k} + \frac {\left(4 L _ {y} ^ {2} + L _ {F}\right) \tilde {\sigma} _ {f} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \alpha_ {b, k} ^ {2} \\ \stackrel {(b)} {=} \frac {2 V ^ {0}}{\alpha_ {l , K - 1} K} + \frac {2 T L _ {f} ^ {2} \sigma_ {g} ^ {2} \alpha_ {b , 0} \beta_ {b , 0} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{(k + 1) ^ {3 / 2}} + \frac {4 T \sigma_ {g} ^ {2} \beta_ {b , 0} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{k + 1} + \\ \frac {2 B ^ {2} \alpha_ {b , 0}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{(k + 1) ^ {0 . 5}} + \frac {\left(4 L _ {y} ^ {2} + L _ {F}\right) \tilde {\sigma} _ {f} ^ {2} \alpha_ {b , 0} ^ {2}}{\alpha_ {l , K - 1} K} \sum_ {k = 0} ^ {K - 1} \frac {1}{k + 1} \\ \stackrel {(c)} {\leq} \frac {2 V ^ {0}}{\alpha_ {l , K - 1} K} + \frac {2 T L _ {f} ^ {2} \sigma_ {g} ^ {2} \alpha_ {b , 0} \beta_ {b , 0} ^ {2}}{\alpha_ {b , 0} \sqrt {K}} + \frac {4 T \sigma_ {g} ^ {2} \beta_ {b , 0} ^ {2} \log (K)}{\alpha_ {b , 0} \sqrt {K}} + \\ + \frac {2 B ^ {2} \alpha_ {b , 0}}{\alpha_ {l , 0}} + \frac {(4 L _ {y} ^ {2} + L _ {F}) \tilde {\sigma} _ {f} ^ {2} \alpha_ {b , 0} ^ {2} \log (K)}{\alpha_ {l , 0} \sqrt {K}}, \\ \end{array}
$$
where in (b) we have substituted  $\alpha_{b,k} = \frac{\alpha_{b,0}}{\sqrt{k + 1}}$  and  $\beta_{b,k} = \frac{\beta_{b,0}}{\sqrt{k + 1}}$ , and (c) is by  $\sum_{k=0}^{K-1} \frac{1}{(k+1)^{3/2}} \leq 2$ ,  $\sum_{k=0}^{K-1} \frac{1}{k+1} \leq \log(K)$ , and  $\sum_{k=0}^{K-1} \frac{1}{(k+1)^{1/2}} \leq \sqrt{K}$ . Similar to the proof of Theorem 3, we choose  $\alpha_{l,0} \sim \alpha_{b,0} \sim \mathcal{O}(\kappa^{-3})$ ,  $T \sim \mathcal{O}(\kappa)$ , and  $\beta_{b,0} \sim \mathcal{O}(\kappa^{-1})$  to obtain  $\frac{1}{K} \sum_{k=0}^{K-1} \mathbb{E}[\|\nabla F(x^k)\|^2] \leq \mathcal{O}\left(\frac{\kappa^3}{\sqrt{K}} + \frac{\kappa^2 \log(K)}{\sqrt{K}}\right)$ .
B Additional Experiment Results
This section is organized as follows. First, we discuss synthetic quadratics experiments. Second, we provide more details on the sensitivity of the algorithms to the choices of  $\delta$  in (14), on the reset procedure, and on the search cost of BiSLS. Third, we compare the empirical performance of 1-sample vs 2-samples implementations of our algorithms for single-level convex and bi-level optimization. Some additional results for hyper-representation learning and data distillation experiments are also presented. We run 5 independent runs for all our experiments.
B.1 Synthetic Quadratics
The experiments on quadratic functions are adapted from Loizou et al. [29]. The training objective is as follows:
$$
f (x) = \frac {1}{2} \left(x - x _ {1} ^ {*}\right) ^ {T} H _ {1} (x - x _ {1} ^ {*}) + \frac {1}{2} \left(x - x _ {2} ^ {*}\right) ^ {T} H _ {2} (x - x _ {2} ^ {*}),
$$
where  $H_{i}$  ( $i = 1,2$ ) are positive definite. The optimal solutions  $x_{i}^{*}$  ( $i = 1,2$ ) are generated randomly from a standard normal distribution. Specifically,  $H_{i}$  is defined as follows:
$$
H _ {i} = O ^ {T} \cdot \operatorname {D i a g} (\log (1 + \lambda_ {i})) \cdot O, \quad i = 1, 2,
$$
where  $O$  and  $\lambda_{i}$  are taken from the spectral decomposition of  $P^T P$ , and  $P$  is generated from the standard normal distribution. Figure 9 shows the convergence of various algorithms with different starting points. Interestingly, both SPSB with either 1 sample or 2 samples (1 sample for computing the gradient and the other for computing the step size) converge to the optimal solution (labelled with a star).
Figure 9: Iterate trajectories of different starting points for the synthetic quadratic experiments.
B.2 Sensitivity of  $\delta$ , reset, and search cost
In this section, we discuss the effects of  $\delta$  in (14),  $\eta$  in Algorithm 2, and comment on the search cost of the options in the Reset Algorithm 2. Recall that the line-search condition (8) assumes that we can find a largest  $\gamma_{k} \leq \gamma_{b,0}$  to satisfy it. However, in practice, we apply a backtracking procedure, i.e.  $\gamma_{k} = \gamma_{k} * w$ ,  $0 < w < 1$ , until  $\gamma_{k}$  satisfies (8). Therefore, the found learning rate is not guaranteed to be the largest. Nonetheless, we assume that  $\gamma_{k}$  is the largest to simplify our analysis given above (similar arguments apply to line-search at both upper and lower-level in the bi-level optimization). The experiments in this section are based on hyper-representation learning [42]. In this case, the objective of the induced bilevel-optimization problem can be written as:
$$
\min _ {w} F (w) = \frac {1}{2 D _ {X _ {1}}} \| \tilde {f} (X _ {1}; w) c ^ {*} (w) - Y _ {1} \| ^ {2}
$$
$$
s. t. \quad c ^ {*} (w) = \underset {c} {\operatorname {a r g m i n}} \frac {1}{2 D _ {X _ {2}}} \| \tilde {f} (X _ {2}; w) c - Y _ {2} \| ^ {2} + \frac {\lambda}{2} \| c \| ^ {2},
$$