SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recall that the hypergradient surrogate defined in (17) based on  $(x^{k},y^{k + 1})$  is
$$
\bar {\nabla} f \left(x ^ {k}, y ^ {k + 1}\right) = \nabla_ {x} f \left(x ^ {k}, y ^ {k + 1}\right) + \nabla_ {x y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) \left[ \nabla_ {y y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) \right] ^ {- 1} \nabla_ {y} f \left(x ^ {k}, y ^ {k + 1}\right). \tag {23}
$$
Given a filtration  $\mathcal{F}_k^\prime$  up to and including  $x^{k}$  and  $y^{k + 1}$ , the bias of the stochastic hypergradient is defined as  $B = \bar{\nabla} f(x^{k},y^{k + 1}) - \mathbb{E}[h_{f}^{k}|\mathcal{F}_{k}^{\prime}]$ , and the variance is defined as  $\mathbb{E}[\| \bar{\nabla} f(x^k,y^{k + 1}) - \mathbb{E}[h_f^k |\mathcal{F}_k']\| ^2 ]$ . Lemma 3 has been proven in [19, Lem 1.]. (also see [4, Lem 5.]). Lemmas 1, 2, and 3 will be used in the proofs of Theorems 3 and 4.
Lemma 3. Under Assumptions 3, 4, and 5, the bias and variance of the stochastic hypergradient  $h_f^k$  satisfy the following
$$
\text {B i a s :} \| \bar {\nabla} f (x ^ {k}, y ^ {k + 1}) - \mathbb {E} [ h _ {f} ^ {k} | \mathcal {F} _ {k} ^ {\prime} ] \| \leq \frac {C _ {g} C _ {f}}{\mu_ {g}} (1 - \frac {\mu_ {g}}{L _ {g}}) ^ {N}, \forall k
$$
$$
V a r i a n c e: \mathbb {E} [ \| \bar {\nabla} f (x ^ {k}, y ^ {k + 1}) - \mathbb {E} [ h _ {f} ^ {k} | \mathcal {F} _ {k} ^ {\prime} ] \| ^ {2} ] \leq \tilde {\sigma} _ {f} ^ {2}, \forall k,
$$
where  $N$  is the total number of samples, and  $\tilde{\sigma}_f^2 = \sigma_f^2 +[(\sigma_f^2 +L_1^2)(\sigma_G^2 +2L_g^2) + \sigma_f^2 L_g^2 ]\frac{3}{\mu_g^2}\sim \mathcal{O}(\kappa^2)$ .
Lemma 4 is on the descent of the quantity  $V^{k} \coloneqq F(x^{k}) - F(x^{*}) + \| y^{k} - y^{*}(x^{k})\|^{2}$ . It will be used in the proofs of Theorem 3 and 4.
Lemma 4. Suppose  $F$  satisfies Assumptions 3, 4, and 5, sequences  $\alpha_{b,k} = \frac{\alpha_{b,0}}{\sqrt{k + 1}}$  and  $\alpha_{l,k} = \frac{\alpha_{l,0}}{\sqrt{k + 1}}$  with  $\alpha_{b,0}$  and  $\alpha_{l,0}$  satisfying  $\frac{1}{L_F + 4L_y^2} \geq \frac{\alpha_{b,0}^2}{\alpha_{l,0}}$  and  $\alpha_{l,0} \leq \alpha_{b,0}$ . Further assume that  $\alpha_k$  is independent of  $h_f^k$ . Then step sizes of the form (10) and (11) achieve the following:
$$
E [ V ^ {k + 1} ] \leq \mathbb {E} [ V ^ {k} ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] + (\alpha_ {b, k} L _ {f} ^ {2} + 2) \mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] +
$$
$$
\alpha_ {b, k} B ^ {2} + \left(2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2}\right) \tilde {\sigma} _ {f} ^ {2} - \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ], \forall k, \tag {24}
$$
where  $V^{k} = F(x^{k}) - F(x^{*}) + \| y^{k + 1} - y^{*}(x^{k + 1})\|^{2}$  and recall that  $F$  is the upper-level loss defined in (1).
Proof. We denote  $\mathbb{E}[h_f^k |\mathcal{F}_k'] = \bar{h}_f^k$ . By the  $L_{F}$ -smoothness of the objective  $F$ :
$$
F (x ^ {k + 1}) \leq F (x ^ {k}) + \langle \nabla F (x ^ {k}), x ^ {k + 1} - x ^ {k} \rangle + \frac {L _ {F}}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2}.
$$
Take expectation conditioned on a filtration of past iterates  $\mathcal{F}_k^\prime$  (up to and include  $x^{k},y^{k + 1}$  ..
$$
\begin{array}{l} \mathbb {E} [ F (x ^ {k + 1}) | \mathcal {F} _ {k} ^ {\prime} ] \leq F (x ^ {k}) + \mathbb {E} [ \langle \nabla F (x ^ {k}), x ^ {k + 1} - x ^ {k} \rangle | \mathcal {F} _ {k} ^ {\prime} ] + \frac {L _ {F}}{2} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} | \mathcal {F} _ {k} ^ {\prime} ] \\ = F (x ^ {k}) - \mathbb {E} [ \alpha_ {k} \langle \nabla F (x ^ {k}), h _ {f} ^ {k} \rangle | \mathcal {F} _ {k} ^ {\prime} ] + \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \mathbb {E} [ \| h _ {f} ^ {k} - \bar {h} _ {f} ^ {k} + \bar {h} _ {f} ^ {k} \| ^ {2} | \mathcal {F} _ {k} ^ {\prime} ] \\ \stackrel {(a)} {=} F (x ^ {k}) - \mathbb {E} [ \alpha_ {k} | \mathcal {F} _ {k} ^ {\prime} ] \langle \nabla F (x ^ {k}), \bar {h} _ {f} ^ {k} \rangle + \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \tilde {\sigma} _ {f} ^ {2} \\ = F (x ^ {k}) - \frac {\mathbb {E} [ \alpha_ {k} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \| \nabla F (x ^ {k}) \| ^ {2} - \frac {\mathbb {E} [ \alpha_ {k} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \frac {\mathbb {E} [ \alpha_ {k} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \| \nabla F (x ^ {k}) - \bar {h} _ {f} ^ {k} \| ^ {2} + \\ \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \tilde {\sigma} _ {f} ^ {2}, \\ \end{array}
$$
where (a) is by the assumption that  $\alpha_{k}$  is independent of  $h_f^k$ . Then expand the term  $\| \nabla F(x^{k}) - \bar{h}_{f}^{k}\|^{2}$  as follows:
$$
\begin{array}{l} \left\| \nabla F (x ^ {k}) - \bar {h} _ {f} ^ {k} \right\| ^ {2} = \left\| \nabla F (x ^ {k}) - \bar {\nabla} f (x ^ {k}, y ^ {k + 1}) + \bar {\nabla} f (x ^ {k}, y ^ {k + 1}) - \bar {h} _ {f} ^ {k} \right\| ^ {2} \\ \leq 2 \| \nabla F (x ^ {k}) - \bar {\nabla} f (x ^ {k}, y ^ {k + 1}) \| ^ {2} + 2 \| \bar {\nabla} f (x ^ {k}, y ^ {k + 1}) - \bar {h} _ {f} ^ {k} \| ^ {2} \\ \stackrel {(b)} {\leq} 2 L _ {f} ^ {2} \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 B ^ {2}, \\ \end{array}
$$
where (b) is by Lemma 2 and 3. Substituting this into the above:
$$
\begin{array}{l} \mathbb {E} [ F (x ^ {k + 1}) | F _ {k} ^ {\prime} ] \leq F (x ^ {k}) - \frac {\mathbb {E} [ \alpha_ {k} | F _ {k} ^ {\prime} ]}{2} \| \nabla F (x ^ {k}) \| ^ {2} - \frac {\mathbb {E} [ \alpha_ {k} | F _ {k} ^ {\prime} ]}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | F _ {k} ^ {\prime} ]}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \\ E [ \alpha_ {k} | \mathcal {F} _ {k} ^ {\prime} ] L _ {f} ^ {2} \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + E [ \alpha_ {k} | \mathcal {F} _ {k} ^ {\prime} ] B ^ {2} + \frac {L _ {F} \mathbb {E} [ \alpha_ {k} ^ {2} | \mathcal {F} _ {k} ^ {\prime} ]}{2} \tilde {\sigma} _ {f} ^ {2} \\ \stackrel {(c)} {\leq} F (x ^ {k}) - \frac {\alpha_ {l , k}}{2} \| \nabla F (x ^ {k}) \| ^ {2} - \frac {\alpha_ {l , k}}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + \\ \alpha_ {b, k} L _ {f} ^ {2} \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + \alpha_ {b, k} B ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2} \tilde {\sigma} _ {f} ^ {2}, \\ \end{array}
$$
where (c) is by  $\alpha_{l,k} \leq \alpha_k$  and  $\alpha_{b,k} \geq \alpha_k$ . Then take total expectations and subtract  $F(x^*)$ :
$$
\begin{array}{l} \mathbb {E} [ F (x ^ {k + 1}) - F (x ^ {*}) ] \leq \mathbb {E} [ F (x ^ {k}) - F (x ^ {*}) ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \bar {h} _ {f} ^ {k} \| ^ {2} ] + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2} \mathbb {E} [ \| \bar {h} _ {f} ^ {k} \| ^ {2} ] + \\ \alpha_ {b, k} L _ {f} ^ {2} \mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \alpha_ {b, k} B ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2} \tilde {\sigma} _ {f} ^ {2} \tag {25} \\ \end{array}
$$
Now define the potential function  $V^{k} \coloneqq F(x^{k}) - F(x^{*}) + \| y^{k + 1} - y^{*}(x^{k + 1})\|^{2}$  and expand the term  $\| y^{k + 1} - y^{*}(x^{k + 1})\|^{2}$  as follows:
$$
\begin{array}{l} \left\| y ^ {k + 1} - y ^ {*} \left(x ^ {k + 1}\right) \right\| ^ {2} = \left\| y ^ {k + 1} - y ^ {*} \left(x ^ {k}\right) + y ^ {*} \left(x ^ {k}\right) - y ^ {*} \left(x ^ {k + 1}\right) \right\| ^ {2} \\ \leq 2 \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 \| y ^ {*} (x ^ {k}) - y ^ {*} (x ^ {k + 1}) \| ^ {2} \\ \stackrel {(d)} {\leq} 2 \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 L _ {y} ^ {2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} \\ = 2 \| y ^ {k + 1} - y ^ {*} \left(x ^ {k}\right) \| ^ {2} + 2 L _ {y} ^ {2} \alpha_ {k} ^ {2} \| h _ {f} ^ {k} \| ^ {2} \\ = 2 \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 L _ {y} ^ {2} \alpha_ {k} ^ {2} \| h _ {f} ^ {k} - \bar {h} _ {f} ^ {k} + \bar {h} _ {f} ^ {k} \| ^ {2}, \\ \end{array}
$$
where (d) is by Lemma 2. Take expectation conditioned on  $\mathcal{F}_k^{\prime}$ :
$$
\begin{array}{l} \mathbb {E} \left[ \| y ^ {k + 1} - y ^ {*} \left(x ^ {k + 1}\right) \| ^ {2} \mid \mathcal {F} _ {k} ^ {\prime} \right] \leq 2 \| y ^ {k + 1} - y ^ {*} \left(x ^ {k}\right) \| ^ {2} + 2 L _ {y} ^ {2} \alpha_ {k} ^ {2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + 2 L _ {y} ^ {2} \alpha_ {k} ^ {2} \tilde {\sigma} _ {f} ^ {2} \\ \leq 2 \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} \| \bar {h} _ {f} ^ {k} \| ^ {2} + 2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2}. \\ \end{array}
$$
Then, take total expectations:
$$
\mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k + 1}) \| ^ {2} ] \leq 2 \mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] + 2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} \mathbb {E} [ \| \bar {h} _ {f} ^ {k} \| ^ {2} ] + 2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} \tilde {\sigma} _ {f} ^ {2}. \tag {26}
$$
Now, based on the definition of  $V^k$  and combining (25) and (26):
$$
\begin{array}{l} E [ V ^ {k + 1} ] \leq \mathbb {E} [ F (x ^ {k}) - F (x ^ {*}) ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \\ - \frac {\mathbb {E} [ \| \bar {h} _ {f} ^ {k} \| ^ {2} ]}{2} (\alpha_ {l, k} - L _ {F} \alpha_ {b, k} ^ {2} - 4 L _ {y} ^ {2} \alpha_ {b, k} ^ {2}) + (\alpha_ {b, k} L _ {f} ^ {2} + 2) \mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \\ \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2}) \tilde {\sigma} _ {f} ^ {2} \\ \stackrel {(e)} {\leq} \mathbb {E} [ F (x ^ {k}) - F (x ^ {*}) ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] + (\alpha_ {b, k} L _ {f} ^ {2} + 2) \mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \\ \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2}) \tilde {\sigma} _ {f} ^ {2} \\ = \mathbb {E} [ V ^ {k} ] - \frac {\alpha_ {l , k}}{2} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] + (\alpha_ {b, k} L _ {f} ^ {2} + 2) \mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \\ \alpha_ {b, k} B ^ {2} + (2 L _ {y} ^ {2} \alpha_ {b, k} ^ {2} + \frac {L _ {F} \alpha_ {b , k} ^ {2}}{2}) \tilde {\sigma} _ {f} ^ {2} - \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ], \\ \end{array}
$$
where (e) is because  $\frac{1}{L_F + 4L_y^2} \geq \frac{\alpha_{b,0}^2}{\alpha_{l,0}}$ , which guarantees that  $\alpha_{l,k} = \frac{\alpha_{l,0}}{\sqrt{k + 1}} \geq (L_F + 4L_y^2)\alpha_{b,k} =$
$$
\left(L _ {F} + 4 L _ {y} ^ {2}\right) \frac {\alpha_ {b , 0} ^ {2}}{k + 1}.
$$
Lemma 5 and Lemma 6 give two alternatives for bounding the term  $\| y^{k + 1} - y^* (x^k)\| ^2$ . Lemma 5 is based on the assumption  $\mathbb{E}_{\psi}[g(x,y^{*}(x);\psi) - g(x,y_{x,\psi}^{*};\psi)]\leq \sigma_{g}^{2},\forall x$ . The proof for its one-variable analogous assumption is given in Loizou et al. [29]. Here we follow a similar approach for the two-variable function  $g(x,y)$ . Lemma 5 will be used in the proof of Theorem 3.
Lemma 5. Suppose Assumptions 3, 5, and the bounded optimal function value assumption (15) hold. Further assume that each sampled function  $g(x,y;\psi)$  is convex, then step size of the form 11 achieves the following:
$$
\mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] \leq \left(1 - \mu_ {g} C _ {k}\right) ^ {T} \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ] + 2 \beta_ {b, k} T \sigma_ {g} ^ {2},
$$
where  $C_k = \min \left\{\frac{1}{2pL_g},\beta_{b,k}\right\} .$
Proof. We denote  $h_g^{k,t} = \nabla_y g(x^k, y^{k,t}; \psi)$ , and  $\mathcal{F}_{k,t}'$  be a filtration up to and including  $x^k$  and  $y^{k,t}$ . We have,
$$
\begin{array}{l} \left\| y ^ {k, t + 1} - y ^ {*} \left(x ^ {k}\right) \right\| ^ {2} = \left\| y ^ {k, t} - \beta_ {k, t} h _ {g} ^ {k, t} - y ^ {*} \left(x ^ {k}\right) \right\| ^ {2} \\ = \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} - 2 \beta_ {k, t} \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + \beta_ {k, t} ^ {2} \| h _ {g} ^ {k, t} \| ^ {2} \\ \stackrel {(a)} {\leq} \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} - 2 \beta_ {k, t} \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + \frac {\beta_ {k , t}}{p} [ g (x ^ {k}, y ^ {k, t}; \psi) - g (x ^ {k}, y _ {x ^ {k}, \psi} ^ {*}; \psi) ] \\ \stackrel {(b)} {\leq} \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} - 2 \beta_ {k, t} \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + 2 \beta_ {k, t} [ g (x ^ {k}, y ^ {k, t}; \psi) - g (x ^ {k}, y _ {x ^ {k}, \psi} ^ {*}; \psi) ] \\ = \| y ^ {k, t} - y ^ {*} \left(x ^ {k}\right) \| ^ {2} - 2 \beta_ {k, t} \left\langle y ^ {k, t} - y ^ {*} \left(x ^ {k}\right), h _ {g} ^ {k, t} \right\rangle + \\ 2 \beta_ {k, t} \left[ g \left(x ^ {k}, y ^ {k, t}; \psi\right) - g \left(x ^ {k}, y ^ {*} \left(x ^ {k}\right); \psi\right) + g \left(x ^ {k}, y ^ {*} \left(x ^ {k}\right); \psi\right) - g \left(x ^ {k}, y _ {x ^ {k}, \psi} ^ {*}; \psi\right) \right] \\ = \| y ^ {k, t} - y ^ {*} \left(x ^ {k}\right) \| ^ {2} + 2 \beta_ {k, t} \left[ - \left\langle y ^ {k, t} - y ^ {*} \left(x ^ {k}\right), h _ {q} ^ {k, t} \right\rangle + g \left(x ^ {k}, y ^ {k, t}; \psi\right) - g \left(x ^ {k}, y ^ {*} \left(x ^ {k}\right); \psi\right) \right] + \\ 2 \beta_ {k, \ell} [ g (x ^ {k}, y ^ {*} (x ^ {k}); \psi) - g (x ^ {k}, y _ {x ^ {k}, \psi} ^ {*}; \psi) ] \\ \stackrel {(c)} {\leq} \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 C _ {k} [ - \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + g (x ^ {k}, y ^ {k, t}; \psi) - g (x ^ {k}, y ^ {*} (x ^ {k}); \psi) ] + \\ 2 \beta_ {b, k} [ g (x ^ {k}, y ^ {*} (x ^ {k}); \psi) - g (x ^ {k}, y _ {x ^ {k}, \psi} ^ {*}; \psi) ], \\ \end{array}
$$
where (a) is by Lemma 1, (b) is by choosing  $p \geq \frac{1}{2}$ , and, (c) is by individual convexity of  $g(x,y;\psi)$  such that  $-\langle y^{k,t} - y^{*}(x^{k}), h_{g}^{k,t} \rangle + g(x^{k}, y^{k,t}; \psi) - g(x^{k}, y^{*}(x^{k}); \psi) \leq 0$  and recalling that  $C_k = \min \left\{\frac{1}{2pL_g}, \beta_{b,k}\right\} \leq \beta_{k,t}$  by Lemma 1. Take expectation conditioned on  $\mathcal{F}_{k,t}'$  and note that  $\mathbb{E}[h_{g}^{k,t} | \mathcal{F}_{k,t}'] = \nabla_y g(x^k, y^{k+1})$ ,  $\mathbb{E}[g(x^k, y^{k,t}; \psi) | \mathcal{F}_{k,t}'] = g(x^k, y^{k,t})$ , and  $\mathbb{E}[g(x^k, y^*(x^k); \psi)] = g(x^k, y^*(x^k))$ :
$$
\mathbb {E} \left[ \| y ^ {k, t + 1} - y ^ {*} (x ^ {k}) \| ^ {2} \mid \mathcal {F} _ {k, t} ^ {\prime} \right] \leq \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} + 2 C _ {k} \left[ - \langle y ^ {k, t} - y ^ {*} (x ^ {k}), \nabla_ {y} g (x ^ {k}, y ^ {k + 1}) \rangle + \right.
$$
$$
\left. g \left(x ^ {k}, y ^ {k, t}\right) - g \left(x ^ {k}, y ^ {*} \left(x ^ {k}\right)\right) \right] + 2 \beta_ {b, k} \sigma_ {g} ^ {2}. \tag {27}
$$
Now, based on the strong convexity of  $g$  w.r.t.  $y$ ,
$$
- \langle y ^ {k, t} - y ^ {*} (x ^ {k}), \nabla_ {y} g (x ^ {k}, y ^ {k + 1}) \rangle + g (x ^ {k}, y ^ {k, t}) - g (x ^ {k}, y ^ {*} (x ^ {k})) \leq \frac {\mu_ {g}}{2} \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2},
$$
we can further obtain (by taking total expectations of (27) and using strong-convexity):
$$
\mathbb {E} [ \| y ^ {k, t + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] \leq (1 - \mu_ {g} C _ {k}) \mathbb {E} [ \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} ] + 2 \beta_ {b, k} \sigma_ {g} ^ {2}.
$$
Solve this recursively from  $t = 0$  to  $t = T - 1$  (recall  $T$  is the total number of lower-level steps,  $y^{k,0} = y^k$  and  $y^{k + 1} = y^{k,T}$ ):
$$
\begin{array}{l} \mathbb {E} \left[ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} \right] \leq \left(1 - \mu_ {g} C _ {k}\right) ^ {T} \mathbb {E} \left[ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} \right] + 2 \beta_ {b, k} \sigma_ {g} ^ {2} \sum_ {j = 0} ^ {T - 1} \left(1 - \mu_ {g} C _ {k}\right) ^ {j} \\ \leq \left(1 - \mu_ {g} C _ {k}\right) ^ {T} \mathbb {E} \left[ \left\| y ^ {k} - y ^ {*} \left(x ^ {k}\right) \right\| ^ {2} \right] + 2 \beta_ {b, k} T \sigma_ {g} ^ {2}. \\ \end{array}
$$
Lemma 6 is based on the standard bounded variance assumption  $\mathbb{E}_{\psi}[\| \nabla_y g(x,y^* (x);\psi) - \nabla_yg(x,y^* (x))\| ^2 ]\leq \sigma_g^2,\forall x,y$  in the bi-level optimization literature [19, 4]. Lemma 6 will be used in the proof of Theorem 4.
Lemma 6. Suppose Assumptions 3, 5 and the bounded variance assumption (16) hold. Suppose that  $p \geq \max \left\{\frac{\mu_g}{\mu_g + L_g}, \frac{\mu_g + L_g}{4L_g}\right\}$ ,  $\beta_{b,0} \leq \min \left\{\frac{2}{\mu_g + L_g}, \frac{\mu_g + L_g}{2\mu_gL_g}, \frac{1}{2pLg - \frac{2\mugL_g}{\mu_g + L_g}}\right\}$ . Then step size of the form 11 achieves the following:
$$
\mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] \leq \left(\frac {\beta_ {b , k}}{C _ {k}} - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}} \beta_ {b, k}\right) ^ {T} \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ] + T \beta_ {b, k} ^ {2} \sigma_ {g} ^ {2},
$$
Proof. Similar to the proof of Lemma 5, we can start with
$$
\| y ^ {k, t + 1} - y ^ {*} (x ^ {k}) \| ^ {2} = \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} - 2 \beta_ {k, t} \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + \beta_ {k, t} ^ {2} \| h _ {g} ^ {k, t} \| ^ {2}.
$$
Divide both sides by  $\beta_{k,t}$
$$
\frac {\| y ^ {k , t + 1} - y ^ {*} (x ^ {k}) \| ^ {2}}{\beta_ {k , t}} = \frac {\| y ^ {k , t} - y ^ {*} (x ^ {k}) \| ^ {2}}{\beta_ {k , t}} - 2 \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + \beta_ {k, t} \| h _ {g} ^ {k, t} \| ^ {2}.
$$
Then use the facts that  $\beta_{k,t} \leq \beta_{b,k}$  and  $\beta_{k,t} \geq C_k$ ,
$$
\frac {\| y ^ {k , t + 1} - y ^ {*} (x ^ {k}) \| ^ {2}}{\beta_ {b , k}} \leq \frac {\| y ^ {k , t} - y ^ {*} (x ^ {k}) \| ^ {2}}{C _ {k}} - 2 \langle y ^ {k, t} - y ^ {*} (x ^ {k}), h _ {g} ^ {k, t} \rangle + \beta_ {b, k} \| h _ {g} ^ {k, t} \| ^ {2}.
$$
Next, take expectation conditioned on the Filtration  $\mathcal{F}_{k,t}^{\prime}$  up to and including  $x^{k}$  and  $y^{k,t}$
$$
\begin{array}{l} \frac {1}{\beta_ {b , k}} \mathbb {E} [ \| y ^ {k, t + 1} - y ^ {*} (x ^ {k}) \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] \leq \frac {1}{C _ {k}} \mathbb {E} [ \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] - 2 \langle y ^ {k, t} - y ^ {*} (x ^ {k}), \nabla g (x ^ {k}, y ^ {k, t}) \rangle \\ + \beta_ {b, k} \mathbb {E} [ \| h _ {g} ^ {k, t} \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] \\ = \frac {1}{C _ {k}} \mathbb {E} [ \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] - 2 \langle y ^ {k, t} - y ^ {*} (x ^ {k}), \nabla g (x ^ {k}, y ^ {k, t}) \rangle \\ + \beta_ {b, k} \mathbb {E} [ \| h _ {g} ^ {k, t} - \nabla g (x ^ {k}, y ^ {k, t}) + \nabla g (x ^ {k}, y ^ {k, t}) \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \frac {1}{C _ {k}} \mathbb {E} [ \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] - 2 \langle y ^ {k, t} - y ^ {*} (x ^ {k}), \nabla g (x ^ {k}, y ^ {k, t}) \rangle \\ + \beta_ {b, k} \sigma_ {g} ^ {2} + \beta_ {b, k} \| \nabla g (x ^ {k}, y ^ {k, t}) \| ^ {2}, \\ \end{array}
$$
where (a) is by the bounded variance assumption (16). Based on strong-convexity of  $g$  [34, Theorem 2.1.11], we have
$$
\begin{array}{l} \frac {1}{\beta_ {b , k}} \mathbb {E} [ \| y ^ {k, t + 1} - y ^ {*} (x ^ {k}) \| ^ {2} | \mathcal {F} _ {k, t} ^ {\prime} ] \leq \frac {1}{C _ {k}} \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}} \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} \\ - \frac {2}{\mu_ {g} + L _ {g}} \| \nabla g (x ^ {k}, y ^ {k, t}) \| ^ {2} + \beta_ {b, k} \sigma_ {g} ^ {2} + \beta_ {b, k} \| \nabla g (x ^ {k}, y ^ {k, t}) \| ^ {2} \\ \end{array}
$$
Multiply by  $\beta_{b,k}$  in both sides, group terms, and take total expectations to reach
$$
\begin{array}{l} \mathbb {E} [ \| y ^ {k, t + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] \leq (\frac {\beta_ {b , k}}{C _ {k}} - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}} \beta_ {b, k}) \mathbb {E} [ \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \beta_ {b, k} ^ {2} \sigma_ {g} ^ {2} \\ + \beta_ {b, k} (\beta_ {b, k} - \frac {2}{\mu_ {g} + L _ {g}}) \| \nabla g (x ^ {k}, y ^ {k, t}) \| ^ {2} \\ \stackrel {(b)} {\leq} \left(\frac {\beta_ {b , k}}{C _ {k}} - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}} \beta_ {b, k}\right) \mathbb {E} [ \| y ^ {k, t} - y ^ {*} (x ^ {k}) \| ^ {2} ] + \beta_ {b, k} ^ {2} \sigma_ {g} ^ {2}, \\ \end{array}
$$
where in (b) we have chosen  $\beta_{b,k} \leq \beta_{b,0} \leq \frac{2}{\mu_g + L_g}, \forall k$ . Solving this recursion similar to Lemma 5, we obtain:
$$
\mathbb {E} [ \| y ^ {k + 1} - y ^ {*} (x ^ {k}) \| ^ {2} ] \leq \left(\frac {\beta_ {b , k}}{C _ {k}} - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}} \beta_ {b, k}\right) ^ {T} \mathbb {E} [ \| y ^ {k} - y ^ {*} (x ^ {k}) \| ^ {2} ] + T \beta_ {b, k} ^ {2} \sigma_ {g} ^ {2}. \tag {28}
$$
In (28), we require  $\frac{\beta_{b,k}}{C_k} -\frac{2\mu_gL_g}{\mu_g + L_g}\beta_{b,k}\geq 0$  (recall  $C_k = \min \{\frac{1}{2pL_g},\beta_{b,k}\} \leq \beta_{k,t}\leq \beta_{b,k})$  , this is equivalent to  $C_k\leq \frac{\mu_g + L_g}{2\mugL_g}$  . In case of  $C_k = \frac{1}{2pL_g}$  , we choose  $p\geq \frac{\mu_g}{\mu_g + L_g}$  (to avoid contradictions, we also choose  $p\geq \frac{\mu_g + L_g}{4L_g}$  ). In case of  $C_k = \beta_{b,k}$  , we choose  $\beta_{b,k}\leq \beta_{b,0}\leq \frac{\mu_g + L_g}{2\mugL_g}$
We also require  $\frac{\beta_{b,k}}{C_k} -\frac{2\mu_gL_g}{\mu_g + L_g}\beta_{b,k}\leq 1$ . In case of  $C_k = \beta_{b,k}$ , this is equivalent to  $\frac{2\mu_gL_g}{\mu_g + L_g}\beta_{b,k}\geq 0$ , which is satisfied by all  $\beta_{b,k}$ . In case of  $C_k = \frac{1}{2pL_g}$ , we choose  $\beta_{b,k}\leq \beta_{b,0}\leq \frac{1}{2pL_g - \frac{2\mu_gL_g}{\mu_g + L_g}}$ . Putting everything together, we have  $p\geq \max \{\frac{\mu_g}{\mu_g + L_g},\frac{\mu_g + L_g}{4L_g}\}$ , and
$$
\beta_ {b, 0} \leq \min \{\frac {2}{\mu_ {g} + L _ {g}}, \frac {\mu_ {g} + L _ {g}}{2 \mu_ {g} L _ {g}}, \frac {1}{2 p L g - \frac {2 \mu_ {g} L _ {g}}{\mu_ {g} + L _ {g}}} \}.
$$
A.2 Single-level Convex Proofs
A.2.1 Proof of Theorem 1
The proof of Theorem 1 is similar to the standard proof of decaying-step SGD (GD) that can be found in e.g. Beck [1]. Here, we give the proof for completeness.
$$
\begin{array}{l} \left\| x ^ {k + 1} - x ^ {*} \right\| ^ {2} = \left\| x ^ {k} - x ^ {*} \right\| ^ {2} - 2 \gamma_ {k} \langle x ^ {k} - x ^ {*}, \nabla f _ {i} \left(x ^ {k}\right) \rangle + \gamma_ {k} ^ {2} \left\| \nabla f _ {i} \left(x ^ {k}\right) \right\| ^ {2} \\ \stackrel {(a)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \gamma_ {k} \left(f _ {i} \left(x ^ {k}\right) - f _ {i} \left(x ^ {*}\right)\right) + \gamma_ {k} ^ {2} \| \nabla f _ {i} \left(x ^ {k}\right) \| ^ {2} \\ \stackrel {(b)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \gamma_ {k} \left(f _ {i} \left(x ^ {k}\right) - f _ {i} \left(x ^ {*}\right)\right) + \gamma_ {b, k} ^ {2} G ^ {2}, \\ \end{array}
$$
where (a) is by individual convexity of  $f_{i}$ , and (b) is by Assumption 2. Take conditional expectation and assume that  $\gamma_{k}$  is independent of sample  $k$ :
$$
\begin{array}{l} \mathbb {E} [ \| x ^ {k + 1} - x ^ {*} \| ^ {2} | x ^ {k} ] \stackrel {(c)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - 2 E [ \gamma_ {k} | x ^ {k} ] (f (x ^ {k}) - f (x ^ {*})) + \gamma_ {b, k} ^ {2} G ^ {2} \\ \stackrel {(d)} {\leq} \| x ^ {k} - x ^ {*} \| ^ {2} - 2 \gamma_ {l, k} (f (x ^ {k}) - f (x ^ {*})) + \gamma_ {b, k} ^ {2} G ^ {2}, \\ \end{array}
$$
where (c) is by independence of  $\gamma_{k}$  and  $x^{k}$ , and (d) is because  $\gamma_{l,k} \leq \gamma_{k}, \forall k$ . Take total expectation and rearrange