SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similar to the single-level Adam case, the matrix  $G_{k}$  in the bi-level setting is defined as  $G_{k} = (\beta_{2}G_{k - 1} + (1 - \beta_{2})\mathrm{diag}(h_{f}^{k}h_{f}^{kT})) / (1 - \beta_{2}^{k})$  [25, 43]. Moreover, BiSLS-Adam takes the following steps for updating the variable  $x$ :  $x^{k + 1} = x^k -\alpha_kA_k^{-1}m_k$  where  $m^{k + 1} = \beta_1m^k -(1 - \beta_1)h_f^k$  . The details are given in Algorithms 1 and 2. We denote the search starting point for the upper-level as  $\alpha_{b,k}$  at iteration  $k$  , and denote it as  $\beta_{b,k}^{t}$  at step  $t$  within iteration  $k$  for the lower-level. We remark the following key benefits
Figure 4: Results on data distillation experiments adapted from Lorraine et al. [30] (see Sec 4 for details). We compare BiSPS and decaying-step SGD for different values of  $\alpha_{b,0}$  where Hessian inverse in (2) is computed based on the Identity matrix (left) or Neumann series (right). The lower-level learning rate is fixed at  $10^{-4}$ .
of resetting  $\alpha_{b,k}$  and  $\beta_{b,k}^{\bar{t}}$  (by using Algorithm 2) to larger values with reference to  $\alpha_{k}$  and  $\beta_{k}^{t}$  (respectively) at each step: (1) we avoid always searching from  $\alpha_{b,0}$  or  $\beta_{b,0}^{0}$ , thus reducing computation cost and (2) preserving an overall non-increasing (not necessarily monotonic) trend for  $\alpha_{b,k}$  and  $\beta_{b,k}^{t}$  (improving training stability). We found that different values of  $\eta$  all give good empirical performances (see Appendix B). The key algorithmic challenge we are facing is that during the backtracking process, for any candidate  $\alpha_{k}$ , we need to compute  $\hat{x}^{k} := x^{k} - \alpha_{k}h_{f}^{k}$  and approximate  $y^{*}(\hat{x}^{k})$  with  $\hat{y}^{k + 1}$  (see Algorithm 1). To limit the cost induced by this nested loop, we limit the number of steps to obtain  $\hat{y}^{k + 1}$  to be 1.
Moreover,  $\delta$  in (14) plays the role of a safeguard that ensures a step size can be found. We set  $\delta$  to be small to avoid finding unrealistically large learning rates while tolerating some error in the
hypergradient estimation (see Appendix B for experiments on the sensitivity of  $\delta$ ). In practice, we found that simply setting  $\delta = 0$  works well. In Figure 5a, we observe that BiSLS-Adam outperforms fine-tuned Adam or SGD. Surprisingly, its training is stable even when the search starting point  $\alpha_{b,0}$  is 5 orders of magnitude larger than a fine-tuned learning rate  $(\mathcal{O}(10^{-4}))$ . Importantly, BiSLS-Adam finds large upper and lower-level learning rates in early phase (see Figure 5b, 5c) for different values of  $\alpha_{b,0}$  and  $\beta_{b,0}$  that span 3 orders of magnitudes. Interestingly, the learning rates naturally decay with training (also see Figure 6c and 6d). In essence, BiSLS is a truly adaptive (no knowledge of initialization required) and robust (different initialization works) method that finds large  $\alpha$  and  $\beta$  without tuning. In the next section, we give the convergence results of the envelope-type step size.
3 Convergence Results
3.1 Envelope-type step size for single-level optimization
We first state the assumptions, which are standard in the literature, that will be used for analyzing single-level problems. Assumption 1 is on the Lipschitz continuity of  $f$  and  $f_{i}$  in Problem 4.
Assumption 1. The individual function  $f_{i}$  is convex and  $L_{i}$ -smooth such that  $\| \nabla f_{i}(x) - \nabla f_{i}(x^{\prime}) \| \leq L_{i} \| x - x^{\prime} \|, \forall i, \forall x \in \operatorname{dom} f$  and the overall function  $f$  is  $L$ -smooth. We denote  $L_{\max} \triangleq \max_i L_i$ . Furthermore, we assume there exists  $l_{i}^{*}$  such that  $l_{i}^{*} \leq f_{i}^{*} := \inf_x f_i(x), \forall i$ , and  $f$  is lower bounded by  $f^{*}$  obtained by some  $x^{*}$  such that  $f^{*} = f(x^{*})$ .
The following bounded gradient assumption is also used in the analysis of convex problems [41, 33].
Assumption 2. There exists  $G > 0$  such that  $\| \nabla f_i(x) \|^2 \leq G, \forall i$ .
We first state the theorem for the envelope-type step size defined in (9) for convex functions.
Theorem 1. Suppose Assumption 1, 2 hold, each  $f_{i}$  is convex,  $C = \mathrm{dom}f$ ,  $\gamma_{k}$  is independent of the sample  $\nabla f_{k}(x^{k})$ , and choose  $\gamma_{b,k} = \frac{\gamma_{b,0}}{\sqrt{k + 1}}$ . Then, the envelope-type step size in (9) achieves the following rate,
$$
\mathbb {E} [ f (\bar {x} ^ {K}) - f (x ^ {*}) ] \leq \frac {\| x ^ {0} - x ^ {*} \| ^ {2}}{2 \gamma_ {l , K - 1} K} + \frac {\gamma_ {b , 0} ^ {2} G ^ {2} \log (K)}{2 \gamma_ {l , K - 1} K},
$$
where  $\gamma_{l,K - 1} = \min \{\omega ,\frac{\gamma_{b,0}}{\sqrt{K}}\}$  and  $\bar{x}^K = \frac{1}{K}\sum_{k = 0}^{K}x^k$
We were not able to give a convergence result that uses the same sample for computing the step size and the gradient. However, we empirically observe that the performance is similar when using either one or two independent samples per iteration (see Figure 2 and Appendix B). When two independent samples  $i_k$  and  $j_k$  are used per iteration, the first computes the gradient sample  $\nabla f_{i_k}(x^k)$ , and the other computes the step-size  $\gamma_k$ . For example, for SPSB this gives  $\gamma_k = \min \{ \frac{f_{j_k}(x^k) - l_{j_k}^*}{c_k \| \nabla f_{j_k}(x^k) \|^2}, \gamma_{b,k} \}$ . This type of assumption has been used in several other works for analyzing adaptive step sizes [27, 44, 29]. Under this assumption, we specialize the results of Theorem 1 to SPSB and SLSB, where  $\gamma_{l,K-1} = \min \{ \frac{1}{2cL_{\max}}, \frac{\gamma_{b,0}}{\sqrt{K}} \}$  and  $\gamma_{l,K-1} = \min \{ \frac{2(1-\bar{c})}{L_{\max}}, \frac{\gamma_{b,0}}{\sqrt{K}} \}$  respectively. Concretely, for  $K \geq \gamma_{b,0}^2 L_{\max}^2$ , SLSB and SPSB with  $\gamma_{b,k} = \frac{\gamma_{b,0}}{\sqrt{k+1}}$  and  $c = \bar{c} = \frac{1}{2}$  achieve the following rate:  $\mathbb{E}[f(\bar{x}^K) - f(x^*)] \leq \frac{\| x^0 - x^*\|^2}{2\gamma_{b,0}\sqrt{K}} + \frac{\gamma_{b,0}G^2\log(K)}{2\sqrt{K}}$ . Next, we state the result for the envelop-type step size when  $f$  is  $\mu$ -strongly convex.
Theorem 2. Suppose a  $\mu$ -strongly convex function  $f$  satisfying Assumptions 1 and 2, assume  $\mathcal{C}$  is a closed and convex set, and  $\gamma_{k}$  is independent of the sample  $\nabla f_{k}(x^{k})$ . Then an envelope-type step size as in (9) with  $\gamma_{b,k} = \frac{\gamma_{b,0}}{k + 1}$ ,  $\gamma_{b,0} \geq \frac{1}{\mu}$ , and  $\omega \mu < 1$  achieves the following rate
$$
\mathbb {E} [ f (\bar {x} _ {K}) - f (x ^ {*}) ] \leq \frac {\mu k _ {0}}{2 (K - k _ {0})} \big (e ^ {- k _ {0} \mu \omega} \| x _ {0} - x ^ {*} \| ^ {2} + \gamma_ {b, 0} ^ {2} G ^ {2} \big) + \frac {\gamma_ {b , 0} G ^ {2} \log K}{2 (K - k _ {0})},
$$
where  $\bar{x}_K = \frac{1}{K - k_0}\sum_{k = k_0}^{K - 1}x^k$  and  $k_{0} = \max \{1,\lceil \gamma_{b,0} / \omega \rceil -1\}$
We can again apply the result of Theorem 2 to SPSB and SLSB with  $\gamma_{b,k} = \frac{\gamma_{b,0}}{k + 1}$ ,  $\gamma_{b,0} \geq \frac{1}{\mu}$ ,  $\omega = 1 / L_{\max}$ , and  $c = \bar{c} = \frac{1}{2}$  to get an explicit rate:  $\mathbb{E}[f(\bar{x}_K) - f(x^*)] \leq \frac{\mu k_0}{2(K - k_0)} \left( e^{\frac{-k_0 \mu}{L_{\max}}} \| x_0 - x^* \|^2 + \gamma_{b,0}^2 G^2 \right) + \frac{\gamma_{b,0} G^2 \log K}{2(K - k_0)}$ , where  $k_0 = \max \{1, \lceil \gamma_{b,0} L_{\max} \rceil - 1\}$ .
Remark 1. Under the envelop-type step size framework and the assumption of two independent samples, SLSB and SPSB share the same convergence rates of  $\mathcal{O}\left(\frac{1}{\sqrt{K}}\right)$  and  $\mathcal{O}\left(\frac{1}{K}\right)$  as SGD with decaying step-size for convex and strongly-convex losses respectively. For the latter, a projection operation is required to stay in the closed and convex set  $\mathcal{C}$ . These rates are not surprising because of the structure of the envelope step-size in (9). Indeed, the proof is similar to the standard proof of analogous rate for SGD with decaying step-size. Nonetheless, we include it here for completeness.
3.2 Envelope-type step size for bi-level optimization
We start with recalling standard assumptions in BO [22, 19, 15, 4]. We denote  $z = [x; y]$  and recall the bi-level problem in (1). The first assumption is on the lower-level objective  $g$ .
Assumption 3. The function  $g(x, y)$  is  $\mu_g$  strongly convex in  $y$  for any given  $x$ . Moreover,  $\nabla g$  is Lipschitz continuous:  $\| \nabla g(x_1, y_1) - \nabla g(x_2, y_2) \| \leq L_g \| z_1 - z_2 \|$  (also assume that this holds true for each sampled function  $g(x, y; \psi)$ ), and  $\nabla^2 g$  is Lipschitz continuous:  $\| \nabla^2 g(x_1, y_1) - \nabla^2 g(x_2, y_2) \| \leq L_G \| z_1 - z_2 \|$ . We further assume that  $\| \nabla_{xy}^2 g(x, y) \| \leq C_g$ , and the condition number is defined as  $\kappa = \frac{L_g}{\mu_g}$ .
Next, we state the assumptions on the upper objective  $f$ .
Assumption 4. The function  $f$  and its gradients are Lipschitz continuous. That is:  $\| f(x_1, y_1) - f(x_2, y_2) \| \leq L_1 \| z_1 - z_2 \|$  and  $\| \nabla f(x_1, y_1) - \nabla f(x_2, y_2) \| \leq L_{f,1} \| z_1 - z_2 \|$ . We also assume that  $\| \nabla_y f(x, y) \| \leq C_f$ .
Furthermore, we make the following standard assumptions on the estimates of  $\nabla f$ ,  $\nabla g$ , and  $\nabla^2 g$ .
Assumption 5. The stochastic gradients are unbiased:  $\mathbb{E}_{\phi}[\nabla f(x,y;\phi)] = \nabla f(x,y)$ ,  $\mathbb{E}_{\psi}[\nabla g(x,y;\psi)] = \nabla g(x,y)$ , and  $\mathbb{E}_{\psi}[\nabla^2 g(x,y;\psi)] = \nabla^2 g(x,y)$ . The variances of  $\nabla f(x,y;\phi)$  and  $\nabla^2 g(x,y;\psi)$  are bounded:  $\mathbb{E}_{\phi}[\| \nabla f(x,y;\phi) - \nabla f(x,y)\|^2] \leq \sigma_f^2$  and  $\mathbb{E}_{\psi}[\| \nabla^2 g(x,y;\psi) - \nabla^2 g(x,y)\|^2] \leq \sigma_G^2$ .
Finally, we introduce the bounded optimal function value assumption in (15), which is used specifically for analyzing step size of the form (11) in the bi-level setting:
$$
\mathbb {E} _ {\psi} \left[ g \left(x, y ^ {*} (x); \psi\right) - g \left(x, y _ {x, \psi} ^ {*}; \psi\right) \right] \leq \sigma_ {g} ^ {2}, \forall x, \tag {15}
$$
$$
\mathbb {E} \left[ \| \nabla_ {y} g (x, y) - \nabla_ {y} g (x, y; \phi) \| ^ {2} \right] \leq \sigma_ {g} ^ {2}, \forall x, y, \tag {16}
$$
where  $y^{*}(x) = \min_{y} g(x, y)$  and  $y_{x,\psi}^{*} = \min_{y} g(x, y; \psi)$  for a given  $x$  (recall that at any iteration  $k$ , the lower-level steps in BiSPS are  $\mathrm{SPS}_{\max}$  with an upper bound  $\beta_{b,k}$ ; furthermore,  $\beta_{b,k}$  is non-increasing w.r.t. upper iteration  $k$ ). The one-variable analogous assumption of (15) has been used in the analysis of  $\mathrm{SPS}_{\max}$  [29]. Here, we extend it to a two-variable function. Unlike the bounded variance assumption (16), which needs to hold true for all  $x$  and  $y$ , we require (15) to hold at  $y^{*}(x)$  for any given  $x$ . As mentioned previously, the closed form solution  $y^{*}(x)$  is difficult to obtain. Thus, we define the following expression by replacing  $y^{*}(x)$  with  $y$  in (2):
$$
\bar {\nabla} f (x, y) = \nabla_ {x} f (x, y) - \nabla_ {x y} ^ {2} g (x, y) [ \nabla_ {y y} ^ {2} g (x, y) ] ^ {- 1} \nabla_ {y} f (x, y). \tag {17}
$$
A stochastic Neumann series in (13) approximates (17) with  $x$  and  $y$  being  $x^{k}$  and  $y^{k + 1}$  (respectively), also recall that  $y^{k + 1}$  is an approximation of  $y^{*}(x^{k})$  by running  $T$  lower-level SGD steps to minimize  $g$  w.r.t.  $y$  for a fixed  $x^k$ . Based on Assumptions 3, 4, and 5, we have the following results to be used in the analysis [15]:  $\| \nabla F(x_1) - \nabla F(x_2)\| \leq L_F\| x_1 - x_2\|$ ,  $\| y^{*}(x_{1}) - y^{*}(x_{2})\| \leq L_{y}\| x_{1} - x_{2}\|$ , and  $\| \bar{\nabla} f(x,y^{*}(x)) - \bar{\nabla} f(x,y)\| \leq L_f\| y^* (x) - y\|$ . Furthermore, the bias in the stochastic hypergradient in (13) (denoted as  $B$ ) decays exponentially with  $N$  and its variance is bounded, i.e.  $\mathbb{E}[\| h_f^k -\mathbb{E}[h_f^k ]\| ^2 ]\leq \tilde{\sigma}_f^2$  (see Appendix A for details) [19].
Now, we state our main theorem based on step size of the form (10) and (11).
Figure 6: Validation loss (a) and accuracy (b) against iterations. (a) Comparisons between whether to use or not use line-search at the upper or lower level; (b) Generalization performance of BiSLS-Adam/SGD and fine-tuned Adam/SGD; (c) Validation loss against iterations for different values of  $\alpha_{b,0}$  ( $\beta_{b,0}$  fixed at 100). (d) Same plot as (c) but for different values of  $\beta_{b,0}$  ( $\alpha_{b,0}$  fixed at 10).
(b) Validation Accuracy
(c) Different  $\alpha_{b,0}$
(d) Different  $\beta_{b,0}$
Theorem 3. Suppose  $f$  and  $g$  satisfy Assumptions 3, 4, and 5, learning-rate upper bounds  $\alpha_{b,k} = \frac{\alpha_{b,0}}{\sqrt{k + 1}}$  and  $\alpha_{l,k} = \frac{\alpha_{l,0}}{\sqrt{k + 1}}$  with  $\alpha_{b,0}$  and  $\alpha_{l,0}$  satisfying  $\frac{1}{L_F + 4L_y^2} \geq \frac{\alpha_{b,0}^2}{\alpha_{l,0}}$  and  $\alpha_{l,0} \leq \alpha_{b,0}$ . Further assume that  $\alpha_k$  is independent of the stochastic hypergradient  $h_f^k$ , and each sampled function  $g(x,y;\psi)$  is convex. Then under the Assumption (15) with  $p \geq \frac{1}{2}$ ,  $C_k = \min \left\{\frac{1}{2pL_g}, \beta_{b,k}\right\}$ ,  $T \geq \frac{\log(\alpha_{b,0}L_f^2 + 2)}{-\log(1 - \mu_gC_K - 1)}$ , and  $\beta_{b,k} = \frac{\beta_{b,0}}{k + 1}$ , BiSPS achieves the rate:
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \| \nabla F (x ^ {k}) \| ^ {2} ] \leq \tilde {\mathcal {O}} \left(\frac {\kappa^ {3}}{\sqrt {K}} + \frac {\kappa^ {2} \log K}{\sqrt {K}}\right). \tag {18}
$$
Remark 2. We further give the convergence result under the bounded variance assumption (16) in Appendix A. Theorem 3 shows that BiSPS matches the optimal rate of SGD up to a logarithmic factor without a growing batch size. We notice that the assumption (15) largely simplifies the expression on  $T$  and does not require an explicit upper bound on  $\beta_{b,0}$ . As in the single-level case, whether using one sample or two samples (which makes upper-level step-size independent of gradient) gives similar empirical performances (see Appendix B). Note that the independence assumption is only needed for the upper-level. Thus, the two-sample requirement of theorem does not apply to the lower-level problem. This is useful from computational standpoint as typical bi-level algorithms run multiple lower-level updates for each upper-level iteration.
4 Additional Hyper-Representation and Data Distillation Experiments
Hyper-representation learning: The experiments are performed on MNIST dataset using LeNet [26, 42]. We use conjugate gradient method for solving system of equations when computing the hypergradient [17]. The upper and lower-level objectives are to optimize the embedding layers and the classifier (i.e. the last layer of the neural net), respectively (see Appendix B for details). For constant-step SGD and Adam, we tune the lower-level learning rate  $\beta \in \{10.0, 5.0, 1.0, 0.5, 0.1, 0.05, 0.01\}$ . For the upper-level learning rate, we tune  $\alpha \in \{0.001, 0.0025, 0.005, 0.01, 0.05, 0.1\}$  for SGD, and  $\alpha \in \{10^{-5}, 5 \cdot 10^{-5}, 10^{-4}, 5 \cdot 10^{-4}, 0.001, 0.01\}$  for Adam (recall that  $\delta$  in (14) is set to 0). Based on the results of Figure 6, we make the following key observations: ① line-search at the upper-level is essential for achieving the optimal performance (Figure 6a); ② BiSLS-Adam/SGD not only converges fast but also generalizes well (Figure 6b); ③ BiSLS-Adam/SGD is highly robust to search starting points  $\alpha_{b,0}$  and  $\beta_{b,0}$  (Figure 6c, 6d). It addresses the fundamental question of how to tune  $\alpha$  and  $\beta$  in bi-level optimization (see Appendix B for additional results on search cost).
Search cost and run-time comparison: The search cost of reset option 1 in Algorithm 2 for BiSLS-SGD and BiSLS-Adam is  $89 \pm 15$  and  $115 \pm 16$  per iteration, respectively, measured in the number of condition-checking via (14). These costs are significantly reduced when option 3 is used. Concretely, Figure 7b suggests that choosing  $\eta = 2$  in option 3 results in  $\sim 9$  evaluations of (14) per iteration for both BiSLS-Adam and BiSLS-SGD. Choosing  $\eta = 1$  in reset option 3 is equivalent to reset option 2, which further cuts down the cost to  $\sim 4$ . However, this choice forces the learning rate to be monotonically-decreasing, which leads to a slower convergence compared to other  $\eta$ 's (see Appendix B for more details). Besides this, Figure 7a shows that a single step for
(a) Approximation of  $y^{*}(x)$
(b) Search cost
(c) Run time
(a) Neumann
(b) Identity
Figure 7: (a) Validation accuracy and run time (in seconds) against different number of gradient steps for approximating  $y^{*}(x)$  in (14). (b) Search cost (i.e. number of evaluations of (14) per iteration) against different  $\eta$ 's for reset option 3 in Algorithm 2. (c) Run time (in seconds) of BiSLS-Adam and fine-tuned Adam to reach  $85\%$  validation accuracy against different number of conjugate gradient steps for computing the hypergradient.
Figure 8: (a)(b): Comparison between BiSLS-SGD and Adam/SGD for Data Distillation on MNIST dataset. Validation loss plotted against iterations. (a) Hypergradient computed using Neumann series; (b) Inverse Hessian in (2) treated as the Identity [30] when computing the hypergradient; (c) Distilled MNIST images after 3000 iterations of BiSLS-SGD.
(c) Distilled MNIST Images
approximating the  $y^{*}(x)$  in (14) is sufficient, considering the trade-off between performance gain and extra computation overhead introduced. Most importantly, BiSPS-Adam takes less time to reach a threshold test accuracy than fine-tuned Adam (Figure 7c) due to a suitable (and potentially large) learning rate found using (14) (note that Figure 7c excludes the cost of tuning the learning rate of Adam).
Data distillation: The goal of data distillation is to generate a small set of synthetic data from an original dataset that preserves the performance of a model when trained using the generated data [46, 50]. We adapted the experiment set up from Lorraine et al. [30] to distill MNIST digits. We present the results in Figure 8, where we observe that BiSLS-SGD converges significantly faster than fine-tuned Adam or SGD, and generate realistic MNIST images (see Appendix B for more results).
In this work, we have given simple alternatives to SLS and SPS that show good empirical performance in non-interpolating scenario without requiring the step size to be monotonic. We unify their analysis based on a simplified envelope-type step size, and extend the analysis to the bi-level setting while designing a SPS-based bi-level algorithm. In the end, we propose bi-level line-search algorithm BiSLS-Adam/SGD that is empirically truly robust and adaptive to learning rate initialization. Our work opens several possible future directions. Given the superior performance of BiSLS, we prioritize an analysis of its convergence rates. The difficulty stems from: (a) the bias in hypergradient estimation; (b) the dual updates in  $x$  and  $y^{*}(x)$  (incurring nested loop structures); (c) the error in estimating  $y^{*}(x)$ . On single-level optimization, we remark as an important direction to relax the two-sample assumption on SPSB /SLSB. Ultimately, we hope to promote further research on bi-level optimization algorithms with minimal tuning.
Acknowledgement This work is funded partially by the NSERC Discovery Grant RGPIN-2021-03677, the NSERC Discovery Grant RGPIN-2022-03669, and the Canada CIFAR AI Chair Program.
[46] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020.
[47] Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes, 2021.
[48] Jorge Nocedal Stephen J Wright. Numerical optimization, 2006.
[49] Junjie Yang, Kaiyi Ji, and Yingbin Liang. Provably faster algorithms for bilevel optimization, 2021.
[50] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review, 2023.
[51] Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning, pages 26693-26712. PMLR, 2022.
[52] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization, 2023.
[53] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719, 2022.
A Proofs of Theorems and Additional Convergence Results
A.1 Useful Lemmas
Lemma 1 provides more details on the envelope structure of SPSB and SLSB given in (10) and (11). The lower bound in (19) will also be used in Lemma 5 (for bounding the term  $\| y^{k + 1} - y^* (x^k)\|^2$ ).
Lemma 1. Under the Assumption 1, we have the following:
$$
S P S B: \quad \min  \left\{\frac {1}{2 c L _ {\max }}, \gamma_ {b, k} \right\} \leq \gamma_ {k} = \min  \left\{\frac {f _ {i _ {k}} \left(x ^ {k}\right) - l _ {i _ {k}} ^ {*}}{c \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, \gamma_ {b, k} \right\}, \quad 0 <   c, \tag {19}
$$
$$
S L S B: \quad \min  \left\{\frac {2 (1 - \bar {c})}{L _ {\max }}, \gamma_ {b, k} \right\} \leq \gamma_ {k} \leq \min  \left\{\frac {f _ {i _ {k}} \left(x ^ {k}\right) - l _ {i _ {k}} ^ {*}}{\bar {c} \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, \gamma_ {b, k} \right\}, \quad 0 <   \bar {c} <   1. \tag {20}
$$
Proof. The bounds in (19) have been shown in [29, 36]. For (20), the first part of the inequality has been shown in [44]. For the second part, recall the Armijo condition (14):
$$
f _ {i _ {k}} (x _ {k} - \gamma_ {k} \nabla f _ {i _ {k}} (x _ {k})) \leq f _ {i _ {k}} (x _ {k}) - \bar {c} \cdot \gamma_ {k} \| \nabla f _ {i _ {k}} (x _ {k}) \| ^ {2}, 0 <   \bar {c} <   1.
$$
We can then rearrange this to obtain
$$
\gamma_ {k} \leq \frac {f _ {i _ {k}} \left(x _ {k}\right) - f _ {i _ {k}} \left(x _ {k} - \gamma_ {k} \nabla f _ {i _ {k}} \left(x _ {k}\right)\right)}{\bar {c} \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}} \leq \frac {f _ {i _ {k}} \left(x _ {k}\right) - f _ {i _ {k}} ^ {*}}{\bar {c} \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}} \leq \frac {f _ {i _ {k}} \left(x _ {k}\right) - l _ {i _ {k}} ^ {*}}{\bar {c} \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, \tag {21}
$$
where  $l_{i_k}^*$  is any lower bound for  $f_{i_k}^*$ . Also recall that  $\gamma_{b,k}$  is the search starting point at iteration  $k$ , hence (20) holds for SLSB.
Lemma 2 gives the expressions for the constants  $L_{F}, L_{y}$ , and  $L_{f}$ . The proof can be found in [15, Lem 2.2].
Lemma 2. Under Assumptions 3 and 4, we have the following:
$$
\| \nabla F (x _ {1}) - \nabla F (x _ {2}) \| \leq L _ {F} \| x _ {1} - x _ {2} \|,
$$
$$
\left\| y ^ {*} \left(x _ {1}\right) - y ^ {*} \left(x _ {2}\right) \right\| \leq L _ {y} \left\| x _ {1} - x _ {2} \right\|,
$$
$$
\| \bar {\nabla} f (x, y ^ {*} (x)) - \bar {\nabla} f (x, y) \| \leq L _ {f} \| y ^ {*} (x) - y \|,
$$
$$
L _ {f} = L _ {f, 1} + \frac {L _ {f , 1} L _ {g}}{\mu_ {g}} + \frac {L _ {1}}{\mu_ {g}} (L _ {G} + \frac {L _ {g} L _ {G}}{\mu_ {g}}) \sim \mathcal {O} (\kappa^ {2})
$$
$$
L _ {y} = \frac {L _ {g}}{\mu_ {g}} \sim \mathcal {O} (\kappa)
$$
$$
L _ {F} = L _ {f, 1} + \frac {L _ {g} (L _ {f , 1} + L _ {f})}{\mu_ {g}} + \frac {L _ {1}}{\mu_ {g}} (L _ {G} + \frac {L _ {g} L _ {G}}{\mu_ {g}}) \sim \mathcal {O} (\kappa^ {3})
$$
Lemma 3 is on the bias and variance of the stochastic hypergradient in (13), which has the following form [19]
$$
h _ {f} ^ {k} = \nabla_ {x} f \left(x ^ {k}, y ^ {k + 1}; \phi\right) - \nabla_ {x y} g \left(x ^ {k}, y ^ {k + 1}; \psi_ {0}\right) \left[ \frac {N}{L _ {g}} \prod_ {j = 1} ^ {\bar {N}} \left(I - \nabla_ {y y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}; \psi_ {j}\right)\right) \right] \nabla_ {y} f \left(x ^ {k}, y ^ {k + 1}; \phi\right). \tag {22}
$$