SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chen Fan<sup>1</sup>, Gaspard Chone-Ducasse<sup>2</sup>, Mark Schmidt<sup>1,3</sup>, and Christos Thrampoulidis<sup>1</sup>
<sup>1</sup>University of British Columbia  
<sup>2</sup>Ecole Normale Supérieure  
<sup>3</sup>Canada CIFAR AI Chair (Amii)
The popularity of bi-level optimization (BO) in deep learning has spurred a growing interest in studying gradient-based BO algorithms. However, existing algorithms involve two coupled learning rates that can be affected by approximation errors when computing hypergradients, making careful fine-tuning necessary to ensure fast convergence. To alleviate this issue, we investigate the use of recently proposed adaptive step-size methods, namely stochastic line search (SLS) and stochastic Polyak step size (SPS), for computing both the upper and lower-level learning rates. First, we revisit the use of SLS and SPS in single-level optimization without the additional interpolation condition that is typically assumed in prior works. For such settings, we investigate new variants of SLS and SPS that improve upon existing suggestions in the literature and are simpler to implement. Importantly, these two variants can be seen as special instances of general family of methods with an envelope-type step-size. This unified envelope strategy allows for the extension of the algorithms and their convergence guarantees to BO settings. Finally, our extensive experiments demonstrate that the new algorithms, which are available in both SGD and Adam versions, can find large learning rates with minimal tuning and converge faster than corresponding vanilla SGD or Adam BO algorithms that require fine-tuning.
Bi-level optimization (BO) has found its applications in various fields of machine learning such as hyperparameter optimization [14, 17, 30, 40], adversarial training [51], data distillation [2, 53], neural architecture search [28, 39], neural-network pruning [52], and meta-learning [13, 37, 11]. Specifically, BO is used widely for problems that exhibit a hierarchical structure of the following form:
$$
\min  _ {x \in X} F (x) = \mathbb {E} _ {\phi} [ f (x, y ^ {*} (x); \phi) ] \quad \text {s . t .} \quad y ^ {*} (x) = \underset {y \in Y} {\operatorname {a r g m i n}} \mathbb {E} _ {\psi} [ g (x, y; \psi) ]. \tag {1}
$$
Here, the solution to the lower-level objective  $g$  becomes the input to the upper-level objective  $f$ , and in (1) the upper-level variable  $x$  is fixed when optimizing the lower-level variable  $y$ . To solve such bi-level problems using gradient-based methods requires computing the hypergradient of  $F$ , which based on the chain rule is given as [15]:
$$
\nabla F (x) = \nabla_ {x} f (x, y ^ {*} (x)) - \nabla_ {x y} ^ {2} g (x, y ^ {*} (x)) [ \nabla_ {y y} ^ {2} g (x, y ^ {*} (x)) ] ^ {- 1} \nabla_ {y} f (x, y ^ {*} (x)). \qquad (2)
$$
In practice, the closed-form solution  $y^{*}(x)$  can be difficult to obtain, and one strategy is to run a few steps of (stochastic) gradient descent on  $g$  with respect to  $y$  to get an approximation  $\bar{y}$ , and
use  $\bar{y}$  in places of  $y^{*}(x)$ . We denote the stochastic hypergradient based on  $\bar{y}$  as  $h_f(x,\bar{y})$  and the stochastic gradient of  $g$  with respect to  $y$  as  $h_g$ . This leads to a general gradient-based framework for solving bi-level optimization [15, 19, 4]. At each iteration  $k$ , run T (can be one or more) steps of SGD on  $y$  with a step size  $\beta$ ,  $y^{k,t+1} = y^{k,t} - \beta h_g^{k,t}$ , then run one step on  $x$  using the approximated hypergradient:
$$
x ^ {k + 1} = x ^ {k} - \alpha h _ {f} \left(x ^ {k}, y ^ {k + 1}\right), \quad \text {w h e r e} \quad y ^ {k + 1} = y ^ {k, T}. \tag {3}
$$
Based on this framework, a series of stochastic algorithms have been developed to achieve the optimal or near-optimal rate of their deterministic counterparts [7, 8]. These algorithms can be broadly divided into single-loop ( $T = 1$ ) or double-loop ( $T > 1$ ) categories [23].
Unlike minimizing the single-level finite-sum (convex) problem
$$
F (x) := \min  _ {x \in \mathcal {C}} \frac {1}{N} \sum_ {i = 1} ^ {N} f _ {i} (x), \tag {4}
$$
where only one learning rate is involved when using SGD, bi-level optimization involves tuning both the lower and upper-level learning rates  $(\beta$  and  $\alpha$  respectively). This poses a significant challenge due to the potential correlation between these learning rates [19]. Thus, as
Figure 1: Results based on hyper-representation learning task (see Sec 4 for details). Validation loss against upper-level iterations for different values of  $\beta$  (left,  $\alpha = 0.005$ ) and  $\alpha$  (right,  $\beta = 0.01$ ). Unless carefully tuned, vanilla SGD-based methods for BO are very unstable.
observed in Figure 1, algorithm divergence can occur when either  $\alpha$  or  $\beta$  is large. While there is considerable literature on achieving faster rates in bi-level optimization [24, 5, 7, 8], only a few studies have focused on stabilizing its training and automating the tuning of  $\alpha$  and  $\beta$ .
This work addresses the question: Is it possible to utilize large  $\alpha$  and  $\beta$  without manual tuning? In doing so, we explore the use of stochastic adaptive-step size methods, namely stochastic Polyak step size (SPS) and stochastic line search (SLS), which utilize gradient information to adjust the learning rate at each iteration [44, 29]. These methods have been demonstrated to perform well in interpolation settings with strong convergence guarantees [44, 29]. However, applying them to bi-level optimization (BO) introduces significant challenges, as follows. BO requires tuning two correlated learning rates (for lower and upper-level). The bias in the stochastic approximation of the hypergradient complicates the practical performance and convergence analysis of SLS and SPS. Other algorithmic challenges arise for both algorithms. For SLS, verifying the stochastic Armijo condition at the upper-level involves evaluating the objective at a new  $(x,y^{*}(x))$  pair, while  $y^{*}(x)$  is only approximately known; For SPS, most existing variants guarantee good performance only in interpolating settings, which are typically not satisfied for the upper-level objective in BO [22]. Before presenting our solutions to the challenges above in Sec 2, we first review the most closely related literature.
Gradient-Based Bi-level Optimization Penalty or gradient-based approaches have been used for solving bi-level optimization problems [10, 45, 21]. Here we focus our discussions on stochastic gradient-based methods as they are closely related to this work. For double-loop algorithms, an early work (BSA) by Ghadimi and Wang [15] has derived the sample complexity of  $\phi$  in achieving an  $\epsilon$ -stationary point to be  $\mathcal{O}(\epsilon^{-2})$ , but require the number of lower-level steps to satisfy  $T \sim \mathcal{O}(\epsilon^{-1})$ . Using a warm start strategy (stocBiO), Ji et al. [22] removed this requirement on  $T$ . However, to achieve the same sample complexity, the batch size of stocBiO grows as  $\mathcal{O}(\epsilon^{-1})$ . Chen et al. [4] removed both requirements on  $T$  and batch size by using the smoothness properties of  $y^{*}(x)$  and setting the step sizes  $\alpha$  and  $\beta$  at the same scale. For single-loop algorithms, a pioneering work by Hong et al. [19] gave a sample complexity of  $\mathcal{O}(\epsilon^{-2.5})$ , provided  $\alpha$  and  $\beta$  are on two different scales (TTSA). By making corrections to the  $y$  variable update (STABLE), Chen et al. [5] improved the rate to  $\mathcal{O}(\epsilon^{-2})$ . However, extra matrix projections required by STABLE can incur high computation cost [5, 4]. By incorporating momentum into the updates of  $x$  and  $y$  (SUSTAIN), Khanduri et al.
Figure 2: Experiments on quadratic functions adapted from [29]. The objective is the sum of two-dimensional functions  $f_{i} = \frac{1}{2}(x - x_{i}^{*})^{T}H_{i}(x - x_{i}^{*})$ , where  $H_{i}$  is positive definite and  $i = 1,2$  (see Appendix B for more details). From left to right, we show: the objective value, distance to optimum, step size, and iterate trajectories.
[24] further improved the rate to  $\mathcal{O}(\epsilon^{-1.5})$  [6]. Besides these single or double-loop algorithms, a series of works have drawn ideas from variance reduction to achieve faster convergence rates for BO. For example, Yang et al. [49] designed the VRBO algorithm based on SPIDER [12]. Dagréou et al. [7, 8] designed the SABA and SRBA algorithms based on SAGA and SARAH respectively, and demonstrate that they can achieve the optimal rate of  $\mathcal{O}(\epsilon^{-1})$  [9, 35]. Huang et al. [20] proposes to use Adam-type step sizes in BO. However, it introduces three sequences of learning rates  $(\alpha_{k},\beta_{k},\eta_{k})$  that require tuning, which limits its practical usage. To our knowledge, none of these works have explicitly addressed the fundamental problem of how to select  $\alpha$  and  $\beta$  in bi-level optimization. In this work, we focus on the alternating SGD framework (T can be 1 or larger), and design efficient algorithms that find large  $\alpha$  and  $\beta$  without tuning, while ensuring the stability of training.
Adaptive Step Size Adaptive step-size such as Adam has found great success in modern machine learning, and different variants have been proposed [25, 38, 47, 31, 32]. Here, we limit our discussions on two adaptive step sizes that are most relevant to this work. The Armijo line search is a classic way for finding step sizes for gradient descent [48]. Vaswani et al. [44] extends it to the stochastic setting (SLS) and demonstrates that the algorithm works well with minimal tuning required under interpolation, where the model fits the data perfectly. This method is adaptive to local smoothness of the objective, which is typically difficult to predict a priori. However, the theoretical guarantee of SLS in the non-interpolating regime is lacking. In fact, the results in Figure 3 suggest that SLS can perform poorly for convex losses when interpolation is not satisfied. Besides SLS, another adaptive method derived from the Polyak step size is proposed by Loizou et al. [29] with the name stochastic Polyak step size (SPS). Loizou et al. [29] further places an upper bound on the step size resulting in the  $\mathrm{SPS}_{\mathrm{max}}$  variant. Similar to SLS, the algorithm performs well when the model is over-parametrized. Without interpolation, the algorithm converges to a neighborhood of the solution whose size depends on this upper
Algorithm 1 BiSLS-Adam/SGD  
Input:  $x^0, y^0, K, T, \delta, \alpha_{b,0}, \beta_{b,0}, \eta, w \in (0,1)$   
Output:  $x$   
1: for  $k = 0, 1, \ldots, K - 1$  do  
2:  $y^{k,0} = y^k$   
3: for  $t = 0, 1, \ldots, T - 1$  do  
4:  $\beta_{b,k}^t \leftarrow \text{reset}(\beta, \beta_{b,0}, \eta, \text{opt}) \triangleright$  see Algorithm 2  
5:  $\beta \gets$  line-search based on (8) starting from  $\beta_{b,k}^t$   
6:  $y^{k,t + 1} = y^{k,t} - \beta h_g^{k,t}$ ,  
7: end for  
8:  $y^{k + 1} = y^{k,T - 1}; \hat{x}^k = x^k; \hat{y}^{k + 1} = y^{k + 1}$   
9:  $\alpha \gets \text{reset}(\alpha, \alpha_{b,0}, \eta, \text{opt})$   
10: while (14) based on  $(\hat{x}^k, \hat{y}^{k + 1}, \alpha, \delta)$  does not hold. do  
11:  $\alpha = \alpha * w$   
12:  $\hat{x}^k = x^k - \alpha h_f(x^k, y^{k + 1})$  or  
13:  $\hat{x}^k = x^k - \alpha A_k^{-1} h_f(x^k, y^{k + 1})$   
14:  $\hat{y}^{k + 1} = y^{k + 1} - \beta h_g(\hat{x}^k, y^{k + 1})$   
15: end while  
16:  $x^{k + 1} = x^k - \alpha h_f^k$   
17: end for
In a later work, Orvieto et al. [36] make the SPS converge to the exact solution by ensuring the step size and its upper bound are both non-increasing (DecSPS). However, enforcing monotonicity may result in the step size being smaller than decaying-step SGD and losing the adaptive features of SPS (see Figure 2, 3). In this work, we propose new versions of SLS and SPS that do not require monotonicity and extend them into the alternating SGD bi-level optimization framework (3).
(a) Minimum train loss vs.  $\gamma_{b,0}$
Figure 3: Binary linear classification on w8a dataset using logistic loss [3]. (a) Minimum train loss of decaying-step SGD and SPSB for different  $\gamma_{b,0}$ 's. (b)(c) Train loss and step size against iterations, respectively. We choose  $c = 1$  and  $\bar{c} = 1$  for  $\mathrm{SPS}_{\mathrm{max}}$  and SLS respectively;  $c_k = \sqrt{k + 1}$  for DecSPS;  $c_k = 1$  and  $\gamma_{b,k} = \frac{\gamma_{b,0}}{\sqrt{k + 1}}$  for SPSB;  $\bar{c} = 0.1$  and  $\gamma_{b,k} = \frac{\gamma_{b,0}}{\sqrt{k + 1}}$  for SLSB;  $\gamma_{b,k} = \frac{\gamma_{b,0}}{\sqrt{k + 1}}$  for decaying-step SGD. For (b) and (c), we set  $\gamma_{b,0} = 1000$  for all algorithms.
(b) Train loss vs. iterations
(c) Step size vs. iterations
2 Summary of Contributions
We discuss our main contributions in this section, which is organized as follows. First, we discuss our variants of SPS and SLS, and unify them under the umbrella of "envelope-type step-size". Then, we extend the envelope-type step size to the bi-level setting. Finally, we discuss our bi-level line-search algorithms based on Adam and SGD.
Converging SPSB and SLSB by Envelope Approach We first propose simple variants of SLS and SPS that converge in the non-interpolating setting while not requiring the step size to be monotonic. To this end, we introduce a new stochastic Polyak step size (SPSB). For comparison, we also recall the step-sizes of  $\mathrm{SPS}_{\mathrm{max}}$  and DecSPS. For all methods, the iterate updates are given as  $x_{k + 1} = x_k - \gamma_k\nabla f_{i_k}(x^k)$  where  $i_k$  is sampled uniformly from  $[n] = \{1,\dots ,n\}$  at each iteration  $k$ . The step-sizes  $\gamma_{k}$  are then defined as follows:
$$
\mathrm {S P S} _ {\max } [ 2 9 ]: \quad \gamma_ {k} = \min  \left\{\frac {f _ {i _ {k}} \left(x ^ {k}\right) - f _ {i _ {k}} ^ {*}}{c \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, \gamma_ {b, 0} \right\} \tag {5}
$$
$$
\text {D e c S P S} [ 3 6 ]: \quad \gamma_ {0} = \bar {\gamma} \quad \gamma_ {k} = \frac {1}{c _ {k}} \min  \left\{\frac {f _ {i _ {k}} \left(x ^ {k}\right) - l _ {i _ {k}} ^ {*}}{\| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, c _ {k - 1} \gamma_ {k - 1} \right\} \quad \forall k \geq 1 \tag {6}
$$
$$
\text {S P S B} (\text {o u r s}): \quad \gamma_ {k} = \min  \left\{\frac {f _ {i _ {k}} \left(x ^ {k}\right) - l _ {i _ {k}} ^ {*}}{c _ {k} \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, \gamma_ {b, k} \right\}, \tag {7}
$$
where  $f_{i}^{*} = \inf_{x}f_{i}(x)$ ,  $\bar{\gamma} = \frac{1}{c_0}\min \left\{\frac{f_{i_0}(x^0) - l_{i_0}^*}{\|\nabla f_{i_0}(x^0)\|^2}, c_0\gamma_{b,0}\right\}$ ,  $c_k$  is non-decreasing,  $\gamma_{b,k}$  is non-increasing, and  $l_{i}^{*} \leq f_{i}^{*}$  is any lower bound.
Unlike  $\mathrm{SPS}_{\mathrm{max}}$  in which  $\gamma_{b,0}$  is a constant, our upper bound  $\gamma_{b,k}$  is non-increasing. Also, unlike DecSPS in which both the step size and the upper bound are non-increasing (this is because  $\gamma_{k} \leq \frac{c_{k-1}}{c_{k}} \gamma_{k-1}$  and  $\min \left\{\frac{1}{2cL_{\mathrm{max}}}, \frac{c_{0} \gamma_{b,0}}{c_{k}}\right\} \leq \gamma_{k} \leq \frac{c_{0} \gamma_{b,0}}{c_{k}}$  [36, Lemma 1]), we simplify the recursive structure and do not require the step-size to be monotonic. As we empirically observe in Figure 3, the step size of DecSPS is similar to that of decaying SGD and in fact can be much smaller. Interestingly, the resulting performance of DecSPS is worse than  $\mathrm{SPS}_{\mathrm{max}}$  despite  $\mathrm{SPS}_{\mathrm{max}}$  eventually becoming unstable once the iterates get closer to the neighborhood of a solution and the step-size naturally behaves erratically. This is not unexpected due to small gradient norms (note the division by gradient-norm in (5)) and dissimilarity between samples in the non-interpolating scenario. Moreover, note that the adaptivity of SPS in the early stage seems to be lost in DecSPS due to the monotonicity of the latter. On the other hand, SPSB not only takes advantage of the large SPS steps that leads to fast convergence, but also stays regularized due to the non-increasing upper bound  $\gamma_{b,k}$  in (19). These observations are further supported by the experiments on quadratic functions given in Figure 2, where we observe the fast convergence of SPSB and the instability of  $\mathrm{SPS}_{\mathrm{max}}$ . Furthermore, SPSB is significantly more robust to the choice of  $\gamma_{b,0}$  than decaying-step SGD as shown by Figure 3a. Motivated by the good practical performance of SPSB, we take a similar approach for SLS. The
SLS proposed and analyzed by Vaswani et al. [44] starts with  $\gamma_{b,0}$  and in each iteration  $k$  finds the largest  $\gamma_k \leq \gamma_{b,0}$  that satisfies:
$$
f _ {i _ {k}} \left(x _ {k} - \gamma_ {k} \nabla f _ {i _ {k}} \left(x _ {k}\right)\right) \leq f _ {i _ {k}} \left(x _ {k}\right) - \bar {c} \cdot \gamma_ {k} \| \nabla f _ {i _ {k}} \left(x _ {k}\right) \| ^ {2}, \quad 0 <   \bar {c} <   1. \tag {8}
$$
To ensure its convergence without interpolation, we replace  $\gamma_{b,0}$  with appropriate non-increasing sequence  $\gamma_{b,k}$ . We name this variant of SLS as SLSB. Interestingly, the empirical performance and step size of SLSB are similar to those of SPSB (see Figure 3). This can be explained by observing that the step sizes of SPSB and SLSB share similar envelope structures, as follows (see Lemma 1 in Appendix A):
$$
\mathrm {S P S B}: \quad \min  \{\frac {1}{2 c L _ {\max }}, \gamma_ {b, k} \} \leq \gamma_ {k} = \min  \{\frac {f _ {i _ {k}} (x ^ {k}) - l _ {i _ {k}} ^ {*}}{c \| \nabla f _ {i _ {k}} (x ^ {k}) \| ^ {2}}, \gamma_ {b, k} \}, \quad 0 <   c,
$$
$$
\text {S L S B}: \quad \min  \left\{\frac {2 (1 - \bar {c})}{L _ {\max }}, \gamma_ {b, k} \right\} \leq \gamma_ {k} \leq \min  \left\{\frac {f _ {i _ {k}} \left(x ^ {k}\right) - l _ {i _ {k}} ^ {*}}{\bar {c} \| \nabla f _ {i _ {k}} \left(x ^ {k}\right) \| ^ {2}}, \gamma_ {b, k} \right\}, \quad 0 <   \bar {c} <   1.
$$
Therefore, we unify their analysis based on the following generic envelope-type step size:
$$
\gamma_ {k} = \min  \left\{\max  \left\{\gamma_ {l, k}, \tilde {\gamma} _ {k} \right\}, \gamma_ {b, k} \right\}, \quad \gamma_ {l, k} = \min  \{w, \gamma_ {b, k} \}, \tag {9}
$$
where  $\omega > 0$ ,  $\gamma_{b,k}$  is non-increasing, and  $\tilde{\gamma}_k$  satisfies  $\gamma_{l,k} := \min \{\omega, \gamma_{b,k}\} \leq \tilde{\gamma}_k \leq \gamma_{b,k}$ . We show that this envelope-type step size converges at a rate  $\mathcal{O}\left(\frac{1}{\sqrt{K}}\right)$  and  $\mathcal{O}\left(\frac{1}{K}\right)$  for convex and strongly-convex losses respectively.
Envelope Step Size for Bi-level Optimization (BiSPS) We extend the use of envelope-type step sizes to the bi-level setting. The step sizes for upper and lower-level objectives of our general envelope-type method are:
Upper:  $\alpha_{k} = \min \{\max \{\alpha_{l,k},\tilde{\alpha}_{k}\} ,\alpha_{b,k}\}$  hence  $\alpha_{l,k}\leq \tilde{\alpha}_k\leq \alpha_{b,k}$  (10)
Lower:  $\beta_{k,t} = \min \{\frac{g(x^k,y^{k,t};\psi) - g(x^k,y_{x^k,\psi}^*;\psi)}{p\|\nabla_yg(x^k,y^{k,t};\psi)\|^2},\beta_{b,k}\}\quad \forall t,$  (11)
where  $y_{x^k,\psi}^*$  is the minimizer of the function  $g(x^k, \cdot; \psi)$ , and  $\alpha_{l,k}, \alpha_{b,k},$  and  $\beta_{b,k}$  are three non-increasing sequences. Note that  $\beta_{b,k}$  is fixed over the lower-level iterations for a given  $k$ , therefore this is equivalent to running  $T$  steps of  $\mathrm{SPS}_{\mathrm{max}}$  to minimize the function  $g$  at each upper iteration  $k$ . However, the decrease in the upper bound  $\beta_{b,k}$  with  $k$  is crucial to guarantee the overall convergence of the algorithm (see Theorem 3). Starting from the general step-size rules in (10) and (11), our bi-level extension of SPS (BiSPS) follows by setting  $\alpha_k$  in the form of SPS computed using stochastic hypergradient  $h_f^k$ . That is,
Algorithm 2 reset
Input:  $p,q,\eta \geq 1$  opt
Output:  $p$
1: if opt = 1 then
2:  $p\gets q$
3: else if opt = 2 then
4:  $p\gets p$
5: else if opt = 3 then
6:  $p\gets \eta \cdot p$
7: end if
$$
\bar {\alpha} _ {k} = \frac {f \left(x ^ {k} , y ^ {k + 1} ; \phi\right) - l _ {f (\cdot , y ^ {k + 1}; \phi} ^ {*}}{p \| h _ {f} ^ {k} \| ^ {2}}, \quad \alpha_ {l, k} = \frac {\alpha_ {l , 0}}{\sqrt {k + 1}}, \quad \alpha_ {b, k} = \frac {\alpha_ {b , 0}}{\sqrt {k + 1}}, \tag {12}
$$
where  $\alpha_{l,0} \leq \alpha_{b,0}$  and  $l_{f(\cdot ,y^{k + 1};\phi)}^*$  is a lower bound for  $\inf_x f(x,y^{k + 1};\phi)$ . For computing  $h_f^k$ , we can take a similar approach as previous works [15, 19, 4] that use a Neumann series approximation
$$
h _ {f} ^ {k} = \nabla_ {x} f \left(x ^ {k}, y ^ {k + 1}; \phi\right) - \nabla_ {x y} g \left(x ^ {k}, y ^ {k + 1}; \psi_ {0}\right) \left[ \frac {N}{L _ {g}} \prod_ {j = 1} ^ {\bar {N}} \left(I - \nabla_ {y y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}; \psi_ {j}\right)\right) \right] \nabla_ {y} f \left(x ^ {k}, y ^ {k + 1}; \phi\right), \tag {13}
$$
where  $\bar{N}$  is sampled uniformly from  $[N]$  and  $N$  is the total number of samples. For BiSPS, we use the same sample for  $f(x^{k},y^{k + 1};\phi)$  and  $\nabla_{x}f(x^{k},y^{k + 1};\phi)$  when evaluating  $\bar{\alpha}_k$  in (12). Interestingly, we also empirically observe that using independent samples for computing  $\bar{\alpha}_k$  and  $h_f^k$  results in similar performance as using the same sample. The optimal rate of SGD for non-convex bi-level optimization is  $\mathcal{O}(\frac{1}{\sqrt{K}})$  without a growing batch size [4]. We show that BiSPS can obtain the same rate (see Theorem 3) by taking the envelope-type step-size of the form (10) and (11). We implement BiSPS according to (12) and observe that it has better performances over decaying-step SGD with less variations across different values of  $\alpha_{b,0}$  (see Figure 4 and note that decaying-step SGD is of the form  $\frac{\alpha_{b,0}}{\sqrt{k + 1}}$ ).
(a) Validation Loss
(b) Upper-level step sizes
Figure 5: Results on hyper-representation learning task (see Sec 4 for details). (a) Validation loss against upper-level iterations for comparing BiSLS-Adam/SGD to fine-tuned Adam/SGD. (b)(c) Upper (left) and lower-level (right) learning rates found by BiSLS-Adam. For the fine-tuned Adam, the optimal lower and upper-level learning rates are  $\mathcal{O}(1)$  and  $\mathcal{O}(10^{-4})$ , respectively. BiSLS-Adam outperforms fine-tuned Adam/SGD with a starting point that is 5 orders of magnitude larger than the optimal step size.
(c) Lower-level step sizes
Stochastic Line-Search Algorithms for Bi-level Optimization The challenge of extending SLS to bi-level optimization is rooted in the term  $y^{*}(x)$ . In fact, we realize that some of the bi-level objectives are of the form  $F(x) = f(y^{*}(x))$ . That is,  $f$  does not have an explicit dependence on  $x$  as in the data hyper-cleaning task [22]. This implies that when SLS takes a potential step on  $x$ , the approximation of  $y^{*}(x)$  (i.e.,  $\bar{y}(x)$ ) also needs to be updated, otherwise there is no change in function values. Moreover, the use of approximation  $\bar{y}(x)$  and the stochastic estimation error in hypergradient would not guarantee a step size can be always found. To this end, we modify the Armijo line-search rule to be:
BiSLS-SGD:  $f\big(x^{k} - \alpha_{k}h_{f}^{k},\hat{y}^{k + 1}(x^{k} - \alpha_{k}h_{f}^{k})\big)\leq f(x^{k},y^{k + 1}) - p\alpha_{k}\| h_{f}^{k}\|^{2} + \delta ,$
BiSLS-Adam:  $f\big(x^{k} - \alpha_{k}A_{k}^{-1}h_{f}^{k},\dot{y}^{k + 1}(x^{k} - \alpha_{k}A_{k}^{-1}h_{f}^{k})\big)\leq f(x^{k},y^{k + 1}) - p\alpha_{k}\| h_{f}^{k}\|_{A_{k}^{-1}}^{2} + \delta ,$  (14)
where  $p, \delta > 0$  and  $A_{k}$  is a positive definite matrix such that  $A_{k}^{2} = G_{k}$ .