Title: BiSLS/SPS: Auto-tune Step Sizes for Stable Bi-level Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adaptive step sizes

bi-level optimization

convergence rates

line searches
Abstract: The popularity of bi-level optimization (BO) in deep learning has spurred a growing interest in studying gradient-based BO algorithms.
However, existing algorithms involve two coupled learning rates that can be affected by approximation errors when computing hypergradients, making careful fine-tuning necessary to ensure fast convergence. To alleviate this issue, we investigate the use of recently proposed adaptive step-size methods, namely stochastic line search (SLS) and stochastic Polyak step size (SPS), for computing both the upper and lower-level learning rates. First, we revisit the use of SLS and SPS in single-level optimization without the additional interpolation condition that is typically assumed in prior works. For such settings, we investigate new variants of SLS and SPS that improve upon existing suggestions in the literature and are simpler to implement. Importantly, these two variants can be seen as special instances of general family of methods with an envelope-type step-size. This unified envelope strategy allows for the extension of the algorithms and their convergence guarantees to BO settings. Finally, our extensive experiments demonstrate that the new algorithms, which are available in both SGD and Adam versions, can find large learning rates with minimal tuning and converge faster than corresponding vanilla SGD or Adam BO algorithms that require fine-tuning.

BiSLS/SPS: Auto-tune Step Sizes for Stable Bi-level Optimization

Abstract

The popularity of bi-level optimization (BO) in deep learning has spurred a growing interest in studying gradient-based BO algorithms. However, existing algorithms involve two coupled learning rates that can be affected by approximation errors when computing hypergradients, making careful fine-tuning necessary to ensure fast convergence. To alleviate this issue, we investigate the use of recently proposed adaptive step-size methods, namely stochastic line search (SLS) and stochastic Polyak step size (SPS), for computing both the upper and lower-level learning rates. First, we revisit the use of SLS and SPS in single-level optimization without the additional interpolation condition that is typically assumed in prior works. For such settings, we investigate new variants of SLS and SPS that improve upon existing suggestions in the literature and are simpler to implement. Importantly, these two variants can be seen as special instances of general family of methods with an envelope-type step-size. This unified envelope strategy allows for the extension of the algorithms and their convergence guarantees to BO settings. Finally, our extensive experiments demonstrate that the new algorithms, which are available in both SGD and Adam versions, can find large learning rates with minimal tuning and converge faster than corresponding vanilla SGD or Adam BO algorithms that require fine-tuning.

1 Introduction

Bi-level optimization (BO) has found its applications in various fields of machine learning such as hyperparameter optimization [14, 17, 30, 40], adversarial training [51], data distillation [2, 53], neural architecture search [28, 39], neural-network pruning [52], and meta-learning [13, 37, 11]. Specifically, BO is used widely for problems that exhibit a hierarchical structure of the following form:

Here, the solution to the lower-level objective g becomes the input to the upper-level objective f , and in (1) the upper-level variable x is fixed when optimizing the lower-level variable y . To solve such bi-level problems using gradient-based methods requires computing the hypergradient of F , which based on the chain rule is given as [15]:

In practice, the closed-form solution y ∗ ( x ) can be difficult to obtain, and one strategy is to run a few steps of (stochastic) gradient descent on g with respect to y to get an approximation ¯ y , and

use ¯ y in places of y ∗ ( x ) . We denote the stochastic hypergradient based on ¯ y as h f ( x, ¯ y ) and the stochastic gradient of g with respect to y as h g . This leads to a general gradient-based framework for solving bi-level optimization [15, 19, 4]. At each iteration k , run T (can be one or more) steps of SGD on y with a step size β , y k,t +1 = y k,t -βh k,t g , then run one step on x using the approximated hypergradient:

Based on this framework, a series of stochastic algorithms have been developed to achieve the optimal or near-optimal rate of their deterministic counterparts [7, 8]. These algorithms can be broadly divided into single-loop ( T = 1 ) or double-loop ( T > 1 ) categories [23].

Unlike minimizing the single-level finite-sum (convex) problem

where only one learning rate is involved when using SGD, bi-level optimization involves tuning both the lower and upper-level learning rates ( β and α respectively). This poses a significant challenge due to the potential correlation between these learning rates [19]. Thus, as

0

10

20

30

40

50

Upper Iterations

10 0

4×10 1

6×10 1

2×10 0

Validation Loss

=0.1

=1.0

=5.0

=10.0

0

10

20

30

40

50

Upper Iterations

10 0

6×10 1

2×10 0

Validation Loss

=0.001

=0.0025

=0.005

=0.01

=0.05 (Diverge)

=0.1 (Diverge)

observed in Figure 1, algorithm divergence can occur when either α or β is large. While there is considerable literature on achieving faster rates in bi-level optimization [24, 5, 7, 8], only a few studies have focused on stabilizing its training and automating the tuning of α and β .

This work addresses the question: Is it possible to utilize large α and β without manual tuning? In doing so, we explore the use of stochastic adaptive-step size methods, namely stochastic Polyak step size (SPS) and stochastic line search (SLS), which utilize gradient information to adjust the learning rate at each iteration [44, 29]. These methods have been demonstrated to perform well in interpolation settings with strong convergence guarantees [44, 29]. However, applying them to bi-level optimization (BO) introduces significant challenges, as follows. BO requires tuning two correlated learning rates (for lower and upper-level). The bias in the stochastic approximation of the hypergradient complicates the practical performance and convergence analysis of SLS and SPS. Other algorithmic challenges arise for both algorithms. For SLS, verifying the stochastic Armijo condition at the upper-level involves evaluating the objective at a new ( x, y ∗ ( x )) pair, while y ∗ ( x ) is only approximately known; For SPS, most existing variants guarantee good performance only in interpolating settings, which are typically not satisfied for the upper-level objective in BO [22]. Before presenting our solutions to the challenges above in Sec 2, we first review the most closely related literature.

1.1 Related Work

Gradient-Based Bi-level Optimization Penalty or gradient-based approaches have been used for solving bi-level optimization problems [10, 45, 21]. Here we focus our discussions on stochastic gradient-based methods as they are closely related to this work. For double-loop algorithms, an early work (BSA) by Ghadimi and Wang [15] has derived the sample complexity of ϕ in achieving an ϵ -stationary point to be O ( ϵ -2 ) , but require the number of lower-level steps to satisfy T ∼ O ( ϵ -1 ) . Using a warm start strategy (stocBiO), Ji et al. [22] removed this requirement on T . However, to achieve the same sample complexity, the batch size of stocBiO grows as O ( ϵ -1 ) . Chen et al. [4] removed both requirements on T and batch size by using the smoothness properties of y ∗ ( x ) and setting the step sizes α and β at the same scale. For single-loop algorithms, a pioneering work by Hong et al. [19] gave a sample complexity of O ( ϵ -2 . 5 ) , provided α and β are on two different scales (TTSA). By making corrections to the y variable update (STABLE), Chen et al. [5] improved the rate to O ( ϵ -2 ) . However, extra matrix projections required by STABLE can incur high computation cost [5, 4]. By incorporating momentum into the updates of x and y (SUSTAIN), Khanduri et al.

0

50

100

150

200

Iterations

10

2

10

1

10 0

Objective value

DecSPS

SPSmax

SPSB

SPSB (2 samples)

0

200

400

600

800

1000

Iterations

10

3

10

2

10

1

10 0

x

x

*

10 1

10 2

10 3

Iterations

10

2

10

1

10 0

k

0.0

0.5

x

0.0

0.5

1.0

1.5

y

Starting point

x

*

1

and

x

*

2

Minimum

[24] further improved the rate to O ( ϵ -1 . 5 ) [6]. Besides these single or double-loop algorithms, a series of works have drawn ideas from variance reduction to achieve faster convergence rates for BO. For example, Yang et al. [49] designed the VRBO algorithm based on SPIDER [12]. Dagréou et al. [7, 8] designed the SABA and SRBA algorithms based on SAGA and SARAH respectively, and demonstrate that they can achieve the optimal rate of O ( ϵ -1 ) [9, 35]. Huang et al. [20] proposes to use Adam-type step sizes in BO. However, it introduces three sequences of learning rates ( α k , β k , η k ) that require tuning, which limits its practical usage. To our knowledge, none of these works have explicitly addressed the fundamental problem of how to select α and β in bi-level optimization. In this work, we focus on the alternating SGD framework (T can be 1 or larger), and design efficient algorithms that find large α and β without tuning, while ensuring the stability of training.

Algorithm 1 BiSLS-Adam/SGD

Input: x 0 , y 0 , K , T , δ , α b, 0 , β b, 0 , η , w ∈ (0 , 1) Output: x

1: for k = 0 , 1 , . . . , K -1 do

2: y k, 0 = y k

3: for t = 0 , 1 , . . . , T -1 do

4: β t b,k ← reset( β , β b, 0 , η , opt) ▷ see Algorithm 2

5: β ← line-search based on (8) starting from β t b,k

6: y k,t +1 = y k,t -β h k,t g ,

7: end for

8: y k +1 = y k,T -1 ; ˆ x k = x k ; ˆ y k +1 = y k +1

9: α ← reset( α , α b, 0 , η , opt)

10: while (14) based on ( ˆ x k , ˆ y k +1 , α , δ ) does not hold. do

11: α = α ∗ w

12: ˆ x k = x k -αh f ( x k , y k +1 ) or

13: ˆ x k = x k -αA -1 k h f ( x k , y k +1 )

14: ˆ y k +1 = y k +1 -β h g (ˆ x k , y k +1 )

15: end while

16: x k +1 = x k -αh k f

17: end for

Adaptive Step Size Adaptive step-size such as Adam has found great success in modern machine learning, and different variants have been proposed [25, 38, 47, 31, 32]. Here, we limit our discussions on two adaptive step sizes that are most relevant to this work. The Armijo line search is a classic way for finding step sizes for gradient descent [48]. Vaswani et al. [44] extends it to the stochastic setting (SLS) and demonstrates that the algorithm works well with minimal tuning required under interpolation, where the model fits the data perfectly. This method is adaptive to local smoothness of the objective, which is typically difficult to predict a priori. However, the theoretical guarantee of SLS in the non-interpolating regime is lacking. In fact, the results in Figure 3 suggest that SLS can perform poorly for convex losses when interpolation is not satisfied. Besides SLS, another adaptive method derived from the Polyak step size is proposed by Loizou et al. [29] with the name stochastic Polyak step size (SPS). Loizou et al. [29] further places an upper bound on the step size resulting in the SPS max variant. Similar to SLS, the algorithm performs well when the model is over-parametrized. Without interpolation, the algorithm converges to a neighborhood of the solution whose size depends on this upper bound.

In a later work, Orvieto et al. [36] make the SPS converge to the exact solution by ensuring the step size and its upper bound are both non-increasing (DecSPS ). However, enforcing monotonicity may result in the step size being smaller than decaying-step SGD and losing the adaptive features of SPS (see Figure 2, 3). In this work, we propose new versions of SLS and SPS that do not require monotonicity and extend them into the alternating SGD bi-level optimization framework (3).

100

500

1000

1500

2000

b

, 0

0.050

0.075

0.100

0.125

0.150

0.175

0.200

0.225

Minimum Train Loss

SGD

SPSB

(a) Minimum train loss vs.

γ

b,

0

0

1000

2000

3000

4000

5000

Iterations

10

1

10 0

10 1

Train Loss

Const SGD

SLS

SPSmax

DecSPS

SLSB

SPSB

(b) Train loss vs. iterations

10 0

10 1

10 2

10 3

10 4

Iterations

10

2

10

1

10 0

10 1

10 2

10 3

Step Sizes

Const SGD

SLS

SPSmax

DecSPS

SLSB

SPSB

(c) Step size vs. iterations

2 Summary of Contributions

We discuss our main contributions in this section, which is organized as follows. First, we discuss our variants of SPS and SLS, and unify them under the umbrella of 'envelope-type step-size'. Then, we extend the envelope-type step size to the bi-level setting. Finally, we discuss our bi-level line-search algorithms based on Adam and SGD.

Converging SPSB and SLSB by Envelope Approach We first propose simple variants of SLS and SPS that converge in the non-interpolating setting while not requiring the step size to be monotonic. To this end, we introduce a new stochastic Polyak step size (SPSB). For comparison, we also recall the step-sizees of SPS max and DecSPS . For all methods, the iterate updates are given as x k +1 = x k -γ k ∇ f i k ( x k ) where i k is sampled uniformly from [ n ] = { 1 , . . . , n } at each iteration k . The step-sizes γ k are then defined as follows:

where f ∗ i = inf x f i ( x ) , ¯ γ = 1 c 0 min { f i 0 ( x 0 ) -l ∗ i 0 ∥∇ f i 0 ( x 0 ) ∥ 2 , c 0 γ b, 0 } , c k is non-decreasing, γ b,k is nonincreasing, and l ∗ i ≤ f ∗ i is any lower bound.

Unlike SPS max in which γ b, 0 is a constant, our upper bound γ b,k is non-increasing. Also, unlike DecSPS in which both the step size and the upper bound are non-increasing (this is because γ k ≤ c k -1 c k γ k -1 and min { 1 2 cL max , c 0 γ b, 0 c k } ≤ γ k ≤ c 0 γ b, 0 c k [36, Lemma 1]), we simplify the recursive structure and do not require the step-size to be monotonic. As we empirically observe in Figure 3, the step size of DecSPS is similar to that of decaying SGD and in fact can be much smaller. Interestingly, the resulting performance of DecSPS is worse than SPS max despite SPS max eventually becoming unstable once the iterates get closer to the neighborhood of a solution and the step-size naturally behaves erratically. This is not unexpected due to small gradient norms (note the division by gradient-norm in (5)) and dissimilarity between samples in the non-interpolating scenario. Moreover, note that the adaptivity of SPS in the early stage seems to be lost in DecSPS due to the monotonicity of the latter. On the other hand, SPSB not only takes advantage of the large SPS steps that leads to fast convergence, but also stays regularized due to the non-increasing upper bound γ b,k in (19). These observations are further supported by the experiments on quadratic functions given in Figure 2, where we observe the fast convergence of SPSB and the instability of SPS max . Furthermore, SPSB is significantly more robust to the choice of γ b, 0 than decaying-step SGD as shown by Figure 3a. Motivated by the good practical performance of SPSB , we take a similar approach for SLS. The

SLS proposed and analyzed by Vaswani et al. [44] starts with γ b, 0 and in each iteration k finds the largest γ k ≤ γ b, 0 that satisfies:

To ensure its convergence without interpolation, we replace γ b, 0 with appropriate non-increasing sequence γ b,k . We name this variant of SLS as SLSB . Interestingly, the empirical performance and step size of SLSB are similar to those of SPSB (see Figure 3). This can be explained by observing that the step sizes of SPSB and SLSB share similar envelope structures, as follows (see Lemma 1 in Appendix A):

Therefore, we unify their analysis based on the following generic envelope-type step size :

where ω > 0 , γ b,k is non-increasing, and ˜ γ k satisfies γ l,k := min { ω, γ b,k } ≤ ˜ γ k ≤ γ b,k . We show that this envelope-type step size converges at a rate O ( 1 √ K ) and O ( 1 K ) for convex and strongly-convex losses respectively.

Envelope Step Size for Bi-level Optimization (BiSPS) We extend the use of envelope-type step sizes to the bi-level setting. The step sizes for upper and lower-level objectives of our general envelope-type method are:

where y ∗ x k ,ψ is the minimizer of the function g ( x k , · ; ψ ) , and α l,k , α b,k , and β b,k are three non-increasing sequences. Note that β b,k is fixed over the lowerlevel iterations for a given k , therefore this is equivalent to running T steps of SPS max to minimize the function g at each upper iteration k . However, the decrease in the upper bound β b,k with k is crucial to guarantee the overall convergence of the algorithm (see Theorem 3). Starting from the general step-size rules in (10) and (11), our bi-level extension of SPS (BiSPS) follows by setting α k in the form of SPS computed using stochastic hypergradient h k f . That is,

where α l, 0 ≤ α b, 0 and l ∗ f ( · ,y k +1 ; ϕ ) is a lower bound for inf x f ( x, y k +1 ; ϕ ) . For computing h k f , we can take a similar approach as previous works [15, 19, 4] that use a Neumann series approximation

where ¯ N is sampled uniformly from [ N ] and N is the total number of samples. For BiSPS, we use the same sample for f ( x k , y k +1 ; ϕ ) and ∇ x f ( x k , y k +1 ; ϕ ) when evaluating ¯ α k in (12). Interestingly, we also empirically observe that using independent samples for computing ¯ α k and h k f results in similar performance as using the same sample. The optimal rate of SGD for non-convex bi-level optimization is O ( 1 √ K ) without a growing batch size [4]. We show that BiSPS can obtain the same rate (see Theorem 3) by taking the envelope-type step-size of the form (10) and (11). We implement BiSPS according to (12) and observe that it has better performances over decaying-step SGD with less variations across different values of α b, 0 (see Figure 4 and note that decaying-step SGD is of the form α b, 0 √ k +1 ).

0

10

20

30

40

50

Upper Iterations

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

Validation Loss

SGD

=5.0

=0.0025

SGD

=1.0

=0.005

SGD

=0.5

=0.005

Adam

=10.0

=0.0001

Adam

=1.0

=0.0005

Adam

=0.5

=0.0005

BiSLS-SGD (

b

, 0 =100.0,

b

, 0 =10.0)

BiSLS-Adam (

b

, 0 =100.0,

b

, 0 =10.0)

(a) Validation Loss

0

10

20

30

40

50

Upper Iterations

10

6

10

5

10

4

10

3

Upper Step Size

b

, 0 =0.1

b

, 0 =0.5

b

, 0 =1.0

b

, 0 =5.0

b

, 0 =10.0

(b) Upper-level step sizes

0

500

1000

1500

2000

2500

Iterations

10 1

6×10 0

2×10 1

Lower Step Size

b

, 0=100

b

, 0=500

b

, 0 =1000

b

, 0 =5000

(c) Lower-level step sizes

Stochastic Line-Search Algorithms for Bi-level Optimization The challenge of extending SLS to bi-level optimization is rooted in the term y ∗ ( x ) . In fact, we realize that some of the bi-level objectives are of the form F ( x ) = f ( y ∗ ( x )) . That is, f does not have an explicit dependence on x as in the data hyper-cleaning task [22]. This implies that when SLS takes a potential step on x , the approximation of y ∗ ( x ) (i.e, ¯ y ( x ) ) also needs to be updated, otherwise there is no change in function values. Moreover, the use of approximation ¯ y ( x ) and the stochastic estimation error in hypergradient would not gaurantee a step size can be always found. To this end, we modify the Armijo line-search rule to be:

where p, δ > 0 and A k is a positive definite matrix such that A 2 k = G k .

Similar to the single-level Adam case, the matrix G k in the bi-level setting is defined as G k = ( β 2 G k -1 +(1 -β 2 ) diag( h k f h k f T )) / (1 -β k 2 ) [25, 43]. Moreover, BiSLS-Adam takes the following steps for updating the variable x : x k +1 = x k -α k A -1 k m k where m k +1 = β 1 m k -(1 -β 1 ) h k f . The details are given in Algorithms 1 and 2. We denote the search starting point for the upper-level as α b,k at iteration k , and denote it as β t b,k at step t within iteration k for the lower-level. We remark the following key benefits

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

2

10

1

10 0

Validation Loss

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

2

10

1

10 0

Validation Loss

of resetting α b,k and β t b,k (by using Algorithm 2) to larger values with reference to α k and β t k (respectively) at each step: (1) we avoid always searching from α b, 0 or β 0 b, 0 , thus reducing computation cost and (2) preserving an overall non-increasing (not necessarily monotonic) trend for α b,k and β t b,k (improving training stability). We found that different values of η all give good empirical performances (see Appendix B). The key algorithmic challenge we are facing is that during the backtracking process, for any candidate α k , we need to compute ˆ x k := x k -α k h k f and approximate y ∗ (ˆ x k ) with ˆ y k +1 (see Algorithm 1). To limit the cost induced by this nested loop, we limit the number of steps to obtain ˆ y k +1 to be 1.

Moreover, δ in (14) plays the role of a safeguard that ensures a step size can be found. We set δ to be small to avoid finding unrealistically large learning rates while tolerating some error in the

hypergradient estimation (see Appendix B for experiments on the sensitivity of δ ). In practice, we found that simply setting δ = 0 works well. In Figure 5a, we observe that BiSLS-Adam outperforms fine-tuned Adam or SGD. Surprisingly, its training is stable even when the search starting point α b, 0 is 5 orders of magnitude larger than a fine-tuned learning rate ( O (10 -4 ) ). Importantly, BiSLS-Adam finds large upper and lower-level learning rates in early phase (see Figure 5b, 5c) for different values of α b, 0 and β b, 0 that span 3 orders of magnitudes. Interestingly, the learning rates naturally decay with training (also see Figure 6c and 6d). In essence, BiSLS is a truly adaptive (no knowledge of initialization required) and robust (different initialization works) method that finds large α and β without tuning . In the next section, we give the convergence results of the envelope-type step size.

3 Convergence Results

3.1 Envelope-type step size for single-level optimization

We first state the assumptions, which are standard in the literature, that will be used for analyzing single-level problems. Assumption 1 is on the Lipschitz continuity of f and f i in Problem 4.

Assumption 1. The individual function f i is convex and L i -smooth such that ∥∇ f i ( x ) -∇ f i ( x ′ ) ∥ ≤ L i ∥ x -x ′ ∥ , ∀ i, ∀ x ∈ dom f and the overall function f is L -smooth. We denote L max ≜ max i L i . Furthermore, we assume there exists l ∗ i such that l ∗ i ≤ f ∗ i := inf x f i ( x ) , ∀ i , and f is lower bounded by f ∗ obtained by some x ∗ such that f ∗ = f ( x ∗ ) .

The following bounded gradient assumption is also used in the analysis of convex problems [41, 33]. Assumption 2. There exists G > 0 such that ∥∇ f i ( x ) ∥ 2 ≤ G, ∀ i .

We first state the theorem for the envelope-type step size defined in (9) for convex functions.

Theorem 1. Suppose Assumption 1, 2 hold, each f i is convex, C = dom f , γ k is independent of the sample ∇ f k ( x k ) , and choose γ b,k = γ b, 0 √ k +1 . Then, the envelope-type step size in (9) achieves the following rate,

We were not able to give a convergence result that uses the same sample for computing the step size and the gradient. However, we empirically observe that the performance is similar when using either one or two independent samples per iteration (see Figure 2 and Appendix B). When two independent samples i k and j k are used per iteration, the first computes the gradient sample ∇ f i k ( x k ) , and the

k

other computes the step-size γ k . For example, for SPSB this gives γ k = min { f j k ( x ) -l j k c k ∥∇ f j k ( x k ) ∥ 2 , γ b,k } . This type of assumption has been used in several other works for analyzing adaptive step sizes [27, 44, 29]. Under this assumption, we specialize the results of Theorem 1 to SPSB and SLSB , where γ l,K -1 = min { 1 2 cL max , γ b, 0 √ K } and γ l,K -1 = min { 2(1 -¯ c ) L max , γ b, 0 √ K } respectively. Concretely, for K ≥ γ 2 b, 0 L 2 max , SLSB and SPSB with γ b,k = γ b, 0 √ k +1 and c = ¯ c = 1 2 achieve the following rate: E [ f (¯ x K ) -f ( x ∗ )] ≤ ∥ x 0 -x ∗ ∥ 2 2 γ b, 0 √ K + γ b, 0 G 2 log( K ) 2 √ K . Next, we state the result for the envelop-type step size when f is µ -strongly convex.

∗

Theorem 2. Suppose a µ -strongly convex function f satisfying Assumptions 1 and 2, assume C is a closed and convex set, and γ k is independent of the sample ∇ f k ( x k ) . Then an envelope-type step size as in (9) with γ b,k = γ b, 0 k +1 , γ b, 0 ≥ 1 µ , and ωµ < 1 achieves the following rate

where ¯ x K = 1 K -k 0 ∑ K -1 k = k 0 x k and k 0 = max { 1 , ⌈ γ b, 0 /ω ⌉ -1 } .

We can again apply the result of Theorem 2 to SPSB and SLSB with γ b,k = γ b, 0 k +1 , γ b, 0 ≥ 1 µ , ω = 1 /L max , and c = ¯ c = 1 2 to get an explicit rate: E [ f (¯ x K ) -f ( x ∗ )] ≤ µk 0 2( K -k 0 ) ( e -k 0 µ L max ∥ x 0 -x ∗ ∥ 2 + γ 2 b, 0 G 2 ) + γ b, 0 G 2 log K 2( K -k 0 ) , where k 0 = max { 1 , ⌈ γ b, 0 L max ⌉ -1 } .

Remark 1. Under the envelop-type step size framework and the assumption of two independent samples, SLSB and SPSB share the same convergence rates of O ( 1 √ K ) and O ( 1 K ) as SGD with decaying step-size for convex and strongly-convex losses respectively. For the latter, a projection operation is required to stay in the closed and convex set C . These rates are not surprising because of the structure of the envelope step-size in (9) . Indeed, the proof is similar to the standard proof of analogous rate for SGD with decaying step-size. Nonetheless, we include it here for completeness.

3.2 Envelope-type step size for bi-level optimization

We start with recalling standard assumptions in BO [22, 19, 15, 4]. We denote z = [ x ; y ] and recall the bi-level problem in (1). The first assumption is on the lower-level objective g .

Assumption 3. The function g ( x, y ) is µ g strongly convex in y for any given x . Moreover, ∇ g is Lipschitz continuous: ∥∇ g ( x 1 , y 1 ) -∇ g ( x 2 , y 2 ) ∥ ≤ L g ∥ z 1 -z 2 ∥ (also assume that this holds true for each sampled function g ( x, y ; ψ ) ), and ∇ 2 g is Lipschitz continuous: ∥∇ 2 g ( x 1 , y 1 ) -∇ 2 g ( x 2 , y 2 ) ∥ ≤ L G ∥ z 1 -z 2 ∥ . We further assume that ∥∇ 2 xy g ( x, y ) ∥ ≤ C g , and the condition number is defined as κ = L g µ g .

Next, we state the assumptions on the upper objective f .

Assumption 4. The function f and its gradients are Lipschitz continuous. That is: ∥ f ( x 1 , y 1 ) -f ( x 2 , y 2 ) ∥ ≤ L 1 ∥ z 1 -z 2 ∥ and ∥∇ f ( x 1 , y 1 ) -∇ f ( x 2 , y 2 ) ∥ ≤ L f, 1 ∥ z 1 -z 2 ∥ . We also assume that ∥∇ y f ( x, y ) ∥ ≤ C f .

Furthermore, we make the following standard assumptions on the estimates of ∇ f , ∇ g , and ∇ 2 g .

Assumption 5. The stochastic gradients are unbiased: E ϕ [ ∇ f ( x, y ; ϕ )] = ∇ f ( x, y ) , E ψ [ ∇ g ( x, y ; ψ )] = ∇ g ( x, y ) , and E ψ [ ∇ 2 g ( x, y ; ψ )] = ∇ 2 g ( x, y ) . The variances of ∇ f ( x, y ; ϕ ) and ∇ 2 g ( x, y ; ψ ]) are bounded: E ϕ [ ∥∇ f ( x, y ; ϕ ) - ∇ f ( x, y ) ∥ 2 ] ≤ σ 2 f and E ψ [ ∥∇ 2 g ( x, y ; ψ ) -∇ 2 g ( x, y ) ∥ 2 ] ≤ σ 2 G .

Finally, we introduce the bounded optimal function value assumption in (15), which is used specifically for analyzing step size of the form (11) in the bi-level setting:

where y ∗ ( x ) = min y g ( x, y ) and y ∗ x,ψ = min y g ( x, y ; ψ ) for a given x (recall that at any iteration k , the lower-level steps in BiSPS are SPS max with an upper bound β b,k ; furthermore, β b,k is nonincreasing w.r.t. upper iteration k ). The one-variable analogous assumption of (15) has been used in the analysis of SPS max [29]. Here, we extend it to a two-variable function. Unlike the bounded variance assumption (16), which needs to hold true for all x and y , we require (15) to hold at y ∗ ( x ) for any given x . As mentioned previously, the closed form solution y ∗ ( x ) is difficult to obtain. Thus, we define the following expression by replacing y ∗ ( x ) with y in (2):

Astochastic Neumann series in (13) approximates (17) with x and y being x k and y k +1 (respectively), also recall that y k +1 is an approximation of y ∗ ( x k ) by running T lower-level SGD steps to minimize g w.r.t. y for a fixed x k . Based on Assumptions 3, 4, and 5, we have the following results to be used in the analysis [15]: ∥∇ F ( x 1 ) - ∇ F ( x 2 ) ∥ ≤ L F ∥ x 1 -x 2 ∥ , ∥ y ∗ ( x 1 ) -y ∗ ( x 2 ) ∥ ≤ L y ∥ x 1 -x 2 ∥ , and ∥ ¯ ∇ f ( x, y ∗ ( x )) -¯ ∇ f ( x, y ) ∥ ≤ L f ∥ y ∗ ( x ) -y ∥ . Furthermore, the bias in the stochastic hypergradient in (13) (denoted as B ) decays exponentially with N and its variance is bounded, i.e. E [ ∥ h k f -E [ h k f ] ∥ 2 ] ≤ ˜ σ 2 f (see Appendix A for details) [19].

Now, we state our main theorem based on step size of the form (10) and (11).

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

BiSLS-SGD Lower-Search=True Upper-Search=False

BiSLS-SGD Lower-Search=True Upper-Search=True

BiSLS-Adam Lower-Search=True Upper-Search=False

BiSLS-Adam Lower-Search=True Upper-Search=True

(a) Validation Loss

0

10

20

30

40

50

Upper Iterations

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Validation Accuracy

SGD

=5.0

=0.0025

SGD

=1.0

=0.005

SGD

=0.5

=0.005

Adam

=10.0

=0.0001

Adam

=1.0

=0.0005

Adam

=0.5

=0.0005

BiSLS-SGD (

b

, 0 =100.0,

b

, 0 =10.0)

BiSLS-Adam (

b

, 0 =100.0,

b

, 0 =10.0)

(b) Validation Accuracy

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

b

, 0 =0.1

b

, 0 =0.5

b

, 0 =1.0

b

, 0 =5.0

b

, 0 =10.0

BiSLS-SGD

b

, 0 =0.1

BiSLS-SGD

b

, 0 =0.5

BiSLS-SGD

b

, 0 =1.0

BiSLS-SGD

b

, 0 =5.0

BiSLS-SGD

b

, 0 =10.0

(c) Different

α

b,

0

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

b

, 0=100

b

, 0=500

b

, 0 =1000

b

, 0 =5000

(d) Different

β

b,

0

Theorem 3. Suppose f and g satisfy Assumptions 3, 4, and 5, learning-rate upper bounds α b,k = α b, 0 √ k +1 and α l,k = α l, 0 √ k +1 with α b, 0 and α l, 0 satisfying 1 L F +4 L 2 y ≥ α 2 b, 0 α l, 0 and α l, 0 ≤ α b, 0 . Further assume that α k is independent of the stochastic hypergradient h k f , and each sampled function g ( x, y ; ψ ) is convex. Then under the Assumption (15) with p ≥ 1 2 , C k = min { 1 2 pL g , β b,k } , T ≥ log( α b, 0 L 2 f +2) -log(1 -µ g C K -1 ) , and β b,k = β b, 0 k +1 , BiSPS achieves the rate:

4 Additional Hyper-Representation and Data Distillation Experiments

Hyper-representation learning: The experiments are performed on MNIST dataset using LeNet [26, 42]. We use conjugate gradient method for solving system of equations when computing the hypergradient [17]. The upper and lower-level objectives are to optimize the embedding layers and the classifier (i.e. the last layer of the neural net), respectively (see Appendix B for details). For constantstep SGD and Adam, we tune the lower-level learning rate β ∈ { 10 . 0 , 5 . 0 , 1 . 0 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 } . For the upper-level learning rate, we tune α ∈ { 0 . 001 , 0 . 0025 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 } for SGD, and α ∈ { 10 -5 , 5 · 10 -5 , 10 -4 , 5 · 10 -4 , 0 . 001 , 0 . 01 } for Adam (recall that δ in (14) is set to 0). Based on the results of Figure 6, we make the following key observations: ① line-search at the upper-level is essential for achieving the optimal performance (Figure 6a); ② BiSLS-Adam/SGD not only converges fast but also generalizes well (Figure 6b); ③ BiSLS-Adam/SGD is highly robust to search starting points α b, 0 and β b, 0 (Figure 6c, 6d). It addresses the fundamental question of how to tune α and β in bi-level optimization (see Appendix B for additional results on search cost).

Search cost and run-time comparison: The search cost of reset option 1 in Algorithm 2 for BiSLS-SGD and BiSLS-Adam is 89 ± 15 and 115 ± 16 per iteration, respectively, measured in the number of condition-checking via (14). These costs are significantly reduced when option 3 is used. Concretely, Figure 7b suggests that choosing η = 2 in option 3 results in ∼ 9 evaluations of (14) per iteration for both BiSLS-Adam and BiSLS-SGD. Choosing η = 1 in reset option 3 is equivalent to reset option 2, which further cuts down the cost to ∼ 4 . However, this choice forces the learning rate to be monotonically-decreasing, which leads to a slower convergence compared to other η 's (see Appendix B for more details). Besides this, Figure 7a shows that a single step for

1

2

4

6

8

10

# steps to approximate

y

*

(

x

)

91.0

91.5

92.0

92.5

93.0

Validation Accuracy (%)

100

125

150

175

200

225

250

275

Run Time (s)

(a) Approximation of

y

∗

(

x

)

2

4

6

8

10

5

10

15

20

25

Upper-Level Search Cost

BiSLS-SGD

(b) Search cost

5

10

15

20

25

# CG Steps

80

100

120

140

160

180

Run Time (s)

Adam

(c) Run time

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

3

10

2

10

1

10 0

Validation Loss

Adam

=0.001

Adam

=0.01

Adam

=0.1

SGD

=0.001

SGD

=0.01

SGD

=0.1

BiSLS-SGD

(a) Neumann

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

3

10

2

10

1

10 0

Validation Loss

Adam

=0.001

Adam

=0.01

Adam

=0.1

SGD

=0.001

SGD

=0.01

SGD

=0.1

BiSLS-SGD

(b) Identity

(c) Distilled MNIST Images

approximating the y ∗ ( x ) in (14) is sufficient, considering the trade-off between performance gain and extra computation overhead introduced. Most importantly, BiSPS-Adam takes less time to reach a threshold test accuracy than fine-tuned Adam (Figure 7c) due to a suitable (and potentially large) learning rate found using (14) (note that Figure 7c excludes the cost of tuning the learning rate of Adam).

Data distillation: The goal of data distillation is to generate a small set of synthetic data from an original dataset that preserves the performance of a model when trained using the generated data [46, 50]. We adapted the experiment set up from Lorraine et al. [30] to distill MNIST digits. We present the results in Figure 8, where we observe that BiSLS-SGD converges significantly faster than fine-tuned Adam or SGD, and generate realistic MNIST images (see Appendix B for more results).

5 Conclusion

In this work, we have given simple alternatives to SLS and SPS that show good empirical performance in non-interpolating scenario without requiring the step size to be monotonic. We unify their analysis based on a simplified envelope-type step size, and extend the analysis to the bi-level setting while designing a SPS-based bi-level algorithm. In the end, we propose bi-level line-search algorithm BiSLS-Adam/SGD that is empirically truly robust and adaptive to learning rate initialization. Our work opens several possible future directions. Given the superior performance of BiSLS, we prioritize an analysis of its convergence rates. The difficulty stems from: (a) the bias in hypergradient estimation; (b) the dual updates in x and y ∗ ( x ) (incurring nested loop structures); (c) the error in estimating y ∗ ( x ) . On single-level optimization, we remark as an important direction to relax the two-sample assumption on SPSB /SLSB . Ultimately, we hope to promote further research on bi-level optimization algorithms with minimal tuning.

Acknowledgement This work is funded partially by the NSERC Discovery Grant RGPIN-202103677, the NSERC Discovery Grant RGPIN-2022-03669, and the Canada CIFAR AI Chair Program.

References

Amir Beck. First-order methods in optimization . SIAM, 2017.

Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems , 33: 14879-14890, 2020.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Tighter analysis of alternating stochastic gradient method for stochastic nested problems, 2021.

Tianyi Chen, Yuejiao Sun, Quan Xiao, and Wotao Yin. A single-timescale method for stochastic bilevel optimization, 2022.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd, 2020.

Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms, 2022.

Mathieu Dagréou, Thomas Moreau, Samuel Vaiter, and Pierre Ablin. A lower bound and a near-optimal algorithm for bilevel empirical risk minimization, 2023.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives, 2014.

James E Falk and Jiming Liu. On bilevel programming, part i: general nonlinear cases. Mathematical Programming , 70:47-72, 1995.

Chen Fan, Parikshit Ram, and Sijia Liu. Sign-maml: Efficient model-agnostic meta-learning by signsgd, 2021.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex optimization via stochastic path integrated differential estimator, 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017.

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization, 2017.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming, 2018.

Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. Sgd: General analysis and improved rates. In International conference on machine learning , pages 5200-5209. PMLR, 2019.

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation, 2020.

Riccardo Grazzi, Massimiliano Pontil, and Saverio Salzo. Convergence properties of stochastic hypergradients, 2021.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic, 2022.

Feihu Huang, Junyi Li, and Shangqian Gao. Biadam: Fast adaptive bilevel optimization methods, 2023.

Yo Ishizuka and Eitaro Aiyoshi. Double penalty method for bilevel optimization problems. Annals of Operations Research , 34(1):73-88, 1992.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design, 2021.

Kaiyi Ji, Mingrui Liu, Yingbin Liang, and Lei Ying. Will bilevel optimizers benefit from loops, 2022.

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes, 2019.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2019.

Nicolas Loizou, Sharan Vaswani, Issam Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence, 2021.

Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics , pages 1540-1552. PMLR, 2020.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate, 2019.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4): 1574-1609, 2009.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáˇ c. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International Conference on Machine Learning , pages 2613-2621. PMLR, 2017.

Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution, 2022.

Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients, 2019.

Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 , 2019.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1-34, 2021.

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated backpropagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1723-1732. PMLR, 2019.

Shai Shalev-Shwartz, Yoram Singer, and Nathan Srebro. Pegasos: Primal estimated sub-gradient solver for svm. In Proceedings of the 24th international conference on Machine learning , pages 807-814, 2007.

Daouda Sow, Kaiyi Ji, and Yingbin Liang. On the convergence theory for hessian-free bilevel algorithms, 2022.

Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive gradient methods converge faster with over-parameterization (but you should do a line-search). arXiv preprint arXiv:2006.06835 , 2020.

Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates, 2021.

Luis Vicente, Gilles Savard, and Joaquim Júdice. Descent approaches for quadratic bilevel programming. Journal of optimization theory and applications , 81(2):379-399, 1994.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020.

Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes, 2021.

Jorge Nocedal Stephen J Wright. Numerical optimization, 2006.

Junjie Yang, Kaiyi Ji, and Yingbin Liang. Provably faster algorithms for bilevel optimization, 2021.

Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review, 2023.

Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning , pages 26693-26712. PMLR, 2022.

Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization, 2023.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719 , 2022.

A Proofs of Theorems and Additional Convergence Results

A.1 Useful Lemmas

Lemma 1 provides more details on the envelope structure of SPSB and SLSB given in (10) and (11). The lower bound in (19) will also be used in Lemma 5 (for bounding the term ∥ y k +1 -y ∗ ( x k ) ∥ 2 ).

Lemma 1. Under the Assumption 1, we have the following:

Proof. The bounds in (19) have been shown in [29, 36]. For (20), the first part of the inequality has been shown in [44]. For the second part, recall the Armijo condition (14):

We can then rearrange this to obtain

where l ∗ i k is any lower bound for f ∗ i k . Also recall that γ b,k is the search starting point at iteration k , hence (20) holds for SLSB .

Lemma 2 gives the expressions for the constants L F , L y , and L f . The proof can be found in [15, Lem 2.2].

Lemma 2. Under Assumptions 3 and 4, we have the following:

where

Lemma 3 is on the bias and variance of the stochastic hypergradient in (13), which has the following form [19]

Recall that the hypergradient surrogate defined in (17) based on ( x k , y k +1 ) is

Given a filtration F ′ k up to and including x k and y k +1 , the bias of the stochastic hypergradient is defined as B = ¯ ∇ f ( x k , y k +1 ) -E [ h k f |F ′ k ] , and the variance is defined as E [ ∥ ¯ ∇ f ( x k , y k +1 ) -E [ h k f |F ′ k ] ∥ 2 ] . Lemma 3 has been proven in [19, Lem 1.] (also see [4, Lem 5.]). Lemmas 1, 2, and 3 will be used in the proofs of Theorems 3 and 4.

Lemma 3. Under Assumptions 3, 4, and 5, the bias and variance of the stochastic hypergradient h k f satisfy the following

where N is the total number of samples, and ˜ σ 2 f = σ 2 f +[( σ 2 f + L 2 1 )( σ 2 G +2 L 2 g ) + σ 2 f L 2 g ] 3 µ 2 g ∼ O ( κ 2 ) .

Lemma 4 is on the descent of the quantity V k := F ( x k ) -F ( x ∗ ) + ∥ y k -y ∗ ( x k ) ∥ 2 . It will be used in the proofs of Theorem 3 and 4.

Lemma 4. Suppose F satisfies Assumptions 3, 4, and 5, sequences α b,k = α b, 0 √ k +1 and α l,k = α l, 0 √ k +1 with α b, 0 and α l, 0 satisfying 1 L F +4 L 2 y ≥ α 2 b, 0 α l, 0 and α l, 0 ≤ α b, 0 . Further assume that α k is independent of h k f . Then step sizes of the form (10) and (11) achieve the following:

where V k = F ( x k ) -F ( x ∗ ) + ∥ y k +1 -y ∗ ( x k +1 ) ∥ 2 and recall that F is the upper-level loss defined in (1) .

Proof. We denote E [ h k f |F ′ k ] = ¯ h k f . By the L F -smoothness of the objective F :

Take expectation conditioned on a filtration of past iterates F ′ k (up to and include x k , y k +1 ):

where (a) is by the assumption that α k is independent of h k f . Then expand the term ∥∇ F ( x k ) -¯ h k f ∥ 2 as follows:

where (b) is by Lemma 2 and 3. Substituting this into the above:

where (c) is by α l,k ≤ α k and α b,k ≥ α k . Then take total expectations and subtract F ( x ∗ ) :

Now define the potential function V k := F ( x k ) -F ( x ∗ ) + ∥ y k +1 -y ∗ ( x k +1 ) ∥ 2 and expand the term ∥ y k +1 -y ∗ ( x k +1 ) ∥ 2 as follows:

where (d) is by Lemma 2. Take expectation conditioned on F ′ k :

Then, take total expectations:

Now, based on the definition of V k and combining (25) and (26):

where (e) is because 1 L F +4 L 2 y ≥ α 2 b, 0 α l, 0 , which guarantees that α l,k = α l, 0 √ k +1 ≥ ( L F +4 L 2 y ) α b,k = ( L F +4 L 2 y ) α 2 b, 0 k +1 .

Lemma 5 and Lemma 6 give two alternatives for bounding the term ∥ y k +1 -y ∗ ( x k ) ∥ 2 . Lemma 5 is based on the assumption E ψ [ g ( x, y ∗ ( x ); ψ ) -g ( x, y ∗ x,ψ ; ψ )] ≤ σ 2 g , ∀ x . The proof for its one-variable analogous assumption is given in Loizou et al. [29]. Here we follow a similar approach for the two-variable function g ( x, y ) . Lemma 5 will be used in the proof of Theorem 3.

Lemma 5. Suppose Assumptions 3, 5, and the bounded optimal function value assumption (15) hold. Further assume that each sampled function g ( x, y ; ψ ) is convex, then step size of the form 11 achieves the following:

where C k = min { 1 2 pL g , β b,k }

Proof. We denote h k,t g = ∇ y g ( x k , y k,t ; ψ ) , and F ′ k,t be a filtration up to and including x k and y k,t . We have,

where (a) is by Lemma 1, (b) is by choosing p ≥ 1 2 , and, (c) is by individual convexity of g ( x, y ; ψ ) such that -⟨ y k,t -y ∗ ( x k ) , h k,t g ⟩ + g ( x k , y k,t ; ψ ) -g ( x k , y ∗ ( x k ); ψ ) ≤ 0 and recalling that C k = min { 1 2 pL g , β b,k } ≤ β k,t by Lemma 1. Take expectation conditioned on F ′ k,t and note that E [ h k,t g |F ′ k,t ] = ∇ y g ( x k , y k +1 ) , E [ g ( x k , y k,t ; ψ ) |F ′ k,t ] = g ( x k , y k,t ) , and E [ g ( x k , y ∗ ( x k ); ψ )] = g ( x k , y ∗ ( x k )) :

Now, based on the strong convexity of g w.r.t. y ,

we can further obtain (by taking total expectations of (27) and using strong-convexity):

Solve this recursively from t = 0 to t = T -1 (recall T is the total number of lower-level steps, y k, 0 = y k and y k +1 = y k,T ):

Lemma 6 is based on the standard bounded variance assumption E ψ [ ∥∇ y g ( x, y ∗ ( x ); ψ ) -∇ y g ( x, y ∗ ( x )) ∥ 2 ] ≤ σ 2 g , ∀ x, y in the bi-level optimization literature [19, 4]. Lemma 6 will be used in the proof of Theorem 4.

Lemma 6. Suppose Assumptions 3, 5 and the bounded variance assumption (16) hold. Suppose that p ≥ max { µ g µ g + L g , µ g + L g 4 L g } , β b, 0 ≤ min { 2 µ g + L g , µ g + L g 2 µ g L g , 1 2 pLg -2 µgLg µg + Lg } . Then step size of the form 11 achieves the following:

where C k = min { 1 2 pL g , β b,k } .

Proof. Similar to the proof of Lemma 5, we can start with

Divide both sides by β k,t

Then use the facts that β k,t ≤ β b,k and β k,t ≥ C k ,

Next, take expectation conditioned on the Filtration F ′ k,t up to and including x k and y k,t

where (a) is by the bounded variance assumption (16). Based on strong-convexity of g [34, Theorem 2.1.11], we have

Multiply by β b,k in both sides, group terms, and take total expectations to reach

where in (b) we have chosen β b,k ≤ β b, 0 ≤ 2 µ g + L g , ∀ k . Solving this recursion similar to Lemma 5, we obtain:

In (28), we require β b,k C k -2 µ g L g µ g + L g β b,k ≥ 0 (recall C k = min { 1 2 pL g , β b,k } ≤ β k,t ≤ β b,k ), this is equivalent to C k ≤ µ g + L g 2 µ g L g . In case of C k = 1 2 pL g , we choose p ≥ µ g µ g + L g (to avoid contradictions, we also choose p ≥ µ g + L g 4 L g ). In case of C k = β b,k , we choose β b,k ≤ β b, 0 ≤ µ g + L g 2 µ g L g .

We also require β b,k C k -2 µ g L g µ g + L g β b,k ≤ 1 . In case of C k = β b,k , this is equivalent to 2 µ g L g µ g + L g β b,k ≥ 0 , which is satisfied by all β b,k . In case of C k = 1 2 pL g , we choose β b,k ≤ β b, 0 ≤ 1 2 pL g -2 µgLg µg + Lg . Puting everything together, we have p ≥ max { µ g µ g + L g , µ g + L g 4 L g } , and 2 µ g + L g 1

A.2 Single-level Convex Proofs

A.2.1 Proof of Theorem 1

The proof of Theorem 1 is similar to the standard proof of decaying-step SGD (GD) that can be found in e.g. Beck [1]. Here, we give the proof for completeness.

where (a) is by individual convexity of f i , and (b) is by Assumption 2. Take conditional expectation and assume that γ k is independent of sample k

where (c) is by independence of γ k and x k , and (d) is because γ l,k ≤ γ k , ∀ k . Take total expectation and rearrange

Using the fact that γ l,K -1 ≤ γ l,k ∀ k ∈ [ K -1] and set γ b,k = γ 0 √ k +1 , we obtain

Summing over k = 0 to k = K -1 we obtain

Define ¯ x = 1 K ∑ K -1 k =0 x k , apply Jensen's inequality and rearrange

A.3 Proof of Theorem 2

The approach of Theorem 2 is similar to [16, Theorem 3.2]. The crucial difference is that the step size in [16, Theorem 3.2] is constant ( γ k = 1 L max ) for k ≤ 4 ⌈ κ ⌉ , whereas for envelope-type step size it is of the form:

where ¯ γ k can be (e.g. in the case of SPSB ):

The proof of Theorem 2 suggests that the step size can be either f i k ( x k ) -l ∗ i k c k ∥∇ f i k ( x k ) ∥ 2 or γ b, 0 k +1 depending on their magnitudes for k ≤ k 0 -1 ( k 0 = max { 1 , ⌈ γ 0 /ω ⌉ -1 } ). After k 0 iterations, the step size is γ k = γ b, 0 k +1 . This finding is numerically confirmed by the experimental results in Section B.

To proceed with the proof, we have:

where (a) is because γ k ≤ γ b,k , ∀ k . Take conditional expectations

where (b) is by bounded gradients assumption and strong convexity of f . Take total expectation and rearrange

Choose k 0 = max { 1 , ⌈ γ 0 /ω ⌉ -1 } , then for ∀ k s.t. k ≥ k 0 , we have γ l,k = min { ω, γ b,k } = γ b,k = γ 0 k +1 , which means γ k = γ 0 k +1 after k 0 steps. Within the first k 0 steps, the step size is γ k = min { ω, γ b,k } . Hence, for k ≥ k 0 we have

Now, sum from k = k 0 to K -1

For the first term in (29), call it A , we expand it as

where (d) is because γ 0 ≥ 1 µ , we have k +1 γ 0 -k γ 0 -µ ≤ 0 , ∀ k and k 0 +1 γ 0 -µ ≤ µk 0 . For the second term in (29), call it B, we have

Putting A and B together, we obtain

Within the first k 0 -1 iterations, similarly to the above, we have

For the first k ≤ k 0 -1 iterations, γ l,k = ω where ωµ < 1 ; thus, we obtain the following E [ ∥ x k +1 -x ∗ ∥ ] ≤ (1 -µω ) E [ ∥ x k -x ∗ ∥ 2 ] + γ 2 b,k G 2 .

Solve this recursively,

Putting this into (30), we obtain

Define ¯ x K = 1 K -k 0 ∑ K -1 k = k 0 x k , then by Jensen's inequality we have

A.4 Bi-level Proofs

A.4.1 Proof of Theorem 3

We substitute the result of Lemma 5 for the expression E [ ∥ y k +1 -y ∗ ( x k ) ∥ 2 ] ,

E [ V k +1 ] ≤ E [ V k ] -α l,k 2 E [ ∥∇ F ( x k ) ∥ 2 ] + [( α b,k L 2 f +2)(1 -µ g C k ) T -1] E [ ∥ y k -y ∗ ( x k ) ∥ 2 ]+ 2 α b,k β b,k TL 2 f σ 2 g +4 β b,k Tσ 2 g + α b,k B 2 +(2 L 2 y + L F 2 ) α 2 b,k ˜ σ 2 f ( a ) ≤ E [ V k ] -α l,k 2 E [ ∥∇ F ( x k ) ∥ 2 ] + [( α b, 0 L 2 f +2)(1 -µ g C K -1 ) T -1] E [ ∥ y k -y ∗ ( x k ) ∥ 2 ]+ 2 α b,k β b,k TL 2 f σ 2 g +4 β b,k Tσ 2 g + α b,k B 2 +(2 L 2 y + L F 2 ) α 2 b,k ˜ σ 2 f ( b ) ≤ E [ V k ] -α l,k 2 E [ ∥∇ F ( x k ) ∥ 2 ] + 2 α b,k β b,k TL 2 f σ 2 g +4 β b,k Tσ 2 g + α b,k B 2 +(2 L 2 y + L F 2 ) α 2 b,k ˜ σ 2 f , where (a) is by α b, 0 ≥ α b,k , C K -1 ≤ C k , ∀ k , hence ( α b, 0 L 2 f )(1 -µ g C K -1 ) T ≥ ( α b,k L 2 f )(1 -µ g C k ) T (recall that C k = min { 1 2 pL g , β b,k } in Lemma 1); (b) is by T ≥ log[ α b, 0 L 2 f +2] -log(1 -µC K -1 ) , which implies that ( α b, 0 L 2 f +2)(1 -µ g C K -1 ) T ≤ 1 . Now, rearrange and use the fact that α l,K -1 ≤ α l,k ,

Then sum over k = 0 to K -1 :

where we substituted α b,k = α b, 0 √ k +1 and β b,k = β b, 0 k +1 in (c); (d) is based on ∑ K -1 k =0 1 ( k +1) 1 . 5 ≤ 2 , ∑ K -1 k =0 1 k +1 ≤ log( K ) , and ∑ K -1 k =0 1 √ k +1 ≤ √ K . Recall that in Lemma 2, we have L f ∼ O ( κ 2 ) , L y ∼ O ( κ ) , and L F ∼ O ( κ 3 ) . Also recall that α b,k = α b, 0 √ k +1 and α l,k = α l, 0 √ k +1 , hence we choose α l, 0 ∼ α b, 0 ∼ O ( κ -3 ) , T ∼ O ( κ ) , and β b, 0 ∼ O ( κ -2 ) . Then we can obtain 1 K ∑ K -1 k =0 E [ ∥∇ F ( x k ) ∥ 2 ] ≤ O ( κ 3 √ K + κ 2 log( K ) √ K ) .

A.4.2 Theorem 4 and its proof

Theorem 4. Suppose f and g satisfy Assumptions 3, 4, and 5, and, learning-rate upper bounds α b,k = α b, 0 √ k +1 and α l,k = α l, 0 √ k +1 with α b, 0 and α l, 0 satisfying 1 L F +4 L 2 y ≥ α 2 b, 0 α l, 0 and α l, 0 ≤ α b, 0 . Further assume that α k is independent of the stochastic hypergradient h k f . Then, under the Bounded-variance assumption in (16) with p ≥ max { µ g µ g + L g , µ g + L g 4 L g } , β b, 0 ≤ min { 2 µ g + L g , µ g + L g 2 µ g L g , 1 2 pLg -2 µgLg µg + Lg } , and

Proof. Start with Lemma 4:

We substitute the result of Lemma 6 for the expression E [ ∥ y k +1 -y ∗ ( x k ) ∥ 2 ] ,

Now, rearrange and use the fact that α l,K -1 ≤ α l,k ,

Sum over k = 0 to k = K -1 :

where in (b) we have substituted α b,k = α b, 0 √ k +1 and β b,k = β b, 0 √ k +1 , and (c) is by ∑ K -1 k =0 1 ( k +1) 3 / 2 ≤ 2 , ∑ K -1 k =0 1 k +1 ≤ log( K ) , and ∑ K -1 k =0 1 ( k +1) 1 / 2 ≤ √ K . Similar to the proof of Theorem 3, we choose α l, 0 ∼ α b, 0 ∼ O ( κ -3 ) , T ∼ O ( κ ) , and β b, 0 ∼ O ( κ -1 ) to obtain 1 K ∑ K -1 k =0 E [ ∥∇ F ( x k ) ∥ 2 ] ≤ O ( κ 3 √ K + κ 2 log( K ) √ K ) .

B Additional Experiment Results

This section is organized as follows. First, we discuss synthetic quadratics experiments. Second, we provide more details on the sensitivity of the algorithms to the choices of δ in (14), on the reset procedure, and on the search cost of BiSLS. Third, we compare the empirical performance of 1-sample vs 2-samples implementations of our algorithms for single-level convex and bi-level optimization. Some additional results for hyper-representation learning and data distillation experiments are also presented. We run 5 independent runs for all our experiments.

B.1 Synthetic Quadratics

The experiments on quadratic functions are adapted from Loizou et al. [29]. The training objective is as follows:

where H i ( i = 1 , 2 ) are positive definite. The optimal solutions x ∗ i ( i = 1 , 2 ) are generated randomly from a standard normal distribution. Specifically, H i is defined as follows:

where O and λ i are taken from the spectral decomposition of P T P , and P is generated from the standard normal distribution. Figure 9 shows the convergence of various algorithms with different starting points. Interestingly, both SPSB with either 1 sample or 2 samples (1 sample for computing the gradient and the other for computing the step size) converge to the optimal solution (labelled with a star).

2

0

2

x

2

0

2

y

DecSPS

SPSB

SPSB (2 samples)

Starting points

x

*

1

and

x

*

2

Minimum

B.2 Sensitivity of δ , reset, and search cost

In this section, we discuss the effects of δ in (14), η in Algorithm 2, and comment on the search cost of the options in the Reset Algorithm 2. Recall that the line-search condition (8) assumes that we can find a largest γ k ≤ γ b, 0 to satisfy it. However, in practice, we apply a backtracking procedure, i.e. γ k = γ k ∗ w, 0 < w < 1 , until γ k satisfies (8). Therefore, the found learning rate is not guaranteed to be the largest. Nonetheless, we assume that γ k is the largest to simplify our analysis given above (similar arguments apply to line-search at both upper and lower-level in the bi-level optimization). The experiments in this section are based on hyper-representation learning [42]. In this case, the objective of the induced bilevel-optimization problem can be written as:

where ( X 1 , Y 1 ) and ( X 2 , Y 2 ) are validation and training data sets with sizes D X 1 and D X 2 , respectively; ˜ f ( · ; w ) are the embedding layers of the model parameterized by w ; and, c is the classification layer. Moreover, we use conjugate gradient methods (CG) [17, 18] to solve the linear system when computing the hypergradient for hyper-representation learning experiments.

Reset While Algorithm 2 (reset) can be applied to both upper and lower-level problems, we focus our discussions here on the upper-level learning rate ( α k ). This is because we empirically find it to be more critical for the convergence performance (see Figure 6a). As shown in Algorithm 2, reset has 3 options. Options 1 , 2 , and 3 search starting from α b, 0 , α k -1 , and ηα k -1 , at iteration k respectively. Option 1 has the highest search cost as it always starts from the same initial upper bound ( α b, 0 ). Option 2 ensures the monotonicity of the learning rate due to α k ≤ α b,k = α k -1 . Option 3 chooses the search starting point at iteration k ( α b,k ) by multiplying the previous learning rate ( α k -1 ) by a factor η ≥ 1 . As in the single-level convex case where monotonicity in the step size can potentially lead to slow convergence (see Figure 3), we again observe that monotonicity in the upper learning rate (i.e. option 2) leads to poorer performance when compared against options 1 or 3 as shown in Figure 10. Finally, we compare the performance of different η s in option 3 (note that η = 1 in option 3 is equivalent to option 2). We observe in Figure 11 that different η s perform equally well. This shows the robustness of our algorithm to the choice of η . As mentioned previously, the choice of option 3 over option 1 are due to 2 reasons: (a) reduced search cost; (b) provides an overall non-increasing and non-monotonic trend of upper bound α b,k . We discuss search cost of different η s in option 3 below.

Search Cost Based on the results in Figure 7b, we observe the use of option 3 in reset can significantly reduce the search cost for both BiSLS-Adam and BiSLS-SGD. Figure 10 further suggests that options 1 and 3 have nearly the same performance. Therefore, option 3 is an efficient algorithm that maintains good performance while reducing computation cost. Option 2 has the lowest search cost ( ∼ 4 rounds per iteration). However, its performance is not as good as option 1 or 3 as

0

10

20

30

40

50

Upper Iterations

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=0.0 (Opt-1)

=0.0 (Opt-2)

=0.0 (Opt-3)

(a) Different Opt (BiSLS-SGD)

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=0.0 (Opt-1)

=0.0 (Opt-2)

=0.0 (Opt-3)

(b) Different Opt (BiSLS-Adam)

0

10

20

30

40

50

Upper Iterations

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

BiSLS-SGD

=1.0

BiSLS-SGD

=2.0

BiSLS-SGD

=5.0

BiSLS-SGD

=7.0

BiSLS-SGD

=10.0

(a) Different

η

s (BiSLS-SGD)

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=1.0

=2.0

=5.0

=7.0

=10.0

(b) Different

η

s (BiSLS-Adam)

observed in Figure 10. Moreover, the average lower-level search cost is only 1 round per iteration when option 1 is used (see Figure 12).

Sensitivity on δ As mentioned in Sec 2, due to the stochastic error in hypergradient computation, further complicated by the approximation error of y ∗ ( x ) (see (14)), a learning rate is not guaranteed to be found in the bi-level case. Specifically, this is in contrast to the single-level convex problems. To avoid this, we introduce in (14) a δ slack to give some tolerance to such errors. Here, we give a thorough investigation of the effects of δ on performance. We vary its magnitude across 6 orders for both reset options 1 and 3 (see Algorithm 2 and discussions on reset above). We observe that despite a large difference on the magnitudes of δ , they all share very similar performance for both BiSLS-SGD and BiSLS-Adam: see Figures 13 and 14. We summarize the key fins in this section as follows: ① The option 3 in reset has good empirical performance (outperforms option 2) and is an effective way to reduce search cost (Figure 10, ??); ② BiSLS is highly robust to different choices of η in option 3 and δ in (14) (Figure 11, 13, 14).

0

1000

2000

3000

4000

5000

b

, 0

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

Lower-Level Search Cost

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=0.0 (Opt-1)

=10 6  (Opt-1)

=10 5  (Opt-1)

=10 4  (Opt-1)

=10 3  (Opt-1)

=10 2  (Opt-1)

(a) Different

δ

s/Opt-1 (BiSLS-SGD)

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=0.0 (Opt-1)

=10 6  (Opt-1)

=10 5  (Opt-1)

=10 4  (Opt-1)

=10 3  (Opt-1)

=10 2  (Opt-1)

(b) Different δ s/Opt-1 (BiSLS-Adam)

B.3 Data distillation objective and additional results

We let L S ( w ) denote the loss evaluated on dataset S with model weights w . The objective of data distillation can be expressed as a BO problem as follows:

where ˜ V is of the same size as D and subsampled from the entire (original) dataset V . The solution D ∗ is the distilled data, e.g. 9 MNIST digits each corresponding to a different label. In figure 15a, we show the performance of BiSPS for different values of α b, 0 in comparison with BiSLS-SGD, and observe that BiSLS-SGD has better performance. In 15b, we show the results when we increase the number of lower-level iterations (T) from 20 to 50. As observed for T = 20 (in Figure 8), BiSLS-SGD here also outperforms a fine-tuned Adam or SGD.

B.4 1-sample or 2-samples versions of algorithms for convex and bi-level optimization

We provide additional results to compare the performance of 1-sample and 2-samples (one for computing the gradient and the other for computing the step size) versions of our algorithms for

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=0.0 (Opt-3)

=10 6  (Opt-3)

=10 5  (Opt-3)

=10 4  (Opt-3)

=10 3  (Opt-3)

=10 2  (Opt-3)

(a) Different

δ

s/Opt-3 (BiSLS-SGD)

0

10

20

30

40

50

Upper Iterations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Validation Loss

=0.0 (Opt-3)

=10 6  (Opt-3)

=10 5  (Opt-3)

=10 4  (Opt-3)

=10 3  (Opt-3)

=10 2  (Opt-3)

(b) Different

δ

s/Opt-3 (BiSLS-Adam)

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

4

10

3

10

2

10

1

10 0

Validation Loss

BiSPS - 0.1/

k

+1

BiSPS - 1.0/

k

+1

BiSPS - 10.0/

k

+1

BiSPS - 100.0/

k

+1

BiSLS-SGD

(a) Different

α

b,

0

s (BiSPS)

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

4

10

3

10

2

10

1

10 0

Validation Loss

Adam

=0.001

Adam

=0.01

Adam

=0.1

SGD

=0.001

SGD

=0.01

SGD

=0.1

BiSLS-SGD

(b) T = 50, Identity

SPSB and BiSPS used for single-level and bi-level optimization, respectively. In the single-level case (Figure 16), we observe that 2-samples SPSB performs just as well as 1-sample SPSB . Interestingly, we observe that their step sizes also follow a similar pattern. That is: an initial increase followed by a regime where γ k = f i k ( x k ) -l ∗ i k c ∥∇ f i k ( x k ) ∥ 2 is frequently used, and eventually changes to decayingstep SGD. This seems to also match with Theorem 2 where a transition point for SPSB ( k 0 = max { 1 , ⌈ γ 0 /ω ⌉ -1 } , w = 1 2 cL max , γ 0 ≥ 1 µ ) is predicted. At the same time, we also note that (perhaps, unsurprisingly) the 2-samples version seems to have a slightly more oscillatory behavior than the 1-sample version as shown in Figure 16. SLSB with either 1-sample or 2-samples also result in a similar performance and step size. Overall, despite the requirements of Theorems 1 and 2 for a 2-samples assumption, the empirical performance of 1-sample and 2-samples for either SPSB or SLSB appears to be very similar. Moving on to the bi-level case, recall that Theorems 3 and 4 require the 2-samples assumption (i.e., α k independent of h k f ) for the upper-level learning rate. We empirically verify this assumption with both hyper-representation learning and data distillation experiments. For hyper-representation learning experiments in Figure 17, BiSPS with either 1-sample or 2-samples for different values of α b, 0 show similar performance. In fact, for α b, 0 = 0 . 1 we even observe that the 2-samples variant outperforms the 1-sample BiSPS. For data distillation experiments in Figure 18, the performances of 1-sample and 2-samples BiSPS are similar to each other when

α b, 0 = 10 . 0 or α b, 0 = 50 . 0 . In general, the performance difference between 1-sample and 2-samples in the single-level or bi-level settings is small.

0

1000

2000

3000

4000

5000

Iterations

10

1

10 0

10 1

Train Loss

Const SGD

SLSB

SLSB (2 samples)

SPSB

SPSB (2 samples)

10 0

10 1

10 2

10 3

10 4

Iterations

10 0

10 1

10 2

10 3

Step Sizes

Const SGD

SLSB

SLSB (2 samples)

SPSB

SPSB (2 samples)

(a) Train loss

(b) Step size

0

10

20

30

40

50

Upper Iterations

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Validation Loss

Decaying-step SGD

BiSPS (1-sample)

b

, 0 =0.01

BiSPS (1-sample)

b

, 0 =0.05

BiSPS (1-sample)

b

, 0 =0.1

BiSPS (2-sample)

b

, 0 =0.01

BiSPS (2-sample)

b

, 0 =0.05

BiSPS (2-sample)

b

, 0 =0.1

0

250

500

750

1000

1250

1500

1750

2000

Upper Iterations

10

2

10

1

10 0

Validation Loss

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the bilevel optimization problem, an in particular, focuses on developing effective learning rate schemes for bilevel optimization. In specific, the authors propose two adaptive step-sizes methods named stochastic line search (SLS) and stochastic Polyak step size (SPS) as variants of methods such as SPSmax and DecSPS. Compared to existing approaches, the proposed methods do not require the step-sizes to be monotonic by replacing the constant $\\gamma_{b,0}$ by a non-increasing $\\gamma_{b,k}$. The adaptive stepsizes are further applied to bilevel optimization on both upper and lower levels. The convergence analysis is provided for SPS and SLS on single-level problems and also for bilevel problems. Simple experiments are provided.\n\nSTRENGTHS:\n1. Studying adaptive learning rates in bilevel optimization is interesting, and has not been explored well in the literature. \n\n2. The proposed adaptive learning rates are more practical than existing methods like SPSmax and DecSPS with milder requirements.\n\nWEAKNESSES:\n1. The paper is well written and hard to follow. For example, there are quite a few assumptions and requirements in the paragraphs of describing the proposed methods. There are also many notations and inequalities in Section 2. It makes me a little bit hard to follow the principle of the designs. \n\n2. The adaptive learning rates seem to introduce more hyperparameters such as $l^*_{i_k}, c_{k},\\gamma_{b,k}$. It makes me wonder how it can be meaningful in practice given more efforts in tuning such parameters. In addition, no sensitivity analysis is provided for such parameters in experiments. \n\n3. How to find the approximate $l^*_{I,k}$ of the lowest function value? Is the choice of this quantity important in terms of performance? A more comprehensive empirical justification should be provided.\n\n4. In terms of the complexity, it seems to me the proposed stepsizes cannot provide improved sample or computational complexity in theory. Maybe I miss something, but I think the authors can elaborate on this\n\n5. The experiments are not convincing enough. Since the biggest motivation lies in the design of adaptive learning rates, the benefits may come from the empirical side. Thus, it would be better to provide more comprehensive experiments on practical NN architecture and larger datasets for the validation. However, the current experiments are rather toy examples. If the contribution lies in the theoretical side, it fails to show improved complexity performance. \n\nOverall, I like the topic of adaptive learning rate for bilevel problems. However, the current theory and experiments are not convincing enough, and how important the proposed methods also are not clear to me. For this reason, I am on the negative side for the current version. I am also open to change my mind if more convincing experiments (e.g., how to select extra parameters, more datasets and backbones, sensitivity analysis, more problems) can be provided.\n\nQUESTIONS:\nPlease see Weakness part.\n\nLIMITATIONS:\nPlease see Weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an adaptive step size algorithm for bi-level optimization, which improves the shortcomings of the existing BO method that requires careful adjustment of the upper and lower learning rate, and gives a proof of convergence. Experiments also verify the robustness of the method to learning rate.\n\nSTRENGTHS:\n1.This paper presents an adaptive and robust bi-level optimization algorithm for adaptive step size, which can obtain a good set of step sizes without prior knowledge and careful modulation.\n2.This algorithm is compatible with the accelerated solver Adam in addition to SGD, improving computational efficiency.\n3.The analysis framework of this algorithm more generally unifies SPS and SLS.\n\nWEAKNESSES:\n1.Theorem 1 only states that f is convex rather than strongly convex, but the full text lacks explanation for the case of multiple solutions at the lower level\n2.Potential computational burden. Although the author states that only a small number of matrix vector multiplication and addition operations are usually used to approximate matrix inverses, this still implies additional computational complexity and there is a problem: the trade-off between the performance loss caused by approximation and the improvement of computational efficiency. Specifically, the lack of time analysis in the charts in the paper exacerbates this concern.\n\nQUESTIONS:\n1.The drawing of  figure is difficult to read. For example, the line of beta=10.0 in Figure 1 is not fully drawn, different colors and lines are mixed in Figure 2 but have no additional meaning, and the second half of the right subgraph in Figure 3 is too complex, resulting in some lines being unrecognizable.\n2. The light colored parts in the figure do not seem to represent the three standard deviations, but rather the upper and lower bounds?\n3.I noticed that the sequence of upper and lower bounds for the declared step size in the paper needs to be appropriately controlled, so my point of interest is whether the decay rate is not 1/ sqrt {k+1}? Or can I change the decay rate to change the rate of convergence?\n\nLIMITATIONS:\nYes, the author clearly stated the assumptions of the algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies adaptive step-size methods for both single-level optimization and bilevel optimization. The authors propose two novel variants of stochastic line search (SLS) and stochastic Polyak step size (SPS), and they unify these variants into a general envelop strategy. Importantly, these variants are simpler to implement and demonstrate good empirical performance, particularly in non-interpolating scenario. By using the unified envelop strategy, the authors also propose a bi-level line-search algorithm BiSLS-Adam/SGD with convergence guarantees, which demonstrates empirical robustness and generalizes well.\n\nSTRENGTHS:\n1. Both adaptive step-size methods and bilevel optimization are currently active topics. The investigation of auto-tune step sizes for bilevel optimization algorithms is under-explored, and the studied topic in work is interesting and important. \n2. The illustrations in Figures 1-5 are helpful in understanding the contributions.\n3. The newly proposed variants of stochastic line search (SLS) and stochastic Polyak step size (SPS) in this work are novel and easy to understand. Moreover, these variants can be unified into a general envelop-type step size, and their effectiveness in the context of single-level optimization and bilevel optimization is well-supported by theoretical results and extensive experiments.\n\nWEAKNESSES:\nSee the Limitations below.\n\nQUESTIONS:\nQ1: Can the authors explain in more details why it is possible to set the number of inner steps to be 1 in Line 213? \n\nMinor Comments:\n1. For Equation (2), the $+$ should be $-$.\n\n2. In Equation (13), $\\nabla_y^2 yg$ should be $\\nabla_{yy}^2 g$.\n\n3. In Line 175, $\\nabla f_x$ should be $\\nabla_x f$.\n\n4. The quadratic functions in Figure 2 and Section B.1 do not satisfy the gradient bounded assumption in Assumption 2. \n\n5. The $+$ in Equation (17) should be $-$.\n\nLIMITATIONS:\nThe authors discussed some of the limitations on single-level optimization in the conclusion section of the paper. \n\nOn bilevel optimization, a limitation is the lower-level strong convexity in Assumption 3. It would be of interest to investigate whether this condition can be relaxed or removed, taking into account recent advancements in the field, such as those presented in [1, 2,3].\n\n[1] B. Liu et al. “Bome! bilevel optimization made easy: A simple first-order approach.” NeurIPS 2022.\n\n[2] R. Liu et al. “Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity.” ICML 2023.\n\n[3] H. Shen and T. Chen, “On penalty-based bilevel gradient descent method.” ICML 2023.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper introduces the use of stochastic adaptive-step size methods, namely stochastic Polyak step size (SPS) and stochastic line search (SLS), for bi-level optimization. This approach addresses the challenge of tuning both the lower and upper-level learning rates.

STRENGTHS:
1. SLS and SPS can be seen as special instances of general family of methods with an envelope-type step-size. 
2. The unified envelope strategy enables the algorithm development and convergence analysis.

WEAKNESSES:
The paper compares the proposed algorithms to vanilla SGD or Adam versions, but it does not provide a comprehensive comparison with other existing algorithms for bi-level optimization. This may limit the understanding of the relative performance of the proposed algorithms in the broader context.

QUESTIONS:
What is the overhead of using the proposed algorithm to tune the step size?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07