Title: Achieving $\mathcal{O}(\epsilon^{-1.5})$ Complexity in Hessian/Jacobian-free Stochastic Bilevel Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stochastic bilevel optimization

Hessian-free algorithms

near-optimal complexity
Abstract: In this paper, we revisit the bilevel optimization problem, in which the upper-level objective function is generally nonconvex and the lower-level objective function is strongly convex. Although this type of problem has been studied extensively, it still remains an open question how to achieve an $\mathcal{O}(\epsilon^{-1.5})$ sample complexity in Hessian/Jacobian-free stochastic bilevel optimization without any second-order derivative computation. To fill this gap, we propose a novel Hessian/Jacobian-free bilevel optimizer named FdeHBO, which features a simple fully single-loop structure, a projection-aided finite-difference Hessian/Jacobian-vector approximation, and momentum-based updates. Theoretically, we show that FdeHBO requires $\mathcal{O}(\epsilon^{-1.5})$ iterations (each using $\mathcal{O}(1)$ samples and only first-order gradient information) to find an $\epsilon$-accurate stationary point. As far as we know, this is the first Hessian/Jacobian-free method with an $\mathcal{O}(\epsilon^{-1.5})$ sample complexity for nonconvex-strongly-convex stochastic bilevel optimization.

Achieving O ( ϵ -1 . 5 ) Complexity in Hessian/Jacobian-free Stochastic Bilevel Optimization

Abstract

In this paper, we revisit the bilevel optimization problem, in which the upper-level objective function is generally nonconvex and the lower-level objective function is strongly convex. Although this type of problem has been studied extensively, it still remains an open question how to achieve an O ( ϵ -1 . 5 ) sample complexity in Hessian/Jacobian-free stochastic bilevel optimization without any second-order derivative computation. To fill this gap, we propose a novel Hessian/Jacobianfree bilevel optimizer named FdeHBO, which features a simple fully single-loop structure, a projection-aided finite-difference Hessian/Jacobian-vector approximation, and momentum-based updates. Theoretically, we show that FdeHBO requires O ( ϵ -1 . 5 ) iterations (each using O (1) samples and only first-order gradient information) to find an ϵ -accurate stationary point. As far as we know, this is the first Hessian/Jacobian-free method with an O ( ϵ -1 . 5 ) sample complexity for nonconvex-strongly-convex stochastic bilevel optimization.

1 Introduction

Bilevel optimization has drawn intensive attention due to its wide applications in meta-learning [18, 4, 50], hyperparameter optimization [18, 52, 14], reinforcement learning [35, 27], signal process [36, 16] and communication [31] and federated learning [59]. In this paper, we study the following stochastic bilevel optimization problem.

where the upper- and lower-level objective functions f ( x, y ) and g ( x, y ) take the expectation form w.r.t. the random variables ξ and ζ , and are jointly continuously differentiable. In this paper, we focus on the nonconvex-strongly-convex bilevel setting, where the lower-level function g ( x, · ) is strongly convex and the upper-level function Φ( x ) is nonconvex. This class of bilevel problems has been studied extensively from the theoretical perspective in recent years. Among them, [19, 30, 3, 62] proposed bilevel approaches with a double-loop structure, which update x and y in a nested manner. Single-loop bilevel algorithms have also attracted significant attention recently [27, 62, 34, 25, 9, 40, 11] due to the simple updates on all variables simultaneously. Among them, the approaches in [62, 34, 25] have been shown to achieve an O ( ϵ -1 . 5 ) sample complexity, but with expensive evaluations of Hessian/Jacobian matrices or Hessian/Jacobian-vector products.

Hessian/Jacobian-free bilevel optimization has received increasing attention due to its high efficiency and feasibility in practical large-scale settings. In particular, [15, 48, 61] directly ignored the

computation of all second-order derivatives. However, such eliminations may lead to performance degeneration [2, 13], and can vanish the hypergradient for bilevel problems with single-variable upperlevel function, i.e., Φ( x ) = f ( y ∗ ( x )) . [56, 23] proposed zeroth-order approaches that approximate the hypergradient using only function values. These methods do not have a convergence rate guarantee. Recently, several Hessian/Jacobian-free bilevel algorithms were proposed by [42, 57, 53, 8] by reformulating the lower-level problem into the optimality-based constraints such as g ( x, y ) ≤ min y g ( x, y ) . However, these approaches all focus on the deterministic setting, and their extensions to the stochastic setting remain unclear. In the stochastic case, [58] proposed evolution strategies based bilevel method, which achieves a high sample complexity of O ( p 2 ϵ -2 ) , where p is the problem dimension. Most recently, [37] proposed two fully first-order (i.e., Hessian/Jacobian-free) valuefunction-based stochastic bilevel optimizer named F 2 SA and its momentum-based version F 3 SA with a single-loop structure, which achieves sample complexities of O ( ϵ -3 . 5 ) and O ( ϵ -2 . 5 ) , respectively. However, there is still a large gap of ϵ -1 , compared to the optimal complexity of O ( ϵ -1 . 5 ) . Then, an important open question, as recently proposed by [37], is:

Can we achieve an O ( ϵ -1 . 5 ) sample/gradient complexity for nonconvex-strongly-convex bilevel optimization using only first-order gradient information?

1.1 Our Contributions

In this paper, we provide an affirmative answer to the above question by proposing a new Hessian/Jacobian-free stochastic bilevel optimizer named FdeHBO with three main features. First, FdeHBO takes the fully single-loop structure with momentum-based updates on three variables y, v and x for optimizing the lower-level objective, the linear system (LS) of the Hessian-inverse-vector approximation, and the upper-level objective, respectively. Second, FdeHBO contains only a single matrix-vector product at each iteration, which admits a simple first-order finite-difference estimation. Third, FdeHBO involves an auxiliary projection on v updates to ensure the boundedness of the Hessian-vector approximation error, the variance on momentum-based iterates, and the smoothness of the LS loss function. Our detailed contributions are summarized below.

Theoretically, we show that FdeHBO achieves a sample/gradient complexity of O ( ϵ -1 . 5 ) and an iteration complexity of O ( ϵ -1 . 5 ) to achieve an ϵ -accurate stationary point, both of which outperforms existing results by a large margin. As far as we know, this is the first-known method with an O ( ϵ -1 . 5 ) sample complexity for nonconvex-strongly-convex stochastic bilevel optimization using only first-order gradient information.

Technically, we show that the auxiliary projection can provide more accurate iterates on v in solving the LS problem without affecting the overall convergence behavior, and in addition, provide a novel characterization of the gradient estimation error and the iterative progress during the v updates, as well as the impact of the y and v updates on the momentum-based hypergradient estimation, all of which do not exist in previous studies. In addition, the finite-different approximations make the unbiased assumptions in the momentum-based gradients no longer hold, and hence a more careful analysis is required.

As a byproduct, we further propose a fully single-loop momentum-based method named FMBO in the small-dimensional case with matrix-vector-based hypergradient computations. Differently from existing momentum-based bilevel methods with O (log 1 ϵ ) Hessian-vector evaluations per iteration, FMBO contains only a single Hessian-vector computation per iteration with the same O ( ϵ -1 . 5 ) sample complexity.

We also want to emphasize our technical differences from previous works as below.

Comparison to existing momentum-based methods. Previous momentum-based methods [62, 34] solve the linear system (LS) to a high accuracy of O ( ϵ ) , whereas our algorithm includes a new estimation error by the single-step momentum update on LS, and this error is also correlated with the lower-level updating error and the hypergradient estimation error. In addition, due to the finitedifference approximation, the stochastic gradients in all three updates on y, v, x are no longer unbiased. Non-trivial efforts need to be taken to deal with such challenges and derive the optimal complexity.

Comparison to existing fully single-loop methods. The analysis of the single-step momentum update in solving the LS requires the smoothness of the LS loss function and the boundedness of LS gradient variance, both of which may not be satisfied. To this end, we include an auxiliary projection and show it not only guarantees these crucial properties, but also, in theory, provides an improved per-iteration progress. As a comparison, existing works on fully single-loop stochastic bilevel optimization such as SOBA/SABA [11] and FLSA [40] with a new time scale to update the LS problem often assume that the iterates on v are bounded during the process. We do not require such assumptions. In addition, an O ( ϵ -1 . 5 ) complexity has not been established for fully single-loop bilevel algorithms yet.

1.2 Related Work

Bilevel optimization methods. Bilevel optimization, which was first introduced by [6], has been studied for decades. By replacing the lower-level problem with its optimality conditions, [26, 20, 54, 55] reformulated the bilevel problem to the single-level problem. Gradient-based bilevel methods have shown great promise recently, which can be divided into approximate implicit differentiation (AID) [12, 49, 41, 3] and iterative differentiation (ITD) [47, 17, 15, 52, 21] based approaches. Recently, a bunch of stochastic bilevel algorithms has been proposed via Neumann series [9, 30], recursive momentum [62, 28, 25] and variance reduction [62, 11]. Theoretically, the convergence of bilevel optimization has been analyzed by [18, 52, 45, 19, 30, 27, 3, 11]. Among them, [29] provides the lower complexity bounds for deterministic bilevel optimization with (strongly-)convex upper-level functions. [25, 9, 62, 34] achieved the near-optimal sample complexity with second-order derivative computations. Some works studied deterministic bilevel optimization with convex or Polyak-Lojasiewicz (PL) lower-level problems via mixed gradient aggregation [51, 46, 39], logbarrier regularization [45], primal-dual method [57] and dynamic barrier [63]. More results and details can be found in the survey by [44].

Hessian/Jacobian-free bilevel optimization. Some Hessian/Jacobian-free bilevel optimization methods have been proposed recently by [58, 43, 15, 23, 56, 48]. Among them, FOMAML [15, 48] and MUMOMAML [61] directly ignore the computation of all second-order derivatives. Several Hessian/Jacobian-free bilevel algorithms were proposed by [42, 57, 53, 8] by replacing the lowerlevel problem with the optimality conditions as the constraints. However, these approaches focus only on the deterministic setting. Recently, zeroth-order stochastic approaches have been proposed for the hypergradient estimation [56, 23, 58]. Theoretically, [58] analyzed the convergence rate for their method. [37] proposed fully first-order stochastic bilevel optimization algorithms based on the valuefunction-based lower-level problem reformulation. This paper proposes a new Hessian/Jacobian-free stochastic bilevel algorithm that for the first time achieves an O ( ϵ -1 . 5 ) sample complexity.

Momentum-based bilevel approaches. The recursive momentum technique was first introduced by [10, 60] for minimization problems to improve the SGD-based updates in theory and in practice. This technique has been incorporated in stochastic bilevel optimization [34, 9, 24, 25, 62]. These approaches involve either Hessian-inverse matrix computations or a subloop of a number of iterations in the Hessian-inverse-vector approximation. As a comparison, our proposed method takes the simpler fully single-loop structure, and only uses the first-order gradient information.

Finite-difference matrix-vector approximation. The finite-difference matrix-vector estimation has been studied extensively in the problems of escaping from saddle points [1] [7] (some other related works can be found therein), neural architecture search (NAS) [43] and meta-learning [13]. However, such finite-different estimation can be sensitive to the selection of the smoothing constant, and may suffer from some numerical issues in practice [32][33], such as rounding errors. It is interesting but still open to developing a fully first-order stochastic bilevel optimizer without the finite-different matrix-vector estimation. We would like to leave it for future study.

Algorithm 1 Hessian/Jacobian-free Bilevel Optimizer via Projection-aided Finite-difference Estimation

1: Input: { α t , β t , λ t } T -1 t =0 and r v .

2: Initialize:

3: for t = 0 , 1 , 2 , ..., T -1 do

4: Compute the gradient estimator h g t by eq. (6) and update y t +1 = y t -β t h g t .

5: Compute the gradient estimator h R t by eq. (7) and update w t +1 = v t -λ t ˜ h R t .

6: Set v t +1 = { w t +1 , ∥ w t +1 ∥ ≤ r v ; r v w t +1 ∥ w t +1 ∥ , ∥ w t +1 ∥ > r v .

7: Compute the gradient estimator h f t by eq. (10) and update x t +1 = x t -α t ˜ h f t .

8: end for

2 Algorithms

In this section, we first describe the hypergradient computation in bilevel optimization, and then present the proposed Hessian/Jacobian-free bilevel method.

2.1 Hypergradient Computation

One major challenge in bilevel optimization lies in computing the hypergradient ∇ Φ( x ) due to the implicit and complex dependence of the lower-level minimizer y ∗ on x . To see this, if g is twice differentiable, ∇ y g is continuously differentiable and the Hessian ∇ 2 yy g ( x, y ∗ ( x )) is invertible, using the implicit function theorem (IFT) [22, 5], the hypergradient ∇ Φ( x ) takes the form of

Note that the hypergradient in eq. (2) requires computing the exact solution y ∗ and the expensive Hessian inverse [ ∇ 2 yy g ( x, y ∗ )] -1 . To approximate this hypergradient efficiently, we define the following (stochastic) hypergradient surrogates as

where v ∈ R q is an auxiliary vector to approximate the Hessian-inverse-vector product in eq. (2), and ¯ ∇ f ( x, y, v ; ξ ) can be regarded as a stochastic version of ¯ ∇ f ( x, y, v ) . Based on eq. (3), one needs to find an efficient estimate y of y ∗ , e.g., via an iterative optimization procedure, as well as a feasible estimate v of the solution v ∗ = [ ∇ 2 yy g ( x, y )] -1 ∇ y f ( x, y ) of a linear system (LS) (equivalently quadratic programming) whose generic loss function is given by

where the gradient of R ( x, y, v ) w.r.t. v is given by

Similarly to eq. (3), we also define ∇ v R ( x, y, v ; ψ ) = ∇ 2 yy g ( x, y ; ψ ) v - ∇ y f ( x, y ; ψ ) over any sample ψ as a stochastic version of ∇ v R ( x, y, v ) in eq. (5). It can be seen from eq. (3), eq. (4) and eq. (5) that the updates on the LS system involve the Hessian- and Jacobian-vector products, which can be computationally intractable in the high-dimensional case. In the next section, we propose a novel stochastic Hessian/Jacobian-free bilevel algorithm.

2.2 Hessian/Jacobian-free Bilevel Optimizer via Projection-aided Finite-difference Estimation

As shown in Algorithm 1, we propose a fully single-loop stochastic Hessian/Jacobian-free bilevel optimizer named FdeHBO via projection-aided finite-difference estimation. It can be seen that FdeHBO first minimizes the lower-level objective function g ( x, y ) w.r.t. y by running a single-step momentum-based update as y t +1 = y t -β t h g t , where β t is the stepsize and h g t is the momentum-based gradient estimator that takes the form of

where η g t ∈ [0 , 1] is a tuning parameter. The next key step is to deal with the LS problem via solving the quadratic problem eq. (4) as w t +1 = v t -λ t ˜ h R t , with the momentum-based gradient ˜ h R t given by

where ˜ ∇ v R is a Hessian-free version of the LS gradient ∇ v R in eq. (5), given by

Note that in the above eq. (8), ˜ H ( x t , y t , v t , δ ϵ ; ψ t ) is the finite-difference estimation of the Hessianvector product ∇ 2 yy g ( x t , y t ; ψ t ) v t , which takes the form of

where δ ϵ > 0 is a small constant. Note that in eq. (9), if the iterative v t is unbounded, the approximation error between ˜ H and ∇ 2 yy g ( x t , y t ; ψ t ) v t can be uncontrollable as well. We further prove lemma 5 in appendix B that the bound of this gap relies on ∥ v t ∥ and δ but it is independent of the dimension of y t . To this end, after obtaining w t +1 , our key step in line 6 introduces an auxiliary projection on a ball (which can be generalized to any convex and bounded domain) with a radius of r v as

This auxiliary projection guarantees the boundedness of v t , t = 0 , ..., T -1 , which serves three important purposes. First, it ensures the smoothness of the LS loss function R ( x, y, v ) in eq. (5) w.r.t. all x, y and v , which is crucial in the convergence analysis of the momentum-based updates. Second, the boundedness of v t also ensures that the estimation variance of the stochastic LS gradient ∇ v R ( x t , y t , v t ; ψ t ) does not explode. Third, it guarantees the error of the finite-difference Hessianvector approximation to be sufficiently small with proper δ ϵ . We will show later that under a proper choice of the radius r v , this auxiliary projection provides better per-step progress, and the proposed algorithm achieves a stronger convergence performance. Finally, for the upper-level problem, the momentum-based hypergradient estimate ˜ h f t is designed as

where ˜ ∇ f ( x, y, v, δ ϵ ; ¯ ξ t ) is the fully first-order hypergradient estimate evaluated at two consecutive iterates ( x t , y t , v t ) and ( x t -1 , y t -1 , v t -1 ) is given by

and ˜ J ( x, y, v, δ ϵ ; ¯ ξ t ) is the finite-difference Jacobian-vector approximation given by

Note that ˜ ∇ v R and ˜ ∇ f are biased estimators of the gradients ∇ v R and ¯ ∇ f , which further complicates the convergence analysis on the momentum-based updates because the conventional analysis on the recursive momentum requires the unbiased gradient estimation to ensure the variance reduction effect. By controlling the perturbation δ ϵ properly, we will show that FdeHBO can achieve an O ( ϵ -1 . 5 ) convergence and complexity performance without any second-order derivative computation.

2.3 Extension to Small-Dimensional Case

As a byproduct of our proposed FdeHBO, we further propose a fully single-loop momentum-based bilevel optimizer (FMBO), which is more suitable in the small-dimensional case without finitedifference approximation. As shown in Algorithm 2, FMBO first takes the same lower-level updates

Algorithm 2 Fully Single-loop Momentum-based Bilevel Optimizer (FMBO)

1: Input: { α t , β t , λ t } T -1 t =0 , and r v . 2: Initialize: 3: for t = 0 , 1 , 2 , ..., T -1 do 4: Compute the gradient estimator h g t by eq. (6) and update y t +1 = y t -β t h g t . 5: Compute the gradient estimator h R t by eq. (12) and update w t +1 = v t -λ t h R t . 6: Set v t +1 = { w t +1 , ∥ w t +1 ∥ ≤ r v ; r v w t +1 ∥ w t +1 ∥ , ∥ w t +1 ∥ > r v . 7: Compute the gradient estimator h f t by eq. (13) and update x t +1 = x t -α t h f t . 8: end for

on y t as in eq. (6). Then, it solves the LS problem as w t +1 = v t -λ t h R t , where the momentum-based gradient estimator is given by

where differently from FdeHBO, we here use the precise gradient ∇ v R without finite-difference approximation. Similarly to FdeHBO, we add an auxiliary projection on the v t updates to ensure the LS smoothness and bounded variance. Finally, for the upper-level problem, we optimize x t based on a momentum-based update as x t +1 = x t -α t h f t with the hypergradient estimator

where η f t ∈ [0 , 1] is a tuning parameter. Similarly, we directly use the hypergradient estimate in eq. (3) without the finite-difference estimation. We note that compared to existing momentum-based algorithms [62, 34] that contains O (log 1 ϵ ) steps in solving the LS problem, FMBO takes the fully single-loop structure with a single-step momentum-based acceleration on the LS updates.

3 Main Results

3.1 Assumptions and Definitions

We make the following standard assumptions for the upper- and lower-level objective functions, as also adopted by [30, 9, 34]. The following assumption imposes the Lipschitz condition on the upper-level function f ( x, y ) .

Assumption 1. For any x ∈ R d x and y ∈ R d y , there exist positive constants L f x , L f y , C f x and C f y such that ∇ x f ( x, y ) and ∇ y f ( x, y ) are L f x - and L f y -Lipschitz continuous w.r.t. ( x, y ) , and ∥∇ x f ( x, y ) ∥ 2 ≤ C f x , ∥∇ y f ( x, y ) ∥ 2 ≤ C f y .

The following assumption imposes the Lipschitz condition on the lower-level function g ( x, y ) .

Assumption 2. For any x ∈ R d x and y ∈ R d y , there exist positive constants µ g , L g , L g xy , L g yy , C g xy , C g yy such that

Function g ( x, y ) is twice continuously differentiable;

Function g ( x, · ) is µ g -strongly-convex;

The derivatives ∇ y g ( x, y ) , ∇ 2 xy g ( x, y ) and ∇ 2 yy g ( x, y ) are L g -, L g xy - and L g yy -Lipschitz continuous w.r.t. ( x, y ) ;

∥∇ 2 xy g ( x, y ) ∥ 2 ≤ C g xy and ∥∇ 2 yy g ( x, y ) ∥ 2 ≤ C g yy .

The following assumption is adopted for the stochastic functions f ( x, y ; ξ ) and g ( x, y ; ζ ) .

Assumption 3. Assumptions 1 and 2 hold for f ( x, y ; ξ ) and g ( x, y ; ζ ) for ∀ ξ and ζ . Moreover, we assume that there exist positive constants σ f x , σ f y , σ g , σ g xy and σ g yy such that

Definition 1. We say ¯ x is an ϵ -accurate stationary point of a function Φ( x ) if E ∥∇ Φ(¯ x ) ∥ 2 ≤ ϵ , where ¯ x is the output of an optimization algorithm.

3.2 Convergence and Complexity Analysis of FdeHBO

We further provide the convergence analysis for the proposed Hessian/Jacobian-free FdeHBO algorithm. We first characterize several estimation properties of FdeHBO. Let e f t := ˜ h f t -∇ f ( x t , y t , v t ) -∆( x t , y t , v t ) denote the hypergradient estimation error.

Proposition 1. Under Assumption 3, the iterates of the outer problem by Algorithm 1 satisfy

The hypergradient estimator error O ( E ∥ e f t +1 ∥ 2 ) contains three main components. The first term [(1 -η f t +1 ) 2 +4 L g xy r 2 v δ ϵ ] E ∥ e f t ∥ 2 indicates the per-iteration improvement induced by the momentumbased update, the error term α 2 t E ∥ h f t ∥ 2 is caused by the x t updates, the error term O ( β 2 t E ( ∥ e g t ∥ 2 + ∥∇ y g ( x t , y t ) ∥ 2 )) is caused by solving the lower-level problem, and the new error term O ( λ 2 t E ( ∥ e R t ∥ 2 + L 2 g ∥ v t -v ∗ t ∥ 2 )) is induced by the one-step momentum update on the LS problem, which does not exist in previous momentum-based bilevel methods [62, 34, 25] that solve the LS problem to a high accuracy. Also note that the errors 4 L g xy r 2 v δ ϵ E ∥ e f t ∥ 2 and 4 L g xy r 2 v δ ϵ +16 L 2 g xy r 4 v δ 2 ϵ are caused by the finite-difference approximation error. Fortunately, by choosing the perturbation level δ ϵ in these two terms to be properly small, it can guarantee the descent factor (1 -η f t +1 ) 2 +4 L g xy r 2 v δ ϵ to be at an order of (1 -O ( η f t +1 )) 2 , and hence the momentum-based variance reduction effect is still applied.

As shown in Proposition 2, the LS gradient estimation error e R t +1 contains an iteratively improved error component [ (1 -η R t +1 ) 2 (1+96 L 4 g λ 2 t )+4 L g yy r 2 v δ ϵ ] E ∥ e R t ∥ 2 for the stepsize λ t and the approximation factor δ ϵ sufficiently small, a finite-difference approximation error O ( δ ϵ ) as well as an approximation error O ( λ 2 t E ∥ v t -v ∗ t ∥ 2 ) for solving the LS problem. The next step is to upper-bound E ∥ v t -v ∗ t ∥ 2 .

Proposition 3. Under the Assumption 1, 2, the iterates of the LS problem by Algorithm 1 satisfy

E ∥ v t +1 -v ∗ t +1 ∥ 2

Based on the above important properties, we now provide the general convergence theorem for FdeHBO.

Theorem 1. Suppose Assumptions 1, 2, 3 and Lemma 3 are satisfied. Choose r v ≥ C fy µ g and set

and δ ϵ ≤ min { c η f ,c η R } 8( L gxy r 2 v ( w + T -1) 2 / 3 ) , where the constants w , c β , c λ , c η f , c η R and c η g are defined in eq. (67) in the appendix. Then, the iterates generated by Algorithm 1 satisfy

Corollary 1. Under the same setting of Theorem 1, FdeHBO requires ˜ O ( ϵ -1 . 5 ) samples and gradient evaluations, respectively, to achieve an ϵ -accurate stationary point.

It can be seen from Corollary 1 that the proposed FdeHBO achieves an ˜ O ( ϵ -1 . 5 ) sample complexity without any second-order derivative computation. As far as we know, this is the first Hessian/Jacobianfree stochastic bilevel optimizer with an ˜ O ( ϵ -1 . 5 ) sample complexity.

3.3 Convergence and Complexity Analysis of FMBO

In this section, we analyze the convergence and complexity of the simplified FMBO method.

Theorem 2. Suppose Assumptions 1, 2 and 3 are satisfied. Choose r v ≥ C fy µ g and set parameters

where w , c β , c λ , c η f , c η R and c η g are defined in eq. (33) in the appendix. The iterates generated by Algorithm 2 satisfy

Theorem 2 shows that the proposed fully single-loop FMBO achieves a convergence rate of 1 T 2 / 3 , which further yields the following complexity result.

Corollary 2. Under the same setting of Theorem 2, FMBO requires totally ˜ O ( ϵ -1 . 5 ) data samples, gradient and matrix-vector evaluations, respectively, to achieve an ϵ -accurate stationary point.

Corollary 2 shows that FMBO requires a total number ˜ O ( ϵ -1 . 5 ) of data samples, which matches the best sample complexity in [34, 62, 28]. More importantly, each iteration of FMBO contains only one Hessian-vector computation due to the simple fully single-loop implementation, whereas other momentum-based approaches require O (log 1 ϵ ) Hessian-vector computations in a nested manner per iteration. Also, note that FMBO is the first fully single-loop bilevel optimizer that achieves the ˜ O ( ϵ -1 . 5 ) sample complexity.

4 Experiments

In this section, we test the performance of the proposed FdeHBO and FMBO on two applications: hyper-representation and data hyper-cleaning, respectively.

100

F2SA

F3SA

95

2.0

FdeHBO

90

PZOBO-S

Outer Loss

1.5

85

80

1.0

75

F2SA

70

F3SA

0.5

FdeHBO

65 -

PZOBO-S

0.0

0

10

20

30

40

50

60

60 ~

0

10

20

30

40

50

60

Running Time /s

Running Time /s

4.1 Hyper-representation on MNIST Dataset

We now compare the performance of our Hessian/Jacobian-free FdeHBO with the relevant Hessian/Jacobian-free methods PZOBO-S [58], F 2 SA [37] and F 3 SA [37]. We perform the hyperrepresentation with the 7 -layer LeNet network [38], which aims to solve the following bilevel problem.

where L CE denotes the cross-entropy loss, S ν and S τ denote the training data and validation data, and f ( λ ; x i ) denotes the features extracted from the data x i . More details of the experimental setups are specified in Appendix A.1.

As shown in Figure 1, our FdeHBO converges much faster and more stably than PZOBO-S, F 2 SA and F 3 SA, while achieving a higher training accuracy. This is consistent with our theoretical results, and validates the momentum-based approaches in reducing the variance during the entire training.

4.2 Hyper-cleaning on MNIST Dataset

We compare the performance of our FMBO to various bilevel algorithms including AID-FP [21], reverse[17], SUSTAIN [34], MRBO and VRBO [62], BSA [19], stocBiO [30], FSLA [40] and SOBA [11], on a low-dimensional data hyper-cleaning problem with a linear classifier on MNIST dataset, which takes the following formulation.

where L CE denotes the cross-entropy loss, S ν and S τ denote the training data and validation data, whose sizes are set to 20000 and 5000, respectively, λ = { λ i } i ∈ S τ and C are the regularization parameters, and σ ( · ) is the sigmoid function. AmIGO [3] is not included in the figures because it performs similarly to stocBiO. The experimental details can be found in Appendix A.2.

As shown in Figure 2(a), FMBO, stocBiO and AID-FP converge much faster and more stable than other algorithms. Compared to stocBiO and AID-FP, FMBO achieves a lower training loss. This demonstrates the effectiveness of momentum-based variance reduction in finding more accurate iterates. It can be seen from Figure 2(b) that FMBO converges faster than existing fully single-loop FSLA and SOBA algorithms with a lower training loss.

5 Conclusion

In this paper, we propose a novel Hessian/Jacobian-free bilevel optimizer named FdeHBO. We show that FdeHBO achieves an O ( ϵ -1 . 5 ) sample complexity, which outperforms existing algorithms of the

(a) noise p = 0.1

(b) Single-loop methods

S7

TTSA

25

TTSA

1.25

SOBA

BSA

BSA

Optimality log[f(x, y)-f°]

FSLA

20

stocBio

20

stocBio

1.00 -

FMBO

AID-FP

AID-FP

0.75

Train Loss

15

VRBO

SUSTAIN

Test Loss

15

VRBO

SUSTAIN

0.50

10

FMBO

MRBO

10

FMBO

MRBO

0.25

0.00

0.25

0.50

D

0

50

100

Running Time /s

150

200

250

300

350

400

D

0

50

100

Running Time /s

150

200

250

300

350

400

0

50

100

150

Running Time /s

200

250

300

350

400

same type by a large margin. Our experiments validate the theoretical results and the effectiveness of the proposed algorithms. We anticipate that the developed analysis will shed light on developing provable Hessian/Jacobian-free bilevel optimization algorithms and the proposed algorithms may be applied to other applications such as fair machine learning.

References

Z. Allen-Zhu and Y. Li. Neon2: Finding local minima via first-order oracles. Advances in Neural Information Processing Systems , 31, 2018.

A. Antoniou, H. Edwards, and A. Storkey. How to train your MAML. In International Conference on Learning Representations (ICLR) , 2019.

M. Arbel and J. Mairal. Amortized implicit differentiation for stochastic bilevel optimization. In International Conference on Learning Representations (ICLR) , 2022.

L. Bertinetto, J. F. Henriques, P. Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR) , 2018.

M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-López, F. Pedregosa, and J.-P. Vert. Efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15183 , 2021.

J. Bracken and J. T. McGill. Mathematical programs with optimization problems in the constraints. Operations Research , 21(1):37-44, 1973.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Accelerated methods for nonconvex optimization. SIAM Journal on Optimization , 28(2):1751-1772, 2018.

L. Chen, J. Xu, and J. Zhang. On bilevel optimization without lower-level strong convexity. arXiv preprint arXiv:2301.00712 , 2023.

T. Chen, Y. Sun, and W. Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671 , 2021.

A. Cutkosky and F. Orabona. Momentum-based variance reduction in non-convex SGD. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. arXiv preprint arXiv:2201.13409 , 2022.

J. Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics (AISTATS) , pages 318-326, 2012.

A. Fallah, A. Mokhtari, and A. Ozdaglar. On the convergence theory of gradient-based modelagnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1082-1092. PMLR, 2020.

M. Feurer and F. Hutter. Hyperparameter optimization. In Automated Machine Learning , pages 3-33. Springer, Cham, 2019.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. International Conference on Machine Learning (ICML) , pages 1126-1135, 2017.

R. Flamary, A. Rakotomamonjy, and G. Gasso. Learning constrained task similarities in graphregularized multi-task learning. Regularization, Optimization, Kernels, and Support Vector Machines , page 103, 2014.

L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning (ICML) , pages 1165-1173, 2017.

L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning (ICML) , pages 1568-1577, 2018.

S. Ghadimi and M. Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

S. Gould, B. Fernando, A. Cherian, P. Anderson, R. S. Cruz, and E. Guo. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447 , 2016.

R. Grazzi, L. Franceschi, M. Pontil, and S. Salzo. On the iteration complexity of hypergradient computation. In Proc. International Conference on Machine Learning (ICML) , 2020.

A. Griewank and A. Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation . SIAM, 2008.

B. Gu, G. Liu, Y. Zhang, X. Geng, and H. Huang. Optimizing large-scale hyperparameters via automated learning algorithm. arXiv preprint arXiv:2102.09026 , 2021.

Z. Guo, Y. Xu, W. Yin, R. Jin, and T. Yang. On stochastic moving-average estimators for non-convex optimization. arXiv preprint arXiv:2104.14840 , 2021.

Z. Guo and T. Yang. Randomized stochastic variance-reduced methods for stochastic bilevel optimization. arXiv preprint arXiv:2105.02266 , 2021.

P. Hansen, B. Jaumard, and G. Savard. New branch-and-bound rules for linear bilevel programming. SIAM Journal on Scientific and Statistical Computing , 13(5):1194-1217, 1992.

M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05170 , 2020.

F. Huang and H. Huang. Biadam: Fast adaptive bilevel optimization methods. arXiv preprint arXiv:2106.11396 , 2021.

K. Ji and Y. Liang. Lower bounds and accelerated algorithms for bilevel optimization. arXiv preprint arXiv:2102.03926 , 2021.

K. Ji, J. Yang, and Y . Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning (ICML) , pages 4882-4892. PMLR, 2021.

K. Ji and L. Ying. Network utility maximization with general and unknown utility functions: A distributed, data-driven bilevel optimization approach. Submitted , 2022.

C. Jin, R. Ge, P. Netrapalli, S. M. Kakade, and M. I. Jordan. How to escape saddle points efficiently. In International conference on machine learning , pages 1724-1732. PMLR, 2017.

N. Jorge and J. W. Stephen. Numerical optimization . Spinger, 2006.

P. Khanduri, S. Zeng, M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems (NeurIPS) , 34:30271-30283, 2021.

V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems (NeurIPS) , pages 1008-1014, 2000.

G. Kunapuli, K. P. Bennett, J. Hu, and J.-S. Pang. Classification model selection via bilevel programming. Optimization Methods & Software , 23(4):475-489, 2008.

J. Kwon, D. Kwon, S. Wright, and R. Nowak. A fully first-order method for stochastic bilevel optimization. arXiv preprint arXiv:2301.10945 , 2023.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

J. Li, B. Gu, and H. Huang. Improved bilevel model: Fast and optimal algorithm with theoretical guarantee. arXiv preprint arXiv:2009.00690 , 2020.

J. Li, B. Gu, and H. Huang. A fully single loop algorithm for bilevel optimization without hessian inverse. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7426-7434, 2022.

R. Liao, Y. Xiong, E. Fetaya, L. Zhang, K. Yoon, X. Pitkow, R. Urtasun, and R. Zemel. Reviving and improving recurrent back-propagation. In Proc. International Conference on Machine Learning (ICML) , 2018.

B. Liu, M. Ye, S. Wright, P. Stone, and Q. Liu. Bome! bilevel optimization made easy: A simple first-order approach. Advances in Neural Information Processing Systems , 35:17248-17262, 2022.

H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR) , 2018.

R. Liu, J. Gao, J. Zhang, D. Meng, and Z. Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

R. Liu, X. Liu, X. Yuan, S. Zeng, and J. Zhang. A value-function-based interior-point method for non-convex bi-level optimization. In International Conference on Machine Learning (ICML) , 2021.

R. Liu, P. Mu, X. Yuan, S. Zeng, and J. Zhang. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International Conference on Machine Learning (ICML) , 2020.

D. Maclaurin, D. Duvenaud, and R. Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning (ICML) , pages 2113-2122, 2015.

A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

F. Pedregosa. Hyperparameter optimization with approximate gradient. In International Conference on Machine Learning (ICML) , pages 737-746, 2016.

A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems (NeurIPS) , pages 113-124, 2019.

S. Sabach and S. Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots. Truncated back-propagation for bilevel optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1723-1732, 2019.

H. Shen and T. Chen. On penalty-based bilevel gradient descent method. arXiv preprint arXiv:2302.05185 , 2023.

C. Shi, J. Lu, and G. Zhang. An extended kuhn-tucker approach for linear bilevel programming. Applied Mathematics and Computation , 162(1):51-63, 2005.

A. Sinha, P. Malo, and K. Deb. A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation , 22(2):276-295, 2017.

X. Song, W. Gao, Y. Yang, K. Choromanski, A. Pacchiano, and Y. Tang. ES-MAML: Simple Hessian-free meta learning. In International Conference on Learning Representations (ICLR) , 2019.

D. Sow, K. Ji, Z. Guan, and Y. Liang. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123 , 2022.

D. Sow, K. Ji, and Y. Liang. On the convergence theory for Hessian-free bilevel algorithms. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

D. A. Tarzanagh, M. Li, C. Thrampoulidis, and S. Oymak. Fednest: Federated bilevel, minimax, and compositional optimization. arXiv preprint arXiv:2205.02215 , 2022.

Q. Tran-Dinh, N. H. Pham, D. T. Phan, and L. M. Nguyen. Hybrid stochastic gradient descent algorithms for stochastic nonconvex optimization. arXiv preprint arXiv:1905.05920 , 2019.

R. Vuorio, S.-H. Sun, H. Hu, and J. J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

J. Yang, K. Ji, and Y. Liang. Provably faster algorithms for bilevel optimization. Advances in Neural Information Processing Systems (NeurIPS) , 34:13670-13682, 2021.

M. Ye, B. Liu, S. Wright, P. Stone, and Q. Liu. Bome! bilevel optimization made easy: A simple first-order approach. In Conference on Neural Information Processing Systems (NeurIPS) , page 1, 2022.

Peer reviews:
comments: SUMMARY:
This paper proposed faster first-order stochastic algorithms for bilevel optimization, based on the idea that a Hessian-vector product can be implemented by finite difference of the gradient along the given direction.

STRENGTHS:
The paper studies a very important problem and the provided complexity upper bound strictly improves previous works.

WEAKNESSES:
1. It is well-known that the Hessian-vector product can be implemented by the finite difference of the gradient along the given direction.
And the computation of a Hessian-vector product is known to be as easy as the computation of gradients. Therefore the results and algorithms seem not to be novel compared to the existing methods (AID/ITD) that use Hessian-vector products as oracles.

2. The paper uses three different names for the same baseline, including F${}^2$SA, F2SA, $F{}^2$SA  (Line 442, Table 1, Figure 1). Should use a unified name.

QUESTIONS:
I have a question about the claim of "near-optimal''.
Assumption 3 supposes that "Assumptions 1 and 2 hold for $f(x,y;\xi)$ and $g(x, y;\zeta)$ for $\forall \xi$ and $\forall \zeta$."
But according to "Lower Bounds for Non-Convex Stochastic  Optimization"(https://arxiv.org/pdf/1912.02365.pdf), the lower bound function is constructed under the mean-square-smooth Assumption. Therefore it seems that the mean-square-smoothness should be imposed for $f$, which is slightly stronger than the smoothness of $f(x,y;\xi)$ for $\forall \xi$?

I think it may be a fixable bug. But the paper may need some modifications.

Please let me know if I am wrong. I will adjust my ratings based on the rebuttal.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-22

comments: SUMMARY:
This paper provides a novel Hessian-free bilevel optimization algorithm called FdeHBO, which improves upon previous state of the art F3SA with a better iteration complexity. The author provide convergence analysis and empirical results to show the strength of the proposed method. However, I have some concerns regarding the proof, which is detailed in the weaknesses part.

STRENGTHS:
The algorithm is novel, as far as I am concerned, and the improved convergence rate seems promising. The paper is well organized and easy to follow. The experiment seems convincing.

WEAKNESSES:
The proof of Lemma 5 is incorrect. In the proof, 
the authors attempt to use the mean value theorem for a function $\mathbb{R}\to \mathbb{R}^n$, but this is in general wrong. The estimation error of the finite difference of in the lemma is a key step, without which the proof of the main theorem is incorrect. 

In general, a zeroth order optimization algorithm should have a dimension dependency, but this paper magically removes this. I believe the one of the reason is the mistake in Lemma 5. I will be glad to change my evaluation if I am wrong.

QUESTIONS:
See weakness part above.

LIMITATIONS:
I do not see the discussion of limitations, but I think this is not a major problem. I do not see potential negative societal impact as this paper mainly focuses on the theoretical side.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on addressing a stochastic bilevel optimization problem characterized by a strongly convex lower-level objective function and a nonconvex upper-level objective. The authors specifically investigate a Hessian-free (fully first-order) approach to bilevel optimization, which avoids the need for Hessian or Jacobian-vector computations. They introduce a new method called FdeHBO, which adopts a single-loop structure and incorporates momentum-based updates for the upper-level problem, linear system, and lower-level problem.\nFdeHBO achieves a sample and gradient complexity of $O(\\epsilon^{-1.5})$, which is the first result for nonconvex-strongly-convex Hessian-free bilevel optimization. Additionally, the authors propose a simplified and nearly optimal variant called FMBO, which is specifically designed for scenarios with small-dimensional problems. To validate the effectiveness of their proposed methods, the authors conduct several experiments.\n\nSTRENGTHS:\n1. The paper is overall easy to follow. The topic of Hessian-free bilevel optimization, particularly in the nonconvex-strongly-convex setting, has not been extensively explored. Therefore, the investigation into the development of simple and nearly optimal stochastic Hessian-free bilevel algorithms is both interesting and significant.\n2. To the best of my knowledge, the proposed FdeHBO algorithm achieves near-optimal sample/gradient complexity for the first time. The utilization of the projection-aided finite-difference estimator in bilevel optimization is interesting. The experimental results depicted in Figure 1 provide substantial evidence of FdeHBO's superior performance compared to other approaches of the same kind.\n3. From a technical standpoint, addressing issues such as momentum-based errors, biased estimation caused by the finite-difference estimator, and auxiliary projection errors requires nontrivial treatments, further highlighting the sophistication of this work.\n\nWEAKNESSES:\n1. The paper's projection technique is limited to projecting onto a ball. Is it possible that  this approach can be generalized to encompass projections onto more general convex and closed sets?\n\n2. While this work primarily focuses on the momentum-based estimator, it appears that the approach can be simplified to a more general stochastic gradient descent (SGD)-type method by setting $\\eta_t^f = \\eta_t^g = \\eta_t^R = 0$. Does the analysis presented in the paper adequately cover this specific case? It is worth considering the importance of simpler SGD updates with reduced tuning requirements, as they often hold practical significance.\n\n3. The authors should provide further elaboration on why the auxiliary projection does not impact the final convergence guarantee. Although I have not thoroughly examined all the proofs, a clearer explanation on this aspect would be beneficial.\n\nQUESTIONS:\nSee Weakness. \n\n$\\nabla_y^2 g$ in Line 126 should be corrected to $\\nabla_{yy}^2 g$?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper studies stochastic bilevel optimization, where the first level objective function depends on the optimizer of the second level objective function. The paper provides an optimization procedure involving gradient computations and projections. The paper provides rates of convergence for the method. Since convexity is not assumed, convergence is expressed in terms of the gradient becoming small (not in terms of converging to a global optimizer). The benefit of the algorithm is demonstrated on various data sets.

STRENGTHS:
The technical contribution of the paper appears to be impressive. The algorithm suggested is involved. Its novelties and benefits compared to the state of the art are discussed extensively. The convergence proofs appear to be involved and innovative. The numerical performances are very good, especially when compared to the state of the art.

WEAKNESSES:
The content is very technical, and it is not very easy to follow, especially given the space constraint in the main text. 
In the appendix, the description of the experiments is also difficult to follow. There there are no space constraints, so I think the description could have been more pedagogical there.

QUESTIONS:
Algorithm 1: Hessain -> Hessian.

Conclusion, why is it written that the sample complexity is O(epsilon^-2), while Corollary 1 writes O(epsilon^-1.5)?

Appendix A.1. What are lambda and w? What are the dimensions?

LIMITATIONS:
The authors adequately discuss limitations.
I think there is not concern about societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-24