Title: First-Order Methods for Linearly Constrained Bilevel Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Constrained bilevel optimization

first-order methods

nonsmooth nonconvex optimization

bilevel optimization

nonconvex optimization

nonsmooth optimization

Goldstein stationarity
Abstract: Algorithms for bilevel optimization often encounter Hessian computations, which are prohibitive in high dimensions. While recent works offer first-order methods for unconstrained bilevel problems, the constrained setting remains relatively underexplored. We present  first-order linearly constrained optimization methods with finite-time hypergradient stationarity guarantees. For linear equality constraints, we attain $\epsilon$-stationarity in $\widetilde{O}(\epsilon^{-2})$ gradient oracle calls, which is nearly-optimal. 
For linear inequality constraints, we attain $(\delta,\epsilon)$-Goldstein stationarity in $\widetilde{O}(d{\delta^{-1} \epsilon^{-3}})$ gradient oracle calls, where $d$ is the upper-level dimension. 
Finally, we obtain for the linear inequality setting dimension-free rates of $\widetilde{O}({\delta^{-1} \epsilon^{-4}})$ oracle complexity under the additional assumption of oracle access to the optimal dual variable. Along the way, we develop new nonsmooth nonconvex optimization methods with inexact oracles. Our numerical experiments verify these guarantees.

First-Order Methods for Linearly Constrained Bilevel Optimization

Abstract

Algorithms for bilevel optimization often encounter Hessian computations, which are prohibitive in high dimensions. While recent works offer first-order methods for unconstrained bilevel problems, the constrained setting remains relatively underexplored. We present first-order linearly constrained optimization methods with finite-time hypergradient stationarity guarantees. For linear equality constraints, we attain ϵ -stationarity in ˜ O ( ϵ -2 ) gradient oracle calls, which is nearly-optimal. For linear inequality constraints, we attain ( δ, ϵ ) -Goldstein stationarity in ˜ O ( dδ -1 ϵ -3 ) gradient oracle calls, where d is the upper-level dimension. Finally, we obtain for the linear inequality setting dimension-free rates of ˜ O ( δ -1 ϵ -4 ) oracle complexity under the additional assumption of oracle access to the optimal dual variable. Along the way, we develop new nonsmooth nonconvex optimization methods with inexact oracles. Our numerical experiments verify these guarantees.

1 Introduction

Bilevel optimization [1-4], an important problem in optimization, is defined as follows:

Here, the value of the upper-level problem at any point x depends on the solution of the lowerlevel problem. This framework has recently found numerous applications in meta-learning [5-8], hyperparameter optimization [9-11], and reinforcement learning [12-15]. Its growing importance has spurred increasing efforts towards designing computationally efficient algorithms for it.

As demonstrated by [16], a key computational step in algorithms for bilevel optimization is estimating dy ∗ ( x ) /dx , the gradient of the lower-level solution. This gradient estimation problem has been extensively studied in differentiable optimization [17, 18] by applying the implicit function theorem to the KKT system of the given problem [19-24]. However, this technique typically entails computing (or estimating) second-order derivatives, which can be prohibitive in high dimensions [25-27].

Recently, [28] made a big leap forward towards addressing this computational bottleneck. Restricting themselves to the class of unconstrained bilevel optimization, they proposed a fully first-order method with finite-time stationarity guarantees. While a remarkable breakthrough, [28] does not directly extend to the important setting of constrained bilevel optimization. This motivates the question:

Can we develop a first-order algorithm for constrained bilevel optimization?

Besides being natural from the viewpoint of complexity theory, this question is well-grounded in applications such as mechanism design [29, 30], resource allocation [31-34], and decision-making under uncertainty [20, 35, 36]. Our primary contribution is an affirmative answer to the highlighted question for bilevel programs with linear constraints , an important problem class often arising in adversarial training, decentralized meta learning, and sensor networks (see [37]). While there have been some other recent works [37-39] on this problem, our work is first-order (as opposed to [37]) and offers, in our view, a stronger guarantee on stationarity (compared to [38, 39])- cf. Section 1.2.

1.1 Our contributions

We provide first-order algorithms (with associated finite-time convergence guarantees) for linearly constrained bilevel programs (Problem 1.1). By 'first-order', we mean that we use only zeroth and first-order oracle access to f and g . Our assumptions for each of our contributions are in Section 2.1.

(1) Linear equality constraints. As our first contribution, we design first-order algorithms for solving Problem 1.1 where the lower-level constraint set S ( x ) := { y : Ax -By -b = 0 } comprises linear equality constraints, and X a convex compact set. With appropriate regularity assumptions on f and g , we show in this case smoothness in x of the hyperobjective F . Inspired by ideas from Kwon et al. [40], we use implicit differentiation of the KKT matrix of a slightly perturbed version of the lower-level problem to design a first-order approximation to ∇ F . Constructing our first-order approximation entails solving a strongly convex optimization problem on affine constraints, which can be done efficiently. With this inexact gradient oracle in hand, we then run projected gradient descent, which converges in ˜ O ( ϵ -2 ) iterations for smooth functions.

Theorem 1.1 (Informal; cf. Theorem 3.1) . Given Problem 1.1 with linear equality constraints S ( x ) := { y : Ax -By -b = 0 } and X a convex compact set, under regularity assumptions on f and g (Assumptions 2.2 and 2.3), there exists an algorithm, which in ˜ O ( ϵ -2 ) oracle calls to f and g , converges to an ϵ -stationary point of F .

For linear equality constrained bilevel optimization, this is the first first-order result attaining ϵ -stationarity of F with assumptions solely on the constituent functions f and g and none on F -cf. Section 1.2 for a discussion of the results of Khanduri et al. [37] for this setting.

(2) Linear inequality constraints. Next, we provide first-order algorithms for solving Problem 1.1 where the lower-level constraint set S ( x ) := { y : Ax -By -b ≤ 0 } comprises linear inequality constraints, and the upper-level variable is unconstrained.

Our measure of convergence of algorithms in this case is that of ( δ, ϵ ) -stationarity [41]: for a Lipschitz function, we say that a point x is ( δ, ϵ ) -stationary if within a δ -ball around x there exists a convex combination of subgradients of the function with norm at most ϵ (cf. Definition 2.1).

To motivate this notion of convergence, we note that the hyperobjective F (in Problem 1.1) as a function of x could be nonsmooth and nonconvex (and Lipschitz, as we later prove). Minimizing such a function in general is known to be intractable [42], necessitating local notions of stationarity. Indeed, not only is it impossible to attain ϵ -stationarity in finite time [43], even getting near an approximate stationary point of an arbitrary Lipschitz function is impossible unless the number of queries is exponential in the dimension [44]. Consequently, for this function class, ( δ, ϵ ) -stationarity has recently emerged [43] to be a natural and algorithmically tractable notion of stationarity. We give the following guarantee under regularity assumptions on f and g .

Theorem 1.2 (Informal; Theorem 4.1) . Consider Problem 1.1 with linear inequality constraints S ( x ) := { y : Ax -By -b ≤ 0 } . Under mild assumptions on f and g (Assumption 2.2) and the lower-level primal solution y ∗ (Assumption 2.4), there exists an algorithm, which converges to a ( δ, ϵ ) -stationary point of F in ˜ O ( dδ -1 ϵ -3 ) oracle calls to f and g , where d is the upper-level variable dimension.

To the best of our knowledge, this is the first result to offer a first-order finite-time stationarity guarantee on the hyperobjective for linear inequality constrained bilevel optimization (cf. Section 1.2 for a discussion of related work [37-39]). We obtain our guarantee in Theorem 1.2 by first invoking a result by Zhang and Lan [45] to obtain inexact hyperobjective values of F using only ˜ O (1) oracle calls to f and g . We also show (Lemma 4.3) that this hyperobjective F is Lipschitz. We then employ our inexact zeroth-order oracle for F in Algorithm 2 designed to minimize Lipschitz nonsmooth nonconvex functions (in particular, F ), with the following convergence guarantee.

Theorem 1.3 (see Theorem C.1) . Given L -Lipschitz F : R d → R and | ˜ F ( · ) -F ( · ) | ≤ ϵ , there exists an algorithm, which, in ˜ O ( dδ -1 ϵ -3 ) calls to ˜ F ( · ) , outputs x out with E [dist(0 , ∂ δ F ( x out ))] ≤ 2 ϵ .

While such algorithms using exact zeroth-order access already exist [46], extending them to the inexact gradient setting is non-trivial; we leverage recent ideas connecting online learning to nonsmooth nonconvex optimization by Cutkosky, Mehta, and Orabona [47] (cf. Section 4).

(3) Linear inequality under assumptions on dual variable access. For the inequality setting (i.e., Problem 1.1 with the lower-level constraint set S ( x ) := { y : Ax -By -b ≤ 0 } ), we obtain dimension-free rates under an additional assumption (Assumption 2.5) on oracle access to the optimal dual variable λ ∗ of the lower-level problem. We are not aware of a method to obtain this dual variable in a first-order fashion (though in practice, highly accurate approximations to λ ∗ are readily available), hence the need for imposing this assumption. We believe that removing this assumption and obtaining dimension-free first-order rates in this setting would be an important direction for future work. Our guarantee for this setting is summarized below.

Theorem 1.4 (Informal; Theorem 4.4 combined with Theorem 5.3) . Consider Problem 1.1 with linear inequality constraints S ( x ) := { y : Ax -By -b ≤ 0 } and unconstrained upper-level variable. Under mild regularity assumptions on f and g (Assumption 2.2), on y ∗ (Assumption 2.4), and assuming oracle access to the optimal dual variable λ ∗ (Assumption 2.5), there exists an algorithm, which in ˜ O ( δ -1 ϵ -4 ) oracle calls to f and g converges to a ( δ, ϵ ) -stationary point for F .

We obtain this result by first reformulating Problem 4.1 via the penalty method and constructing an inexact gradient oracle for the hyperobjective F (cf. Section 5). We then employ this inexact gradient oracle within an algorithm (Algorithm 3) designed to minimize Lipschitz nonsmooth nonconvex functions (in particular, F ), with the following convergence guarantee.

Theorem 1.5 (Informal; Theorem 4.4) . Given Lipschitz F : R d → R and ∥ ˜ ∇ F ( · ) -∇ F ( · ) ∥ ≤ ϵ , there exists an algorithm that, in T = O ( δ -1 ϵ -3 ) calls to ˜ ∇ F , outputs a ( δ, 2 ϵ ) -stationary point of F .

Our Algorithm 3 is essentially a 'first-order' version of Algorithm 2. Similar to Algorithm 2, despite the existence of algorithms with these guarantees with access to exact gradients [48], their extensions to the inexact gradient setting are not trivial and also make use of the new framework of Cutkosky, Mehta, and Orabona [47]. We believe our analysis for this general task can be of independent interest to the broader optimization community. Lastly, we also use a more implementation-friendly variant of Algorithm 3 (with slightly worse theoretical guarantees) in numerical experiments.

1.2 Related work

The vast body of work on asymptotic results for bilevel programming, starting with classical works such as Anandalingam and White [49], Ishizuka and Aiyoshi [50], White and Anandalingam [51], Vicente, Savard, and Júdice [52], Zhu [53], and Ye and Zhu [54], typically fall into two categories: those based on approximate implicit differentiation: Amos and Kolter [17], Agrawal et al. [18], Domke [55], Pedregosa [56], Gould et al. [57], Liao et al. [58], Grazzi et al. [59], and Lorraine, Vicol, and Duvenaud [60] and those via iterative differentiation: Franceschi et al. [9], Shaban et al. [10], Domke [55], Grazzi et al. [59], Maclaurin, Duvenaud, and Adams [61], and Franceschi et al. [62]. Another recent line of work in this category includes Khanduri et al. [37], Liu et al. [63], Ye et al. [64], and Gao et al. [65], which use various smoothing techniques.

The first non-asymptotic result for bilevel programming was provided by Ghadimi and Wang [16], which was followed by a flurry of work: for example, algorithms that are single-loop stochastic: Chen, Sun, and Yin [66], Chen et al. [67], and Hong et al. [68], projection-free: Akhtar et al. [69], Jiang et al. [70], Abolfazli et al. [71], and Cao et al. [72], use variance-reduction and momentum: Khanduri et al. [73], Guo et al. [74], Yang, Ji, and Liang [75], and Dagréou et al. [76], those for single-variable bilevel programs: Jiang et al. [70], Sabach and Shtern [77], Amini and Yousefian [78, 79], and Merchav and Sabach [80], and for bilevel programs with special constraints: Khanduri et al. [37], Abolfazli et al. [71], Tsaknakis, Khanduri, and Hong [81], and Xu and Zhu [82].

The most direct predecessors of our work are those by Khanduri et al. [37], Yao et al. [38], Lu and Mei [39], Kwon et al. [40], and Liu et al. [63]. As alluded to earlier, Liu et al. [28] recently made a significant contribution by providing for bilevel programming a fully first-order algorithm with finite-time stationarity guarantees. This was extended to the stochastic setting by Kwon et al. [40]

(which we build upon), simplified and improved by Chen, Ma, and Zhang [83], and extended to the constrained setting by Khanduri et al. [37], Yao et al. [38], and Lu and Mei [39].

The works of Yao et al. [38] and Lu and Mei [39] study the more general problem of bilevel programming with general convex constraints. However, they use KKT stationarity as a proxy to the hypergradient stationarity. Our Theorem 1.4 is restricted to linear inequality constraints, we provide stationarity guarantees directly in terms of the objective of interest. Moreover, Yao et al. [38] assumes joint convexity of the lower-level constraints in upper and lower variables to allow for efficient projections, while we require convexity only in the lower-level variable.

The current best result for the linearly constrained setting is that of Khanduri et al. [37]. However, this work requires Hessian computations (and is therefore not fully first-order). Moreover, Khanduri et al. [37] imposes strong regularity assumptions on the hyperobjective F , which are, in general, impossible to verify. In contrast, Theorem 1.1 imposes assumptions solely on the constituent functions f and g , none directly on F , thus making substantial progress on these two fronts.

2 Preliminaries

We follow standard notation (see Appendix A), with only the following crucial definition stated here. Definition 2.1. Consider a locally Lipschitz function f : R d → R , a point x ∈ R d , and a parameter δ > 0 . The Goldstein subdifferential [41] of f at x is the set ∂ δ f ( x ) := conv( ∪ y ∈ B δ ( x ) ∂f ( y )) , where ∂f ( x ) = conv { lim n →∞ ∇ f ( x n ) : x n → x, x n ∈ dom( ∇ f ) } is the Clarke subdifferential [84] of f and B δ ( x ) denotes the Euclidean ball of radius δ around x . A point x is called ( δ, ϵ ) -stationary if dist(0 , ∂ δ f ( x )) ≤ ϵ , where dist( x, S ) := inf y ∈ S ∥ x -y ∥ .

2.1 Assumptions

We consider Problem 1.1 with linear equality constraints (Section 3) under Assumptions 2.2 and 2.3 and linear inequality constraints (Sections 4 and 5) under Assumptions 2.2, 2.4 and 2.5. We assume the upper-level (UL) variable x ∈ R d x , lower-level (LL) variable y ∈ R d y , and A ∈ R d h × d x .

Assumption 2.2. For Problem 1.1, we assume the following for both settings we study:

(i) Upper-level: The objective f is C f -smooth and L f -Lipschitz continuous in ( x, y ) .

(ii) Lower-level: The objective g is C g -smooth. Fixing any x ∈ X , g ( x, · ) is µ g -strongly convex.

(iii) We assume that the linear independence constraint qualification (LICQ) condition holds for the LL problem at every x and y , i.e., the constraint h ( x, y ) := Ax -By -b has a full row rank B .

Assumption 2.3. For Problem 3.1 (with linear equality constraints), we additionally assume that the set X is convex and compact, and that the objective g is S g -Hessian smooth, that is, ∥ ∥ ∇ 2 g ( x, y ) -∇ 2 g (¯ x, ¯ y ) ∥ ∥ ≤ S g ∥ ( x, y ) -(¯ x, ¯ y ) ∥ ∀ x, ¯ x ∈ X , and y, ¯ y ∈ R d y .

Assumption 2.4. For Problem 4.1 (with linear inequality constraints), we additionally assume that y ∗ is L y -Lipschitz in x , where y ∗ is the LL primal solution y ∗ ( x ) , λ ∗ ( x ) = arg max y min β ≥ 0 g ( x, y ) + β ⊤ h ( x, y ) , where h ( x, y ) := Ax -By -b .

Assumption 2.5. We provide additional results for Problem 4.1 under additional stronger assumptions stated here: Denote the LL primal and dual solution y ∗ ( x ) , λ ∗ ( x ) = arg max y min β ≥ 0 g ( x, y ) + β ⊤ h ( x, y ) , where h ( x, y ) := Ax -By -b ; then, we assume exact access to λ ∗ and that ∥ λ ∗ ( x ) ∥ ≤ R .

Assumptions 2.2(i) and 2.2(ii) are standard in bilevel optimization. Assumption 2.2(iii) is the same as the complete recourse assumption in stochastic programming [85], that is, the LL problem is feasible y for every x ∈ R d x . Assumption 2.3 is used only in the equality case and guarantees smoothness of F . Assumption 2.4 is used in the inequality case and implies Lipschitzness of F . We need the stronger assumption in Assumption 2.5 for our dimension-free result for the linear inequality case.

3 Lower-level problem with linear equality constraint

We first obtain improved results for the setting of bilevel programs with linear equality constraints in the lower-level problem. Our formal problem statement is:

where f , g , h ( x, y ) := Ax -By -b , and X satisfy Assumptions 2.2 and 2.3. The previous best result on Problem 3.1 providing finite-time ϵ -stationarity guarantees, by Khanduri et al. [37], required certain regularity assumptions on F as well as Hessian computations. In contrast, our finite-time guarantees require assumptions only on f and g , not on F ; indeed, in our work, these desirable properties of F are naturally implied by our analysis. Specifically, our key insight is that the hypergradient

∇ F ( x ) := ∇ x f ( x, y ∗ )+ ( dy ∗ ( x ) dx ) ⊤ ∇ y f ( x, y ∗ ) for Problem 3.1 is Lipschitz-continuous and admits an easily computable - yet highly accurate - finite-difference approximation. Therefore, O ( ϵ -2 ) iterations of gradient descent on F with this finite-difference gradient proxy yield an ϵ -stationary point. Specifically, for any fixed x ∈ X , our proposed finite-difference gradient proxy approximating the non-trivial-to-compute component ( dy ∗ ( x ) dx ) ⊤ ∇ y f ( x, y ∗ ) of the hypergradient is given by

where ( y ∗ δ , λ ∗ δ ) are the primal and dual solutions to the perturbed lower-level problem:

We show in Lemma 3.2 that v in (3.2) approximates ( dy ∗ ( x ) dx ) ⊤ ∇ y f ( x, y ∗ ) up to an O ( δ ) -additive error, implying the gradient oracle construction outlined in the pseudocode presented in Algorithm 1. Our full implementable algorithm for solving Problem 3.1 is displayed in Algorithm 5.

Algorithm 1 Inexact Gradient Oracle for Bilevel Program with Linear Equality Constraint

1: Input: Current x , accuracy ϵ , perturbation δ = ϵ 2 .

2: Compute y ∗ (as in Problem 3.1) and corresponding optimal dual λ ∗ (as in (B.1))

3: Compute y ∗ δ (as in (3.3)) and and corresponding optimal dual λ ∗ δ (as in (B.7))

4: Compute v x

as in (3.2) )

5: Output: ˜ ∇ F = v x + ∇ x f ( x, y ∗ )

Notice that the finite-difference term in (3.2) avoids differentiating through the implicit function y ∗ . Instead, all we need to evaluate it are the values of ( y ∗ , λ ∗ , y ∗ δ , λ ∗ δ ) (and gradients of g and h ). Since ( y ∗ , λ ∗ ) are solutions to a smooth strongly convex linearly constrained problem, they can be approximated at a linear rate. Similarly, since the approximation error in (3.2) is proportional to δ (cf. Lemma 3.2), a small enough δ in the perturbed objective g + δf in (3.3) ensures that it is dominated by the strongly convex and smooth g , whereby accurate approximates to ( y ∗ δ , λ ∗ δ ) can also be readily obtained. Putting it all together, the proposed finite-difference hypergradient proxy in (3.2) is efficiently computable, yielding the following guarantee.

Theorem 3.1. Consider Problem 3.1 under Assumption 2.2, and let κ = C g /µ g be the condition number of g . Then Algorithm 5 finds an ϵ -stationary point (in terms of gradient mapping, see (B.14)) after T = ˜ O ( C F ( F ( x 0 ) -inf F ) √ κϵ -2 ) oracle calls to f and g , where C F := 2( L f + C f + C g ) C 3 H S g ( L g + ∥ A ∥ ) 2 is the smoothness constant of the hyperobjective F .

We now sketch the proof of Theorem 3.1. The complete proofs may be found in Appendix B.

3.1 Main technical ideas

We briefly outline the two key technical building blocks alluded to above, that together give us Theorem 3.1: the approximation guarantee of our finite-difference gradient proxy ((3.2)) and the smoothness of hyperobjective F (for Problem 3.1). The starting point for both these results is the following simple observation obtained by implicitly differentiating, with respect to x , the KKT system associated with y ∗ = arg min h ( x,y )=0 g ( x, y ) and optimal dual variable λ ∗ :

▷ Approximates ( dy ∗ ( x ) /dx ) ⊤ ∇ y f ( x, y ∗

The invertibility of the matrix in the preceding equation is proved in Corollary B.3: essentially, this invertibility is implied by strong convexity of g and ∇ y h ( x, y ∗ ) = B having full row rank. This in conjunction with the compactness of X implies that the inverse of the matrix is bounded by some constant C H (cf. Corollary B.3 for details). Our hypergradient approximation guarantee follows:

Lemma 3.2. For Problem 3.1 under Assumption 2.2, with v x as in (3.2), the following holds:

Proof sketch; see Appendix B. The main idea is that the two terms being compared are essentially the same by the implicit function theorem. First, we use the expression for dy ∗ ( x ) dx from (3.4):

We now examine v x . For simplicity of exposition, we instead consider lim δ → 0 ∇ x [ g ( x,y ∗ δ )+ ⟨ λ ∗ δ ,h ( x,y ∗ ) ⟩ ] -∇ x [ g ( x,y ∗ )+ ⟨ λ ∗ ,h ( x,y ∗ ) ⟩ ] δ , which, by the fundamental theorem of calculus and Assumption 2.3 , equals v x up to an O ( δ ) -additive error. Note that this expression is:

Since y ∗ δ is minimizes a strongly convex function over a linear equality constraint (3.3), the reasoning that yields (3.4) also gives the following, which, when combined with (3.5), finishes the proof:

Having shown the construction of the hypergradient approximation, we now state smoothness of the hyperobjective F (proof in Appendix B), crucial to getting our claimed rate.

Lemma 3.3. The solution y ∗ (as defined in Problem 3.1) is O ( C H · ( C g + ∥ A ∥ )) -Lipschitz continuous and O ( C 3 H · S g · ( C g + ∥ A ∥ ) 2 ) -smooth as a function of x . Thus the hyper-objective F is gradientLipschitz with a smoothness constant of C F := O { ( L f + C f + C g ) C 3 H S g ( L g + ∥ A ∥ ) 2 } .

4 Nonsmooth nonconvex optimization with inexact oracle

We now shift gears from the case of linear equality constraints to that of linear inequality constraints. Specifically, defining h ( x, y ) = Ax -By -b , the problem we now consider is

As noted earlier, for this larger problem class, the hyperobjective F can be nonsmooth nonconvex, necessitating our measure of convergence to be the now popular notion of Goldstein stationarity [43].

Our first algorithm for solving Problem 4.1 is presented in Algorithm 2, with its convergence guarantee in Theorem 4.1. At a high level, this algorithm first obtains access to an inexact zeroth-order oracle to F (we shortly explain how this is done) and uses this oracle to construct a (biased) gradient estimate of F . It then uses this gradient estimate to update the iterates with a rule motivated by recent works reducing nonconvex optimization to online optimization [47]. We explain this in Section 4.1.

Theorem 4.1. Consider Problem 4.1 under Assumptions 2.2 and 2.4. Let κ = C g /µ g be the condition number of g . Then combining the procedure for Lemma 4.2 with Algorithm 2 run with ρ = min { δ 2 , F ( x 0 ) -inf F L f L y } , ν = δ -ρ, D = Θ ( νϵ 2 ρ 2 d x ρ 2 L 2 f L 2 y + α 2 d 2 x ) , and η = Θ ( νϵ 3 ρ 4 ( d x ρ 2 L 2 f L 2 y + α 2 d 2 x ) 2 ) outputs x out such that E [dist(0 , ∂ δ F ( x out ))] ≤ ϵ + α with T oracle calls to f and g , where:

Algorithm 2 Nonsmooth Nonconvex Algorithm with Inexact Zero-Order oracle 1: Input: Initialization x 0 ∈ R d , clipping parameter D > 0 , step size η > 0 , smoothing parameter ρ > 0 , accuracy parameter ν > 0 , iteration budget T ∈ N , inexact zero-order oracle ˜ F : R d → R . 2: Initialize: ∆ 1 = 0 3: for t = 1 , . . . , T do 4: Sample s t ∼ Unif[0 , 1] , w t ∼ Unif( S d -1 ) 5: x t = x t -1 +∆ t , z t = x t -1 + s t ∆ t 6: ˜ g t = d 2 ρ ( ˜ F ( z t + ρw t ) -˜ F ( z t -ρw t )) w t 7: ∆ t +1 = clip D (∆ t -η ˜ g t ) ▷ clip D ( z ) := min { 1 , D ∥ z ∥ } · z 8: M = ⌊ ν D ⌋ , K = ⌊ T M ⌋ 9: for k = 1 , . . . , K do 10: x k = 1 M ∑ M m =1 z ( k -1) M + m 11: Sample x out ∼ Unif { x 1 , . . . , x K } 12: Output: x out .

Algorithm 2 is a variant of gradient descent with momentum and clipping, with ˜ g t the inexact gradient, ∆ t a clipped accumulated gradient (hence accounts for past gradients, which serve as a momentum), and the clipping ensuring that consecutive iterates of the algorithm reside within a δ -ball of each other. While similar algorithms have appeared in prior work on nonsmooth nonconvex optimization (e.g. [47]), none of them account for inexactness in the gradient, crucial in our setting.

4.1 Nonsmooth nonconvex optimization with inexact zeroth-order oracle

We can obtain inexact zeroth-order oracle access to F because (as formalized in Lemma 4.2) despite potential nonsmoothness and nonconvexity of F in Problem 4.1, estimating its value F ( x ) at any point x amounts to solving a single smooth and strongly convex optimization problem, which can be done can be done in ˜ O (1) oracle calls to f and g by appealing to a result by Zhang and Lan [45].

Lemma 4.2 (Proof in Appendix C.1) . Given any x , we can return ˜ F ( x ) such that | F ( x ) -˜ F ( x ) | ≤ α using O ( √ C g /µ g log( L f /α )) first-order oracle calls to f and g .

Having computed an inexact value of the hyperobjective F , we now show how to use it to develop an algorithm for solving Problem 4.1. To this end, we first note that F , despite being possibly nonsmooth and nonconvex, is Lipschitz and hence amenable to the use of recent algorithmic developments in nonsmooth nonconvex optimization pertaining to Goldstein stationarity.

Lemma 4.3. Under Assumption 2.2 and 2.5, F in Problem 4.1 is O ( L f L y ) -Lipschitz in x .

With this guarantee on the Lipschitzness of F , we prove Theorem C.1 for attaining Goldstein stationarity using the inexact zeroth-order oracle of a Lipschitz function. Our proof of Theorem C.1 crucially uses the recent online-to-nonconvex framework of Cutkosky, Mehta, and Orabona [47]. Combining Lemma 4.2 and Theorem C.1 then immediately implies Theorem 4.1.

4.2 Nonsmooth nonconvex optimization with inexact gradient oracle

In Section 5, we provide a way to generate approximate gradients of F . Here, we present an algorithm that attains Goldstein stationarity of Problem 4.1 using this inexact gradient oracle. While there has been a long line of recent work on algorithms for nonsmooth nonconvex optimization with convergence to Goldstein stationarity [43, 48, 86-88], these results necessarily require exact gradients. This brittleness to any error in gradients renders them ineffective in our setting, where our computed (hyper)gradient necessarily suffers from an additive error. While inexact oracles are known to be effective for smooth or convex objectives [89], utilizing inexact gradients in the nonsmooth nonconvex regime presents a nontrivial challenge. Indeed, without any local bound on gradient variation due to smoothness, or convexity that ensures that gradients are everywhere correlated with the direction pointing at the minimum, it is not clear a priori how to control the accumulating price of inexactness throughout the run of an algorithm. To derive such results, we use the recently proposed connection between online learning and nonsmooth nonconvex optimization by Cutkosky, Mehta,

and Orabona [47]. By controlling the accumulated error suffered by online gradient descent for linear losses (cf. Lemma C.3), we derive guarantees for our setting of interest, providing Lipschitz optimization algorithms that converge to Goldstein stationary points even with inexact gradients.

This algorithm matches the best known complexity in first-order nonsmooth nonconvex optimization [43, 47, 48], merely replacing the convergence to a ( δ, ϵ ) -stationary point by ( δ, ϵ + α ) -stationarity, where α is the inexactness of the gradient oracle.

Algorithm 3 Nonsmooth Nonconvex Algorithm with Inexact Gradient Oracle

1: Input: Initialization x 0 ∈ R d , clipping parameter D > 0 , step size η > 0 , accuracy parameter δ > 0 , iteration budget T ∈ N , inexact gradient oracle ˜ ∇ F : R d → R d .

2: Initialize: ∆ 1 = 0

3: for t = 1 , . . . , T do

4: Sample s t ∼ Unif[0 , 1]

5: x t = x t -1 +∆ t , z t = x t -1 + s t ∆ t

6: ˜ g t = ˜ ∇ F ( z t )

7: ∆ t +1 = clip D (∆ t -η ˜ g t )

8: M = ⌊ δ D ⌋ , K = ⌊ T M

⌋

9: for k = 1 , . . . , K do

10: x k = 1 M ∑ M m =1 z ( k -1) M + m

11: Sample x out ∼ Unif { x 1 , . . . , x K }

12: Output: x out .

Theorem 4.4. Suppose F : R d → R is L -Lipschitz and that ∥ ˜ ∇ F ( · ) -∇ F ( · ) ∥ ≤ α . Then running Algorithm 3 with D = Θ( δϵ 2 L 2 ) , η = Θ( δϵ 3 L 4 ) , outputs a point x out such that E [dist(0 , ∂ δ F ( x out ))] ≤ ϵ + α , with T = O ( ( F ( x 0 ) -inf F ) L 2 δϵ 3 ) calls to ˜ ∇ F ( · ) .

We defer the proof of Theorem 4.4 to Appendix C.1. Plugging the complexity of computing inexact gradients, as given by Theorem 5.3, into the result above, we immediately obtain convergence to a ( δ, ϵ ) -stationary point of Problem 1.1 with ˜ O ( δ -1 ϵ -4 ) gradient calls overall.

Implementation-friendly algorithm. While Algorithm 3 matches the best-known results in nonsmooth nonconvex optimization, it could be impractical due to several hyperparameters which need tuning. Arguably, a more natural application of the hypergradient estimates would be simply plugging them into gradient descent, which requires tuning only the stepsize. Since F is neither smooth nor convex, perturbations are required to guarantee differentiability along the trajectory. We therefore complement Theorem 4.4 by analyzing perturbed (inexact) gradient descent in the nonsmooth nonconvex setting (Algorithm 7) and state its theoretical guarantee in Theorem C.4. Despite its suboptimal worst-case theoretical guarantees, we find this algorithm easier to implement in practice.

5 Inequality constraints: constructing the inexact gradient oracle

Computing a stationary point of F of Problem 4.1 via any first-order method would require:

for which the key challenge lies in computing dy ∗ ( x ) /dx . This requires differentiating through an argmin operator, which typically requires second-order derivatives. Instead, here we differentiate (using the implicit function theorem) through the KKT conditions describing y ∗ ( x ) and get:

where given x , we assume efficient access to the optimal dual solution λ ∗ ( x ) of the LL problem in Problem 4.1. In (5.2), we use I := { i ∈ [ d h ] : h i ( x, y ) = 0 , λ ∗ i > 0 } to denote the set of active

constraints with non-zero dual solution, with h I := [ h i ] i ∈I and λ ∗ I := [ λ ∗ i ] i ∈I being the constraints and dual variables, respectively, corresponding to I .

Observe that as currently stated, (5.2) leads to a second-order computation of dy ∗ ( x ) /dx . In the rest of the section, we provide a fully first-order approximate hypergradient oracle by constructing an equivalent reformulation of Problem 4.1 using a penalty function.

5.1 Reformulation via the penalty method

We begin by reformulating Problem 4.1 into a single level constrained optimization problem:

where g ∗ ( x ) := min y : h ( x,y ) ≤ 0 g ( x, y ) = g ( x, y ∗ ( x )) and λ ∗ ( x ) is the optimal dual solution. The equivalence of this reformulation to Problem 4.1 is spelled out in Appendix D. From (5.3), we define the following penalty function, crucial to our analysis:

where α = [ α 1 , α 2 ] ≥ 0 are the penalty parameters. Notably, we can compute its derivative with respect to x of (5.4) in a fully first-order fashion by the following expression:

To give some intuition for our choice of penalties in (5.4), we note that the two constraints in (5.3) behave quite differently. The first constraint g ( x, y ) + λ ∗ ( x ) ⊤ h ( x, y ) ≤ g ∗ ( x ) is one-sided, i.e., can only be violated or met, and hence just needs a penalty parameter α 1 to weight the 'violation'. As to the second constraint h ( x, y ) ≤ 0 , it can be arbitrary. To allow for such a 'two-sided' constraint, we penalize only the active constraints I , i.e., we use ∥ h I ( x, y ) ∥ 2 to penalize deviation.

Next, we define the optimal solutions to the penalty function optimization by:

We now show that this minimizer is close to the optimal solution of the LL problem, while suffering only a small constraint violation.

Lemma 5.1. Given any x , the corresponding dual solution λ ∗ ( x ) , primal solution y ∗ ( x ) of the lower optimization problem in Problem 4.1, and y ∗ λ ∗ , α ( x ) as in (5.5), satisfy:

The proof of Lemma 5.1 is based on the Lipschitzness of f and strong convexity of g for sufficiently large α 1 . The aforementioned constraint violation bound on h I ( x, y ) is later used in Lemma 5.2 to bound the inexactness of our proposed gradient oracle.

5.2 Main result: approximating the hypergradient

The main export of this section is the following bound on the approximation of the hypergradient. This, together with the bounds in Lemma 5.1, validate our use of the penalty function in (5.4).

Lemma 5.2. Consider F as in Problem 4.1, L as in (5.4), a fixed x , and y ∗ λ ∗ , α as in (5.5). Then under Assumptions 2.2 and 2.5, we have:

The proof can be found in Appendix G. With this hypergradient approximation guarantee, we design Algorithm 4 to compute an inexact gradient oracle for the hyperobjective F .

Theorem 5.3. Given any accuracy parameter α > 0 , Algorithm 4 outputs ˜ ∇ x F ( x ) such that ∥ ˜ ∇ F ( x ) -∇ F ( x ) ∥ ≤ α within ˜ O ( α -1 ) gradient oracle evaluations.

The full proof of this result may be found in Appendix H.

Algorithm 4 Inexact Gradient Oracle for General Inequality Constraints

1: Input: Upper level variable x , accuracy α , penalty parameters α 1 = α -2 , α 2 = α -4 .

2: Compute y ∗ , λ ∗ , and active constraints I of the constrained LL problem min y : h ( x,y ) ≤ 0 g ( x, y ) .

3: Define penalty function L λ ∗ , α ( x, y ) by (5.4)

4: Compute the minimizer y ∗ λ ∗ , α = arg min y L λ ∗ , α ( x, y ) (as in (5.5)).

5: Output: ˜ ∇ F := ∇ x L λ ∗ , α ( x, y ∗ λ ∗ , α ) .

0.0

0.2

0.4

0.6

0.8

1.0

Gradient error

0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

850

900

950

Iteration

0.0

0.2

0.4

0.6

0.8

1.0

Optimality gap

cvxpylayer

F2CBA (Algorithm 3)

(a) Convergence and gradient error of Fully First-order Constrained Bilevel Algorithm (F2CBA) compared to cvxpylayer [18].

Figure 1: with d x = 100 , d y = 200 , n const iterations, gradient exactness α , and

0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

850

900

950

Iteration

0.0

0.5

1.0

1.5

2.0

Optimality gap

F2CBA

2 =0.0001

F2CBA

2 =0.001

F2CBA

2 =0.01

F2CBA

2 =0.1

F2CBA

2 =1.0

100

200

300

400

500

600

700

800

900

1000

Inner level dimension

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Time (s)

cvxpylayer

F2CBA

We run Algorithm 3 using Algorithm 4 on the bilevel optimization in the toy example in Problem L.1 = d y / 5 , and accuracy α = 0 . 1 . Figure 2a, Figure 2b, Figure 2c vary # of d y , respectively, to compare the performance under different settings.

6 Experiments

We generate instances of the following constrained bilevel optimization problem:

where h ( x, y ) = Ay -b is a d h -dim linear constraint. The PSD matrix Q ∈ R d y × d y , c ∈ R d y , P ∈ R d x × d y , and constraints A ∈ R d h × d y , b ∈ R d h are randomly generated from normal distributions (cf. Appendix K). We compare Algorithm 3 with a non-fully first-order method using cvxpyLayer [18]. Both algorithms use Adam [90] to control the learning rate, and are averaged over 10 random seeds.

Figure 2a shows that both the algorithms converge to the same optimal solution at the same rate. Simultaneously, the colorful bars represent the gradient differences between two methods, showing the inexactness of our gradients. Figure 2b additionally varies this inexactness to demonstrate its impact on convergence with standard deviation plotted. Figure 2c compares the computation costs for different lower-level problem sizes. Our fully first-order method significantly outperforms, in computation cost, the non-fully first-order method implemented using differentiable optimization method. The implementation can be found in https://github.com/guaguakai/constrained-bilevel-optimization .

7 Limitations and future directions

One limitation to our approach is that the inexact gradient oracle we constructed in Section 5 requires access to the exact dual multiplier λ ∗ . For a first-order algorithm, the closest proxy one could get to this would be a highly accurate approximation (which could be computed up to ϵ error in O (log(1 /ϵ )) gradient oracle evaluations). Removing this 'exact dual access' assumption (Assumption 2.5) would be an important result.

Another important direction for future work would be to extend the hypergradient stationarity guarantee of Theorem 1.4 to bilevel programs with general convex constraints. To this end, we conjecture that the use of a primal-only gradient approximation oracle could be potentially effective.

Finally, our current rate of ˜ O ( δ -1 ϵ -4 ) oracle calls for reaching ( δ, ϵ ) -Goldstein stationarity is not necessarily inherent to the problem; indeed, it might be the case that an alternate approach could improve it to the best known rate of O ( δ -1 ϵ -3 ) for nonsmooth nonconvex optimization.

Acknowledgments

GKis supported by an Azrieli Foundation graduate fellowship. KW is supported by Schmidt Sciences AI2050 Fellowship and NSF IIS-2403240. SS acknowledges generous support from the Alexander von Humboldt Foundation. SP is supported by NSF CCF-2112665 (TILOS AI Research Institute) and gratefully acknowledges Ali Jadbabaie for many useful discussions. ZZ is supported by NSF FODSI Fellowship.

References

Jerome Bracken and James T McGill. 'Mathematical programs with optimization problems in the constraints'. In: Operations research 21.1 (1973), pp. 37-44 (cit. on p. 1).

Benoît Colson, Patrice Marcotte, and Gilles Savard. 'An overview of bilevel optimization'. In: Annals of operations research 153 (2007), pp. 235-256 (cit. on p. 1).

Jonathan F Bard. Practical bilevel optimization: algorithms and applications . Vol. 30. Springer Science & Business Media, 2013 (cit. on p. 1).

Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. 'A review on bilevel optimization: From classical to evolutionary approaches and applications'. In: IEEE transactions on evolutionary computation 22.2 (2017), pp. 276-295 (cit. on p. 1).

Jake Snell, Kevin Swersky, and Richard Zemel. 'Prototypical networks for few-shot learning'. In: Advances in neural information processing systems 30 (2017) (cit. on p. 1).

Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. 'Meta-learning with differentiable closed-form solvers'. In: arXiv preprint arXiv:1805.08136 (2018) (cit. on p. 1).

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. 'Meta-learning with implicit gradients'. In: Advances in neural information processing systems 32 (2019) (cit. on p. 1).

Kaiyi Ji, Jason D Lee, Yingbin Liang, and H Vincent Poor. 'Convergence of meta-learning with task-specific adaptation over partial parameters'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 11490-11500 (cit. on p. 1).

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. 'Bilevel programming for hyperparameter optimization and meta-learning'. In: International conference on machine learning . PMLR. 2018, pp. 1568-1577 (cit. on pp. 1, 3).

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. 'Truncated backpropagation for bilevel optimization'. In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR. 2019, pp. 1723-1732 (cit. on pp. 1, 3).

Matthias Feurer and Frank Hutter. 'Hyperparameter optimization'. In: Automated machine learning: Methods, systems, challenges (2019), pp. 3-33 (cit. on p. 1).

Vijay Konda and John Tsitsiklis. 'Actor-critic algorithms'. In: Advances in neural information processing systems 12 (1999) (cit. on p. 1).

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018 (cit. on p. 1).

MHong, H Wai, Z Wang, and Z Yang. 'A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv e-prints, art'. In: arXiv preprint arXiv:2007.05170 (2020) (cit. on p. 1).

Haifeng Zhang, Weizhe Chen, Zeren Huang, Minne Li, Yaodong Yang, Weinan Zhang, and Jun Wang. 'Bi-level actor-critic for multi-agent coordination'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 34. 05. 2020, pp. 7325-7332 (cit. on p. 1).

Saeed Ghadimi and Mengdi Wang. 'Approximation methods for bilevel programming'. In: arXiv preprint arXiv:1802.02246 (2018) (cit. on pp. 1, 3, 24).

Brandon Amos and J Zico Kolter. 'Optnet: Differentiable optimization as a layer in neural networks'. In: International Conference on Machine Learning . PMLR. 2017, pp. 136-145 (cit. on pp. 1, 3).

Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. 'Differentiable convex optimization layers'. In: Advances in neural information processing systems 32 (2019) (cit. on pp. 1, 3, 10, 37, 38).

Priya Donti, Brandon Amos, and J Zico Kolter. 'Task-based end-to-end model learning in stochastic optimization'. In: Advances in neural information processing systems 30 (2017) (cit. on p. 1).

Bryan Wilder, Bistra Dilkina, and Milind Tambe. 'Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 33. 01. 2019, pp. 1658-1665 (cit. on pp. 1, 2).

James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder. 'End-to-end constrained optimization learning: A survey'. In: arXiv preprint arXiv:2103.16378 (2021) (cit. on p. 1).

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. 'Meta-learning with differentiable convex optimization'. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019, pp. 10657-10665 (cit. on p. 1).

Bo Tang and Elias B Khalil. 'Pyepo: A pytorch-based end-to-end predict-then-optimize library for linear and integer programming'. In: arXiv preprint arXiv:2206.14234 (2022) (cit. on p. 1).

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 'Deep equilibrium models'. In: Advances in neural information processing systems 32 (2019) (cit. on p. 1).

Akshay Mehra and Jihun Hamm. 'Penalty method for inversion-free deep bilevel optimization'. In: Asian conference on machine learning . PMLR. 2021, pp. 347-362 (cit. on p. 1).

Kaiyi Ji, Junjie Yang, and Yingbin Liang. 'Bilevel optimization: Convergence analysis and enhanced design'. In: International conference on machine learning . PMLR. 2021, pp. 48824892 (cit. on p. 1).

Kai Wang, Sanket Shah, Haipeng Chen, Andrew Perrault, Finale Doshi-Velez, and Milind Tambe. 'Learning mdps from features: Predict-then-optimize for sequential decision making by reinforcement learning'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 8795-8806 (cit. on p. 1).

Bo Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. 'Bome! bilevel optimization made easy: A simple first-order approach'. In: Advances in neural information processing systems 35 (2022), pp. 17248-17262 (cit. on pp. 1, 3).

Kai Wang, Lily Xu, Andrew Perrault, Michael K Reiter, and Milind Tambe. 'Coordinating followers to reach better equilibria: End-to-end gradient descent for stackelberg games'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 36. 5. 2022, pp. 5219-5227 (cit. on p. 2).

Paul Dütting, Zhe Feng, Harikrishna Narasimhan, David C Parkes, and Sai S Ravindranath. 'Optimal auctions through deep learning'. In: Communications of the ACM 64.8 (2021), pp. 109-116 (cit. on p. 2).

Jiuping Xu, Yan Tu, and Ziqiang Zeng. 'Bilevel optimization of regional water resources allocation problem under fuzzy random environment'. In: Journal of Water Resources Planning and Management 139.3 (2013), pp. 246-264 (cit. on p. 2).

Walter J Gutjahr and Nada Dzubur. 'Bi-objective bilevel optimization of distribution center locations considering user equilibria'. In: Transportation Research Part E: Logistics and Transportation Review 85 (2016), pp. 1-22 (cit. on p. 2).

Yue Zhang, Oded Berman, Patrice Marcotte, and Vedat Verter. 'A bilevel model for preventive healthcare facility network design with congestion'. In: IIE Transactions 42.12 (2010), pp. 865880 (cit. on p. 2).

Amir M Fathollahi-Fard, Mostafa Hajiaghaei-Keshteli, Reza Tavakkoli-Moghaddam, and Neale R Smith. 'Bi-level programming for home health care supply chain considering outsourcing'. In: Journal of Industrial Information Integration 25 (2022), p. 100246 (cit. on p. 2).

Adam N Elmachtoub and Paul Grigas. 'Smart 'predict, then optimize''. In: Management Science 68.1 (2022), pp. 9-26 (cit. on p. 2).

Miguel Angel Muñoz, Salvador Pineda, and Juan Miguel Morales. 'A bilevel framework for decision-making under uncertainty with contextual information'. In: Omega 108 (2022), p. 102575 (cit. on p. 2).

Prashant Khanduri, Ioannis Tsaknakis, Yihua Zhang, Jia Liu, Sijia Liu, Jiawei Zhang, and Mingyi Hong. 'Linearly constrained bilevel optimization: A smoothed implicit gradient approach'. In: International Conference on Machine Learning . PMLR. 2023, pp. 1629116325 (cit. on pp. 2-5).

Wei Yao, Chengming Yu, Shangzhi Zeng, and Jin Zhang. Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithm . 2024. arXiv: 2401.16164 [cs.LG] (cit. on pp. 2-4).

Zhaosong Lu and Sanyou Mei. First-order penalty methods for bilevel optimization . 2024. arXiv: 2301.01716 [math.OC] (cit. on pp. 2-4).

Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. 'A fully first-order method for stochastic bilevel optimization'. In: International Conference on Machine Learning . PMLR. 2023, pp. 18083-18113 (cit. on pp. 2, 3).

AA Goldstein. 'Optimization of Lipschitz continuous functions'. In: Mathematical Programming 13 (1977), pp. 14-22 (cit. on pp. 2, 4).

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. 'Problem complexity and method efficiency in optimization'. In: (1983) (cit. on p. 2).

Jingzhao Zhang, Hongzhou Lin, Stefanie Jegelka, Suvrit Sra, and Ali Jadbabaie. 'Complexity of finding stationary points of nonconvex nonsmooth functions'. In: International Conference on Machine Learning . PMLR. 2020, pp. 11173-11182 (cit. on pp. 2, 6-8).

Guy Kornowski and Ohad Shamir. 'Oracle complexity in nonsmooth nonconvex optimization'. In: Journal of Machine Learning Research 23.314 (2022), pp. 1-44 (cit. on p. 2).

Zhe Zhang and Guanghui Lan. 'Solving Convex Smooth Function Constrained Optimization Is Almost As Easy As Unconstrained Optimization'. In: arXiv preprint arXiv:2210.05807 (2022) (cit. on pp. 2, 7, 24, 34).

Guy Kornowski and Ohad Shamir. 'An algorithm with optimal dimension-dependence for zero-order nonsmooth nonconvex stochastic optimization'. In: Journal of Machine Learning Research 25.122 (2024), pp. 1-14 (cit. on pp. 3, 25).

Ashok Cutkosky, Harsh Mehta, and Francesco Orabona. In: International Conference on Machine Learning . PMLR. 2023, pp. 6643-6670 (cit. on pp. 3, 6-8, 26).

Damek Davis, Dmitriy Drusvyatskiy, Yin Tat Lee, Swati Padmanabhan, and Guanghao Ye. 'A gradient sampling method with complexity guarantees for Lipschitz functions in high and low dimensions'. In: Advances in neural information processing systems 35 (2022), pp. 6692-6703 (cit. on pp. 3, 7, 8).

G Anandalingam and DJ White. 'A solution method for the linear static Stackelberg problem using penalty functions'. In: IEEE Transactions on automatic control 35.10 (1990), pp. 11701173 (cit. on p. 3).

Yo Ishizuka and Eitaro Aiyoshi. 'Double penalty method for bilevel optimization problems'. In: Annals of Operations Research 34.1 (1992), pp. 73-88 (cit. on p. 3).

Douglas J White and G Anandalingam. 'A penalty function approach for solving bi-level linear programs'. In: Journal of Global Optimization 3 (1993), pp. 397-419 (cit. on p. 3).

Luis Vicente, Gilles Savard, and Joaquim Júdice. 'Descent approaches for quadratic bilevel programming'. In: Journal of Optimization theory and applications 81.2 (1994), pp. 379-399 (cit. on p. 3).

DL Zhu. 'Optimality conditions for bilevel programming problems'. In: Optimization 33.1 (1995), pp. 9-27 (cit. on p. 3).

JJ Ye and DL Zhu. 'Exact penalization and necessary optimality conditions for generalized bilevel programming problems'. In: SIAM Journal on optimization 7.2 (1997), pp. 481-507 (cit. on p. 3).

Justin Domke. 'Generic methods for optimization-based modeling'. In: Artificial Intelligence and Statistics . PMLR. 2012, pp. 318-326 (cit. on p. 3).

Fabian Pedregosa. 'Hyperparameter optimization with approximate gradient'. In: International conference on machine learning . PMLR. 2016, pp. 737-746 (cit. on p. 3).

Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. 'On differentiating parameterized argmin and argmax problems with application to bi-level optimization'. In: arXiv preprint arXiv:1607.05447 (2016) (cit. on p. 3).

Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard Zemel. 'Reviving and improving recurrent back-propagation'. In: International Conference on Machine Learning . PMLR. 2018, pp. 3082-3091 (cit. on p. 3).

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. 'On the iteration complexity of hypergradient computation'. In: International Conference on Machine Learning . PMLR. 2020, pp. 3748-3758 (cit. on p. 3).

Jonathan Lorraine, Paul Vicol, and David Duvenaud. 'Optimizing millions of hyperparameters by implicit differentiation'. In: International conference on artificial intelligence and statistics . PMLR. 2020, pp. 1540-1552 (cit. on p. 3).

Dougal Maclaurin, David Duvenaud, and Ryan Adams. 'Gradient-based hyperparameter optimization through reversible learning'. In: International conference on machine learning . PMLR. 2015, pp. 2113-2122 (cit. on p. 3).

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 'Forward and reverse gradient-based hyperparameter optimization'. In: International Conference on Machine Learning . PMLR. 2017, pp. 1165-1173 (cit. on p. 3).

Risheng Liu, Xuan Liu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. 'A value-functionbased interior-point method for non-convex bi-level optimization'. In: International Conference on Machine Learning . PMLR. 2021, pp. 6882-6892 (cit. on p. 3).

Jane J Ye, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. 'Difference of convex algorithms for bilevel programs with applications in hyperparameter selection'. In: Mathematical Programming 198.2 (2023), pp. 1583-1616 (cit. on p. 3).

Lucy L. Gao, Jane J. Ye, Haian Yin, Shangzhi Zeng, and Jin Zhang. Moreau Envelope Based Difference-of-weakly-Convex Reformulation and Algorithm for Bilevel Programs . 2024. arXiv: 2306.16761 [math.OC] (cit. on p. 3).

Tianyi Chen, Yuejiao Sun, and Wotao Yin. 'Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 25294-25307 (cit. on p. 3).

Tianyi Chen, Yuejiao Sun, Quan Xiao, and Wotao Yin. 'A single-timescale method for stochastic bilevel optimization'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2022, pp. 2466-2488 (cit. on p. 3).

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. 'A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actorcritic'. In: SIAM Journal on Optimization 33.1 (2023), pp. 147-180 (cit. on p. 3).

Zeeshan Akhtar, Amrit Singh Bedi, Srujan Teja Thomdapu, and Ketan Rajawat. 'Projectionfree stochastic bi-level optimization'. In: IEEE Transactions on Signal Processing 70 (2022), pp. 6332-6347 (cit. on p. 3).

Ruichen Jiang, Nazanin Abolfazli, Aryan Mokhtari, and Erfan Yazdandoost Hamedani. 'A conditional gradient-based method for simple bilevel optimization with convex lower-level problem'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2023, pp. 10305-10323 (cit. on p. 3).

Nazanin Abolfazli, Ruichen Jiang, Aryan Mokhtari, and Erfan Yazdandoost Hamedani. 'An Inexact Conditional Gradient Method for Constrained Bilevel Optimization'. In: arXiv preprint arXiv:2306.02429 (2023) (cit. on p. 3).

Jincheng Cao, Ruichen Jiang, Nazanin Abolfazli, Erfan Yazdandoost Hamedani, and Aryan Mokhtari. 'Projection-free methods for stochastic simple bilevel optimization with convex lower-level problem'. In: Advances in Neural Information Processing Systems 36 (2024) (cit. on p. 3).

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. 'A near-optimal algorithm for stochastic bilevel optimization via double-momentum'. In: Advances in neural information processing systems 34 (2021), pp. 30271-30283 (cit. on p. 3).

Zhishuai Guo, Quanqi Hu, Lijun Zhang, and Tianbao Yang. 'Randomized stochastic variance-reduced methods for multi-task stochastic bilevel optimization'. In: arXiv preprint arXiv:2105.02266 (2021) (cit. on p. 3).

Junjie Yang, Kaiyi Ji, and Yingbin Liang. 'Provably faster algorithms for bilevel optimization'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 13670-13682 (cit. on p. 3).

Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. 'A framework for bilevel optimization that enables stochastic and global variance reduction algorithms'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 26698-26710 (cit. on p. 3).

Shoham Sabach and Shimrit Shtern. 'A first order method for solving convex bilevel optimization problems'. In: SIAM Journal on Optimization 27.2 (2017), pp. 640-660 (cit. on p. 3).

Mostafa Amini and Farzad Yousefian. 'An iterative regularized incremental projected subgradient method for a class of bilevel optimization problems'. In: 2019 American Control Conference (ACC) . IEEE. 2019, pp. 4069-4074 (cit. on p. 3).

Mostafa Amini and Farzad Yousefian. 'An iterative regularized mirror descent method for ill-posed nondifferentiable stochastic optimization'. In: arXiv preprint arXiv:1901.09506 (2019) (cit. on p. 3).

Roey Merchav and Shoham Sabach. 'Convex Bi-level Optimization Problems with Nonsmooth Outer Objective Function'. In: SIAM Journal on Optimization 33.4 (2023), pp. 3114-3142 (cit. on p. 3).

Ioannis Tsaknakis, Prashant Khanduri, and Mingyi Hong. 'An implicit gradient-type method for linearly constrained bilevel problems'. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2022, pp. 5438-5442 (cit. on p. 3).

Siyuan Xu and Minghui Zhu. 'Efficient gradient approximation method for constrained bilevel optimization'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 37. 10. 2023, pp. 12509-12517 (cit. on p. 3).

Lesi Chen, Yaohua Ma, and Jingzhao Zhang. 'Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization'. In: arXiv preprint arXiv:2306.14853 (2023) (cit. on p. 4).

Frank H Clarke. 'Generalized gradients of Lipschitz functionals'. In: Advances in Mathematics 40.1 (1981), pp. 52-67 (cit. on p. 4).

Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. Lectures on stochastic programming: modeling and theory . SIAM, 2021 (cit. on p. 4).

Michael Jordan, Guy Kornowski, Tianyi Lin, Ohad Shamir, and Manolis Zampetakis. 'Deterministic nonsmooth nonconvex optimization'. In: The Thirty Sixth Annual Conference on Learning Theory . PMLR. 2023, pp. 4570-4597 (cit. on p. 7).

Siyu Kong and AS Lewis. 'The cost of nonconvexity in deterministic nonsmooth optimization'. In: Mathematics of Operations Research (2023) (cit. on p. 7).

Benjamin Grimmer and Zhichao Jia. 'Goldstein Stationarity in Lipschitz Constrained Optimization'. In: arXiv preprint arXiv:2310.03690 (2023) (cit. on p. 7).

Olivier Devolder, François Glineur, and Yurii Nesterov. 'First-order methods of smooth convex optimization with inexact oracle'. In: Mathematical Programming 146 (2014), pp. 37-75 (cit. on p. 7).

Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: arXiv preprint arXiv:1412.6980 (2014) (cit. on pp. 10, 38).

Guanghui Lan. First-order and stochastic optimization methods for machine learning . Vol. 1. Springer, 2020 (cit. on p. 23).

Ohad Shamir. 'An optimal algorithm for bandit and zero-order convex optimization with twopoint feedback'. In: The Journal of Machine Learning Research 18.1 (2017), pp. 1703-1713 (cit. on p. 24).

Simon S Du and Wei Hu. 'Linear convergence of the primal-dual gradient method for convexconcave saddle point problems without strong convexity'. In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR. 2019, pp. 196-205 (cit. on p. 36).

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 'Pytorch: An imperative style, high-performance deep learning library'. In: Advances in neural information processing systems 32 (2019) (cit. on p. 37).

Steven Diamond and Stephen Boyd. 'CVXPY: A Python-embedded modeling language for convex optimization'. In: Journal of Machine Learning Research 17.83 (2016), pp. 1-5 (cit. on p. 37).

Appendix

A Notation

We use ⟨· , ·⟩ to denote inner products and ∥ · ∥ for the Euclidean norm. Unless transposed, all vectors are column vectors. For f : R d 2 → R d 1 its Jacobian with respect to x ∈ R d 2 is ∇ f ∈ R d 1 × d 2 . For f : R d → R , we overload ∇ f to refer to its gradient (the transposed Jacobian), a column vector. We use ∇ x to denote partial derivatives with respect to x .

A function f : R n → R m is L -Lipschitz if for any x, y , we have ∥ f ( x ) -f ( y ) ∥ ≤ L ∥ x -y ∥ . A differentiable function f : R n → R is convex if for any x, y ∈ R n we have f ( y ) ≥ f ( x ) + ∇ f ( x ) ⊤ ( y -x ) ; it is µ -strongly convex if f -µ 2 ∥ · ∥ 2 is convex; it is β -smooth if ∇ f is β -Lipschitz.

For a Lipschitz function f , a point x is ( δ, ϵ ) -stationary if within a δ -ball around x , there exists a convex combination of subgradients of f with norm at most ϵ . For a differentiable function f , we say that x is ϵ -stationary if ∥∇ f ( x ) ∥ ≤ ϵ .

B Proofs from Section 3

In this section, we provide the full proofs of claims for bilevel programs with linear equality constraints, as stated in Section 3. We first state a few technical results using the implicit function theorem that we repeatedly invoke in our results for this setting.

Lemma B.1. Fix a point x . Given y ∗ = arg min y : h ( x,y )=0 g ( x, y ) where g is strongly convex in y and λ ∗ is the dual optimal variable for this problem, define L eq ( x, y, λ ) = g ( x, y ) + ⟨ λ, h ( x, y ) ⟩ . Then, we have

︷︷ H for linear equality constraints

Proof. Since g is strongly convex, by linear constraint qualification, the KKT condition is both sufficient and necessary condition for optimality. Hence, consider the following KKT system obtained via first order optimality of y ∗ , with dual optimal variable λ ∗ :

Differentiating the system of equations in (B.1) with respect to x and rearranging terms in a matrixvector format yields:

Noting that ∇ 2 yy L eq ( x, y, λ ) = ∇ 2 yy g ( x, y ) + ∇ 2 yy ⟨ λ, h ( x, y ) ⟩ , we can write (B.2) in the form shown in the lemma.

Lemma B.2. Consider the setup in Lemma B.1. The matrix H defined in (3.4) is invertible if the Hessian ∇ 2 yy L eq ( x, y ∗ , λ ∗ ) := ∇ 2 yy g ( x, y ∗ ) + ∇ 2 yy ⟨ λ ∗ , h ( x, y ∗ ) ⟩ satisfies ∇ 2 yy L eq ( x, y ∗ , λ ∗ ) ≻ 0 over the tangent plane T := { y : ∇ y h ( x, y ∗ ) y = 0 } and ∇ y h has full rank.

̸

̸

Proof. Let u = [ y, λ ] . We show that Hu = 0 implies u = 0 , which in turn implies invertibility of H . If ∇ y h ( x, y ∗ ) y = 0 , then by construction of u and H , we must also have Hu = 0 . Otherwise if ∇ y h ( x, y ∗ ) y = 0 and y = 0 , the quadratic form u ⊤ Hu is positive, as seen by

where the final step is by the assumption of L eq being positive definite over the defined tangent plane T = { y : ∇ y h ( x, y ∗ ) y = 0 } . If y = 0 while Hu = 0 , then ∇ y h having full rank implies λ = 0 . Combined with y = 0 , this means u = 0 , as required when Hu = 0 . This concludes the proof.

̸

Corollary B.3. For Problem 3.1 under Assumption 2.2 and Assumption 2.3, the matrix H (as defined in (3.4)) is non-singular. Further, there exists a finite C H such that ∥ H -1 ∥ ≤ C H .

Proof. Since we are assuming strong convexity of g , Lemma B.2 applies, yielding the claimed invertibility of H . Combined with the boundedness of variables x (per Assumption 2.3) and continuity of the inverse implies a bound on ∥ H -1 ∥ .

B.1 Construction of the inexact gradient oracle

We now show how to construct the inexact gradient oracle for the objective F in Problem 3.1. As sketched in Section 3, we then use this oracle in a projected gradient descent algorithm to get the claimed guarantee.

Lemma B.4. Consider Problem 3.1 under Assumption 2.2 and Assumption 2.3. Let y ∗ δ be as defined in (3.3). Then, for any δ ∈ [0 , ∆] with ∆ ≤ µ g / 2 C f , the following relation is valid:

Proof. The first-order optimality condition applied to g ( x, y ) + δf ( x, y ) at y ∗ and y ∗ δ gives

which upon adding and subtracting ∇ y f ( x, y ∗ ) transforms into

Similarly, the first-order optimality condition applied to g at y ∗ and y ∗ δ gives

Adding Inequality (B.3) and Inequality (B.4) and rearranging yields

Applying to the left side above a lower bound via strong convexity of g + δf and to the right hand side an upper bound via Cauchy-Schwarz inequality, we have

where s is the strong convexity of g + δf . Since f is C f -smooth, the worst case value of this is s = µ g -δC f = µ g -µ g 2 C f C f = µ g / 2 , which when plugged in Inequality (B.5) then gives the claimed bound.

Lemma B.5. Consider Problem 3.1 under Assumption 2.2 and Assumption 2.3. Then the following relation is valid.

Proof. Recall that by definition, g is strongly convex and y ∗ = arg min y : h ( x,y )=0 g ( x, y ) . Hence, we can apply Lemma B.1. Combining this with Lemma B.2 and further applying that linearity of h implies ∇ 2 yy h = 0 and ∇ 2 xy h = 0 , we obtain the following:

So we can express the right-hand side of the claimed equation in the lemma statement by

which can be further simplified to

We now apply Lemma B.1 to the perturbed problem defined in (3.3). We know from Lemma B.4 that lim δ → 0 y ∗ δ = y ∗ . The associated KKT system is given by

Taking the derivative with respect of (B.7) gives the following implicit system, where we used the fact that h is linear and hence ∇ 2 yy h = 0 :

For a sufficiently small δ , we have ∇ 2 yy g ( x, y ∗ δ ) + δ ∇ 2 yy f ( x, y ∗ δ ) ⪰ µ g 2 I , which implies invertibility of H δ by an application of Lemma B.2. Since Lemma B.4 implies lim δ → 0 y ∗ δ = y ∗ , we get

So we can express the left-hand side of the expression in the lemma statement by

which matches (B.6) (since ( ∇ 2 g ) ⊤ = ∇ 2 g ), thus concluding the proof.

yx xy

Lemma 3.3. The solution y ∗ (as defined in Problem 3.1) is O ( C H · ( C g + ∥ A ∥ )) -Lipschitz continuous and O ( C 3 H · S g · ( C g + ∥ A ∥ ) 2 ) -smooth as a function of x . Thus the hyper-objective F is gradientLipschitz with a smoothness constant of C F := O { ( L f + C f + C g ) C 3 H S g ( L g + ∥ A ∥ ) 2 } .

Proof. Rearranging (3.4) and applying Corollary B.3, we have

This implies a Lipschitz bound of C H · ( C g + ∥ A ∥ ) . Next, note that in the case with linear equality constraints, the terms in (B.2) involving second-order derivatives of h are all zero; differentiating (B.2) with respect to x , we notice that the linear system we get again has the same matrix H from before. We can therefore again perform the same inversion and apply the bound on ∥ H -1 ∥ and on the third-order derivatives of g (Assumption 2.3) to observe that ∥ d 2 y ∗ dx 2 ∥ ≤ O ( C H · S g ∥ dy ∗ dx ∥ 2 ) = O ( C 3 H · S g · ( C g + ∥ A ∥ ) 2 ) , where we are hiding numerical constants in the Big-Oh notation.

As a result, we can calculate the Lipschitz smoothness constant associated with the hyper-objective F by

Lemma 3.2. For Problem 3.1 under Assumption 2.2, with v x as in (3.2), the following holds:

Proof. For simplicity, we adopt the following notation throughout this proof: g xy ( x, y ) = ∇ 2 xy g, and g xyy denotes the tensor such that its ijk entry is given by ∂ 3 g ∂x i ∂y j ∂y k . We first consider the terms involving g . By the fundamental theorem of calculus, we have

As a result, we have

We now bound each of the terms on the right-hand side of (B.9). For the first term, we have

where M y is the Lipschitz bound on y ∗ as shown in Lemma 3.3, and S g is the smoothness of g from Assumption 2.3. For the second term on the right-hand side of (B.9), we have

where C g is the bound on smoothness of g as in Assumption 2.3, and C y is the bound on ∥ d 2 y ∗ dx 2 ∥ from Lemma 3.3. For the terms involving the function h , we have

where C ℓ is the bound on ∥ d 2 λ ∗ ds 2 ∥ from Lemma 3.3. Combining (B.9), Inequality (B.10), Inequality (B.11), and Inequality (B.12), along with Lemma B.5, Corollary B.3, and Lemma 3.3, we have that overall bound is

B.2 Cost of linear equality constrained bilevel program

Algorithm 5 The Fully First-Order Method for Bilevel Equality Constrained Problem

1: Input: Current x 0 , accuracy ϵ , perturbation δ = ϵ 2 / 8 C 2 F R X with C F = 2( L f + C f + C g ) C 3 H S g ( L g + ∥ A ∥ ) 2 , accuracy for the lower level problem ˜ δ = 2( C g + ∥ A ∥ ) δ 2 .

2: for t=0,1,2,... do

3: Run Algorithm 6 to generate ˜ δ -accurate primal and dual solutions (ˆ y ∗ , ˆ λ ∗ ) for

4: Run Algorithm 6 to generate ˜ δ -accurate primal and dual solutions (ˆ y ∗ δ , ˆ λ ∗ δ ) for

5: Compute ˆ v t := ∇ x [ g ( x t , ˆ y ∗ δ )+ ˆ λ ∗ δ h ( x, ˆ y ∗ )] -∇ x [ g ( x t , ˆ y ∗ )+ ˆ λ ∗ h ( x, ˆ y ∗ )] δ , set

6: Set x t +1 ← arg min z ∈X ∥ z -( x t -1 C F ˜ ∇ F ( x t )) ∥ 2 .

Theorem 3.1. Consider Problem 3.1 under Assumption 2.2, and let κ = C g /µ g be the condition number of g . Then Algorithm 5 finds an ϵ -stationary point (in terms of gradient mapping, see (B.14)) after T = ˜ O ( C F ( F ( x 0 ) -inf F ) √ κϵ -2 ) oracle calls to f and g , where C F := 2( L f + C f + C g ) C 3 H S g ( L g + ∥ A ∥ ) 2 is the smoothness constant of the hyperobjective F .

Proof. We first show the inexact gradient ˜ ∇ F ( x t ) generated in Algorithm 5 is an δ -accurate approximation to the hyper-gradient ∇ F ( x t ) . Consider the inexact gradient defined in (3.2)

Thus we get

Applied to the C F -smooth hyper-objective F , such an inexact gradient oracle satisfies the requirement for Proposition B.6. Thus an ϵ -stationary point with ∥G F ( x t ) ∥ ≤ ϵ (see Eq. (B.14) for the definition of gradient mapping) must be found in N = O ( C F ( F ( x 0 ) -F ∗ ) ϵ 2 ) iterations. Noting the evaluation of inexact solutions (ˆ y ∗ , ˆ λ ∗ , ˆ y ∗ δ , ˆ λ ∗ δ ) requires ˜ O ( √ C g /µ g ) first order oracle evaluations, we arrive at the total oracle complexity of ˜ O ( √ C g /µ g C F ( F ( x 0 ) -F ∗ ) ϵ 2 ) for finding an ϵ -stationary point.

B.3 The cost of inexact projected gradient descent method

In this subsection, we state the number of iterations required by projected gradient descent method to find an ϵ -stationary point using inexact gradient oracles. Specifically, we consider the following non-convex smooth problem where the objective F is assumed to be C F -Lipschitz smooth:

Since the feasible region X is compact, we use the norm of the following gradient mapping G F ( x ) as the stationarity criterion

Initialized to some x 0 and the inexact gradient oracle ˜ ∇ F , the updates of the inexact projected gradient descent method is given by

For t=1,2,..., N do :

The next proposition calculates the complexity result.

Proposition B.6. Consider the constrained optimization problem in (B.13) with F being C F -Lipschitz smooth and X having a radius of R . When supplied with a δ = ϵ 2 / 4 C F R -inexact gradient oracle ˜ ∇ F , that is, ∥∇ F ( x ) -˜ ∇ F ( x ) ∥ ≤ δ , the solution generated by the projected gradient descent method (B.15) satisfies

that is, it takes at most O ( C F ( F ( x 0 ) -F ∗ ) ϵ 2 ) iterations to generate some ¯ x with ∥G F ( x ) ∥ ≤ ϵ .

Proof. By C F -smoothness of F , we have

We now show that 1 β ˜ G F ( x t ) ⊤ ( ˜ G F ( x t ) - ∇ f ( x t )) ≤ 0 . Let ˜ y t = x t -1 C F ˜ ∇ F ( x t ) , and let y t = x t -1 C F ∇ f ( x t ) . Then have that )) ))

where the penultimate inequality uses the fact that X is a convex set, and R is the diameter of the set X . Combining this with Inequality (B.16), we have that the function decrease per iteration is

Summing over N iterations telescopes the terms, we get

Substituting in N = 4 ϵ 2 C F ( F ( x 0 ) -F ∗ ) and the choice of δ = ϵ 2 / 4 C F R , we get

Taking into account the fact that ∥ ˜ G F ( x t ) -G F ( x t ) ∥ ≤ ∥∇ F ( x t ) -˜ ∇ F ( x t ) ∥ ≤ δ , we obtain the desired result.

B.4 The cost of generating approximate solutions to the linearly constrained LL problem

In this subsection, we address the issue of generating approximations to the primal and dual solutions ( y ∗ , λ ∗ ) associated with the lower-level problem in Problem 3.1. These approximations are required for computing the approximate hypergradient in Algorithm 1. For notational simplicity, we are going to consider the following constrained strongly convex problem:

We propose the following simple scheme to generate approximate solutions to Problem B.17.

The following lemma tells us that ˆ λ is close to λ ∗ if B has full row rank.

Lemma B.7. Suppose g in Problem B.17 is a C g -Lipschitz smooth, and the matrix B has full row rank such that the following matrix M B is invertible

Then the approximate solution ( ˆ λ, ˆ y ) from (B.18) satisfies ∥ ˆ λ -λ ∗ ∥ ≤ ∥ M -1 B ∥ (1 + C g ) δ .

Proof. Since ( λ ∗ , y ∗ ) satisfy the KKT conditions, they are the solution to the following linear system

That is

Thus we get

Now we can just use the AGD method to generate a close enough approximate solution ˆ y and call up the Subroutine in (B.18) to generate the approximate dual solution ˆ λ.

Algorithm 6 The Projected Gradient Method to Generate Primal and Dual Solutions for a Linearly Constrained Problem

1: Input : accuracy requirement ϵ > 0 and linearly constrained problem min y : By = b g ( y ) .

2: Starting from y 0 = 0 and using Y := { y ∈ R d : By = b } as the simple feasible region.

3: Run the Accelerated Gradient Descent (AGD) Method (Section 3.3 in [91]) for N = ⌈ 4 √ C g /µ g log ∥ y ∗ ∥∥ M -1 B ∥ ( C g +1) µ g ϵ ⌉ iterations.

4: Use the y N as the approximate solution ˆ y to generate ˆ λ according to (B.18).

5: return (ˆ y, ˆ λ )

Proposition B.8. Suppose the objective function g is both L g -smooth and µ g -strongly convex, and that the constraint satisfies the assumption in Lemma B.7. Fix an ϵ > 0 , the solution (ˆ y, ˆ λ ) returned by the above procedure satisfies ∥ y ∗ -ˆ y ∥ ≤ ϵ and ∥ ˆ λ -λ ∗ ∥ ≤ ϵ . In another words, the cost of generating ϵ -close primal and dual solutions are bounded by O ( √ C g µ g log 1 ϵ ) .

Proof. With N := ⌈ 4 √ C g /µ g log ∥ y ∗ ∥∥ M -1 B ∥ ( L g +1) µ g ϵ ⌉ , Theorem 3.7 in [91] shows that ∥ y N -ˆ y ∥ ≤ ϵ/ ∥ M -1 B ∥ (1 + L g ) . Then we can apply Lemma B.7 to obtain the desired bound.

On the other hand, the approximate solutions (ˆ y, ˆ λ ) in (B.18) satisfies

We show the right hand side (r.h.s) of the above equation to be close to the r.h.s of (B.19). Let S := { B ⊤ λ : λ ∈ R m } denote the subspace spanned by the rows of B . We can rewrite B ⊤ ˆ λ as the projection of ∇ g (ˆ y ) onto S , that is,

where the second relation follows from the KKT conditon associated with ( λ ∗ , y ∗ ) . Since the projection is an non-expansive operation, we have

We can rewrite (ˆ y, ˆ λ ) as solutions to the following linear system with some ∥ τ ∥ ≤ (1 + C g ) δ ,

C Proofs for Section 4

Our algorithms are based on the Lipschitzness of F , which we prove below.

Lemma 4.3. Under Assumption 2.2 and 2.5, F in Problem 4.1 is O ( L f L y ) -Lipschitz in x .

Proof. By Lemma 2 . 1 of [16], the hypergradient of F computed with respect to the variable x may be expressed as ∇ x F ( x ) = ∇ x f ( x, y ∗ ( x )) + ( dy ∗ ( x ) dx ) ⊤ · ∇ y f ( x, y ∗ ( x )) . Since we impose Lipschitzness on f and y ∗ , we can bound each of the terms of ∇ x F ( x ) by the claimed bound.

C.1 Faster algorithm for low upper-level dimensions

In this section we analyze Algorithm 2, which as stated in Section 4, requires evaluating only the hyperobjective F (as opposed to estimating the hypergradient in Algorithm 3).

The motivation for designing such an algorithm, is that while evaluating ∇ F up to α accuracy requires O ( α -1 ) gradient evaluations, the hyperobjective value can be estimated at a linear rate:

Lemma 4.2 (Proof in Appendix C.1) . Given any x , we can return ˜ F ( x ) such that | F ( x ) -˜ F ( x ) | ≤ α using O ( √ C g /µ g log( L f /α )) first-order oracle calls to f and g .

Proof of Lemma 4.2. We note that it suffices to find ˜ y ∗ such that ∥ ˜ y ∗ -y ∗ ( x ) ∥ ≤ α/L f , since setting ˜ F ( x ) := f ( x, ˜ y ∗ ) will then satisfy | ˜ F ( x ) -F ( x ) | = | f ( x, ˜ y ∗ ) -f ( x, y ∗ ( x )) | ≤ L f · α L f = α by Lispchitzness of f , as required. Noting that y ∗ ( x ) = arg min h ( x,y ) ≤ 0 g ( x, y ) is the solution to a constrained smooth, strongly-convex problem with condition number C g /µ g , it is possible to approximate it up to α/L f with O ( √ C g /µ g log( L f /α )) first-order oracle calls using the result of Zhang and Lan [45].

Accordingly, we consider Algorithm 2, which is a zero-order variant of Algorithm 3, whose guarantee is summarized is the theorem below.

Theorem C.1. Suppose F : R d → R is L -Lipschitz, and that | ˜ F ( · ) -F ( · ) | ≤ α . Then running Algorithm 3 with ρ = min { δ 2 , F ( x 0 ) -inf F L } , ν = δ -ρ, D = Θ ( νϵ 2 ρ 2 dρ 2 L 2 + α 2 d 2 ) , η =

Θ

(

dρ

2

νϵ

L

2

3

ρ

+

4

α

2

d

2

)

2

, outputs a point

x

such that

E

[dist(0

, ∂

δ

F

(

x

))]

ϵ

+

α

with

Combining the result of Theorem C.1 with the complexity of hyperobjective estimation, as given by Lemma 4.2, we obtain convergence to a ( δ, ϵ ) -stationary point of Problem 4.1 with ˜ O ( d x δ -1 ϵ -3 ) gradient calls overall.

C.1.1 Proof of Theorem C.1

Denoting the uniform randomized smoothing F ρ ( x ) := E ∥ z ∥≤ 1 [ F ( x + ρ · z )] where the expectation, here and in what follows, is taken with respect to the uniform measure, it is well known [92, Lemma 10] that

We first show that replacing the gradient estimator with the inexact evaluations ˜ F ( · ) leads to a biased gradient estimator of F .

(

)

out out

it holds that

Proof. For the first bound, we have

while for the second bound

where the last step invoked (C.1).

We are now ready to analyze Algorithm 2. We denote α ′ = αd ρ , ˜ G = √ α 2 d 2 ρ 2 + dL 2 . Since x t = x t -1 +∆ t , we have

By summing over t ∈ [ T ] = [ K × M ] , we get for any fixed sequence u 1 , . . . , u K ∈ R d :

where the last inequality follows by combining Lemma C.2 and Lemma C.3. By setting u k := -∑ M

Finally, note that for all m ∈ [ M ] : ∥ ∥ z ( k -1) M + m -x k ∥ ∥ ≤ MD ≤ ν , therefore ∇ F ρ ( z ( k -1) M + m ) ∈ ∂ ν F ρ ( x k ) ⊂ ∂ δ F ( x k ) , where the last containment is due to [46, Lemma 4] by using our assignment ρ + ν = δ . Invoking the convexity of the Goldstein subdifferential, this implies that

thus it suffices to bound the first three summands in (C.2) by ϵ in order to finish the proof. This happens as long as F ρ ( x 0 ) -inf F ρ Kν ≤ ϵ 3 , αd ρ √ M ≤ ϵ 3 , and L √ d √ M ≤ ϵ 3 , which imply K ≳ F ρ ( x 0 ) -inf F ρ νϵ , M ≳ α 2 d 2 ρ 2 ϵ 2 , and M ≳ L 2 d ϵ 2 . By our assignments of ρ and ν , these result in

completing the proof.

C.2 Proof of Theorem 4.4

We recall Theorem 4.4 below to keep this section self-contained.

Theorem 4.4. Suppose F : R d → R is L -Lipschitz and that ∥ ˜ ∇ F ( · ) -∇ F ( · ) ∥ ≤ α . Then running Algorithm 3 with D = Θ( δϵ 2 L 2 ) , η = Θ( δϵ 3 L 4 ) , outputs a point x out such that E [dist(0 , ∂ δ F ( x out ))] ≤ ϵ + α , with T = O ( ( F ( x 0 ) -inf F ) L 2 δϵ 3 ) calls to ˜ ∇ F ( · ) .

Our analysis is inspired by the reduction from online learning to nonconvex optimization given by [47]. To that end, we start by proving a seemingly unrelated result, asserting that online gradient descent minimizes the regret with respect to inexact evaluations. Recalling standard definitions from online learning, given a sequence of linear losses ℓ m ( · ) = ⟨ g m , ·⟩ , if an algorithm chooses ∆ 1 , . . . , ∆ M we denote the regret with respect to u as

Consider an update rule according to online projected inexact gradient descent:

Lemma C.3 (Inexact Online Gradient Descent) . In the setting above, suppose that ( ˜ g m ) M m =1 are possibly randomized vectors, such that E ∥ ˜ g m -g m ∥ ≤ α and E ∥ ˜ g m ∥ 2 ≤ ˜ G 2 for all m ∈ [ M ] . Then for any ∥ u ∥ ≤ D it holds that

where the expectation is with respect to the (possible) randomness of ( ˜ g m ) M m =1 . In particular, setting η m ≡ D ˜ G √ M yields √

Proof. For any m ∈ [ M ] :

thus

from which we get that

Summing over m ∈ [ M ] , we see that

The simplification for η m ≡ D ˜ G √ M readily follows.

We are now ready to analyze Algorithm 3 in the inexact gradient setting.

Proof of Theorem 4.4. Since Algorithm 3 has x t = x t -1 +∆ t , we have

By summing over t ∈ [ T ] = [ K × M ] , we get for any fixed sequence u 1 , . . . , u K ∈ R d :

where the last inequality follows from Lemma C.3 for ˜ G = L 2 + α 2 , η = D ˜ G √ M , since ∥ ˜ g t -∇ F ( z t ) ∥ ≤ α (deterministically) for all t ∈ [ T ] by assumption. Letting u k := -D ∑ M m =1 ∇ F ( z ( k -1) M + m ) ∥ ∑ M m =1 ∇ F ( z ( k -1) M + m ) ∥ , rearranging and dividing by DT = DKM , we obtain

Finally, note that for all k ∈ [ K ] , m ∈ [ M ] : ∥ ∥ z ( k -1) M + m -x k ∥ ∥ ≤ MD ≤ δ , therefore ∇ F ( z ( k -1) M + m ) ∈ ∂ δ F ( x k ) . Invoking the convexity of the Goldstein subdifferential, we see that

thus it suffices to bound the first two summands on the right-hand side in (C.3) by ϵ in order to finish the proof. This happens as long as F ( x 0 ) -inf F Kδ ≤ ϵ 2 and ˜ G √ M ≤ ϵ 2 . These are equivalent to

C.3 An implementation-friendly algorithm and its analysis

Algorithm 7 Perturbed Inexact GD

1: Input: Inexact gradient oracle ˜ ∇ F : R d → R d , initialization x 0 ∈ R d , spatial parameter δ > 0 , step size η > 0 , iteration budget T ∈ N .

2: for t = 0 , . . . , T -1 do

3: Sample w t ∼ Unif( S d -1 )

4: ˜ g t = ˜ ∇ F ( x t + δ · w t )

6: Output: x out ∼ Unif { x 0 , . . . , x T -1 } .

5: x t +1 = x t -η ˜ g t

Theorem C.4. Suppose F : R d → R is L -Lipschitz, and that ∥ ˜ ∇ F ( · ) - ∇ F ( · ) ∥ ≤ α . Then running Algorithm 7 with η = Θ ( (( F ( x 0 ) -inf F )+ δL ) 1 / 2 δ 1 / 2 T 1 / 2 L 1 / 2 d 1 / 4 ( α + L ) ) outputs a point x out such that E [dist(0 , ∂ δ F ( x out ))] ≤ ϵ + √ αL , with

Proof. Throughout the proof we denote z t = x t + δ · w t . Since F is L -Lipschitz, F δ ( x ) := E w ∼ Unif( S d -1 ) [ F ( x + δ · w )] is L -Lipschitz and O ( L √ d/δ ) -smooth. By smoothness we get

Noting that E [ ∇ F ( z t )] = ∇ F δ ( x t ) and that ∥ ˜ g t ∥ ≤ ∥ ˜ g t -∇ F ( z t ) ∥ + ∥∇ F ( z t ) ∥ ≤ α + L , we see that

which implies

Averaging over t = 0 , . . . , T -1 and noting that F δ ( x 0 ) -inf F δ ≤ ( F ( x 0 ) -inf F ) + δL results in

By Jensen's inequality and the sub-additivity of the square root,

D Reformulation equivalence

Theorem D.1 (Reformulation equivalence) . When λ ∗ matches to an optimal dual solution to the lower level problem y ∗ = arg min y g ( x, y ) s.t. h ( x, y ) ≤ 0 , we show that for each x , the reformulation has the same feasible region of y .

Proof. We first show that lower-level feasibility implies feasibility of the reformulated problem. Let y ∗ , λ ∗ = min y max β ≥ 0 g ( x, y )+ β ⊤ h ( x, y ) be the primal and the dual solution to the lower level problem with parameter x . We can verify that y ∗ satisfies all the constraints in the reformulation problem. The feasibility condition h ( x, y ∗ ) is automatically satisfied. We just need to check:

Therefore, x, y ∗ is a feasible point to the reformulation problem.

We now show the other direction, i.e., that feasibility of the reformulaed problem implies that of the lower-level problem. Given λ ∗ , let us assume y satisfies g ( x, y ) ≤ g ∗ λ ∗ ( x ) and h ( x, y ) ≤ 0 . On the other hand, assume y ∗ , λ ∗ = min y max β ≥ 0 g ( x, y ) + β ⊤ h ( x, y ) be the primal and the dual solution. We can show that:

By the strong convexity of g +( λ ∗ ) ⊤ h , we know that y matches to the unique minimum y ∗ , which implies that y = y ∗ is also a feasible point to the original bilevel problem.

E Active constraints in differentiable optimization

By computing the derivative of the KKT conditions in Section 2.1, we get:

̸

Let I = { i ∈ [ d h ] | h ( x, y ∗ ) i = 0 , λ ∗ i > 0 } be the set of active constraints with positive dual solution, and I 1 = { i | h ( x, y ∗ ) i = 0 } be the set of inactive constraints and I 2 = { i | h ( x, y ∗ ) i = 0 , λ ∗ i = 0 } . We know that ¯ I = I 1 ∪ I 2 . For each i ∈ I 1 , due to complementary slackness, we know that λ ∗ i = 0 .

For i ∈ I 1 in (E.1), we have λ ∗ i ∇ x h ( x, y ∗ ) i + λ ∗ i ∇ y h ( x, y ∗ ) i dy ∗ dx + h ( x, y ∗ ) i dλ ∗ i dx = 0 , which implies h ( x, y ∗ ) i dλ ∗ i dx = 0 because λ ∗ i = 0 . This in turn implies dλ ∗ i dx = 0 because h ( x, y ∗ ) i < 0 . That means the dual variable λ ∗ i = 0 and has zero gradient dλ ∗ i dx = 0 for any index i ∈ I 1 . Therefore, we can remove row i ∈ I 1 in (E.2) and obtain λ ∗ i = 0 and dλ ∗ i dx = 0 .

For i ∈ I 2 , the KKT condition in (E.2) is degenerate. Therefore, dλ ∗ i dx can be arbitrary, i.e., nondifferentiable. As a subgradient choice, we can set dλ ∗ i dx = 0 for such i . This choice will also eliminate

its impact on the KKT condition in (E.1) because dλ ∗ i dx is set to be 0 . By this choice of subgradient, we can also remove row i ∈ I 2 (E.2).

Thus (E.2) can be written as the following set of equations, for h I = [ h i ] i ∈I and λ ∗ I = [ λ ∗ i ] i ∈I :

In (E.1), due to dλ ∗ i dx = 0 for all i ∈ ¯ I , we can remove dλ ∗ i dx ∀ i ∈ ¯ I in (E.1) by:

Combining (E.4) and (E.3), we get:

which can be written in its matrix form:

This concludes the derivation of the derivative of constrained optimization in (5.2).

F Inequality case: bounds on primal solution error and constraint violation

Lemma 5.1. Given any x , the corresponding dual solution λ ∗ ( x ) , primal solution y ∗ ( x ) of the lower optimization problem in Problem 4.1, and y ∗ λ ∗ , α ( x ) as in (5.5), satisfy:

Proof. We first provide the claimed bound on ∥ y ∗ α 1 ,α 2 -y ∗ ( x ) ∥ .

Part 1: Bound on the convergence of y .

Since y ∗ λ ∗ , α minimizes L α ,λ ∗ ( x, y ) , the first-order condition gives us:

Similarly, we can compute the gradient of L α ,λ ∗ ( x, y ) at y ∗ :

where the second step is due to the property of the primal and dual solution: ∇ y g ( x, y ∗ ) + ( λ ∗ ) ⊤ ∇ y h ( x, y ∗ ) = 0 by the stationarity condition in the KKT conditions, and by definition of the active constraints h I where the optimal y ∗ must have h I ( x, y ∗ ) = 0 .

Since, for a sufficiently large α 1 , the penalty function is α 1 µ g -L f ≥ α 1 µ g 2 strongly convex in y , we have:

Therefore, upon rearranging the terms, we obtain the claimed bound:

Part 2: bound on the constraint violation.

When we plug y ∗ into (5.4), we get:

Plugging in y ∗ α ,λ ∗ , we may obtain:

where we used the strong convexity (with respect to y ) of g ( x, y ) + ( λ ∗ ) ⊤ h ( x, y ) and the optimality of y ∗ for g ( x, y ) + ( λ ∗ ) ⊤ h ( x, y ) . By the optimality of y ∗ λ ∗ , α for L α ,λ ∗ , we know that

Therefore, by the Lipschitzness of the function f in terms of y , and the bound ∥ y ∗ -y ∗ λ ∗ , α ∥ ≤ 2 L f α 1 µ g , we know that:

Rearranging terms then gives the claimed bound.

The bound on the constraint violation in Lemma 5.1 is an important step in the following theorem.

G Proof of Lemma 5.2: gradient approximation for inequality constraints

Lemma 5.2. Consider F as in Problem 4.1, L as in (5.4), a fixed x , and y ∗ λ ∗ , α as in (5.5). Then under Assumptions 2.2 and 2.5, we have:

Proof. First, we recall (5.4) here:

Next, recall from Equation D.1, we can express g ∗ ( x ) = g ( x, y ∗ ) + ( λ ∗ ) ⊤ h ( x, y ∗ ) , which we use in the first step below:

According to (5.2) and (E.5), we let

which is invertible by Assumption 2.2(iii) and by the fact that we remove all the inactive constraints. We now bound the terms in (G.1), (G.2), (G.3), (G.4), and (G.5).

Bounding (G.1) and (G.2): (G.1) can be easily bounded by the smoothness of f in terms of x and y , and the bound on ∥ y ∗ -y ∗ ( λ ∗ ) ∗ , α ∥ ≤ O ( α -1 1 ) from Lemma 5.1. Therefore, we know:

Similarly, given Assumption 2.5 by which y ∗ ( x ) is L y -Lipschitz in x , we have the bound ∥ ∥ ∥ dy ∗ dx ∥ ∥ ∥ ≤ L y . Therefore, (G.2) can be bounded by:

Bounding (G.3):

where we use the optimality of y ∗ ( λ ∗ ) ∗ , α from (5.5):

Further, recall that H is non-degenerate by Assumption 2.2, as a result of which, the added term 1 in (G.3) can be modified as follows:

The added term 2 in (G.3) can be expanded to:

Therefore, we can compute the difference between (G.6), (G.8), and (G.9) to bound (G.3), and use the fact that ∇ y g ( x, y ∗ ) + ( λ ∗ ) ⊤ ∇ y h ( x, y ∗ ) = 0 :

The terms in (G.10) and (G.11) can both be bounded by α 1 C g L y ∥ y ∗ λ ∗ , α -y ∗ ∥ 2 and α 1 RC h L y ∥ y ∗ λ ∗ , α -y ∗ ∥ 2 by the smoothness of g and h ⊤ λ ∗ . Further, plugging in ∥ y ∗ -y ∗ λ ∗ , α ∥ ≤ O ( α -1 1 ) from Lemma 5.1 bounds both these terms by O ( α -1 1 ) .

To bound the term in (G.12), we use:

Therefore, we have:

which upon scaling by α 1 gives us the following bound on the term in (G.12):

The term in (G.13) can be bounded by:

Bounding (G.4): This can be easily bounded by the smoothness of g and h , and the bound on the dual solution ∥ λ ∗ ∥ ≤ R . Thus (G.4) can be bounded by R · O ( α -1 1 ) = O ( α -1 1 ) .

Bounding (G.5): By the same argument in (G.14), we get:

Combining all upper bounds gives the claimed bound.

H Proof of the main result (Theorem 5.3): convergence and computation cost

Theorem 5.3. Given any accuracy parameter α > 0 , Algorithm 4 outputs ˜ ∇ x F ( x ) such that ∥ ˜ ∇ F ( x ) -∇ F ( x ) ∥ ≤ α within ˜ O ( α -1 ) gradient oracle evaluations.

Proof. First, given the bound in Lemma 5.2, we choose α 1 = α -2 and α 2 = α -4 to ensure the inexactness of the gradient oracle is bounded by α . In the later analysis, we will still use α 1 and α 2 in the penalty function for clarity.

Now we estimate the computation cost of the inexact gradient oracle:

Lower-level problem. Given the oracle access to the optimal dual solution λ ∗ ( x ) , we can recover the primal solution y ∗ ( x ) efficiently (e.g, by [45]). Therefore, we can use the primal and dual solutions to construct the penalty function L λ ∗ ,α ( x, y ) in (5.4).

Penalty function minimization problem. The second main optimization problem is the penalty minimization problem in Line 4 of Algorithm 4. Recall from (5.4) that

where we use the approximate dual solution λ as opposed to the optimal dual solution λ ∗ .

Given (H.1), we solve the penalty minimization problem:

The penalty minimization is a unconstrained strongly convex optimization problem, which is known to have linear convergence rate. We further analyze its convexity and smoothness below to precisely estimate the computation cost:

The strong convexity of L λ, α ( x, y ) is lower bounded by α 1 µ g 2 = O ( α 1 ) .

The smoothness of L λ, α ( x, y ) is dominated by the smoothness of α 2 ∥ h I ( x, y ) ∥ 2 since α 2 ≫ α 1 . By Lemma 5.2, we know that the optimal solution must lie in an open ball B ( y ∗ , O (1 /α 1 )) with center y ∗ (inner optimization primal solution) and a radius of the order of O ( 1 α 1 ) . This implies that we just need to search over a bounded feasible set of y , which we can bound ∥∇ y h ( x, y ) ∥ ≤ L h and h ( x, y ) ≤ H within the bounded region y ∈ B ( y ∗ , O (1 /α 1 )) . We can show that h 2 is smooth (gradient Lipschitz) within the bounded region by the following:

which also implies h 2 I is also smooth (gradient Lipschitz). Therefore, αh 2 I is ( HC h + L 2 h ) α 2 = O ( α 2 ) smooth.

Choosing α 1 = 1 α 2 and α 2 = 1 α 4 , the condition number of L α ,λ ( x, y ) becomes κ = O ( α 2 /α 1 ) = O ( 1 α 2 ) . Therefore, by the linear convergence of gradient descent in strongly convex smooth optimization, the number of iterations needed to get to α accuracy is O ( √ α -2 × log( 1 α )) = O ( 1 α log( 1 α )) . Therefore, we can get a near optimal solution y ′ λ,α with inexactness α in O ( 1 α ) oracle calls.

Computation cost and results. Overall, for the inner optimization, we can invoke the efficient optimal dual solution oracle to get the optimal dual solution λ ∗ ( x ) and recover the optimal primal solution y ∗ ( x ) from there. For the penalty minimization, we need O ( 1 α ) oracle calls to solve an unconstrained strongly convex smooth optimization problem to get to α accuracy. In conclusion, combining everything in Appendix H, we run O ( 1 α ) oracle calls to obtain an α accurate gradient oracle to approximate the hyperobjective gradient ∇ x F ( x ) . This concludes the proof of Theorem 5.3.

Remark H.1. The following analysis quantifies how the error in the optimal dual solution propagates to the inexact gradient estimate. This is not needed if such a dual solution oracle exists. But in practice, the oracle may come with some error, for which we bound the error.

Bounding the error propagation in error in dual solution and the penalty minimization. First, if we do not get an exact optimal dual solution, the error in the dual solution λ with ∥ λ -λ ∗ ∥ ≤ α will slightly impact the analysis in Lemma 5.2. Specifically, in Appendix G, the approximate λ will impact the inexact gradient ∇ x L λ,α ( x, y ′ λ,α ) computation and the analysis in (G.4) and (G.7). In (G.4), to change λ to λ ∗ , we get an additional error:

where the last inequality is due to ∥ ∥ ∥ y ′ -y ′ λ,α ∥ ∥ ∥ ≤ O ( α -1 1 ) that is based on a similar analysis in Lemma 5.1 with a near-optimal y ′ λ,α under α 2 = α 1 accuracy.

Therefore, the error incurred by inexact λ in (G.4) is at most O ( α ) , which is of the same rate as the current gradient inexactness O ( α ) .

In (G.7), the optimality holds approximately for the approximate λ . Therefore, by the near optimality of y ′ λ, α (strongly convex optimization), we know that the following gradient is also α -close to 0 , i.e.,

whose inexactness matches the inexactness of the gradient oracle α , and thus we do not incur additional order of inexactness here.

Moreover, there is an additional error because we need λ ∗ as opposed to a near-optimal λ to make the analysis in Appendix G work. The error between using λ and λ ∗ in (H.3) can be bounded by:

where we use the local Lipschitzness of the function h in an open ball near y ∗ . Therefore, the additional error is also O ( α ) , which matches the inexactness of the inexact gradient oracle.

Therefore, we conclude that in order to bound the inexactness of the gradient oracle, we just need an efficient inexact dual solution with α accuracy.

I Practical oracle to optimal (approximate) dual solution

Here we discuss how practical the assumption on the oracle access to the optimal dual solution is.

For linear inequality constraint h ( x, y ) = Ax -By -b , the LL problem is a constrained strongly convex smooth optimization problem. To show that we can compute an approximate solution to the optimal dual solution for linear inequality constraints, we apply the result from [93]:

Corollary I.1 (Application of Corollary 3.1 in [93]) . When h ( x, y ) = Ax -By + b is linear in y , the primal and dual solutions can be written as:

where g is strongly convex in y and B is of full rank by Assumption 2.2. According to Corollary 3.1 from [93], the primal-dual gradient method guarantees a linear convergence. More precisely, in t = O (log 1 α ) iterations, we get:

Given Corollary I.1, we can efficiently approximate the primal and dual solutions up to high accuracy with O (log 1 α ) oracle calls when the inequality constraints are linear. This gives us an efficient approximate oracle access to the dual solution.

Remark I.2. Under the assumption of an optimal dual solution oracle, all the analyses mentioned in Section 5 hold for the general convex inequality constraints. However, the main technical challenge is that the dual solution oracle for general convex inequality cannot be guaranteed in practice. In fact, to the best of our knowledge, there is no iterate convergence in the dual solution λ for general convex inequality constraints. Most of the literature in strongly-convex-concave saddle point convergence only guarantees dual solution convergence in terms of its duality gap or some other merit functions. We are not aware of any successful bound on the dual solution iterate convergence, which is an important research question to answer by itself. This is the main technical bottleneck for general convex inequality constraints as well.

Remark I.3. On the other hand, we need the dual solution iterate convergence with rate O (1 /α ) to ensure the error to be bounded. But this is not a necessary condition. To ensure a bound on the error propagation, we just need to bound some forms of merit functions ((H.2) and (H.4)) of the dual solutions, which we believe that this is much more tractable than the actual iterate dual solution convergence. We leave this as a future direction and this will generalize the analysis from linear inequality constraints to general convex inequality constraints.

J The role of λ ∗ ( x ) in the derivative of Equation (5.4)

Notice that Equation (5.4), we treat the dual solution λ ∗ ( x ) as a constant to define the penalty function derivative. Yet, the dual solution λ ∗ ( x ) is in fact also a function of x . Therefore, in theory, we should also compute its derivative with respect to x .

However, notice that the following:

The later term in Equation (J.1) can be divided into two cases:

For active constraint i ∈ I with h ( x, y ∗ ) = 0 , we know that y ∗ λ,α is close to y ∗ by Lemma 5.1. Therefore, the derivative ∥ ∥ ∥ dλ ( x ) dx ⊤ h ( x, y ∗ λ,α ) ∥ ∥ ∥ ≤ L h L λ α 1 = O ( α 1 ) = O ( α 2 ) by the local smoothness of h near y ∗ and the Lipschitzness assumption of λ ∗ in Assumption 2.5.

For inactive constraint i ∈ ¯ I and λ ∗ i > 0 , we can solve the KKT conditions and get dλ ( x ) dx = 0 . Therefore, the second term becomes 0 .

For inactive constraint i ∈ ¯ I and λ ∗ i = 0 , the KKT system degenerates and we need to use subgradient. By solving the KKT system, we find that dλ ( x ) dx = 0 is a valid subgradient. Therefore, by choosing this subgradient, the second term also vanishes.

Therefore, we do not need to compute the derivative of λ ∗ as the terms involved its derivative is negligible compared to other major terms.

K Experimental setup

All experiments were run on a computing cluster with Dual Intel Xeon Gold 6226 CPUs @ 2.7 GHz and DDR4-2933 MHz DRAM. No GPU was used, and we used 1 core with 8GB RAM per instance of the experiment. The cutoff time for running the algorithms is set to be 6 hours. All experiments were run and averaged over 10 different random seeds. All parameters in the constrained bilevel optimization in Section 6, including the objective parameters and the constrain parameters, are randomly generated from a normal distribution with 0 -mean and standard deviation 1 .

For our fully first-order algorithm, we implement Algorithm 3, where the inexact gradient oracle subroutine is provided by implementing Algorithm 4. All algorithms are implemented in PyTorch [94] to compute gradients, and using Cvxpy [95] to solve the LL problem and the penalty minimization problem. We implement our fully first-order method based on the solutions returned by Cvxpy with certain accuracy requirement, and use PyTorch to compute the inexact gradient discussed in Section 5. We implement the non-fully first-order method using the CvxpyLayer [18], which is a Cvxpy compatible library that can differentiate through the LL convex optimization problem.

L Additional Experimental Results

0.0

0.2

0.4

0.6

0.8

1.0

0

100 200 300 400 500 600 700 800 900

Iteration

0.0

0.5

1.0

1.5

2.0

Optimality gap

cvxpylayer

F2CBA (Algorithm 3)

0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

850

900

950

Iteration

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Optimality gap

F2CBA

2 =0.0001

F2CBA

2 =0.001

F2CBA

2 =0.01

F2CBA

2 =0.1

F2CBA

2 =1.0

100

200

300

400

500

600

700

800

900 1000

Inner level dimension

0

10

20

30

Time (s)

cvxpylayer

F2CBA

Figure 2: We run Algorithm 3 using Algorithm 4 on the bilevel optimization in the toy example in Problem L.1 with varying upper-level variable dimensions d x , a fixed lower-level variable dimension d y = 200 , and the number of constraints n const = d y / 5 = 40 , and accuracy α = 0 . 1 . Figure 2a, Figure 2b, Figure 2c vary # of iterations, gradient exactness α , and d y , respectively, to compare the performance under different settings.

We generate instances of the following constrained bilevel optimization problem:

where h i ( x, y ) = x ⊤ A i y -b ⊤ i x ∀ i ∈ [ d h ] is a d h -dim bilinear constraint, where the constraint bilinear matrix A i ∈ R d x × d y , b i ∈ R d x for all i ∈ [ d h ] are randomly generated from normal

distributions. The bilinear (nonlinear) constraint of the lower-level problem is the major difference compared to the experiment in Section 6. We are interested in whether our algorithms work beyond the linear constraints where our theory guarantees.

The rest of the parameters are the same as in Section 6. The PSD matrix Q ∈ R d y × d y , c ∈ R d y , P ∈ R d x × d y . We compare our Algorithm 3 with a non-fully first-order method implemented using cvxpyLayer [18]. Both algorithms use Adam [90] to control the learning rate in gradient descent. All the experiments are averaged over ten random seeds.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide algorithms and corresponding theoretical guarantees for all our claims in the abstract. We provide experiments (and relevant code) as claimed.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is discussed in Section 7

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions, theorem statements, and proof sketches are included in the main paper. The full proofs are included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide our full code in the supplemental material, and it can be used to reproduce the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is included in the supplemental material and can be used to reproduce the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide this information in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide this in Section 6

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide this information in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects is involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies fully first-order methods for bilevel optimization with strongly convex lower-level objectives and linear lower-level constraints, including both inequality constraints and equality constraints. For linear inequality constraints, the paper used a penalty methods to construct hypergradient estimators and demonstrate the approximation error. Then combined with a gradient method for nonsmooth nonconvex optimization to obtain a Goldstein stationarity point. While for linear inequality constraints, the paper proposed to use zeroth-order approximation of the hyper-gradient and demonstrate complexity.\n\nSTRENGTHS:\n1. The paper considered both linear inequality and equality constraints. \n2. The hypergradient construction in the linear inequality constraint setting is interesting and the paper provides approximation guarantees. It would be useful for follow-up papers. \n3. Obviously, the complexity bounds are not yet optimal. But the paper has contributed several interesting directions that worth further exploration. \n3. Algorithm 5 uses zeroth-order oracle is very smart as in many bilevel problems, the dimension of the upper-level is small such as hyperparameter optimization. \n3. The finite-difference gradient proxy for the linear equality constrained case is also interesting.\n\nWEAKNESSES:\n1. How to solve linear inequality constrained case with simple gradient methods to achieve optimal complexity bounds still remain unknown. The current algorithm is a bit complicated and hard to tune in practice. \n2. It is unclear if the complexity bounds are optimal for both linearly inequality and equality constrained settings. Following the conventional notation for stationarity such that $||\\nabla F(x)||\\leq \\epsilon^2$, the complexity bound for Alg 1 is $O(\\epsilon^{-8})$, for Alg 2 is $O(\\epsilon^{-10})$ if further taking into account the complexity to get $tilde \\nabla F$, and for Alg 3 is $O(\\epsilon^{-4})$. Note that the paper consider deterministic bilevel optimization, it implies that there are a lot of room for improvements. The paper should mention comparison to unconstrained lower-level bilevel optimization so that readers know the results can be improved. Despite the complexity bounds, I still believe that this paper makes good contribution to the community of bilevel optimization.\n3. The methods proposed in the paper are not aligning with each other well. Despite the constrained lower-level bilevel optimization setting, the paper lacks a coherent narrative in terms of algorithmic design.\n\nQUESTIONS:\n1. In the Alg 3 perturbed GD, what is the intuition to use perturbed GD instead of just inexact GD $\\tilde \\nabla F(x_t)$. In other words, why one needs to perturb using samples from sphere distribution?\n2. In Sec 3, although the problem is nonsmooth and nonconvex, given Thm 3.3, I would assume that the $\\nabla F(x)$ exists but it is not Lipschitz continuous? \n3. Please be specific about Lipschitz, i.e., does it mean Lipschitz continuous or Lipschitz smooth.\n4. What is the reason to consider zeroth-order methods in the linear equality constrained setting? Is it possible to derive fully first-order gradient estimator for the linear equality constrained setting? If possible, why not do it? \n5. In Sec 3.1, assuming knowing $\\lambda^*$ means that one also knows $y^*$ and $g^*$. \n6. The definition of complexity is not very clear. In line 229, it says solving a single smooth and strongly convex optimization problems amounts to $\\tilde O(1)$ oracle calls. But line 226 implies that one of such oracle call requires $\\epsilon^{-1}$ gradient evaluations. Then in line 322, the complexity seems to become $O(log(1/\\epsilon))$. Please specify how expensive to obtain $y^*$ and $\\lambda^*$ in an $\\epsilon$ error and clean oracle calls and complexity bounds in a consistent manner. \n7. What is the motivation of (5.3) the perturbed lower-level problem? \n8. Alg 1 requires computing $y^*_{\\lambda^*,\\alpha}$, this requires either $f$ to be convex in $y$ or requires $\\alpha$ to be large enough to ensure $\\mathcal{L}$ is convex in $y$. Please specify explicitly the requirements. Is $\\mathcal{L}$ strongly convex in $y$?\n\nI am happy to raise the score if these issues are well-addressed.\n\nLIMITATIONS:\nThe paper includes several settings. Yet why adapting a specific strategy to solve each setting is not sufficiently motivated. Such as why one needs to use Alg 2 and Alg 3 instead of directly using the hypergradient estimator with gradient descent. Why one needs to use zeroth-order oracle for the linearly equality constrained setting instead of first-order oracle?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works deals with constrained bilevel optimization problems where the lower-level has linear inequality or equality constraints. A set of algorithms is developed that does not require access to the Hessian, but only to zeroth and first-order information. In the case of inequality constraints, convergence is established to an $(\\delta,\\epsilon)$-Goldstein stationary point, due to the non-smoothness of the problem, using $O(\\delta^{-1}\\epsilon^{-4})$ or $O(d\\delta^{-1}\\epsilon^{-3})$ oracle calls (d: dimension of the upper-level).For equality-constrained problems, the convergence rate is nearly optimal with a rate $O(\\epsilon^{-2})$.\n\nSTRENGTHS:\n* This work makes progress in the topic of constrained (in the lower-level) bilevel problems, which is a challenging class of problems, much more difficult than unconstrained problems.\n* The development of first-order algorithms for bilevel problems with linear constraints in the lower-level. On the contrary, typically implicit gradient methods for (unconstrained or constrained) bilevel problems require access to the Hessian. In addition, for the case of inequality constraints there are no assumptions involving second-order derivatives.\n* The design of algorithms that deal with the non-differentiability of the hyperobjective F.\n* Some of the proposed algorithms (e.g., Algorithm 3) appears to be simple and easy to implement.\n\nWEAKNESSES:\n* A major weakness of this work is the small number of experiments. There is only a single example bilevel problem over which experiments are performed. In addition, there are no experiments on real applications (e.g., in machine learning).\n* In the experiments section the proposed method is compared only with a single baseline, which does not correspond to any published bilevel method. Why aren’t there any comparisons with other bilevel algorithms that deal with constrained bilevel problems? There are both value function and implicit gradient methods that can, at least in theory, deal with the example problem used here, e.g. [37,38]. \n* The algorithms require access to exact solutions of certain problems, such as $y^{\\ast}(x)$ of the lower-level or $y_{\\lambda^{\\ast},\\alpha}^{\\ast}(x)$ of problem (3.6). This is not the case in practice.\n\nQUESTIONS:\n* In the gradient of the penalty function (3.5.), there is the term $\\nabla_x g^{\\ast}(x)$. How do you compute this gradient given that $g^{\\ast}(x)$ is a min function? The authors should explain this. \n* The authors should explain in more detail Algorithm 2. It is not clear what the utility of each step is.\n* The authors derive the equations for the gradient of $F$ and $y^{\\ast}(x)$ (eq. 3.3). However, as they note in section 1.1, the hyperfunction $F$ is non-smooth. Then, how can we derive a formula for the gradient of $y^{\\ast}(x)$? How do we determine if a given point is differentiable?\n* Perhaps the least common assumption in literature is Assumption 2.3. The authors should justify whether this assumption is easy to be satisfied. For instance, is it typically the case that the dual solution is bounded? How about the Lipschitz property of $y^{\\ast}(x)$? Are there other relevant (bilevel) works that use the same or similar assumption?\n* How does the cvxpyLayer baseline work? This is not explained clearly in the text, besides a minor reference in the Appendix. As this is the only baseline currently used, I believe that some more details are required.\n\nLIMITATIONS:\n* See weaknesses above.\n* The authors discuss limitations of their work in section 7.\n* No negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides algorithms for bilevel optimization with linear equality and inequality constraints. The main contribution is that the algorithms are fully first order and do not require Hessian computations. This is achieved by reformulating the linearly constrained bilevel optimization problem using the penalty method and assuming access to the upper-level variable $x$ and approximate dual optimal solution. In this setting an inexact gradient oracle for the problem is constructed. The authors provide stationarity guarantees for their algorithms under certain standard assumptions. They also describe how to use their inexact gradient oracle for non-convex non smooth optimization. The algorithms and theoretical results are backed by small set of proof-of-concept experiments.\n\nSTRENGTHS:\n1) Typically, hessian inverse computation is quite expensive and hence coming up with a fully first order method with guarantees for an optimization problem has both good theoretical and practical significance. The paper will be of interest to the community.\n2) The paper is written very nicely. There is a lot of clarity regarding the related works, notations, main contributions and techniques. Proof sketches are also provided for some results in the main paper. Though the paper is heavy on technical material, the organization makes it somewhat easier for the reader to follow the arguments. I am not exactly from the same research area but could follow most of the paper. \n3) I could not check all the proofs in details, but the main claims of the paper appear correct.\n4) Nonconvex non smooth optimization is encountered in many ML problems these days. The applicability of the inexact gradient oracle to these problems with guarantees may be of practical significance.\n\nWEAKNESSES:\n1) There are no experiments highlighting the effectiveness of Algorithm 2.\n2) The experiments given here are also only of a proof -of -concept nature and are not comprehensive. Some more comparisons with existing algorithms (based on time) and ablation studies may provide more insights regarding the practical applicability of the methods.\n\nQUESTIONS:\nThere is a lot of literature available on approximating Hessian computations faster for 2nd order methods (for example using iterative sketching etc.) though not necessarily for bilevel optimization. I would appreciate it if you can comment on such methods when compared to yours in terms of practicality.\n\nLIMITATIONS:\nYes, the limitations have been clarified in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper studies the linearly constrained bilevel optimization problem and provides a fully first-order method with solid theoretical analysis. To approximate hypergradient, the penalty method seems novel to me and it is applied in two settings where the LL problem is linear inequality or equality constraints.

STRENGTHS:
1. The presentation is pretty good and I feel easy to follow.
2. Theoretical analysis is solid and rigorous.
3. Code is provided.

WEAKNESSES:
1. The experiment does not seem sufficient.
2. The checklist should be behind the appendix as I memorize.

QUESTIONS:
1. Could the authors explain the design of Algorithm 2? Why is there a clip? 
2. Why do authors not include the experiment of Algorithm 4? 
3. Could authors compare the method in [a] though it studies the unconstrained bilevel optimization problem?
[a] Yang Y, Xiao P, Ji K. Achieving ${O}(\epsilon^{-1.5}) $ Complexity in Hessian/Jacobian-free Stochastic Bilevel Optimization.

LIMITATIONS:
Please check weaknesses and problems.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12