Title: Penalty-based Methods for Simple Bilevel Optimization under Hölderian Error Bounds
Venue: NeurIPS 2024
Accepted: yes
Keywords: Simple Bilevel Optimization

Hölderian Error Bound

Penalization

Complexity
Abstract: This paper investigates simple bilevel optimization problems where we minimize a convex upper-level objective over the optimal solution set of a convex lower-level objective. Existing methods for such problems either only guarantee asymptotic convergence, have slow sublinear rates, or require strong assumptions. To address these challenges, we propose a penalization framework that delineates the relationship between approximate solutions of the original problem and its reformulated counterparts. This framework accommodates varying assumptions regarding smoothness and convexity, enabling the application of specific methods with different complexity results. 
Specifically, when both upper- and lower-level objectives are composite convex functions, under an $\alpha$-Hölderian error bound condition and certain mild assumptions, our algorithm attains an $(\epsilon,\epsilon^{\beta})$-optimal solution of the original problem for any $\beta> 0$ within $\mathcal{O}\left(\sqrt{{1}/{\epsilon^{\max\\{\alpha,\beta\\}}}}\right)$ iterations. The result can be improved further if the smooth part of the upper-level objective is strongly convex. We also establish complexity results when the upper- and lower-level objectives are general nonsmooth functions. Numerical experiments demonstrate the effectiveness of our algorithms.

Penalty-based Methods for Simple Bilevel Optimization under Hölderian Error Bounds

Abstract

This paper investigates simple bilevel optimization problems where we minimize an upper-level objective over the optimal solution set of a convex lower-level objective. Existing methods for such problems either only guarantee asymptotic convergence, have slow sublinear rates, or require strong assumptions. To address these challenges, we propose a penalization framework that delineates the relationship between approximate solutions of the original problem and its reformulated counterparts. This framework accommodates varying assumptions regarding smoothness and convexity, enabling the application of specific methods with different complexity results. Specifically, when both upper- and lower-level objectives are composite convex functions, under an α -Hölderian error bound condition and certain mild assumptions, our algorithm attains an ( ϵ, ϵ β ) -optimal solution of the original problem for any β > 0 within O ( √ 1 /ϵ max { α,β } ) iterations. The result can be improved further if the smooth part of the upper-level objective is strongly convex. We also establish complexity results when the upper- and lowerlevel objectives are general nonsmooth functions. Numerical experiments demonstrate the effectiveness of our algorithms.

1 Introduction

Bilevel optimization involves embedding one optimization problem within another, creating a hierarchical structure where the upper-level problem's feasible set is influenced by the lower-level problem. This framework frequently occurs in various real-world scenarios, such as meta-learning [Bertinetto et al., 2018, Rajeswaran et al., 2019], hyper-parameter optimization [Chen et al., 2024, Franceschi et al., 2018, Shaban et al., 2019], reinforcement learning [Mingyi et al., 2020] and adversarial learning [Bishop et al., 2020, Wang et al., 2021, 2022]. In this paper, we concentrate on a subset of bilevel optimization known as simple bilevel optimization (SBO), which has garnered significant interest in the machine learning community due to its relevance in dictionary learning [Beck and Sabach, 2014, Jiang et al., 2023], lexicographic optimization [Kissel et al., 2020, Gong et al., 2021], lifelong learning [Malitsky, 2017, Jiang et al., 2023]; see more details in Appendix A.

Rujun Jiang †

SBO aims to find an optimal solution that minimizes the upper-level objective over the solution set of the lower-level problem. In other words, we are interested in solving the following problem:

Here F, G : R n → R ∪ {∞} are proper, convex, and lower semi-continuous functions. We also assume that the optimal solution set of the lower-level problem, denoted as X opt, is nonempty. Moreover, since G is convex and lower semi-continuous, it holds that X opt is closed and convex [Bertsekas et al., 2003, Proposition 1.2.2 and Page 49].

In this paper, we first reformulate problem (P) into the constrained form:

where G ∗ represents the optimal value of the unconstrained lower-level problem.

Based on this reformulation, we consider the following penalization of (P Val ),

where p ( x ) := G ( x ) -G ∗ is the so-called residual function and γ > 0 is the penalized parameter. Obviously, we have p ( x ) ≥ 0 , and p ( x ) = 0 if and only if x ∈ X opt .

Denote F ∗ and G ∗ as the optimal values of problem (P) and the lower-level problem min x ∈ R n G ( x ) , respectively. We aim to find an ( ϵ F , ϵ G ) -optimal solution ˜ x ∗ of problem (P), which satisfies

Moreover, a point ˜ x ∗ γ is said to be an ϵ -optimal solution of problem (P γ ) if

where Φ ∗ γ is the optimal value of problem (P γ ).

1.1 Related work

Various approaches have been developed to solve problem (P) [Cabot, 2005, Solodov, 2007, Sabach and Shtern, 2017, Dutta and Pandit, 2020, Gong et al., 2021]. Among those, one category that is the most related to penalization formulation (P γ ) is the regularization method, which integrates the upper- and lower-level objectives through Tikhonov regularization [Tikhonov and Arsenin, 1977]

where σ is the so-called regularization parameter. When F is strongly convex and its domain is compact, Amini and Yousefian [2019] extended the IR-PG method from Solodov [2007], which achieved a asymptotic convergence rate for the upper-level problem and a convergence rate of O ( 1 /K 0 . 5 -b ) for the lower-level problem, where b ∈ (0 , 0 . 5) . Malitsky [2017] studied a version of Tseng's accelerated gradient method and showed a convergence rate of O (1 /K ) for the lower-level problem, while the convergence rate for the upper-level objective is not explicitly provided. Kaushik and Yousefian [2021] proposed an iteratively regularized gradient (a-IRG) method which obtains a complexity of O ( 1 /K 0 . 5 -b ) and O ( 1 /K b ) for the upper- and lower-level objective, respectively, where b ∈ (0 , 0 . 5) . Inspired by this research, and under a quasi-Lipschitz assumption for F , Merchav and Sabach [2023] introduced a bi-subgradient (Bi-SG) method. This method demonstrates convergence rates of O (1 /K b ) and O (1 /K 1 -b ) for the lower- and upper-level objectives, respectively, where b ∈ (0 . 5 , 1) . In their framework, the convergence rate of the upper-level objective can be improved to be linear when F is strongly convex. Recently, under the weak-sharp minima assumption of the lower-level problem, Samadi et al. [2023] proposed a regularized accelerated proximal method (RAPM), showing a convergence rate of O (1 /K 2 ) for both upper- and lower-level objectives. When the domain is compact and F, G are both smooth, Giang-Tran et al. [2023] proposed an iteratively regularized conditional gradient (IR-CG) method, which ensures convergence rates of O (1 /K p ) and O (1 /K 1 -p ) for upper- and lower-level objectives, respectively, where p ∈ (0 , 1) .

Despite the abundance of existing methodologies yielding non-asymptotic convergence outcomes, their efficacy is frequently contingent upon additional assumptions. Denote L f 1 and L g 1 as the Lipschitz constants for the gradients of the smooth components in the upper- and lower-level objectives,

respectively. Specifically, when F is strongly convex and G is smooth, Beck and Sabach [2014] presented the Minimal Norm Gradient (MNG) method and provided the asymptotic convergence to the optimal solution set and a convergence rate of O ( L 2 g 1 /ϵ 2 ) for the lower-level problem. When F is assumed to be smooth, Jiang et al. [2023] introduced a conditional gradient-based bilevel optimization (CG-BiO) method, which invokes at most O (max { L f 1 /ϵ F , L g 1 /ϵ G } ) of linear optimization oracles to achieve an ( ϵ F , ϵ G ) -optimal solution. Shen et al. [2023] combined an online framework with the mirror descent algorithm and established a convergence rate of O (1 /ϵ 3 ) for both upper- and lower-level objectives, assuming a compact domain and boundedness of the functions and gradients at both levels. Furthermore, they showed that the convergence rate can be improved to O (1 /ϵ 2 ) under additional structural assumptions. For a concise overview of overall methodologies, including their assumptions and convergence outcomes, refer to Table 1 in Appendix B.

For general bilevel optimization problems, there have been recent results on convergent guarantees [Shen and Chen, 2023, Sow et al., 2022, Chen et al., 2023, Huang, 2023]. Among those, the one that is the most related to ours is [Shen and Chen, 2023]. It investigates the case when the upper-level objective is nonconvex and gives convergence results under additional assumptions [Shen and Chen, 2023, Theorem 3 and 4]. However, as the general bilevel optimization problem is nonconvex, the algorithms in the literature often converge to weak stationary points, while our method for SBO converges to global optimal solution.

1.2 Our approach

Our approach is straightforward. Firstly, we introduce a penalization framework delineating the connection between approximate solutions of problems (P) and (P γ ). This framework enables the attainment of an ( ϵ F , ϵ G ) -optimal solution by solving problem (P γ ) approximately. Subsequently, our focus shifts solely to resolving the unconstrained problem (P γ ). Depending on varying assumptions regarding smoothness and convexity, we can employ different methods such as the accelerated proximal gradient (APG) methods [Beck and Teboulle, 2009, Nesterov, 2013, Lin and Xiao, 2014] to solve problem (P γ ). We summarize our main contributions as follows.

We propose a framework that explicitly examines the relationship between an ϵ -optimal solution of penalty formulation (P γ ) and an ( ϵ F , ϵ G ) -optimal solution of problem (P). We also provide a lower bound for the metric F ( x ) -F ∗ .

We present adaptive versions of PB-APG and PB-APG-sc with warm-start, which dynamically adjust the penalty parameters, and solve the associated penalized problem with adaptive accuracy. The adaptive ones have similar complexity results as their primal counterparts but can achieve superior performance in some experiments.

When F and G are both composite convex functions, we provide a penalty-based APG algorithm that attains an ( ϵ, ϵ β ) -optimal solution of problem (P) within O ( √ 1 /ϵ max { α,β } ) iterations. If the upper-level objective is strongly convex, the complexity can be improved to O ( √ 1 /ϵ max { α -1 ,β -1 } log 1 ϵ ) . We also apply our method for the scenario where both the upper- and lower-level objectives are generalized nonsmooth convex functions.

Utilizing the penalization method to address the original SBO problem is a novel approach. While Tikhonov regularization may seem similar to our framework, its principles differ. Implementing Tikhonov regularization necessitates the "slow condition" ( lim k →∞ σ k = 0 , ∑ ∞ k =0 σ k = + ∞ ), which requires iterative solutions for each iteration. In contrast, our method simply involves solving a single optimization problem (P γ ) for a given γ . Furthermore, we establish a relationship between the approximate solutions of the original bilevel problem and those of the reformulated single-level problem (P γ ) for a specific γ . This is the first theoretical result connecting the original bilevel problem to the penalization problem, accompanied by an optimal non-asymptotic complexity result.

2 The penalization framework

We begin by outlining specific assumptions for F and G , as detailed below.

Assumption 2.1. The set S := x ∈ X opt ∂F ( x ) is bounded with a diameter l F := max ξ ∈ S ∥ ξ ∥ .

Note that the type of subdifferential ∂F used here is the most general form for a convex function, as detailed in [Bertsekas et al., 2003, Section 4.2]. When the upper-level objective F is non-convex,

∪

we replace the assumption with the condition that the upper-level objective is Lipschitz continuous (cf. Theorems 2.7 and 2.8).

Assumption 2.2 (Hölderian error bound) . The function p ( x ) := G ( x ) -G ∗ satisfies the Hölderian error bound with exponent α ≥ 1 and ρ > 0 . Namely,

where dist ( x , X opt ) := inf y ∈ X opt ∥ x -y ∥ .

We remark that Hölderian error bounds are satisfied by many practical problems and widely used in optimization literature [Pang, 1997, Bolte et al., 2017, Zhou and So, 2017, Roulet and d'Aspremont, 2020, Jiang and Li, 2022]. There are two notable special cases: (i) when α = 1 , we often refer to X opt as a set of weak sharp minima of G [Burke and Ferris, 1993, Studniarski and Ward, 1999, Burke and Deng, 2005, Samadi et al., 2023]; (ii) when α = 2 , Assumption 2.2 is known as the quadratic growth condition [Drusvyatskiy and Lewis, 2018a]. Additional examples of functions exhibiting Hölderian error bound, along with their corresponding parameters, are presented in Appendix C.

We are now ready to establish the connection between approximate solutions of problems (P) and (P γ ). The subsequent two lemmas build upon the work of Shen and Chen [2023] for (general) bilevel optimization. Compared with their work, we generalize the exponent α from 2 to α ≥ 1 , providing a more general result. Furthermore, we also derive a lower bound for the penalized parameter for all α ≥ 1 and present a theoretical framework for these scenarios.

Lemma2.3. Suppose that Assumptions 2.1 and 2.2 hold with α > 1 . Then, for any ϵ > 0 , an optimal solution of problem (P) is an ϵ -optimal solution of problem (P γ ) when γ ≥ ρl α F ( α -1) α -1 α -α ϵ 1 -α .

Lemma 2.3 establishes the relationship between an optimal solution of problem (P) and an ϵ -optimal solution of problem (P γ ) when α > 1 . It also provides a lower bound for γ , which plays a pivotal role in the complexity results. The proofs of this paper are deferred to Appendix E.

The lemma presented below yields a more favorable outcome when α = 1 , which is referred to as exact penalization. Notably, this specific result is not discussed in Shen and Chen [2023].

Lemma 2.4. Suppose that Assumptions 2.1 and 2.2 hold with α = 1 . Then an optimal solution of problem (P) is also an optimal solution of problem (P γ ) if γ ≥ ρl F , and vice versa if γ > ρl F . In this case, we say that there is an exact penalization between problems (P) and (P γ ) .

For simplicity, we define

Based on Lemmas 2.3 and 2.4, we give an overall relationship of approximate solutions between problems (P γ ) and (P).

Theorem 2.5. Suppose that Assumptions 2.1 and 2.2 hold. For any given ϵ > 0 and β > 0 , let

with γ ∗ defined in (2) . If ˜ x ∗ γ is an ϵ -optimal solution of problem (P γ ) , then ˜ x ∗ γ is an ( ϵ, l -β F ϵ β ) -optimal solution of problem (P) .

Particularly, we are also able to establish a lower bound for F (˜ x ∗ γ ) -F ∗ under the same conditions outlined in Theorem 2.5.

Theorem 2.6. Suppose that the conditions in Theorem 2.5 hold. Then, ˜ x ∗ γ satisfies the following suboptimality lower bound,

By setting β = α , we obtain F (˜ x ∗ γ ) -F ∗ ≥ -ρ 1 α ϵ . which along with Theorem 2.5 gives

We emphasize that the lower bound established in Theorem 2.6 is an intrinsic property of problem (P) under Assumptions 2.1 and 2.2. This property is independent of the algorithms we present.

2.1 The upper-level function is non-convex

Note that the upper-level objective F is required to be convex in the above context (cf. Theorem 2.5). This raises a question: while Theorem 2.5 establishes the relationship between approximate solutions of problems (P) and (P γ ), the distinction between the global or local optimal solutions of problem (P) and (P γ ) remains unclear when F is non-convex.

We first establish the relationship between global optimal solutions of problems (P) and (P γ ) when F is non-convex, which is similar to Theorem 2.5.

Theorem 2.7. Suppose that Assumption 2.2 holds, G is convex, and F is l -Lipschitz continuous on dom( F ) . For any given ϵ > 0 and β > 0 , let

where γ ∗ is given by (2) . If ˜ x ∗ γ is an ϵ -global optimal solution of problem (P γ ) , then ˜ x ∗ γ is an ( ϵ, l -β ϵ β ) -global optimal solution of problem (P) .

Theorem 2.7 provides the relationship between the global optimal solutions of problems (P γ ) and (P). However, the relationship between local optimal solutions of these problems is more intricate than those of the global ones [Shen and Chen, 2023]. Given r > 0 and z ∈ R n , define B ( z , r ) := { x ∈ R n : ∥ x -z ∥ ≤ r } . We present the following theorem, which demonstrates that local optimal solutions of problem (P γ ) can serve as approximate local optimal solutions of problem (P).

Theorem 2.8. Suppose that Assumption 2.2 holds and G is convex. Let x ∗ γ be a local optimal solution of problem (P γ ) on B ( x ∗ γ , r ) . Assume F is l -Lipschitz continuous on B ( x ∗ γ , r ) . Then x ∗ γ is an approximate local optimal solution of problem (P) that satisfies F ( x ∗ γ ) -F ∗ B ≤ 0 and G ( x ∗ γ ) -G ∗ ≤ ϵ when α > 1 and γ ≥ ( ρl α ϵ α -1 ) 1 α , where F ∗ B is the optimal value of problem (P) on B ( x ∗ γ , r ) ∩ X opt . Furthermore, x ∗ γ is a local optimal solution of problem (P) when α = 1 and γ > ρl .

Indeed, the relationship between approximate local optimal solutions of problems (P γ ) and (P) is more intricate than the connection among global solutions presented in Theorem 2.5. These interactions will be the focus of our future work. The proofs of Theorems 2.7 and 2.8 are presented in Appendixes E.5 and E.6.

3 Main algorithms

In this section, we concentrate on addressing problem (P), making various assumptions, and offering distinct convergence outcomes.

3.1 Both objectives are convex composite functions

In this scenario, we address problem (P) where F and G are both composite functions, i.e., F = f 1 + f 2 and G = g 1 + g 2 .

Assumption 3.1. F and G satisfy the following assumptions.

The gradient of f 1 ( x ) , denoted as ∇ f 1 , is L f 1 -Lipschitz continuous on dom( F ) ;

The gradient of g 1 ( x ) , denoted as ∇ g 1 , is L g 1 -Lipschitz continuous on dom( G ) ;

f 2 and g 2 are proper, convex, lower semicontinuous, and possibly non-smooth.

We remark that Assumption 3.1(1)(3) is more general than many existing papers in the literature. Specifically, while previous works such as Beck and Sabach [2014], Amini and Yousefian [2019], Jiang et al. [2023], Giang-Tran et al. [2023] require the upper-level objective to be smooth or strongly convex, we simply assume that F is a composite function composed of a smooth convex function and a possibly non-smooth convex function. For the lower-level objective, previous works such as Beck and Sabach [2014], Amini and Yousefian [2019], Jiang et al. [2023], Giang-Tran et al. [2023] impose smoothness assumptions and, in some cases, convexity and compactness constraints on the domain; while our approach does not require these additional constraints, allowing for more flexibility and generality as presented in Assumption 3.1(2)(3).

We are now prepared to introduce two algorithms: the penalty-based accelerated proximal gradient (PB-APG) algorithm and its adaptive counterpart, the aPB-APG to solve problem (P γ ) and, subsequently, to obtain an ( ϵ F , ϵ G ) -optimal solution of problem (P).

To simplify notations, we omit the constant term -γG ∗ , and rewrite problem (P γ ) as follows,

where ϕ γ ( x ) = f 1 ( x ) + γg 1 ( x ) and ψ γ ( x ) = f 2 ( x ) + γg 2 ( x ) represent the smooth and nonsmooth parts, respectively. Then, it follows that the gradient of ϕ γ ( x ) is L γ -Lipschitz continuous with L γ = L f 1 + γL g 1 .

To implement the APG methods, we need another assumption concerning ψ γ ( x ) .

Assumption 3.2. For any γ > 0 , the function ψ γ ( x ) is prox-friendly, i.e., the proximal mapping

is easy to compute for any t > 0 .

The function ψ γ ( x ) represents the sum of two non-smooth functions, and proximal mapping for such function sums is widely studied and used in the literature [Yu, 2013, Pustelnik and Condat, 2017, Adly et al., 2019, Boob et al., 2023, Latafat et al., 2023]. This assumption is also a more general requirement compared to many existing algorithms [Sabach and Shtern, 2017, Giang-Tran et al., 2023]. It is important to note that our assumption is more general than existing literature. In the simple bilevel literature, when employing proximal mappings, researchers often consider the scenario where only one level contains a nonsmooth term (see, e.g., [Jiang et al., 2023, Doron and Shtern, 2023, Samadi et al., 2023, Merchav and Sabach, 2023]). In this case, the proximal mapping of the sum f 2 + γg 2 is then reduced to the proximal mapping of either f 2 or g 2 , which is a more easily satisfied condition.

3.1.1 Accelerated proximal gradient-based algorithm

We apply the APG algorithm [Beck and Teboulle, 2009, Lin and Xiao, 2014, Nesterov, 2013] to solve problem (P Φ ), as outlined in Algorithm 1. Moreover, if the Lipschitz constant L γ is unknown or computationally infeasible, line search [Beck and Teboulle, 2009] can be adopted and will yield almost the same complexity bound. For brevity, we denote Algorithm 1 as ˆ x = PB-APG ( ϕ γ , ψ γ , L f 1 , L g 1 , x 0 , ϵ ) , where ˆ x represents an ϵ -optimal solution of (P Φ ).

Algorithm 1 Penalty-based APG (PB-APG)

1: Input: γ, L γ = L f 1 + γL g 1 , x -1 = x 0 ∈ R n , R > 0 , t -1 = t 0 = 1 , k = 0 , ϵ > 0 and { t k } .

5: end for

2: for k ≥ 0 do 3: y k = x k + t k ( t -1 k -1 -1 ) ( x k -x k -1 ) 4: x k +1 = prox L -1 γ ψ γ ( y k -L -1 γ ∇ ϕ γ ( y k ))

In Algorithm 1, we stop the loop of Line. 3 - 4 if the number of iterations satisfies that:

where R is a constant that satisfies ∥ x 0 -x ∗ ∥ ≤ R .

Combining Theorem 2.5 and [Tseng, 2008, Corollary 2], we establish the following complexity result for problem (P).

Theorem 3.3. Suppose that Assumptions 2.1, 2.2, 3.1 and 3.2 hold and the sequence { t k } in Algorithm 1 satisfies 1 -t k +1 t 2 k +1 ≤ 1 t 2 k . Let γ be given as in Theorem 2.5. Algorithm 1 generates an ( ϵ, l -β F ϵ β ) -optimal solution of problem (P) after at most K iterations, where

Note that Theorem 3.3 encompasses all possible relationships between the magnitudes of ϵ F and ϵ G in (1), as α ≥ 1 and β > 0 are arbitrary. Specially, if α = 1 and β ≤ α , the number of iterations is K = O ( √ ( L f 1 + l F L g 1 ) /ϵ ) . This result matches the lower bound complexity for unconstrained smooth or convex composite optimization [Nemirovsky and Yudin, 1983, Woodworth and Srebro, 2016]. Additionally, if g 1 ≡ 0 , the number of iterations for obtaining an ( ϵ, ϵ β ) -optimal solution of problems (P) is independent of γ , which can be improved to K = O ( √ L f 1 /ϵ ) .

Remark 3.4. It is noteworthy that Theorem 1 in a previous paper Samadi et al. [2023] provides the first method that needs O ( √ ( L g 1 + l F L g 1 ) /ϵ ) iterations to achieve an ( ϵ, ϵ ) solution if α = 1 and F is smooth. Nevertheless, our methodology diverges in various respects. First, our approach is rooted in the penalization formulation of problem (P Val ), while the approach proposed by Samadi et al. [2023] is based on the Tikhonov regularization [Tikhonov and Arsenin, 1977]. Second, we provide a theoretical framework that clearly delineates the relationship between approximate solutions of problems (P) and (P γ ) for all cases of α ≥ 1 and F is non-convex, as indicated in Lemmas 2.3, 2.4 and Theorems 2.5, 2.7, 2.8. Therefore, we can first shift our focus from (P) to (P γ ) based on the penalization framework and then use various methods to solve (P γ ), not limited to using the APG methods. Besides, the association between approximate solutions of problem (P) and (P γ ) differs significantly based on whether α > 1 or α = 1 . For the case of α > 1 , the lower bound comprehensively integrates the accuracy parameter ϵ , which results in a more sophisticated analysis of the convergence result, while Samadi et al. [2023] did not consider the situation when α > 1 . Third, our method applies to the case that F is composite, while Samadi et al. [2023] requires F to be smooth. Finally, we also propose an adaptive version of our algorithm (see Algorithm 2) that does not require an estimate of γ .

3.1.2 Adaptive version with warm-start mechanism

In practice, the penalty parameter γ might be difficult to determine. This motivates us to propose Algorithm 2, which adaptively updates γ and invokes PB-APG with dynamic γ and solution accuracies.

Algorithm 2 Adaptive PB-APG method (aPB-APG)

1: Input: x 0 ∈ R n , γ 0 = γ 1 > 0 , L f 1 , L g 1 , ν > 1 , η > 1 , ϵ 0 > 0 . 2: for k ≥ 0 do 3: ϕ k ( x ) = f 1 ( x ) + γ k g 1 ( x ) 4: ψ k ( x ) = f 2 ( x ) + γ k g 2 ( x ) 5: Invoke x k = PB-APG ( ϕ k , ψ k , L f 1 , L g 1 , x k -1 , ϵ k ) 6: ϵ k +1 = ϵ k /η 7: γ k +1 = νγ k 8: end for

In Algorithm 2, we adaptively update the penalty parameter γ k , and invoke the PB-APG to generate an approximate solution for (P γ ) with accuracy ϵ = ϵ k . Meanwhile, a warm-start mechanism is employed, meaning that the initial point for each subproblem is the output of the preceding subproblem. The convergence result of Algorithm 2 is as follows.

Theorem 3.5. Suppose that Assumptions 2.1, 2.2, 3.1, and 3.2 hold. Also assume that for every outcome of inner loop in Algorithm 2, ∥ x k -x ∗ k ∥ ≤ R . Let ϵ 0 > 0 be given.

When α > 1 , set ν > η α -1 , and define N := ⌈ log η 1 -α ν ( ρL α F ( α -1) α -1 α -α ϵ 1 -α 0 /γ 0 ) ⌉ + and γ ∗ k := ρL α F ( α -1) α -1 α -α ϵ 1 -α 0 η k ( α -1) .

When α = 1 , set ν > 1 , and define N := ⌈ log ν ( ρl F /γ 0 ) ⌉ + and γ ∗ k := ρL F .

Then, for any k ≥ N , Algorithm 2 generates an ( ϵ 0 η k , 2 ϵ 0 η k ( γ 0 ν k -γ ∗ k ) ) -optimal solution of problem (P) after at most K iterations, where K satisfies

Theorem 3.5 shows that for any given initial accuracy ϵ 0 > 0 , Algorithm 2 can produce an approximate solution of problem (P) with the desired accuracy.

Remark 3.6. From Theorem 3.5, one can obtain an ( ϵ, ϵ γ 0 ν k -γ ∗ k ) -optimal solution of problem (P) within O ( √ L f 1 /ϵ + √ L g 1 /ϵ α ) iterations when ϵ/η ≤ ϵ 0 /η k ≤ ϵ , which is similar to the complexity results in Theorem 3.3.

3.1.3 The upper-level objective is strongly convex

We investigate the convergence outcomes when the smooth part of the upper-level objective exhibits strong convexity.

Assumption 3.7. f 1 ( x ) is µ -strongly convex on dom( F ) with µ > 0 .

Assumption 3.7 is another widely adopted setting in the existing SBO literature [Beck and Sabach, 2014, Sabach and Shtern, 2017, Amini and Yousefian, 2019, Merchav and Sabach, 2023]. Here, we propose a variant of PB-APG that can provide better complexity results than existing methods. Our main integration is an APG-based algorithm, which has been studied in the existing literature [Nesterov, 2013, Lin and Xiao, 2014, Xu, 2022]. In this paper, we adopt the algorithm proposed in Lin and Xiao [2014] and modify it with a constant step-size for simplicity as in Algorithm 3. Similar to Algorithm 1, we denote Algorithm 3 by ˆ x = PB-APG-sc ( ϕ γ , ψ γ , µ, L f 1 , L g 1 , y 0 , ϵ ) .

Algorithm 3 PB-APG method for Strong Convexity Case (PB-APG-sc)

The convergence analysis of Algorithm 3 is in the existing literature [Nesterov, 2013, Lin and Xiao, 2014]. Combining [Lin and Xiao, 2014, Theorem 1] and Theorem 2.5, we have the following complexity result.

Theorem 3.8. Suppose that Assumptions 2.1, 2.2, 3.1, 3.2, and 3.7 hold. Algorithm 3 can produce an ( ϵ, l -β F ϵ β ) -optimal solution of problem (P) after at most K iterations, where K satisfies

Theorem 3.8 improves the complexity results of Theorem 3.3 significantly. Specifically, when 0 < β ≤ α = 1 , the convergence rate can be improved to be linear, i.e., K = O ( √ L f 1 /µ log 1 ϵ ) .

Additionally, we present an adaptive variant of PB-APG-sc, termed aPB-APG-sc, which adaptively executes x k = PB-APG-sc ( ϕ k , ψ k , µ, L f 1 , L g 1 , x k -1 , ϵ k ) and enjoys the similar complexity results of Algorithm 3, as delineated in Algorithm 4 within Appendix D.1.

3.2 Both objectives are non-smooth

In this section, we focus on the scenario where both the upper- and lower-level objectives are non-smooth, namely, f 1 = g 1 ≡ 0 . Additionally, we assume that there is a point x ∈ C in the lower level problem, where C is either R n (the unconstrained case) or a nonempty closed and convex set satisfying C ⊆ int (dom( F ) ∩ dom( G )) .

It is worth noting that in the case where both F and G are non-smooth, the convergence result may not be as favorable as those in the previous scenarios. This is primarily due to the limited availability of information and unfavorable properties concerning F and G . In this case, we employ a subgradient method to solve problem (P γ ), which has been extensively studied in the existing literature [Shor, 2012, Bubeck et al., 2015, Beck, 2017, Nesterov, 2018]. Specifically, we update

where ξ k ∈ ∂ Φ γ ( x k ) is an subgradient of Φ γ ( x k ) , and Proj C ( x ) is the projection of x onto C .

Let x ∗ γ be an optimal solution of problem (P γ ) and suppose that there exists a constant R such that ∥ x 0 -x ∗ γ ∥ ≤ R . Motivated by Theorem 8.28 in Beck [2017], we establish the subsequent complexity result for problem (P).

Theorem 3.9. Suppose that Assumption 3.1(3) holds, f 2 and g 2 are l f 2 - and l g 2 -Lipschitz continuous, respectively. Set step-size η k = R l γ √ k +1 in (4) . Then, the subgradient method produces an ( ϵ, l -β f 2 ϵ β ) -optimal

solution of problem (P) after at most K iterations, where K satisfies

For non-smooth SBO problems, our method has lower complexity compared to existing approaches. Specifically, under a bounded domain assumption, Helou and Simões [2017] simply proposed an ϵ -subgradient method with an asymptotic rate towards the optimal solution set. The a-IRG method in Kaushik and Yousefian [2021] achieved convergence rates of O (1 /ϵ 1 0 . 5 -b ) and O (1 /ϵ 1 b ) for the upper- and lower-level objectives, respectively, where b ∈ (0 , 0 . 5) . Setting b = 0 . 25 yields the convergence rates of O (1 /ϵ 4 ) for both upper- and lower-level objectives, which indicates that our complexity is more efficient than theirs when α < 2 and β ≤ α . Furthermore, the online framework proposed in Shen et al. [2023] performed a complexity of O (1 /ϵ 3 ) for both upper- and lower-level objectives. Similarly, our approach prevails over theirs when α < 1 . 5 and β ≤ α .

Strongly convex upper-level objective. Based on Theorem 8.31 in Beck [2017], we next explore the improved complexity result for problem (P) when f 2 is additionally strongly convex.

Theorem 3.10. Suppose that Assumption 3.1(3) holds, C ⊆ int (dom( F ) ∩ dom( G )) , f 2 is l f 2 -Lipschitz continuous and µ f 2 -strongly convex 3 , and g 2 is l g 2 -Lipschitz continuous. Choose step-size η k = 2 µ f 2 ( k +1) in (4) . Then, the subgradient method produces an ( ϵ, l -β f 2 ϵ β ) -optimal solution of problem (P) after at most K iterations, where K satisfies

To our knowledge, within the context of Theorem 3.10, current findings fail to exploit strong convexity to enhance results. However, our approach capitalizes on distinct structural characteristics that yield superior complexity outcomes relative to Theorem 3.9 in cases where α < 2 and β ≤ α .

4 Numerical experiments

We apply our Algorithms 1, 2, 3 and 4 to two simple bilevel optimization problems from the motivating examples in Appendix A. The performances of our methods are compared with several existing methods: MNG [Beck and Sabach, 2014], BiG-SAM [Sabach and Shtern, 2017], DBGD [Gong et al., 2021], a-IRG [Kaushik and Yousefian, 2021], CG-BiO [Jiang et al., 2023], Bi-SG [Merchav and Sabach, 2023] and R-APM [Samadi et al., 2023]. For practical efficiency, we use the Greedy FISTA algorithm proposed in Liang et al. [2022] as the APG method in our approach. Detailed settings and additional experimental results are presented in Appendix F.

0

0.1

0.2

0.3

0.4

0.5

0.6

10

-10

10

-5

10

0

0

0.1

0.2

0.3

0.4

0.5

0.6

0

5

10

15

20

0

10

20

30

40

10

-8

10

-6

10

-4

10

-2

10

0

10

2

0

10

20

30

40

0

20

40

60

80

100

4.1 Logistic regression problem (LRP)

The LRP reads

where I C ( x ) is the indicator function of the set C = { x ∈ R n : ∥ x ∥ 1 ≤ θ } with θ = 10 . Our goal is to find a solution to the lower-level problem with the smallest Euclidean norm. The upper-level objective only

consists of the smooth part, which is 1 -strongly convex and 1 -smooth; meanwhile, the lower-level objective is a composite function, where the smooth part is 1 4 m λ max ( A T A ) -smooth, and the nonsmooth part is prox-friendly [Duchi et al., 2008].

In this experiment, we compare our methods with MNG, BiG-SAM, DBGD, a-IRG, CG-BiO, and Bi-SG. We plot the values of residuals of the lower-level objective G ( x k ) -G ∗ and the upper-level objective over time in Figure 1.

As shown in Figure 1, the PB-APG, aPB-APG, PB-APG-sc, and aPB-APG-sc algorithms exhibit significantly faster convergence performance than the other methods for both lower- and upper-level objectives, although R-APM attains similar outcomes, our PB-APG and PB-APG-sc ensure a more rapid decline than it, as shown in the first subfigure of Figure 1. This is because our methods achieve lower optimal gaps and desired function values of the lower- and upper-level objectives with less execution time. This observation confirms the improved complexity results stated in the theorems above. Although the high exactness of our methods for the lower-level problem leads to larger upper-level objectives, Table 3 in Appendix F.1 shows that our methods are much closer to the optimal value. This is reasonable because the other methods exhibit lower accuracy at the lower-level problem, resulting in larger feasible sets compared to the lower-level optimal solution set X opt. In addition, Figure 1 demonstrates that aPB-APG and aPB-APG-sc outperform PB-APG and PB-APG-sc in terms of convergence rate. This improvement can be attributed to the adaptiveness incorporated in Algorithms 2 and 4.

4.2 Least squares regression problem (LSRP)

The LSRP has the following form:

where τ = 0 . 02 regulates the trade-off between ℓ 1 and ℓ 2 norms. We aim to find a sparse solution for the lowerlevel problem. The upper-level objective is formulated as a composite function, which consists of a τ -strongly convex and τ -smooth component, along with a proximal-friendly non-smooth component [Beck, 2017]. The lower-level objective is a smooth function with a smoothness parameter of 1 m λ max ( A T A ) .

In this experiment, we compare the performances of our methods with a-IRG, BiG-SAM, and Bi-SG. We plot the values of residuals of lower-level objective G ( x k ) -G ∗ and the upper-level objective over time in Figure 2.

Figure 2 shows that the proposed PB-APG, aPB-APG, PB-APG-sc, and aPB-APG-sc converge faster than the compared methods for both the lower- and upper-level objectives, as well. For the upper-level objective, our methods achieve larger function values than other methods, except BiG-SAM ( δ = 0 . 01 ). This is because our methods attain higher accuracy for the lower-level objective than other methods. We have similar observations in Section 4.1. Furthermore, Figure 1 also demonstrates that the adaptive mechanism produces staircase-shaped curves for aPB-APG and aPB-APG-sc, which might prevent undesirable fluctuations in PB-APG and PB-APGsc.

5 Conclusion

This paper proposes a penalization framework that effectively addresses the challenges inherent in simple bilevel optimization problems. By delineating the relationship between approximate solutions of the original problem and its penalized reformulation, we enable the application of specific methods under varying assumptions for the original problem. Under the Hölderian error bound condition, our methods achieve superior complexity results compared to the existing methods. The performance is further improved when the smooth component of the upper-level objective is strongly convex. Additionally, we extend our framework to scenarios involving general nonsmooth objectives. Numerical experiments also validate the effectiveness of our algorithms.

References

Samir Adly, Loïc Bourdin, and Fabien Caubet. On a decomposition formula for the proximal operator of the sum of two convex functions. Journal of Convex Analysis , 26(2):699-718, 2019.

Mostafa Amini and Farzad Yousefian. An iterative regularized incremental projected subgradient method for a class of bilevel optimization problems. In 2019 American Control Conference (ACC) , pages 4069-4074. IEEE, 2019.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Amir Beck and Shoham Sabach. A first order method for finding minimal norm-like solutions of convex optimization problems. Mathematical Programming , 147(1-2):25-46, 2014.

Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences , 2(1):183-202, 2009.

Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.08136 , 2018.

Dimitri Bertsekas, Angelia Nedic, and Asuman Ozdaglar. Convex analysis and optimization , volume 1. Athena Scientific, 2003.

Nicholas Bishop, Long Tran-Thanh, and Enrico Gerding. Optimal learning from verified training data. Advances in Neural Information Processing Systems , 33:9520-9529, 2020.

Jérôme Bolte, Trong Phong Nguyen, Juan Peypouquet, and Bruce W Suter. From error bounds to the complexity of first-order descent methods for convex functions. Mathematical Programming , 165:471-507, 2017.

Digvijay Boob, Qi Deng, and Guanghui Lan. Stochastic first-order methods for convex and nonconvex functional constrained optimization. Mathematical Programming , 197(1):215-279, 2023.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

James V. Burke and Sien Deng. Weak sharp minima revisited, part ii: application to linear regularity and error bounds. Mathematical Programming , 104(2-3):235-261, 2005.

James V Burke and Michael C Ferris. Weak sharp minima in mathematical programming. SIAM Journal on Control and Optimization , 31(5):1340-1359, 1993.

Alexandre Cabot. Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization. SIAM Journal on Optimization , 15(2):555-572, 2005.

Chen, H. Xu, R. Jiang, et al. Lower-level duality based reformulation and majorization minimization algorithm for hyperparameter optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 784-792. PMLR, 2024.

Chen, J. Xu, and J. Zhang. Bilevel optimization without lower-level strong convexity from the hyperobjective perspective. arXiv preprint arXiv:2301.00712 , 2023.

Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239, 2019.

Stephan Dempe, Nguyen Dinh, Joydeep Dutta, and Tanushree Pandit. Simple bilevel programming and extensions. Mathematical Programming , 188:227-253, 2021.

Lior Doron and Shimrit Shtern. Methodology and first-order algorithms for solving nonsmooth and nonstrongly convex bilevel optimization problems. Mathematical Programming , 201:521-558, 2023.

Dmitriy Drusvyatskiy and Adrian S. Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of operations research , 43(3):919-948, 2018a.

Dmitriy Drusvyatskiy and Adrian S Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of Operations Research , 43(3):919-948, 2018b.

John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the l 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning , pages 272-279, 2008.

Joydeep Dutta and Tanushree Pandit. Algorithms for simple bilevel programming. Bilevel Optimization: Advances and Next Challenges , pages 253-291, 2020.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning , pages 1568-1577. PMLR, 2018.

Michael P Friedlander and Paul Tseng. Exact regularization of convex programs. SIAM Journal on Optimization , 18(4):1326-1350, 2008.

Khanh-Hung Giang-Tran, Nam Ho-Nguyen, and Dabeen Lee. Projection-free methods for solving convex bilevel optimization problems. arXiv preprint arXiv:2311.09738 , 2023.

Chengyue Gong, Xingchao Liu, and Qiang Liu. Bi-objective trade-off with dynamic barrier gradient descent. In International Conference on Neural Information Processing Systems , pages 29630-29642, 2021.

Elias S Helou and Lucas EA Simões. ϵ -subgradient algorithms for bilevel convex optimization. Inverse Problems , 33(5):055020, 2017.

Huang. On momentum-based gradient methods for bilevel optimization with nonconvex lower-level. arXiv preprint arXiv:2303.03944 , 2023.

Ruichen Jiang, Nazanin Abolfazli, Aryan Mokhtari, and Erfan Yazdandoost Hamedani. A conditional gradientbased method for simple bilevel optimization with convex lower-level problem. In International Conference on Artificial Intelligence and Statistics , pages 10305-10323. PMLR, 2023.

Rujun Jiang and Xudong Li. Hölderian error bounds and kurdyka-łojasiewicz inequality for the trust region subproblem. Mathematics of Operations Research , 47(4):3025-3050, 2022.

Harshal D. Kaushik and Farzad Yousefian. A method with convergence rates for optimization problems with variational inequality constraints. SIAM Journal on Optimization , 31(3):2171-2198, 2021.

Matthias Kissel, Martin Gottwald, and Klaus Diepold. Neural network training with safe regularization in the null space of batch activations. In Artificial Neural Networks and Machine Learning-ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15-18, 2020, Proceedings, Part II 29 , pages 217-228. Springer, 2020.

Puya Latafat, Andreas Themelis, Silvia Villa, and Panagiotis Patrinos. Adabim: An adaptive proximal gradient method for structured convex bilevel optimization. arXiv preprint arXiv:2305.03559 , 2023.

Jingwei Liang, Tao Luo, and Carola-Bibiane Schonlieb. Improving fast iterative shrinkage-thresholding algorithm: Faster, smarter, and greedier. SIAM Journal on Scientific Computing , 44(3):A1069-A1091, 2022.

Qihang Lin and Lin Xiao. An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization. In International Conference on Machine Learning , pages 73-81. PMLR, 2014.

Zhi-Quan Luo, Jong-Shi Pang, Daniel Ralph, and Shi-Quan Wu. Exact penalization and stationarity conditions of mathematical programs with equilibrium constraints. Mathematical Programming , 75(1):19-76, 1996.

Yura Malitsky. Chambolle-Pock and Tsengs methods: relationship and extension to the bilevel optimization. arXiv preprint arXiv:1706.02602 , 2017.

Roey Merchav and Shoham Sabach. Convex bi-level optimization problems with nonsmooth outer objective function. SIAM Journal on Optimization , 33(4):3114-3142, 2023.

Hong Mingyi, Wai Hoi-To, Wang Zhaoran, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05170 , 2020.

Arkadij Semenoviˇ c Nemirovsky and David Borisovich Yudin. Problem complexity and method efficiency in optimization . Wiley, 1983.

Yurii Nesterov. Gradient methods for minimizing composite functions. Mathematical programming , 140(1): 125-161, 2013.

Yurii Nesterov. Lectures on convex optimization , volume 137. Springer, 2018.

Jong Shi Pang. Error bounds in mathematical programming. Mathematical Programming , 79(1-3):299-332, 1997.

Nelly Pustelnik and Laurent Condat. Proximity operator of a sum of functions; application to depth map estimation. IEEE Signal Processing Letters , 24(12):1827-1831, 2017.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems , 32, 2019.

Vincent Roulet and Alexandre d'Aspremont. Sharpness, restart and acceleration. SIAM Journal on Optimization , 30(1):262-289, 2020.

Shoham Sabach and Shimrit Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

Sepideh Samadi, Daniel Burbano, and Farzad Yousefian. Achieving optimal complexity guarantees for a class of bilevel convex optimization problems. arXiv preprint arXiv:2310.12247 , 2023.

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 17231732. PMLR, 2019.

Han Shen and Tianyi Chen. On penalty-based bilevel gradient descent method. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 30992-31015. PMLR, 23-29 Jul 2023.

Lingqing Shen, Nam Ho-Nguyen, and Fatma Kılınç-Karzan. An online convex optimization-based framework for convex bilevel optimization. Mathematical Programming , 198(2):1519-1582, 2023.

Naum Zuselevich Shor. Minimization methods for non-differentiable functions , volume 3. Springer Science & Business Media, 2012.

Mikhail Solodov. An explicit descent method for bilevel convex optimization. Journal of Convex Analysis , 14 (2):227-237, 2007.

Sow, K. Ji, Z. Guan, and et al. A primal-dual approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123 , 2022.

Marcin Studniarski and Doug E Ward. Weak sharp minima: characterizations and sufficient conditions. SIAM Journal on Control and Optimization , 38(1):219-236, 1999.

Andre Nikolaevich Tikhonov and V. I. A. K. Arsenin. Solutions of ill-posed problems . Wiley, 1977.

Paul Tseng. On accelerated proximal gradient methods for convex-concave optimization. unpublished manuscript , 2008.

Jiali Wang, He Chen, Rujun Jiang, Xudong Li, and Zihao Li. Fast algorithms for stackelberg prediction game with least squares loss. In International Conference on Machine Learning , pages 10708-10716. PMLR, 2021.

Jiali Wang, Wen Huang, Rujun Jiang, Xudong Li, and Alex L Wang. Solving stackelberg prediction game with least squares loss via spherically constrained least squares reformulation. In International Conference on Machine Learning , pages 22665-22679. PMLR, 2022.

Blake E Woodworth and Nati Srebro. Tight complexity bounds for optimizing composite objectives. Advances in neural information processing systems , 29, 2016.

Yangyang Xu. First-order methods for problems with o (1) functional constraints can have almost the same convergence rate as for unconstrained problems. SIAM Journal on Optimization , 32(3):1759-1790, 2022.

Yao-Liang Yu. On decomposing the proximal map. Advances in neural information processing systems , 26, 2013.

Jinshan Zeng, Tim Tsz-Kit Lau, Shaobo Lin, and Yuan Yao. Global convergence of block coordinate descent in deep learning. In International conference on machine learning , pages 7313-7323. PMLR, 2019.

Zirui Zhou and Anthony Man-Cho So. A unified approach to error bounds for structured convex optimization problems. Mathematical Programming , 165:689-728, 2017.

A Motivating examples

Many machine learning applications involve a primary objective G , which usually represents the training loss, and a secondary objective F , which can be a regularization term or an auxiliary loss. A common approach for such problems is to optimize G fully and then use F to select the optimal solutions from the ones obtained for G . This is called lexicographic optimization [Kissel et al., 2020, Gong et al., 2021]. Two classes of lexicographic optimization problems are the regularized problem, also known as the ill-posed optimization problem [Amini and Yousefian, 2019, Jiang et al., 2023], and the over-parameterized regression [Jiang et al., 2023], where the upper-level objectives are the regularization terms or loss functions, and the lower-level objectives are the loss functions and the constraint terms. We present some examples of these classes of problems as follows.

Example A.1 (Linear Inverse Problems) . Linear inverse problems aim to reconstruct a vector x ∈ R n from measurements b ∈ R m that satisfy b = A x + ρε , where A : R n → R m is a linear mapping, ε ∈ R m is unknown noise, and ρ > 0 is its magnitude. Various optimization techniques can address these problems. We focus on the bilevel formulation, widely adopted in the literature [Beck and Sabach, 2014, Sabach and Shtern, 2017, Dempe et al., 2021, Latafat et al., 2023, Merchav and Sabach, 2023].

The lower-level objective in the bilevel formulation is given by

where I C ( x ) is the indicator function of a set C that satifies I C ( x ) = 0 if x ∈ C , and I C ( x ) = + ∞ if x / ∈ C . The set C is a closed, convex set that can be chosen as C = R n , C = { x ∈ R n : x ≥ 0 } , or C = { x ∈ R n : ∥ x ∥ 1 ≤ θ } for some θ > 0 .

This problem may have multiple minimizer solutions. Hence, a reasonable option is to consider the minimal norm solution problem, i.e., find the optimal solution with the smallest Euclidean norm [Beck and Sabach, 2014, Sabach and Shtern, 2017, Latafat et al., 2023]:

We need to solve the simple bilevel optimization problem:

Example A.2 (Sparse Solution of Linear Inverse Problems) . Consider the same setting as in Example A.1, but with the additional goal of finding a sparse solution among all the minimizers of the linear inverse problem (7). This can simplify the model and improve computational efficiency. To achieve sparsity, we can use any function that encourages it. One such function is the well-known elastic net regularization [Friedlander and Tseng, 2008, Amini and Yousefian, 2019, Merchav and Sabach, 2023], which is defined as

where τ > 0 regulates the trade-off between ℓ 1 and ℓ 2 norms.

This example corresponds to our second experiment in Section 4.2.

Example A.3 (Logistic Regression Problem) . The logistic regression problem aims to map the feature vectors a i to the target labels b i . A standard machine learning technique for this problem is to minimize the logistic loss function over the given dataset [Amini and Yousefian, 2019, Gong et al., 2021, Jiang et al., 2023, Latafat et al., 2023, Merchav and Sabach, 2023]. We assume that the dataset consists of a feature matrix A ∈ R m × n and a label vector b ∈ R m , with b i ∈ {-1 , 1 } for each i . The logistic loss function is defined as

Over-fitting is a common issue when the number of features is large compared to the number of instances m . A possible approach is to regularize the logistic objective function with a specific function or a constraint [Jiang et al., 2023, Merchav and Sabach, 2023]. For instance, we can use g 2 ( x ) = I C ( x ) , where I C ( x ) is the indicator of the set C = { x ∈ R n : ∥ x ∥ 1 ≤ θ } , as in Example A.1.

This problem may also have multiple optimal solutions. Hence, a natural extension is to consider the minimal norm solution problem [Gong et al., 2021, Jiang et al., 2023, Latafat et al., 2023], as in Example A.1. This requires solving the following problem:

When choosing C = { x ∈ R n : ∥ x ∥ 1 ≤ θ } for some θ > 0 , it corresponds to our first experiment in Section 4.1.

Example A.4 (Over-parameterized Regression Problem) . The linear regression problem aims to find a parameter vector x ∈ R n that minimizes the training loss ℓ tr ( x ) over the training dataset D tr . Without explicit regularization, the over-parameterized regression problem has multiple minima. However, these minima may have different generalization performance. Therefore, we introduce a secondary objective, such as the validation loss over a validation set D val , to select one of the global minima of the training loss. This results in the following bilevel problem:

For instance, we can consider the sparse linear regression problem, where the lower-level objective consists of the training error and a regularization term, namely, G ( x ) = 1 2 ∥ A tr x -b tr ∥ 2 + I C ( x ) . Here, I C ( x ) denotes the indicator of a convex set, as in Example A.2. The upper-level objective is the validation error, i.e., F ( x ) = 1 2 ∥ A val x -b val ∥ 2 . The linear regression problem is over-parameterized when the number of features n is larger than the number of data instances in the training set.

B Comparison between simple bilevel optimization methods

C Examples of functions satisfying the Hölderian error bound

We present several examples of functions that satisfy the Hölderian error bound Assumption 2.2 and their corresponding exponent parameter α in Table 2. We also provide some clarifications for Table 2 below. The abbreviations ' Q ∈ S n ' and ' Q ≻ 0 ' stand for ' Q is a symmetric matrix of order n and a positive definite matrix, respectively. We refer the reader to Pang [1997], Bolte et al. [2017], Zhou and So [2017], Jiang and Li [2022], Doron and Shtern [2023] and the references therein for more examples of functions that satisfy Hölderian error bound Assumption 2.2. Furthermore, it is noteworthy that numerous applications in neural networks, such as deep neural networks (DNNs), also comply with this assumption, as discussed in Bolte et al. [2017], Zeng et al. [2019].

∥

∥

D Supplementary results

D.1 Adaptive version of PB-APG method with strong convexity assumption

Algorithm 4 Adaptive PB-APG-sc method (aPB-APG-sc)

1: Input: x -1 = x 0 ∈ R n , γ 0 = γ 1 > 0 , L f 1 , L g 1 , ν > 1 , η > 1 , ϵ 0 > 0 .

3: ϕ k ( x ) = f 1 ( x ) + γ k g 1 ( x )

2: for k ≥ 0 do

4: ψ k ( x ) = f 2 ( x ) + γ k g 2 ( x )

5: Invoke x k = PB-APG-sc ( ϕ k , ψ k , µ, L f 1 , L g 1 , x k -1 , ϵ k )

7: γ k +1 = νγ k

6: ϵ k +1 = 1 η ϵ k

8: end for

Similar to Algorithm 2, we have the following convergence results of Algorithm 4.

Theorem D.1. Suppose that Assumptions 2.1, 2.2, 3.1, 3.2, and 3.7 hold. Let ϵ 0 > 0 be given.

When α > 1 , set ν > η α -1 , N = ⌈ log η 1 -α ν ( ρL α F ( α -1) α -1 α -α ϵ 1 -α 0 /γ 0 ) ⌉ + and γ ∗ k = ρL α F ( α -1) α -1 α -α ϵ 1 -α 0 η k ( α -1) ;

When α = 1 , set ν > 1 , N = ⌈ log ν ( ρl F /γ 0 ) ⌉ + and γ ∗ k = ρL F .

Then, for any k ≥ N , Algorithm 2 generates an ( ϵ 0 η k , 2 ϵ 0 η k ( γ 0 ν k -γ ∗ k ) ) -optimal solution of problem (P) after at most K iterations, where K satisfies

The proof is similar to the proof of Theorem 3.5 in Appendix E.8. So we omit it here.

E Proofs of main results

In this section, we propose the proofs of our main convergence results in this paper.

E.1 Proof of Lemma 2.3

Proof. Since X opt is closed and convex [Beck and Sabach, 2014], the projection of any x ∈ R n onto X opt , denoted as ¯ x , exists and is unique. Furthermore, it holds that dist ( x , X opt ) = ∥ x -¯ x ∥ .

Then, by Assumption 2.1, we have

Choosing γ ∗ = ρl α F ( α -1) α -1 α -α ϵ 1 -α , it follows that

where ( a ) follows from the Hölderian error bound assumption of p ( x ) , and ( b ) is from the fact that y = -l F z + γ ∗ ρ z α attains its minimum at z ∗ = ( ρl F αγ ∗ ) 1 α -1 .

Since ¯ x ∈ X opt is feasible for problem (P), we have F (¯ x ) ≥ F ∗ . This along with (11) indicates

Let x ∗ be an optimal solution of (P) so that F ( x ∗ ) = F ∗ . In addition, since x ∗ ∈ X opt, we have p ( x ∗ ) = 0 . Combine these results with (12), we have

This demonstrates that an optimal solution of (P) is an ϵ -optimal solution for (P γ ).

E.2 Proof of Lemma 2.4

Proof. The proof is motivated by Theorem 1 in Luo et al. [1996]. Denote x ∗ , x ∗ γ as optimal solutions of problem (P) and (P γ ), respectively.

For any x ∈ R n , let ¯ x be the projection of x onto X opt. Then ¯ x is a feasible solution of (P) and F (¯ x ) ≥ F ( x ∗ ) holds. Then we have

where ( a ) follows from (10) and the Hölderian error bound assumption of p ( x ) , and ( b ) follows from γ ≥ ρl F . Therefore, we conclude that x ∗ is an optimal solution of (P γ ).

For the converse part, let ¯ x ∗ γ be the projection of x ∗ γ onto X opt. Then ¯ x ∗ γ is a feasible solution of (P). Therefore, it holds that F (¯ x ∗ γ ) ≥ F ( x ∗ ) . Similarly, we have

where the inequality ( c ) follows from (10) and the Hölderian error bound assumption of p ( x ) .

Therefore, all inequalities in (15) become equalities. We deduce that ∥ x ∗ γ -¯ x ∗ γ ∥ = 0 if γ > ρl F , implying that x ∗ γ is in X opt , i.e., p ( x ∗ γ ) = 0 . Furthermore, as the first inequality of (15) becomes an equality, we obtain

Therefore, x ∗ γ is also an optimal solution of (P).

E.3 Proof of Theorem 2.5

Proof. Denote x ∗ , x ∗ γ as optimal solutions of problem (P) and (P γ ), respectively.

Note that the arguments in the proof of Lemma 2.3 still hold. Substituting x = x ∗ into (16) and utilizing p ( x ∗ ) = 0 , we have

where the last inequality follows from setting x = ˜ x ∗ γ in (13). Then, it holds that

By setting x = x ∗ in (16), we have

Using the fact that p ( x ∗ ) = 0 ≤ p (˜ x ∗ γ ) , we have

Combing (18) with (17), we conclude that ˜ x ∗ γ is an ( ϵ, l -β F ϵ β ) -optimal solution of (P).

Case of α = 1 . Since ˜ x ∗ γ is an ϵ -optimal solution of (P γ ), we have

On the one hand, as γ = γ ∗ + l β F ϵ 1 -β > γ ∗ , by Lemma 2.4, x ∗ γ is an optimal solution of (P). On the other hand, since γ ≥ γ ∗ , according to Lemma 2.4, x ∗ is also an optimal solution of (P γ ). Therefore, p ( x ∗ ) = 0 and p ( x ∗ γ ) = 0 , it holds that

The second inequality of (20) and p (˜ x ∗ γ ) ≥ 0 imply that where the first inequality follows from the fact that x ∗ is an optimal solution of (P γ ), and the last inequality follows from the optimality of x ∗ to (P γ ) when γ ≥ γ ∗ .

That is, it holds that

In addition, from (20), we have F (˜ x ∗ γ ) + γp (˜ x ∗ γ ) ≤ F (˜ x ∗ γ ) + γ ∗ p (˜ x ∗ γ ) + ϵ , which implies that

This result along with (21) demonstrate that ˜ x ∗ γ is an ( ϵ, l -β F ϵ β ) -optimal solution of (P).

E.4 Proof of Theorem 2.6

Proof. Let ˆ x ∗ γ be the projection of ˜ x ∗ γ on X opt, we have ∥ ˜ x ∗ γ -ˆ x ∗ γ ∥ = dist(˜ x ∗ γ , X opt ) .

By Assumption 2.2, the following inequality holds,

where ( a ) follows from (17) when α > 1 or from (22) when α = 1 .

where the first inequality follows from F (ˆ x ∗ γ ) ≥ F ∗ and ˆ x ∗ γ ∈ X opt .

E.5 Proof of Theorem 2.7

Proof. For any x ∈ dom( F ) , let ¯ x be the projection of x onto X opt, where the existence and uniqueness of ¯ x follows from that X opt is closed and convex. Since F is l -Lipschitz continuous, similar to (10), we have

Therefore, all the requirements of (10) in equations (11), (14) and (15) can be replaced by (24). This implies that Lemmas 2.3 and 2.4 also hold for the global solutions of problems (P) and (P γ ) when F is non-convex. Then, the final result follows a similar pattern to Theorem 2.5. Here we omit it.

E.6 Proof of Theorem 2.8

Proof. Let ¯ x ∗ γ be the projection of x ∗ γ onto X opt and ˆ x ∗ γ = c x ∗ γ +(1 -c )¯ x ∗ γ with c = min { 1 , 1 -r ∥ x ∗ γ -¯ x ∗ γ ∥ } , which implies that ˆ x ∗ γ ∈ B ( x ∗ γ , r ) . Then, we have

F ( x ∗ γ ) + γp ( x ∗ γ ) ≤ F (ˆ x ∗ γ ) + γp (ˆ x ∗ γ ) ( i ) ≤ F (ˆ x ∗ γ ) + γ ( cp ( x ∗ γ ) + (1 -c ) p (¯ x ∗ γ )) = F (ˆ x ∗ γ ) + γcp ( x ∗ γ ) , (25) where inequality ( i ) follows from the convexity of p ( x ) .

Inequality (25) demonstrates that

where the second inequality follows from the l -Lipschitz continuity of F on ( x , r ) . Therefore, it holds that

-≤ -≤ ∥ -∥ -∥ -∥ ≤ -B ∗ γ

Case of α > 1 . By (26), we have p ( x ∗ γ ) ≤ ( ρl α γ α ) 1 α -1 , which demonstrates that p ( x ∗ γ ) ≤ ϵ if γ ≥ ( ρl α ϵ α -1 ) 1 α .

Then, for any x γ ∈ B ( x ∗ γ , r ) that also satisfies p ( x γ ) ≤ p ( x ∗ γ ) ≤ ϵ , we have which implies that F ( x ∗ γ ) -F ( x γ ) ≤ γ ( p ( x γ ) -p ( x ∗ γ )) ≤ 0 . The desired result follows.

Case of α = 1 . By (26), we have p ( x ∗ γ ) = 0 if γ > ρl . Therefore, for any x γ ∈ B ( x ∗ γ , r ) ∩ X opt , by the definition of x ∗ γ , it holds that

which demonstrates that F ( x ∗ γ ) ≤ F ( x γ ) . The desired result follows.

E.7 Proof of Theorem 3.3

Proof. From [Beck, 2017, Theorem 10.34], the objective value after K iterations can be bounded by

where L γ = L f 1 + γL g 1 .

Combining this with our stopping criterion, we find that after K iterations,

This indicates that we obtain an ϵ -optimal solution to problem (P γ ). The value of K satisfies:

Specifically, we analyze the value of K in various scenarios in the form of O ( · ) .

Case of α > 1 . In this case, γ = γ ∗ + 2 l β F ϵ 1 -β comprises two components: γ ∗ and 2 l β F ϵ 1 -β . Therefore, it is natural to discuss which of these two components plays the dominant role in the complexity results. First, we write K in the form:

If β < α , the dominating term in γ is γ ∗ = ρl α F ( α -1) α -1 α -α ϵ 1 -α . Then, the number of iterations is

If β = α , we have γ = ( ρ ( α -1) α -1 α -α +2 ) l α F ϵ 1 -α . Then, the number of iterations is

If β > α , the dominating term in γ is 2 l β F ϵ 1 -β . Then, the number of iterations is

Case of α = 1 . In this case, γ = γ ∗ + l β F ϵ 1 -β , where γ ∗ = ρl F . Similarly, we explore which of these two elements plays a more significant role.

If β < 1 , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = 1 , we have γ = ( ρ +1) l F ϵ 1 -α . Then the number of iterations is

If β > 1 , the dominating term in γ is l β F ϵ 1 -β . Then, the number of iterations is

Combining the above results, we conclude that

E.8 Proof of Theorem 3.5

Proof. In this proof, we denote Φ ∗ k as the optimal value of problem (P γ ) when γ = γ k , and x k as the output of PB-APG (Algorithm 1) in the k -th iteration.

Case of α > 1 . Suppose that N is the smallest nonnegative integer such that γ N ≥ γ ∗ N := ρl α F ( α -1) α -1 α -α ϵ 1 -α N . In this case, we have

which is equivalent to

From (29), after at most N := ⌈ log η 1 -α ν ( ρl α F ( α -1) α -1 α -α ϵ 1 -α 0 γ 0 ) ⌉ + iterations, (28) holds.

Since x N = PB-APG ( ϕ N , ψ N , L f 1 , L g 1 , x N -1 , ϵ N ) , we have

which shows that x N is an ϵ N -optimal solution of (P γ ) with γ = γ N . From the proof in Theorem 2.5 (see inequalities (17) and (18) in Appendix E.3), x N is also an ( ϵ 0 η N , 2 ϵ 0 η N ( γ 0 ν N -γ ∗ N ) ) -optimal solution of problem (P).

Furthermore, note that for any iteration k ≥ N , inequality (29) always holds, which means that the following statement holds for any k ≥ N :

Let I k be the number of iterations of PB-APG required to satisfy (30) at the k -th iteration of aPBAPG. Then, for any k ≥ N , the total number of iterations is

From [Beck, 2017, Theorem 10.34], the number of iterations in i -th inner loop satisfies:

where x ∗ i is the optimal solution in i -th inner loop. Then we have that

For simplicity, we can also use O ( · ) to show the value of K .

Case of α = 1 . Suppose that after N updates, we have γ N ≥ ρl F , i.e.,

Similar to the case of α > 1 , the total iteration number is:

This demonstrates that after for all k ≥ N := log ν ( ρl F γ 0 ) , (31) always holds.

E.9 Proof of Theorem 3.8

Before proving Theorem 3.8, we need the following lemma that is modified from Theorem 1 in Lin and Xiao [2014], we state it in the subsequent lemma for completeness.

Lemma E.1. Suppose that Assumptions 2.1, 3.1, 3.2, and 3.7 hold. Let x ∗ γ be an optimal solution of problem (P γ ) and suppose that there exists a constant R such that max {∥ y 0 -x ∗ γ ∥ , ∥ ˜ x -x ∗ γ ∥} ≤ R . Then, the sequence { x k } generated by Algorithm 3 satisfy

Proof. Denote L γ = L f 1 + γL g 1 . By Theorem 3.1 in Beck and Teboulle [2009], we have

By Lemma E.1, we are now prepared to prove Theorem 3.8.

Proof. By Lemma E.1, the number of iterations required to achieve an ϵ -optimal solution for problem (P γ ) is

· Case of α > 1 . In this case, γ = γ ∗ +2 l β F ϵ 1 -β , where γ ∗ = ρl α F ( α -1) α -1 α -α ϵ 1 -α . If β < α , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = α , we have γ = ( ρ ( α -1) α -1 α -α +2 ) l α F ϵ 1 -α . Then, the number of iterations is

If β > α , the dominating term in γ is 2 l β F ϵ 1 -β . Then, the number of iterations is

· Case of α = 1 . When α = 1 , γ can be written as γ = γ ∗ + l β F ϵ 1 -β , where γ ∗ = ρl F . If β < 1 , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = 1 , we have γ = ( ρ +1) l F ϵ 1 -α . Then, the number of iterations is

If β > 1 , the dominating term in γ is l β F ϵ 1 -β . Then, we have

Combining the above results, we conclude that

E.10 Proof of Theorem 3.9

Proof. Denote l γ = l f 2 + γl g 2 . Define Φ K γ,best = min i =0 ,...,K Φ γ ( x i ) and ˆ Φ K,j γ,best = min i = j,...,K Φ γ ( x i ) for all 0 ≤ j ≤ K . We claim that the sequence generated by the subgradient method satisfies

Specifically, from Lemma 8.24 in Beck [2017], for all 0 ≤ j ≤ K , we have

Define ⌊·⌋ and ⌈·⌉ as rounding up and rounding down, respectively. Let j = ⌊ K 2 ⌋ in (36), by the definition of step-size η k = R l γ √ k +1 , we have

From the fact that Φ K γ,best ≤ ˆ Φ K,j γ,best , The desired result of (35) follows.

where the second inequality follows from that ∑ K k = ⌊ K 2 ⌋ 1 k +1 ≤ ∫ K ⌈ K 2 ⌉-1 1 s +1 ds ≤ 2 log 2 and ∑ K k = ⌊ K 2 ⌋ 1 √ k +1 ≥ ∫ K +1 ⌈ K 2 ⌉ 1 √ s +1 ds ≥ 1 2 √ K +2 .

Then, inequality (35) demonstrates that the number of iterations to obtain an ϵ -optimal solution for problem (P γ ) is

· Case of α > 1 . we have γ = γ ∗ +2 l β f 2 ϵ 1 -β and γ ∗ = ρl α f 2 ( α -1) α -1 α -α ϵ 1 -α . If β < α , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = α , we have γ = ( ρ ( α -1) α -1 α -α +2 ) l α F ϵ 1 -α . Then, the number of iterations is

If β > α , the dominating term in γ is 2 l β F ϵ 1 -β . Then, the number of iterations is

Case of α = 1 . we have γ = γ ∗ + l β f 2 ϵ 1 -β and γ ∗ = ρl f 2 .

If β < 1 , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = 1 , we have γ = ( ρ +1) l F ϵ 1 -α . Then, the number of iterations is

If β > 1 , the dominating term in γ is l β f 2 ϵ 1 -β . Then, the number of iterations is

Combining the above results, we conclude that

E.11 Proof of Theorem 3.10

Proof. Denote l γ = l f 2 + γl g 2 , define Φ K γ,best = min i =0 ,...,K Φ γ ( x i ) . From Theorem 8.31 in Beck [2017], the sequence generated by the subgradient method satisfies

This demonstrates that the number of iterations to obtain an ϵ -optimal solution for problem (P γ ) is

· Case of α > 1 . we have γ = γ ∗ +2 l β f 2 ϵ 1 -β and γ ∗ = ρl α f 2 ( α -1) α -1 α -α ϵ 1 -α . If β < α , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = α , we have γ = ( ρ ( α -1) α -1 α -α +2 ) l α F ϵ 1 -α . Then, the number of iterations is

If β > α , the dominating term in γ is 2 l β F ϵ 1 -β . Then, the number of iterations is

Case of α > 1 . we have γ = γ ∗ + l β f 2 ϵ 1 -β and γ ∗ = ρl f 2 .

If β < 1 , the dominating term in γ is γ ∗ . Then, the number of iterations is

If β = 1 , we have γ = ( ρ +1) l F ϵ 1 -α . Then, the number of iterations is

If β > 1 , the dominating term in γ is l β f 2 ϵ 1 -β . Then, the number of iterations is

Combining the above results, we conclude that

F Implementation details

In this section, we provide supplementary experiment settings and results. Specifically, in Appendix F.1, we present the detailed experimental settings, and in Appendix F.2, we provide the detailed experimental results. Additionally, in Appendix F.3 and F.4, we conduct experiments with different values of penalty parameter γ and solution accuracy ϵ , respectively.

F.1 Experiment setting

All simulations are implemented using MATLAB R2023a on a PC running Windows 11 with an AMD (R) Ryzen (TM) R7-7840H CPU (3.80GHz) and 16GB RAM.

F.1.1 Experiment setting of Section 4.1

We conduct the first experiment using the a1a.t data from LIBSVM datasets 5 . This data consists of 30 , 956 instances, each with n = 123 features. For this experiment, a sample of 1 , 000 instances is taken from the data, denoted as A . The corresponding labels for these instances are denoted as b , where each label b i is either -1 or 1 , corresponding to the i -th instance a i .

The Greedy FISTA algorithm [Liang et al., 2022] is used as a benchmark to compute G ∗ . To compute the proximal mapping of f 2 ( x )+ γg 2 ( x ) in problem (P γ ), i.e, projection onto a 1 -norm ball, we utilize the method proposed in Duchi et al. [2008], which performs exact projection in O ( n ) expected time, where n is the dimension of x .

For the PB-APG and PB-APG-sc algorithms, we set the value of γ = 10 5 , and we terminate the algorithms when ∥ x k +1 -x k ∥ ≤ 10 -10 . For the aPB-APG and aPB-APG-sc algorithms, we set γ 0 = 1 2 5 , ν = 20 , η = 10 , and ϵ 0 = 10 -6 . The iterations of these two algorithms continue until ϵ k reaches 10 -10 (meanwhile, γ = 10 5 ).

We compare our methods with MNG, BiG-SAM, DBGD, a-IRG, CG-BiO, Bi-SG, and R-APM in this experiment. Specifically, for R-APM [Samadi et al., 2023], the regularization parameter η is set to η = 1 /γ , reflecting the equivalence of the penalty formulation (P γ ) to (PReg), with σ = 1 /γ , as previously discussed.

We note that the termination criterion ∥ x k +1 -x k ∥ ≤ 10 -10 used in our experiments is different from the one proposed in our algorithms since the parameters required for the latter are not easily measurable. Nevertheless, this termination criterion is also widely used in the literature, as it corresponds to a gradient mapping [Beck, 2017, Nesterov, 2018, Davis and Drusvyatskiy, 2019]. Furthermore, Theorem 3.5 of Drusvyatskiy and Lewis [2018b] implies that ∥ x k +1 -x k ∥ also measures the distance to the optimal solution set.

F.1.2 Experiment setting of Section 4.2

In the second experiment, we address the problem of least squares regression using the YearPredictionMSD data from the UCI Machine Learning Repository 6 . This data consists of 515 , 345 songs with release years ranging from 1992 to 2011 . Each song has 90 features, and the corresponding release year is used as the label.

For this experiment, a sample of m = 1 , 000 songs is taken from the data, and the feature matrix and release years vector are denoted as A and b , respectively.

Following Section 5.2 in Merchav and Sabach [2023], we apply the min-max scaling technique to normalize the feature matrix A . Additionally, we add an intercept term and 90 collinear features to A such that the resulting matrix A T A becomes positive semi-definite, which implies that the feasible set X opt is not a singleton.

We compare our methods with a-IRG, BiG-SAM, and Bi-SG in this experiment. Specifically, for BiG-SAM [Sabach and Shtern, 2017], we consider the accuracy parameter δ for the Moreau envelope with two values, namely δ = 1 and δ = 0 . 01 .

To benchmark the performance, we utilize the MATLAB function lsqminnorm to compute G ∗ . Moreover, we follow the parameter settings outlined in Section 4.1.

F.2 Detailed results of experiments

To approximate the optimal value F ∗ , we use the MATLAB function fmincon to solve a relaxed version of the function-value-based reformulations in equation (P Val ). In this relaxed version, we replace the constraint in (P Val ) with G ( x ) -G ∗ ≤ ε , where ε = 10 -10 . This allows us to obtain an approximation of the optimal value while allowing for a small deviation from the true optimal value G ∗ .

We gather the total number of iterations for our methods, as well as the lower- and upper-level objective values and the optimal gaps for all the methods, in Table 3. Subsequently, we compare the optimal gaps of all methods, which are defined as G ( x ) -G ∗ and F ( x ) -F ∗ for the lower- and upper-level optimal gaps, respectively.

Table 3 reveals that for the logistic regression problem (5), our PB-APG, aPB-APG, PB-APG-sc, and aPB-APGsc exhibit almost identical function values for both objectives, surpassing other methods in terms of optimal gaps for the lower- and upper-level objectives (measured by the numerical value of the upper-level objective). In the case of the least squares regression problem (6), aPB-APG achieves the smallest optimal gaps for both objectives, followed by PB-APG and PB-APG-sc. These results demonstrate that our methods, despite yielding larger upper-level function values, generate solutions that are significantly closer to the optimal solution, as depicted in Figure 1. Additionally, for the problem in (5), both aPB-APG and aPB-APG-sc require fewer iterations than PB-APG and PB-APG-sc, respectively. This can be attributed to the warm-start mechanism employed in aPB-APG and aPB-APG-sc. Moreover, for the problem in (6), both aPB-APG and aPB-APG-sc require more iterations than PB-APG and PB-APG-sc, respectively. However, they exhibit staircase-shaped curves, which avoid the unwanted oscillations in PB-APG and PB-APG-sc, we have a similar observation in Figure 2.

F.3 Supplementary experiments for different penalty parameters

In this section, we investigate the impact of different values of penalty parameter γ on the experimental results of problems (5) and (6). We set γ to be either 2 × 10 4 or 5 × 10 5 for PB-APG and PB-APG-sc, and choose the

corresponding γ 0 values as 0 . 2 2 5 or 5 2 5 for aPB-APG and aPB-APG-sc, respectively. The remaining settings are the same as in Section 4.

We plot the values of the residuals of the lower-level objective G ( x k ) -G ∗ and the upper-level objective over time in Figures 3 and 4. Additionally, we also collect the total number of iterations, the lower- and upper-level objective values, and the optimal gaps of our methods in Table 4 for problems (5) and (6) with different values of γ .

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10

-8

10

-6

10

-4

10

-2

10

0

10

2

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0

5

10

15

20

0

0.2

0.4

0.6

0.8

1

10

-10

10

-5

10

0

0

0.2

0.4

0.6

0.8

1

0

5

10

15

20

0

5

10

15

20

25

30

10

-6

10

-4

10

-2

10

0

10

2

0

5

10

15

20

25

30

0

20

40

60

80

100

0

20

40

60

80

100

10

-10

10

-5

10

0

0

20

40

60

80

100

0

20

40

60

80

100

As Figures 3 and 4 show, our methods consistently outperform the other methods for both the lower- and upperlevel objectives, irrespective of the penalty parameter γ , since our methods achieve lower optimal gaps and desired function values for the lower- and upper-level objectives, respectively. The only exception is problem (6) with γ = 5 × 10 5 , as the third subfigure of Figure 4 shows, since we do not set the solution accuracy of BiG-SAM ( δ = 0 . 01 ), it attains a lower optimal gap than our PB-APG-sc and aPB-APG-sc for the lower-level objective. However, BiG-SAM ( δ = 0 . 01 ) produces significantly worse upper-level objective values, which are much larger than the objective values of our methods.

Tables 3, 4, and 5 reveal that the number of iterations for our methods increases as penalty parameter γ increases. However, it is worth noting that the accuracy of the obtained solutions also increases, as indicated by the decreasing optimal gaps of the lower- and upper-level objectives. This observation confirms that the complexity results and solution accuracies of our methods are indeed dependent on the choice of penalty parameters, specifically, L γ , as demonstrated in corresponding Theorem 3.3 and other related theorems.

F.4 Supplementary experiments for different solution accuracies

In this section, we investigate the impact of different solution accuracies on the experimental results of problems (5) and (6). We set ϵ to be either 10 -4 or 10 -7 and terminate the algorithms for PB-APG and PB-APG-sc when ∥ x k +1 -x k ∥ ≤ ϵ . For aPB-APG and aPB-APG-sc, we choose the corresponding ϵ 0 values as 1 or 10 -3 . The remaining settings are the same as in Section 4.

We also plot the values of the residuals of the lower-level objective G ( x k ) -G ∗ and the upper-level objective over time in Figures 5 and 6. Additionally, we also collect the total number of iterations, the lower- and upperlevel objective values, and the optimal gaps of our methods in Table 6 for problems (5) and (6) with different solution accuracies.

0

0.05

0.1

0.15

0.2

10

-10

10

-5

10

0

0

0.05

0.1

0.15

0.2

0

5

10

15

20

0

0.05

0.1

0.15

0.2

0.25

0.3

10

-10

10

-5

10

0

0

0.05

0.1

0.15

0.2

0.25

0.3

0

5

10

15

20

0

0.05

0.1

0.15

0.2

0.25

0.3

10

-4

10

-2

10

0

10

2

0

0.05

0.1

0.15

0.2

0.25

0.3

0

20

40

60

80

100

(ax)

102

109

PB-APG-

aPB-APG-sc

PB-APG

aPB-APG

BiG-SAM(6 = 1)

BiG-SAM(# = 0.01)

aIRG

Bi-SG

100

60

80

PB-APG-c

aPB-APG-sc

PB-APG

aPB-APG

BiG-SAM(6 = 1)

BIG-SAM(6 = 0.01)

aIRG

Bi-SG

10~

40

20

0

4

time (s)

6

8

10

12

2

4

time (s)

b

8

10

12

From Figures 5 and 6, it is evident that in most cases, our methods outperform the other methods in terms of both the lower- and upper-level objectives. However, there is an exception in the case of the upper-level objective for problem (6) when ϵ = 10 -4 . As illustrated in the second subfigure in Figure 6, our methods exhibit larger function values for the upper-level objective compared to the other methods (except BiG-SAM ( δ = 0 . 01 )), despite still achieving smaller optimal gaps for the lower-level objective. This discrepancy actually indicates that our methods have not yet achieved the desired accuracy when ϵ = 10 -4 , and it is important to note that ∥ x k +1 -x k ∥ ≤ ϵ is not the termination criterion in our proposed algorithms, as explained in Appendix F.1. Therefore, the larger optimality gaps for the upper-level objective in this case may be attributed to the termination criterion.

Tables 3, 6, and 7 demonstrate that the number of iterations for our methods also increases with the solution accuracy, while the optimal gaps of the lower- and upper-level objectives decrease correspondingly. This finding confirms that the number of iterations and the optimal gaps are influenced by the solution accuracy, as illustrated in the expressions for the number of iterations provided by Theorem 3.3 and other related theorems.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the assumptions adopted in this paper (e.g. Assumptions 2.1, 2.2, and 3.2), our study is based on these assumptions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to the theorems and the proofs of them in this paper, please refer to Appendix E. For example, Theorem 3.3 and its proof in Appendix E.7.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 4 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to Section 4 and Appendix F and the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The error bars are not applicable in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section 4 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Please refer to Section 4 and Appendix F.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

Werecognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the creators or original owners of assets are properly credited, and the license and terms of use are explicitly mentioned and properly respected, please refer to Section 4 and Appendix F.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please refer to the supplemental materials and the 'README.m' file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article presents a nice extension to the broad class of penalty method problems, particularly for the case when the objective functions have a Hölderian error bound. This assumption for this class of problems appears new, and a thorough coverage of interesting results are claimed.\n\nSTRENGTHS:\n- New results presented for bilevel optimization for the Hölderian error bound setting\n- New relationships between the minimizers of a penalty problem and the exact problem; this is a particularly mature field, and finding new results here is meritorious.\n- Presentation of a standard prox-based method for solving these bilevel problems with their new theory, with complexity details explained.\n\nWEAKNESSES:\n- The article does not distinguish between what type of subdifferential is used. Since the authors also make a mention of the applicability of their results to nonconvex objectives, this is crucial point. Many classical results in optimization and convex analysis are proven using the *convex* subdifferential; however, even for differentiable nonconvex functions (e.g., $-x^2$), the convex subdifferential is empty, while other notions like the Clarke subdifferential is nonempty.\n- Assumption 3.2 is a very strong one; while I agree with the author(s) that the prox of a sum is widely studied, the list of settings in which one can compute this is still quite limited. However, this is not a major drawback for me, since handling HEB bilevel optimization even in the L-smooth setting appears to be new.\n- In my opinion, leaving lines 140--142 to stake claim to results for non-convex which are exclusively in the appendix seems improper. The appendix is technically \"unreviewed\" material; if the formal statement of the result cannot fit into this article, then I do not think it is worth including. I suggest that the authors either try to find a way to incorporate these results into the article (since they are indeed quite interesting), or save the results for another work.\n- The graphs of the numeric are too small for me to read, especially in-print.\n\nMinor comments:\n- The fact that the argmin of  a convex lsc function is closed and convex is a classical result which has been known for decades, so it seems odd to cite [Beck and Sabach, 2014] instead of a classical book, or the article where it was first proven.\n- The article makes mention to functions $f_2$, $g_2$ far before they are introduced, which is confusiung to the reader.\n- Names \"Pock, Tseng, etc.\" are not capitalized in the citations.\n- Citation on lines 391-392 is missing volume number.\n\nQUESTIONS:\nThe classical article on penalty methods,\n\nDmitri P. Bertsekas, \"Necessary and Sufficient conditions for a penalty method to be exact\", Math. Program., vol. 9, pp. 87 - 99, 1975.\n\nbasically states that, if the penalty function is smooth near the solution set, then the penalty must go to infinity in order to achieve exact penalization; and, if the penalty function is nonsmooth near the solution set, then a finite penalty will suffice. My question is: are your results consistent with this article? In particular,\n  - In the setting where $G$ is smooth, can you confirm that $\\lim_{\\varepsilon\\to 0}\\gamma^*\\to\\infty$?\n  - On the other hand, if $G$ is nonsmooth at its solution set, does $\\lim_{\\varepsilon \\to 0} \\gamma^*$ approach a finite value? Violating this second requirement would not contradict the aforementioned article, but it would significantly strengthen your results.\n\nLIMITATIONS:\nThe authors do a good job of explaining the required assumptions for their results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with a simple bilevel (the lower-level problem has no dependence on the upper-level variable) optimization problem, where both the upper- and the lower-level objectives are convex and potentially non-smooth. To simplify the original bilevel problem the authors consider a penalty-based single-level reformulation and establish the relationship between the solutions of the two problems. An accelerated proximal gradient method (PB-APG)  is proposed to solve the penalty problem. Under a Holderian error bound condition and certain additional assumptions about the convexity and smoothness of the objectives, convergence of the PB-APG is shown to approximate global optimal solutions of the original bilevel problem.\n\nSTRENGTHS:\n* Differently from most of the works in literature, this paper deals with a problem in which the global solutions of the original bilevel problem are attainable. As a result, from a theoretical perspective, this is an interesting problem class. In addition, the proposed method is shown to (approximately) converge to these global solutions (of the original bilevel problem) and not to the solutions of some reformulation, as it is usually the case in other bilevel works.  \n* The authors provide a comprehensive and detailed summary of algorithms developed for solving simple bilevel optimization problems (in table 1).\n\nWEAKNESSES:\n* From an applications perspective, the problem class considered here does not seem very interesting. The bilevel problem is simple, the objectives are (strongly) convex, a Holderian error bound holds and the non-smooth term is required to be proximal-friendly. This significantly restricts the number and complexity of potential applications. Indeed, the applications presented in the appendix seem to revolve around (regularized) linear least-squares problems and simple learning models like logistic regression.\n* The proposed algorithm solves the reformulated minimization problem $P_{\\Phi}$ rather than the original bilevel. This minimization problem looks fairly standard, and it seems that known algorithms are applied directly for its solution. Thus, it appears that there is no significant novelty in terms of algorithm design.\n\nQUESTIONS:\n* Are Holderian error bounds satisfied in problems involving training of complex learning models, like neural networks.\n* In table 1 some of the algorithms depend on parameters $\\alpha$ and b, but the property to which these parameters correspond to is not specified. Please clarify this.\n\nLIMITATIONS:\nNo negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a penalty based algorithm for simple bilevel optimization problems. The paper studies the relationship between the solutions of the penalized problem and the original bilevel problem. It extends the existing results on general bilevel optimization problem, which are established under PL condition, to the more generic Hölderian error bounds. The results are also extended to non-smooth upper and lower-level functions.\n\nSTRENGTHS:\n1) The paper is well written and easy to follow. \n2) The generalization of the existing results on PL condition to Hölderian error bound is interesting.\n3) The extended results on non-smooth objectives are good contributions.\n\nWEAKNESSES:\n1. Missing literature comparison/review:  The method proposed in this paper is a penalty-based method. There is a large body of work for penalty-based bilevel optimization methods. While in the related work section in the main text, there is no detailed review/comparison of the previous penalty-based bilevel optimization algorithms. It might be beneficial if the authors could discuss about the advantage of the proposed method compared with the existing ones.\n\n2. In the current state of the paper, the experiments seem to be purely synthetic and might be a little too toy.  Though the paper is majorly a theoretical one, it would benefit to have more practical experiments. It is also of great importance to find meaningful applications for the simple bilevel optimization problem.\n\n3. The method proposed in this work falls into the penalty-based methods. It might be interesting if the authors include a comparison with existing penalty-based bilevel optimization methods in, e.g.,\n\nQUESTIONS:\n1. The proposed method is for convex simple bilevel optimization problems, which might be restrictive. Can the author explain how the algorithm can be extended to the non-convex case?\n\n2. How does this algorithm compare to the penalty-based algorithms for general bilevel optimization problems? Are there any advantages demonstrated by this algorithm as compared to the generic ones?\n\nLIMITATIONS:\nno potential social impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose algorithms to solve simple bilevel optimization problems of the form $\\min_x F(x)$ s.t. $x$ minimizes $G$, in the case where $F$ and $G$ are composite convex functions (i.e. some of 2 convex functions, one of which is also smooth), and assuming Lipschitz continuity of $F$ on the set of minimizers of $G$ and $G$ verifying a Holder error bound.\n\nThe proposed algorithms actually tackle a simple minimization problem, solving a relaxed version of the original problem that authors previously proved to be equivalent.\n\nSTRENGTHS:\nThe link between the solutions of the original problem and the ones of the relaxed one is interesting and allows considering a simpler problem to tackle.\n\nWEAKNESSES:\nI am not very familiar with this specific piece of literature, so I cannot pretend to be sure of the novelty of the results.\nIn particular, in line 47, the authors say « Tikhonov regularization most related to $P_\\gamma$ », I do not see any difference between the two formulations. Can the authors develop on this? What is exactly done in Tikhonov's paper? And what is new here?\n\n\nMinors:\n- Usually, a proximal-based algorithm can be applied to the particular case where the non-smooth part of the problem is null, then the prox map is simply the identity map, and all the guarantees hold. But in this paper's setting, the non-smooth part is essential and cannot be considered null as a particular case. Indeed, otherwise, $G$ would be smooth, therefore upper bounded by a quadratic function, and $G$ is also assumed to grow faster than $||x-\\bar{x}||^{alpha}$ which cannot happen (if alpha<2).\n- A natural oracle assumption in the case where $F$ and $G$ contain each a non-smooth part would be to assume access to the proximal map associated with each. However, authors assume access to the proximal of all the linear combinations of the 2 non-smooth parts, and this choice seems to be only motivated by the reformulation of the problem. Doing this makes the derivation of PB-APG trivial once Thm 2.5. has been proven. But how could we tackle the same problem with the sole knowledge of the 2 prox maps independently?\n\nQUESTIONS:\n- l.32: Why exclude the particular case where $X_{opt}$ is a singleton? This assumption is not used, therefore if we do not know in advance whether $X_{opt}$ is a singleton or not, one can still run this paper's algorithms, and guarantees will follow.\n- l.39: Even if what $G^*$ is seems clear, I think it is worth mentioning it before using it in line 35.\n- l.50: What about the upper problem?\n- l.50: « where b in (0, 0.5) ». The authors should be more specific here. Is it true for one specific b in the interval, depending on some parameters of the problem? Is it true for any (one proposed algorithm for each value of b)?\n- l.55: same remark\n- l.61: It would be more convenient if the authors keep a consistent way of displaying complexity.\n- l.68: What is g_2? Not defined yet at this stage. Moreover, note that the authors could not look at what happens in their case when $g_2=0$, as discussed in the \"weakness\" section.\n- l.101: What is f_2? Again not defined. And I think here that the authors meant $F$.\n- l.193: typo: « an » is repeated.\n- l.205: « while samadi … » If they consider the smooth case only, studying $\\alpha<2$ would be pointless because it is empty.\n- It would be great if authors could add explicit stopping criteria in their algorithms. Indeed, if $F^*$ and $G^*$ are not known, I guess we stop PB-APG after K iterations where K is set according to Th3.3. But then, PB-APG is not only a function of $\\phi$, $\\psi$, $L_{f_1}$, $L_{g_1}, x_0, \\epsilon$, but also $\\ell_F$, $\\alpha$ and $\\beta$. Which is important to actually understand the iterates of aPB-APG.\n\nLIMITATIONS:\nNo limitation", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}