Title: Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bilevel optimization

stochastic optimization
Abstract: In this paper, we study a class of stochastic bilevel optimization problems, also known as stochastic simple bilevel optimization, where we minimize a smooth stochastic objective function over the optimal solution set of another stochastic convex optimization problem. We introduce novel stochastic bilevel optimization methods that locally approximate the solution set of the lower-level problem via a stochastic cutting plane, and then run a conditional gradient update with variance reduction techniques to control the error induced by using stochastic gradients. For the case that the upper-level function is convex, our method requires $\mathcal{O}(\max\\{1/\epsilon_f^{2},1/\epsilon_g^{2}\\}) $ stochastic oracle queries to obtain a solution that is $\epsilon_f$-optimal for the upper-level and $\epsilon_g$-optimal for the lower-level. This guarantee improves the previous best-known complexity of $\mathcal{O}(\max\\{1/\epsilon_f^{4},1/\epsilon_g^{4}\\})$. Moreover, for the case that the upper-level function is non-convex, our method requires at most $\mathcal{O}(\max\\{1/\epsilon_f^{3},1/\epsilon_g^{3}\\}) $ stochastic oracle queries to find an $(\epsilon_f, \epsilon_g)$-stationary point. In the finite-sum setting, we show that the number of stochastic oracle calls required by our method are  $\mathcal{O}(\sqrt{n}/\epsilon)$ and $\mathcal{O}(\sqrt{n}/\epsilon^{2})$ for the convex and non-convex settings, respectively, where $\epsilon=\min \\{\epsilon_f,\epsilon_g\\}$.

Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem

Abstract

In this paper, we study a class of stochastic bilevel optimization problems, also known as stochastic simple bilevel optimization, where we minimize a smooth stochastic objective function over the optimal solution set of another stochastic convex optimization problem. We introduce novel stochastic bilevel optimization methods that locally approximate the solution set of the lower-level problem via a stochastic cutting plane, and then run a conditional gradient update with variance reduction techniques to control the error induced by using stochastic gradients. For the case that the upper-level function is convex, our method requires ˜ O (max { 1 /ϵ 2 f , 1 /ϵ 2 g } ) stochastic oracle queries to obtain a solution that is ϵ f -optimal for the upper-level and ϵ g -optimal for the lower-level. This guarantee improves the previous best-known complexity of O (max { 1 /ϵ 4 f , 1 /ϵ 4 g } ) . Moreover, for the case that the upper-level function is non-convex, our method requires at most ˜ O (max { 1 /ϵ 3 f , 1 /ϵ 3 g } ) stochastic oracle queries to find an ( ϵ f , ϵ g ) -stationary point. In the finite-sum setting, we show that the number of stochastic oracle calls required by our method are ˜ O ( √ n/ϵ ) and ˜ O ( √ n/ϵ 2 ) for the convex and non-convex settings, respectively, where ϵ = min { ϵ f , ϵ g } .

1 Introduction

An important class of bilevel optimization problems is simple bilevel optimization in which we aim to minimize an upper-level objective function over the solution set of a lower-level problem [1-4]. Recently this class of problems has attracted great attention in machine learning society due to their applications in continual learning [5], hyper-parameter optimization [6, 7], meta-learning [8, 9], and reinforcement learning [10, 11]. Motivated by large-scale learning problems, in this paper, we are particularly interested in the stochastic variant of the simple bilevel optimization where the upper and lower-level objective functions are the expectations of some random functions with unknown distributions and are accessible only through their samples. Hence, the computation of the objective function values or their gradients is not computationally tractable. Specifically, we focus on the stochastic simple bilevel problem defined as

where Z is compact and convex and ˜ f, ˜ g : R d → R are continuously differentiable functions on an open set containing Z , and θ and ξ are some independent random variables drawn from some possibly unknown probability distributions. As a result, the functions f, g : R d → R are also continuously differentiable functions on Z . We assume that g is convex but not necessarily strongly convex, and hence the solution set of the lower-level problem in (1) is in general not a singleton. We also study the finite sum version of the above problem where both functions can be written as the average of n component functions, i.e., f ( x ) = (1 /n ) ∑ n i =1 ˜ f ( x , θ i ) and g ( z ) = (1 /n ) ∑ n i =1 ˜ g ( z , ξ i ) .

The main challenge in solving problem (1), which is inherited from its deterministic variant, is the absence of access to the feasible set, i.e., the lower-level solution set. This issue eliminates the possibility of using any projection-based or projection-free methods. There have been some efforts to overcome this issue in the deterministic setting (where access to f and g and their gradients is possible), including [12-15], however, there is little done on the stochastic setting described above. In fact, the only work that addresses the stochastic problem in (1) is [16], where the authors present an iterative regularization-based stochastic extra gradient algorithm and show that it requires O (1 /ϵ 4 f ) and O (1 /ϵ 4 g ) queries to the stochastic gradient of the upper-level and lower-level function, respectively, to obtain a solution that is ϵ f -optimal for the upper-level and ϵ g -optimal for the lower-level. We improve these bounds and also extend our results to nonconvex settings.

Contributions. In this paper, we present novel projection-free stochastic bilevel optimization methods with tight non-asymptotic guarantees for both upper and lower-level problems. At each iteration, the algorithms use a small number of samples to build unbiased and low variance estimates and construct a cutting plane to locally approximate the solution set of the lower-level problem and then combine it with a Frank-Wolfe-type update on the upper-level objective. Our methods require careful construction of the cutting plane so that with high probability it contains the solution set of the lower-level problem, which is obtained by selecting proper function and gradient estimators to achieve the obtained optimal convergence guarantees. Next, we summarize our main theoretical results for the proposed Stochastic Bilevel Conditional Gradient methods for Infinite and Finite sample settings denoted by SBCGI and SBCGF, respectively.

(Stochastic setting) We show that SBCGI (Algorithm 1), in the convex setting, finds a solution ˆ x that satisfies f (ˆ x ) -f ∗ ≤ ϵ f and g (ˆ x ) -g ∗ ≤ ϵ g with probability 1 -δ within O (log( d/δϵ ) /ϵ 2 ) stochastic oracle queries, where ϵ = min { ϵ f , ϵ g } , f ∗ is the optimal value of problem (1) and g ∗ is the optimal value of the lower-level problem. Moreover, in the non-convex setting, it finds ˆ x satisfying G (ˆ x ) ≤ ϵ f and g (ˆ x ) -g ∗ ≤ ϵ g with probability 1 -δ within O ((log( d/δϵ )) 3 / 2 /ϵ 3 ) stochastic oracle queries, where G (ˆ x ) is the Frank-Wolfe (FW) gap.

(Finite-sum setting) We show that SBCGF (Algorithm 2), in the convex setting, finds ˆ x that satisfies f (ˆ x ) -f ∗ ≤ ϵ f and g (ˆ x ) -g ∗ ≤ ϵ g with probability 1 -δ within O ( √ n (log(1 /δϵ )) 3 / 2 /ϵ ) stochastic oracle queries, where n is the number of samples of finite-sum problem. Moreover, in the nonconvex setting, it finds ˆ x that satisfies G (ˆ x ) ≤ ϵ f and g (ˆ x ) -g ∗ ≤ ϵ g with probability 1 -δ within O ( √ n log(1 /δϵ ) /ϵ 2 ) stochastic oracle queries.

1.1 Related work

General stochastic bilevel. In a general format of stochastic bilevel problems, the upper-level function f also depends on an extra variable y ∈ R p which also affects the lower-level objective,

There have been several works including [10, 17-21] on solving the general stochastic bilevel problem (2). However, they only focus on the setting where the lower-level problem is strongly convex, i.e., g ( z , y ) is strongly convex with respect to z for any value of y . In fact, (2) with a convex lower-level problem is known to be NP-hard [22]. Hence, the results of these works are not directly comparable with our work as we focus on a simpler setting, but our assumption on the lower-level objective function is weaker and it only requires the function to be convex.

Deterministic simple bilevel. There have been some recent results on non-asymptotic guarantees for the deterministic variant of problem (1). The BiG-SAM algorithm was presented in [14], and it was shown that its lower-level objective error converges to zero at a rate of O (1 /t ) , while the upper-level error asymptotically converges to zero. In [15], the authors achieved the first non-asymptotic rate for both upper- and lower-level problems by introducing an iterative regularization-based method which achieves an ( ϵ f , ϵ g ) -optimal solution after O (max { 1 /ϵ 4 f , 1 /ϵ 4 g } ) iterations. In [12], the authors proposed a projection-free method for deterministic simple bilevel problems that has a complexity of O (max { 1 /ϵ f , 1 /ϵ g } ) for convex upper-level and complexity of O (max { 1 /ϵ 2 f , 1 / ( ϵ f ϵ g ) } ) for non-convex upper-level. Moreover, in [23] the authors presented a switching gradient method to solve simple bilevel problems with convex smooth functions for both upper- and lower-level problems with complexity O (1 /ϵ ) . However, all the above results are limited to the deterministic setting.

General bilevel without lower-level strong convexity. Recently, there are several recent works on general bilevel optimization problems without lower-level strong convexity including [24-27, 23]. However, they either have a weaker theoretical results like asymptotic convergence rate in [24] or have some additional assumptions. Specifically, in [25], the authors reformulated the problem as a constrained optimization problem and further assumes such problem to be a convex program. Moreover, in [26, 27], the authors in both papers assumed that the lower-level objective satisfies the PL inequality, while we assumed that the lower-level objective is convex. In [23], the authors used a looser convergence criterion that only guarantees convergence to a Goldstein stationary point. Since these works consider a more general class of problems, we argue that their theoretical results when applied to our setting are necessarily weaker.

2 Preliminaries

2.1 Motivating examples

Example 1: Over-parameterized regression. A general form of problem (1) is when the lower-level problem represents training loss and the upper-level represents test loss. The goal is to minimize the test loss by selecting one of the optimal solutions for the training loss [13]. An instance of that is the constrained regression problem, where we intend to find an optimal parameter vector β ∈ R d that minimizes the loss ℓ tr ( β ) over the training dataset D tr . To represent some prior knowledge, we usually constrain β to be in some subsets Z ⊆ R d , e.g., Z = { β | ∥ β ∥ 1 ≤ λ } for some λ > 0 to induce sparsity. To handle multiple global minima, we adopt the over-parameterized approach, where the number of samples is less than the parameters. Although achieving one of these global minima is possible, not all optimal solutions perform equally on other datasets. Hence, we introduce an upper-level objective: the loss on a validation set D val . This helps select a training loss optimizer that performs well on both training and validation sets. It leads to the following bilevel problem:

In this case, both the upper- and lower-level losses are smooth and convex if ℓ is smooth and convex.

Example 2: Dictionary learning. Problem (1) also appears in lifelong learning, where the learner takes a series of tasks sequentially and tries to accumulate knowledge from past tasks to improve performance in new tasks. Here we focus on continual dictionary learning. The aim of dictionary

learning is to obtain a compact representation of the input data. Let A = { a 1 , . . . , a n } ∈ R m × n denote a dataset of n points. We seek a dictionary D = [ d 1 , . . . , d p ] ∈ R m × p such that all data points a i can be represented by a linear combination of basis vectors in D which can be cast as [28-31]:

Moreover, we denote X = [ x 1 , . . . , x n ] ∈ R p × n as the coefficient matrix. In practice, data points usually arrive sequentially and the representation evolves gradually. Hence, the dictionary must be updated sequentially as well. Assume that we already have learned a dictionary ˆ D ∈ R m × p and the corresponding coefficient matrix ˆ X ∈ R p × n for the dataset A . As a new dataset A ′ = { a ′ 1 , . . . , a ′ n ′ } arrives, we intend to enrich our dictionary by learning ˜ D ∈ R m × q ( q > p ) and the coefficient matrix ˜ X ∈ R q × n ′ for the new dataset while maintaining good performance of ˜ D on the old dataset A as well as the learned coefficient matrix ˆ X . This leads to the following stochastic bilevel problem:

where f ( ˜ D , ˜ X ) ≜ 1 2 n ′ ∑ n ′ k =1 ∥ a ′ k -˜ D ˜ x k ∥ 2 2 represents the average reconstruction error on the new dataset A ′ , and g ( ˜ D ) ≜ 1 2 n ∑ n i =1 ∥ a i -˜ D ˆ x i ∥ 2 2 represents the error on the old dataset A . Note that we denote ˆ x i as the prolonged vector in R q by appending zeros at the end. In problem (5), the upper-level objective is non-convex, while the lower-level loss is convex with multiple minima.

2.2 Assumptions and definitions

Next, we formally state the assumptions required in this work.

Assumption 2.1. Z is convex and compact with diameter D , i.e., ∀ x , y ∈ Z , we have ∥ x -y ∥ ≤ D.

Assumption 2.2. The upper-level stochastic function ˜ f satisfies the following conditions:

(i) ∇ ˜ f is Lipschitz with constant L f , i.e., ∀ x , y ∈ Z , ∀ θ , ∥∇ ˜ f ( x , θ ) -∇ ˜ f ( y , θ ) ∥ ≤ L f ∥ x -y ∥ .

(ii) The stochastic gradients noise is sub-Gaussian, E [exp {∥∇ ˜ f ( x , θ ) -∇ f ( x ) ∥ 2 /σ 2 f } ] ≤ exp { 1 } .

Assumption 2.3. The lower-level stochastic function ˜ g satisfies the following conditions:

(i) g is convex and ∇ ˜ g is L g -Lipschitz, i.e., ∀ x , y ∈ Z , ∀ ξ , ∥∇ ˜ g ( x , ξ ) -∇ ˜ g ( y , ξ ) ∥ ≤ L g ∥ x -y ∥ .

(ii) The stochastic gradients noise is sub-Gaussian, E [exp {∥∇ ˜ g ( x , ξ ) -∇ g ( x ) ∥ 2 /σ 2 g } ] ≤ exp { 1 } .

(iii) The stochastic functions noise is sub-Gaussian, E [exp {| ˜ g ( x , ξ ) -g ( x ) | 2 /σ 2 l } ] ≤ exp { 1 } .

Remark 2.1 . Assumptions 2.1 and 2.3(i) imply that ˜ g is Lipschitz continuous on an open set containing Z with some constant L l , i.e., for all x , y ∈ Z , we have | ˜ g ( x ) -˜ g ( y ) | ≤ L l ∥ x -y ∥ .

In the paper, we denote g ∗ ≜ min z ∈Z g ( z ) and X ∗ g ≜ arg min z ∈Z g ( z ) as the optimal value and the optimal solution set of the lower-level problem, respectively. Note that by Assumption 2.3, the set X ∗ g is nonempty, convex, and compact, but typically not a singleton as g potentially has multiple minima on Z . Furthermore, we denote f ∗ and x ∗ as the optimal value and an optimal solution of problem (1), which are assured to exist since f is continuous and X ∗ g is compact.

Definition 2.1. When f is convex, a point ˆ x ∈ Z is ( ϵ f , ϵ g ) -optimal if f (ˆ x ) -f ∗ ≤ ϵ f and g (ˆ x ) -g ∗ ≤ ϵ g . When f is non-convex, ˆ x ∈ Z is ( ϵ f , ϵ g ) -optimal if G (ˆ x ) ≤ ϵ f and g (ˆ x ) -g ∗ ≤ ϵ g , where G (ˆ x ) is the FW gap [32, 33] defined as G (ˆ x ) ≜ max s ∈X ∗ g {⟨∇ f (ˆ x ) , ˆ x -s ⟩} .

3 Algorithms

Conditional gradient for simple bilevel optimization. A variant of the conditional gradient (CG) method for solving bilevel problems has been introduced in [12] which uses a cutting plane idea [34]

to approximate the solution set of the lower-level problem denoted by X ∗ g . More precisely, if one has access to X ∗ g , it is possible to run the FW update with stepsize γ t as

However, the set X ∗ g is not explicitly given and the above method is not implementable. In [12], the authors suggested the use of the following set: X t = { s ∈ Z : ⟨∇ g ( x t ) , s -x t ⟩ ≤ g ( x 0 ) -g ( x t ) } instead of the set X ∗ g in the FW update given in (6). Note that x 0 is selected in a way that g ( x 0 ) -g ∗ is smaller than ϵ g / 2 , and such a point can be efficiently computed. A crucial property of the above set is that it always contains the solution set of the lower-level problem denoted by X ∗ g . This can be easily verified by the fact that for any x ∗ g in X ∗ g we have x ∗ g ∈ Z and

where the second inequality holds as g ( x 0 ) ≥ g ( x ∗ g ) . As shown in [12], this condition is sufficient to show that if one follows the update in (6) with X t instead of X ∗ g , the iterates will converge to the optimal solution. However, this framework is not applicable to the stochastic setting as we cannot access the functions or their gradients. Next, we present our main idea to address this delicate issue.

Random set for the subproblem. A natural idea to address stochasticity is to replace all gradients and functions with their stochastic estimators for both the subproblem in (6), i.e., ∇ f ( x t ) , as well as the construction of the cutting plane X t , i.e., g ( x t ) , and ∇ g ( x t ) . However, this simple idea fails since the set X t may no longer contain the solution set X ∗ g . More precisely, if ˆ g t and ̂ ∇ g t are unbiased estimators of g ( x t ) and ∇ g ( x t ) , respectively, for the following approximation set

we can not argue that it contains X ∗ g , as the second inequality in (7) does not hold, i.e., ⟨ ̂ ∇ g ( x t ) , x ∗ g -x t ⟩ ≤ ˆ g ( x ∗ g ) -ˆ g ( x t ) ≰ g ( x 0 ) -ˆ g ( x t ) . In the appendix, we numerically illustrate this point.

To address this issue, we tune the cutting plane by only moving it but not rotating it, i.e., adding another term to tolerate the noise from stochastic estimates. We introduce the stochastic cutting plane

where ̂ ∇ g t and ˆ g t are gradient and function value estimators, respectively, that we formally define later. In the above expression, the addition of the term K t , which is a sequence of constants converging to zero as t → ∞ , allows us to ensure that with high probability the random set ˆ X t contains all optimal solutions of the lower-level problem. Choosing suitable values for the sequence K t is a crucial task. If we select a large value for K t then the probability of ˆ X t containing X ∗ g goes up at the price allowing points with larger values g in the set. As a result, once we perform an update similar to the one in (6), the lower-level function value could increase significantly. On the other hand, selecting small values for K t would allow us to show that the lower level objective function is not growing, while the probability of ˆ X t containing X ∗ g becomes smaller which could even lead to a case that the set becomes empty and the bilevel problem becomes infeasible.

Remark 3.1 . How to compute g ( x 0 ) ? In the finite sum setting, we can accurately compute g ( x 0 ) , and the additional cost of n function evaluations will be dominated by the overall complexity. In the stochastic setting, we could use a large batch of samples to compute g ( x 0 ) with high precision at the beginning of the process. This additional operation will not affect the overall sample complexity of the proposed method, as the additional cost is negligible compared to the overall sample complexity. Specifically, we need to take a batch size of b = ˜ O ( ϵ -2 ) to estimate ˆ g ( x 0 ) . Using the Hoeffding inequality for subgaussian random variables, we have the following bound: | ˆ g ( x 0 ) -g ( x 0 ) | ≤ √ 2 σ l ( T +1) -ω/ 2 √ log(2 /δ ) , with a probability of at least 1 -δ , where T is the maximum number of iterations. Comparing this with Lemma 4.1.3, we can further derive: | ˆ g ( x 0 ) -g ( x 0 ) | ≤ √ 2 σ l ( T + 1) -ω/ 2 √ log(2 /δ ) ≤ √ 2(2 L l D + 3 ω 3 ω -1 σ l )( t + 1) -ω/ 2 √ log(6 /δ ) , with a probability of at least 1 -δ for all 0 ≤ t ≤ T . Consequently, the introduced error term would be absorbed in K 0 ,t and will not affect any parts of the analysis.

Variance reduced estimators. As mentioned above, a key point in the design of our stochastic bilevel algorithms is to select K t properly such that ˆ X t contains X ∗ g with high probability, for all t ≥ 0 . To

Algorithm 1 SBCGI

achieve such a guarantee, we first need to characterize the error of our gradient and function value estimators. More precisely, suppose that for our function estimator we have that P ( | ˆ g t -g ( x t ) | ≤ K 0 ,t ) ≥ 1 -δ ′ and for the gradient estimator we have P ( ∥ ̂ ∇ g t - ∇ g ( x t ) ∥ ≤ K 1 ,t ) ≥ 1 -δ ′ , for some δ ′ ∈ (0 , 1) . Then, by setting K t = K 0 ,t + DK 1 ,t , we can guarantee that the conditions required for the inequalities in (7) hold with probability at least (1 -2 δ ′ ) .

Using simple sample average estimators would not allow for the selection of a diminishing K t , as the variance is not vanishing, but by using proper variance-reduced estimators the variance of the estimators vanishes over time and eventually, we can send K t to zero. In this section, we focus on two different variance reduction estimators. For the stochastic setting in (1) we use the STOchastic Recursive Momentum estimator (STORM), proposed in [35], and for the finite-sum setting, we utilize the Stochastic Path-Integrated Differential EstimatoR (SPIDER) proposed in [36]. If v t -1 is the gradient estimator of STORM at time t -1 , the next estimator is computed as

where ∇ ˜ f ( x , θ ) is the stochastic gradient evaluated at x with sample θ . The main advantage of the above estimator is that it can be implemented even with one sample per iteration. Unlike STORM, for the SPIDER estimator, we need a larger batch of samples per update. More precisely, if we consider v t -1 as the estimator of SPIDER for ∇ f ( x t ) , it is updated according to

where ∇ f S ( x ) = (1 /S ) ∑ i ∈S ∇ f ( x , θ i ) is the average sub-sampled stochastic gradient computed using samples that are in the set S . As we will discuss later, in the finite sum case that we use SPIDER, the size of batch S depends on n which is the number of component functions. We delay establishing a high probability error bound for these estimators to section 4.1.

3.1 Conditional gradient algorithms with random sets: stochastic and finite-sum

Next, we present our Stochastic Bilevel Conditional Gradient method for Infinite sample case abbreviated by SBCGI for solving (1) and its finite sum variant denoted by SBCGF. In both cases, we first find a point x 0 that satisfies g ( x 0 ) -g ∗ ≤ ϵ g / 2 , for some accuracy ϵ g . The cost of finding such a point is negligible compared to the cost of the main algorithm as we discuss later. At each iteration t , we first update the gradient estimator of the upper-level and the function and gradient estimators of the lower-level problem. In SBCGI, we follow the STORM idea as described in steps 4-6 of Algorithm 1, while in SBCGF, we use the SPIDER technique as presented in steps 7-10 of Algorithm 2.In the case of SBCGF, we need to compute the exact gradient and function values once every q iteration as presented in steps 4-6 of Algorithm 2. Once the estimators are updated, we can define the random set ˆ X t as in (9) and solve the following subproblem over the set ˆ X t ,

Algorithm 2 SBCGF

where ̂ ∇ f t is the unbiased estimator of ∇ f ( x t ) . Note that we implicitly assume that we have access to a linear optimization oracle that returns a solution of the subproblem in (12), which is standard for projection-free methods [32, 33]. In particular, if Z can be described by a system of linear inequalities, then problem (12) corresponds to a linear program and can be solved by a standard solver as we show in our experiments. Once, s t is calculated we simply update the iterate

with stepsize γ t +1 ∈ [0 , 1] . The only missing part for the implementation of our methods is the choice of K t in the random set and the stepsize parameters. We address these points in the next section.

Remark 3.2 . SBCGI can be implemented with a batch size as small as S = 1 . However, this does not imply that the batch size "has to be" S = 1 . In other words, the main advantage of SBCGI, compared to SBCGF, is its capability to be implemented with any mini-batch size, even as small as S = 1 . Therefore, for SBCGI, the batch size can be set arbitrarily, whereas for SBCGF, it must be √ n .

Remark 3.3 . In the finite-sum setting, if the numbers of functions in the upper- and lower-level losses are different, we could simply modify SBCGF 2 by choosing S u = q u = √ n u and S l = q l = √ n l , where n u and n l are the number of functions in the upper- and lower-level, respectively.

4 Convergence analysis

In this section, we characterize the sample complexity of our methods for stochastic and finite-sum settings. Before stating our results, we first characterize a high probability bound for the estimators of our algorithms, which are crucial in the selection of parameter K t and the overall sample complexity.

4.1 High probability bound for the error terms

To achieve a high probability bound, it is common to assume that the noise of gradient or function is uniformly bounded as in [36, 37], but such assumptions may not be realistic for most machine learning applications. Hence, in our analysis, we consider a milder assumption and assume the noise of function and gradient are sub-Gaussian as in Assumptions 2.2 and 2.3, respectively. Given these assumptions, we next establish a high probability error bound for the estimators in SBCGI.

Lemma 4.1. Consider SBCGI in Algorithm 1 with parameters α t = β t = ρ t = γ t = 1 / ( t +1) ω where ω ∈ (0 , 1] . If Assumptions 2.1, 2.2, and 2.3 are satisfied, for any t ≥ 1 and δ ∈ (0 , 1) , with probability at least 1 -δ , for some absolute constant c , ( d is the number of dimension), we have

Lemma 4.1 shows that for any ω ∈ (0 , 1] , if we set α t = β t = ρ t = γ t = 1 / ( t +1) ω , then with high probability the gradient and function approximation errors converge to zero at a sublinear rate of ˜ O (1 /t ω/ 2 ) . Moreover, the above result characterizes the choice of K t . Specifically, if we define K 1 ,t as the upper bound in (15) and K 0 ,t as the upper bound in (16), by setting K t = K 0 ,t + DK 1 ,t , then with probability (1 -δ ) the random set ˆ X t contains X ∗ g . Later, we show that ω = 1 leads to the best complexity bound for the convex setting and ω = 2 / 3 is the best choice for the nonconvex setting.

Next, we establish a similar result for the estimators in SBCGF.

Lemma 4.2. Consider SBCGF with stepsize γ and S = q = √ n . If Assumptions 2.1-2.3 hold, for any t ≥ 1 and δ ∈ (0 , 1) , with probability 1 -δ we have ∥ ̂ ∇ f t -∇ f ( x t ) ∥ ≤ 4 L g Dγ √ log(12 /δ ) , ∥ ̂ ∇ g t -∇ g ( x t ) ∥ ≤ 4 L g Dγ √ log(12 /δ ) , and | ˆ g t -g ( x t ) | ≤ 4 L l Dγ √ log(12 /δ ) .

Similarly, for SBCGF, we set K 1 ,t = 4 L g Dγ √ log(12 /δ ) and K 0 ,t = 4 L l Dγ √ log(12 /δ ) and choose K t = K 0 ,t + DK 1 ,t , then the random set ˆ X t contains X ∗ g with probability 1 -δ .

Next, we formalize our claim about the random set with the above choice of K t .

Lemma 4.3. If X ∗ g is the solution set of the lower-level problem and ˆ X t is the feasible set constructed by cutting plane at iteration t , then for any t ≥ 0 and δ ∈ (0 , 1) , we have P ( X ∗ g ⊆ ˆ X t ) ≥ 1 -δ .

This lemma shows all X ∗ g is a subset of the constructed feasible set ˆ X t with a high probability of 1 -δ . Indeed, using a union bound one can show that the above statement holds for all iterations up to time t with probability 1 -tδ .

4.2 Convergence and complexity results for the stochastic setting

Next, we characterize the iteration and sample complexity of the proposed method in SBCGI for the stochastic setting. First, we present the result for the case that f is convex.

Theorem 4.4 (Stochastic setting with convex upper-level) . Consider SBCGI in Algorithm 1 for solving problem (1) . Suppose Assumptions2.1, 2.2, and 2.3 hold and f is convex. If the stepsizes of SBCGI are selected as α t = β t = ρ t = γ t = ( t + 1) -1 , and the cutting plane parameter is K t = c ((2 L l D + 3 2 σ l ) √ 2 log(6 t/δ ) + D (2 L g D + 3 2 σ g ) √ 2 log(6 td/δ ))( t +1) -1 / 2 , then after T iterations,

with probability 1 -δ for some absolute constants C 1 and C 2 and ζ := √ log (6 Td/δ ) .

Theorem 4.4 shows a convergence rate of O ( √ log( Td/δ ) / √ T ) . As a corollary, SBCGI returns an ( ϵ f , ϵ g ) -optimal solution with probability 1 -δ after O (log( d/δϵ ) /ϵ 2 ) iterations, where ϵ = min { ϵ f , ϵ g } . Since we use one sample per iteration, the overall sample complexity is also O (log( d/δϵ ) /ϵ 2 ) . Note that the iteration complexity and sample complexity of our method outperform the ones in [16], as they require O (1 /ϵ 4 ) iterations and sample to achieve the same guarantee.

Remark 4.1 . The task of finding x 0 which is equivalent to a single-level stochastic optimization problem requires O (1 /ϵ 2 g ) iterations and samples. As a result, this additional cost does not affect the overall complexity of our method. The same argument also holds in the non-convex case.

Theorem 4.5 (Stochastic setting with non-convex upper level) . Consider SBCGI for solving problem (1) . Suppose Assumptions 2.1-2.3 hold, f is nonconvex, and define f = min x ∈Z f ( x ) . If the stepsizes of SBCGI are selected as α t = β t = ρ t = ( t +1) -2 / 3 , γ t = ( T +1) -2 / 3 , and the cutting plane parameter is K t = c ((2 L l D + 3 2 / 3 3 2 / 3 -1 σ l ) √ 2 log(6 T/δ )+ D (2 L g D + 3 2 / 3 3 2 / 3 -1 σ g ) √ 2 log(6 Td/δ ))( t + 1) -1 / 3 , then after T iterations, there exists an iterate x t ∗ in the set ∈ { x 0 , x 1 , . . . , x T -1 } for which

with probability 1 -δ for some absolute constants C 3 and C 4 and ζ := √ log (6 Td/δ ) .

As a corollary of Theorem 4.5, the number of iterations required to find an ( ϵ f , ϵ g )-optimal solution can be upper bounded by O (log( d/δϵ ) 3 / 2 /ϵ 3 ) , where ϵ = min { ϵ f , ϵ g } . We note that the dependence on the upper-level accuracy ϵ f also matches that in the standard CG method for a single-level nonconvex problem [33, 38]. Moreover, as we only need one stochastic oracle query per iteration, SBCGI only requires O (log ( d/δϵ ) 3 / 2 /ϵ 3 ) stochastic oracle queries to find an ( ϵ f , ϵ g )-optimal.

4.3 Convergence and complexity results for the finite-sum setting

Similarly, we present iteration and sample complexity for algorithm 2 under the finite-sum setting.

Theorem 4.6 (Finite-sum setting with convex upper-level) . Consider SBCGF presented in Algorithm 2 for solving the finite-sum version of (1) . Suppose Assumptions 2.1, 2.2, and 2.3 hold and f is convex. If we set the stepsizes of SBCGF as γ = log T/T , S = q = √ n , and the cutting plane parameter as K t = 4 D ( L l √ log(12 T/δ ) + L g D √ log(12 T/δ )) log T/T , then after T iterations,

with probability at least 1 -δ , for some absolute constant C 5 and C 6 , and ζ ′ = √ log(12 T/δ ) .

Theorem 4.6 implies the number of stochastic oracle queries is O (log(1 /δϵ ) 3 / 2 √ n/ϵ ) , where ϵ = min { ϵ f , ϵ g } , which matches the optimal sample complexity of single-level problems [39].

Theorem 4.7 (Finite-sum setting with non-convex upper-level) . Consider SBCGF presented in Algorithm 2 for solving the finite-sum version of (1) . Suppose Assumptions2.1-2.3 hold, and f is non-convex. Define f = min x ∈Z f ( x ) . If the parameters of SBCGF are selected as γ = 1 / √ T , S = q = √ n , and the cutting plane parameter is K t = 4 D ( L l √ log(12 T/δ ) + L g D √ log(12 T/δ )) / √ T , then after T iterations, there exists an iterate x t ∗ ∈ { x 0 , x 1 , . . . , x T -1 } for which,

with probability at least 1 -δ , for some absolute constants C 7 and C 8 , and ζ ′ = √ log(12 T/δ ) .

As a corollary of Theorem 4.7, the number of stochastic oracle queries is O (log(1 /δϵ ) √ n/ϵ 2 ) , where ϵ = min { ϵ f , ϵ g } , which matches the state-of-the-art single-level result O ( √ n/ϵ 2 ) in [40]. SBCGF also improves the number of linear minimization oracle queries of SBCGI from O (1 /ϵ 2 ) to O (1 /ϵ ) for convex upper-level and from O (1 /ϵ 3 ) to O (1 /ϵ 2 ) for non-convex upper-level.

5 Numerical experiments

In this section, we test our methods on two different stochastic bilevel optimization problems with real and synthetic datasets and compare them with other existing stochastic methods in [16] and [13].

Over-parameterized regression. We consider the bilevel problem corresponding to sparse linear regression introduced in (3). We apply the Wikipedia Math Essential dataset [30] which composes of a data matrix A ∈ R n × d with n = 1068 samples and d = 730 features and an output vector b ∈ R n . To ensure the problem is over-parameterized, we assign 1 / 3 of the dataset as the training set ( A tr , b tr ) , 1 / 3 as the validation set ( A val , b val ) and the remaining 1 / 3 as the test set ( A test , b test ) . For both upper- and lower-level loss functions we use the least squared loss, and we set λ = 10 . We compare the performance of our methods with the aR-IP-SeG method by [16] and the stochastic version of DBGD introduced by [13]. We employ CVX [41, 42] to solve the lower-level problem and the reformulation of the bilevel problem to obtain g ∗ and f ∗ , respectively. We also include the additional cost of finding x 0 in SBCGI and SBCGF in our comparisons.

In Figure 1(a)(b), we observe that SBCGF maintains a smaller lower-level gap than other methods and converges faster than the rest in terms of upper-level error. SBCGI has the second-best performance in terms of lower- and upper-level gaps, while aR-IP-SeG performs poorly in terms of both lower- and upper-level objectives. The performance of DBGD-sto for the upper-level objective is well, however, it underperforms in terms of lower-level error. In Figure 1(c), SBCGF, SBCGI, and DBGD-sto achieve almost equally small test errors, while aR-IP-SeG fails to achieve a low test error. Note

0

2

4

6

8

10

10 5

10 -6

10 -4

10 -2

10 0

(a) Lower-level gap

0

2

4

6

8

10

10 5

10 -4

10 -2

10 0

10 2

(b) Upper-level gap

0

2

4

6

8

10

10 5

10 -1

10 0

10 1

(c) Test error

0

1

2

3

4

5

6

7

10 5

10 -8

10 -6

10 -4

10 -2

10 0

(a) Lower-level gap

0

1

2

3

4

5

6

7

10 5

10 -2

10 -1

10 0

(b) Upper-level gap

0

1

2

3

4

5

6

7

10 5

0

0.2

0.4

0.6

0.8

1

(c) Recovery rate

that after the initial stage, SBCGI increases slightly in terms of all the performance criteria, because SBCGI (1) only takes one sample per iteration and uses a decreasing step-size while SBCGF takes √ n samples per iteration and uses a small constant stepsize, demonstrating a more robust performance.

Dictionary learning. To test our methods on problems with non-convex upper-level we consider problem (5) on a synthetic dataset with a similar setup to [12]. We first construct the true dictionary ˜ D ∗ ∈ R 25 × 50 comprising of 50 basis vectors in R 25 . All entries of these basis vectors are drawn from the standard Gaussian distribution and then normalized to have unit ℓ 2 -norm. We also generate two more dictionaries D ∗ and D ′∗ consisting of 40 and 20 basis vectors in ˜ D ∗ , respectively (thus they share at least 10 bases). These two datasets A = { a 1 , . . . , a 250 } and A ′ = { a ′ 1 , . . . , a ′ 250 } are constructed as a i = D ∗ x i + n i , for i = 1 , . . . , 250 , and a ′ k = D ′∗ x ′ k + n ′ k , for k = 1 , . . . , 250 , where { x i } 250 i =1 , { x ′ k } 250 k =1 are coefficient vectors and { n i } 250 i =1 , { n ′ k } 250 k =1 are random Gaussian noises. As neither A nor A ′ includes all the elements of ˜ D ∗ , it is important to renew our dictionary by using the new dataset A ′ while maintaining the knowledge from the old dataset A .

In our experiment, we initially solve the standard dictionary learning problem employing dataset A , achieving the initial dictionary ˆ D and coefficient vectors { ˆ x } 250 i =1 . We define the lower-level objective as the reconstruction error on A using { ˆ x } 250 i =1 , and the upper-level objective as the error on new dataset A ′ . We compare our algorithms with aR-IP-SeG and DBGD (stochastic version), measuring performance with the recovery rate of true basis vectors. Note that a basis vector ˜ d ∗ i in ˜ D ∗ is considered as successfully recovered if there exists ˜ d j in ˜ D such that |⟨ ˜ d ∗ i , ˜ d j ⟩| > 0 . 9 (for more details of the experiment setup see Appendix F). In Figure 2(a), we observe SBCGF converges faster than any other method regarding the lower-level objective. While SBCGI has the second-best performance in terms of the lower-level gap, aR-IP-SeG and DBGD-sto perform poorly compared with SBCGI and SBCGF. In Figures 2(b) and (c), we see that SBCGI, SBCGF, and DBGD-sto achieve good results in terms of the upper-level objective and the recovery rate. However, aR-IP-SeG still performs poorly in terms of both criteria, which matches the theoretical results in Table 1.

References

Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations research , 21(1):37-44, 1973.

Stephen Dempe, Nguyen Dinh, and Joydeep Dutta. Optimality conditions for a simple convex bilevel programming problem. Variational Analysis and Generalized Differentiation in Optimization and Control: In Honor of Boris S. Mordukhovich , pages 149-161, 2010.

Joydeep Dutta and Tanushree Pandit. Algorithms for simple bilevel programming. Bilevel Optimization: Advances and Next Challenges , pages 253-291, 2020.

Yekini Shehu, Phan Tu Vuong, and Alain Zemkoho. An inertial extrapolation method for convex simple bilevel optimization. Optimization Methods and Software , 36(1):1-19, 2021.

Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems , 33:1487914890, 2020.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , pages 1568-1577. PMLR, 2018.

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated backpropagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1723-1732. PMLR, 2019.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems , 32, 2019.

Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.08136 , 2018.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05170 , 2020.

Haifeng Zhang, Weizhe Chen, Zeren Huang, Minne Li, Yaodong Yang, Weinan Zhang, and Jun Wang. Bi-level actor-critic for multi-agent coordination. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 7325-7332, 2020.

Ruichen Jiang, Nazanin Abolfazli, Aryan Mokhtari, and Erfan Yazdandoost Hamedani. A conditional gradient-based method for simple bilevel optimization with convex lower-level problem. In International Conference on Artificial Intelligence and Statistics , pages 1030510323. PMLR, 2023.

Chengyue Gong and Xingchao Liu. Bi-objective trade-off with dynamic barrier gradient descent. NeurIPS 2021 , 2021.

Shoham Sabach and Shimrit Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

Harshal D Kaushik and Farzad Yousefian. A method with convergence rates for optimization problems with variational inequality constraints. SIAM Journal on Optimization , 31(3):21712198, 2021.

Afrooz Jalilzadeh and Farzad Yousefian. Stochastic approximation for estimating the price of stability in stochastic nash games. arXiv preprint arXiv:2203.01271 , 2022.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

Junjie Yang, Kaiyi Ji, and Yingbin Liang. Provably faster algorithms for bilevel optimization. Advances in Neural Information Processing Systems , 34:13670-13682, 2021.

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in neural information processing systems , 34:30271-30283, 2021.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Tighter analysis of alternating stochastic gradient method for stochastic nested problems. arXiv preprint arXiv:2106.13781 , 2021.

Zeeshan Akhtar, Amrit Singh Bedi, Srujan Teja Thomdapu, and Ketan Rajawat. Projection-free stochastic bi-level optimization. arXiv preprint arXiv:2110.11721 , 2021.

Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research , 153:235-256, 2007.

Lesi Chen, Jing Xu, and Jingzhao Zhang. On bilevel optimization without lower-level strong convexity. arXiv preprint arXiv:2301.00712 , 2023.

Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang. Towards gradient-based bilevel optimization with non-convex followers and beyond. Advances in Neural Information Processing Systems , 34:8662-8675, 2021.

Daouda Sow, Kaiyi Ji, Ziwei Guan, and Yingbin Liang. A primal-dual approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123 , 2022.

Han Shen and Tianyi Chen. On penalty-based bilevel gradient descent method. arXiv preprint arXiv:2302.05185 , 2023.

Feihu Huang. On momentum-based gradient methods for bilevel optimization with nonconvex lower-level. arXiv preprint arXiv:2303.03944 , 2023.

Kenneth Kreutz-Delgado, Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te-Won Lee, and Terrence J Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation , 15(2):349-396, 2003.

Mehrdad Yaghoobi, Thomas Blumensath, and Mike E Davies. Dictionary learning for sparse approximations with the majorization method. IEEE Transactions on Signal Processing , 57(6):2178-2191, 2009.

Benedek Rozemberczki, Paul Scherer, Yixuan He, George Panagopoulos, Alexander Riedel, Maria Astefanoaei, Oliver Kiss, Ferenc Beres, Guzmán López, Nicolas Collignon, et al. Pytorch geometric temporal: Spatiotemporal signal processing with neural machine learning models. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 4564-4573, 2021.

Chenglong Bao, Hui Ji, Yuhui Quan, and Zuowei Shen. Dictionary learning for sparse coding: Algorithms and convergence analysis. IEEE transactions on pattern analysis and machine intelligence , 38(7):1356-1369, 2015.

Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International Conference on Machine Learning , pages 427-435. PMLR, 2013.

Simon Lacoste-Julien. Convergence rate of frank-wolfe for non-convex objectives. arXiv preprint arXiv:1607.00345 , 2016.

Stephen Boyd and Lieven Vandenberghe. Localization and cutting-plane methods. From Stanford EE 364b lecture notes , 2007.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems , 32, 2019.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal nonconvex optimization via stochastic path-integrated differential estimator. Advances in Neural Information Processing Systems , 31, 2018.

Jiahao Xie, Zebang Shen, Chao Zhang, Boyu Wang, and Hui Qian. Efficient projection-free online methods with stochastic recursive gradient. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6446-6453, 2020.

Aryan Mokhtari, Asuman Ozdaglar, and Ali Jadbabaie. Escaping saddle points in constrained optimization. Advances in Neural Information Processing Systems , 31, 2018.

Aleksandr Beznosikov, David Dobre, and Gauthier Gidel. Sarah frank-wolfe: Methods for constrained optimization with best rates and practical features. arXiv preprint arXiv:2304.11737 , 2023.

Alp Yurtsever, Suvrit Sra, and Volkan Cevher. Conditional gradient methods via stochastic path-integrated differential estimator. In International Conference on Machine Learning , pages 7282-7291. PMLR, 2019.

Michael C Grant and Stephen P Boyd. Graph implementations for nonsmooth convex programs. In Recent advances in learning and control , pages 95-110. Springer, 2008.

Michael Grant and Stephen Boyd. Cvx: Matlab software for disciplined convex programming, version 2.1, 2014.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability , pages 1679-1706, 1994.

Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint arXiv:1902.03736 , 2019.

Tong Zhang. Learning bounds for kernel regression using effective data dimensionality. Neural Computation , 17(9):2077-2098, 2005.

Supplementary material

A Additional Motivating Examples

The bilevel optimization problem in (1) provides a versatile framework that covers a broad class of optimization problems. In addition to the motivating examples provided in the main body of the paper, here we also provide a generic example of stochastic convex constrained optimization that can be formulated as (1). We further present a more general form of the examples covered in the main body.

Stochastic convex optimization with many conic constraints: Consider the following convex optimization problem

where K ⊆ R d is a closed convex cone. This problem can be formulated as a special case of (1) by letting ˜ g ( x , ξ ) = 1 2 d 2 -K ( h ( x , ξ )) where d -K ( · ) ≜ ∥ · -P -K ( · ) ∥ denotes the distance function and P -K ( · ) denotes the projection map. Our proposed framework provides an efficient method for solving this class of problems when the projections onto K can be computed efficiently, while the projection onto the preimage h -1 ( -K , ξ ) is not practical, e.g., when K is the positive semidefinite cone, computing a projection onto the preimage set requires solving a nonlinear SDP.

A.1 Lexicographic optimization

Example 1 (over-parameterized regression) can be generalized as a broader class of problem, which is known as lexicographic optimization [13] and uses the secondary loss to improve generalization. The problem can be formulated as the following stochastic simple bilevel optimization problem,

In general, the lower-level problem could have multiple optimal solutions and be very sensitive to small perturbations. To tackle the issue, we use a secondary criterion L ( · ) to select some of the optimal solutions with our desired properties. For instance, we can find the optimal solutions with minimal ℓ 2 -norm by letting L ( β ) = ∥ β ∥ 2 , which is also known as Lexicographic ℓ 2 Regularization .

A.2 Lifelong learning

Example 2 (dictionary learning) is an instance of a popular framework known as lifelong learning, which can be formulated as follows,

In this problem, the objective is the training loss on the current tasks D t = { ( x ′ i , y ′ i ) } n ′ i =1 . While the constraint enforces that the model parameterized by β performs no worse than the previous one on the episodic memory M (i.e., data samples from all the past tasks).

In the paper, we discuss a variant of the problem above, where we slightly change the constraint and ensure that the current model also minimizes the error on the past tasks. It can be formulated as the following finite-sum/stochastic simple bilevel optimization problem [12],

B Supporting lemmas

B.1 Proof of Lemma 4.1

Before we proceed to the proof for Lemma 4.1, we present the following technical lemma, which gives us an upper bound for a complex term appearing in the following analysis.

Lemma B.1. Define ρ t = 1 / ( t + 1) ω where ω ∈ (0 , 1] and t ≥ 1 . For all t ≥ 2 , let { s t } be a sequence of real numbers given by

Then it holds that

Proof. We prove the result by induction. For t = 2 , we can verify that

Now we suppose that the inequality in (20) holds when t = T for some T ≥ 2 , i.e.,

First note that the sequence { s t } satisfies the following recurrence relation:

Moreover, since ω ∈ (0 , 1] , we have ( T +2) ω -1 ≤ ( t +1) ω . Therefore, we obtain

By induction, the inequality in (20) holds for all t ≥ 2 .

Now we proceed to prove Lemma 4.1.

Proof of Lemma 4.1. We show the proof of part (i) here. The proof of part (ii) is very similar to part (i). The first step is to reformulate e t = ̂ ∇ g t -∇ g ( x t ) as the sum of a martingale difference sequence. For t ≥ 1 , by unrolling the reucurrence we have

Thus, we can write e t as the sum e t = ∑ t τ =1 ζ τ , where we define ζ 1 = ∏ t k =2 (1 -β k ) e 1 and

for τ > 1 . Recall that e 1 = ∇ ˜ g ( x t , ζ 1 ) -∇ g ( x 1 ) . We observe that E [ ζ τ |F τ -1 ] = 0 where F τ -1 is the σ -field generated by { x 1 , ξ 1 , . . . , x τ -1 , ξ τ -1 } . Therefore, { ζ τ } t τ =1 is a martingale difference sequence.

Next, we derive upper bounds of ∥ ζ τ ∥ . To begin with, we observe that for any τ = 1 , 2 , . . . , t ,

where we used the fact that ( k +1) ω -1 ≤ k ω in the last inequality. By using the above inequality, we can bound ∥ ζ 1 ∥ as follows:

Define c 1 = 2 ω σ g ( T +1) ω , then by Assumption 2.3(ii) we have E [exp ( ∥ ζ 1 ∥ 2 /c 2 1 )] ≤ exp(1) . Moreover, for τ > 1 , by triangle inequality, ∥ ζ τ ∥ can be bounded by

Define c τ = (2 L g D + 3 ω σ g 3 ω -1 ) β τ ∏ t k = τ (1 -β k ) . Note that if we have E [exp( X 2 1 /c 2 1 )] ≤ 1 and E [exp( X 2 2 /c 2 2 )] ≤ 1 , then we have E [exp(( X 1 + X 2 ) 2 / ( c 1 + c 2 ) 2 )] ≤ 1 [43]. Thus, we have E [exp ( ∥ ζ τ ∥ 2 /c 2 τ )] ≤ exp(1) for all 1 ≤ τ ≤ t . Hence by proposition E.2, with probability 1 -δ ′

where c is an absolute constant, d is the number of dimension, and ∑ T τ =1 c 2 τ can be bounded by Lemma B.1 as follows,

where the last inequality follows from the fact that ( √ 2) ω ≤ 3 ω / (3 ω -1) for any ω ∈ (0 , 1] . Combining (26) and (27), we have with probability at least 1 -δ ′ ,

Similarly with probability at least 1 -δ ′ ,

and with probability at least 1 -δ ′ ,

where c is an absolute constant and d is the dimension of vectors. We can use union bound to obtain that these three inequalities hold for at least probability 1 -3 δ ′ = 1 -δ . For simplicity, we define constant A ω 1 and A ω 0 such that,

B.2 Proof of Lemma 4.2

Proof. Let us define t 0 ≜ ⌊ t/q ⌋ for any t ∈ { 0 , . . . , T -1 } , then whenever t = t 0 q according to the Algorithm 2 a full batch of sample gradients are selected, hence, ̂ ∇ g t = ∇ g ( x t ) ; otherwise, the error of computing a sample gradient can be expressed as follows

where i is the index with S ( i ) denoting the i -th random component function selected at iteration t . Furthermore, from the update rule of x t we have ∥ x t -x t -1 ∥ = γ t ∥ s t -1 -x t -1 ∥ ≤ Dγ for any t ≥ 0 , therefore,

̸

for all t ∈ { t 0 +1 , . . . , t 0 + q } and i ∈ { 1 , . . . , S } . On the other hand, from the update of ̂ ∇ g t and (32) we have that for any t = t 0 q , ̂ ∇ g t -∇ g ( x t ) = ̂ ∇ g t -1 -∇ g ( x t -1 ) + ∑ S i =1 ϵ t,i . Therefore, by continuing the recursive relation and taking the norm from both sides of the equality we obtain

where the last equality follows from ̂ ∇ g ( x t 0 ) = ∇ g ( x t 0 ) . Then by Proposition E.1, we have

where the last inequality follows from the fact S = √ n and t -t 0 ≤ q = √ n . By setting λ = (4 L g Dγ √ log(4 /δ ′ )) for some δ ′ ∈ (0 , 1) , we have with probability at least 1 -δ ′ ,

Similarly, with probability at least 1 -δ ′ ,

Then by union bound and δ = 3 δ ′ , we show these three equalities hold with probability 1 -δ .

B.3 Proof of Lemma 4.3

Proof. Let x ∗ g be any point in X ∗ g , i.e., any optimal solution of the lower-level problem. By definition, we have g ( x ∗ g ) = g ∗ . Since g is convex and g ∗ ≤ g ( x 0 ) , we have

Add and subtract terms in inequality above, we have,

Considering the random hyperplane we used in (9), we want to prove the following inequality holds with high probability,

Recall K t = K 0 ,t + DK 1 ,t . And K 0 ,t and K 1 ,t were set as the high probability bounds of ∥ ̂ ∇ g t -∇ g ( x t ) ∥ and | ˆ g t -g ( x t ) | in Lemma 4.1 for Algorithm 1 or Lemma 4.2 for Algorithm 2. Then compare the two inequalities above and use Jensen's inequality, |⟨ ̂ ∇ g t , x ∗ g -x t ⟩| + | ˆ g t -g ( x 0 ) | ≤ K t holds with high probability 1 -δ for all t ≥ 0 . Hence, Lemma 4.3 holds with probability 1 -δ for all t ≥ 0 .

B.4 Improvement in one step

The following lemma characterizes the improvement of both the upper-level and lower-level objective values after one step of the algorithms.

Lemma B.2. If Assumptions 2.1, 2.2, 2.3 are satisfied,

(i) For all t ≥ 0 , assume that X ∗ g ⊂ X t . Then we have

As a corollary, if f is convex, we further have

and with probability 1 -δ ′ ,

(ii) We have

Proof. (i) Based on the L f -smoothness of the expected function f we show that f ( x t +1 ) is bounded by

Replace the terms x t +1 -x t by γ t +1 ( s t -x t ) and add and subtract the term γ t +1 ̂ ∇ f T t ( s t -x t ) to the right hand side to obtain,

By Lemma 4.3, X ∗ g ⊆ X t with high probability 1 -δ , for all t = 1 , . . . , T . Note that if we define s ′ t = arg max s ∈X t {⟨∇ f ( x t ) , x t -s ⟩} . Recall that FW gap is G (ˆ x ) = max s ∈X ∗ g {⟨∇ f (ˆ x ) , ˆ x -s ⟩} . We can replace the inner product ⟨ ̂ ∇ f t , s t ⟩ by its upper bound ⟨ ̂ ∇ f t , s ′ t ⟩ . Applying this substitution leads to

Rearrange the terms for the inequality above, we can obtain,

∗

As a simple corollary, since G ( x t ) ≥ f ( x t ) -f when f is convex, we have,

(ii) Based on the L g -smoothness of the expected function g we show that g ( x t +1 ) is bounded by

Replace the terms x t +1 -x t by γ t +1 ( s t -x t ) and add and subtract the term γ t +1 ̂ ∇ g ⊤ t ( s t -x t ) to the right-hand side to obtain,

Now by definition of the set X t , using ⟨ ̂ ∇ g t , s t -x t ⟩ ≤ g ( x 0 ) -ˆ g t + K 0 ,t + DK 1 ,t . In addition, we could use Cauchy-Schwarz inequality to upper bound the second term. Then add and subtract γ t +1 g ( x 0 ) on the right hand side to obtain,

Then subtract g ( x 0 ) on both sides,

g

0

t

+1

t

+1

x

x

x

g

(

)

(1

(

)

γ

)(

g

(

-

-

+

t

+1

t

x

γ

(

D

g

(

∥∇

and the claim in the lemma follows.

t

)

)

-

-

g

(

∇

g

̂

x

0

t

∥

))

+

∥

g

(

x

t

)

-

ˆ

g

t

∥

+

K

0

,t

+

DK

1

,t

) +

L

g

2

∥

x

t

+1

-

t

x

∥

(53)

2

C Proof of Theorem for Algorithm 1

C.1 Proof of Theorem 4.4

Proof. For lower-level , by Lemma B.2, we have

By Lemma 4.1, we have ∥∇ g ( x t ) -̂ ∇ g t ∥ ≤ K 1 ,t and ∥ g ( x t ) -ˆ g t ∥ ≤ K 0 ,t with probability 1 -δ ′ . Plug them in the inequality above to obtain,

with probability 1 -δ ′ for all t . Let C 1 = 4( DA 1 1 + A 1 0 ) and δ = Tδ ′ . Then, by applying the inequality recursively for t = 1 , . . . , T -1 , we obtain that

with probability 1 -δ .

For upper-level , by Lemma B.2, we have

By Lemma 4.1, we have ∥∇ f ( x t ) -̂ ∇ f t ∥ ≤ A 1 2 √ log (6 d/δ ′ ) ( t +1) 1 / 2 with probability 1 -δ ′ . Plug it in the inequality above to obtain,

with probability 1 -δ ′ for all t . Let C 2 = 2 DA 1 2 and δ = Tδ ′ . Then, by applying the inequality recursively for t = 1 , . . . , T -1 , we obtain that

with probability 1 -δ . The theorem is obtained.

C.2 Proof of Theorem 4.5

Proof. For lower-level , by Lemma B.2, we have

By Lemma 4.1, we have ∥∇ g ( x t ) -̂ ∇ g t ∥ ≤ K 1 ,t and ∥ g ( x t ) -ˆ g t ∥ ≤ K 0 ,t with probability 1 -δ ′ . Plug them in the inequality above to obtain,

with probability 1 -δ ′ for all t . Let C 3 = 2( DA 2 / 3 1 + A 2 / 3 0 ) .Then we can sum all the inequality up for all t to obtain,

By induction, we have for all t ≥ 1 ,

with probability 1 -δ , where δ = Tδ ′ .

For upper-level , by Lemma B.2, we have

By Lemma 4.1, we have ∥∇ f ( x t ) -̂ ∇ f t ∥ ≤ A 2 / 3 2 √ log (6 d/δ ′ ) ( t +1) 1 / 3 with probability 1 -δ ′ . Plug it and γ t +1 = 1 / ( T +1) 2 / 3 in inequality above to obtain,

Let x t ∗ = arg min 1 ≤ t ≤ T G ( x t ) , then

with probability 1 -δ , where δ = Tδ ′ . By letting C 4 = 3 2 DA 2 / 3 2 , the theorem is obtained.

D Proof of Theorem for Algorithm 2

D.1 Proof of Theorem 4.6

Proof. For lower-level By Lemma B.2, we have

By Lemma 4.2, we have ∥∇ g ( x t ) -̂ ∇ g t ∥ ≤ 4 L g Dγ √ log(12 /δ ′ ) and ∥ g ( x t ) -ˆ g t ∥ ≤ 4 L l Dγ √ log(12 /δ ′ ) with probability 1 -δ ′ . Let C 5 = 8 D ( DL g + L l ) and δ = Tδ ′ . Plug them in inequality above and let γ t = γ = log T/T to obtain,

with probability 1 -δ .

For upper-level , by Lemma B.2, we have,

Now we proceed by replacing the terms ∥∇ f ( x t ) -̂ ∇ f t ∥ by its upper bounds from Lemma 4.2, i.e. ∥∇ f ( x t ) -̂ ∇ f t ∥ ≤ 4 L f Dγ √ log (12 /δ ′ ) ,

with probability (1 -δ ′ ) . And we can choose δ = 3 Tδ ′ Then by telescope, with γ = log T T , we can obtain,

with probability 1 -δ . Note that without loss of generality, we can assume f ( x 0 ) -f ∗ ≥ 0 . If it is less than 0, we can bound it by 0. By letting C 6 = 5 L f D 2 , the theorem is obtained.

D.2 Proof of Theorem 4.7

Proof. For lower-level , by Lemma B.2, we have

By Lemma 4.2, we have ∥∇ g ( x t ) -̂ ∇ g t ∥ ≤ 4 L g Dγ √ log(12 /δ ′ ) and ∥ g ( x t ) -ˆ g t ∥ ≤ 4 L l Dγ √ log(12 /δ ′ ) with probability 1 -δ ′ . Let C 7 = 8 D ( DL g + L l ) and δ = Tδ ′ . Plug them in inequality above and let γ t = 1 / √ T to obtain,

with probability 1 -δ/T . Sum up the inequalities for all t ≥ 1 to get,

with probability 1 -δ .

For upper-level , by Lemma B.2, we have

By Lemma 4.2, we have ∥∇ f ( x t ) -̂ ∇ f t ∥ ≤ 4 L f Dγ √ log(12 /δ ′ ) with probability 1 -δ ′ . Plug it and γ t +1 = 1 / √ T in inequality above to obtain,

Divide both sides by √ T , we can get, Let x t ∗ ≜ arg min 1 ≤ t ≤ T G ( x t ) , then

with probability 1 -δ . By letting C 8 = 5 L f D 2 , the theorem is obtained.

E Azuma-Hoeffding-type inequalities

In this section, we present two useful vector versions of Azuma-Hoeffding-type concentration inequalities with uniform bound assumption or sub-gaussian assumption. They are crucial in our high probability analysis.

Proposition E.1. (Pinelis and other 1994 [44], Theorem 3.5) Let ζ 1 , . . . , ζ t ∈ R d be a vectorvalued martingale difference sequence w.r.t. a filtration {F t } , i.e. for each τ ∈ 1 , . . . , t , we have E [ ζ τ |F τ -1 ] = 0 . Suppose that ∥ ζ τ ∥ ≤ c τ almost surely. Then ∀ t ≥ 1 ,

Proposition E.2. (Jin et al. [45], Corollary 7) Let ζ 1 , . . . , ζ t ∈ R d be a vector-valued martingale difference sequence w.r.t. a filtration {F t } , i.e. for each τ ∈ 1 , . . . , t , we have E [ ζ τ |F τ -1 ] = 0 . Suppose that E [exp( ∥ ζ τ ∥ 2 /c 2 τ )] ≤ exp(1) . Then there exists a absolute constant c such that, for any δ > 0 , with probability at least 1 -δ ,

This proposition was also used in previous literature including [46] and [37]. It is common to use such martingale inequality to obtain some high-probability results recently.

F Experiment details

In this section, we include more details about the numerical experiments in Section 5. For completeness, we briefly introduce the update rules of aR-IP-SeG in [16] and DBGD in [13]. In the following, we use the notation Π Z ( · ) to denote the Euclidean projection onto the set Z . The aR-IP-SeG algorithm is given by,

where γ t is the stepsize, ρ t is the regularization parameter, and ¯ y T is the output of the algorithm. In this experiment, we choose γ t = γ 0 / ( t +1) 3 / 4 and ρ t = ρ 0 ( t +1) 1 / 4 for some constants γ 0 and ρ 0 . The DBGD-sto is a stochastic version of DBGD, which simply replaces the gradients in DBGD with stochastic gradients. Although the stochastic version of DBGD does not have a theoretical guarantee, it has been used to solve stochastic simple bilevel optimization problems in [13], which worked pretty well empirically. Hence, we use it as a baseline for solving stochastic simple bilevel problems and compare it with our proposed algorithms. The DBGD algorithm is given by

where γ k is the stepsize and we set λ k as

where α and β are hyperparameters and ˆ g is a lower bound of g ∗ . In this experiment, we choose ˆ g = 0 . We also note that [13] only considered unconstrained simple bilevel optimization, i.e. Z = R d . We further project x t onto Z for each iteration to ensure the constraints are satisfied.

F.1 Over-parameterized regression

Dataset generation. The original Wikipedia Math Essential dataset [30] composes of a data matrix of size 1068 × 731 . We randomly select one of the columns as the outcome vector b ∈ R 1068 and the rest to be a new matrix A ∈ R 1068 × 731 . We set constraint parameter λ = 10 in this experiment. Initialization. We run the algorithm, SPIDER-FW [40], with stepsize chosen as γ t = 0 . 1 / ( t +1) on the lower-level problem in (1). We terminate the process to get x 0 as the initial point for both SBCGI 1 and SBCGF 2 after 10 5 stochastic oracle queries.

Implementation details. We query stochastic oracle 9 × 10 5 times with stepsize γ t = 0 . 01 / ( t +1) and γ = 10 -5 for SBCGI 1 and SBCGF 2 with K t = 10 -4 / √ t +1 , respectively. In each iteration, we need to solve the following subproblem induced by the methods,

Introduce s + , s -≥ 0 such that s = s + -s -. Then we can reformulate the problem above as follows,

where 1 ∈ R d is the all-one vector.

For aR-IP-SeG, we choose γ 0 = 10 -7 and ρ 0 = 10 3 . For DBGD, we set α = β = 1 and γ t = 10 -6 .

F.2 Dictionary learning

Dataset generation. We generate 500 sparse coefficient vectors { x i } 250 i =1 and { x ′ k } 250 k =1 with 5 random nonzero entries, whose absolute values are drawn uniformly from [0 . 2 , 1] . The entries of the random noise vectors { n i } 250 i =1 and { n ′ k } 250 k =1 are drawn from i.i.d. Gaussian distribution with mean 0 and standard deviation 0.01.

Initialization. We use a similar initialization procedure as [12], which consists of two phases. In the first phase, we run the standard Frank-Wolfe algorithm on both the variables D ∈ R 25 × 40 and X ∈ R 40 × 250 for 10 4 iterations with the stepsize γ t = 1 / √ t +1 . Next, in the second phase, we fix the variable X and only update D using the Frank-Wolfe algorithm with exact line search for additional 10 4 iterations to obtain ˆ D and ˆ X as the initial point for the full bilevel problem.

Implementation Details. We choose δ = 3 in both problems (5). To be fair, all four algorithms start from the same initial point. We slightly modify the initial point by letting ˜ D ∈ R 25 × 50 be the concatenation of ˆ D ∈ R 25 × 40 and 10 columns of all zeros vectors. Furthermore, we initialize another variable ˜ X randomly by choosing its entries from a standard Gaussian distribution and then normalizing each column to have a ℓ 1 -norm of δ . We choose the stepsize as γ t = 0 . 1 / ( t +1) 2 / 3 and γ = 10 -3 for our SBCGI 1 and SBCGF2 with K t = 0 . 01 / ( t +1) 1 / 3 , respectively. Empirically, we observe that taking one sample per iteration leads to a very unstable process in this problem. In this case, we choose a mini-batch of size 8 for SBCGI, aR-IP-SeG, and the stochastic version of DBGD. For each iteration, we will solve the following subproblem,

The above problem can be reformulated by using the KKT condition, which is equivalent to get a root of the following one-dimensional nonlinear equation involving λ ≥ 0 :

0

100

200

300

10 -6

10 -4

10 -2

10 0

(a) Lower-level gap

0

100

200

300

10 -4

10 -2

10 0

10 2

(b) Upper-level gap

0

100

200

300

10 -1

10 0

10 1

(c) Test error

0

20

40

60

80

100

120

10 -8

10 -6

10 -4

10 -2

10 0

(a) Lower-level gap

0

20

40

60

80

100

120

10 -2

10 -1

10 0

(b) Upper-level gap

0

20

40

60

80

100

120

0

0.2

0.4

0.6

0.8

1

(c) Recovery rate

where the projection onto Z = { ˜ D ∈ R 25 × 50 : ∥ ∥ ∥ ˜ d i ∥ ∥ ∥ 2 ≤ 1 , i = 1 , . . . , 50 } is equivalent to project each column on the Euclidean ball. In practice, the reformulated problem can be solved efficiently by MATLAB's root-finding solver.

For aR-IP-SeG, we choose γ 0 = 10 and ρ 0 = 1 . For the stochastic version of DBGD, we set -3

-4 α = β = 100 and γ t = 5 × 10 .

Additional plots illustrating the comparison of the studied methods in terms of runtime rather than the number of sample used are provided in Figure 3 and Figure 4.

F.3 Experiments with different random seeds

We further repeat the experiment 10 times with different random seeds to see more realizations of the stochastic algorithms. The results are reported in Figure 5 and Figure 6. The solid lines denote the average statistics over 10 trials of the algorithms. While the shaded regions surrounding each line reflect the span of all the random instances involved. Figure 5 and Figure 6 present similar results as Figure 1 and Figure 2, which eliminates the possibility of choosing a particularly good instance.

(a) Lower-level gap

(b) Upper-level gap

(c) Test error

10

SBCGI

SBCGF

100

10-1

DBGD-sto

aR-IP-SeG

10-1

(g)6]

10-2

10-2

Test error

10-3

10-3

104

SBCGF

SBCGI

10-1

SBCGF

SBCGI

aR-IP-SeG

aR-IP-SeG

DBGD-sto

DBGD-sto

10~4

0

2

4

6

8

10

10-5

0

2

4

6

8

10

Q

2

4

6

8

10

12

number of samples

×105

number of samples

x105

number of samples

×105

(a) Lower-level gap

(b) Upper-level gap

(c) Recovery rate

10

100

0.8

（Do）

10-2

Recovery rate

SBCGI

0.6

10

SBCGI

aR-IP-SeG

SBCGF

aR-IP-SeG

DBGD-sto

SBCGF

0.4

106

DBGD-sto

0.2

SBCGF

SBCGI

aR-IP-SeG

DBGD-sto

10-8

1

2

3

4

5

6

102

2

4

6

8

0

2

4

6

8

number of samples

×105

number of samples

×105

number of samples

×105

0

2

4

6

8

10 5

10 -20

10 -15

10 -10

10 -5

10 0

(a) Lower-level gap

0

2

4

6

8

10 5

10 -3

10 -2

10 -1

10 0

(b) Upper-level gap

0

2

4

6

8

10 5

0

0.2

0.4

0.6

0.8

1

(c) Recovery rate

F.4 Importance of the right cutting plane

In this section, we numerically illustrate the importance of choosing the right cutting plane on Example 2 (dictionary learning) . Specifically, we compare our proposed methods with the ones without a cutting plane and with an unregularized cutting plane (without additional term K t ).

If we replace the stochastic cutting plane (9) with the unregularized cutting plane (8) in SBCGI 1 and SBCGF 2, then the algorithm usually fail at some point in the process, depending on the datasets and parameters chosen, based on our experimental observations. More specifically, algorithms' failure means that the subproblem of dictionary learning (85) is infeasible. So we slightly modify it by adding a checkpoint before solving the subproblem. If the subproblem is infeasible at the current iteration, then we choose the update direction s t = ̂ ∇ g t . This adjustment prevents unnecessary interruptions during the process and enforce the algorithms to focus only on the lower-level problem when the subproblem is infeasible. We denote the modified algorithms SBCGI-M and SBCGF-M. Moreover, we also take SBCGI and SBCGF without cutting planes into consideration, denoted as STORM-FW and SPIDER-FW. In fact, in this case, the bilevel algorithms degenerate to single-level projection-free algorithms similar to algorithms in [37] and [40].

Figure 7 (a) indicates that SBCGI-M and SBCGF-M focus more on the lower-level problem due to the design of the algorithms and extremely unstable as we can see in Figure 7 (b)(c). While STORM-FW and SPIDER-FW only focus on the upper-level problem, which leads to terrible results on the lower-level gap and recovery rate.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA nice and clean result for stochastic simple bilevel optimization. The manuscript extends the algorithm design from the deterministic simple bilevel optimization with convex lower-level problem to the stochastic setting and demonstrate the sample complexity, which seems to be near-optimal (ignoring the log factor) when restricting the problem to classical stochastic optimization setting.\n\nSTRENGTHS:\nNew results with good quality that are demonstrated in a nice and clean way. The motivation is very clear. It may push forward the study of general stochastic bilevel optimization when the lower-level is non-singleton.\n\nWEAKNESSES:\nThe manuscript has no major flaws. The idea to add a $K_t$ and to estimate g using samples are rather straightforward though. The motivation for using variance reduction are not well-explained. Using variance reduction for lower level is to ensure that decaying $K_t$ makes sense. This step I doubt if it is possible to only use moving average estimator [Chen et al., 2021 NeurIPS, Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems] on the lower-level problem to achieve the same effect. Using variance reduction for upper level is to obtain the current complexity bound. Otherwise, using vanilla Frank-wolfe on the upper level will result in larger complexity. But nevertheless, the manuscript already provides a full story. So that the algorithm would requires less hyperparameter to tune. See my questions below.\n\nQUESTIONS:\n1. Throughout your discussion of related literature and contributions, please accurately denote if the reference or contribution pertains to simple stochastic bilevel optimization or the broader general stochastic bilevel optimization. This applies to numerous instances, such as Line 139 and Line 45.\n\n2. Regarding Line 103, is it $A\\subset \\mathbb{R}^m$ or $a_i\\in\\mathbb{R}^m$? There appears to be a typographical error.\n\n3. How are $p$ and $q$ determined in continual dictionary learning? Do the old dataset $A$ and the new dataset $A^\\prime$ have any correlation? If uncorrelated, why merge them? If correlated, would the results still hold?\n\n4. In the 'Random set for the subproblem' subsection, is there a missing $\\hat $ on $g(x_0)$? How is $g(x_0)$ known? This query also applies to equations (8)(9) and Algorithms 1 and 2.\n\n5. Lemma 4.1 seems to miss a space in \"c, (d is \". Additionally, why does the estimation error in Lemma 1 depend on the dimension? In other variance reduction papers, like STORM and SPIDER, the estimation error is generally dimension-independent.\n\n6. In Theorem 1 and 2, $\\mathcal{G}$ is not defined anywhere in the main context.\n\n7. My understanding of general bilevel optimization implies variance reduction aims for $O(\\epsilon^{-4})$ complexity bound in the nonconvex regime ($O(\\epsilon^{-3})$ with variance reduction). However, Chen et al., 2021 [NeurIPS, closing the gap, tighter analysis] indicated that variance reduction isn't essential for achieving optimal complexity in strong and convex settings. Moreover, for nonconvex settings, an $O(\\epsilon^{-4})$ rate can still be obtained without variance reduction. Therefore, I question whether variance reduction is genuinely necessary when $\\hat{\\nabla g}$ can be adaptively estimated using a moving average estimator, possibly achieving the same rate. This could be an avenue for future consideration.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and there is no societal impact involved as it is a pure theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach for solving stochastic simple bilevel optimization problems. The authors develop new methods that locally approximate the solution set of the lower-level problem using a stochastic cutting plane and then employ a conditional gradient update with variance reduction techniques to manage error from stochastic gradients.\n\nThe specific contributions are:\n- For the convex upper-level function case, the proposed method requires ${\\tilde{\\mathcal{O}}}(\\max ( 1/\\epsilon_f^2, 1/\\epsilon_g^2 ) )$ stochastic oracle queries to find a solution that is $\\epsilon_f$-optimal for the upper-level and $\\epsilon_g$-optimal for the lower-level. This is a substantial improvement over the previous complexity of $\\mathcal{O}(\\max(1/\\epsilon_f^4, 1/\\epsilon_g^4))$.\n- For the non-convex upper-level function case, the method requires at most ${\\tilde{\\mathcal{O}}}(\\max(1/\\epsilon_f^3, 1/\\epsilon_g^3))$ stochastic oracle queries to find an $(\\epsilon_f, \\epsilon_g)$-stationary point.\n- In the finite-sum setting, the method requires ${\\tilde{\\mathcal{O}}}(\\sqrt{n}/\\epsilon)$ and ${\\tilde{\\mathcal{O}}}(\\sqrt{n}/\\epsilon^2)$ stochastic oracle calls for the convex and non-convex settings, respectively, where $\\epsilon= \\min(\\epsilon_f,\\epsilon_g)$.\n\nSTRENGTHS:\n- Originality: In this paper, the idea of using cutting planes to approximate the optimal solution set for solving deterministic simple bilevel optimization is extended to the stochastic version. Due to the stochastic nature of the problem, this generalization is not trivial.\n- Quality: From a theoretical standpoint, the quality of the research is notable, with the authors demonstrating a substantial improvement over previous complexity results.\n- Clarity: The overall structure of the paper is clear.\n- Significance: This research provide lower complexity guarantees for stochastic simple bilevel optimization.\n\nWEAKNESSES:\n- The authors make an assumption that the constraint set $\\mathcal Z$ is compact and convex, which may not hold for many real applications. This strong assumption limits the method's applicability. \n\n- In their experiments, the authors employ the number of samples as the measure on the abscissa axis without providing a clear rationale for this choice. Furthermore, a comparative analysis of the time efficiency across various methods is missing, which should ideally be included to give a more comprehensive evaluation of their approach. \n\n- The paper contains numerous typographical errors which, although not affecting the technical content, detract from the overall readability and professionalism of the paper.\n\n- In Example 1, the authors do not clearly delineate how randomness is incorporated and why it necessitates the use of stochastic simple bilevel optimization as a modeling approach for this particular problem.\n\nQUESTIONS:\n- Line 15 and elsewhere: The authors discard the logarithmic terms in the complexity results. I suggest the authors use $\\tilde{\\mathcal{O}}$ to hide the logarithmic terms.\n- Line 46: The authors state that their approach is ``tight\". Why?\n- Example 1: The paper doesn't clarify where the element of randomness lies, nor does it explain why the problem specifically requires a stochastic simple bilevel optimization model.\n- Line 156, the formula (8): How to compute the exact $g(x_0)$?\n- Line 158: Why is there $\\hat{g}(x_0)$ instead of $g(x_0)$?\n- Algorithm 1, Line 7; Line 160, the formula (9);  Line 163 and elsewhere: The notations $\\mathcal{X}_t$, $\\hat{\\mathcal{X}}_t$, $\\mathcal{X}_t^{S}$ should be unified.\n- Line 177: The citation (7) should be (9).\n- Line 226, the formula (16): The term $\\sqrt{\\log(6/\\delta)}$ should be changed to $\\sqrt{\\log(6d/\\delta)}$.\n- The authors state that $\\omega=1$ and $\\omega=2/3$ are the ``best'' choice for the convex and nonconvex settings, respectively. How to ensure that the choices are the best?\n- Lemma 4.2 and Lemma 4.3: The authors could swap the places of these two lemmas and rewrite Lemma 4.2 to be more precise.\n- Line 238, the last sentence: add \"we\" before \"establish\".\n- In numerical experiments: Why do the authors use the number of samples be the abscissa axis? The paper doesn't include a comparative analysis of the time efficiencies across different algorithms, an omission that needs clarification.\n- Line 316: The matrix $D^{'*}$ is not clearly defined.\n- In Supplementary material: Many formulas are numbered but not cited, please consider removing these numbers. Sometimes the paper doesn't provide an explanation for the validity of two adjacent formulas. An appropriate clarification would be helpful for understanding.\n- Line 479: The $S$ in the formula should be changed to $s_2$.\n- Line 480 and elsewhere: The use of both \"t\" and \"T\" in the paper could lead to confusion. To avoid this, the authors could consider replacing one of them, such as changing \"T\" to \"M\".\n- Line 482 and elsewhere: The \"$\\alpha$\" should be replaced by \"$\\omega$\".\n- Line 526: the citation (47) is not correct.\n- Line 563, the formula (55) and elsewhere: The term $g(x_t)-g(x_0)$ should be $(g(x_t)-g(x_0))$.\n- Line 565, the formulas and elsewhere: The subscript $t+1$ in $g(x_{t+1})$ in the formula does not match the subscript $t$ in $g(x_t)$ in the theorem. This situation occurs many times in the proofs of theorems.\n- Line 567, the formula (57) and elsewhere: The Lipchitz constant $L_g$ should be replaced by $L_f$.\n- Line 677: The number $40$ should be replaced by $50$.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper introduces novel projection-free optimization methods for solving a class of *simple* stochastic bilevel optimization problems. The proposed methods are based on the conditional gradient algorithm and use variance reduction techniques to approximate the solution set of the lower-level problem. The paper provides theoretical guarantees for the convergence of the proposed methods and analyzes their sample complexity. The methods are shown to be efficient for both convex (global convergence) and non-convex upper-level (local convergence) functions.

STRENGTHS:
The paper introduces a new framework for solving a class of stochastic bilevel optimization problems, which is an important and challenging problem in optimization. The proposed methods are shown to be efficient in experiments and have theoretical guarantees.

WEAKNESSES:
I am concerned that the paper makes no reference to the game theory literature on the topic (general-sum Stackelberg games).  Related work on the topic would make the work accessible to a broader audience.

QUESTIONS:
Questions: Can you describe how we can see this optimization problem from a game theory perspective?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes methods for solving nonconvex/convex-convex simple bilevel optimization problems. The proposed methods are proven to have better non-asymptotic convergence guarantee than existing works. Numerical experiments are done on two applications to verify the effectiveness of the proposed methods.\n\nSTRENGTHS:\nThe main contribution of this work is the improved convergence rates for solving nonconvex/convex-convex simple bilevel optimization problems. The proposed methods use a cutting plane to approximate the solution set of the lower-level problem, of which the solution may not be unique. In order to have a more accurate approximation, the work proposes to use variance reduced estimators (STORM for stochastic setting and SPIDER for finite-sum setting) for gradient and function value estimations.\n\nWEAKNESSES:\n1. In the related work section, line 69~71 says that existing works on solving general bilevel optimization only consider strongly convex lower-level problems. This statement is not true. There have several works done on studying general bilevel optimization without strong convexity condition of lower-level problems. Just to name a few: (Liu et al., 2021), (Sow et al., 2022), (Shen et al., 2023), etc. Since these works are solving general bilevel optimization, they are potentially applicable to the simple bilevel optimization. I believe it is reasonable to provide a more detailed discussion on how these existing works are/aren't applicable to the target formulation of this draft.\n2. I notice that the algorithm SBCGI uses only one sample data per iteration, and the sample data batch size in algorithm SBCGF is fixed to $\\sqrt{n}$. In general, fixed batch size is often not very practical, especially for large dataset. I wonder if the authors have any insights on why the batch size cannot be set arbitrarily and be a dependency parameter in the convergence rate.\n3. Typo: in line 73, one->on.\n\n\n\nReferences:\n\nRisheng Liu, Yaohua Liu, Shangzhi Zeng, Jin Zhang, Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyond, 2021.\n\nDaouda Sow, Kaiyi Ji, Ziwei Guan, Yingbin Liang, A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima, 2022.\n\nHan Shen, Quan Xiao, Tianyi Chen, On Penalty-based Bilevel Gradient Descent Method, 2023.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nThe fixed batch size limits the proposed methods from applying on large-scale datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}