Title: An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bilevel Optimization

Acceleration

Unbounded Smoothness

Nonconvex Optimization
Abstract: This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(\epsilon^{-4})$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(\epsilon^{-3})$ to find an $\epsilon$-stationary point, when the lower-level stochastic gradient has a small variance $O(\epsilon)$. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

Abstract

This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require ˜ O ( ϵ -4 ) oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an ϵ -stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of ˜ O ( ϵ -3 ) to find an ϵ -stationary point, when the lower-level stochastic gradient has a small variance O ( ϵ ) . Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization. The code is available here.

1 Introduction

Bilevel optimization receives tremendous attention recently in the machine learning community, due to its applications in meta-learning [27, 59], hyperparameter optimization [27, 25], data hypercleaning [44], continual learning [7, 37], and reinforcement learning [47]. The bilevel optimization problem has the following formulation:

where f and g are upper-level and lower-level functions respectively. For example, in metalearning [26, 27], x denotes the layers of neural networks for shared representation learning, y denotes the task-specific head encoded in the last layer, and the formulation (1) aims to learn the a

common representation learning encoder x such that it can be quickly adapted to downstream tasks by only updating the task-specific head y . In machine learning, people typically consider stochastic optimization setting such that f ( x, y ) = E ξ ∼D f [ F ( x, y ; ξ )] and g ( x, y ) = E ζ ∼D g [ G ( x, y ; ζ )] , where D f and D g are the underlying unknown data distributions for f and g respectively, and one can access noisy observations of f and g based on sampling from D f and D g .

There emerges a wave of studies for algorithmic design and analysis for solving the bilevel optimization problem (1) under different assumptions of f and g . Most theoretical work assumes the upper-level function is smooth (i.e., gradient is Lipschitz) and nonconvex, and the lower-level function is strongly convex [30, 44, 41, 33, 48]. However, as pointed out by [75, 15], certain neural networks such as recurrent neural networks [22], long-short term memory networks [40] and transformers [65] have smoothness constants that scale with gradient norm, potentially leading to unbounded smoothness constants (i.e., gradient Lipschitz constant can be infinity). Motivated by this, Hao et al. [38] designed the first bilevel optimization algorithm to handle the cases where f is nonconvex with potentially unbounded smoothness and g is strongly convex. The algorithm in [38] achieves ˜ O ( ϵ -4 ) oracle complexity for finding an ϵ -stationary point (i.e., a point x such that ∥∇ Φ( x ) ∥ ≤ ϵ ). Gong et al. [32] proposed an single-loop algorithm under the same setting as in [38] and also achieved ˜ O ( ϵ -4 ) oracle complexity. This complexity result is worse than the ˜ O ( ϵ -3 ) oracle complexity under the relatively easier setting where f has a Lipschitz gradient, and each realization of the stochastic oracle calls is Lipschitz with respect to its argument (e.g., almost-sure Lipschitz oracle) [71, 46, 18, 34, 43]. This naturally motivates us to study the following question:

Is it possible to improve the ˜ O ( ϵ -4 ) oracle complexity for bilevel optimization problems where the upper-level function is nonconvex with unbounded smoothness and the lower-level function is strongly convex, by assuming that the properties of the function at the population level also hold almost surely for each random realization?

In this paper, we give a positive answer to this question by designing a new algorithm named AccBO with an improved oracle complexity of ˜ O ( ϵ -3 ) , when the lower-level stochastic gradient has a small variance O ( ϵ ) . Our algorithm is inspired by momentum-based variance reduction techniques used in nonconvex smooth optimization [18] under the almost-sure Lipschitz stochastic gradient oracle framework. The innovation of AccBO lies in its update rules: it employs normalized stochastic gradient descent with recursive momentum for the upper-level variable and stochastic Nesterov accelerated gradient descent with averaging for the lower-level variable. Our approach differs from existing accelerated bilevel optimization algorithms, such as those proposed by [71, 46] in two key ways: (i) while these algorithms use recursive momentum for the upper-level variable update, AccBO utilizes normalized recursive momentum to address the unbounded smoothness of the upper-level function; (ii) for the lower-level variable update, we use stochastic Nesterov accelerated gradient descent with averaging, in contrast to the recursive momentum method used by the other algorithms. The primary challenge in analyzing the convergence rate of AccBO arises from the need to simultaneously control errors from both upper-level and lower-level variables, given the unbounded smoothness, large learning rate, and recursive momentum in the upper-level problem. Our main contributions are summarized as follows.

We design a new algorithm named AccBO for solving bilevel optimization problems where the upper-level function is nonconvex with unbounded smoothness and the lower-level function is strongly convex. AccBO leverages normalized recursive momentum for the upperlevel variable and Nesterov momentum for the lower-level variable under the stochastic setting to achieve acceleration. To the best of our knowledge, the simultaneous usage of these two techniques in stochastic bilevel optimization is novel and has not been previously explored in the bilevel optimization literature.

We prove that the AccBO algorithm requires ˜ O ( ϵ -3 ) oracle calls for finding an ϵ -stationary point, when the variance of the lower-level stochastic gradient is O ( ϵ ) . This complexity strictly improves the state-of-the-art oracle complexity for unbounded smooth nonconvex upper-level problem and strongly-convex lower-level problem as described in [38, 32] 2 . To achieve this result, we introduce novel proof techniques for analyzing the dynamics of stochastic Nesterov accelerated gradient descent under distribution drift with high probability

for the lower-level variable, which are crucial for analyzing the hypergradient error and also of independent interest.

We empirically verify the effectiveness of our proposed algorithm on various tasks, including deep AUC maximization and data hypercleaning. Our algorithm indeed achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

2 Related Work

Relaxed Smoothness. The concept of relaxed smoothness was initially introduced by [75], inspired by the loss landscapes observed in recurrent neural networks and long-short term memory networks. They show that techniques such as gradient clipping and normalization could improve the performance compared with gradient descent in these scenarios. It inspired further investigations that concentrate on various aspects, including improved analysis on gradient clipping and normalization [74, 45], adaptive algorithms [15, 51, 68, 24], federated algorithms [54, 16, 17], generalized assumptions [15, 14], and recursive momentum based methods with faster rates [60, 56]. The work of [38, 32] considered a relaxed smoothness condition for the upper-level problem in the bilevel optimization setting.

Bilevel Optimization. Bilevel optimization refers to a special kind of optimization where one problem is embedded within another. It was first introduced by [9]. Early works developed specific bilevel optimization algorithms with asymptotic convergence analysis [67, 1, 69]. Ghadimi and Wang [30] initiated the study of non-asymptotic convergence for gradient-based methods in bilevel optimization where the upper-level problem is smooth and the lower-level problem is strongly convex. This field saw further advancements with improved complexity results [41, 44, 11, 21, 12] and fully first-order algorithms [48, 52]. There is a line of work which leverages almost-sure Lipschitz oracle (e.g., stochastic gradient) to obtain improved convergence rates of bilevel optimization algorithms [71, 46]. When the lower-level function is not strongly convex, several algorithmic framework and approximation schemes were proposed [61, 63, 49, 62, 55, 10]. The setting considered in this paper is very close to [38, 32], where the upper-level function is nonconvex and unbounded smooth, and the lower-level function is strongly convex. However, the work of [38, 32] do not have an accelerated rate ˜ O ( ϵ -3 ) for finding an ϵ -stationary point as established in this paper.

Nesterov Accelerated Gradient and Variants. Nesterov Accelerated Gradient (NAG) method was introduced by [58] for deterministic convex optimization problems. The stochastic version of NAG (SNAG) was extensively studied in the literature [3, 5, 66, 13]. To the best of our knowledge, none of them provide a high probability analysis for SNAG. In the online learning setting, there is a line of work focusing on the perspective of sequential stochastic/online optimization with distributional drift [6, 70, 57, 20]. While these studies provide valuable insights into adaptive techniques and performance bounds under distributional drift, they do not explore the potential integration of such methods with bilevel optimization problems, nor do they consider the application of SNAG within this framework.

3 Problem Setup and Preliminaries

Define ⟨· , ·⟩ and ∥ · ∥ as the inner product and the Euclidean norm. Throughout the paper, we use asymptotic notation ˜ O ( · ) , ˜ Θ( · ) , ˜ Ω( · ) to hide polylog factors in ϵ -1 and 1 /δ . Denote f : R d x × R d y → R as the upper-level function, and g : R d x × R d y → R as the lower-level function. The hypergradient ∇ Φ( x ) has the following form [30]:

To avoid the Hessian inverse computation, we typically use the following Neumann series to approximate the hypergradient [30, 46, 41]. In particular, for the stochastic setting, define

where q ( Q ) ∼ Uniform { 0 , . . . , Q -1 } , ¯ ξ := { ξ, ζ (0) , . . . , ζ ( q ( Q )) } and we use the convention that ∏ j i =1 A i = I if j = 0 . Then E ¯ ξ [ ¯ ∇ f ( x, y ; ¯ ξ )] is a good approximation of ∇ Φ( x ) if y and y ∗ ( x ) are close [30].

Throughout the paper, we make the following assumptions.

Assumption 3.1 ( ( L x, 0 , L x, 1 , L y, 0 , L y, 1 ) -smoothness [38]) . Let z = ( x, y ) and z ′ = ( x ′ , y ′ ) , there exists L x, 0 , L x, 1 , L y, 0 , L y, 1 > 0 such that for all z, z ′ , if ∥ z -z ′ ∥ ≤ 1 / √ 2( L 2 x, 1 + L 2 y, 1 ) , then ∥∇ x f ( z ) - ∇ x f ( z ′ ) ∥ ≤ ( L x, 0 + L x, 1 ∥∇ x f ( z ) ∥ ) ∥ z -z ′ ∥ and ∥∇ y f ( z ) - ∇ y f ( z ′ ) ∥ ≤ ( L y, 0 + L y, 1 ∥∇ y f ( z ) ∥ ) ∥ z -z ′ ∥ .

Remark : Assumption 3.1 is introduced by [38] for describing the bilevel optimization problems with recurrent neural networks. This assumption can be regarded as a block-wise relaxed smoothness assumptions for two blocks x and y , which is a variant of the relaxed smoothness assumption [75] and the coordinate-wise relaxed smooth assumption [15].

Assumption 3.2. Suppose the followings hold for objective functions f and g : (i) f is continuously differentiable and ( L x, 0 , L x, 1 , L y, 0 , L y, 1 ) -smooth in ( x, y ) ; (ii) For every x , ∥∇ y f ( x, y ) ∥ ≤ l f, 0 for all y ; (iii) For every x , g ( x, y ) is µ -strongly-convex in y for µ > 0 ; (iv) g is l g, 1 -smooth jointly in ( x, y ) ; (v) g is twice continuously differentiable, and ∇ 2 xy g, ∇ 2 yy g are l g, 2 -Lipschitz jointly in ( x, y ) .

Remark : Assumption 3.2 is standard in the bilevel optimization literature [48, 38, 30]. Assumption 3.2 (i) characterizes the unbounded smoothness of the upper-level function and is empirically observed in recurrent neural networks [38].

Assumption 3.3. The following stochastic estimators are unbiased and have the following properties:

Remark: Assumption 3.3 assumes the stochastic oracle for the upper-level problem has bounded variance, which is standard in nonconvex stochastic optimization [28-30]. It also assumes the stochastic oracle for the lower-level problem is light-tailed, which is common for the high probability analysis for the lower-level problem [50, 39]. Note that the same assumption is also made in [38, 32] for the bilevel problems with a unbounded smooth upper-level function.

Assumption 3.4. F ( x, y ; ξ ) and G ( x, y ; ζ ) satisfy Assumption 3.2 for every ξ and ζ almost surely.

Remark: Assumption 3.4 assumes that each random realization of the upper- and lower-level functions satisfies the same property as in the population level. Note that this condition is the key to achieve an improved ˜ O ( ϵ -3 ) oracle complexity under various settings, including both single-level nonconvex smooth problems [23, 18, 64] and bilevel problems with nonconvex smooth upper-level objectives [71, 46]. Furthermore, this assumption is shown to be necessary for achieving improved oracle complexity in single-level problems [2].

4 Algorithm and Analysis

4.1 Main Challenges and Algorithm Design

Main Challenges. We begin by explaining why existing bilevel optimization algorithms and their corresponding analysis techniques are insufficient in our setting. First, most algorithms developed for bilevel optimization require the upper-level function is smooth (i.e., the gradient of the upperlevel function is Lipschitz) [30, 44, 41, 71, 46, 21, 48]. They characterize the estimation error of the optimal solution for the lower-level problem, utilize an approximate hypergradient descent approach and the descent lemma for L -smooth functions to prove the convergence. In particular, they demonstrate that a potential function, incorporating both the function value and the bilevel error from the lower-level problem, progressively decreases in expectation. However, when the upper-level function is ( L x, 0 , L x, 1 , L y, 0 , L y, 1 ) -smooth as illustrated in Assumption 3.1, the previous algorithms and analyses relying on L -smoothness do not work. The reason is that the hypergradient bias depends on the approximation error of the lower-level variable as well as the hypergradient itself: these elements are statistically dependent and the standard potential function argument with an expectationbased analysis would not work. To address this issue, the work of [38, 32] requires a careful high probability analysis in the unbounded smoothness setting and obtains ˜ O ( ϵ -4 ) oracle complexity. Such a requirement of high probability analysis prevents us from leveraging the momentum-based

variance reduction technique for updating the lower-level variable. For example, the work [46] which has ˜ O ( ϵ -3 ) oracle complexity in the smooth case leverages the momentum-based variance reduction technique [18] for updating the lower-level variable with an expectation-based analysis, but it seems difficult to establish a high probability analysis for the momentum-based variance reduction algorithm in terms of the lower-level variable. Second, the recent work of Hao et al. [38] and Gong et al. [32] considered that the upper-level function is unbounded smooth and addressed this issue by performing normalized stochastic gradient with momentum for the upper-level variable and periodic updates or stochastic gradient descent for the lower-level variable, but their oracle complexity is not better than ˜ O ( ϵ -4 ) . These facts indicate that we need new algorithm design and analysis techniques to get potential acceleration.

Algorithm Design. To obtain potential acceleration and enable a high probability analysis for the lower-level variable, our key idea is to update the upper-level variable by normalized stochastic gradient descent with recursive momentum and update the lower-level variable by the stochastic Nesterov accelerated gradient (SNAG) method. Different from [38, 32], the key innovation of our algorithm design is that we achieve acceleration for both upper-level and lower-level problems simultaneously but without affecting each other. The upper-level update rule can be regarded as a generalization of the acceleration technique (e.g., the momentum-based variance reduction technique) [18, 56] from single-level to bilevel problems. The main challenge is that we need to deal with the accumulated error of the recursive momentum over time due to the hypergradient bias, which is caused by the inaccurate estimation of the optimal lower-level variable. Therefore we require a very small tracking error between the iterate of the lower-level variable and the optimal lower-level solution defined by the upper-level variable (i.e., y ∗ ( x ) ) at every iteration. This requirement is satisfied by executing SNAG method under the distribution drift for the lower-level problem, where the drift is caused by the change of the upper-level variable over time. Note that we can provide a high probability analysis of the SNAG method under distributional drift, which strictly improves the analysis of SGD under distributional drift in [19] in the small stochastic gradient noise regime.

The detailed description of our algorithm is illustrated in Algorithm 2. At the very beginning, we run a certain number of iterations of SNAG for the fixed upper-level variable x 0 (line 2 ) as the warm-start stage, and then update the lower-level variable by SNAG (line 8 ∼ 20 ) with averaging (line 21 ) and update the upper-level variable by normalized stochastic gradient descent with recursive momentum (line 23 ∼ 24 ). Note that we have two options for implementing SNAG. In Option I (line 8 ∼ 9 ), the algorithm simply runs SNAG under distribution drift caused by the sequence { x t } . Option I is specifically designed for a particular subset of bilevel optimization problems where the lower-level function is a quadratic function. Option II (line 11 ∼ 20 ) is designed for a broader range of bilvel optimization problems, accommodating general strongly-convex lower-level functions. In Option II, we run SNAG with periodic updates: the lower-level update is performed for N iterations only when the iteration number t is a multiple of I .

4.2 Main Results

We first introduce some useful notations. Let σ ( · ) be the σ -algebra generated by the random variables in the argument. We define the following filtrations for t ≥ 1 : F init = σ (˜ π 0 , . . . , ˜ π T 0 -1 ) , F t = σ ( ¯ ξ 0 , . . . , ¯ ξ t -1 ) , ˜ F 1 t = σ ( π 0 , . . . , π t -1 ) , and we also define ˜ F 2 t = σ ( π 0 t , . . . , π N -1 t ) when t is a multiple of I . We use E t , E F t and E to denote the conditional expectation E [ · | F t ] , the expectation over F t and the total expectation over F T respectively.

Theorem 4.1. Suppose Assumptions 3.1 to 3.4 hold. Let { x t } be the iterates produced by Algorithm 2. For any given δ ∈ (0 , 1) and small enough ϵ (see exact choice in (54) ), if σ g, 1 = O ( √ ϵ ) as defined in (55) , and we set parameters α init , α, β, γ, η, τ, I, N, S, Q, T 0 (see exact choices in (56) , (57) , (58) ,

(59) , and (60) ) as

Then with probability at least 1 -2 δ over the randomness in σ ( F init ∪ ˜ F 1 T ) (for Option I) or σ ( F init ∪ ( ∪ t ≤ T ˜ F 2 t )) (for Option II), Algorithm 2 guarantees 1 T ∑ T t =1 E ∥∇ Φ( x t ) ∥ ≤ 20 ϵ within T = 4 d 0 ηϵ = ˜ O ( ϵ -3 ) iterations, where d 0 := Φ( x 0 ) -inf x Φ( x ) and the expectation is taken over the randomness over F T . For Option I, it requires T 0 + SQT = ˜ O ( ϵ -3 ) oracle calls of stochastic gradient or Hessian/Jacobian vector product. For Option II, it requires T 0 + NT I + SQT = ˜ O ( ϵ -3 ) oracle calls of stochastic gradient or Hessian/Jacobian vector product.

Remark: Theorem 4.1 established an improved ˜ O ( ϵ -3 ) oracle complexity for finding an ϵ -stationary point when the lower-level standard deviation σ g, 1 = O ( √ ϵ ) . This complexity result strictly improves the ˜ O ( ϵ -4 ) obtained by [38, 32] when the upper-level function is nonconvex and unbounded smooth. This complexity result also matches that in the single-level unbounded smooth setting [56] and is nearly optimal in terms of the dependency on ϵ [2]. The full statement of Theorem 4.1 is included in Theorem E.2.

4.3 Proof Sketch

In this section, we provide a roadmap of proving Theorem 4.1 and the main steps. The detailed proofs can be found in Appendix D and E. The key idea is to prove two things: (1) the lower-level iterate is very close to the optimal lower-level variable at every iteration; (2) two consecutive iterates of the lower-level iterates are close to each other. In particular, define y ∗ t = y ∗ ( x t ) , and we aim to prove that ∥ ˆ y t -y ∗ t ∥ ≤ O ( ϵ ) and ∥ ˆ y t +1 -ˆ y t ∥ ≤ O ( ϵ 2 ) for every t . These two requirements are essential to control the hypergradient estimation error (i.e., ∥ m t -∇ Φ( x t ) ∥ ) caused by inaccurate estimate of the lower-level problem. Lemma 4.7 provides the guarantee for the lower-level problem, and Lemma 4.8

characterizes the hypergradient estimation error. Equipped with these two lemmas, we can adapt the momentum-based variance reduction techniques [18, 56] to the upper-level problem and prove the main theorem.

The main technical contribution of this paper is to provide a general framework for proving the convergence of SNAG under distributional drift in Section 4.3.1, which can be leveraged as a tool to control the lower-level error in bilevel optimization and derive the Lemma 4.7, as illustrated in Section 4.3.2. In particular, we can regard the change of the upper-level variable x at each iteration as the distributional drift for the lower-level problem: the drift is small due to the normalization operator of the upper-level update rule and also the Lipschitzness of y ∗ ( x ) . Once we have the general lemma for tracking the minimizer for any fixed distributional drift over time, this lemma can be applied to our algorithm analysis and establish guarantees for the bilevel problem.

4.3.1 Stochastic Nesterov Accelerated Gradient Descent under Distributional Drift

In this section, we study the sequences of stochastic optimization problems min w ∈ R d ϕ t ( w ) indexed by time t ∈ N . We denote the minimizer and the minimal value of ϕ t as w ∗ t and ϕ ∗ t , and we define the minimizer drift at time t to be ∆ t := ∥ w ∗ t -w ∗ t +1 ∥ . With a slight abuse of notation 3 , we consider the SNAG algorithm applied to the sequence { ϕ t } T t =1 , where T is the total number of iterations:

where g t = ∇ ϕ t ( z t ; ξ t ) is the stochastic gradient evaluated at z t with random sample ξ t . Define ε t = g t -∇ ϕ t ( z t ) as the stochastic gradient noise at t -th iteration. Define H t = σ ( ξ 1 , . . . , ξ t -1 ) as the filtration, which is the σ -algebra generated by all random variables until t -th iteration. We make the following assumption, which is the same as Assumption 3 in [20] for high probability analysis.

Assumption 4.2. Function ϕ t : R d → R is µ -strongly convex and L -smooth for constants µ, L > 0 . Also, there exists constants ∆ , σ > 0 such that the drift ∆ 2 t is sub-exponential conditioned on H t with parameter ∆ 2 and the noise ε t is norm sub-Gaussian conditioned on H t with parameter σ/ 2 . Lemma 4.3. Suppose Assumption 4.2 holds and let { w t } be the iterates produced by the update rule (3) with constant learning rate α ≤ 1 / 25 L , and set γ = 1 - √ µα 1+ √ µα . Define θ t = [( w t -w ∗ t ) ⊤ , ( w t -1 -w ∗ t ) ⊤ ] ⊤ ∈ R 2 d , and the potential function V t as

Then for any given δ ∈ (0 , 1) and all t ≥ 0 , the following holds with probability at least 1 -δ over the randomness in H t (here e denotes the base of natural logarithms):

(ii) (Without drift) Let ϕ t ( w ) ≡ ϕ ( w ) be any general strongly convex functions with ∆ = 0 , then V t ≤ ( 1 -√ µα 4 ) t V 0 + 5 √ ασ 2 √ µ ln eT δ .

Remark : When { ϕ t } T t =1 is a sequence of quadratic functions with moving minimizers, Lemma 4.3 provides a high probability tracking guarantee for SNAG with distributional drift, which is useful to provide guarantees for Option I in Algorithm 2. Note that this guarantee strictly improves the guarantee of stochastic gradient descent with distributional drift (e.g., [20, Theorem 6]) in the small stochastic gradient noise regime and therefore is of independent interest. In particular, for small α , the decaying factor in the first term is improved from 1 -µα 2 to 1 -√ µα 4 , the drift term is improved from ∆ 2 µα 2 to ∆ 2 α , and the variance term becomes a bit worse (from ασ 2 to √ ασ 2 √ µ ). When σ is small enough, the variance term becomes insignificant compared with the drift term, then Lemma 4.3 provides an improved convergence rate with high probability. To the best of our knowledge, such an improved guarantee for SNAG with distributional drift is first shown in this work. When there is no drift, Lemma 4.3 also provides a high probability guarantee for SNAG. It holds for any smooth and strongly convex function ϕ , and it is useful to provide guarantees for Option II of Algorithm 2.

4.3.2 Application of Stochastic Nesterov Accelerated Gradient to Bilevel Optimization

Inspired by [38, 32], we can regard ϕ t ( · ) as ϕ t ( · ) := g ( x t , · ) in the bilevel setting, and then we have ∆ t = ηl g, 1 /µ for every t due to the upper-level update rule and the Lipschitzness of y ∗ ( x ) . Therefore we can focus on the high probability analysis on the lower-level variable without worrying about the randomness from the upper-level. Throughout, we assume Assumption 3.1, 3.2, 3.3 and 3.4 hold. In addition, the failure probability δ ∈ (0 , 1) and ϵ > 0 are chosen in the same way as in Theorem 4.1.

Lemma 4.4 (Warm-start) . Let { y init t } be the iterates produced by line 2 of Algorithm 2. Set α init = ˜ Θ( ϵ 4 ) , σ g, 1 = ( µα ) 1 / 4 ˜ σ g, 1 , and ϕ t ( y ) ≡ g ( x 0 , y ) . Then ∥ y init T 0 -y ∗ 0 ∥ ≤ √ µα 32 ϵ L 0 holds with probability at least 1 -δ over the randomness in ˜ F init (we denote this event as E init ) in T 0 = ˜ O ( ϵ -2 ) iterations.

Remark: Lemma 4.4 shows that for fixed initialization x 0 , running SNAG for at most T 0 = ˜ O ( ϵ -2 ) iterations can guarantee that the Euclidean distance between the lower-level variable y init T 0 and the optimal solution y ∗ ( x 0 ) is at most O ( ϵ ) , with high probability.

Lemma 4.5 (Option I) . Under event E init , let { y t } be the iterates produced by Option I. Set α = ˜ Θ( ϵ 2 ) , σ g, 1 = ( µα ) 1 / 4 ˜ σ g, 1 , and ϕ t ( y ) = g ( x t , y ) = µ 2 ∥ y -y ∗ t ∥ 2 . Then for any t ∈ [ T ] , Algorithm 2 guarantees with probability at least 1 -δ over the randomness in ˜ F 1 T (we denote this event as E 1 y ) that ∥ y t -y ∗ t ∥ ≤ ϵ/ 2 L 0 .

Lemma 4.6 (Option II) . Under event E init , let { y t } be the iterates produced by Option II. Set α = ˜ Θ( ϵ 2 ) , N = ˜ O ( ϵ -1 ) , I = ˜ O ( ϵ -1 ) , σ g, 1 = ( µα ) 1 / 4 ˜ σ g, 1 , and ϕ t ( y ) = g ( x t , y ) when t is a multiple of I (i.e., x t is fixed for each update round of Option II so g can be general functions). Then for any t ∈ [ T ] , Algorithm 2 guarantees with probability at least 1 -δ over the randomness in σ ( ∪ t ≤ T ˜ F 2 t ) (we denote this event as E 2 y ) that ∥ y t -y ∗ t ∥ ≤ ϵ/L 0 .

Remark : Lemma 4.5 and Lemma 4.6 show that, under event E init and both option I and option II, the algorithm guarantees that each iterate y t is O ( ϵ ) -close to the the optimal lower-level variable y ∗ t at every iteration t with high probability.

Lemma 4.7 (Averaging) . Under event E init ∩ E 1 y (Option I) or E init ∩ E 2 y (Option II), set τ = √ µα in the averaging step (line 21 of Algorithm 2). Then for any t ≥ 0 we have ∥ ˆ y t -y ∗ t ∥ ≤ 2 ϵ L 0 and

Remark: Lemma 4.7 shows that after performing averaging operations over the sequence { y t } T t =1 , the averaged sequence enjoys stronger guarantees. First, each averaged iterate ˆ y t is still O ( ϵ ) -close to the optimal lower-level variable y ∗ t ; Second, two consecutive averaged iterates (i.e., ˆ y t and ˆ y t +1 ) is O ( ϵ 2 ) -close to each other. The stronger guarantees are crucial to control the hypergradient estimation error as described in Lemma 4.8.

Lemma 4.8. Under event E init ∩ E 1 y (Option I) or E init ∩ E 2 y (Option II), define ϵ t = m t -E t [ ¯ ∇ f ( x t , ˆ y t ; ¯ ξ t )] , then we have the following averaged cumulative error bound:

where S denotes the batch size, and ¯ σ, ¯ L 0 , ¯ L 1 are defined in Lemmas B.4 and B.6.

Remark: Lemma 4.8 characterizes the upper-level hypergradient estimation error under the good event that the lower-level error can be controlled. One can choose hyperparameters appropriately such that the cumulative error (i.e., LHS) grows only sublinearly in terms of T , which is important for establishing the fast convergence of our algorithm.

5 Experiments

Deep AUC Maximization with Recurrent Neural Networks . AUC (Area Under the ROC Curve) [36] is a critical metric in evaluating the performance of binary classification models. It measures the ability of the model to distinguish between positive and negative classes, and

0

5

10

15

20

25

Epoch

0.5

0.6

0.7

0.8

0.9

Train AUC

Train AUC vs. Epoch

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(a)

Training AUC

0

5

10

15

20

25

Epoch

0.55

0.60

0.65

0.70

0.75

Test AUC

Test AUC vs. Epoch

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(b)

Test AUC

0

100

200

300

400

500

600

running time /s

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Train AUC

Train AUC vs. running time (s)

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(c)

Training AUC vs. running time

0

100

200

300

400

500

600

running time /s

0.55

0.60

0.65

0.70

0.75

Test AUC

Test AUC vs. running time (s)

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(d)

Test AUC vs. running time

it is defined as the probability that the prediction score of a positive example is higher than that is a negative example [35]. Deep AUC maximization [53, 72] can be formulated as a minmax optimization problem [53]: min w ∈ R d , ( a,b ) ∈ R 2 max α ∈ R f ( w , a, b, α ) := E z [ F ( w , a, b, α ; z )] , where F ( w , a, b, α ; z ) = (1 -r )( h ( w ; x ) -a ) 2 I [ c =1] + r ( h ( w ; x ) -b ) 2 I [ c = -1] + 2(1 + α )( rh ( w ; x ) I [ c = -1] -(1 -r ) h ( w ; x ) I [ c =1] ) -r (1 -r ) α 2 , w denotes the model parameter, z = ( x , c ) is the random data sample ( x denote the feature vector and c ∈ { +1 , -1 } denotes the label), h ( w , x ) is the score function defined by a neural network, and r = Pr ( c = 1) denotes the ratio of positive samples in the population. This min-max formulation is an special case of the bilevel problem with g = -f in (1), which can be reformulated as the following:

where ( w , a, b ) denotes the upper-level variable, and α denotes the lower-level variable. In this case, the lower-level is a quadratic function in terms of α and is strongly convex, and the upper-level function is non-convex function with potential unbounded smoothness when using a recurrent neural network as the predictive model.

We aim to perform imbalanced text classification task and maximize the AUC metric. The Deep AUC maximization experiment is performed on imbalanced Sentiment140 [31] dataset (under the license of CC BY 4.0), which is a binary text classification task. Specifically, we follow [73] to make training set imbalanced with a pre-defined imbalanced ratio ( r ), and leave the test set unchanged. Given r , we randomly discard the positive samples (with label 1) in original training set until the portion of positive samples equals to r . The imbalance ratio r is set to 0.2 in our experiment, which means only 20% data is positive in the training set. We use a two-layer recurrent neural network with input dimension=300, hidden dimension=4096, and output dimension=2 for the model prediction.

We compare with some bilevel optimization baselines, including StocBio [44], TTSA [41], SABA [21], MA-SOBA [12], SUSTAIN [46], VRBO [71] and BO-REP [38]. We show the training and test AUC result with 25 epochs in (a) (b) of Figure 1 and running time in (c), (d) of Figure 1. Our algorithm AccBO achieves highest AUC score among all the baselines over epochs and running time. The running time figure shows AccBO converges to a good result faster than other baselines. The detailed parameter tuning and selection are included in Appendix G.

Data Hypercleaning. The Data hypercleaning task tries to learn a set of weights λ for the corrupted training data D tr , such that the model trained on the weighted corrupted training set can achieve good performance on the clean validation set D val , where the corrupted training set D tr := { x i , ¯ y i } and the label ¯ y i is randomly flipped to one of other labels with probability 0 < p < 1 . The data hyper-cleaning can be formulated as a bilevel optimization problem,

where w is the model parameter of a neural network, and σ ( x ) = 1 1+ e -x is the sigmoid function. We perform bilevel optimization algorithms on the noisy text classification dataset Stanford Natural Language Inference (SNLI) [8] (under the license of CC BY 4.0) with a three-layer recurrent neural network with input dimension=300, hidden dimension=4096, and output dimension=3 for the label prediction. Each of sentence-pairs manually labeled as entailment, contradiction, and neutral.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Epoch

0.4

0.5

0.6

0.7

0.8

Train Acc

Train Acc vs. Epoch

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(a)

Training ACC

0

5

10

15

Epoch

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Test Acc

Test Acc vs. Epoch

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(b)

Test ACC

0

500

1000

1500

2000

2500

running time /s

0.4

0.5

0.6

0.7

Train Acc

Train Acc vs. running time (s)

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(c)

Training ACC vs. running time

0

500

1000

1500

2000

2500

running time /s

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Test Acc

Test Acc vs. running time (s)

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(d)

Test ACC vs. running time

0

5

10

15

Epoch

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Train Acc

Train Acc vs. Epoch

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(e)

Training ACC

0

5

10

15

Epoch

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Test Acc

Test Acc vs. Epoch

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(f)

Test ACC

0

500

1000

1500

2000

2500

running time /s

0.35

0.40

0.45

0.50

0.55

0.60

Train Acc

Train Acc vs. running time (s)

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(g) Training ACC vs. running time

0

500

1000

1500

2000

2500

running time /s

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Test Acc

Test Acc vs. running time (s)

StocBio

TTSA

MA-SOBA

SUSTAIN

VRBO

BO-REP

AccBO

(h)

Test ACC vs. running time

Specifically, the label of each training data is randomly flipped to one of the other two labels with probability p . We set p = 0 . 1 and p = 0 . 2 in the experiments, respectively. We compare all the baselines used in the deep AUC maximization experiment. Different from the formulation (4) for the deep AUC maximization, the lower-level function in (5) is not quadratic function of the lower-level variable. Therefore we choose Option II in Algorithm 2, i.e., periodic updates for the lower-level variable. The results are presented in Figure 2 ( p = 0 . 1 and p = 0 . 2 ). Our algorithm AccBO exhibits the highest classification accuracy on training and test set among all the bilevel baselines, and also shows a high runtime efficiency. More detailed parameter tuning and selection can be found in Appendix G. All the experiments are run on the device of NVIDIA A6000 (48GB memory) GPU and AMDEPYC 7513 32-Core CPU.

6 Conclusion

In this paper, we propose a new algorithm named AccBO for solving bilevel optimization problems where the upper-level is nonconvex and unbounded smooth and the lower-level problem is strongly convex. The algorithm achieved ˜ O ( ϵ -3 ) oracle complexity for finding an ϵ -stationary point when the lower-level stochastic gradient variance is O ( ϵ ) , which matches the rate of the state-of-the-art single-level relaxed smooth optimization [56] and is nearly optimal in terms of dependency on ϵ [2].

Limitations. There are two limitations of our work. One limitation of our work is that the convergence analysis for the Option I of our algorithm relies on the lower-level problem being a quadratic function: only under this case the algorithm becomes a single-loop procedure. Another limitation is that we require the lower-level stochastic gradient has variance O ( ϵ ) . It remains unclear how to design single-loop algorithms for more general lower-level strongly convex functions and get rid of the small stochastic gradient variance assumption for the lower-level variable.

Acknowledgments and Disclosure of Funding

We would like to thank the anonymous reviewers for their helpful comments. We would like to thank Tianbao Yang and Qihang Lin for helpful discussions for the quadratic function with distributional drift in the earlier version of our paper. This work has been supported by the Presidential Scholarship, the ORIEI seed funding, and the IDIA P3 fellowship from George Mason University, the Cisco Faculty Research Award, and NSF award #2436217, #2425687. The Computations were run on ARGO, a research computing cluster provided by the Office of Research Computing at George Mason University (URL: https://orc.gmu.edu).

References

G Anandalingam and DJ White. A solution method for the linear static stackelberg problem using penalty functions. IEEE Transactions on automatic control , 35(10):1170-1173, 1990.

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199 (1-2):165-214, 2023.

Mahmoud Assran and Michael Rabbat. On the convergence of nesterov's accelerated gradient method in stochastic settings. arXiv preprint arXiv:2002.12414 , 2020.

Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban, and Asuman Ozdaglar. A universally optimal multistage accelerated stochastic gradient method. Advances in neural information processing systems , 32, 2019.

Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban, and Asuman Ozdaglar. Robust accelerated gradient methods for smooth strongly convex functions. SIAM Journal on Optimization , 30(1):717-751, 2020.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research , 63(5):1227-1244, 2015.

Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems , 33: 14879-14890, 2020.

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 , 2015.

Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations research , 21(1):37-44, 1973.

Lesi Chen, Jing Xu, and Jingzhao Zhang. On bilevel optimization without lower-level strong convexity. arXiv preprint arXiv:2301.00712 , 2023.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems , 34:25294-25307, 2021.

Xuxing Chen, Tesi Xiao, and Krishnakumar Balasubramanian. Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions. arXiv preprint arXiv:2306.12067 , 2023.

You-Lin Chen, Sen Na, and Mladen Kolar. Convergence analysis of accelerated stochastic gradient descent under the growth condition. Mathematics of Operations Research , 2023.

Ziyi Chen, Yi Zhou, Yingbin Liang, and Zhaosong Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. arXiv preprint arXiv:2303.02854 , 2023.

Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in neural information processing systems , 2022.

Michael Crawshaw, Yajie Bao, and Mingrui Liu. Episode: Episodic gradient clipping with periodic resampled corrections for federated learning with heterogeneous data. In The Eleventh International Conference on Learning Representations , 2023.

Michael Crawshaw, Yajie Bao, and Mingrui Liu. Federated learning with client subsampling, data heterogeneity, and unbounded smoothness: A new algorithm and lower bounds. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems , 32, 2019.

Joshua Cutler, Dmitriy Drusvyatskiy, and Zaid Harchaoui. Stochastic optimization under time drift: iterate averaging, step-decay schedules, and high probability guarantees. Advances in neural information processing systems , 34:11859-11869, 2021.

Joshua Cutler, Dmitriy Drusvyatskiy, and Zaid Harchaoui. Stochastic optimization under distributional drift. Journal of Machine Learning Research , 24(147):1-56, 2023.

Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems , 35:26698-26710, 2022.

Jeffrey L Elman. Finding structure in time. Cognitive science , 14(2):179-211, 1990.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal nonconvex optimization via stochastic path-integrated differential estimator. Advances in neural information processing systems , 31, 2018.

Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. arXiv preprint arXiv:2302.06570 , 2023.

Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated Machine Learning , pages 3-33. Springer, Cham, 2019.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning , pages 1568-1577. PMLR, 2018.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming , 156(1-2):59-99, 2016.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford , 1(12):2009, 2009.

Xiaochuan Gong, Jie Hao, and Mingrui Liu. A nearly optimal single loop algorithm for stochastic bilevel optimization under unbounded smoothness. In Forty-first International Conference on Machine Learning , 2024.

Riccardo Grazzi, Massimiliano Pontil, and Saverio Salzo. Bilevel optimization with a lower-level contraction: Optimal sample complexity without warm-start. arXiv preprint arXiv:2202.03397 , 2022.

Zhishuai Guo, Quanqi Hu, Lijun Zhang, and Tianbao Yang. Randomized stochastic variance-reduced methods for multi-task stochastic bilevel optimization. arXiv preprint arXiv:2105.02266 , 2021.

James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology , 143(1):29-36, 1982.

James A Hanley and Barbara J McNeil. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology , 148(3):839-843, 1983.

Jie Hao, Kaiyi Ji, and Mingrui Liu. Bilevel coreset selection in continual learning: A new formulation and algorithm. Advances in Neural Information Processing Systems , 36, 2023.

Jie Hao, Xiaochuan Gong, and Mingrui Liu. Bilevel optimization under unbounded smoothness: Anewalgorithm and convergence analysis. In The Twelfth International Conference on Learning Representations , 2024.

Elad Hazan and Satyen Kale. Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. Journal of Machine Learning Research , 15(1): 2489-2512, 2014.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actorcritic. SIAM Journal on Optimization , 33(1):147-180, 2023.

Bin Hu and Laurent Lessard. Dissipativity theory for nesterov's accelerated method. In International Conference on Machine Learning , pages 1549-1557. PMLR, 2017.

Quanqi Hu, Zi-Hao Qiu, Zhishuai Guo, Lijun Zhang, and Tianbao Yang. Blockwise stochastic variance-reduced methods with parallel speedup for multi-block bilevel optimization. In International Conference on Machine Learning , pages 13550-13583. PMLR, 2023.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning , pages 4882-4892. PMLR, 2021.

Jikai Jin, Bohang Zhang, Haiyang Wang, and Liwei Wang. Non-convex distributionally robust optimization: Non-asymptotic analysis. Advances in Neural Information Processing Systems , 34:2771-2782, 2021.

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems (NeurIPS) , 34:30271-30283, 2021.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. A fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning , pages 18083-18113. PMLR, 2023.

Jeongyeol Kwon, Dohyun Kwon, Steve Wright, and Robert Nowak. On penalty methods for nonconvex bilevel optimization and first-order stochastic approximation. arXiv preprint arXiv:2309.01753 , 2023.

Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133(1-2):365-397, 2012.

Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of adam under relaxed assumptions. arXiv preprint arXiv:2304.13972 , 2023.

Bo Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. Bome! bilevel optimization made easy: A simple first-order approach. Advances in Neural Information Processing Systems , 35:17248-17262, 2022.

Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. Stochastic auc maximization with deep neural networks. ICLR , 2020.

Mingrui Liu, Zhenxun Zhuang, Yunwen Lei, and Chunyang Liao. A communication-efficient distributed gradient clipping algorithm for training deep neural networks. Advances in Neural Information Processing Systems , 35:26204-26217, 2022.

Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International Conference on Machine Learning , pages 6305-6315. PMLR, 2020.

Zijian Liu, Srikanth Jagabathula, and Zhengyuan Zhou. Near-optimal non-convex stochastic optimization under generalized smoothness. arXiv preprint arXiv:2302.06032 , 2023.

Liam Madden, Stephen Becker, and Emiliano Dall'Anese. Bounds for the tracking error of first-order online optimization methods. Journal of Optimization Theory and Applications , 189: 437-457, 2021.

Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady , volume 27, pages 372-376, 1983.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems (NeurIPS) , pages 113-124, 2019.

Amirhossein Reisizadeh, Haochuan Li, Subhro Das, and Ali Jadbabaie. Variance-reduced clipping for non-convex optimization. arXiv preprint arXiv:2303.00883 , 2023.

Shoham Sabach and Shimrit Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

Han Shen and Tianyi Chen. On penalty-based bilevel gradient descent method. arXiv preprint arXiv:2302.05185 , 2023.

Daouda Sow, Kaiyi Ji, Ziwei Guan, and Yingbin Liang. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123 , 2022.

Quoc Tran-Dinh, Nhan H Pham, Dzung T Phan, and Lam M Nguyen. Hybrid stochastic gradient descent algorithms for stochastic nonconvex optimization. arXiv preprint arXiv:1905.05920 , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Sharan Vaswani, Benjamin Dubois-Taine, and Reza Babanezhad. Towards noise-adaptive, problem-adaptive (accelerated) stochastic gradient descent. In International conference on machine learning , pages 22015-22059. PMLR, 2022.

Luis Vicente, Gilles Savard, and Joaquim Júdice. Descent approaches for quadratic bilevel programming. Journal of optimization theory and applications , 81(2):379-399, 1994.

Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In The Thirty Sixth Annual Conference on Learning Theory , pages 161-190. PMLR, 2023.

Douglas J White and G Anandalingam. A penalty function approach for solving bi-level linear programs. Journal of Global Optimization , 3:397-419, 1993.

Craig Wilson, Venugopal V Veeravalli, and Angelia Nedi´ c. Adaptive sequential stochastic optimization. IEEE Transactions on Automatic Control , 64(2):496-509, 2018.

Junjie Yang, Kaiyi Ji, and Yingbin Liang. Provably faster algorithms for bilevel optimization. Advances in Neural Information Processing Systems , 34:13670-13682, 2021.

Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. In Advances in Neural Information Processing Systems , pages 451-459, 2016.

Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3040-3049, 2021.

Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. Advances in Neural Information Processing Systems , 33:1551115521, 2020.

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. International Conference on Learning Representations , 2020.

A Technical Lemmas

In this section, we will introduce a few useful lemmas. The following technical lemma on recursive control is crucial for providing high probability guarantee of the lower-level variables y t and ˆ y t in Algorithm 2 at anytime . We follow a similar argument as in [20, Proposition 29] with a slight generalization.

Lemma A.1 (Recursive control on MGF) . Consider scalar stochastic processes ( V t ) , ( V ′ t, 1 ) , ( V ′ t, 2 ) , ( D t, 1 ) , ( D t, 2 ) and ( X t ) on a probability space with filtration ( H t ) , which are linked by the inequality

for some deterministic constants α t ∈ ( -∞ , 1] and κ t ∈ R . Suppose the following properties hold.

V t , V ′ t, 1 and V ′ t, 2 are non-negative and H t -measurable.

D t,i is mean-zero sub-Gaussian conditioned on H t with deterministic parameter σ i , and V ′ t,i ≤ V t for i = 1 , 2 :

X t is non-negative and sub-exponential conditioned on H t with deterministic parameter ν t E [exp( λX t ) |H t ] ≤ exp( λν t ) for all 0 ≤ λ ≤ 1 /ν t .

Then the estimate

holds for any λ satisfying 0 ≤ λ ≤ min { 1 -α t 2( σ 2 1 + σ 2 2 ) , 1 2 ν t } .

Proof of Lemma A.1. For any index t ≥ 0 and any scalar λ ≥ 0 , the law of total expectation implies

Hölder's inequality in turn yields

provided 0 ≤ λ ≤ 1 / 2 ν t , where we use V ′ t,i ≤ V t for i = 1 , 2 in the last inequality. Therefore, under the condition that

the following estimate hold for all t ≥ 0 :

where the last inequality follows by the given range of λ . Thus the proof is completed.

Next, we introduce the following Young's inequality beyond Euclidean norm cases. This lemma serves as an important role when dealing with distributional drift for high probability SNAG analysis.

:

Lemma A.2 (Young's inequality) . For any vectors v 1 , v 2 ∈ R d , positive semidefinite (PSD) matrix Q ∈ R d × d , and scalar c > 0 , it holds that 4

Proof of Lemma A.2. By definition of ∥ · ∥ Q , we have

Since Q ∈ R d × d is PSD, let Q = UU ⊤ be the Cholesky decomposition, then

where we use Young's inequality and definition of ∥ · ∥ Q for the second and third lines, respectively. Combing (6) and (7) gives the result as claimed.

B Auxiliary Lemmas for Bilevel Optimization

In this section, we provide important properties of the objective function Φ in bilevel optimization problems, as well as characterizations (such as variance and bias) for stochastic hypergradient estimator ¯ ∇ f ( x, y ; ¯ ξ ) based on Neumann series. For readers' convenience, we only list the results here and defer the detailed proofs to Appendix F.

Lemma B.1 (Lipschitz property, [38, Lemma 8]) . Under Assumptions 3.1 and 3.2, y ∗ ( x ) is ( l g, 1 /µ ) -Lipschitz continuous.

Lemma B.2 ( ( L 0 , L 1 ) -smoothness, [38, Lemma 9]) . Under Assumptions 3.1 and 3.2, for any x, x ′ we have

where ( L 0 , L 1 ) -smoothness constant L 0 and L 1 are defined as

Lemma B.3 (Descent inequality, [38, Lemma 10]) . Suppose Assumptions 3.1 and 3.2 and 3.2 hold. Then for any x, x ′ we have

Lemma B.4 ([46, Lemma B.1]) . Under Assumptions 3.1 to 3.4, the bias of the stochastic hypergradient estimate of the upper-level objective satisfies

where Q is the number of samples chosen to approximate the Hessian inverse. Moreover, we have

Lemma B.5. Under Assumptions 3.1 to 3.4, we have

where constant ¯ L is defined as

Lemma B.6. Under Assumptions 3.1 to 3.4, we have

(ii) For any fixed x ∈ R d x and any y, y ′ ∈ R d y ,

In the above expressions, we define ¯ L 0 and ¯ L 1 as

Note that in Lemma B.6, constant ¯ L 0 depends on the value of ∥ y -y ∗ ( x ) ∥ . When we consider this term in Algorithm 2, it turns into ∥ y t -y ∗ t ∥ or ∥ ˆ y t -y ∗ t ∥ , which are both as small as O ( ϵ ) (and thus bounded) with high probability by Lemmas 4.5 to 4.7. In other words, we can treat this term as another constant for our algorithm and analysis.

C Proofs of Results in Section 4.3.1

For convenience, we will restate a few concepts included in Section 4.3.1 here. We consider the sequences of stochastic optimization problems

indexed by time t ∈ N . We denote the minimizer and the minimal value of ϕ t as w ∗ t and ϕ ∗ t , and we define the minimizer drift at time t to be ∆ t := ∥ w ∗ t -w ∗ t +1 ∥ . With a slight abuse of notation, we consider the SNAG algorithm applied to the sequence { ϕ t } T t =1 , where T is the total number of iterations:

where g t = ∇ ϕ t ( z t ; ξ t ) is the stochastic gradient evaluated at z t with random sample ξ t . Define ε t = g t -∇ ϕ t ( z t ) as the stochastic gradient noise at t -th iteration. Define H t = σ ( ξ 1 , . . . , ξ t -1 ) as the filtration, which is the σ -algebra generated by all random variables until t -th iteration. We will make the following standard assumption, as illustrated below 5 .

Assumption C.1. The sequences of time-varying functions satisfy that, each function ϕ t : R d → R is µ -strongly convex and L -smooth for some constants µ, L > 0 .

Assumption C.2 (Sub-Gaussian drift and noise) . There exists constants ∆ , σ > 0 such that the following holds for all t ≥ 0 :

(i) ( Drift ) The drift ∆ 2 t is sub-exponential conditioned on H t with parameter ∆ 2 :

(ii) ( Noise ) The noise ε t is norm sub-Gaussian conditioned on H t with parameter σ/ 2 :

The following lemma characterize the one-step improvement for stochastic Nesterov accelerated gradient method. Although part of our analysis is similar to [13, 5], our final goal is quite different: we aim to derive a careful formulation (see (13)) such that we can apply Lemma A.1 to recursively control the moment generating function of V t with distributional drift, thus leading to a high probability bound for V t at anytime (see Lemma C.5), while [13, 5] only show the convergence in expectation without distributional drift.

Lemma C.3 (Distance recursion, with drift) . Suppose that Assumptions C.1 and C.2 hold. Let { w t } be the iterates produced by update rule (9) with constant learning rate α ≤ 1 / 2 L and set constants γ, ρ > 0 , and matrix P ∈ R 2 d × 2 d as

Define θ t = [( w t -w ∗ t ) ⊤ , ( w t -1 -w ∗ t ) ⊤ ] ⊤ ∈ R 2 d , also define the potential function and u t, 1 , u t, 2 as

Then for all t ≥ 0 , it holds that

Specifically, if ϕ t ( w ) := µ 2 ∥ w -w ∗ t ∥ 2 , then we have

Proof of Lemma C.3. We first apply Lemma A.2 with

to obtain

Now we bound terms ( A ) , ( B ) and ( C ) individually.

Bounding ( A ) . Let us define vector ω t ∈ R d and matrices A ∈ R 2 d × 2 d , B ∈ R 2 d × d as

By (9) we have

Since ϕ t is µ -strongly convex, then

By L -smoothness of ϕ t and the fact that w t +1 = z t -αg t , we have

where we use g t = ∇ ϕ t ( z t ) + ε t in the second equality. Noting that by (9) we have

and combining (15) and (16) we obtain

where matrix X 1 ∈ R 3 d × 3 d is defined as

Then applying the strong convexity of ϕ t again gives

Noting that

and combining (16) and (18) we obtain

where matrix X 2 ∈ R 3 d × 3 d is defined as

Next, we multiply (17) by ρ 2 and (19) by 1 -ρ 2 , then sum them up to get ρ 2 ( ϕ t ( w t ) -ϕ t ( w ∗ t )) -( ϕ t ( w t +1 ) -ϕ t ( w ∗ t )) ≥ [ θ t ω t ] ⊤ ( ρ 2 X 1 +(1 -ρ 2 ) X 2 ) [ θ t ω t ] -Lα 2 2 ∥ ε t ∥ 2 + α (1 -Lα ) ⟨∇ ϕ t ( z t ) , ε t ⟩ . (20)

By [42, Section 3.1] (see also [5, Corollary 4.9], [4, Theorem 2.3]), we have the following fact:

which, combined with (20) yields

where the first inequality uses (14) and (21), along with the fact that B ⊤ PB = ( α/ 2) ⊗ I d and hence λ max ( B ⊤ PB ) = α/ 2 ; and the last inequality follows by (20). Rearrange the above inequality and by definition of the potential function V t , we obtain

Now recall that in (8) our objective function has the form of ϕ t ( w ) = µ 2 ∥ w -w ∗ t ∥ 2 , hence ∇ ϕ t ( z t ) = µ ( z t -w ∗ t ) . Plugging this into the above inequality gives

where u t, 1 and u t, 2 are defined as

Therefore, we conclude that

where the last inequality follows from the definition of ρ and simple calculation

Bounding ( B ) . Recall that under distributional drift, the objective function in (8) has the form of ϕ t ( w ) = µ 2 ∥ w -w ∗ t ∥ 2 , then we have

Since ϕ t +1 is µ -strongly convex and matrix P is PSD, then

Plugging the above fact back into the upper bound for ( B ) gives

Bounding ( C ) . For this part, we handle the distributional drift. By definition of P in (10), we have

where in the last equality we use the basic algebra of block matrix multiplication and the definition of ∆ t = ∥ w ∗ t -w ∗ t +1 ∥ .

Final Bound for V t +1 . Now we are ready to derive the upper bound for V t +1 . Combining (24), (25) and (26) together yields

For simplicity, we define D as the following

Hence (27) turns into

Solving the above inequality we get

Then an application of Young's inequality reveals

where we plug in the definition of D for the first equality. Since the learning rate α ≤ 1 /L and thus µα ≤ 1 , then we have

Therefore, we finally conclude that

which is as claimed in (13).

When there is no drift, the following lemma holds for any general strongly convex functions ϕ in R d . Lemma C.4 (Distance recursion, without drift) . Under the same settings as in Lemma C.3 with ϕ t ( w ) ≡ ϕ ( w ) , and w ∗ t ≡ w ∗ , where ϕ ( w ) can be any general strongly functions in R d . We redefine u t, 1 , u t, 2 as

Then for all t ≥ 0 , it holds that

Proof of Lemma C.4. By (12) in Lemma C.3 with ϕ t ( w ) ≡ ϕ ( w ) and w ∗ t ≡ w ∗ , we have

Hence the proof is completed.

The following result shows the first part of Lemma 4.3. To the best of our knowledge, this is the first high probability guarantee with improved rate for SNAG under distributional drift.

Lemma C.5 (High-probability distance tracking, with drift) . Under the same setting as in Lemma C.3 with α ≤ 1 / 25 L , for any given δ ∈ (0 , 1) and all t ∈ [ T ] , the following holds with probability at least 1 -δ over the randomness in H t :

Proof of Lemma C.5. We will invoke Lemma A.1 to show the results. To apply Lemma A.1, we first need to show the following two facts:

Fact (I) verification. Since ϕ t is µ -strongly convex and matrix P is PSD, then

Fact (II) verification. By definition of matrix P and simple calculation we have

Rearrange the above inequality, and recall the update rule of stochastic Nesterov accelerated gradient method, we have

where for the last equality we use the definition of γ as in (10). Rearrange it gives (31).

By Lemma C.3 and the choice of α ≤ 1 /L , we have

Note that under Assumption C.2, there exists an absolute constant c ≥ 1 such that for all t ≥ 0 , ∥ ε t ∥ 2 is sub-exponential conditioned on H t with parameter cσ 2 , and ε t is mean-zero sub-Gaussian conditioned on H t with parameter cσ [20, Theorem 30]. For convenience we simply let c = 1 here. Thus ⟨ u t, 1 , -ε t ⟩ is mean-zero sub-Gaussian conditioned on H t with parameter σ , and ∆ 2 t is sub-exponential conditioned on H t with parameter ∆ 2 by assumption. Hence, in light of (32), we apply Lemma A.1 with

yielding the following recursion

for all λ satisfying

We then apply (33) recursively to deduce

for all λ satisfying

Moreover, setting

and taking into account α ≤ 1 / 25 L , then we have

and

Thus we obtain

Taking λ = 1 /ν and applying Markov's inequality and union bound completes the proof.

The following result shows the second part of Lemma 4.3.

Lemma C.6 (High-probability distance tracking, without drift) . Under the same setting as in Lemma C.4 with α ≤ 1 / 25 L , for any given δ ∈ (0 , 1) and all t ∈ [ T ] , the following holds with probability at least 1 -δ over the randomness in H t :

Proof of Lemma C.6. First it is easy to verify that

Then we apply Lemma A.1 to obtain the final result. We omit the detailed proof here since it follows the same procedure as in proof of Lemma C.5.

D Proofs of Results in Section 4.3.2

We first present the following algebraic fact under suitable choice of parameters.

Lemma D.1 (Parameter choice, informal) . For any given δ ∈ (0 , 1) and any small ϵ satisfying

if we set parameters α, β, η, T as

where ¯ σ is defined in Lemma B.4, and d 0 and P are defined as

Then the following holds for all t ∈ [ T ] :

Proof of Lemma D.1. By Lemma B.1, we have ∆ t = ∥ y ∗ t -y ∗ t +1 ∥ ≤ l g, 1 µ ∥ x t -x t +1 ∥ = ηl g, 1 /µ . Thus in our bilevel setting, we choose ∆ = ηl g, 1 /µ , where ∆ is defined in Section 4.3.1. By choice of α, η, T as in (36), we have

Now we choose β to be

Then we have

where we use the fact that ln( P ln( P )) ≤ ln( P ) ≤ ln 2 ( P ) for any P ≥ 4 by choice of ϵ as in (35).

In the rest of this section, we assume Assumptions 3.1 to 3.4 hold. In addition, the failure probability δ ∈ (0 , 1) and ϵ > 0 are chosen in the same way as in Theorem 4.1.

D.1 Proof of Lemma 4.4

Lemma D.2 (Warm-start, Restatement of Lemma 4.4) . Let { y init t } be the iterates produced by line 2 of Algorithm 2. Set α init = µα 2 = ˜ Θ( ϵ 4 ) with α defined in (36) , σ g, 1 = ( µα ) 1 / 4 ˜ σ g, 1 , and ϕ t ( y ) ≡ g ( x 0 , y ) . Then ∥ y init T 0 -y ∗ 0 ∥ ≤ √ µα 32 ϵ L 0 holds with probability at least 1 -δ over the randomness in ˜ F init (we denote this event as E init ) in T 0 = ˜ O ( ϵ -2 ) iterations, where

Proof of Lemma D.2. By Lemmas C.6 and D.1 and µ -strong convexity of g in y , we have with probability at least 1 -δ over the randomness in F init that

where the first inequality uses the choice of α init = µα 2 . By l g, 1 -smoothness of g we have

where the last inequality uses α ≤ 1 /l g, 1 . Now we set

By choice of α as in (36) and simple calculation we obtain T 0 = ˜ O ( ϵ -2 ) when ϵ is small.

D.2 Proof of Lemma 4.5

Lemma D.3 (Option I, Restatement of Lemma 4.5) . Under event E init , let { y t } be the iterates produced by Option I. Set α = ˜ Θ( ϵ 2 ) as in (36) , σ g, 1 = ( µα ) 1 / 4 ˜ σ g, 1 , and ϕ t ( y ) = g ( x t , y ) = µ 2 ∥ y -y ∗ t ∥ 2 . Then for any t ∈ [ T ] , Algorithm 2 guarantees with probability at least 1 -δ over the randomness in ˜ F 1 T (we denote this event as E 1 y ) that ∥ y t -y ∗ t ∥ ≤ ϵ/ 2 L 0 .

Proof of Lemma D.3. By Lemmas C.5 and D.2 we have

Thus we conclude that for all t ∈ [ T ] , we have ∥ y t -y ∗ t ∥ ≤ ϵ/ 2 L 0 .

which gives

D.3 Proof of Lemma 4.6

Lemma D.4 (Option II, Restatement of Lemma 4.6) . Under event E init , let { y t } be the iterates produced by Option II. Set α = ˜ Θ( ϵ 2 ) as in (36) , σ g, 1 = ( µα ) 1 / 4 ˜ σ g, 1 , and run SNAG in each update round for

steps in every I = µϵ 2(1 -β ) L 0 ˜ σ g, 1 = ˜ O ( ϵ -1 ) iterations, set ϕ t ( y ) = g ( x t , y ) when t is a multiple of I (i.e., x t is fixed for each update round of Option II so g can be general functions). Then for any t ∈ [ T ] , Algorithm 2 guarantees with probability at least 1 -δ over the randomness in σ ( ∪ t ≤ T ˜ F 2 t ) (we denote this event as E 2 y ) that ∥ y t -y ∗ t ∥ ≤ ϵ/L 0 .

Proof of Lemma D.4. At the beginning of the first round, by Lemmas D.2 and D.3 we have ∥ y 0 -y ∗ 0 ∥ ≤ ϵ/ 2 L 0 , then we do not update the lower-level variable until t = I -th iteration, then for t = I , we have

where in the last equality we plug in the definition of η and I . By l g, 1 -smoothness of g we have

Then for N steps update in the inner loops of t = I -th iteration, we set

which gives

By choice of α as in (36) and simple calculation we obtain N = ˜ O ( ϵ -1 ) when ϵ is small. Now we have

which yields

where we choose 1 -β to be small (see (56) for details) such that η is small enough to make above inequality holds. Repeating the same process yields the result.

D.4 Proof of Lemma 4.7

Lemma D.5 (Averaging, Restatement of Lemma 4.7) . Under Assumptions 3.1 to 3.4 and event E init ∩E 1 y (Option I) or E init ∩E 2 y (Option II), we further set τ = √ µα in the averaging step (line 21 of Algorithm 2). Then for any t ≥ 0 we have

Proof of Lemma D.5. We will first show the following result by induction, i.e., for any t ≥ 0 , the averaged sequence { ˆ y t } satisfies

For t = 0 , by Lemma D.2 we have

thus the base case holds. Now suppose (39) holds for some t ≥ 0 , then for time step t +1 we have

where we use induction hypothesis in the second inequality. Therefore, we have that (39) holds for any t ≥ 0 . Also, as a consequence, for any t ≥ 0 we have

Now we plug in the definition of α, β, τ, η as in (36) to obtain

and

D.5 Proof of Lemma 4.8

Lemma D.6 (Restatement of Lemma 4.8) . Under Assumptions 3.1 to 3.4 and event E init ∩E 1 y (Option I) or E init ∩E 2 y (Option II), define ϵ t = m t -E t [ ¯ ∇ f ( x t , ˆ y t ; ¯ ξ t )] , then we have the following averaged cumulative error bound:

Proof of Lemma D.6. Define ϵ t = m t -E t [ ¯ ∇ f ( x t , ˆ y t ; ¯ ξ t )] , also define ˜ ϵ t and ˆ ϵ t as

By definition of ϵ t , ˜ ϵ t and ˆ ϵ t , we have the following recursion for any t ≥ 0 :

Then we apply (40) recursively to obtain

which by triangle inequality and total expectation gives

Bounding Err 1 . By definition of ϵ 0 and Lemma B.4, along with Jensen's inequality, we have

Bounding Err 2 . We apply Lemma B.4 and follow the similar procedure as in [38, Lemma D.9] to obtain

where we use Jensen's inequality for the first step.

Bounding Err 3 . We will first use induction to show that for 0 ≤ i ≤ t +1 , the following inequality holds:

Then it's easy to check that by setting i = t +1 we can obtain the bound. When i = 0 , (44) holds obviously since

Hence the base case stands. Now suppose (44) holds for some i ≥ 0 , and we aim to show that (44) holds for i +1 . In fact, we have

where (46) follows by law of total expectation, (47) follows by Jensen's inequality, (48) uses the fact that ˆ ϵ j for j < t -i are F t -i -1 -measurable, and are uncorrelated with ˆ ϵ t -i ; for (49) we use Lemmas B.6 and D.5 to derive

And (50) follows from the fact that x t -i is F t -i -1 -measurable, (51) uses √ a + b ≤ √ a + √ b for a, b ≥ 0 . Hence the induction proof is completed. We set i = t +1 to obtain

Final Bound. Combining (42), (43) and (52) yields

Taking summation and dividing 1 /T on both sides gives the final result

E Proof of Theorem 4.1

Before starting the proof of main results, i.e., Theorem 4.1, we first need the following lemma.

Lemma E.1. Suppose that Assumptions 3.1 to 3.4 hold. For any η satisfying

it holds that

Proof of Lemma E.1. Define h t = m t -∇ Φ( x t ) . Then we apply Lemma B.3 to obtain

where for the last two lines we use Cauchy-Schwarz inequality and ∥ h t ∥ = ∥∇ Φ( x t ) + h t ∥ ≥ ∥∇ Φ( x t ) ∥ - ∥ h t ∥ . Now expanding h t by triangle inequality, we have

where we use definition of ϵ t , Lemmas B.4 and B.5 in the last inequality. Plugging the above inequality back into (53) we obtain

Dividing 1 /Tη on both sides, then taking telescope sum and total expectation, and rearranging it finally yields

Theorem E.2 (Restatement of Theorem 4.1) . Suppose Assumptions 3.1 to 3.4 hold. Let { x t } be the iterates produced by Algorithm 2. For any given δ ∈ (0 , 1) and any small ϵ > 0 satisfying

if σ g, 1 satisfies

with ˜ σ g, 1 = O (1) , and we set parameters α, α init , β, γ, η, τ, I, N, S, Q, T 0 as

where d 0 and P are defined in (37) . Then with probability at least 1 -2 δ over the randomness in σ ( F init ∪ ˜ F 1 T ) (for Option I) or σ ( F init ∪ ( ∪ t ≤ T ˜ F 2 t )) (for Option II), Algorithm 2 guarantees 1 T ∑ T t =1 E ∥∇ Φ( x t ) ∥ ≤ 20 ϵ within T = 4 d 0 ηϵ = ˜ O (1 /ϵ 3 ) iterations, where the expectation is taken over the randomness in F T . For Option I, it requires T 0 + SQT = ˜ O (1 /ϵ 3 ) oracle calls of stochastic gradient or Hessian/Jacobian vector product. For Option II, it requires T 0 + NT I + SQT = ˜ O (1 /ϵ 3 ) oracle calls of stochastic gradient or Hessian/Jacobian vector product.

Proof of Theorem E.2. By Lemmas D.6 and E.1, we have

Rearranging the above inequality gives

Bounding (LHS). By Lemma D.5, we have

Bounding (RHS). By choice of parameters, we have

Combining (61) and (62) finally yields

F Omitted Proofs in Appendix B

F.1 Proof of Lemma B.5

Lemma F.1 (Restatement of Lemma B.5) . Under Assumptions 3.1 to 3.4, we have

where constant ¯ L is defined as

Proof of Lemma B.5. Recall that the exact expressions of ¯ ∇ f ( x, y ) and ∇ Φ( x ) are

and

Then by Assumption 3.2 we have

By definition of L we conclude the proof.

F.2 Proof of Lemma B.6

Lemma F.2 (Restatement of Lemma B.6) . Under Assumptions 3.1 to 3.4, we have

In the above expressions, we define ¯ L 0 and ¯ L 1 as

Proof of Lemma B.6. We show statement ( i ) of the lemma, and ( ii ) follows by similar arguments. For any fixed y ∈ R d y and any x 1 , x 2 ∈ R d x , by definition of ¯ ∇ f ( x, y ; ¯ ξ ) we have

For the second term above, we have

Then we take expectation with respect to q and obtain

Finally, taking expectation on both sides yields

Since for any y ∈ R d y , we have

which yields

Therefore, we conclude that

where we use the definition of ¯ L 0 and ¯ L 1 in the last equality.

G Additional Experimental Details

Hyerparameter setting. We tune the best hyperparameters for each algorithm, including upper/lower-level step size, the number of inner loops, momentum parameters, etc. The upper-level learning rate η up and lower-level learing rate η low are tuned in the range of [0 . 001 , 0 . 1] for all the baselines on experiments of AUC maximization and data hyper-cleaning, the best ( η up , η low ) on AUC maximization are summarized as follows: StocBio: ( 0 . 01 , 0 . 001 ), TTSA: (0 . 005 , 0 . 01) , SABA: (0 . 01 , 0 . 005) , MA-SOBA: (0 . 01 , 0 . 005) , SUSTAIN: (0 . 03 , 0 . 01) , VRBO: (0 . 05 , 0 . 01) , BO-REP: (0 . 001 , 0 . 001) , AccBO: (0 . 005 , 0 . 005) . The best learning rate on the experiment of data hyper-cleaning are summarized as follows: Stocbio: (0 . 01 , 0 . 002) , TTSA: (0 . 001 , 0 . 01) , SABA: (0 . 05 , 0 . 02) , MA-SOBA: (0 . 01 , 0 . 01) , SUSTAIN: (0 . 05 , 0 . 05) , VRBO: (0 . 1 , 0 . 05) , BOREP: (0 . 02 , 0 . 01) , AccBO: (0 . 1 , 0 . 1) . Note that SUSTAIN decays its upper-/lower-level step size with epoch ( t ) by η up = η up / ( t +2) 1 / 3 , η low = η up / ( t +2) 1 / 3 , while other algorithms use a constant learning rate. The number for neumann series estimation in StocBiO and VRBO is fixed to 3, while it is uniformly sampled from { 1 , 2 , 3 } in TTSA, SUSTAIN, and AccBO. In AUC maximization, AccBO uses Option I (Option II in data hyper-cleaning) to update the lower-level variable, and sets the Nestrov momentum parameter γ = 0 . 5 , the averaging parameter τ = 0 . 5 ( γ = 0 . 1 and τ = 0 . 5 in data hyper-cleaning). In AUC maximization, the batch size is set to be 32 for all algorithms except VRBO, which uses larger batch size of 64 (tuned in the range of { 32 , 64 , 128 , 256 , 512 } ) at the checkpoint step and 32 otherwise. In data hyper-cleaning, the batch size is set to be 128 for all algorithms except VRBO, which uses larger batch size of 256 (tuned in the range of { 63 , 128 , 256 , 512 , 1024 } ) at the checkpoint step and 128 otherwise. AccBO uses Option II in data hyper-cleaning, and the periodical update for low-level variable sets the iterations N = 3 and update interval I = 2 . Other hyperparameters setting keep the same in AUC maximization and data hyper-cleaning: The momentum parameter β is fixed to 0 . 9 in AccBO, MA-SOBA, BO-REP. The warm start steps for lower-level variable in AccBO is set to 3 . The number of inner loops for StocBio is set to 3 . BO-REP uses the periodical update for low-level variable, and set the iterations N = 3 and the update interval I = 2 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Every claim made in the abstract is specified a section of the paper, including algorithm design and analysis in Section 4 and experiments in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations of our work in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide both assumptions and proofs in Appendices C to E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental details are fully specified in Section 5 and Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data are attached as a supplement with instructions for reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details are included in Section 5 and Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We only run once due to limited computational budget.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The hardware specification is described in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and conformed to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning from algorithmic and theoretical aspects. We do not see any direct paths to negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not involve the release of any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Our paper uses existing text classification datasets and are cited in Section 5 and their licenses are mentioned.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work develops the first algorithm that achieves the near-optimal oracle complexity $\\widetilde{O}(\\epsilon^{-3})$ (the number of evaluations to gradient or Hessian/Jacobian vector product) to achieve an $\\epsilon$-stationary point of a stochastic bilevel optimization problem with unbounded smooth nonconvex upper-level function and strongly-convex lower-level function. The algorithm design is novel which updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and updates the lower-level variable by the stochastic Nesterov accelerated gradient (SNAG) method.\n\nSTRENGTHS:\nThis work studies a specific stochastic bilevel optimization problem with significant applications in sequential data learning. The upper-level function has unbounded smoothness which is more general than the widely used Lipscthiz smoothness assumption. The oracle complexity is improved on this specific bilevel optimization problem compared with existing works. The algorithm design not only combines but also adjusts STORM variance reduction and Nesterov acceleration. The 2 real-world experiments look convincing. The presentation is clean and clear.\n\nWEAKNESSES:\nThe major weakness is Assumption 4.2 as mentioned in the question (1) below. Some points could be clarified as mentioned in other questions below.\n\nQUESTIONS:\n(1) **My major concern** is that Assumption 4.2 is made on algorithm-generated variables not on problem setting, which cannot be verified. Is it possible to bound $\\Delta_t$ in terms of $\\|x_{t+1}-x_t\\|$ to get rid of Assumption 4.2? If not, at least Assumption 4.2 should also be mentioned in the main result (Theorem 4.1). **I would like to raise my rating if all the assumptions become problem-related.**\n\n(2) In line 48 \"each realization of the stochastic oracle calls is Lipschitz with respect to its argument\", do you mean $F(x,y,\\xi)$ and $G(x,y,\\xi)$ are Lipschitz continuous functions of $(x,y)$ for each $\\xi$? You might express in formula to make it more clear. \n\n(3) You could also cite [1] as a paper on relaxed smoothness:  \n[1] Convex and Non-convex Optimization Under Generalized Smoothness\n\n(4) In line 127, do you mean \"... is a good approximation of $\\nabla \\Phi(x)$ **if $y\\approx y^*(x)$**\"? \n\n(5) Is there an intuition why line 21 (averaging of y) is needed in Algorithm 2? Is it possible to implement $N=\\mathcal{O}(1)$ steps of SNAG such that $\\mathbb{E}\\|y_{t+1}-y^*(x_t)\\|\\le \\frac{1}{2}\\mathbb{E}\\|y_t-y^*(x_t)\\|$ and remove averaging of y for every outer iteration t, and obtain convergence result purely in expectation instead of high probability? This yields the same oracle complexity per round since $I,N=\\widetilde{\\mathcal{O}}(\\epsilon^{-1})$ are of the same order, and may get rid of Assumption 4.2 by bounding $\\Delta_t$ in terms of $\\|x_{t+1}-x_t\\|$.\n\n(6) You said your upper-level algorithm design is inspired from [17,49]. However, in line 23 (STORM variance reduction) of Algorithm 2, your second and third additive terms use the same samples $\\overline{\\xi}_t$ while [49] uses different samples. The algorithm design differs from [17] more. Such difference still exists even if Algorithm 2 is reduced to single-level. Why do you use such different design? \n\n(7) Also in line 23, does $\\overline{\\nabla}f(x_t,\\hat{y}_t;\\overline{\\xi}_t)$ denote  the average of the expression right below Eq. (2) over samples $s=1,\\ldots,S$? You could define more clearly. \n\n(8) At the beginning of Section 5 experiments, $ph(w; x)I_{[y=-1]}$ appears in your AIC maximization. Do you mean $rh(w; x)I_{[c=-1]}$? If not, what do $p$ and $y$ mean?\n\nLIMITATIONS:\nThe conclusion section mentions the limitation that the convergence analysis for the Option I of Algorithm 2 relies on the lower-level problem being a one-dimensional quadratic function. Actually I think this is fine since Option II works for more general strongly convex lower-level function. The major limitation from my perspective in mentioned in my above question (1) about the undesired Assumption 4.2. \nThe authors said in the checklist that they do not see any negative societal impacts in their theoretical work, which I agree.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider bilevel optimization problems under the unbounded smoothness assumption of the outer function. They propose AccBO, an AID-based method that uses Stochastic Nesterov Accelerated Gradient to approximate the lower-level solution and Neumann approximations to estimate the inverse Hessian-Vector product involved in the hypergradient expression. Then, a normalized STORM-like update is performed for the outer variable. They show a $\\tilde{\\mathcal{O}}(\\epsilon^{-3})$ complexity of their algorithm. Finally, numerical experiments are provided on the deep AUC maximization and data hypercleaning problems.\n\nSTRENGTHS:\n- **S1**: The paper is well-written\n- **S2**: The algorithms proposed by the authors achieve SOTA complexity for stochastic bilevel optimization under unbounded smoothness assumption.\n\nWEAKNESSES:\n### Major\n\n- **W1**: Some errors in the code may invalidate the experimental results. See the comment **C2** in the Code section for details.\n\n- **W2**: AccBO comes with many hyperparameters, which can be a concern for practical applications.\n\n- **W3**: l.219: \"is nearly optimal in terms of the dependency of $\\epsilon$\". The authors should be careful when talking about optimality of an algorithm. In particular, the algorithm AccBO does not belong to the algorithm class considered in [1] which does not take into account the biasedness of the hypergradient estimate due to the approximation of the inverse Hessian-vector product and the solution of the inner optimization problem. To my knowledge, the only works considering lower bounds for bilevel problems are [2, 5].\n\n- **W4**: The works [2, 3, 4] also use Nesterov acceleration for the inner optimization problem, but in the deterministic setting. They should be mentioned in the paper.\n\n### Code\nEven though it is a theoretical paper, I have several comments regarding the experiments code:\n\n- **C1**: In the checklist, the authors indicate \"The code and data are attached as a supplement with instructions for reproducibility.\" However, the code documentation is insufficient to run the experiments. The data are not present in the folders and there is no instruction indicating where to download them. As a consequence, when I launch the commmand `python main.py --methods accbo ` as indicated in the `README.md`, I get an error.\n\n- **C2**: It seems that the implementation of the different methods do not correspond to the one described in their respective original paper, which might invalidate the empirical results:\n    - For `TTSA`, `SUSTAIN` and `AccBO`, the estimation of the inverse Hessian-vector product is computed by the formula \n    $$\n        v = \\eta\\sum_{q=-1}^Q\\prod_{j=Q-q}^Q(I-\\eta \\nabla^2_{yy}G(x,y;\\zeta^{(j)}))\\nabla_y F(x,y;\\xi)\n    $$\n    while in the original paper (and even the present paper for AccBO), the authors use the formula\n    $$\n        v = \\eta\\prod_{j=1}^{q(Q)}(I-\\eta \\nabla^2_{yy}G(x,y;\\zeta^{(j)}))\\nabla_y F(x,y;\\xi)\n    $$\n    with $q(Q)\\sim\\mathcal{U}{\\{1,\\dots,Q-1\\}}$.\n\n    - For `SABA`, the SAGA-like variance reduction is absent in the code while being the core of the algorithm. Moreover, `SABA` uses constant step sizes.\n\n    - For `VRBO`, the oracles in the case `step % self.args.update_interval == 0` should be computed with a larger batch-size (or even with full batches) than in the inner for-loop.\n\n- **C3**: The code of `VRBO` for the datacleaning task is missing. The method to compute the hypergradient in `ma_soba.py` is missing as well for the datacleaning task (but not AUC experiment, thus I assume it is almost the same).\n\n### Minor\n- l.41: \"transforms\" -> \"transformers\"\n- In `README.md` (both): `python main.py --medthods [algorithm] ` -> `python main.py --methods [algorithm] `\n\nI am inclined to raise my score if my concerns, particularly regarding the experiments, are addressed.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness, and the lower-level problem is strongly convex. To improve the convergence rate, the authors propose a new Accelerated Bilevel Optimization algorithm named AccBO. This algorithm updates the upper-level variable using normalized stochastic gradient descent and the lower-level variable using the stochastic Nesterov accelerated gradient descent algorithm with averaging. The proof shows that AccBO has an oracle complexity of $ \\tilde{O}(1/\\epsilon^3) $, relying on a novel lemma describing the dynamics of the stochastic Nesterov accelerated gradient descent algorithm under distribution drift. Experimental results demonstrate that AccBO significantly outperforms baseline algorithms in various tasks, achieving the predicted theoretical acceleration.\n\nSTRENGTHS:\n1.The paper introduces AccBO, a novel algorithm that leverages normalized recursive momentum for the upper-level variable and Nesterov momentum for the lower-level variable. This dual approach to acceleration in bilevel optimization under stochastic settings is a significant advancement.\n\n2.AccBO achieves an oracle complexity of  $ \\tilde{O}(1/\\epsilon^3) $ for finding an $\\epsilon$-stationary point. This is a substantial improvement over the previously best-known complexity of        $ \\tilde{O}(1/\\epsilon^4)  $            for similar problems.The paper provides new proof techniques, particularly in analyzing the dynamics of stochastic Nesterov accelerated gradient descent under distribution drift. \n\n3.The effectiveness of AccBO is empirically verified across various tasks, including deep AUC maximization and data hypercleaning. The results show that AccBO achieves the predicted theoretical acceleration and significantly outperforms baseline algorithms.\n\n4.The paper extends the bilevel optimization framework to handle upper-level functions with unbounded smoothness, which is a realistic scenario in many neural network applications. This makes the proposed algorithm applicable to a broader range of practical problems.\n\nWEAKNESSES:\nThe convergence analysis for Option I of the algorithm is restricted to the lower-level problem being a one-dimensional quadratic function. This limitation reduces the generality and applicability of the theoretical results for more complex lower-level problems.\n\nQUESTIONS:\nIn reference [33], Assumption 2 states $|| \\nabla_y f(x, y^*(x)) || \\leq M$, while in this paper, Assumption 3.2 states$ || \\nabla_y f(x, y) || \\leq l_{f,0} $ . Can this assumption be relaxed to a less stringent form?\n\nLIMITATIONS:\nThe convergence analysis for Option I of the AccBO algorithm assumes that the lower-level problem is a one-dimensional quadratic function.  This limits the algorithm's applicability to scenarios where the lower-level function deviates significantly from this simple form, especially in higher dimensions or more complex optimization landscapes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper considers a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness, and the lower-level problem is strongly convex. Their novel algorithms achieve an oracle complexity of $O(1/\epsilon^3)$ to find an $\epsilon$-stationary point.

STRENGTHS:
1. The setting is novel: they consider problems with potentially unbounded smoothness.
2. Their complexity strictly improves the state-of-the-art oracle complexity for unbounded smooth nonconvex upper-level problems and strongly convex lower-level problems.

WEAKNESSES:
1. In the algorithm, for the update of the normalized stochastic gradient, it seems that it can be reformulated to tune the learning rate by dividing by the norm of the stochastic gradient. Therefore, the novelty and reasonableness of this step may be unclear.
2. In the experiments, it is common to use the same learning rate for baselines and the proposed algorithms for a fair comparison. However, in the experimental details, the authors provide the best learning rate pairs for different baselines, which seems unusual.

QUESTIONS:
1. Regarding the main challenge, the authors state why previous algorithms and analyses do not work. However, this explanation is not clear to me. Can you elaborate on how your work addresses the problem of hypergradient bias? Also, how does your potential function argument differ from previous work?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13