Title: Non-Convex Bilevel Optimization with Time-Varying Objective Functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bilevel Optimization

Time-Varying Functions

Single-Loop

Sublinear Bilevel Local Regret
Abstract: Bilevel optimization has become a powerful tool in a wide variety of machine learning problems. However, the current nonconvex bilevel optimization considers an offline dataset and static functions, which may not work well in emerging online applications with streaming data and time-varying functions. In this work, we study online bilevel optimization (OBO) where the functions can be time-varying and the agent continuously updates the decisions with online streaming data. To deal with the function variations and the unavailability of the true hypergradients in OBO, we propose a single-loop online bilevel optimizer with window averaging (SOBOW), which updates the outer-level decision based on a window average of the most recent hypergradient estimations stored in the memory. Compared to existing algorithms, SOBOW is computationally efficient and does not need to know previous functions. To handle the unique technical difficulties rooted in single-loop update and function variations for OBO, we develop a novel analytical technique that disentangles the complex couplings between decision variables, and carefully controls the hypergradient estimation error. We show that SOBOW can achieve a sublinear bilevel local regret under mild conditions. Extensive experiments across multiple domains corroborate the effectiveness of SOBOW.

Non-Convex Bilevel Optimization with Time-Varying Objective Functions

Abstract

Bilevel optimization has become a powerful tool in a wide variety of machine learning problems. However, the current nonconvex bilevel optimization considers an offline dataset and static functions, which may not work well in emerging online applications with streaming data and time-varying functions. In this work, we study online bilevel optimization (OBO) where the functions can be time-varying and the agent continuously updates the decisions with online streaming data. To deal with the function variations and the unavailability of the true hypergradients in OBO, we propose a single-loop online bilevel optimizer with window averaging (SOBOW), which updates the outer-level decision based on a window average of the most recent hypergradient estimations stored in the memory. Compared to existing algorithms, SOBOW is computationally efficient and does not need to know previous functions. To handle the unique technical difficulties rooted in single-loop update and function variations for OBO, we develop a novel analytical technique that disentangles the complex couplings between decision variables, and carefully controls the hypergradient estimation error. We show that SOBOW can achieve a sublinear bilevel local regret under mild conditions. Extensive experiments across multiple domains corroborate the effectiveness of SOBOW.

1 Introduction

Bilevel optimization has attracted significant recent attention, which in general studies the following problem:

Here both the outer-level function f and the inner-level function g are continuously differentiable, and the outer optimization problem is solved subject to the optimality of the inner problem. Due to its capability of capturing hierarchical structures in many machine learning problems, this nested optimization framework has been exploited in a wide variety of applications, e.g., meta-learning [4, 31], hyperparameter optimization [13, 3], reinforcement learning [62, 57] and neural architecture search [38, 8].

However, numerous machine learning problems with hierarchical structures are online in nature, e.g., online meta-learning [11] and online hyperparameter optimization [63], where the current bilevel

optimization framework cannot be directly applied due to the following reasons: (1) (streaming data) The nature of online streaming data requires decision making on-the-fly with low regret, whereas the offline framework emphasizes more on the quality of the final solution; (2) (timevarying functions) The objective functions in online applications can be time-varying because of non-stationary environments and changing tasks, in contrast to static functions considered in Equation (1); (3) (limited information) The online learning is nontrivial when the decision maker only has limited information, e.g., regarding the inner-level function, which can be even more challenging with time-varying functions. For example, in wireless network control [37], the controller has to operate under limited knowledge about the time-varying wireless channels (see Appendix).

To reap the success of bilevel optimization in online applications, there is an urgent need to develop a new online bilevel optimization (OBO) framework. Generally speaking, OBO considers the scenario where the data comes in an online manner and the agent continuously updates her outer-level decision based on the estimation of the optimal inner-level decision. Both outer-level and inner-level objective functions can be time-varying to capture the data distribution shifts in many online scenarios. Note that OBO is significantly different from single-level online optimization [22], due to the unavailability of the true outer-level objective function composed by f and y ∗ .

The study of OBO was recently initiated in [59], but much of this new framework still remains under-explored and not well understood. In particular, [59] combines offline bilevel optimization with online optimization and proposes an online alternating time-averaged gradient method. Such an approach suffers from several limitations: 1) Multi-step update is required for inner-level decision variable y t at each time t , which can be problematic when only limited information of the inner-level function g is available. 2) The hypergradient estimation at each time requires the knowledge of previous objective functions in a window, and also evaluates current models on each previous function; such a design can be inefficient and infeasible in online scenarios. In this work, we seek to address these limitations and develop a new OBO algorithm that can work efficiently without the knowledge of previous objective functions.

The main contributions can be summarized as follows.

( Efficient algorithm design ) We propose a new single-loop online bilevel optimizer with window averaging (SOBOW), which works in a fully online manner with limited information about the objective functions. In contrast to the OAGD algorithm in [59], SOBOW has the following major differences (as summarized in Table 1): (1) ( single-loop update ) Compared to the multi-step updates of the inner-level decision y t at each round in OAGD, we only require a one-step update of y t , which is more practical for online applications where only limited information about the inner-level function g t is available. (2) ( estimation error of Hessian inverse-vector product ) Estimating the hypergradient requires the outer-level Hessian inverse-vector product. [59] assumes that the exact Hessian inversevector product can be obtained, which can introduce high computational cost. In contrast, we consider a more practical scenario where there could be an estimation error in the Hessian-inverse vector product calculation in solving the linear system. (3) ( window averaged hypergradient descent ) While a window averaged hypergradient estimation is considered in both OAGD and SOBOW, SOBOW is more realistic and efficient than OAGD. Specifically, OAGD requires the knowledge of the most recent objective functions within a window and evaluates the current model on each previous function at every round, whereas SOBOW only stores the historical hypergradient estimation in the memory without any additional knowledge or evaluations about previous functions.

( Novel regret analysis ) Based on the previous studies in single-level online non-convex optimization [23, 2], we introduce a new bilevel local regret as the performance measure of OBO algorithms. We show that the proposed SOBOW algorithm can achieve a sublinear bilevel local regret under mild conditions. Compared to offline bilevel optimization and OAGD, new technical challenges need to be addressed here: (1) unlike the multi-step update of inner-level variable y t in OAGD, the single-step update in SOBOW will lead to an inaccurate estimation of the optimal inner-level

decision y ∗ t ( x t ) and consequently a large estimation error for the hypergradient; this problem can be addressed in offline bilevel optimization by controlling the gap ∥ y ∗ t ( x t ) -y ∗ t +1 ( x t +1 ) ∥ 2 , which depends only on ∥ x t -x t +1 ∥ 2 for static inner-level functions. But this technique cannot be applied here due to the time-varying g t ; (2) the function variations in OBO can blow up the impact of the hypergradient estimation error if not handled appropriately, whereas offline bilevel optimization does not have this issue due to the static functions therein. Towards this end, we appropriately control the estimation error of the Hessian inverse-vector product at each round, and disentangle the complex couplings between the decision variables through a three-level analysis. This enables the control of the hypergradient estimation error, by using a decaying coefficient to diminish the impact of the large inner-level estimation error and leveraging the historical information to smooth the update of the outer-level decision.

( Extensive experimental evaluations ) OBO has the potential to be used in various online applications, by capturing the hierarchical structures therein in an online manner. In this work, the experimental results clearly validate the effectiveness of SOBOW in online hyperparameter optimization and online hyper-representation learning.

2 Related Work

Online optimization Online optimization has been extensively studied for strongly convex and convex functions in terms of both static regret, e.g., [68, 21, 47] and dynamic regret, e.g., [5, 48, 61, 64, 66]. Recently, there has been increasing interest in studying online optimization for non-convex functions, e.g., [1, 58, 34, 51, 15], where minimizing the standard definitions of regret is computationally intractable. Specifically, [23] introduced a notion of local regret in the spirit of the optimality measure in non-convex optimization, and developed an algorithm that averages the gradients of the most recent loss functions evaluated in the current model. [2] proposed a dynamic local regret to handle the distribution shift and also a computationally efficient SGD update for achieving sublinear regret. [24] and [25] studied the zeroth-order online non-convex optimization where the agent only has access to the actual loss incurred at each round.

Offline bilevel optimization Bilevel optimization was first introduced in the seminal work [6]. Following this work, a number of algorithms have been developed to solve the bilevel optimization problem. Initially, the bilevel problem was reformulated into a single-level constrained problem based on the optimality conditions of the inner-level problem [20, 55, 43, 49], which typically involves many constraints and is difficult to implement in machine learning problems. Recently, gradient-based bilevel optimization algorithms have attracted much attention due to their simplicity and efficiency. These can be roughly classified into two categories, the approximate implicit differentiation (AID) based approach [10, 52, 16, 14, 17, 42, 32] and the iterative differentiation (ITD) based approach [45, 12, 53, 44, 17]. Bilevel optimization has also been studied very recently for the cases with stochastic objective functions [14, 31, 7, 33, 18] and multiple inner minima [35, 40, 39, 56]. Notably, a novel value-function based method was first proposed in [39] to deal with the non-convexity of the inner-level functions. Some recent studies, e.g., [36, 9, 26], have also explored single-level algorithms for offline bilevel optimization with time-invariant objective functions, whereas the timevarying functions in online bilevel optimization make the hypergradient estimation error control more challenging for single-loop updates.

Online bilevel optimization The investigation of OBO is still in the very early stage, and to the best of our knowledge, [59] is the only work so far that has studied OBO.

3 Online Bilevel Optimization

Following the same spirit as the online optimization in [22], the decisions are made iteratively in OBO without knowing their outcomes at the time of decision-making. Let T denote the total number of rounds in OBO. Define x t ∈ X ⊂ R d 1 and f t : X × R d 2 as the decision variable and the online function for the outer level problem, respectively. Define y t ∈ R d 2 and g t ∈ X × R d 2 as the decision variable and the online objective function for the inner level problem, respectively. Given the initial values of ( x 1 , y 1 ) , the general procedure of OBO is described in Algorithm 1.

Let y ∗ t ( x ) = arg min y ∈ R d 2 g t ( x, y ) for any x . The OBO framework in Algorithm 1 can be interpreted from two different perspectives: (1) ( single-player ) The player makes the decision on x t without

knowing the optimal inner-level decision y ∗ t ( x ) . Note, y t serves as an estimation of y ∗ t ( x ) from the player's perspective based on her knowledge of function g t ; (2) ( two-player ) OBO can also be viewed as a leader ( x t ) and follower ( y t ) game, where each player considers an online optimization problem and the leader seeks to play against the optimal decision y ∗ t ( x ) of the follower at each round under limited knowledge of g t .

It is worthwhile noting that OBO is quite different from the single-level online optimization. First, the outer-level objective function with respect to (w.r.t) x t , i.e., f t ( x t , y ∗ t ( x t )) , is not available to update x t , whereas, in standard single-level online optimization, the true loss is revealed immediately after making decisions. Besides, as a composite function of f t ( x, y ) and y ∗ t ( x ) , f t ( x, y ∗ t ( x )) is non-convex in general w.r.t. the outer-level decision variable x . Hence, standard regret definitions in online convex optimization [22] are not directly applicable here.

Motivated by the dynamic local regret defined in online non-convex optimization [2], we consider the following bilevel local regret:

and W = ∑ K -1 i =0 η i , η ∈ (0 , 1) , and f t ( · , · ) = 0 for t ≤ 0 . In contrast, the static regret in [59] evaluates the objective at time slot i using variable updates at different time slot j , which does not properly characterize the online learning performance of the model update for time-varying functions (see Appendix for more discussion). Intuitively, the regret in Equation (2) is defined as a sliding average of the hypergradients w.r.t. the decision variables at the corresponding instant for all rounds in OBO. This indeed approximately computes the exponential average of the outer-level function values f t ( x t , y ∗ t ( x t )) at the corresponding decision variables over a sliding window [2]. Larger weights will be assigned to more recent updates. The objective here is to design efficient OBO algorithms with sublinear bilevel regret BLR w ( T ) in T , which implies that the outer-level decision is becoming better and closer to the local optima for the outer-level optimization problem at each round. This gradient-norm based regret shares the same spirit as the first-order optimality criterion [14, 32], which is widely used in offline bilevel optimization to characterize the convergence to the local optima.

4 Algorithm Design

It is well known that online gradient descent (OGD) [54] has achieved great successes in singlelevel online optimization. On the other hand, gradient-based methods (e.g., [16, 14, 42, 40, 29]) have become extremely popular for solving offline bilevel optimization due to their high efficiency. Thus, we also study the online gradient descent based algorithm to solve the OBO problem. As mentioned earlier, the unique challenges of OBO should be carefully handled in the algorithm design, including 1) the inaccessibility of the objective function and accurate hypergradients compared to single-level online optimization and 2) the time-varying functions and limited information compared to offline bilevel optimization. To this end, our algorithm includes two major designs, i.e., efficient hypergradient estimation with limited information and window averaged outer-level decision update.

Hypergradient estimation In OBO, the exact hypergradient ∇ f t ( x t , y ∗ t ( x t )) w.r.t. x t can be represented as

where v ∗ t solves the linear system ∇ 2 y g t ( x t , y ∗ t ( x t )) v = ∇ y f t ( x t , y ∗ t ( x t )) . The optimal inner-level decision y ∗ t ( x t ) is generally unavailable in OBO. To estimate the hypergradient ∇ f t ( x t , y ∗ t ( x t )) , where

the AID-based approach [10, 14, 32] for offline bilevel optimization can be leveraged here, which will involve the following steps: (1) given x t , run N steps of gradient descent w.r.t. the inner-level objective function g t to find a good approximation y N t close to y ∗ t ( x t ) ; (2) given y N t , obtain v Q t by solving ∇ 2 y g t ( x t , y N t ) v = ∇ y f t ( x t , y N t ) with Q t steps of conjugate gradient. The estimated hypergradient is constructed as

Nevertheless, the N steps gradient descent for estimating y ∗ t ( x t ) require multiple inquiries about the inner-level function g t , which can be inefficient and infeasible for online applications. For the algorithm being used in more practical scenarios with limited information about g t , we consider the extreme case where N = 1 , i.e.,

where α is the inner-level step size. This would lead to an inaccurate estimation of y ∗ t ( x t ) , which can pose critical challenges for making satisfying outer-level decisions in OBO due to the unreliable hypergradient estimation, especially when the objective functions are time-varying.

Window averaged decision update To deal with the non-convex and time-varying functions, inspired by time-smoothed gradient descent in online non-convex optimization [23, 2, 67, 19], we consider a time-smoothed hypergradient descent for updating the outer-level decision variable x t :

where Π X is the projection onto the set X , β is the outer-level step size and

Here ̂ ∇ f t -i ( x t -i , y t +1 -i ) is the hypergradient estimation at the round t -i , as in Equation (4). Intuitively, the update of the current x t is smoothed by the historical hypergradient estimations w.r.t. the decision variables at that time, which is particularly important here for OBO due to the following reasons: (1) To compute the averaged ̂ ∇ F t,η ( x t , y t +1 ) at each round t , we only store the hypergradient estimation for previous rounds in the memory, i.e., store ̂ ∇ f i ( x i , y i +1 ) at each round i , and estimate the current hypergradient ̂ ∇ f t ( x t , y t +1 ) using the available information at current round t . Compared to OAGD in [59], there is no need to access to the previous outer-level and inner-level objective functions and evaluate the current decisions on those functions, which is clearly more efficient and practical for online applications. (2) Leveraging the historical information in the window to update the current decision is helpful to deal with the inaccurate hypergradient estimation for the current round, especially under mild function variations. This indeed shares the same rationale with using past tasks to facilitate forward knowledge transfer in online meta-learning [11] and continual learning [41], for better decision making in the current task and improving the overall performance in non-stationary environments.

Building upon these two major components, we can have our main OBO algorithm, Single-loop Online Bilevel Optimizer with Window averaging (SOBOW), as summarized in Algorithm 2. At the round t , we first estimate y t +1 as in Equation (5) given x t and y t , which will be next leveraged to solve the linear system and construct the hypergradient estimation as in Equation (4). Based on the historical hypergradient estimations stored in the memory for previous rounds, we next update x t based on Equation (6).

5 Theoretical Analysis

In this section, we provide the theoretical analysis of the regret bound for SOBOW.

5.1 Technical Assumptions

Let z = ( x, y ) . Before the regret analysis, we first make the following assumptions.

Assumption 5.1. The inner-level function g t ( x, y ) is µ g -strongly convex w.r.t. y , and the composite objective function f t ( x, y ∗ t ( x )) is non-convex w.r.t x .

Assumption 5.2. The following conditions hold for objective functions f t ( z ) and g t ( z ) , ∀ t ∈ [1 , T ] : (1) The function f t ( z ) is L 0 -Lipschitz continuous; (2) ∇ f t ( z ) and ∇ g t ( z ) are L 1 -Lipschitiz continuous; (3) The high-order derivatives ∇ x ∇ y g t ( z ) and ∇ 2 y g t ( z ) are L 2 -Lipschitz continuous.

1: Initialize decisions x 1 , y 1 , v 0

2: for t = 1 , ..., T do

3: Get information about functions f t and g t

4: Update y t +1 based on Equation (5)

5: Solve the linear system ∇ 2 y g t ( x t , y t +1 ) v = ∇ y f t ( x t , y t +1 ) using Q t steps of conjugate gradient starting from a fixed point v 0 with stepsize λ to obtain v Q t

6: Construct the hypergradient ̂ ∇ f t ( x t , y t +1 ) based on Equation (4) with y N t = y t +1

7: Store ̂ ∇ f t ( x t , y t +1 ) in the memory

8: Update x t +1 based on Equation (6)

9: end for

Note that both Assumption 5.1 and Assumption 5.2 are standard and widely used in the literature of bilevel optimization, e.g., [52, 14, 29, 59, 27].

Assumption 5.3. For any t ∈ [1 , T ] , the function f t ( x, y ∗ t ( x )) is bounded, i.e., | f t ( x, y ∗ t ( x )) | ≤ M with M > 0 . Besides, the closed convex set X is bounded, i.e., ∥ x -x ′ ∥ ≤ D with D > 0 , for any x and x ′ in X .

Assumption 5.3 on the boundedness of the objection functions is also standard in the literature of non-convex optimization, e.g., [23, 2, 50, 59]. Moreover, to guarantee the boundedness of the hypergradient estimation error, previous studies (e.g., [14, 28, 17]) in offline bilevel optimization usually assume that the gradient norm ∥∇ f ( z ) ∥ is bounded from above for all z . In this work, we make a weaker assumption on the feasibility of ∇ y f t ( x, y ∗ t ( x )) , which generally holds since y ∗ t ( x ) is usually assumed to be bounded in bilevel optimization:

Assumption 5.4. There exists at least one ˆ x ∈ X such that ∥∇ y f t (ˆ x, y ∗ t (ˆ x )) ∥ ≤ ρ where ρ > 0 is some constant.

5.2 Theoretical Results

Technical challenges in analysis: To analyze the regret performance of SOBOW, several key and unique technical challenges need to be addressed, compared to offline bilevel optimization and OAGD in [59]: (1) unlike the multi-step update of inner-level variable y t in OAGD, the single-step update in SOBOW will lead to an inaccurate estimation of y ∗ t ( x t ) and consequently a large estimation error for the hypergradient ∇ f t ( x t , y ∗ t ( x t )) ; this problem can be addressed in offline bilevel optimization by controlling the gap ∥ y ∗ t ( x t ) -y ∗ t +1 ( x t +1 ) ∥ 2 , which depends only on ∥ x t -x t +1 ∥ 2 for static inner-level functions, but this technique cannot be applied here due to the time-varying g t ; (2) the function variations in OBO can blow up the impact of the hypergradient estimation error if not handled appropriately, whereas offline bilevel optimization does not have this issue due to the static functions therein; (3) a new three-level analysis is required to understand the involved couplings among the estimation errors about v t , y t and x t in online learning.

Towards this end, the very first step is to understand the estimation error of the optimal solution v ∗ t to the linear system ∇ 2 y g t ( x t , y ∗ t ( x t )) v = ∇ y f t ( x t , y ∗ t ( x t )) . Here v ∗ t = ( ∇ 2 y g t ( x t , y ∗ t ( x t ))) -1 ∇ y f t ( x t , y ∗ t ( x t )) . We have the following lemma about ∥ v Q t -v ∗ t ∥ 2 , where v Q t is obtained by solving ∇ 2 y g t ( x t , y t +1 ) v = ∇ y f t ( x t , y t +1 ) using Q t steps of conjugate gradient.

Lemma 5.5. Suppose Assumptions 5.1-5.4 hold, λ ≤ 1 L 1 , α ≤ 1 L 1 and Q t +1 -Q t ≥ log(1 -αµg 2 ) 2 log(1 -λµ g ) . We can have that

where c 2 > 0 is some constant and the error ϵ 2 t decays with t , i.e., ϵ 2 t +1 ≤ (1 -αµ g / 2) ϵ 2 t .

Lemma 5.5 characterizes the estimation error ∥ v Q t -v ∗ t ∥ 2 in a neat way, by constructing an upper bound with the estimation error of the inner-level optimal decision y ∗ t ( x t ) , i.e., ∥ y t +1 -y ∗ t ( x t ) ∥ 2 , and a decaying error term ϵ 2 t . The way of controlling ∥ v Q t -v ∗ t ∥ 2 here is particularly important, which not only clarifies the coupling between v t and y t but also helps to control the hypergradient estimation

error. Note that solving the linear system with a larger Q t does not require more information about the inner-level function, and the introduced computation cost can be negligible, because the conjugate gradient only involves Hessian-vector product which can be efficiently computed.

Next we seek to bound the hypergradient estimation error ∥∇ f t ( x t , y ∗ t ( x t )) -̂ ∇ f t ( x t , y t +1 ) ∥ 2 at the round t . Intuitively, the hypergradient estimation error depends on both ∥ y t +1 -y ∗ t ( x t ) ∥ 2 and ∥ v Q t -v ∗ t ∥ 2 . Building upon Lemma 5.5, this dependence can be shifted to the joint error of ∥ y t +1 -y ∗ t ( x t ) ∥ 2 and ϵ 2 t , which contains iteratively decreasing components after careful manipulations. Specifically, let G 1 = 1 + c 2 + L 2 2 ( ρµ g + DL 2 1 + DL 1 µ g ) 2 L 2 1 µ 4 g and G 2 = 2 G 1 (1 + 2 αµ g )(1 -αµ g ) . We have the following theorem to characterize the hypergradient estimation error.

Theorem 5.6. Suppose that Assumptions 5.1-5.4 hold, λ ≤ 1 L 1 , α ≤ 1 L 1 and Q t -Q t -1 ≥ log(1 -αµg 2 ) 2 log(1 -λµ g ) . For t ∈ [2 , T ] , we can bound the hypergradient estimation error as follows:

where δ 1 = G 1 ∥ y 2 -y ∗ 1 ( x 1 ) ∥ 2 + ϵ 2 1 .

As shown in Theorem 5.6, the upper bound of the hypergradient estimation error includes three terms: (1) The first term decays with t , which captures the iteratively decreasing component in the joint error of ∥ y t +1 -y ∗ t ( x t ) ∥ 2 and ϵ 2 t ; (2) The second term characterizes the dependence on the variation of the outer-level decision between adjacent rounds in the history; (3) The third term characterizes the dependence on the variation of the optimal inner-level decision between adjacent rounds.

To control the hypergradient estimation error as in Theorem 5.6, the key idea is to decouple the source of the estimation error ∥ y t +1 -y ∗ t ( x t ) ∥ 2 at the current round into three different components, i.e., ∥ y t -y ∗ t -1 ( x t -1 ) ∥ 2 for the previous round, the variation of the out-level decision, and the variation of the optimal inner-level decision. Since the inner-level estimation error is large due to the single-step update, we diminish its impact through a decaying coefficient, which inevitably enlarges the impact of the other two components. The variation of the optimal inner-level decision is due to nature of the OBO problem, which cannot be controlled. One has to impose some regularity constraints on this variation to achieve a sublinear regret, in the same spirit to the regularities on functional variations widely used in the dynamic regret literature (e.g., [5, 66]). Therefore, the key point now becomes the control of the variation of the out-level decision ∥ x t -1 -j -x t -j ∥ 2 , which can be achieved through the window averaged update of x t in Equation (6). Intuitively, by leveraging the historical information, the window averaged hypergradient in Equation (7) smooths the outer-level decision update, which serves as a better update direction compared to the deviated single-round estimation ̂ ∇ f t ( x t , y t +1 ) . Before presenting the main result, we first introduce the following definitions to characterize the variations of the objective function f t ( · , y ∗ t ( · )) and the optimal inner-level decision y ∗ t ( · ) in OBO, respectively:

Intuitively, V 1 ,T measures the overall fluctuations between the adjacent objective functions in all rounds under the same outer-level decision variable, and H 2 ,T can be regarded as the inner-level path length to capture the variation of the optimal inner-level decisions as in [59]. Note that V 1 ,T is a weaker regularity for the functional variation compared to absolute values used in single-level online optimization for dynamic regret [5, 66]. When the functions are static, these variations terms are simply 0. We are interested in the case where both V 1 ,T and H 2 ,T are o ( T ) as in the literature of dynamic regret.

Based on Theorem 5.6, we can have the following theorem to characterize the regret of SOBOW.

Theorem 5.7. Suppose that Assumptions 5.1-5.4 hold. Let λ ≤ 1 L 1 , α ≤ 1 L 1 , Q t +1 -Q t ≥ log(1 -αµg 2 ) 2 log(1 -λµ g ) , η ∈ (1 -αµ g 2 , 1) , and β ≤ min { 1 4 L f , µ 2 g L f W (1 -η )( η -1+ αµ g / 2) 24 L 4 1 G 2 η } where L f is the

smoothness parameter of the function f t ( · , y ∗ t ( · )) . Then we can have

The value of L f can be found in the proof in Appendix. Note that the hypergradient estimation at the current round depends on all previous outer-level decisions as shown in Theorem 5.6. While these decisions may not be good at the early stage in OBO, choosing η ∈ (1 -αµ g 2 , 1) would diminish their impact on the local regret. When the variations V 1 ,T and H 2 ,T are both o ( T ) , a sublinear bilevel local regret can be achieved for an appropriately selected window, e.g., W = o ( T ) when η = 1 -h ( T ) , where h ( T ) → 0 as T →∞ . Note that the value β does not change substantially since W (1 -η ) converges to 1 . Particularly, when η = 1 -o ( 1 T ) , W = ω ( T ) . In this case, we can have the smallest regret O ( V 1 ,T β + H 2 ,T ) that only depends on the function variations in OBO.

6 Experiments

In this section, we conduct experiments in multiple domains to corroborate the utility of the OBO framework and the effectiveness of SOBOW. ×10 6 ×10 6 ×10 6

0.0

0.5

1.0

1.5

2.0

Rounds T

×10 3

0.0

0.5

1.0

1.5

Regret

SOBOW-50

OAGD-50

OGD

(a) static setup

0

1

2

3

Rounds T

×10 3

0

1

2

3

Regret

SOBOW-50

OAGD-50

OGD

(b) dynamic setup

0.5

1.0

1.5

2.0

Rounds T

×10 3

0.0

0.5

1.0

Regret

SOBOW,

=0.99

SOBOW,

=0.95

SOBOW,

=0.9

SOBOW,

=0.8

(c) impact of

η

0.5

1.0

1.5

2.0

Rounds T

×10 3

0.00

0.25

0.50

0.75

Regret

×10 6

SOBOW, N=1

SOBOW, N=2

SOBOW, N=10

(d) impact of

N

Figure 1: Evaluation for online HR. As shown in subfigures (a) and (b), SOBOW performs similarly to OAGD and significantly outperforms OGD in both static and dynamic setups. Subfigure (c) shows the performance of SOBOW under different values of the averaging parameter η for online HR. Better performance is achieved as η → 1 . Subfigure (d) shows the performance of SOBOW under different values of inner steps N when the data stream contains two data points (one in X g t and one in X f t ). The performance saturates at N = 2 .

Specifically, we compare our algorithm SOBOW with the following baseline methods: (1) OAGD [59], which is the only method for OBO in the literature; (2) OGD , a natural method which updates the outer-level decision by using the current hypergradient estimation only without any window averaging. Intuitively, OGD is not only a special case of OAGD when the information of previous functions is not available , but also a direct application of offline bilevel optimization, e.g., AID-based method [32]. We also denote SOBOWK /OAGDK as SOBOW and OAGD with window size K , respectively. And we evaluate the regret using the definition in Equation (2). We also compare the performance using the regret in [59] in Appendix where similar results can be observed.

Online Hyper-representation Learning Representation learning [13, 17] seeks to extract good representations of the data. The learnt representation mapping can be used in downstream tasks to facilitate the learning of task specific model parameters. This formulation is typically encountered in a multi-task setup, where Λ captures the common representation extracted for multiple tasks and w defines the task-specific model parameters. When the data/task arrives in an online manner, the hyper-representation needs to be continuously adapted to incorporate the new knowledge.

Following [17], we study online hyper-representation learning (Online HR) with linear models. Specifically, at each round t , the agent applies the hyper-representation Λ t ∈ R p × d and the linear model prediction w t ∈ R d , and then receives small minibatches ( X f t , Y f t ) and ( X g t , Y g t ) . Based on Λ t and ( X g t , Y g t ) , the agent updates her linear model prediction w t +1 as an estimation of w ∗ (Λ t ) = arg min w ∈ R d g t (Λ t , w ) := ∥ X g t Λ t w -Y g t ∥ 2 + γ 2 ∥ w ∥ 2 . Based on the estimation w t +1 and ( X f t , Y f t ) , the agent further updates her decision Λ t +1 about the hyper-representation to minimize the loss f t (Λ , w ∗ t (Λ)) := ∥ X f t Λ w ∗ t (Λ) -Y f t ∥ 2 . In our experiments, we consider synthetic data generated as in [17] and explore two distinct settings: (i) a static setup where the underlying model generating the minibatches is fixed; and (ii) a staged dynamic setup where the model changes after some steps.

As shown in Figure 1(a) and Figure 1(b), SOBOW achieves comparable regret with OAGD in both static and dynamic setups, without the need of knowing previous functions. In terms of the running

time for 5000 steps with K = 50 , SOBOW takes 11 seconds, OAGD takes 228 seconds and OGD takes 7 seconds. Therefore, SOBOW is much more computationally efficient compared to OAGD, because SOBOW does not need to re-evaluate the previous functions on the current model at each round. On the other hand, SOBOW performs substantially better than OGD (i.e., OAGD when previous functions are not available) with similar running time . These results not only demonstrate the usefulness of SOBOW when the previous functions are not available, but also corroborate the benefit of window-averaged outer-level decision update by leveraging the historical hypergradient estimations in OBO. Figure 1(c) shows the performance of SOBOW under different values of the averaging parameter η . The performance is better as η → 1 , which is also consistent with our theoretical results. Figure 1(d) indicates that a small number of updates for the inner-level variable is indeed enough for online HR.

Online Hyperparameter Optimization The goal of hyperparameter optimization (HO) [13, 17] is to search for the best values of hyperparameters λ , which seeks to minimize the validation loss of the learnt model parameters w and is usually done offline. However, in online applications where the data distribution can dynamically change, e.g., the unusual traffic patterns in online traffic time series prediction problem [63], keeping the hyperparameters static could lead to sub-optimal performance. Therefore, the hyperparameters should be continuously updated together with the model parameters in an online manner.

Specifically, at each online round t , the agent applies the hyperparameters λ t and the model w t , and then receives a small dataset D t = {D tr t , D val t } composed of a training subset D tr t and a validation subset D val t . Based on λ t and D tr t , the agent first updates her model prediction w t +1 as an estimation of w ∗ t ( λ t ) := arg min w L tr t ( λ t , w ) , where L tr t ( λ, w ) := 1 |D tr t | ∑ ζ ∈D tr t L ( w ; ζ ) + Ω( λ, w ) , L ( w,ζ ) is a cost function computed on data point ζ with prediction model w , and Ω( w,λ ) is a regularizer. Based on the model prediction w t +1 and D tr t , the agent updates the hyperparameters λ t +1 to minimize the validation loss L val t ( λ, w ∗ t ( λ )) := 1 val ∑ ∈D val L ( w ∗ t ( λ ); ξ ) .

| D t | ξ t

We consider an online classification setting on the 20 Newsgroup dataset, where the classifier is modeled by an affine transformation and we use the cross-entropy loss as the losscost function. For Ω( λ, w ) , we use one ℓ 2 -regularization parameter for each row of the transformation matrix in w , so that we have one regularization parameter for each data feature (i.e., | λ | is given by the dimension of the data). We remove all news headers in the 20 Newsgroup dataset and pre-process the dataset so as to have data feature vectors of dimension d = 99238 . In our implementations, we approximate the hypergradient using implicit differentiation with the fixed point method [17]. We consider two different setups: (i) a static setup where the agent receives a stream of clean data batches {D t } t ; (ii) a dynamic setting in which the agent receives a stream of corrupted batches {D t } t , where the corruption level changes after some time steps. For both setups the batchsize is fixed to 16 . For the dynamic setting we consider four different corruption levels { 5% , 10% , 20% , 30% } and also optimize the learning rate as an additional hyperparameter.

We evaluate the testing accuracy for SOBOW and OAGD in Table 2 for both static (Left) and dynamic (Right) setups. It can be seen that compared to OAGD, SOBOW achieves similar accuracy but with a much shorter running time. When the window size increases in Table 2, performance of both SOBOW and OAGD increases and the computational advantage of SOBOW becomes more significant. In particular, SOBOW runs around 20 times faster than OAGD when the window size is 50.

7 Conclusions and Discussion

In this work, we study non-convex bilevel optimization where the functions can be time-varying and the agent continuously updates the decisions with online streaming data. We proposed a single-loop

online bilevel optimizer with window averaging (SOBOW) to handle the function variations and the unavailability of the true hypergradients in OBO. Compared to existing algorithms, SOBOW is computationally efficient and does not require previous function information. We next developed a novel analytical technique to tackle the unique challenges in OBO and showed that SOBOW can achieve a sublinear bilevel local regret. Extensive experiments justified the effectiveness of SOBOW. We also discuss the potential applications of the OBO framework in online meta-learning and online adversarial training (see Appendix).

Limitation and future directions The study of online bilevel optimization is still in a very early stage, and much of this new framework still remains under-explored and not well understood. We started with the second-order approach for hypergradient estimation, which is less scalable. One future direction is to leverage the recently developed first order approaches for hypergradient estimation. Another limitation is that we assume that the inner-level objective function is strongly convex. In the future, we will investigate the convex and even non-convex case.

Acknowledgments

This work has been supported in part by NSF grants NSF AI Institute (AI-EDGE) CNS-2112471, CNS-2106933, CNS-2106932, CNS-2312836, CNS-1955535, CNS-1901057, ECCS-2113860, DMS2134145, and 2311274, by Army Research Office under Grant W911NF-21-1-0244, and was sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-23-2-0225. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

References

Naman Agarwal, Alon Gonen, and Elad Hazan. Learning in non-convex games with an optimization oracle. In Conference on Learning Theory , pages 18-29. PMLR, 2019.

Sergul Aydore, Tianhao Zhu, and Dean P Foster. Dynamic local regret for non-convex online forecasting. Advances in neural information processing systems , 32, 2019.

Fan Bao, Guoqiang Wu, Chongxuan Li, Jun Zhu, and Bo Zhang. Stability and generalization of bilevel programming in hyperparameter optimization. Advances in Neural Information Processing Systems , 34:4529-4541, 2021.

Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations , 2018.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research , 63(5):1227-1244, 2015.

Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations Research , 21(1):37-44, 1973.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671 , 2021.

Jiequan Cui, Pengguang Chen, Ruiyu Li, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast and practical neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6509-6518, 2019.

Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems , 35:26698-26710, 2022.

Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics , pages 318-326. PMLR, 2012.

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning , pages 1920-1930. PMLR, 2019.

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning , pages 1165-1173. PMLR, 2017.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , pages 1568-1577. PMLR, 2018.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

Udaya Ghai, Zhou Lu, and Elad Hazan. Non-convex online learning via algorithmic equivalence. arXiv preprint arXiv:2205.15235 , 2022.

Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447 , 2016.

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , pages 3748-3758. PMLR, 2020.

Zhishuai Guo and Tianbao Yang. Randomized stochastic variance-reduced methods for stochastic bilevel optimization. arXiv preprint arXiv:2105.02266 , 2021.

Nadav Hallak, Panayotis Mertikopoulos, and Volkan Cevher. Regret minimization in stochastic non-convex learning via a proximal-gradient approach. In International Conference on Machine Learning , pages 4008-4017. PMLR, 2021.

Pierre Hansen, Brigitte Jaumard, and Gilles Savard. New branch-and-bound rules for linear bilevel programming. SIAM Journal on scientific and Statistical Computing , 13(5):1194-1217, 1992.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2):169-192, 2007.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Elad Hazan, Karan Singh, and Cyril Zhang. Efficient regret minimization in non-convex games. In International Conference on Machine Learning , pages 1433-1441. PMLR, 2017.

Amélie Héliou, Matthieu Martin, Panayotis Mertikopoulos, and Thibaud Rahier. Online nonconvex optimization with imperfect feedback. Advances in Neural Information Processing Systems , 33:17224-17235, 2020.

Amélie Héliou, Matthieu Martin, Panayotis Mertikopoulos, and Thibaud Rahier. Zeroth-order non-convex learning via hierarchical dual averaging. In International Conference on Machine Learning , pages 4192-4202. PMLR, 2021.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actorcritic. SIAM Journal on Optimization , 33(1):147-180, 2023.

Xiaoyin Hu, Nachuan Xiao, Xin Liu, and Kim-Chuan Toh. An improved unconstrained approach for bilevel optimization. arXiv preprint arXiv:2208.00732 , 2022.

Kaiyi Ji, Jason D Lee, Yingbin Liang, and H Vincent Poor. Convergence of meta-learning with task-specific adaptation over partial parameters. Advances in Neural Information Processing Systems , 33:11490-11500, 2020.

Kaiyi Ji and Yingbin Liang. Lower bounds and accelerated algorithms for bilevel optimization. arXiv preprint arXiv:2102.03926 , 2021.

Kaiyi Ji, Mingrui Liu, Yingbin Liang, and Lei Ying. Will bilevel optimizers benefit from loops. arXiv preprint arXiv:2205.14224 , 2022.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Provably faster algorithms for bilevel optimization and applications to meta-learning. 2020.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning , pages 4882-4892. PMLR, 2021.

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems , 34:30271-30283, 2021.

Antoine Lesage-Landry, Joshua A Taylor, and Iman Shames. Second-order online nonconvex optimization. IEEE Transactions on Automatic Control , 66(10):4866-4872, 2020.

Junyi Li, Bin Gu, and Heng Huang. Improved bilevel model: Fast and optimal algorithm with theoretical guarantee. arXiv preprint arXiv:2009.00690 , 2020.

Junyi Li, Bin Gu, and Heng Huang. A fully single loop algorithm for bilevel optimization without hessian inverse. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7426-7434, 2022.

Xiaojun Lin, Ness B Shroff, and Rayadurgam Srikant. A tutorial on cross-layer optimization in wireless networks. IEEE Journal on Selected areas in Communications , 24(8):1452-1463, 2006.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Risheng Liu, Xuan Liu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A value-function-based interior-point method for non-convex bi-level optimization. In International Conference on Machine Learning , pages 6882-6892. PMLR, 2021.

Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International Conference on Machine Learning , pages 6305-6315. PMLR, 2020.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics , pages 1540-1552. PMLR, 2020.

Yibing Lv, Tiesong Hu, Guangmin Wang, and Zhongping Wan. A penalty function method based on kuhn-tucker condition for solving linear bilevel programming. Applied mathematics and computation , 188(1):808-813, 2007.

Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv:1903.03088 , 2019.

Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning , pages 2113-2122. PMLR, 2015.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

H Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908 , 2010.

Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 7195-7201. IEEE, 2016.

Gregory M Moore. Bilevel programming algorithms for machine learning model selection . Rensselaer Polytechnic Institute, 2010.

Parvin Nazari and Esmaile Khorram. Dynamic regret analysis for online meta-learning. arXiv preprint arXiv:2109.14375 , 2021.

SangWoo Park, Julie Mulvaney-Kemp, Ming Jin, and Javad Lavaei. Diminishing regret for online nonconvex optimization. In 2021 American Control Conference (ACC) , pages 978-985. IEEE, 2021.

Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International conference on machine learning , pages 737-746. PMLR, 2016.

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated backpropagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1723-1732. PMLR, 2019.

Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.

Chenggen Shi, Jie Lu, and Guangquan Zhang. An extended kuhn-tucker approach for linear bilevel programming. Applied Mathematics and Computation , 162(1):51-63, 2005.

Daouda Sow, Kaiyi Ji, Ziwei Guan, and Yingbin Liang. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123 , 2022.

Bradly Stadie, Lunjun Zhang, and Jimmy Ba. Learning intrinsic rewards as a bi-level optimization problem. In Conference on Uncertainty in Artificial Intelligence , pages 111-120. PMLR, 2020.

Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory , pages 845-861. PMLR, 2020.

Davoud Ataee Tarzanagh and Laura Balzano. Online bilevel optimization: Regret analysis of online alternating gradient methods. arXiv preprint arXiv:2207.02829 , 2022.

Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 , 2020.

Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In International Conference on Machine Learning , pages 449-457. PMLR, 2016.

Zhuoran Yang, Zuyue Fu, Kaiqing Zhang, and Zhaoran Wang. Convergent reinforcement learning with function approximation: A bilevel optimization perspective. 2018.

Hongyuan Zhan, Gabriel Gomes, Xiaoye S Li, Kamesh Madduri, and Kesheng Wu. Efficient online hyperparameter optimization for kernel ridge regression with applications to traffic time series prediction. arXiv preprint arXiv:1811.00620 , 2018.

Lijun Zhang, Tianbao Yang, Jinfeng Yi, Rong Jin, and Zhi-Hua Zhou. Improved dynamic regret for non-degenerate functions. Advances in Neural Information Processing Systems , 30, 2017.

Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning , pages 26693-26712. PMLR, 2022.

Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Learning for Dynamics and Control , pages 48-59. PMLR, 2021.

Zhenxun Zhuang, Yunlong Wang, Kezi Yu, and Songtao Lu. No-regret non-convex online meta-learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3942-3946. IEEE, 2020.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936, 2003.

Appendix

A Discussion about practical applications of OBO

(1) In the traffic flow prediction problem [63], since the data collected by the traffic sensors arrive at the controller continuously and frequently, the hyperparameters need to be optimized (in the outer level) quickly in an online manner in order to guarantee the performance of the prediction model (in the inner level). Keeping the hyperparameters static for the prediction model may result in sub-optimal performance, because the distribution of the traffic flow can change gradually. Further, the controller may not have global information of the traffic flows, because it is exorbitantly expensive to deploy sensors to cover all traffic flows.

(2) In the wireless network control problem [37], the controller allocates resources, e.g., wireless channel bandwidth, to the users, where each user can have its own utility function depending on the wireless channel conditions given the allocated resources. Since wireless channels are usually time-varying, the controller has to continuously update the resource allocation quickly to maximize the network performance. The rate allocation decisions (deciding what packet rate a user transmits at any given time) and scheduling decisions (which users transmit) are done at a fast time-scale (in the inner level), while determining the utility functions of the users could be done at a slower time scale (in the outer level). Further these decisions need to be made in a distributed fashion, so under local knowledge of the channel conditions and interference levels.

In these applications, the current offline bilevel optimization framework cannot be directly applied because of the streaming data, the time-varying functions and possibly the limited information about the system. In contrast, online bilevel optimization has great potential for these online applications. Moreover, the decision making in these online applications also needs to be fast and efficient without the need of knowing all previous functions. Thus, the regret captures the performance of the learning model over the sequential process rather than just the performance of a final output model in the offline setting. How to design such algorithms with a sublinear regret guarantee is very important for making online bilevel optimization more practical in real applications.

B Applications of OBO in meta-learning and adversarial training

Online meta-learning In online meta-learning, learning tasks arrive one at a time, and the agent aims to learn a good meta-model θ based on the past tasks in a sequential manner, which can be quickly adapted to a good task-specific model ϕ for the current task. Specifically, each task t has a training dataset D tr t and a testing dataset D te t , and given a meta- model θ , the optimal task model is defined as

In the OBO framework of online meta-learning, the meta-model θ t is the outer-level decision variable at round t , and the task model ϕ t is the inner-level decision variable. At round t , the task model ϕ t +1 is first obtained based on g t ( θ t , ϕ ) as an estimation of ϕ ∗ t ( θ t ) ; and then given ϕ t +1 , the meta-model will be further updated w.r.t.

Online adversarial training Adversarial training [46, 60, 65] is usually formulated as a min-max optimization problem. The defender learns a robust model to minimize the worst-case training loss against an attacker, where the attacker aims to maximize the loss by perturbing the training data. A static setting is often considered with full access to the target dataset at all times. Nevertheless, many real-world applications involve streaming data that arrive in an online manner, e.g., the financial markets or real-time sensor networks. A continuously robust model update is more desirable in these applications against potential attacks on the streaming data.

This online adversarial training problem can also be addressed by the OBO framework. Here the model parameters θ is the outer-level decision variable and the adversarial perturbations δ to the data point ( x, y ) is the inner-level decision variable. At each time t , the defender updates the estimate δ t +1 of the worst adversarial perturbations, given her knowledge about the inner-level objection function g t ( θ t , δ t ) = -L ( θ t , x t + δ, y t ) subject to the perturbation constraint. Based

on δ t +1 , the defender updates the model θ t +1 robustly w.r.t. the outer-level objective function f t ( θ, δ t +1 ) = L ( θ, x t + δ t +1 , y t ) .

C Experimental details

We use a grid of values between 10 -5 and 10 to set the stepsizes and did not find the algorithm particularly sensitive to them for the experiments considered. For example, we achieve best performance by setting both the inner and outer stepsizes to 10 -4 for the online hyper-representation learning experiments and small values around that scale yield the same performance. For the dynamic OHO experiments, only the outer step size is set manually to 0 . 01 . The inner step size is optimized along with the other regularization hyperparameters.

OAGD needs to store the previous objective functions, which requires all the previous data points to compute the function values and additional resources to store the knowledge of the function structures. In contrast, our method only stores the previous hypergradient estimates averaged over previous data points. For example, when n data points are used to evaluate the function at each round, the memory requirement for OAGD is O ( nK ) , compared to O ( K ) for our method. Here, K is the window size. Therefore, the memory cost of our method can be lower, especially when the number of data points is large.

D Comparison between the regret definitions

For a clear comparison, we restate the definitions of bilevel local regret in our work and OAGD here: Our definition:

The key difference is that we evaluate the past loss f t -i using the variable updates x t -i and y ∗ t -i ( x t -i ) at exactly same time t -i , while in OAGD the past loss f t -i is evaluated using the most recent updates x t and y ∗ t ( x t ) . As shown in [2], the static regret in OAGD can cause problems for time-varying loss functions. Intuitively, evaluating the objective at time slot i using variable updates at different time slot j can be misleading, because it does not properly characterize the online learning performance of the model update at time slot i , especially when the objective functions vary a lot.

In Figure 2 we compare our algorithm and OAGD using the regret notion proposed in OAGD. The results show that our algorithm still achieves similar regret performance compared to OAGD, but with a much shorter runtime.

E Additional Results

Using path-length regularization to capture the variation of optimal decision variables is very common in the literature of dynamic online learning, e.g., [68, 48, 61, 64, 66]. Note that because this variation of optimal decision variables is not controllable, we do not use this term in the design of the algorithm. Rather, the variation term is only used in the theoretical analysis to understand which factors in the system lead to a tighter bound on the regret. However, we mention here that it is also possible to explicitly analyze the regret in terms of the function variations directly.

Theorem E.1. Suppose that Assumptions 5.1-5.4 hold. Let V g = ∑ T t =1 sup | g t +1 ( x, y ) -g t ( x, y ) | and V f = ∑ T t =1 sup[ f t +1 ( x, y ) -f t ( x, y )] . Under the same conditions on λ , α , Q t , η and β with √

The main proof idea is as follows:

In OAGD:

0

1

2

3

Rounds T

×10 3

0

1

2

3

Regret

×10 6

SOBOW-20

OAGD-20

For the term H 2 ,T , based on the strong convexity of g t , we can show that ∥ y ∗ t +1 ( x ) -y ∗ t ( x ) ∥ 2 ≤ 2 µ g sup | g t +1 ( x, y ) -g t ( x, y ) | ;

For the term V 1 ,T , we can show that f t +1 ( x, y ∗ t +1 ( x )) -f t ( x, y ∗ t ( x )) ≤ L 0 ∥ y ∗ t +1 ( x ) -y ∗ t ( x ) ∥ + sup[ f t +1 ( x, y ) -f t ( x, y )] , such that ∑ T t =1 [ F t,η ( x t , y ∗ t ( x t )) -F t,η ( x t +1 , y ∗ t ( x t +1 ))] in Lemma H.3 can be upper bounded by 2 MT W + L 0 √ 2 T µ g √ ∑ T t =1 sup | g t +1 ( x, y ) -g t ( x, y ) | + ∑ T t =1 sup[ f t +1 ( x, y ) -f t ( x, y )] ;

Based on the above, if we denote V g = ∑ T t =1 sup | g t +1 ( x, y ) -g t ( x, y ) | and V f = ∑ T t =1 sup[ f t +1 ( x, y ) -f t ( x, y )] to capture the function variations, we can have the overall regret as O ( T βW + V f β + V g + √ TV g β ) . In this case, a sublinear regret will be achieved if both V g

and V f are o ( T ) for suitably selected W . As mentioned in our previous response, the condition on the variation of y ∗ t ( x ) is weaker compared to the condition on the variation of g t in order to achieve a small regret. For example, suppose W = ω ( T ) and the function variation of f t is very small, to achieve a regret of O ( T 3 / 4 ) , V 1 ,T = O ( T 3 / 4 ) is sufficient, while we need a stricter condition on the function variation of g t , i.e., V g = O ( T 1 / 2 ) .

F Proof of Lemma 5.5

To prove Lemma 5.5, we first have the following lemma about v ∗ t :

Lemma F.1. Suppose that Assumption 5.2, Assumption 5.3 and Assumption 5.4 hold. We can have the following upper bound on ∥ v ∗ t ∥ :

Proof. Based on Lemma 2.2 in [14], it can be shown that y ∗ t ( x ) is L 1 µ g -Lipschitz continuous in x , i.e.,

for any x and x ′ . According to Assumption 5.4, let ˆ x ∈ X such that ∥∇ y f t (ˆ x, y ∗ t (ˆ x )) ∥ ≤ ρ . Then it follows that

Therefore,

Based on Lemma 1 in [30], we can have that

follows that

where c 2 = ( 1 + 1 λµ g ) C 2 Q , and ϵ 2 t = (1 + λµ g )(1 -λµ g ) 2 Q t (2 ∥ v 0 ∥ 2 +2 M v ) .

G Proof of Theorem 5.6

Based on Equation (3) and Equation (4), it follows that

where (a) is true because of Assumption 5.2, (b) is true because of Lemma F.1 and (c) holds due to Lemma 5.5.

Next, based on Assumption 5.1 and Assumption 5.2, for any t and α < 1 L 1 we can have

where (a) is based on the Young's inequality and (b) is due to Equation (8).

For λ = αµ g 2 , it follows that

Let G 1 = 1+ c 2 + L 2 2 ( ρµ g + DL 2 1 + DL 1 µ g ) 2 L 2 1 µ 4 g and δ t = G 1 ∥ y t +1 -y ∗ t ( x t ) ∥ 2 + ϵ 2 t . Based on Lemma 5.5, we can have that

where G 2 = 2 G 1 (1+ 2 αµ g )(1 -αµ g ) and δ 1 = ( 1 + c 2 + L 2 2 ( ρµ g + DL 2 1 + DL 1 µ g ) 2 L 2 1 µ 4 g ) ∥ y 2 -y ∗ 1 ( x 1 ) ∥ 2 + ϵ 2 1 .

By substituting Equation (10) back into Equation (9), we can obtain that

H Proof of Theorem 5.7

Based on Lemma 2.2 in [14], we can have the following lemma to characterize the smoothness of the function f t ( x, y ∗ t ( x )) w.r.t. x for any t ∈ [1 , T ] .

Lemma H.1. Suppose that Assumption 5.1 and Assumption 5.2 hold. Then for any t ∈ [1 , T ] , the function f t ( x, y ∗ t ( x )) is L f -smooth, i.e., for any x and x ′ ,

where the constant L f is given by

For any t ∈ [1 , T ] we first define

Based on Assumption 5.2, we can show the smoothness of F t,η ( x, y ∗ t ( x )) w.r.t. x .

Lemma H.2. Suppose Assumption 5.2 holds. Then the following holds for function F t,η ( x t +1 , y ∗ t ( x t +1 )) :

Proof. For any x and x ′ , we can know that

where (a) holds because of the smoothness of function f t ( x, y ∗ t ( x )) w.r.t. x .

Next, based on Lemma H.2, we can have that such that

(1) We first have the following lemma to bound the term (a) from above:

Lemma H.3. The following inequality holds:

Proof. First, it is clear that

For the term (a.1), we can obtain that

where the inequality holds because of Assumption 5.3.

For the term (a.2), we can have that

By substituting Equation (13) and Equation (14) back to Equation (12), Lemma H.3 can be proved:

(2) Next, for the term (b) which captures the window-averaged hypergradient estimation error, it follows that

∗ - which boils down to characterize the hypergradient estimation error ∥∇ f t ( x t , y t ( x t )) ̂ ∇ f t ( x t , y t +1 ) ∥ 2 on the outer level objective function at each round. By leveraging Theorem 5.6, it is clear that

Let γ = α 2 . For the first term (b.1), it can be seen that for 1 -αµ g 2 < η < 1

For the second term (b.2), we have

such that

where H 2 ,T = ∑ t =2 sup x ∥ y t -1 ( x ) -y t ( x ) ∥ .

Besides, we know that

.

Following the same analysis for the second term (b.2), the following result can be obtained that for the third term (b.3):

Therefore, by substituting Equation (17), Equation (18) and Equation (19) into Equation (16), we can have that

Based on Equation (11), we can have

where (a) and (c) are because βL f < 1 4 , and (b) is because βL f > G 4 G 3 4 L 2 1 G 2 β 2 µ 2 g .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies bilevel optimization in an online setting where the objective in both the levels are allowed to vary with time, and the goal is to develop an algorithm with sublinear regret. This paper proposes a practical single-loop algorithm that updates the lower-level variable only once for each upper-level variable update. The lower-level variable is updated via vanilla SGD, while the hypergradient for the upper-level update is computed via some specified number of steps of Conjugate Gradient, and averaged over a window of specified size. This design significantly improves the computational and memory overhead relative to the existing baseline which requires access to all the past objectives and gradient oracles in the window; the proposed scheme just needs to maintain the hypergradients in the window. Based a novel notion of bilevel regret, the paper shows that the proposed algorithm is able to achieve sublinear regret under some standard assumptions for appropriately set algorithmic parameters and window sizes. The empirical evaluation on two online bilevel applications show that the proposed scheme can match the performance of the existing baseline while being significantly more efficient both in terms of time and memory requirement -- the computational advantage of the proposed scheme is enhanced when considering larger window sizes.\n\nSTRENGTHS:\n**Practical online solution.**\nIn the online bilevel setting, it seems more practical to have a single-loop algorithm since the objectives/gradients for each of the level will probably be made available to the learner in a sequential alternate manner. However, single loop bilevel algorithms are usually harder to analyse. So it is a significant contribution to have a single loop algorithm that has sublinear regret.\n\n**Intuitive presentation of theoretical analyses.**\nThe authors have presented the theoretical analyses in a very intuitive and clear manner. After specifying the necessary assumptions, the authors discuss the necessary steps to complete the analyses, and gradually build up to each of the results. This presentation is very easy to follow for a reader, and I really appreciate the work done for such a presentation. After the main theorem, the author clearly discuss the conditions under which we might achieve the desired sublinear regret.\n\n**Strong empirical performance against baseline.**\nThe experimental results show clearly show that the proposed SOBOW matches the regret of the OAGD baseline, but is able to do so with significantly lower computational and memory overhead, and without access to the past objectives. The computational gains are very significant, with up to almost $20\\times$ speedup. This is an impressive result, making the solution even more practically useful.\n\nWEAKNESSES:\n**Hyperparameters in the definition of regret.**\nOne of the weaknesses of this paper is that the proposed novel notion of bilevel regret itself (equation (2)) seems to depend on the window size $K$ and the decay rate $\\eta$ (and it is not quite clear what the subscript $w$ in $BLR_w(T)$ denotes). Given that the subsequent analysis shows that these quantities need to be set appropriately for the desired convergence rate, it is odd that the notion of regret itself depends on it. The algorithm can use such hyperparameters, but the term quantifying the regret should not depend on them. Is this standard in dynamic local regret analysis? One would expect that the bilevel local regret would be defined as some term such as $\\sum_{t=1}^T || \\nabla_x f_t(x_t, y_t^*(x_t)) ||^2$ or something similar, where we are computing the per-time-step (local) suboptimality, and we would want this quantity to grow sublinearly with $T$ (with appropriate assumptions regarding the relationship between $f_t, g_t$ and $f_{t+1}, g_{t+1}$). It seems as if such a definition of bilevel regret (equation (2)) was considered because it seems to match the form of the upper-level update that is used in the proposed algorithm.\n\n**No dependence on lower-level suboptimality in the regret.**\nAnother issue with the considered notion of bilevel regret is that it is not clear why it is meaningful for the regret to depend on $(x_t, y_t^*(x_t))$ instead of just $(x_t, y_t)$ or $(x_t, y_{t+1})$. Alternately, it is not clear why the sub-optimality in the lower-level decision variable (that is, having $y_{t+1}$ instead of $y_t^*(x_t)$) does not contribute to the regret in any way. Analyses of static bilevel optimization usually establish convergence of $ || \\nabla_x f(x_t, y_t^*(x_t)) ||^2$ **as well as** that of $|| y_{t+1} - y_t^*(x_t) ||^2$.\n\nQUESTIONS:\n- Given existing single-loop static algorithms such as TTSA [A] and STABLE [B], how is the proposed algorithm positioned against these? One of the challenges with single-loop schemes is that the upper-level updates need to be very slow (that is, have a small upper-level learning rate relative to the lower-level learning rate) if we are just using a single SGD step lower-level update. This is because, otherwise, it is hard to guarantee that $y_{t+1}$ converges to $y^*(x_t)$ since the $x_{t-1} \\to x_t$ update can significantly move the lower-level target from $y^*(x_{t-1}) \\to y^*(x_t)$, which a single SGD step is unable to catch up with. That is why, the more expensive but sophisticated lower-level update is proposed in STABLE, to allow for faster upper-level updates. Does this issue manifest in the proposed SOBOW, resulting in a need for a smaller upper-level learning rate (and thus slower convergence), or is there something in the nature of the online bilevel setup that mitigates this issue?\n- Given that $\\{Q_t, t \\in [T]\\}$ would be an increasing sequence, what is the motivation to not just solve the least-squares problem to sufficient optimality at each step and remove the error term in Lemma 5.5, and simplify the analysis?\n\n>[A] Hong, Mingyi, et al. \"A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic.\" SIAM Journal on Optimization 33.1 (2023): 147-180.\n\n>[B] Chen, Tianyi, et al. \"A single-timescale method for stochastic bilevel optimization.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\nLIMITATIONS:\nThe authors do explicitly discuss some limitations, and I do not anticipate any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider bilevel optimization in the online setting. In this setting, we have access at iteration $t$ to the outer function $f_t$ which is assumed to be differentiable and possibly nonconvex. We also have access to the inner function $g_t$ which is assumed to be twice differentiable and strongly-convex with respect to the inner variable $y$. They propose SOBOW, an algorithm that implement approximate implicit differentiation in a single loop fashion. In SOBOW, at iteration $t$, the inner variable is updated by a gradient step and the solution of the linear system involved in the expression of the hypergradient is approximated by Conjugate Gradient steps. Then, the obtained approximate hypergradient is stored and the outer variable $x$ is updated following the opposite direction given by an average of the $K-1$ last approximate hypergradient computed. The outer variable is then projected on the constrains set $\\mathcal{X}$. \nThe authors show that SOBOW achieves a sublinear local regret.\nSOBOW is numerically compared with OGD and OAGD on an online hyper-representation learning task using a simulated dataset, and on an online hyperparameter optimization problem using the 20newsgroups dataset.\n\nSTRENGTHS:\n* The paper is clearly written\n* The authors study online bilevel optimization which has been very little studied in the literature.\n* The proposed method is theoretically grounded\n* The method improves upon previous work by avoiding the evaluation of the previous functions at the current iterates.\n* Numerical validation is provided on several tasks.\n\nWEAKNESSES:\n* The idea of single-loop updates was already exploited in offline context [1, 2, 3]. The authors should mention it.\n\n* Since the authors consider a projection onto $\\mathcal{X}$, this set has to be assumed closed.\n\n* Under a closedness assumption of $\\mathcal{X}$ and $\\mathcal{X}$ being assumed to be bounded, the boundedness of $\\nabla f$ is automatic making assumption 5.4 unecessary. \n\n* In terms of notations, the notation $\\nabla f_t(x, y^*(x))$ is confusing because it can be thought as the gradient of the function $f_t$ evaluated at the point $(x, y^*(x))$ or as the gradient of the function $x\\mapsto f_t(x,y^*(x))$. Maybe it should be clearer to give a name to the function $x\\mapsto f_t(x,y^*(x))$.\n\n[1] M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic.  arXiv:2007.05170, 2021\n\n[2] M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. NeurIPS, 2022.\n\n[3] J. Li, B. Gu, and H. Huang. A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse. AAAI, 2022\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new method for solving online bilevel problem that only required one-step $y$ update and leveraged the historical information to smooth the update. Extensive experiments are provided to validate their theories.\n\nSTRENGTHS:\n1.\tThis work is the second one considering the online bilevel optimization problem and this problem can motivate many applications. \n2.\tThe algorithm does not require multiple inner updates and the evaluation of the current models on previous functions, so that it is more applicable to online setting.\n\nWEAKNESSES:\n1. One of the challenges unique to online bilevel optimization, as compared to its offline counterpart in this paper, lies in controlling the hypergradient estimation error, which is dependent on $ ||y_t^*(x_t)-y_{t+1}^*(x_{t+1})||^2$. Unlike in the offline case, it cannot be simply bounded by $|| x_{t+1} -x_t ||$ due to the time-variant nature of $g_t$. It is impossible to control this term without making variation assumption on the lower-level objective, so it is intriguing to see how to regulate this term by bounded variational assumption of $g_t$. However, Theorem 5.7 seems to circumvent this challenge by directly converting the hypergradient estimation error term to $V_T$ and $H_T$ — terms over which we cannot directly control. Is it possible to characterize these two terms explicitly by the variation of $g_t$? This would offer more insights into the effect on the overall online bilevel optimization.\n\n2. The state-of-the-art work on offline bilevel optimization also adopts the three-level optimization and treats $v$ as a solution to a quadratic problem and thus can eliminate the conjugate gradient loop. As one of the contributions of this work is to reduce the multiple-step lower-level updates to one-step, it is also intriguing to see whether the conjugate gradient loop can be reduced since it is also time-consuming.\n\nQUESTIONS:\nCould the bounded function value in Assumption 5.3 be relaxed to merely on the feasible set? In this way, it can be derived from the bounded feasible set and Lipschitz continuity assumptions. Otherwise, bounded function value on the whole space is relatively restricted. Also, does the objective in the experiment part satisfy this assumption?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the online bilevel optimization problem with nonstationary and time-varying objective functions. This line of research can cover applications with online nature like online meta-learning, online hyperparameter tuning, wireless networks. Compared to widely studied offline bilevel problem, the studied one has challenges like limited information, hypergradient computation, changing objectives. The authors propose a single-loop online bilevel optimizers called SOBOW based on online nonconvex optimization and window averaging, and further show it can attain a sublinear regret. Some experiments are provided to justify the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. Bilevel optimization has been studied intensively mainly in the offline setting where all objective functions are fixed and known. It has been much less explored in the online and nonstationary setting. This work seems to provide a simple and good solution.\n\n2. The authors have done a good job in discussing the underlying challenges and the drawbacks of existing method in [54]. How to design a good online bilevel optimizer with limited queries, efficient hypergradient computation, guaranteed regret turns out to be nontrivial. \n\n3. Technically, this work needs to cope with 1) the intersection among three variables $x,y,v$ in online manner,  2) time-smoothed gradient updates, 3) biased gradient estimation, which may be straightforward.\n\nWEAKNESSES:\n1. The hypergradient estimation contains second-order derivatives. Will them cost a lot in practical online settings? Is it possible to design fully first-order method without matrix-vector computations? There are some recent progress towards the Hessian-free bilevel optimization. \n\n2. The authors do not compare the rate of their method with OAGD.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}