SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sen Lin*
slin50@central.uh.edu
Daouda Sow
sow.53@osu.edu
Yingbin Liang
liang889@osu.edu
Ness Shroff
Department of ECE & CSE
shroff.11@osu.edu
Bilevel optimization has become a powerful tool in a wide variety of machine learning problems. However, the current nonconvex bilevel optimization considers an offline dataset and static functions, which may not work well in emerging online applications with streaming data and time-varying functions. In this work, we study online bilevel optimization (OBO) where the functions can be time-varying and the agent continuously updates the decisions with online streaming data. To deal with the function variations and the unavailability of the true hypergradients in OBO, we propose a single-loop online bilevel optimizer with window averaging (SOBOW), which updates the outer-level decision based on a window average of the most recent hypergradient estimations stored in the memory. Compared to existing algorithms, SOBOW is computationally efficient and does not need to know previous functions. To handle the unique technical difficulties rooted in single-loop update and function variations for OBO, we develop a novel analytical technique that disentangles the complex couplings between decision variables, and carefully controls the hypergradient estimation error. We show that SOBOW can achieve a sublinear bilevel local regret under mild conditions. Extensive experiments across multiple domains corroborate the effectiveness of SOBOW.
Bilevel optimization has attracted significant recent attention, which in general studies the following problem:
$$
\min  _ {x \in \mathbb {R} ^ {d _ {1}}} f (x, y ^ {*} (x)) \quad \text {s . t .} \quad y ^ {*} (x) = \arg \min  _ {y \in \mathbb {R} ^ {d _ {2}}} g (x, y). \tag {1}
$$
Here both the outer-level function  $f$  and the inner-level function  $g$  are continuously differentiable, and the outer optimization problem is solved subject to the optimality of the inner problem. Due to its capability of capturing hierarchical structures in many machine learning problems, this nested optimization framework has been exploited in a wide variety of applications, e.g., meta-learning [4, 31], hyperparameter optimization [13, 3], reinforcement learning [62, 57] and neural architecture search [38, 8].
However, numerous machine learning problems with hierarchical structures are online in nature, e.g., online meta-learning [11] and online hyperparameter optimization [63], where the current bilevel
*The work was done when Sen Lin was in The Ohio State University
optimization framework cannot be directly applied due to the following reasons: (1) (streaming data) The nature of online streaming data requires decision making on-the-fly with low regret, whereas the offline framework emphasizes more on the quality of the final solution; (2) (time-varying functions) The objective functions in online applications can be time-varying because of non-stationary environments and changing tasks, in contrast to static functions considered in Equation (1); (3) (limited information) The online learning is nontrivial when the decision maker only has limited information, e.g., regarding the inner-level function, which can be even more challenging with time-varying functions. For example, in wireless network control [37], the controller has to operate under limited knowledge about the time-varying wireless channels (see Appendix).
To reap the success of bilevel optimization in online applications, there is an urgent need to develop a new online bilevel optimization (OBO) framework. Generally speaking, OBO considers the scenario where the data comes in an online manner and the agent continuously updates her outer-level decision based on the estimation of the optimal inner-level decision. Both outer-level and inner-level objective functions can be time-varying to capture the data distribution shifts in many online scenarios. Note that OBO is significantly different from single-level online optimization [22], due to the unavailability of the true outer-level objective function composed by  $f$  and  $y^{*}$ .
The study of OBO was recently initiated in [59], but much of this new framework still remains under-explored and not well understood. In particular, [59] combines offline bilevel optimization with online optimization and proposes an online alternating time-averaged gradient method. Such an approach suffers from several limitations: 1) Multi-step update is required for inner-level decision variable  $y_{t}$  at each time  $t$ , which can be problematic when only limited information of the inner-level function  $g$  is available. 2) The hypergradient estimation at each time requires the knowledge of previous objective functions in a window, and also evaluates current models on each previous function; such a design can be inefficient and infeasible in online scenarios. In this work, we seek to address these limitations and develop a new OBO algorithm that can work efficiently without the knowledge of previous objective functions.
Table 1: Comparison of OBO algorithms. 'HV' product refers to the Hessian inverse-vector product in hypergradient estimation. OAGD estimates the hypergradient by  $\frac{1}{W}\sum_{i=0}^{K-1}\eta^i\hat{\nabla}f_{t-i}(x_t,y_{t+1})$  which requires the evaluation of  $f_{t-i}$  on current model  $(x_t,y_{t+1})$ .
Algorithm	Single-loop update	Study estimation error of HV product	Do not require previous function info
OAGD [59]	X	X	X
SOBOW (this paper)	✓	✓	✓
The main contributions can be summarized as follows.
(Efficient algorithm design) We propose a new single-loop online bilevel optimizer with window averaging (SOBOW), which works in a fully online manner with limited information about the objective functions. In contrast to the OAGD algorithm in [59], SOBOW has the following major differences (as summarized in Table 1): (1) (single-loop update) Compared to the multi-step updates of the inner-level decision  $y_{t}$  at each round in OAGD, we only require a one-step update of  $y_{t}$ , which is more practical for online applications where only limited information about the inner-level function  $g_{t}$  is available. (2) (estimation error of Hessian inverse-vector product) Estimating the hypergradient requires the outer-level Hessian inverse-vector product. [59] assumes that the exact Hessian inverse-vector product can be obtained, which can introduce high computational cost. In contrast, we consider a more practical scenario where there could be an estimation error in the Hessian-inverse vector product calculation in solving the linear system. (3) (window averaged hypergradient descent) While a window averaged hypergradient estimation is considered in both OAGD and SOBOW, SOBOW is more realistic and efficient than OAGD. Specifically, OAGD requires the knowledge of the most recent objective functions within a window and evaluates the current model on each previous function at every round, whereas SOBOW only stores the historical hypergradient estimation in the memory without any additional knowledge or evaluations about previous functions.
(Novel regret analysis) Based on the previous studies in single-level online non-convex optimization [23, 2], we introduce a new bilevel local regret as the performance measure of OBO algorithms. We show that the proposed SOBOW algorithm can achieve a sublinear bilevel local regret under mild conditions. Compared to offline bilevel optimization and OAGD, new technical challenges need to be addressed here: (1) unlike the multi-step update of inner-level variable  $y_{t}$  in OAGD, the single-step update in SOBOW will lead to an inaccurate estimation of the optimal inner-level
decision  $y_{t}^{*}(x_{t})$  and consequently a large estimation error for the hypergradient; this problem can be addressed in offline bilevel optimization by controlling the gap  $\| y_{t}^{*}(x_{t}) - y_{t + 1}^{*}(x_{t + 1})\|^{2}$ , which depends only on  $\| x_{t} - x_{t + 1}\|^{2}$  for static inner-level functions. But this technique cannot be applied here due to the time-varying  $g_{t}$ ; (2) the function variations in OBO can blow up the impact of the hypergradient estimation error if not handled appropriately, whereas offline bilevel optimization does not have this issue due to the static functions therein. Towards this end, we appropriately control the estimation error of the Hessian inverse-vector product at each round, and disentangle the complex couplings between the decision variables through a three-level analysis. This enables the control of the hypergradient estimation error, by using a decaying coefficient to diminish the impact of the large inner-level estimation error and leveraging the historical information to smooth the update of the outer-level decision.
(Extensive experimental evaluations) OBO has the potential to be used in various online applications, by capturing the hierarchical structures therein in an online manner. In this work, the experimental results clearly validate the effectiveness of SOBOW in online hyperparameter optimization and online hyper-representation learning.
Online optimization Online optimization has been extensively studied for strongly convex and convex functions in terms of both static regret, e.g., [68, 21, 47] and dynamic regret, e.g., [5, 48, 61, 64, 66]. Recently, there has been increasing interest in studying online optimization for non-convex functions, e.g., [1, 58, 34, 51, 15], where minimizing the standard definitions of regret is computationally intractable. Specifically, [23] introduced a notion of local regret in the spirit of the optimality measure in non-convex optimization, and developed an algorithm that averages the gradients of the most recent loss functions evaluated in the current model. [2] proposed a dynamic local regret to handle the distribution shift and also a computationally efficient SGD update for achieving sublinear regret. [24] and [25] studied the zeroth-order online non-convex optimization where the agent only has access to the actual loss incurred at each round.
Offline bilevel optimization Bilevel optimization was first introduced in the seminal work [6]. Following this work, a number of algorithms have been developed to solve the bilevel optimization problem. Initially, the bilevel problem was reformulated into a single-level constrained problem based on the optimality conditions of the inner-level problem [20, 55, 43, 49], which typically involves many constraints and is difficult to implement in machine learning problems. Recently, gradient-based bilevel optimization algorithms have attracted much attention due to their simplicity and efficiency. These can be roughly classified into two categories, the approximate implicit differentiation (AID) based approach [10, 52, 16, 14, 17, 42, 32] and the iterative differentiation (ITD) based approach [45, 12, 53, 44, 17]. Bilevel optimization has also been studied very recently for the cases with stochastic objective functions [14, 31, 7, 33, 18] and multiple inner minima [35, 40, 39, 56]. Notably, a novel value-function based method was first proposed in [39] to deal with the non-convexity of the inner-level functions. Some recent studies, e.g., [36, 9, 26], have also explored single-level algorithms for offline bilevel optimization with time-invariant objective functions, whereas the time-varying functions in online bilevel optimization make the hypergradient estimation error control more challenging for single-loop updates.
Online bilevel optimization The investigation of OBO is still in the very early stage, and to the best of our knowledge, [59] is the only work so far that has studied OBO.
3 Online Bilevel Optimization
Following the same spirit as the online optimization in [22], the decisions are made iteratively in OBO without knowing their outcomes at the time of decision-making. Let  $T$  denote the total number of rounds in OBO. Define  $x_{t} \in \mathcal{X} \subset \mathbb{R}^{d_{1}}$  and  $f_{t}: \mathcal{X} \times \mathbb{R}^{d_{2}}$  as the decision variable and the online function for the outer level problem, respectively. Define  $y_{t} \in \mathbb{R}^{d_{2}}$  and  $g_{t} \in \mathcal{X} \times \mathbb{R}^{d_{2}}$  as the decision variable and the online objective function for the inner level problem, respectively. Given the initial values of  $(x_{1}, y_{1})$ , the general procedure of OBO is described in Algorithm 1.
Let  $y_{t}^{*}(x) = \arg \min_{y\in \mathbb{R}^{d_{2}}}g_{t}(x,y)$  for any  $x$ . The OBO framework in Algorithm 1 can be interpreted from two different perspectives: (1) (single-player) The player makes the decision on  $x_{t}$  without
Algorithm 1 General procedure of OBO
1: Initialize decisions  $x_{1}$  and  $y_{1}$
2: for  $t = 1, \dots, T$  do
3: Get information about functions  $f_{t}$  and  $g_{t}$
4: Update decision  $y_{t + 1}$  based on  $x_{t}$  and  $g_{t}$
5: Update decision  $x_{t + 1}$  based on  $y_{t + 1}$  and  $f_t$
6: end for
knowing the optimal inner-level decision  $y_{t}^{*}(x)$ . Note,  $y_{t}$  serves as an estimation of  $y_{t}^{*}(x)$  from the player's perspective based on her knowledge of function  $g_{t}$ ; (2) (two-player) OBO can also be viewed as a leader  $(x_{t})$  and follower  $(y_{t})$  game, where each player considers an online optimization problem and the leader seeks to play against the optimal decision  $y_{t}^{*}(x)$  of the follower at each round under limited knowledge of  $g_{t}$ .
It is worthwhile noting that OBO is quite different from the single-level online optimization. First, the outer-level objective function with respect to (w.r.t)  $x_{t}$ , i.e.,  $f_{t}(x_{t},y_{t}^{*}(x_{t}))$ , is not available to update  $x_{t}$ , whereas, in standard single-level online optimization, the true loss is revealed immediately after making decisions. Besides, as a composite function of  $f_{t}(x,y)$  and  $y_{t}^{*}(x)$ ,  $f_{t}(x,y_{t}^{*}(x))$  is non-convex in general w.r.t. the outer-level decision variable  $x$ . Hence, standard regret definitions in online convex optimization [22] are not directly applicable here.
Motivated by the dynamic local regret defined in online non-convex optimization [2], we consider the following bilevel local regret:
$$
B L R _ {w} (T) = \sum_ {t = 1} ^ {T} \| \nabla F _ {t, \eta} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) \| ^ {2} \tag {2}
$$
$$
F _ {t, \eta} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) = \frac {1}{W} \sum_ {i = 1} ^ {K - 1} \eta^ {i} f _ {t - i} \left(x _ {t - i}, y _ {t - i} ^ {*} \left(x _ {t - i}\right)\right),
$$
and  $W = \sum_{i=0}^{K-1} \eta^i$ ,  $\eta \in (0,1)$ , and  $f_t(\cdot, \cdot) = 0$  for  $t \leq 0$ . In contrast, the static regret in [59] evaluates the objective at time slot  $i$  using variable updates at different time slot  $j$ , which does not properly characterize the online learning performance of the model update for time-varying functions (see Appendix for more discussion). Intuitively, the regret in Equation (2) is defined as a sliding average of the hypergradients w.r.t. the decision variables at the corresponding instant for all rounds in OBO. This indeed approximately computes the exponential average of the outer-level function values  $f_t(x_t, y_t^*(x_t))$  at the corresponding decision variables over a sliding window [2]. Larger weights will be assigned to more recent updates. The objective here is to design efficient OBO algorithms with sublinear bilevel regret  $BLR_w(T)$  in  $T$ , which implies that the outer-level decision is becoming better and closer to the local optima for the outer-level optimization problem at each round. This gradient-norm based regret shares the same spirit as the first-order optimality criterion [14, 32], which is widely used in offline bilevel optimization to characterize the convergence to the local optima.
It is well known that online gradient descent (OGD) [54] has achieved great successes in single-level online optimization. On the other hand, gradient-based methods (e.g., [16, 14, 42, 40, 29]) have become extremely popular for solving offline bilevel optimization due to their high efficiency. Thus, we also study the online gradient descent based algorithm to solve the OBO problem. As mentioned earlier, the unique challenges of OBO should be carefully handled in the algorithm design, including 1) the inaccessibility of the objective function and accurate hypergradients compared to single-level online optimization and 2) the time-varying functions and limited information compared to offline bilevel optimization. To this end, our algorithm includes two major designs, i.e., efficient hypergradient estimation with limited information and window averaged outer-level decision update.
Hypergradient estimation In OBO, the exact hypergradient  $\nabla f_t(x_t,y_t^* (x_t))$  w.r.t.  $x_{t}$  can be represented as
$$
\nabla f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) = \nabla_ {x} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) v _ {t} ^ {*} \tag {3}
$$
where  $v_{t}^{*}$  solves the linear system  $\nabla_y^2 g_t(x_t, y_t^*(x_t))v = \nabla_y f_t(x_t, y_t^*(x_t))$ . The optimal inner-level decision  $y_{t}^{*}(x_{t})$  is generally unavailable in OBO. To estimate the hypergradient  $\nabla f_{t}(x_{t},y_{t}^{*}(x_{t}))$ ,
the AID-based approach [10, 14, 32] for offline bilevel optimization can be leveraged here, which will involve the following steps: (1) given  $x_{t}$ , run  $N$  steps of gradient descent w.r.t. the inner-level objective function  $g_{t}$  to find a good approximation  $y_{t}^{N}$  close to  $y_{t}^{*}(x_{t})$ ; (2) given  $y_{t}^{N}$ , obtain  $v_{t}^{Q}$  by solving  $\nabla_y^2 g_t(x_t,y_t^N)v = \nabla_y f_t(x_t,y_t^N)$  with  $Q_{t}$  steps of conjugate gradient. The estimated hypergradient is constructed as
$$
\widehat {\nabla} f _ {t} \left(x _ {t}, y _ {t} ^ {N}\right) = \nabla_ {x} f _ {t} \left(x _ {t}, y _ {t} ^ {N}\right) - \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t} ^ {N}\right) v _ {t} ^ {Q}. \tag {4}
$$
Nevertheless, the  $N$  steps gradient descent for estimating  $y_{t}^{*}(x_{t})$  require multiple inquiries about the inner-level function  $g_{t}$ , which can be inefficient and infeasible for online applications. For the algorithm being used in more practical scenarios with limited information about  $g_{t}$ , we consider the extreme case where  $N = 1$ , i.e.,
$$
y _ {t + 1} = y _ {t} ^ {1} = y _ {t} - \alpha \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t}\right), \tag {5}
$$
where  $\alpha$  is the inner-level step size. This would lead to an inaccurate estimation of  $y_{t}^{*}(x_{t})$ , which can pose critical challenges for making satisfying outer-level decisions in OBO due to the unreliable hypergradient estimation, especially when the objective functions are time-varying.
Window averaged decision update To deal with the non-convex and time-varying functions, inspired by time-smoothed gradient descent in online non-convex optimization [23, 2, 67, 19], we consider a time-smoothed hypergradient descent for updating the outer-level decision variable  $x_{t}$ :
$$
x _ {t + 1} = \Pi_ {\mathcal {X}} \left(x _ {t} - \beta \widehat {\nabla} F _ {t, \eta} \left(x _ {t}, y _ {t + 1}\right)\right) \tag {6}
$$
where  $\Pi_{\mathcal{X}}$  is the projection onto the set  $\mathcal{X}$ ,  $\beta$  is the outer-level step size and
$$
\widehat {\nabla} F _ {t, \eta} \left(x _ {t}, y _ {t + 1}\right) = \frac {1}{W} \sum_ {i = 1} ^ {K - 1} \eta^ {i} \widehat {\nabla} f _ {t - i} \left(x _ {t - i}, y _ {t + 1 - i}\right). \tag {7}
$$
Here  $\widehat{\nabla} f_{t - i}(x_{t - i},y_{t + 1 - i})$  is the hypergradient estimation at the round  $t - i$ , as in Equation (4). Intuitively, the update of the current  $x_{t}$  is smoothed by the historical hypergradient estimations w.r.t. the decision variables at that time, which is particularly important here for OBO due to the following reasons: (1) To compute the averaged  $\widehat{\nabla} F_{t,\eta}(x_t,y_{t + 1})$  at each round  $t$ , we only store the hypergradient estimation for previous rounds in the memory, i.e., store  $\widehat{\nabla} f_i(x_i,y_{i + 1})$  at each round  $i$ , and estimate the current hypergradient  $\widehat{\nabla} f_t(x_t,y_{t + 1})$  using the available information at current round  $t$ . Compared to OAGD in [59], there is no need to access to the previous outer-level and inner-level objective functions and evaluate the current decisions on those functions, which is clearly more efficient and practical for online applications. (2) Leveraging the historical information in the window to update the current decision is helpful to deal with the inaccurate hypergradient estimation for the current round, especially under mild function variations. This indeed shares the same rationale with using past tasks to facilitate forward knowledge transfer in online meta-learning [11] and continual learning [41], for better decision making in the current task and improving the overall performance in non-stationary environments.
Building upon these two major components, we can have our main OBO algorithm, Single-loop Online Bilevel Optimizer with Window averaging (SOBOW), as summarized in Algorithm 2. At the round  $t$ , we first estimate  $y_{t + 1}$  as in Equation (5) given  $x_{t}$  and  $y_{t}$ , which will be next leveraged to solve the linear system and construct the hypergradient estimation as in Equation (4). Based on the historical hypergradient estimations stored in the memory for previous rounds, we next update  $x_{t}$  based on Equation (6).
In this section, we provide the theoretical analysis of the regret bound for SOBOW.
5.1 Technical Assumptions
Let  $z = (x, y)$ . Before the regret analysis, we first make the following assumptions.
Assumption 5.1. The inner-level function  $g_{t}(x,y)$  is  $\mu_g$ -strongly convex w.r.t.  $y$ , and the composite objective function  $f_{t}(x,y_{t}^{*}(x))$  is non-convex w.r.t  $x$ .
Assumption 5.2. The following conditions hold for objective functions  $f_{t}(z)$  and  $g_{t}(z)$ ,  $\forall t \in [1, T]$ : (1) The function  $f_{t}(z)$  is  $L_{0}$ -Lipschitz continuous; (2)  $\nabla f_{t}(z)$  and  $\nabla g_{t}(z)$  are  $L_{1}$ -Lipschitz continuous; (3) The high-order derivatives  $\nabla_{x} \nabla_{y} g_{t}(z)$  and  $\nabla_{y}^{2} g_{t}(z)$  are  $L_{2}$ -Lipschitz continuous.
Algorithm 2 Single-loop Online Bilevel Optimizer with Window averaging (SOBOW)
1: Initialize decisions  $x_{1},y_{1},v^{0}$    
2: for  $t = 1,\dots,T$  do   
3: Get information about functions  $f_{t}$  and  $g_{t}$    
4: Update  $y_{t + 1}$  based on Equation (5)   
5: Solve the linear system  $\nabla_y^2 g_t(x_t,y_{t + 1})v = \nabla_y f_t(x_t,y_{t + 1})$  using  $Q_{t}$  steps of conjugate gradient starting from a fixed point  $v^{0}$  with stepsize  $\lambda$  to obtain  $v_{t}^{Q}$    
6: Construct the hypergradient  $\widehat{\nabla} f_t(x_t,y_{t + 1})$  based on Equation (4) with  $y_{t}^{N} = y_{t + 1}$    
7: Store  $\widehat{\nabla} f_t(x_t,y_{t + 1})$  in the memory   
8: Update  $x_{t + 1}$  based on Equation (6)   
9: end for
Note that both Assumption 5.1 and Assumption 5.2 are standard and widely used in the literature of bilevel optimization, e.g., [52, 14, 29, 59, 27].
Assumption 5.3. For any  $t \in [1, T]$ , the function  $f_{t}(x, y_{t}^{*}(x))$  is bounded, i.e.,  $|f_{t}(x, y_{t}^{*}(x))| \leq M$  with  $M > 0$ . Besides, the closed convex set  $\mathcal{X}$  is bounded, i.e.,  $\| x - x' \| \leq D$  with  $D > 0$ , for any  $x$  and  $x'$  in  $\mathcal{X}$ .
Assumption 5.3 on the boundedness of the objection functions is also standard in the literature of non-convex optimization, e.g., [23, 2, 50, 59]. Moreover, to guarantee the boundedness of the hypergradient estimation error, previous studies (e.g., [14, 28, 17]) in offline bilevel optimization usually assume that the gradient norm  $\| \nabla f(z) \|$  is bounded from above for all  $z$ . In this work, we make a weaker assumption on the feasibility of  $\nabla_y f_t(x, y_t^*(x))$ , which generally holds since  $y_t^*(x)$  is usually assumed to be bounded in bilevel optimization: