SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 5.4. There exists at least one  $\hat{x} \in \mathcal{X}$  such that  $\| \nabla_y f_t(\hat{x}, y_t^*(\hat{x})) \| \leq \rho$  where  $\rho > 0$  is some constant.
5.2 Theoretical Results
Technical challenges in analysis: To analyze the regret performance of SOBOW, several key and unique technical challenges need to be addressed, compared to offline bilevel optimization and OAGD in [59]: (1) unlike the multi-step update of inner-level variable  $y_{t}$  in OAGD, the single-step update in SOBOW will lead to an inaccurate estimation of  $y_{t}^{*}(x_{t})$  and consequently a large estimation error for the hypergradient  $\nabla f_{t}(x_{t},y_{t}^{*}(x_{t}))$ ; this problem can be addressed in offline bilevel optimization by controlling the gap  $\| y_{t}^{*}(x_{t}) - y_{t + 1}^{*}(x_{t + 1})\|^{2}$ , which depends only on  $\| x_{t} - x_{t + 1}\|^{2}$  for static inner-level functions, but this technique cannot be applied here due to the time-varying  $g_{t}$ ; (2) the function variations in OBO can blow up the impact of the hypergradient estimation error if not handled appropriately, whereas offline bilevel optimization does not have this issue due to the static functions therein; (3) a new three-level analysis is required to understand the involved couplings among the estimation errors about  $v_{t}$ ,  $y_{t}$  and  $x_{t}$  in online learning.
Towards this end, the very first step is to understand the estimation error of the optimal solution  $v_{t}^{*}$  to the linear system  $\nabla_y^2 g_t(x_t,y_t^* (x_t))v = \nabla_yf_t(x_t,y_t^* (x_t))$ . Here  $v_{t}^{*} = (\nabla_{y}^{2}g_{t}(x_{t},y_{t}^{*}(x_{t}))^{-1}\nabla_{y}f_{t}(x_{t},y_{t}^{*}(x_{t}))$ . We have the following lemma about  $\| v_{t}^{Q} - v_{t}^{*}\|^{2}$ , where  $v_{t}^{Q}$  is obtained by solving  $\nabla_y^2 g_t(x_t,y_{t + 1})v = \nabla_yf_t(x_t,y_{t + 1})$  using  $Q_{t}$  steps of conjugate gradient.
Lemma 5.5. Suppose Assumptions 5.1-5.4 hold,  $\lambda \leq \frac{1}{L_1}$ ,  $\alpha \leq \frac{1}{L_1}$  and  $Q_{t + 1} - Q_t \geq \frac{\log(1 - \frac{\alpha\mu_g}{2})}{2\log(1 - \lambda\mu_g)}$ . We can have that
$$
\left\| v _ {t} ^ {Q} - v _ {t} ^ {*} \right\| ^ {2} \leq c _ {2} \left\| y _ {t + 1} - y _ {t} ^ {*} \left(x _ {t}\right) \right\| ^ {2} + \epsilon_ {t} ^ {2}
$$
where  $c_{2} > 0$  is some constant and the error  $\epsilon_t^2$  decays with  $t$ , i.e.,  $\epsilon_{t + 1}^{2}\leq (1 - \alpha \mu_{g} / 2)\epsilon_{t}^{2}$ .
Lemma 5.5 characterizes the estimation error  $\| v_{t}^{Q} - v_{t}^{*}\|^{2}$  in a neat way, by constructing an upper bound with the estimation error of the inner-level optimal decision  $y_{t}^{*}(x_{t})$ , i.e.,  $\| y_{t + 1} - y_t^* (x_t)\| ^2$ , and a decaying error term  $\epsilon_t^2$ . The way of controlling  $\| v_{t}^{Q} - v_{t}^{*}\|^{2}$  here is particularly important, which not only clarifies the coupling between  $v_{t}$  and  $y_{t}$  but also helps to control the hypergradient estimation
error. Note that solving the linear system with a larger  $Q_{t}$  does not require more information about the inner-level function, and the introduced computation cost can be negligible, because the conjugate gradient only involves Hessian-vector product which can be efficiently computed.
Next we seek to bound the hypergradient estimation error  $\| \nabla f_t(x_t, y_t^*(x_t)) - \widehat{\nabla} f_t(x_t, y_{t+1})\|^2$  at the round  $t$ . Intuitively, the hypergradient estimation error depends on both  $\| y_{t+1} - y_t^*(x_t)\|^2$  and  $\| v_t^Q - v_t^*\|^2$ . Building upon Lemma 5.5, this dependence can be shifted to the joint error of  $\| y_{t+1} - y_t^*(x_t)\|^2$  and  $\epsilon_t^2$ , which contains iteratively decreasing components after careful manipulations. Specifically, let  $G_1 = 1 + c_2 + \frac{L_2^2 (\rho \mu_g + D L_1^2 + D L_1 \mu_g)^2}{L_1^2 \mu_g^4}$  and  $G_2 = 2 G_1 (1 + \frac{2}{\alpha \mu_g}) (1 - \alpha \mu_g)$ . We have the following theorem to characterize the hypergradient estimation error.
Theorem 5.6. Suppose that Assumptions 5.1-5.4 hold,  $\lambda \leq \frac{1}{L_1}$ ,  $\alpha \leq \frac{1}{L_1}$  and  $Q_{t} - Q_{t - 1} \geq \frac{\log(1 - \frac{\alpha\mu_g}{2})}{2\log(1 - \lambda\mu_g)}$ . For  $t \in [2,T]$ , we can bound the hypergradient estimation error as follows:
$$
\begin{array}{l} \left\| \nabla f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \widehat {\nabla} f _ {t} \left(x _ {t}, y _ {t + 1}\right) \right\| ^ {2} \leq 3 L _ {1} ^ {2} \left\{\frac {2 L _ {1} ^ {2} G _ {2}}{\mu_ {g} ^ {2}} \sum_ {j = 0} ^ {t - 2} \left(1 - \frac {\alpha \mu_ {g}}{2}\right) ^ {j} \left\| x _ {t - 1 - j} - x _ {t - j} \right\| ^ {2} \right. \\ + G _ {2} \sum_ {j = 0} ^ {t - 2} \left(1 - \frac {\alpha \mu_ {g}}{2}\right) ^ {j} \left\| y _ {t - 1 - j} ^ {*} \left(x _ {t - 1 - j}\right) - y _ {t - j} ^ {*} \left(x _ {t - 1 - j}\right) \right\| ^ {2} + \left(1 - \frac {\alpha \mu_ {g}}{2}\right) ^ {t - 1} \delta_ {1} \Biggr \} \\ \end{array}
$$
where  $\delta_{1} = G_{1}\| y_{2} - y_{1}^{*}(x_{1})\|^{2} + \epsilon_{1}^{2}$
As shown in Theorem 5.6, the upper bound of the hypergradient estimation error includes three terms: (1) The first term decays with  $t$ , which captures the iteratively decreasing component in the joint error of  $\| y_{t + 1} - y_t^* (x_t)\| ^2$  and  $\epsilon_t^2$ ; (2) The second term characterizes the dependence on the variation of the outer-level decision between adjacent rounds in the history; (3) The third term characterizes the dependence on the variation of the optimal inner-level decision between adjacent rounds.
To control the hypergradient estimation error as in Theorem 5.6, the key idea is to decouple the source of the estimation error  $\| y_{t + 1} - y_t^* (x_t)\|^2$  at the current round into three different components, i.e.,  $\| y_t - y_{t - 1}^* (x_{t - 1})\|^2$  for the previous round, the variation of the out-level decision, and the variation of the optimal inner-level decision. Since the inner-level estimation error is large due to the single-step update, we diminish its impact through a decaying coefficient, which inevitably enlarges the impact of the other two components. The variation of the optimal inner-level decision is due to nature of the OBO problem, which cannot be controlled. One has to impose some regularity constraints on this variation to achieve a sublinear regret, in the same spirit to the regularities on functional variations widely used in the dynamic regret literature (e.g., [5, 66]). Therefore, the key point now becomes the control of the variation of the out-level decision  $\| x_{t - 1 - j} - x_{t - j}\|^2$ , which can be achieved through the window averaged update of  $x_{t}$  in Equation (6). Intuitively, by leveraging the historical information, the window averaged hypergradient in Equation (7) smooths the outer-level decision update, which serves as a better update direction compared to the deviated single-round estimation  $\widehat{\nabla} f_t(x_t,y_{t + 1})$ . Before presenting the main result, we first introduce the following definitions to characterize the variations of the objective function  $f_{t}(\cdot ,y_{t}^{*}(\cdot))$  and the optimal inner-level decision  $y_{t}^{*}(\cdot)$  in OBO, respectively:
$$
V _ {1, T} = \sum_ {t = 1} ^ {T} \sup  _ {x} [ f _ {t + 1} (x, y _ {t + 1} ^ {*} (x)) - f _ {t} (x, y _ {t} ^ {*} (x)) ], H _ {2, T} = \sum_ {t = 2} ^ {T} \sup  _ {x} \| y _ {t - 1} ^ {*} (x) - y _ {t} ^ {*} (x) \| ^ {2}.
$$
Intuitively,  $V_{1,T}$  measures the overall fluctuations between the adjacent objective functions in all rounds under the same outer-level decision variable, and  $H_{2,T}$  can be regarded as the inner-level path length to capture the variation of the optimal inner-level decisions as in [59]. Note that  $V_{1,T}$  is a weaker regularity for the functional variation compared to absolute values used in single-level online optimization for dynamic regret [5, 66]. When the functions are static, these variations terms are simply 0. We are interested in the case where both  $V_{1,T}$  and  $H_{2,T}$  are  $o(T)$  as in the literature of dynamic regret.
Based on Theorem 5.6, we can have the following theorem to characterize the regret of SOBOW.
Theorem 5.7. Suppose that Assumptions 5.1-5.4 hold. Let  $\lambda \leq \frac{1}{L_1}$ ,  $\alpha \leq \frac{1}{L_1}$ ,  $Q_{t+1} - Q_t \geq \frac{\log(1 - \frac{\alpha\mu_g}{2})}{2\log(1 - \lambda\mu_g)}$ ,  $\eta \in (1 - \frac{\alpha\mu_g}{2}, 1)$ , and  $\beta \leq \min\left\{\frac{1}{4L_f}, \frac{\mu_g^2 L_f W(1 - \eta)(\eta - 1 + \alpha\mu_g / 2)}{24L_1^4 G_2\eta}\right\}$  where  $L_f$  is the
smoothness parameter of the function  $f_{t}(\cdot ,y_{t}^{*}(\cdot))$ . Then we can have
$$
B L R _ {w} (T) \leq O \left(\frac {T}{\beta W} + \frac {V _ {1 , T}}{\beta} + H _ {2, T}\right).
$$
The value of  $L_{f}$  can be found in the proof in Appendix. Note that the hypergradient estimation at the current round depends on all previous outer-level decisions as shown in Theorem 5.6. While these decisions may not be good at the early stage in OBO, choosing  $\eta \in (1 - \frac{\alpha\mu_g}{2}, 1)$  would diminish their impact on the local regret. When the variations  $V_{1,T}$  and  $H_{2,T}$  are both  $o(T)$ , a sublinear bilevel local regret can be achieved for an appropriately selected window, e.g.,  $W = o(T)$  when  $\eta = 1 - h(T)$ , where  $h(T) \to 0$  as  $T \to \infty$ . Note that the value  $\beta$  does not change substantially since  $W(1 - \eta)$  converges to 1. Particularly, when  $\eta = 1 - o(\frac{1}{T})$ ,  $W = \omega(T)$ . In this case, we can have the smallest regret  $O\left(\frac{V_{1,T}}{\beta} + H_{2,T}\right)$  that only depends on the function variations in OBO.
In this section, we conduct experiments in multiple domains to corroborate the utility of the OBO framework and the effectiveness of SOBOW.
(a) static setup
(b) dynamic setup
Figure 1: Evaluation for online HR. As shown in subfigures (a) and (b), SOBOW performs similarly to OAGD and significantly outperforms OGD in both static and dynamic setups. Subfigure (c) shows the performance of SOBOW under different values of the averaging parameter  $\eta$  for online HR. Better performance is achieved as  $\eta \rightarrow 1$ . Subfigure (d) shows the performance of SOBOW under different values of inner steps  $N$  when the data stream contains two data points (one in  $X_{t}^{g}$  and one in  $X_{t}^{f}$ ). The performance saturates at  $N = 2$ .
(c) impact of  $\eta$
(d) impact of  $N$
Specifically, we compare our algorithm SOBOW with the following baseline methods: (1) OAGD [59], which is the only method for OBO in the literature; (2) OGD, a natural method which updates the outer-level decision by using the current hypergradient estimation only without any window averaging. Intuitively,  $OGD$  is not only a special case of OAGD when the information of previous functions is not available, but also a direct application of offline bilevel optimization, e.g., AID-based method [32]. We also denote SOBOW- $K$ /OAGD- $K$  as SOBOW and OAGD with window size  $K$ , respectively. And we evaluate the regret using the definition in Equation (2). We also compare the performance using the regret in [59] in Appendix where similar results can be observed.
Online Hyper-representation Learning Representation learning [13, 17] seeks to extract good representations of the data. The learnt representation mapping can be used in downstream tasks to facilitate the learning of task specific model parameters. This formulation is typically encountered in a multi-task setup, where  $\Lambda$  captures the common representation extracted for multiple tasks and  $w$  defines the task-specific model parameters. When the data/task arrives in an online manner, the hyper-representation needs to be continuously adapted to incorporate the new knowledge.
Following [17], we study online hyper-representation learning (Online HR) with linear models. Specifically, at each round  $t$ , the agent applies the hyper-representation  $\Lambda_t \in \mathbb{R}^{p \times d}$  and the linear model prediction  $w_t \in \mathbb{R}^d$ , and then receives small minibatches  $(X_t^f, Y_t^f)$  and  $(X_t^g, Y_t^g)$ . Based on  $\Lambda_t$  and  $(X_t^g, Y_t^g)$ , the agent updates her linear model prediction  $w_{t+1}$  as an estimation of  $w^*(\Lambda_t) = \arg \min_{w \in \mathbb{R}^d} g_t(\Lambda_t, w) := \| X_t^g \Lambda_t w - Y_t^g\|^2 + \frac{\gamma}{2}\| w\|^2$ . Based on the estimation  $w_{t+1}$  and  $(X_t^f, Y_t^f)$ , the agent further updates her decision  $\Lambda_{t+1}$  about the hyper-representation to minimize the loss  $f_t(\Lambda, w_t^*(\Lambda)) := \| X_t^f \Lambda w_t^*(\Lambda) - Y_t^f\|^2$ . In our experiments, we consider synthetic data generated as in [17] and explore two distinct settings: (i) a static setup where the underlying model generating the minibatches is fixed; and (ii) a staged dynamic setup where the model changes after some steps.
As shown in Figure 1(a) and Figure 1(b), SOBOW achieves comparable regret with OAGD in both static and dynamic setups, without the need of knowing previous functions. In terms of the running
Table 2: Left: Comparison for static online HO. We report accuracy and loss on a separate test split after 12000 steps. Right: Comparison for dynamic online HO. We report accuracy on a separate test split at the end of stream with corruption level  $20\%$  and  $30\%$ . Each level lasts for 4000 steps.
Method	Accuracy (%)	Test Loss	Time (s)
SOBOW-4	65.87	1.287	899
OAGD-4	65.96	1.285	2304
SOBOW-50	66.32	1.28	1188
OAGD-50	66.44	1.273	20161
Method	End 20% stream	End 30% stream	Time (s)
SOBOW-4	58.39	59.70	1198
OAGD-4	62.61	59.26	3072
time for 5000 steps with  $K = 50$ , SOBOW takes 11 seconds, OAGD takes 228 seconds and OGD takes 7 seconds. Therefore, SOBOW is much more computationally efficient compared to OAGD, because SOBOW does not need to re-evaluate the previous functions on the current model at each round. On the other hand, SOBOW performs substantially better than OGD (i.e., OAGD when previous functions are not available) with similar running time. These results not only demonstrate the usefulness of SOBOW when the previous functions are not available, but also corroborate the benefit of window-averaged outer-level decision update by leveraging the historical hypergradient estimations in OBO. Figure 1(c) shows the performance of SOBOW under different values of the averaging parameter  $\eta$ . The performance is better as  $\eta \rightarrow 1$ , which is also consistent with our theoretical results. Figure 1(d) indicates that a small number of updates for the inner-level variable is indeed enough for online HR.
Online Hyperparameter Optimization The goal of hyperparameter optimization (HO) [13, 17] is to search for the best values of hyperparameters  $\lambda$ , which seeks to minimize the validation loss of the learnt model parameters  $w$  and is usually done offline. However, in online applications where the data distribution can dynamically change, e.g., the unusual traffic patterns in online traffic time series prediction problem [63], keeping the hyperparameters static could lead to sub-optimal performance. Therefore, the hyperparameters should be continuously updated together with the model parameters in an online manner.
Specifically, at each online round  $t$ , the agent applies the hyperparameters  $\lambda_{t}$  and the model  $w_{t}$ , and then receives a small dataset  $\mathcal{D}_t = \{\mathcal{D}_t^{\mathrm{tr}}, \mathcal{D}_t^{\mathrm{val}}\}$  composed of a training subset  $\mathcal{D}_t^{\mathrm{tr}}$  and a validation subset  $\mathcal{D}_t^{\mathrm{val}}$ . Based on  $\lambda_{t}$  and  $\mathcal{D}_t^{\mathrm{tr}}$ , the agent first updates her model prediction  $w_{t+1}$  as an estimation of  $w_{t}^{*}(\lambda_{t}) := \arg \min_{w} \mathcal{L}_{t}^{\mathrm{tr}}(\lambda_{t}, w)$ , where  $\mathcal{L}_{t}^{\mathrm{tr}}(\lambda, w) := \frac{1}{|\mathcal{D}_{t}^{\mathrm{tr}}|} \sum_{\zeta \in \mathcal{D}_{t}^{\mathrm{tr}}} \mathcal{L}(w; \zeta) + \Omega(\lambda, w)$ ,  $\mathcal{L}(w, \zeta)$  is a cost function computed on data point  $\zeta$  with prediction model  $w$ , and  $\Omega(w, \lambda)$  is a regularizer. Based on the model prediction  $w_{t+1}$  and  $\mathcal{D}_t^{\mathrm{tr}}$ , the agent updates the hyperparameters  $\lambda_{t+1}$  to minimize the validation loss  $\mathcal{L}_t^{\mathrm{val}}(\lambda, w_t^*(\lambda)) := \frac{1}{|\mathcal{D}_t^{\mathrm{val}}|} \sum_{\xi \in \mathcal{D}_t^{\mathrm{val}}} \mathcal{L}(w_t^*(\lambda); \xi)$ .
We consider an online classification setting on the 20 Newsgroup dataset, where the classifier is modeled by an affine transformation and we use the cross-entropy loss as the losscost function. For  $\Omega (\lambda ,w)$ , we use one  $\ell_2$ -regularization parameter for each row of the transformation matrix in  $w$ , so that we have one regularization parameter for each data feature (i.e.,  $|\lambda|$  is given by the dimension of the data). We remove all news headers in the 20 Newsgroup dataset and pre-process the dataset so as to have data feature vectors of dimension  $d = 99238$ . In our implementations, we approximate the hypergradient using implicit differentiation with the fixed point method [17]. We consider two different setups: (i) a static setup where the agent receives a stream of clean data batches  $\{\mathcal{D}_t\}_t$ ; (ii) a dynamic setting in which the agent receives a stream of corrupted batches  $\{\mathcal{D}_t\}_t$ , where the corruption level changes after some time steps. For both setups the batchsize is fixed to 16. For the dynamic setting we consider four different corruption levels  $\{5\%, 10\%, 20\%, 30\% \}$  and also optimize the learning rate as an additional hyperparameter.
We evaluate the testing accuracy for SOBOW and OAGD in Table 2 for both static (Left) and dynamic (Right) setups. It can be seen that compared to OAGD, SOBOW achieves similar accuracy but with a much shorter running time. When the window size increases in Table 2, performance of both SOBOW and OAGD increases and the computational advantage of SOBOW becomes more significant. In particular, SOBOW runs around 20 times faster than OAGD when the window size is 50.
7 Conclusions and Discussion
In this work, we study non-convex bilevel optimization where the functions can be time-varying and the agent continuously updates the decisions with online streaming data. We proposed a single-loop
online bilevel optimizer with window averaging (SOBOW) to handle the function variations and the unavailability of the true hypergradients in OBO. Compared to existing algorithms, SOBOW is computationally efficient and does not require previous function information. We next developed a novel analytical technique to tackle the unique challenges in OBO and showed that SOBOW can achieve a sublinear bilevel local regret. Extensive experiments justified the effectiveness of SOBOW. We also discuss the potential applications of the OBO framework in online meta-learning and online adversarial training (see Appendix).
Limitation and future directions The study of online bilevel optimization is still in a very early stage, and much of this new framework still remains under-explored and not well understood. We started with the second-order approach for hypergradient estimation, which is less scalable. One future direction is to leverage the recently developed first order approaches for hypergradient estimation. Another limitation is that we assume that the inner-level objective function is strongly convex. In the future, we will investigate the convex and even non-convex case.
This work has been supported in part by NSF grants NSF AI Institute (AI-EDGE) CNS-2112471, CNS-2106933, CNS-2106932, CNS-2312836, CNS-1955535, CNS-1901057, ECCS-2113860, DMS-2134145, and 2311274, by Army Research Office under Grant W911NF-21-1-0244, and was sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-23-2-0225. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
[1] Naman Agarwal, Alon Gonen, and Elad Hazan. Learning in non-convex games with an optimization oracle. In Conference on Learning Theory, pages 18–29. PMLR, 2019.
[2] Sergul Aydore, Tianhao Zhu, and Dean P Foster. Dynamic local regret for non-convex online forecasting. Advances in neural information processing systems, 32, 2019.
[3] Fan Bao, Guoqiang Wu, Chongxuan Li, Jun Zhu, and Bo Zhang. Stability and generalization of bilevel programming in hyperparameter optimization. Advances in Neural Information Processing Systems, 34:4529-4541, 2021.
[4] Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2018.
[5] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research, 63(5):1227-1244, 2015.
[6] Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations Research, 21(1):37-44, 1973.
[7] Tianyi Chen, Yuejiao Sun, and Wotao Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671, 2021.
[8] Jiequan Cui, Pengguang Chen, Ruiyu Li, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast and practical neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6509-6518, 2019.
[9] Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems, 35:26698-26710, 2022.
[10] Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pages 318-326. PMLR, 2012.
[11] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.