SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Achieving  $\mathcal{O}(\epsilon^{-1.5})$  Complexity in Hessian/Jacobian-free Stochastic Bilevel Optimization
Yifan Yang, Peiyao Xiao and Kaiyi Ji
Buffalo, NY 14260
{yyang99,peiyaoxi,kaiyiji}@buffalo.edu
In this paper, we revisit the bilevel optimization problem, in which the upper-level objective function is generally nonconvex and the lower-level objective function is strongly convex. Although this type of problem has been studied extensively, it still remains an open question how to achieve an  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity in Hessian/Jacobian-free stochastic bilevel optimization without any second-order derivative computation. To fill this gap, we propose a novel Hessian/Jacobian-free bilevel optimizer named FdeHBO, which features a simple fully single-loop structure, a projection-aided finite-difference Hessian/Jacobian-vector approximation, and momentum-based updates. Theoretically, we show that FdeHBO requires  $\mathcal{O}(\epsilon^{-1.5})$  iterations (each using  $\mathcal{O}(1)$  samples and only first-order gradient information) to find an  $\epsilon$ -accurate stationary point. As far as we know, this is the first Hessian/Jacobian-free method with an  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity for nonconvex-strongly-convex stochastic bilevel optimization.
Bilevel optimization has drawn intensive attention due to its wide applications in meta-learning [18, 4, 50], hyperparameter optimization [18, 52, 14], reinforcement learning [35, 27], signal process [36, 16] and communication [31] and federated learning [59]. In this paper, we study the following stochastic bilevel optimization problem.
$$
\min  _ {x \in \mathbb {R} ^ {p}} \Phi (x) = f (x, y ^ {*} (x)) := \mathbb {E} _ {\xi} [ f (x, y ^ {*} (x); \xi) ]
$$
$$
\text {s . t .} y ^ {*} (x) = \underset {y \in \mathbb {R} ^ {q}} {\arg \min } g (x, y) := \mathbb {E} _ {\zeta} [ g (x, y ^ {*} (x); \zeta) ] \tag {1}
$$
where the upper- and lower-level objective functions  $f(x,y)$  and  $g(x,y)$  take the expectation form w.r.t. the random variables  $\xi$  and  $\zeta$ , and are jointly continuously differentiable. In this paper, we focus on the nonconvex-strongly-convex bilevel setting, where the lower-level function  $g(x,\cdot)$  is strongly convex and the upper-level function  $\Phi (x)$  is nonconvex. This class of bilevel problems has been studied extensively from the theoretical perspective in recent years. Among them, [19, 30, 3, 62] proposed bilevel approaches with a double-loop structure, which update  $x$  and  $y$  in a nested manner. Single-loop bilevel algorithms have also attracted significant attention recently [27, 62, 34, 25, 9, 40, 11] due to the simple updates on all variables simultaneously. Among them, the approaches in [62, 34, 25] have been shown to achieve an  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity, but with expensive evaluations of Hessian/Jacobian matrices or Hessian/Jacobian-vector products.
Hessian/Jacobian-free bilevel optimization has received increasing attention due to its high efficiency and feasibility in practical large-scale settings. In particular, [15, 48, 61] directly ignored the
Algorithm	Samples	Batch size	# of iterations	Loops per iteration
PZOBO-S [58]	O(p2ε-3)	O(ε-1)	O(p2ε-2)	2
F2SA [37]	O(ε-3.5)	O(1)	O(ε-3.5)	1
F3SA [37]	O(ε-2.5)	O(1)	O(ε-2.5)	1
FdeHBO (this paper)	O(ε-1.5)	O(1)	O(ε-1.5)	1
Table 1: Comparison of stochastic Hessian/Jacobian-free bilevel optimization algorithms.
computation of all second-order derivatives. However, such eliminations may lead to performance degeneration [2, 13], and can vanish the hypergradient for bilevel problems with single-variable upper-level function, i.e.,  $\Phi (x) = f(y^{*}(x))$ . [56, 23] proposed zeroth-order approaches that approximate the hypergradient using only function values. These methods do not have a convergence rate guarantee. Recently, several Hessian/Jacobian-free bilevel algorithms were proposed by [42, 57, 53, 8] by reformulating the lower-level problem into the optimality-based constraints such as  $g(x,y)\leq \min_yg(x,y)$ . However, these approaches all focus on the deterministic setting, and their extensions to the stochastic setting remain unclear. In the stochastic case, [58] proposed evolution strategies based bilevel method, which achieves a high sample complexity of  $\mathcal{O}(p^2\epsilon^{-2})$ , where  $p$  is the problem dimension. Most recently, [37] proposed two fully first-order (i.e., Hessian/Jacobian-free) value-function-based stochastic bilevel optimizer named F $^2$ SA and its momentum-based version F $^3$ SA with a single-loop structure, which achieves sample complexities of  $\mathcal{O}(\epsilon^{-3.5})$  and  $\mathcal{O}(\epsilon^{-2.5})$ , respectively. However, there is still a large gap of  $\epsilon^{-1}$ , compared to the optimal complexity of  $\mathcal{O}(\epsilon^{-1.5})$ . Then, an important open question, as recently proposed by [37], is:
- Can we achieve an  $\mathcal{O}(\epsilon^{-1.5})$  sample/gradient complexity for nonconvex-strongly-convex bilevel optimization using only first-order gradient information?
1.1 Our Contributions
In this paper, we provide an affirmative answer to the above question by proposing a new Hessian/Jacobian-free stochastic bilevel optimizer named FdeHBO with three main features. First, FdeHBO takes the fully single-loop structure with momentum-based updates on three variables  $y$ ,  $v$  and  $x$  for optimizing the lower-level objective, the linear system (LS) of the Hessian-inverse-vector approximation, and the upper-level objective, respectively. Second, FdeHBO contains only a single matrix-vector product at each iteration, which admits a simple first-order finite-difference estimation. Third, FdeHBO involves an auxiliary projection on  $v$  updates to ensure the boundedness of the Hessian-vector approximation error, the variance on momentum-based iterates, and the smoothness of the LS loss function. Our detailed contributions are summarized below.
- Theoretically, we show that FdeHBO achieves a sample/gradient complexity of  $\mathcal{O}(\epsilon^{-1.5})$  and an iteration complexity of  $\mathcal{O}(\epsilon^{-1.5})$  to achieve an  $\epsilon$ -accurate stationary point, both of which outperforms existing results by a large margin. As far as we know, this is the first-known method with an  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity for nonconvex-strongly-convex stochastic bilevel optimization using only first-order gradient information.
- Technically, we show that the auxiliary projection can provide more accurate iterates on  $v$  in solving the LS problem without affecting the overall convergence behavior, and in addition, provide a novel characterization of the gradient estimation error and the iterative progress during the  $v$  updates, as well as the impact of the  $y$  and  $v$  updates on the momentum-based hypergradient estimation, all of which do not exist in previous studies. In addition, the finite-different approximations make the unbiased assumptions in the momentum-based gradients no longer hold, and hence a more careful analysis is required.
- As a byproduct, we further propose a fully single-loop momentum-based method named FMBO in the small-dimensional case with matrix-vector-based hypergradient computations. Differently from existing momentum-based bilevel methods with  $\mathcal{O}(\log \frac{1}{\epsilon})$  Hessian-vector evaluations per iteration, FMBO contains only a single Hessian-vector computation per iteration with the same  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity.
We also want to emphasize our technical differences from previous works as below.
Comparison to existing momentum-based methods. Previous momentum-based methods [62, 34] solve the linear system (LS) to a high accuracy of  $\mathcal{O}(\epsilon)$ , whereas our algorithm includes a new estimation error by the single-step momentum update on LS, and this error is also correlated with the lower-level updating error and the hypergradient estimation error. In addition, due to the finite-difference approximation, the stochastic gradients in all three updates on  $y$ ,  $v$ ,  $x$  are no longer unbiased. Non-trivial efforts need to be taken to deal with such challenges and derive the optimal complexity.
Comparison to existing fully single-loop methods. The analysis of the single-step momentum update in solving the LS requires the smoothness of the LS loss function and the boundedness of LS gradient variance, both of which may not be satisfied. To this end, we include an auxiliary projection and show it not only guarantees these crucial properties, but also, in theory, provides an improved per-iteration progress. As a comparison, existing works on fully single-loop stochastic bilevel optimization such as SOBA/SABA [11] and FLSA [40] with a new time scale to update the LS problem often assume that the iterates on  $v$  are bounded during the process. We do not require such assumptions. In addition, an  $\mathcal{O}(\epsilon^{-1.5})$  complexity has not been established for fully single-loop bilevel algorithms yet.
1.2 Related Work
Bilevel optimization methods. Bilevel optimization, which was first introduced by [6], has been studied for decades. By replacing the lower-level problem with its optimality conditions, [26, 20, 54, 55] reformulated the bilevel problem to the single-level problem. Gradient-based bilevel methods have shown great promise recently, which can be divided into approximate implicit differentiation (AID) [12, 49, 41, 3] and iterative differentiation (ITD) [47, 17, 15, 52, 21] based approaches. Recently, a bunch of stochastic bilevel algorithms has been proposed via Neumann series [9, 30], recursive momentum [62, 28, 25] and variance reduction [62, 11]. Theoretically, the convergence of bilevel optimization has been analyzed by [18, 52, 45, 19, 30, 27, 3, 11]. Among them, [29] provides the lower complexity bounds for deterministic bilevel optimization with (strongly-)convex upper-level functions. [25, 9, 62, 34] achieved the near-optimal sample complexity with second-order derivative computations. Some works studied deterministic bilevel optimization with convex or Polyak-Lojasiewicz (PL) lower-level problems via mixed gradient aggregation [51, 46, 39], log-barrier regularization [45], primal-dual method [57] and dynamic barrier [63]. More results and details can be found in the survey by [44].
Hessian/Jacobian-free bilevel optimization. Some Hessian/Jacobian-free bilevel optimization methods have been proposed recently by [58, 43, 15, 23, 56, 48]. Among them, FOMAML [15, 48] and MUMOMAML [61] directly ignore the computation of all second-order derivatives. Several Hessian/Jacobian-free bilevel algorithms were proposed by [42, 57, 53, 8] by replacing the lower-level problem with the optimality conditions as the constraints. However, these approaches focus only on the deterministic setting. Recently, zeroth-order stochastic approaches have been proposed for the hypergradient estimation [56, 23, 58]. Theoretically, [58] analyzed the convergence rate for their method. [37] proposed fully first-order stochastic bilevel optimization algorithms based on the value-function-based lower-level problem reformulation. This paper proposes a new Hessian/Jacobian-free stochastic bilevel algorithm that for the first time achieves an  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity.
Momentum-based bilevel approaches. The recursive momentum technique was first introduced by [10, 60] for minimization problems to improve the SGD-based updates in theory and in practice. This technique has been incorporated in stochastic bilevel optimization [34, 9, 24, 25, 62]. These approaches involve either Hessian-inverse matrix computations or a subloop of a number of iterations in the Hessian-inverse-vector approximation. As a comparison, our proposed method takes the simpler fully single-loop structure, and only uses the first-order gradient information.
Finite-difference matrix-vector approximation. The finite-difference matrix-vector estimation has been studied extensively in the problems of escaping from saddle points [1] [7] (some other related works can be found therein), neural architecture search (NAS) [43] and meta-learning [13]. However, such finite-different estimation can be sensitive to the selection of the smoothing constant, and may suffer from some numerical issues in practice [32][33], such as rounding errors. It is interesting but still open to developing a fully first-order stochastic bilevel optimizer without the finite-different matrix-vector estimation. We would like to leave it for future study.
Algorithm 1 Hessian/Jacobian-free Bilevel Optimizer via Projection-aided Finite-difference Estimation
1: Input:  $\{\alpha_t, \beta_t, \lambda_t\}_{t=0}^{T-1}$  and  $r_v$ .
2: Initialize:
3: for  $t = 0,1,2,\dots,T - 1$  do
4: Compute the gradient estimator  $h_t^g$  by eq. (6) and update  $y_{t + 1} = y_t - \beta_t h_t^g$ .
5: Compute the gradient estimator  $h_t^R$  by eq. (7) and update  $w_{t + 1} = v_t - \lambda_t\widetilde{h}_t^R$ .
6: Set  $v_{t+1} = \begin{cases} w_{t+1}, & \|w_{t+1}\| \leq r_v; \\ \frac{r_v w_{t+1}}{\|w_{t+1}\|}, & \|w_{t+1}\| > r_v. \end{cases}$
7: Compute the gradient estimator  $h_t^f$  by eq. (10) and update  $x_{t + 1} = x_t - \alpha_t\widetilde{h}_t^f$ .
8: end for
2 Algorithms
In this section, we first describe the hypergradient computation in bilevel optimization, and then present the proposed Hessian/Jacobian-free bilevel method.
2.1 Hypergradient Computation
One major challenge in bilevel optimization lies in computing the hypergradient  $\nabla \Phi(x)$  due to the implicit and complex dependence of the lower-level minimizer  $y^{*}$  on  $x$ . To see this, if  $g$  is twice differentiable,  $\nabla_{y} g$  is continuously differentiable and the Hessian  $\nabla_{yy}^{2} g(x, y^{*}(x))$  is invertible, using the implicit function theorem (IFT) [22, 5], the hypergradient  $\nabla \Phi(x)$  takes the form of
$$
\nabla \Phi (x) = \nabla_ {x} f (x, y ^ {*}) - \nabla_ {x y} ^ {2} g (x, y ^ {*}) \left[ \nabla_ {y y} ^ {2} g (x, y ^ {*}) \right] ^ {- 1} \nabla_ {y} f (x, y ^ {*}). \tag {2}
$$
Note that the hypergradient in eq. (2) requires computing the exact solution  $y^{*}$  and the expensive Hessian inverse  $[\nabla_{yy}^2 g(x,y^*)]^{-1}$ . To approximate this hypergradient efficiently, we define the following (stochastic) hypergradient surrogates as
$$
\bar {\nabla} f (x, y, v) = \nabla_ {x} f (x, y) - \nabla_ {x y} ^ {2} g (x, y) v,
$$
$$
\bar {\nabla} f (x, y, v; \xi) = \nabla_ {x} f (x, y; \xi) - \nabla_ {x y} ^ {2} g (x, y; \xi) v, \tag {3}
$$
where  $v \in \mathbb{R}^q$  is an auxiliary vector to approximate the Hessian-inverse-vector product in eq. (2), and  $\bar{\nabla} f(x,y,v;\xi)$  can be regarded as a stochastic version of  $\bar{\nabla} f(x,y,v)$ . Based on eq. (3), one needs to find an efficient estimate  $y$  of  $y^{*}$ , e.g., via an iterative optimization procedure, as well as a feasible estimate  $v$  of the solution  $v^{*} = [\nabla_{yy}^{2}g(x,y)]^{-1}\nabla_{y}f(x,y)$  of a linear system (LS) (equivalently quadratic programming) whose generic loss function is given by
$$
\text {(L i n e a r s y s t e m l o s s :)} \quad R (x, y, v) = \frac {1}{2} v ^ {T} \nabla_ {y y} ^ {2} g (x, y) v - v ^ {T} \nabla_ {y} f (x, y), \tag {4}
$$
where the gradient of  $R(x,y,v)$  w.r.t.  $v$  is given by
$$
\nabla_ {v} R (x, y, v) = \nabla_ {y y} ^ {2} g (x, y) v - \nabla_ {y} f (x, y). \tag {5}
$$
Similarly to eq. (3), we also define  $\nabla_v R(x,y,v;\psi) = \nabla_{yy}^2 g(x,y;\psi)v - \nabla_y f(x,y;\psi)$  over any sample  $\psi$  as a stochastic version of  $\nabla_v R(x,y,v)$  in eq. (5). It can be seen from eq. (3), eq. (4) and eq. (5) that the updates on the LS system involve the Hessian- and Jacobian-vector products, which can be computationally intractable in the high-dimensional case. In the next section, we propose a novel stochastic Hessian/Jacobian-free bilevel algorithm.
2.2 Hessian/Jacobian-free Bilevel Optimizer via Projection-aided Finite-difference Estimation
As shown in Algorithm 1, we propose a fully single-loop stochastic Hessian/Jacobian-free bilevel optimizer named FdeHBO via projection-aided finite-difference estimation. It can be seen that FdeHBO first minimizes the lower-level objective function  $g(x,y)$  w.r.t.  $y$  by running a single-step momentum-based update as  $y_{t + 1} = y_t - \beta_th_t^g$ , where  $\beta_{t}$  is the stepsize and  $h_t^g$  is the momentum-based gradient estimator that takes the form of
$$
h _ {t} ^ {g} = \eta_ {t} ^ {g} \nabla_ {y} g \left(x _ {t}, y _ {t}; \zeta_ {t}\right) + \left(1 - \eta_ {t} ^ {g}\right) \left(h _ {t - 1} ^ {g} + \nabla_ {y} g \left(x _ {t}, y _ {t}; \zeta_ {t}\right) - \nabla_ {y} g \left(x _ {t - 1}, y _ {t - 1}; \zeta_ {t}\right)\right) \tag {6}
$$
where  $\eta_t^g \in [0,1]$  is a tuning parameter. The next key step is to deal with the LS problem via solving the quadratic problem eq. (4) as  $w_{t + 1} = v_t - \lambda_t\widetilde{h}_t^R$ , with the momentum-based gradient  $\widetilde{h}_t^R$  given by
$$
\begin{array}{l} \widetilde {h} _ {t} ^ {R} = \eta_ {t} ^ {R} \widetilde {\nabla} _ {v} R (x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \psi_ {t}) + (1 - \eta_ {t} ^ {R}) \big (h _ {t - 1} ^ {R} + \widetilde {\nabla} _ {v} R (x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \psi_ {t}) \\ - \widetilde {\nabla} _ {v} R \left(x _ {t - 1}, y _ {t - 1}, v _ {t - 1}, \delta_ {\epsilon}; \psi_ {t}\right)), \tag {7} \\ \end{array}
$$
where  $\tilde{\nabla}_vR$  is a Hessian-free version of the LS gradient  $\nabla_vR$  in eq. (5), given by
$$
(\text {F i r s t - o r d e r L S g r a d i e n t :}) \tilde {\nabla} _ {v} R \left(x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \psi_ {t}\right) = \widetilde {H} \left(x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \psi_ {t}\right) - \nabla_ {y} f \left(x _ {t}, y _ {t}; \psi_ {t}\right). \tag {8}
$$
Note that in the above eq. (8),  $\widetilde{H}(x_t, y_t, v_t, \delta_\epsilon; \psi_t)$  is the finite-difference estimation of the Hessian-vector product  $\nabla_{yy}^2 g(x_t, y_t; \psi_t)v_t$ , which takes the form of
$$
\tilde {H} \left(x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \psi_ {t}\right) = \frac {\nabla_ {y} g \left(x _ {t} , y _ {t} + \delta_ {\epsilon} v _ {t} ; \psi_ {t}\right) - \nabla_ {y} g \left(x _ {t} , y _ {t} - \delta_ {\epsilon} v _ {t} ; \psi_ {t}\right)}{2 \delta_ {\epsilon}}, \tag {9}
$$
where  $\delta_{\epsilon} > 0$  is a small constant. Note that in eq. (9), if the iterative  $v_{t}$  is unbounded, the approximation error between  $\widetilde{H}$  and  $\nabla_{yy}^2 g(x_t,y_t;\psi_t)v_t$  can be uncontrollable as well. We further prove lemma 5 in appendix B that the bound of this gap relies on  $\| v_{t}\|$  and  $\delta$  but it is independent of the dimension of  $y_{t}$ . To this end, after obtaining  $w_{t + 1}$ , our key step in line 6 introduces an auxiliary projection on a ball (which can be generalized to any convex and bounded domain) with a radius of  $r_v$  as
$$
\text {(A u x i l i a r y p r o j e c t i o n)} \quad v _ {t + 1} = \left\{ \begin{array}{l l} w _ {t + 1}, & \| w _ {t + 1} \| \leq r _ {v}; \\ \frac {r _ {v} w _ {t + 1}}{\| w _ {t + 1} \|}, & \| w _ {t + 1} \| > r _ {v}. \end{array} \right.
$$
This auxiliary projection guarantees the boundedness of  $v_{t}, t = 0, \dots, T - 1$ , which serves three important purposes. First, it ensures the smoothness of the LS loss function  $R(x,y,v)$  in eq. (5) w.r.t. all  $x,y$  and  $v$ , which is crucial in the convergence analysis of the momentum-based updates. Second, the boundedness of  $v_{t}$  also ensures that the estimation variance of the stochastic LS gradient  $\nabla_{v}R(x_{t},y_{t},v_{t};\psi_{t})$  does not explode. Third, it guarantees the error of the finite-difference Hessian-vector approximation to be sufficiently small with proper  $\delta_{\epsilon}$ . We will show later that under a proper choice of the radius  $r_v$ , this auxiliary projection provides better per-step progress, and the proposed algorithm achieves a stronger convergence performance. Finally, for the upper-level problem, the momentum-based hypergradient estimate  $\widetilde{h}_t^f$  is designed as
$$
\begin{array}{l} \widetilde {h} _ {t} ^ {f} = \eta_ {t} ^ {f} \widetilde {\nabla} f (x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \bar {\xi} _ {t}) + (1 - \eta_ {t} ^ {f}) \left(h _ {t - 1} ^ {f} + \widetilde {\nabla} f (x _ {t}, y _ {t}, v _ {t}, \delta_ {\epsilon}; \bar {\xi} _ {t}) \right. \\ \left. - \widetilde {\nabla} f \left(x _ {t - 1}, y _ {t - 1}, v _ {t - 1}, \delta_ {\epsilon}; \bar {\xi} _ {t}\right)\right), \tag {10} \\ \end{array}
$$
where  $\widetilde{\nabla} f(x,y,v,\delta_{\epsilon};\bar{\xi}_{t})$  is the fully first-order hypergradient estimate evaluated at two consecutive iterates  $(x_{t},y_{t},v_{t})$  and  $(x_{t - 1},y_{t - 1},v_{t - 1})$  is given by
$$
\widetilde {\nabla} f (x, y, v, \delta_ {\epsilon}; \bar {\xi} _ {t}) = \nabla_ {x} f (x, y; \bar {\xi} _ {t}) - \widetilde {J} (x, y, v, \delta_ {\epsilon}; \bar {\xi} _ {t}),
$$
and  $\widetilde{J}(x,y,v,\delta_{\epsilon};\bar{\xi}_t)$  is the finite-difference Jacobian-vector approximation given by
$$
\widetilde {J} (x, y, v, \delta_ {\epsilon}; \bar {\xi} _ {t}) := \frac {\nabla_ {x} g (x , y + \delta_ {\epsilon} v ; \bar {\xi} _ {t}) - \nabla_ {x} g (x , y - \delta_ {\epsilon} v ; \bar {\xi} _ {t})}{2 \delta_ {\epsilon}}. \tag {11}
$$
Note that  $\widetilde{\nabla}_vR$  and  $\widetilde{\nabla} f$  are biased estimators of the gradients  $\nabla_vR$  and  $\bar{\nabla} f$ , which further complicates the convergence analysis on the momentum-based updates because the conventional analysis on the recursive momentum requires the unbiased gradient estimation to ensure the variance reduction effect. By controlling the perturbation  $\delta_{\epsilon}$  properly, we will show that FdeHBO can achieve an  $\mathcal{O}(\epsilon^{-1.5})$  convergence and complexity performance without any second-order derivative computation.
2.3 Extension to Small-Dimensional Case
As a byproduct of our proposed FdeHBO, we further propose a fully single-loop momentum-based bilevel optimizer (FMBO), which is more suitable in the small-dimensional case without finite-difference approximation. As shown in Algorithm 2, FMBO first takes the same lower-level updates
Algorithm 2 Fully Single-loop Momentum-based Bilevel Optimizer (FMBO)
1: Input:  $\{\alpha_t, \beta_t, \lambda_t\}_{t=0}^{T-1}$ , and  $r_v$ .
2: Initialize:
3: for  $t = 0,1,2,\dots,T - 1$  do
4: Compute the gradient estimator  $h_t^g$  by eq. (6) and update  $y_{t + 1} = y_t - \beta_th_t^g$ .
5: Compute the gradient estimator  $h_t^R$  by eq. (12) and update  $w_{t + 1} = v_t - \lambda_th_t^R$ .
6: Set  $v_{t+1} = \begin{cases} w_{t+1}, & \|w_{t+1}\| \leq r_v; \\ \frac{r_v w_{t+1}}{\|w_{t+1}\|}, & \|w_{t+1}\| > r_v. \end{cases}$
7: Compute the gradient estimator  $h_t^f$  by eq. (13) and update  $x_{t + 1} = x_t - \alpha_th_t^f$ .
8: end for
on  $y_{t}$  as in eq. (6). Then, it solves the LS problem as  $w_{t + 1} = v_t - \lambda_th_t^R$ , where the momentum-based gradient estimator is given by
$$
\begin{array}{l} h _ {t} ^ {R} = \eta_ {t} ^ {R} \nabla_ {v} R (x _ {t}, y _ {t}, v _ {t}; \psi_ {t}) + (1 - \eta_ {t} ^ {R}) \left(h _ {t - 1} ^ {g} + \nabla_ {v} R (x _ {t}, y _ {t}, v _ {t}; \psi_ {t}) \right. \\ \left. - \nabla_ {v} R \left(x _ {t - 1}, y _ {t - 1}, v _ {t - 1}; \psi_ {t}\right)\right), \tag {12} \\ \end{array}
$$
where differently from FdeHBO, we here use the precise gradient  $\nabla_v R$  without finite-difference approximation. Similarly to FdeHBO, we add an auxiliary projection on the  $v_t$  updates to ensure the LS smoothness and bounded variance. Finally, for the upper-level problem, we optimize  $x_t$  based on a momentum-based update as  $x_{t+1} = x_t - \alpha_t h_t^f$  with the hypergradient estimator
$$
h _ {t} ^ {f} = \eta_ {t} ^ {f} \bar {\nabla} f \left(x _ {t}, y _ {t}, v _ {t}; \bar {\xi} _ {t}\right) + \left(1 - \eta_ {t} ^ {f}\right) \left(h _ {t - 1} ^ {f} + \bar {\nabla} f \left(x _ {t}, y _ {t}, v _ {t}; \bar {\xi} _ {t}\right) - \bar {\nabla} f \left(x _ {t - 1}, y _ {t - 1}, v _ {t - 1}; \bar {\xi} _ {t}\right)\right) \tag {13}
$$