SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\eta_t^f \in [0,1]$  is a tuning parameter. Similarly, we directly use the hypergradient estimate in eq. (3) without the finite-difference estimation. We note that compared to existing momentum-based algorithms [62, 34] that contains  $\mathcal{O}(\log \frac{1}{\epsilon})$  steps in solving the LS problem, FMBO takes the fully single-loop structure with a single-step momentum-based acceleration on the LS updates.
3.1 Assumptions and Definitions
We make the following standard assumptions for the upper- and lower-level objective functions, as also adopted by [30, 9, 34]. The following assumption imposes the Lipschitz condition on the upper-level function  $f(x,y)$ .
Assumption 1. For any  $x \in \mathbb{R}^{d_x}$  and  $y \in \mathbb{R}^{d_y}$ , there exist positive constants  $L_{f_x}, L_{f_y}, C_{f_x}$  and  $C_{f_y}$  such that  $\nabla_x f(x, y)$  and  $\nabla_y f(x, y)$  are  $L_{f_x}$ - and  $L_{f_y}$ -Lipschitz continuous w.r.t.  $(x, y)$ , and  $\| \nabla_x f(x, y) \|^2 \leq C_{f_x}$ ,  $\| \nabla_y f(x, y) \|^2 \leq C_{f_y}$ .
The following assumption imposes the Lipschitz condition on the lower-level function  $g(x,y)$ .
Assumption 2. For any  $x \in \mathbb{R}^{d_x}$  and  $y \in \mathbb{R}^{d_y}$ , there exist positive constants  $\mu_g, L_g, L_{g_{xy}}, L_{g_{yy}}$ ,  $C_{g_{xy}}, C_{g_{yy}}$  such that
- Function  $g(x, y)$  is twice continuously differentiable;
- Function  $g(x, \cdot)$  is  $\mu_g$ -strongly-convex;
- The derivatives  $\nabla_y g(x,y)$ ,  $\nabla_{xy}^2 g(x,y)$  and  $\nabla_{yy}^2 g(x,y)$  are  $L_{g^-}$ ,  $L_{g_{xy}}$ - and  $L_{g_{yy}}$ -Lipschitz continuous w.r.t.  $(x,y)$ ;
$\| \nabla_{xy}^2 g(x,y)\| ^2\leq C_{g_{xy}}$  and  $\| \nabla_{yy}^2 g(x,y)\| ^2\leq C_{g_{yy}}$
The following assumption is adopted for the stochastic functions  $f(x,y;\xi)$  and  $g(x,y;\zeta)$ .
Assumption 3. Assumptions 1 and 2 hold for  $f(x,y;\xi)$  and  $g(x,y;\zeta)$  for  $\forall \xi$  and  $\zeta$ . Moreover, we assume that there exist positive constants  $\sigma_{f_x},\sigma_{f_y},\sigma_g,\sigma_{g_{xy}}$  and  $\sigma_{g_{yy}}$  such that
$$
\mathbb {E} \left[ \| \nabla_ {x} f (x, y) - \nabla_ {x} f (x, y; \xi) \| ^ {2} \right] \leq \sigma_ {f _ {x}} ^ {2}, \quad \mathbb {E} \left[ \| \nabla_ {y} f (x, y) - \nabla_ {y} f (x, y; \xi) \| ^ {2} \right] \leq \sigma_ {f _ {y}} ^ {2},
$$
$$
\mathbb {E} \left[ \| \nabla_ {y} g (x, y) - \nabla_ {y} g (x, y; \zeta) \| ^ {2} \right] \leq \sigma_ {g} ^ {2}, \quad \mathbb {E} \left[ \| \nabla_ {x y} ^ {2} g (x, y) - \nabla_ {x y} ^ {2} g (x, y; \xi) \| ^ {2} \right] \leq \sigma_ {g _ {x y}} ^ {2},
$$
$$
\mathbb {E} \left[ \| \nabla_ {y y} ^ {2} g (x, y) - \nabla_ {y y} ^ {2} g (x, y; \xi) \| ^ {2} \right] \leq \sigma_ {g _ {y y}} ^ {2}.
$$
Definition 1. We say  $\bar{x}$  is an  $\epsilon$ -accurate stationary point of a function  $\Phi(x)$  if  $\mathbb{E}\|\nabla\Phi(\bar{x})\|^2 \leq \epsilon$  where  $\bar{x}$  is the output of an optimization algorithm.
3.2 Convergence and Complexity Analysis of FdeHBO
We further provide the convergence analysis for the proposed Hessian/Jacobian-free FdeHBO algorithm. We first characterize several estimation properties of FdeHBO. Let  $e_t^f \coloneqq \widetilde{h}_t^f - \nabla f(x_t, y_t, v_t) - \Delta(x_t, y_t, v_t)$  denote the hypergradient estimation error.
Proposition 1. Under Assumption 3, the iterates of the outer problem by Algorithm 1 satisfy
$$
\begin{array}{l} \mathbb {E} \| e _ {t + 1} ^ {f} \| ^ {2} \leq \left[ (1 - \eta_ {t + 1} ^ {f}) ^ {2} + 4 L _ {g _ {x y}} r _ {v} ^ {2} \delta_ {\epsilon} \right] \mathbb {E} \| e _ {t} ^ {f} \| ^ {2} + 4 (\eta_ {t + 1} ^ {f}) ^ {2} \sigma_ {f} ^ {2} + \left(4 L _ {g _ {x y}} r _ {v} ^ {2} \delta_ {\epsilon} + 1 6 L _ {g _ {x y}} ^ {2} r _ {v} ^ {4} \delta_ {\epsilon} ^ {2}\right) \\ + 6 \left(1 - \eta_ {t + 1} ^ {f}\right) ^ {2} \left[ L _ {F} ^ {2} \alpha_ {t} ^ {2} \mathbb {E} \| \widetilde {h} _ {t} ^ {f} \| ^ {2} + 2 L _ {F} ^ {2} \beta_ {t} ^ {2} \left(\mathbb {E} \| e _ {t} ^ {g} \| ^ {2} + \| \nabla_ {y} g (x _ {t}, y _ {t}) \| ^ {2}\right) \right. \\ \left. + 2 C _ {g _ {x y}} \lambda_ {t} ^ {2} \big (\mathbb {E} \| e _ {t} ^ {R} \| ^ {2} + L _ {g} ^ {2} \mathbb {E} \| v _ {t} - v _ {t} ^ {*} \| ^ {2} \big) \right], \\ \end{array}
$$
for all  $t \in \{0, \dots, T - 1\}$  with  $L_F^2 = 2\left(L_{f_x}^2 + L_{g_{xy}}^2 r_v^2\right)$ .
The hypergradient estimator error  $\mathcal{O}(\mathbb{E}\| e_{t + 1}^{f}\|^{2})$  contains three main components. The first term  $[(1 - \eta_{t + 1}^{f})^{2} + 4L_{g_{xy}}r_{v}^{2}\delta_{\epsilon}]\mathbb{E}\| e_{t}^{f}\|^{2}$  indicates the per-iteration improvement induced by the momentum-based update, the error term  $\alpha_t^2\mathbb{E}\| h_t^f\|^2$  is caused by the  $x_{t}$  updates, the error term  $\mathcal{O}(\beta_t^2\mathbb{E}(\| e_t^g\|^2 +\|\nabla_yg(x_t,y_t)\|^2))$  is caused by solving the lower-level problem, and the new error term  $\mathcal{O}(\lambda_t^2\mathbb{E}(\| e_t^R\|^2 +L_g^2\| v_t - v_t^*\|^2))$  is induced by the one-step momentum update on the LS problem, which does not exist in previous momentum-based bilevel methods [62, 34, 25] that solve the LS problem to a high accuracy. Also note that the errors  $4L_{g_{xy}}r_v^2\delta_\epsilon \mathbb{E}\| e_t^f\|^2$  and  $4L_{g_{xy}}r_v^2\delta_\epsilon +16L_{g_{xy}}^2 r_v^4\delta_\epsilon^2$  are caused by the finite-difference approximation error. Fortunately, by choosing the perturbation level  $\delta_{\epsilon}$  in these two terms to be properly small, it can guarantee the descent factor  $(1 - \eta_{t + 1}^{f})^{2} + 4L_{g_{xy}}r_{v}^{2}\delta_{\epsilon}$  to be at an order of  $(1 - \mathcal{O}(\eta_{t + 1}^{f}))^{2}$ , and hence the momentum-based variance reduction effect is still applied.
Proposition 2. For  $\forall \psi$ , define  $e_t^R \coloneqq \widetilde{h}_t^R - \nabla_v R(x_t, y_t, v_t)$ . Under Assumptions 1, 2, 3, we have
$$
\begin{array}{l} \mathbb {E} \| e _ {t + 1} ^ {R} \| ^ {2} \leq \big [ (1 - \eta_ {t + 1} ^ {R}) ^ {2} (1 + 9 6 L _ {g} ^ {4} \lambda_ {t} ^ {2}) + 4 L _ {g _ {y y}} r _ {v} ^ {2} \delta_ {\epsilon} \big ] \mathbb {E} \| e _ {t} ^ {R} \| ^ {2} + \big (4 L _ {g _ {y y}} r _ {v} ^ {2} \delta_ {\epsilon} + 8 L _ {g _ {y y}} ^ {2} r _ {v} ^ {4} \delta_ {\epsilon} ^ {2} \big) \\ + 8 (\eta_ {t + 1} ^ {R}) ^ {2} (\sigma_ {g _ {y y}} ^ {2} r _ {v} ^ {2} + \sigma_ {f _ {y}} ^ {2}) + 9 6 (1 - \eta_ {t + 1} ^ {R}) ^ {2} L _ {g} ^ {2} \lambda_ {t} ^ {2} \left(\mathbb {E} \| e _ {t} ^ {R} \| ^ {2} + L _ {g} ^ {2} \mathbb {E} \| v _ {t} - v _ {t} ^ {*} \| ^ {2}\right) \\ + 9 6 \left(1 - \eta_ {t + 1} ^ {R}\right) ^ {2} \left(L _ {g _ {y y}} ^ {2} r _ {v} ^ {2} + L _ {f _ {y}} ^ {2}\right) \left[ \alpha_ {t} ^ {2} \mathbb {E} \| \widetilde {h} _ {t} ^ {f} \| ^ {2} + 2 \beta_ {t} ^ {2} (\mathbb {E} \| e _ {t} ^ {g} \| ^ {2} + \mathbb {E} \| \nabla_ {y} g (x _ {t}, y _ {t}) \| ^ {2}) \right] \\ \end{array}
$$
for all  $t\in \{0,1,\dots,T - 1\}$
As shown in Proposition 2, the LS gradient estimation error  $e_{t + 1}^{R}$  contains an iteratively improved error component  $\left[(1 - \eta_{t + 1}^{R})^{2}(1 + 96L_{g}^{4}\lambda_{t}^{2}) + 4L_{g_{yy}}r_{v}^{2}\delta_{\epsilon}\right]\mathbb{E}\| e_{t}^{R}\|^{2}$  for the stepsize  $\lambda_{t}$  and the approximation factor  $\delta_{\epsilon}$  sufficiently small, a finite-difference approximation error  $\mathcal{O}(\delta_{\epsilon})$  as well as an approximation error  $\mathcal{O}(\lambda_t^2\mathbb{E}\| v_t - v_t^*\| ^2)$  for solving the LS problem. The next step is to upper-bound  $\mathbb{E}\| v_t - v_t^*\| ^2$ .
Proposition 3. Under the Assumption 1, 2, the iterates of the LS problem by Algorithm 1 satisfy  $\mathbb{E}\| v_{t + 1} - v_{t + 1}^*\|^2$
$$
\begin{array}{l} \leq \left(1 + \gamma_ {t} ^ {\prime}\right) \left(1 + \delta_ {t} ^ {\prime}\right) \left[ \left(1 - 2 \lambda_ {t} \frac {\left(L _ {g} + L _ {g} ^ {3}\right) \mu_ {g}}{\mu_ {g} + L _ {g}} + \lambda_ {t} ^ {2} L _ {g} ^ {2}\right) \mathbb {E} \| v _ {t} - v _ {t} ^ {*} \| ^ {2} \right] \\ + \left(1 + \gamma_ {t} ^ {\prime}\right) \left(1 + \frac {1}{\delta_ {t} ^ {\prime}}\right) \lambda_ {t} ^ {2} \mathbb {E} \| e _ {t} ^ {R} \| ^ {2} \\ + \left(1 + \frac {1}{\gamma_ {t} ^ {\prime}}\right) \Big (\frac {2 L _ {f _ {y}} ^ {2}}{\mu_ {g} ^ {2}} + \frac {2 C _ {f _ {y}} ^ {2} L _ {g _ {y y}} ^ {2}}{\mu_ {g} ^ {4}} \Big) \left[ \alpha_ {t} ^ {2} \mathbb {E} \| \widetilde {h} _ {t} ^ {f} \| ^ {2} + \beta_ {t} ^ {2} \left(2 \mathbb {E} \| e _ {t} ^ {g} \| ^ {2} + 2 \mathbb {E} \| \nabla_ {y} g (x _ {t}, y _ {t}) \| ^ {2}\right) \right]. \\ \end{array}
$$
for all  $t \in \{0, \dots, T - 1\}$  with some  $\gamma_t' > 0$  and  $\delta_t' > 0$ .
Based on the above important properties, we now provide the general convergence theorem for FdeHBO.
Theorem 1. Suppose Assumptions 1, 2, 3 and Lemma 3 are satisfied. Choose  $r_v \geq \frac{C_{f_y}}{\mu_g}$  and set
$$
\alpha_ {t} = \frac {1}{(w + t) ^ {1 / 3}}, \quad \beta_ {t} = c _ {\beta} \alpha_ {t}, \quad \lambda_ {t} = c _ {\lambda} \alpha_ {t}, \quad \eta_ {t} ^ {f} = c _ {\eta_ {f}} \alpha_ {t} ^ {2}, \quad \eta_ {t} ^ {R} = c _ {\eta_ {R}} \alpha_ {t} ^ {2}, \quad \eta_ {t} ^ {g} = c _ {\eta_ {g}} \alpha_ {t} ^ {2},
$$
and  $\delta_{\epsilon} \leq \frac{\min\{c_{\eta_f}, c_{\eta_R}\}}{8(L_{gxy} r_v^2(w + T - 1)^{2/3})}$ , where the constants  $w, c_{\beta}, c_{\lambda}, c_{\eta_f}, c_{\eta_R}$  and  $c_{\eta_g}$  are defined in eq. (67) in the appendix. Then, the iterates generated by Algorithm 1 satisfy
$$
\begin{array}{l} \mathbb {E} \| \nabla \Phi (x _ {a} (T)) \| ^ {2} \leq \widetilde {\mathcal {O}} \left(\frac {\Phi (x _ {0}) - \Phi^ {*}}{T ^ {2 / 3}} + \frac {\| y _ {0} - y ^ {*} (x _ {0}) \| ^ {2}}{T ^ {2 / 3}} + \frac {\| v _ {0} - v ^ {*} (x _ {0} , y _ {0}) \| ^ {2}}{T ^ {2 / 3}} \right. \\ \left. + \frac {1}{T ^ {2 / 3}} + \frac {\sigma_ {f} ^ {2}}{T ^ {2 / 3}} + \frac {\sigma_ {g} ^ {2}}{T ^ {2 / 3}} + \frac {\sigma_ {R} ^ {2}}{T ^ {2 / 3}}\right). \\ \end{array}
$$
Corollary 1. Under the same setting of Theorem 1, FdeHBO requires  $\widetilde{\mathcal{O}} (\epsilon^{-1.5})$  samples and gradient evaluations, respectively, to achieve an  $\epsilon$ -accurate stationary point.
It can be seen from Corollary 1 that the proposed FdeHBO achieves an  $\widetilde{\mathcal{O}} (\epsilon^{-1.5})$  sample complexity without any second-order derivative computation. As far as we know, this is the first Hessian/Jacobian-free stochastic bilevel optimizer with an  $\widetilde{\mathcal{O}} (\epsilon^{-1.5})$  sample complexity.
3.3 Convergence and Complexity Analysis of FMBO
In this section, we analyze the convergence and complexity of the simplified FMBO method.
Theorem 2. Suppose Assumptions 1, 2 and 3 are satisfied. Choose  $r_v \geq \frac{C_{f_y}}{\mu_g}$  and set parameters
$$
\alpha_ {t} = \frac {1}{(w + t) ^ {1 / 3}}, \quad \beta_ {t} = c _ {\beta} \alpha_ {t}, \quad \lambda_ {t} = c _ {\lambda} \alpha_ {t},
$$
$$
\eta_ {t} ^ {f} = c _ {\eta_ {f}} \alpha_ {t} ^ {2}, \quad \eta_ {t} ^ {R} = c _ {\eta_ {R}} \alpha_ {t} ^ {2}, \quad \eta_ {t} ^ {g} = c _ {\eta_ {g}} \alpha_ {t} ^ {2}
$$
where  $w, c_{\beta}, c_{\lambda}, c_{\eta_f}, c_{\eta_R}$  and  $c_{\eta_g}$  are defined in eq. (33) in the appendix. The iterates generated by Algorithm 2 satisfy
$$
\begin{array}{l} \mathbb {E} \| \nabla \Phi (x _ {a} (T)) \| ^ {2} \leq \widetilde {\mathcal {O}} \left(\frac {\Phi (x _ {0}) - \Phi^ {*}}{T ^ {2 / 3}} + \frac {\| y _ {0} - y ^ {*} (x _ {0}) \| ^ {2}}{T ^ {2 / 3}} \right. \\ + \frac {\| v _ {0} - v ^ {*} (x _ {0} , y _ {0}) \| ^ {2}}{T ^ {2 / 3}} + \frac {\sigma_ {f} ^ {2}}{T ^ {2 / 3}} + \frac {\sigma_ {g} ^ {2}}{T ^ {2 / 3}} + \frac {\sigma_ {R} ^ {2}}{T ^ {2 / 3}} \Biggr). \\ \end{array}
$$
Theorem 2 shows that the proposed fully single-loop FMBO achieves a convergence rate of  $\frac{1}{T^{2/3}}$ , which further yields the following complexity result.
Corollary 2. Under the same setting of Theorem 2, FMBO requires totally  $\widetilde{\mathcal{O}} (\epsilon^{-1.5})$  data samples, gradient and matrix-vector evaluations, respectively, to achieve an  $\epsilon$ -accurate stationary point.
Corollary 2 shows that FMBO requires a total number  $\widetilde{\mathcal{O}} (\epsilon^{-1.5})$  of data samples, which matches the best sample complexity in [34, 62, 28]. More importantly, each iteration of FMBO contains only one Hessian-vector computation due to the simple fully single-loop implementation, whereas other momentum-based approaches require  $\mathcal{O}(\log \frac{1}{\epsilon})$  Hessian-vector computations in a nested manner per iteration. Also, note that FMBO is the first fully single-loop bilevel optimizer that achieves the  $\widetilde{\mathcal{O}} (\epsilon^{-1.5})$  sample complexity.
In this section, we test the performance of the proposed FdeHBO and FMBO on two applications: hyper-representation and data hyper-cleaning, respectively.
Figure 1: Comparison on hyper-representation with the LeNet neural network. Left plot: outer loss v.s. running time; right plot: accuracy v.s. running time.
4.1 Hyper-representation on MNIST Dataset
We now compare the performance of our Hessian/Jacobian-free FdeHBO with the relevant Hessian/Jacobian-free methods PZOBO-S [58],  $\mathrm{F}^2\mathrm{SA}$  [37] and  $\mathrm{F}^3\mathrm{SA}$  [37]. We perform the hyperrepresentation with the 7-layer LeNet network [38], which aims to solve the following bilevel problem.
$$
\min _ {\lambda} L _ {\nu} (\lambda) := \frac {1}{| S _ {\nu} |} \sum_ {(x _ {i}, y _ {i}) \in S _ {\nu}} L _ {C E} (w ^ {*} (\lambda) f (\lambda ; x _ {i}), y _ {i})
$$
$$
s. t. w ^ {*} (\lambda) = \underset {w} {\arg \min } L _ {i n} (\lambda , w), L _ {i n} (\lambda , w) := \frac {1}{| S _ {\tau} |} \sum_ {(\tau , y _ {i}) \in S _ {\tau}} L _ {C E} (w f (\lambda , x _ {i}), y _ {i}),
$$
where  $L_{CE}$  denotes the cross-entropy loss,  $S_{\nu}$  and  $S_{\tau}$  denote the training data and validation data, and  $f(\lambda ;x_i)$  denotes the features extracted from the data  $x_{i}$ . More details of the experimental setups are specified in Appendix A.1.
As shown in Figure 1, our FdeHBO converges much faster and more stably than PZOBO-S,  $\mathrm{F}^2\mathrm{SA}$  and  $\mathrm{F}^3\mathrm{SA}$ , while achieving a higher training accuracy. This is consistent with our theoretical results, and validates the momentum-based approaches in reducing the variance during the entire training.
4.2 Hyper-cleaning on MNIST Dataset
We compare the performance of our FMBO to various bilevel algorithms including AID-FP [21], reverse[17], SUSTAIN [34], MRBO and VRBO [62], BSA [19], stocBiO [30], FSLA [40] and SOBA [11], on a low-dimensional data hyper-cleaning problem with a linear classifier on MNIST dataset, which takes the following formulation.
$$
\min  _ {\lambda} L _ {\nu} (\lambda , w ^ {*}) = \frac {1}{| S _ {\nu} |} \sum_ {(x _ {i}, y _ {i}) \in S _ {\nu}} L _ {C E} ((w ^ {*}) ^ {T} x _ {i}, y _ {i})
$$
$$
s. t. \quad w ^ {*} = \arg \min  _ {w} L (\lambda , w) := \frac {1}{\left| S _ {\tau} \right|} \sum_ {\left(x _ {i}, y _ {i}\right) \in S _ {\tau}} \sigma \left(\lambda_ {i}\right) L _ {C E} \left(w ^ {T} x _ {i}, y _ {i}\right) + C \| w \| ^ {2}, \tag {14}
$$
where  $L_{CE}$  denotes the cross-entropy loss,  $S_{\nu}$  and  $S_{\tau}$  denote the training data and validation data, whose sizes are set to 20000 and 5000, respectively,  $\lambda = \{\lambda_i\}_{i\in S_\tau}$  and  $C$  are the regularization parameters, and  $\sigma (\cdot)$  is the sigmoid function. AmIGO [3] is not included in the figures because it performs similarly to stocBiO. The experimental details can be found in Appendix A.2.
As shown in Figure 2(a), FMBO, stocBiO and AID-FP converge much faster and more stable than other algorithms. Compared to stocBiO and AID-FP, FMBO achieves a lower training loss. This demonstrates the effectiveness of momentum-based variance reduction in finding more accurate iterates. It can be seen from Figure 2(b) that FMBO converges faster than existing fully single-loop FSLA and SOBA algorithms with a lower training loss.
In this paper, we propose a novel Hessian/Jacobian-free bilevel optimizer named FdeHBO. We show that FdeHBO achieves an  $\mathcal{O}(\epsilon^{-1.5})$  sample complexity, which outperforms existing algorithms of the
(a) noise  $\mathrm{p} = 0.1$
Figure 2: (a) Comparison of different algorithms on data hyper-cleaning with noise  $p = 0.1$ . Left plot: test loss v.s. running time; right plot: train loss v.s. running time. (b) Comparison among different single-loop algorithms: training loss v.s. running time.
(b) Single-loop methods
same type by a large margin. Our experiments validate the theoretical results and the effectiveness of the proposed algorithms. We anticipate that the developed analysis will shed light on developing provable Hessian/Jacobian-free bilevel optimization algorithms and the proposed algorithms may be applied to other applications such as fair machine learning.
The work is supported in part by NSF under grants 2326592 and 2311274.
[1] Z. Allen-Zhu and Y. Li. Neon2: Finding local minima via first-order oracles. Advances in Neural Information Processing Systems, 31, 2018.
[2] A. Antoniou, H. Edwards, and A. Storkey. How to train your MAML. In International Conference on Learning Representations (ICLR), 2019.
[3] M. Arbel and J. Mairal. Amortized implicit differentiation for stochastic bilevel optimization. In International Conference on Learning Representations (ICLR), 2022.
[4] L. Bertinetto, J. F. Henriques, P. Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2018.
[5] M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-López, F. Pedregosa, and J.-P. Vert. Efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15183, 2021.
[6] J. Bracken and J. T. McGill. Mathematical programs with optimization problems in the constraints. Operations Research, 21(1):37-44, 1973.
[7] Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Accelerated methods for nonconvex optimization. SIAM Journal on Optimization, 28(2):1751-1772, 2018.
[8] L. Chen, J. Xu, and J. Zhang. On bilevel optimization without lower-level strong convexity. arXiv preprint arXiv:2301.00712, 2023.
[9] T. Chen, Y. Sun, and W. Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671, 2021.
[10] A. Cutkosky and F. Orabona. Momentum-based variance reduction in non-convex SGD. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[11] M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. arXiv preprint arXiv:2201.13409, 2022.
[51] S. Sabach and S. Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization, 27(2):640-660, 2017.
[52] A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots. Truncated back-propagation for bilevel optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1723-1732, 2019.
[53] H. Shen and T. Chen. On penalty-based bilevel gradient descent method. arXiv preprint arXiv:2302.05185, 2023.
[54] C. Shi, J. Lu, and G. Zhang. An extended kuhn-tucker approach for linear bilevel programming. Applied Mathematics and Computation, 162(1):51-63, 2005.
[55] A. Sinha, P. Malo, and K. Deb. A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22(2):276-295, 2017.
[56] X. Song, W. Gao, Y. Yang, K. Choromanski, A. Pacchiano, and Y. Tang. ES-MAML: Simple Hessian-free meta learning. In International Conference on Learning Representations (ICLR), 2019.
[57] D. Sow, K. Ji, Z. Guan, and Y. Liang. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123, 2022.
[58] D. Sow, K. Ji, and Y. Liang. On the convergence theory for Hessian-free bilevel algorithms. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[59] D. A. Tarzanagh, M. Li, C. Thrampoulidis, and S. Oymak. Fednest: Federated bilevel, minimax, and compositional optimization. arXiv preprint arXiv:2205.02215, 2022.
[60] Q. Tran-Dinh, N. H. Pham, D. T. Phan, and L. M. Nguyen. Hybrid stochastic gradient descent algorithms for stochastic nonconvex optimization. arXiv preprint arXiv:1905.05920, 2019.
[61] R. Vuorio, S.-H. Sun, H. Hu, and J. J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[62] J. Yang, K. Ji, and Y. Liang. Provably faster algorithms for bilevel optimization. Advances in Neural Information Processing Systems (NeurIPS), 34:13670-13682, 2021.
[63] M. Ye, B. Liu, S. Wright, P. Stone, and Q. Liu. Bome! bilevel optimization made easy: A simple first-order approach. In Conference on Neural Information Processing Systems (NeurIPS), page 1, 2022.