SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenqian Shen $^{1*}$  Hansi Yang $^{2*}$  Yong Li $^{1}$  James Kwok $^{2}$  Quanming Yao $^{1\dagger}$
$^{1}$ Department of Electronic Engineering, Tsinghua University, Beijing, China
$^{2}$ Department of Computer Science and Engineering,
Hong Kong University of Science and Technology, Hong Kong SAR, China
{szg22@mails, liyong07@mail, qyaoaa@mail}.tsinghua.edu.cn
{hyangbw, jamesk}@cse.ust.hk
As hyper-parameters are ubiquitous and can significantly affect the model performance, hyper-parameter optimization is extremely important in machine learning. In this paper, we consider a sub-class of hyper-parameter optimization problems, where the hyper-gradients are not available. Such problems frequently appear when the performance metric is non-differentiable or the hyper-parameter is not continuous. However, existing algorithms, like Bayesian optimization and reinforcement learning, often get trapped in local optimals with poor performance. To address the above limitations, we propose to use cubic regularization to accelerate convergence and avoid saddle points. First, we adopt stochastic relaxation, which allows obtaining gradient and Hessian information without hyper-gradients. Then, we exploit the rich curvature information by cubic regularization. Theoretically, we prove that the proposed method can converge to approximate second-order stationary points, and the convergence is also guaranteed when the lower-level problem is in exactly solved. Experiments on synthetic and real-world data demonstrate the effectiveness of our proposed method.
In a machine learning system, hyper-parameters are parameters that control the learning process. Well-known examples include learning rate, and choice of loss functions or regularizers. Hyper-parameters play a crucial role in determining the ultimate performance of machine learning models [1; 2; 3], affecting both the convergence of training algorithms and the generalization capabilities of the final models. Therefore, hyper-parameter optimization, as a problem of finding the optimal values for the hyper-parameters in a machine learning system, is fundamental and important.
Since manual hyper-parameter tuning can be time-consuming and often inefficient to find better hyperparameters, there have been many works on hyper-parameter optimization to automatically search for optimal hyper-parameters. Numerous existing hyper-parameter optimization algorithms compute the hyper-gradient with respect to hyper-parameter and utilize it to update current hyper-parameter [4; 5; 6; 7]. However, computing the hyper-gradient requires the objectives to be differentiable, and the hyper-parameters to be optimized must be continuous. As such, hyper-gradients cannot be obtained in many important hyper-parameter optimization problems. For instance, in recommendation system and knowledge graph completion, non-differentiable evaluation metrics such as MRR and NDCG are commonly used [8; 9; 10], which makes it impossible to compute hyper-gradients. Moreover, many
hyper-parameters in machine learning procedures are discrete (e.g. number of layers and hidden dimension size), where computing their hyper-gradient is impossible either.
To solve these problems without the use of hyper-gradients, existing methods often use some derivative-free optimization algorithms, e.g., random search [3], Bayesian optimization [11] or genetic programming [12]. However, these methods are often very time-consuming, and cannot find optimal hyper-parameters. Another type of method is based on stochastic relaxation [13; 14; 15], which constructs another differentiable objective from the original upper-level (i.e. the objective to find the hyper-parameters that have the best performance) objective, and use gradient-based optimization algorithms to optimize this new objective. This framework also covers methods based on reinforcement learning [16]. However, existing works under this framework only considers first-order information (i.e., gradient) of the new objective, which is still not sufficient enough as the upper-level objective can be very complex.
To address the limitations of existing works, in this paper, we propose a novel algorithm for hyperparameter optimization that does not depend on hyper-gradients. Based on stochastic relaxation, we use cubic regularization [17] to optimize the relaxed objective and consider inexact solutions for the lower-level problems. Specially, our contributions can be summarized as follows:
- We propose to utilize cubic regularization based on stochastic relaxation to accelerate convergence and avoid saddle points in hyper-parameter optimization problems.
- We provide theoretical analysis of the proposed method, showing that the proposed method converges to approximate second-order stationary points with only inexactly solving lower level objectives.
- Experiments on synthetic data and diverse applications demonstrate that the proposed method can help find hyper-parameters that lead to models with good performances.
2.1 Hyper-parameter Optimization
Hyper-parameter optimization [18; 19], as a subfield of automated machine learning (AutoML) [20], aims to automate the design and tuning of machine learning models. Most existing works on hyperparameter optimization adopt the bi-level formulation [21; 4], where hyper-parameters are optimized in the upper-level objective, and model parameters are trained in the lower-level objective.
Generally, hyper-parameter optimization algorithms can be divided into two types. The first is derivative-free methods, which only depends on the objective value. For example, grid search [22] and random search [3] are two simple search methods to try different hyper-parameter values and choose the best among them based on the upper-level objective. Another example is model-based Bayesian optimization [11; 2], which builds a probabilistic model to map different hyper-parameters to their evaluation performance and updates this model along with the optimization process. Genetic programming [12] starts with a population of randomly sampled hyper-parameters, and evolves new generations of hyper-parameters by selecting top-performing ones and recombining them. Hyperband [23] improves upon random search by stopping training models with bad performances early before convergence. These methods may suffer from huge computation cost when the number of hyper-parameters is more than a few, and usually do not have a theoretical guarantee for convergence.
Another approach uses hyper-gradient to update hyper-parameters by gradient descent [4; 5; 6; 7]. Despite their solid theoretical analysis and good empirical performances, these methods require both continuous hyper-parameter and differentiable upper-level objective to compute the hyper-gradient. In contrast, our method handles the cases where hyper-gradient is not available, such as when the hyper-parameters are discrete or the objectives are non-differentiable. Moreover, exact computation of the hyper-gradient can be time-consuming, and is only feasible for small models.
2.2 Cubic Regularization
Cubic regularization [17] is a novel technique for general unconstrained and smooth optimization problems. The main idea is to construct a cubic polynomial from the original problem, and update the parameters by finding the minimizer of this polynomial in each iteration. Since the pioneering work [17] in this direction, many variants have been proposed to improve its performance. For
example, ARC [24] proposes to use an approximate global minimizer to reduce the computation cost of cubic regularization method. More recently, cubic regularization is utilized to accelerate convergence in stochastic non-convex optimization problems [25; 26; 27].
Compared to first-order optimization methods, cubic regularization can converge faster [17; 25] and escape saddle points [26]. Despite its solid theoretical foundations and good empirical performances, cubic regularization involves computing the full Hessian matrix, which prevents its application when the number of parameters is large (e.g., training a neural network). Nevertheless, since hyperparameters are often low-dimensional, computing the Hessian matrix will not be a bottleneck. Our proposed method is also the first to use cubic regularization in a bi-level optimization problem.
Mathematically, hyper-parameter optimization with stochastic relaxation [13] can be formulated as the following bi-level optimization problem:
$$
\boldsymbol {\theta} ^ {*} = \underset {\boldsymbol {\theta}} {\arg \min } \left\{\mathcal {J} (\boldsymbol {\theta}) = \mathbb {E} _ {z \sim p _ {\boldsymbol {\theta}} (z)} [ H (w ^ {*} (z), z; D _ {\mathrm {v a l}}) ] \right\} \text {s . t .} w ^ {*} (z) = \underset {w} {\arg \min } F (w, z; D _ {\mathrm {t r a}}), \tag {1}
$$
where  $w$  denotes the model parameter and  $z$  denotes the hyper-parameter. Through stochastic relaxation, the hyper-parameter  $z$  is sampled from the probability distribution  $p_{\theta}(z)$  parameterized by  $\theta$ .  $H$  and  $F$  are the performance evaluation metrics for the training dataset  $D_{\mathrm{tra}}$  and the validation dataset  $D_{\mathrm{val}}$ , respectively.
Different from directly searching for good hyper-parameters, (1) introduces a probability distribution  $p_{\theta}(z)$  on hyper-parameter  $z$ , so that we can optimize  $\theta$  instead of  $z$ . The following Lemma shows that optimizing  $\theta$  is equivalent to finding the best hyper-parameter  $z$  on the validation set:
Lemma 3.1. Assume that for any  $z^*$ , there exists a  $\pmb{\theta}$  (could be infinite) such that  $p_{\pmb{\theta}}(z) = \delta(z - z^*)$  where  $\delta(\cdot)$  is the Dirac's delta function. Then, (i)  $\inf_{\pmb{\theta}} \mathcal{J}(\pmb{\theta}) = \min_{z} H(w^*(z), z; D_{val})$ ; (ii) The optimal  $\pmb{\theta}$  in (1) satisfies  $p_{\bar{\theta}}(z) = \delta(z - \arg \min_{z} H(w^*(z), z; D_{val}))$ .
The proof is in Appendix A. The solution of (1) can be easier as  $\mathcal{J}(\pmb{\theta})$  in (1) is always differentiable, even when we cannot compute the hyper-gradient with respect to  $z$ . As is introduced in Section 1, hyper-parameter optimization problems where hyper-gradient cannot be computed are really common. Here are two examples:
Example 1: Hyper-parameter optimization for knowledge graph completion [9; 28]. A knowledge graph  $G$  is a set of triplets  $\{(h, r, t)\}$  that indicates entity  $h$  has relation  $r$  to another entity  $t$ . Knowledge graph completion is to learn a model to predict unknown facts (i.e., triplets with missing components). Here, we mainly focus on embedding-based models, and we consider the task of tuning hyper-parameters for this type of model. Under the formulation of (1),  $w$  refers to the embedding model parameters,  $z$  refers to the hyper-parameters (e.g. learning rate, regularizer, scoring function).  $H$  is a discrete evaluation metric (e.g. MRR, Hit@10).  $F$  corresponds to the loss function for training.  $D_{\mathrm{tra}}$  and  $D_{\mathrm{val}}$  are formed by training and validation triplets in the knowledge graph.
Example 2: Schedule search for learning with noisy labels [29; 30]. In this application, we have a noisy training dataset (i.e., labels may be incorrect), and we use a proportion schedule to control the proportion of small-loss samples for model training in each epoch, which can avoid over-fitting on incorrect labels. Existing works [29; 30] demonstrate that different schedules have significant impact on the final performance. Therefore, we consider using hyper-parameter  $z$  to express this schedule, and we can optimize  $z$  to help find good schedules that lead to good models. Under the formulation of (1), we let  $w$  be the model parameter,  $w^{*}(z)$  be the final model parameters trained with schedule parameterized by  $z$ .  $F$  denotes the training loss and  $H$  denotes the negative validation set accuracy.  $D_{\mathrm{tra}}$  is the training dataset with noisy label, and  $D_{\mathrm{val}}$  is the clean validation dataset.
3.2 Proposed Method
Since the upper-level objective in (1) can be highly non-convex, common optimization methods such as gradient descent and Newton method may get stuck in saddle points, which leads to poor
performance. As such, we propose to use cubic regularization, which can avoid convergence to saddle points, to optimize the relaxed objective  $\mathcal{J}(\pmb{\theta})$  in (1). We first briefly introduce the main procedure of using cubic regularization to minimize a given objective  $\mathcal{J}(\pmb{\theta})$ . Denote  $\pmb{\theta}^{m}$  as the value of  $\pmb{\theta}$  in the  $m$ -th iteration, cubic regularization will first construct the following surrogate:
$$
c _ {m} (\boldsymbol {\Delta}) = \mathcal {J} \left(\boldsymbol {\theta} ^ {m}\right) + \left(\nabla \mathcal {J} \left(\boldsymbol {\theta} ^ {m}\right)\right) ^ {\top} \boldsymbol {\Delta} + \frac {1}{2} \boldsymbol {\Delta} ^ {\top} \left(\nabla^ {2} \mathcal {J} \left(\boldsymbol {\theta} ^ {m}\right)\right) \boldsymbol {\Delta} + \frac {\rho}{6} \| \boldsymbol {\Delta} \| _ {2} ^ {3}, \tag {2}
$$
where  $\rho$  is a parameter set in advance, We will then update  $\theta^m$  by  $\theta^{m + 1} = \theta^m +\arg \min_{\Delta}c_m(\Delta)$  which involves minimizing the above surrogate as a sub-procedure.
To compute gradient  $\nabla \mathcal{J}(\pmb{\theta}^m)$  and Hessian  $\nabla^2\mathcal{J}(\pmb{\theta}^m)$ , we have the following propositions:
Proposition 3.2.  $\nabla \mathcal{J}(\pmb {\theta}) = \mathbb{E}_{p_{\pmb{\theta}}(z)}\bigl [s^{*}(\pmb {\theta};z)\bigr ]$  where  $s^* (\pmb {\theta};z) = H(w^* (z),z;D_{val})\cdot \nabla \log p_{\pmb{\theta}}(z)$
Proposition 3.3.  $\nabla^2\mathcal{J}(\pmb {\theta}) = \mathbb{E}_{p_\pmb{\theta}(z)}[H^* (\pmb {\theta};z)],$  where  $H^{*}(\pmb {\theta};z) = H(w^{*}(z),z;D_{val})$ $(\nabla \log p_{\pmb{\theta}}(z)\nabla \log p_{\pmb{\theta}}(z)^{\top} + \nabla^{2}\log p_{\pmb{\theta}}(z))$
Proofs of these two propositions are in Appendix A. Note that we do not require the original upper-level objective  $H$  to be differentiable, as we only need to use the value of upper-level objective  $H(w^{*}(z),z;D_{\mathrm{val}})$  to compute the gradient and Hessian of  $\mathcal{J}(\pmb {\theta})$ . Nevertheless, these two propositions still cannot be directly used, as they require infinite number of samples  $z$  as well as the corresponding exact lower-level solutions  $w^{*}(z)$ , and both of them are impossible to obtain in practice. As such, we first define  $w^{\prime}(z)$  to be an approximate solution of the lower-level objective with hyper-parameter  $z$ . Then we define  $\pmb {s}(\pmb {\theta};z) = H(w'(z),z;D_{\mathrm{val}})\cdot \nabla \log p_{\pmb{\theta}}(z)$  and  $\pmb {H}(\pmb {\theta};z) = H(w'(z),z;D_{\mathrm{val}})\cdot$ $(\nabla \log p_{\pmb{\theta}}(z)\nabla \log p_{\pmb{\theta}}(z)^{\top} + \nabla^{2}\log p_{\pmb{\theta}}(z))$  as approximations of  $s^{*}(\pmb {\theta};z)$  and  $\pmb {H}^{*}(\pmb {\theta};z)$ . With these notations, the approximated gradient (denoted as  $\pmb{g}^{m}$ ) and Hessian (denoted as  $\pmb{B}^{m}$ ) in the  $m$ -th iteration are computed as:
$$
\boldsymbol {g} ^ {m} = \frac {1}{K} \sum_ {k = 1} ^ {K} \boldsymbol {s} \left(\boldsymbol {\theta} ^ {m}; z ^ {k}\right), \tag {4}
$$
In other words, we will draw  $K$  different hyper-parameters  $z^k$  from the distribution  $p_{\theta^m}(z)$  in  $m$ -th iteration, and use them to obtain the corresponding approximate solutions  $w'(z^k)$ . Then we can compute the approximate gradient  $\pmb{g}^m$  and Hessian  $\pmb{B}^m$ , and an approximate surrogate  $\tilde{c}_m(\pmb{\theta})$  is then given by:
$$
\tilde {c} _ {m} (\boldsymbol {\Delta}) = \left(\boldsymbol {g} ^ {m}\right) ^ {\top} \boldsymbol {\Delta} + \frac {1}{2} \boldsymbol {\Delta} ^ {\top} \boldsymbol {B} ^ {m} \boldsymbol {\Delta} + \frac {\rho}{6} \| \boldsymbol {\Delta} \| _ {2} ^ {3}. \tag {5}
$$
Compared to  $c_{m}(\pmb{\theta})$  in (2), we remove the constant term  $\mathcal{I}(\pmb{\theta}^{m})$ , which will not affect the minimizer.  $\pmb{\theta}$  is then updated as  $\pmb{\theta}^{m + 1} = \pmb{\theta}^{m} + \pmb{\Delta}^{m}$ , where  $\pmb{\Delta}^{m} = \arg \min_{\pmb{\Delta}}\tilde{c}_{m}(\pmb{\Delta})$  is the minimizer of this new surrogate and can be obtained by existing solvers on cubic regularization (e.g., [24]). The complete algorithm is shown in Algorithm 1.
Algorithm 1 Hyper-parameter optimization with cubic regularization.
1: Initialize  $\pmb{\theta}^{0} = \mathbf{1}$  and  $z^{*}$  randomly sampled from  $p_{\pmb{\theta}^{0}}(z)$ .
2: for  $m = 0, \dots, M - 1$  do
3: for  $k = 0, \dots, K - 1$  do
4: Draw hyper-parameter  $z^k$  from  $p_{\theta^m}(z)$ ;
5: Optimize lower-level objective in (1) to obtain  $w'(z^k)$ ; // most expensive step
6: Update  $z^{*} = z^{k}$  if we have a better validation performance with hyper-parameter  $z^{k}$
7: end for
8: Compute  $g^{m}$  using (3) and  $B^{m}$  using (4);
9: Compute  $\bar{\Delta}^m$  by  $\bar{\Delta}^m = \arg \min_{\Delta}\tilde{c}_m(\Delta)$
0: Update  $\pmb{\theta}^{m + 1} = \pmb{\theta}^m +\pmb{\Delta}^m$
1: end for
12: Perform the training step with  $z^{*}$  and obtain the final model parameter  $w^{*}$ ;
13: return Final model parameter  $w^{*}$
Since the number of hyper-parameters is often not too large,  $\theta$  is also a low-dimensional variable in most cases. For example, for the knowledge graph application in Section 4.2.1, the dimension of
$\theta$  is only 25, as we sum up the number of possible values for each hyper-parameter. This is almost negligible compared with the embedding model, which often has millions of parameters. As such, optimizing (5) takes little time compared with solving the lower-level problem, which will also be empirically verified in Section 4.4.
Previous works that use cubic regularization for stochastic non-convex optimization [25; 26] do not consider bi-level optimization problems, and assume unbiased estimations of gradient and Hessian. As such, their analysis cannot be directly generalized here as our approximated gradient and Hessian are not unbiased due to inexact lower-level solutions. To analyze the convergence of Algorithm 1, we first introduce the definition for saddle point and approximate second-order stationary point, which is also used in [26] for convergence analysis:
Definition 3.4 ([26]). For an objective  $\mathcal{J}$  with  $\rho$ -Lipschitz Hessian,  ${}^3\pmb{\theta}$  is an  $\epsilon$ -second-order stationary point of  $\mathcal{J}$  if  $\| \nabla \mathcal{J}(\pmb{\theta}) \|_2 \leq \epsilon$  and  $\lambda_{\min}(\nabla^2 \mathcal{J}(\pmb{\theta})) \geq -\sqrt{\rho \epsilon}$ .
The definition of  $\epsilon$ -second-order stationary point can be regarded as a extension of  $\epsilon$ -first-order stationary point. For  $\epsilon$ -first-order stationary points, we only require the gradient  $\nabla \mathcal{J}(\pmb{\theta})$  to satisfy  $\| \nabla \mathcal{J}(\pmb{\theta}) \|_2 \leq \epsilon$ . And for  $\epsilon$ -second-order stationary points, we also require the eigenvalues of its Hessian to be large enough, so as to avoid getting stuck in saddle points as in Definition 3.5.
Definition 3.5. For an objective  $\mathcal{J}$  with  $\rho$ -Lipschitz Hessian,  $\pmb{\theta}$  is an  $\epsilon$ -saddle point of  $\mathcal{J}$  if  $\|\nabla \mathcal{J}(\pmb{\theta})\|_2 \leq \epsilon$  and  $\lambda_{\min}(\nabla^2 \mathcal{J}(\pmb{\theta})) < -\sqrt{\rho \epsilon}$ .
Now we make the following assumptions. Assumption 3.6 (i) can be easily satisfied as the upper-level objective  $H$  is naturally lower-bounded in real-world applications, and the parameter  $\rho$  in (2) should match (at least not smaller than) the upper bound  $\rho$  in Assumption 3.6 (i). This can be done by gradually increasing  $\rho$  in (2) to ensure convergence. Assumption 3.6 (ii) is also satisfied by common probability distributions used in our experiments (e.g. sigmoid function for experiments on synthetic data and softmax liked distribution for hyper-parameter optimization for knowledge graph completion). Assumption 3.6 (i), (iii) and (iv) are popularly used in existing works on cubic regularization [25; 26].
Assumption 3.6. Following assumptions on  $\mathcal{I}$  are made:
(i) Objective:  $\inf_{\theta} \mathcal{J}(\theta) > -\infty$  and  $\mathcal{J}$  is second-order differentiable with  $\rho$ -Lipschitz Hessian.
(ii) Bounded gradient and Hessian of probability distribution function  $p_{\pmb{\theta}}(z)$ :  $\| \nabla \log p_{\pmb{\theta}}(z)\| _2\leq Q_1$  and  $\| \nabla^2\log p_\pmb {\theta}(z)\| _2\leq Q_2$ .
(iii) Bounded variance and bias in gradient estimation:  $\mathbb{E}_{p_{\theta}(z)}\big(\| s(\pmb {\theta};z) - \mathbb{E}_{p_{\theta}(z)}(s(\pmb {\theta};z))\| _2^2\big)\leq \sigma_1^2$  and  $\| s(\pmb {\theta};z) - \mathbb{E}_{p_{\theta}(z)}(s(\pmb {\theta};z))\| _2\leq M_1$  for all  $\pmb{\theta}$  and  $z$
(iv) Bounded error in Hessian estimation: denote the spectral norm of a matrix  $\mathbf{A}$  (i.e., the maximum singular value of  $\mathbf{A}$ ) as  $\| \mathbf{A} \|_{\mathrm{sp}}$ , we have  $\mathbb{E}_{p_{\theta}(z)}\big(\| \mathbf{H}(\boldsymbol{\theta}; z) - \mathbb{E}_{p_{\theta}(z)}(\mathbf{H}(\boldsymbol{\theta}; z)) \|_{\mathrm{sp}}\big) \leq \sigma_2$  and  $\| \mathbf{H}(\boldsymbol{\theta}; z) - \mathbb{E}_{p_{\theta}(z)}(\mathbf{H}(\boldsymbol{\theta}; z)) \|_{\mathrm{sp}} \leq M_2$  for all  $\boldsymbol{\theta}$  and  $z$ .
The following theorem guarantees the convergence of Algorithm 1 (proof is in Appendix B).
Theorem 3.7. For any  $\mathcal{J}$  satisfying Assumption 3.6, and approximate lower-level solution  $w'(z)$  with  $|H(w^*(z), z; D_{val}) - H(w'(z), z; D_{val})| \leq \min \left( \frac{\epsilon}{64Q_1}, \frac{\sqrt{\rho\epsilon}}{72(Q_1^2 + Q_2)} \right)$ , there exists  $K = \bar{\mathcal{O}}\left( \frac{1}{\epsilon^2} \log \left( \frac{1}{\delta} \right) \right)$  with probability  $1 - \delta$  for any  $\delta > 0$ , such that Algorithm 1 produces an  $\epsilon$ -second-order stationary point  $\theta^M$  of  $\mathcal{J}$ , with  $M = \mathcal{O}\left( \frac{\sqrt{\rho} (\mathcal{J}(\pmb{\theta}^0) - \mathcal{J}(\pmb{\theta}^*))}{\epsilon^{1.5}} \right)$ .
Theorem 3.7 shows that cubic regularization (Algorithm 1) takes only  $M = \mathcal{O}(1 / \epsilon^{1.5})$  iterations and  $KM = \bar{\mathcal{O}}(1 / \epsilon^{3.5})$  samples to obtain an  $\epsilon$ -second-order stationary point. This is strictly faster than gradient-based optimization methods [14; 15], which require  $\bar{\mathcal{O}}(1 / \epsilon^4)$  samples only to obtain an  $\epsilon$ -first-order stationary point. We also note that the convergence rate depends on two parts: inexactness
3In other words, for any  $z,y$  , we have  $\left\| \nabla^2\mathcal{J}(z) - \nabla^2\mathcal{J}(y)\right\|_{\mathrm{sp}}\leq \rho \| z - y\| _2$  , where  $\| \cdot \|_{\mathrm{sp}}$  denotes the matrix spectral norm.
of lower-level solutions  $|H(w^{*}(z), z; D_{\mathrm{val}}) - H(w'(z), z; D_{\mathrm{val}})|$ , and the number of hyper-parameter samples  $K$  to compute the approximate gradient  $g^{m}$  and Hessian  $B^{m}$ . To better demonstrate their impacts, we consider the following two special cases:
- Set  $K$  to  $\infty$ : in such case, the convergence is solely controlled by the inexactness of lower-level solutions. To obtain a good  $\theta$  (hence good hyper-parameter  $z$ ), we need to obtain good lower-level solutions, which is intuitive as inaccurate lower-level solutions cannot help us identify good hyper-parameters.
- Using exact lower-level solutions: in such case, the convergence is solely controlled by the number of hyper-parameter samples  $K$ . To obtain a good  $\theta$  (hence good hyper-parameter  $z$ ), we need  $K$  to be sufficiently large to accurately compute the approximate gradient  $g^{m}$  and Hessian  $B^{m}$ . This is also intuitive as inaccurate gradient or Hessian cannot lead to good  $\theta$ , and matches previous analysis [25; 26] that focus on single-level optimization problems.