SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The above analysis will also be justified by empirical results in Section 4.3, where we consider the impact of different number of samples  $K$  and different training epochs in lower-level, which causes the inexact solution of the lower-level objective.
Comparison with existing works While there have been many works on hyper-parameter optimization [30; 14; 15] based on stochastic relaxation [13], our method has two significant differences. First, to the best of our knowledge, our method is the first method with convergence guarantee to approximate second-order stationary points (second-order convergence guarantees), while previous methods only have first-order convergence guarantees [30; 15]. As such, our method can avoid convergence to bad saddle points. While a recent work [31] also claims that they are able to escape saddle points in bi-level optimization, they require hyper-gradients to be available and more strict assumptions on objectives. Moreover, our method allows the use of inexact lower-level solutions, while existing methods all assume lower-level solutions are exact. Since lower-level problems are often complex non-convex problems, inexact lower-level solutions are more realistic and suitable for convergence analysis. Detailed comparison upon existing works can also be found in Appendix C.
4.1 Experiment on Synthetic Data
First, we consider a problem of filtering useful features for a linear model [32; 33]. We construct a synthetic dataset on 5-way classification, where the inputs are 50-dimensional vectors. Of the 50 features, 25 have different distributions based on their ground-truth labels, generated from Gaussian distributions with different means for each class and the same variance. The remaining 25 features are filled with i.i.d. Gaussian white noise. The hyper-parameter  $z \in \{0,1\}^{50}$  is a 50-dimensional 0/1 vector to specify which dimensions are masked. We use cross entropy loss as the loss function in the lower-level objective and use AUC to measure the performance of the classifier in the upper-level objective. For the mask of  $i$ -th dimension  $z_{i}$ , we use sigmoid function to represent the probability to mask that dimension, i.e.,  $p_{\theta_i}(z_i = 1) = 1 / (1 + \exp (-\theta_i))$  and  $p_{\theta_i}(z_i = 0) = 1 - p_{\theta_i}(z_i = 1)$ . The complete distribution  $p_{\theta}(z)$  is represented by successive multiplications  $p_{\theta}(z) = \prod_{i=1}^{50} p_{\theta_i}(z_i)$ . Experiments are conducted on a 24GB NVIDIA GeForce RTX 3090 GPU.
We compare the proposed method (Cubic) with methods that are also based on stochastic relaxation, including gradient descent (GD) [34], natural gradient (NG) [35], and Newton's methods (Newton) [36]. The validation AUC of different methods are shown in Figure 1(a). We can see that our proposed method clearly outperforms other methods based on stochastic relaxation.
Furthermore, to empirically verify that our proposed method does escape bad saddle points, we compare the eigenvalues of Hessian matrix  $\nabla^2\mathcal{J}(\pmb{\theta})$  with  $\pmb{\theta}$  found by different methods. For easier comparison, we divide each method's eigenvalues by their largest, which makes different methods have the same largest eigenvalue as 1. As can be seen from Figure 1(b), only our proposed method obtains an optimized  $\pmb{\theta}$  that makes the negative eigenvalues less significant compared to positive eigenvalues, which means only our proposed method can better escape the saddle points in  $\mathcal{J}(\pmb{\theta})$ .
(a) Validation AUC comparison.
(b) Eigenvalue comparison.
Figure 1: Comparison among different gradient based methods for mask learning in synthetic data classification.
4.2 Experiments on Real-world Data
In this section, we mainly consider two applications, hyper-parameter optimization for knowledge graph completion and schedule search on learning with noisy training data, which are both introduced as examples in Section 3.1. For baselines, we choose hyper-parameter optimization methods that are commonly used in hyper-parameter optimization literature [19; 30; 28], including random search (Random) [3], Bayesian optimization (BO) [11], Hyperband [23] and reinforcement learning (RL) [37]. These baseline methods are applied to the original hyper-parameter search problem instead of the problem after stochastic relaxation.
4.2.1 Hyper-parameter Optimization for Knowledge Graph Completion
For this application, we need to tune several discrete hyper-parameters for a knowledge graph embedding model. The hyper-parameters to be tuned include negative sampling number, regularizer, loss function, gamma (for margin ranking loss), initializer, learning rate and score function, and more details can be found in Appendix E.2.
As all hyper-parameters considered in this application are discrete, we choose softmax-liked distributions to represent the probability to select a specific value for each hyper-parameter. For example, consider a hyper-parameter  $z_{i}$  with possible values  $\{z_{i1},\dots,z_{in}\}$ , the probability of  $z_{i} = z_{ij}$  is  $p(z_{i} = z_{ij}) = \frac{\exp(\theta_{ij})}{\sum_{k=1}^{n}\exp(\theta_{ik})}$ , where  $\theta_{ij}$  denote the probability function parameter corresponding to hyper-parameter  $z_{ij}$ . The dimension of distribution parameter is 25. Two well-known knowledge graph datasets, FB15k237 [38] and WN18RR [39], are used in experiments and their statistics are in Appendix E.2.
The experimental results are presented in Figure 2, which shows the achieved maximum reciprocal rank (MRR)  ${}^{4}$  with respect to the number of trained models for different search algorithms. Our proposed method exhibits rapid convergence, as Theorem 3.7 guarantees its fast convergence rate. Moreover, our proposed method outperforms other search algorithms and is more close to the global optimum in terms of MRR because cubic regularization allows our proposed method to escape from bad saddle points, while other methods may get trapped in such saddle points.
4.2.2 Schedule Search on Learning with Noisy Training Data
For this application, we consider the problem of searching for a schedule  $R_{z}(t)$  parameterized by hyper-parameter  $z$  [30]. The hyper-parameter  $z$  is divided into two parts:  $z \equiv (\alpha, \{\beta^i\})$ , and  $R_{z}(t)$  is parameterized as follows:  $R_{z}(t) \equiv \sum_{i=1}^{I} \alpha_{i} r^{i}(t; \beta^{i})$ , where  $\alpha = (\alpha_{1}, \ldots, \alpha_{I})$  can be seen as a set of weights with  $\alpha_{i} \geq 0$  and  $\sum_{i=1}^{I} \alpha_{i} = 1$ . Settings of functions  $r^{i}(t; \beta^{i})$  parameterized with  $\beta^{i}$  are shown in Table 1. Here the dimension of hyper-parameter is 36.
Denote the number of test triplets as  $n$ , and the rank of  $m$ th triplet prediction as  $\mathrm{rank}_m$ ,  $\mathrm{MRR} = \frac{1}{n}\sum_{m=1}^{n}\frac{1}{\mathrm{rank}_m}$ .
(a) FB15k237.
(b) WN18RR.
Figure 2: Validation MRR w.r.t. total number of trained models during search in hyper-parameter optimization for knowledge graph completion.
We use CIFAR-10 dataset for experiments with 50k, 5k, 5k image samples as training, validation, test data, respectively. We consider two different label noise settings:  $50\%$  symmetric noise and  $45\%$  pair flipping noise, and detailed generation process for the two settings is introduced in Appendix E.3.
Figure 3 shows the validation accuracy with respect to the total number of trained models in the search process. As can be seen, our method with cubic regularization is more efficient than the other algorithms compared. Moreover, our proposed method converges to a more accurate model because of the ability of cubic regularization to escape from bad saddle points.
(a) symmetry flipping  $(50\%)$ .
Figure 3: Validation accuracy w.r.t. total number of trained models during search for schedule search on learning with noisy training data.
(b) pair flipping  $(45\%)$ .
Here we conduct ablation study on two important factors of our proposed method: the number of samples  $K$  (which affects the error in estimating the gradient and Hessian of  $\mathcal{J}(\pmb{\theta})$ ) and number of training epochs for each model (which causes the inexact solution of the lower-level objective). The experiments are done on the synthetic data experiment in section 4.1. Results are shown in Figure 4, where in the first row we fix the number of upper-level iterations among different parameter settings, while overall training time is controlled the same in the second row. We fix the epoch number as 40 for experiment on  $K$  and set  $K = 5$  for
experiment on epoch number. In the experiment on the number of samples  $K$  (Figure 4(a)), we can see that our method performs better when  $K$  is larger, which means that when the estimation for gradient and Hessian of  $\mathcal{J}(\pmb{\theta})$  is the more precise, the optimization of  $\mathcal{J}(\pmb{\theta})$  is more effective. However, when the overall training time is limited, excessively large  $K$  (e.g.  $K = 50$ ) causes decline
Table 1: The basis functions used to define the search space in the experiments. Here,  $T$  is the total number of training epochs.
r1(t;β1)	e-β21tβ1+ β31(t/T)β4
r2(t;β2)	e-β22tβ12+ β32log(1+tβ42)/log(1+Tβ42)
r3(t;β3)	1/(1+β32t)β31+ β33(t/T)β43
r4(t;β4)	1/(1+β42t)β41+ β34log(1+tβ44)/log(1+Tβ44)
in performance because the number of parameter update for  $\theta$  is largely reduced. In the experiment for training epoch number (Figure 4(b)), we can see that the performance is better when the epoch number is larger (the solution of lower-level objective is more precise).
(a) number of sample K.
(b) number of training epochs.
Figure 4: Ablation study for our proposed method on synthetic data.
4.4 Time Cost in Experiments
In this section, we evaluate the time cost on different parts of our method in three experiments above. The total time cost of these hyper-parameter optimization problems can be mainly divided into two parts: (i) the model training with a specific hyper-parameter, and (ii) the update step of  $\theta$ . Table 2 shows the the clock time in different datasets, where we conduct each experiment for 5 times. As we can see, time cost in model training is consistently far more than time cost in update step of  $\theta$ , which shows that it is worthwhile to spend more time to exploit the curvature of upper-level optimization objective in (1).
Table 2: Clock time (in seconds) for model training and update step of  $\theta$  .
synthetic data (Section 4.1)	FB15k237 (Section 4.2.1)	WN18RR (Section 4.2.1)	CIFAR10 (Section 4.2.2)
model training	710 ± 37	86298 ± 760	79471 ± 936	163992 ± 801
update step of θ	16.1 ± 4.2	79.1 ± 2.7	72.4 ± 2.3	14.5 ± 1.4
In this paper, we address the challenge of hyper-parameter optimization in scenarios where hypergradients are unavailable. To tackle this problem, we introduce a novel algorithm that leverages cubic regularization based on stochastic relaxation. Our proposed method offers several advantages over existing approaches, including accelerated convergence and enhanced capability to escape from undesirable saddle points. Extensive experiments conducted on both synthetic and real-world datasets demonstrate the effectiveness of our proposed method in solving a wide range of hyper-parameter optimization problems.
This work was supported in part by The National Key Research and Development Program of China under grant 2020AAA0106000, a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. HKU C7004-22G), and NSF of China (No. 92270106).
[1] Ron Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In International Joint Conference on Artificial Intelligence, 1995.
[2] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 2012.
[3] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(2):281-305, 2012.
[4] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, 2018.
[5] Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning, 2020.
[6] Valerii Likhosherstov, Xingyou Song, Krzysztof Choromanski, Jared Q Davis, and Adrian Weller. Debiasing a first-order heuristic for approximate bi-level optimization. In International Conference on Machine Learning, 2021.
[7] Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning, 2021.
[8] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, 2016.
[9] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724-2743, 2017.
[10] Hong-Jian Xue, Xinyu Dai, Jianbing Zhang, Shujian Huang, and Jiajun Chen. Deep matrix factorization models for recommender systems. In International Joint Conference on Artificial Intelligence, 2017.
[11] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in Neural Information Processing Systems, 2011.
[12] Lingxi Xie and Alan Yuille. Genetic CNN. In IEEE International Conference on Computer Vision, 2017.
[13] Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6):721-741, 1984.
[14] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
[15] Youhei Akimoto, Shinichi Shirakawa, Nozomu Yoshinari, Kento Uchida, Shota Saito, and Kouhei Nishida. Adaptive stochastic natural gradient method for one-shot neural architecture search. In International Conference on Machine Learning, 2019.
[16] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
[36] Peng Xu, Fred Roosta, and Michael W Mahoney. Newton-type methods for non-convex optimization under inexact hessian information. Mathematical Programming, 184(1-2):35-70, 2020.
[37] Hadi S. Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint arXiv:1906.11527, 2019.
[38] Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality, 2015.
[39] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence, 2018.
[40] Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1-230, 2015.
[41] Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems, 2021.
[42] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[43] Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations, 2015.
[44] Timothee Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical tensor decomposition for knowledge base completion. In International Conference on Machine Learning, 2018.
[45] Zhanqiu Zhang, Jianyu Cai, and Jie Wang. Duality-induced regularizer for tensor factorization based knowledge graph completion. Advances in Neural Information Processing Systems, 2020.
[46] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems, 2013.
[47] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019.
[48] Théo Trouillon, Christopher R Dance, Éric Gaussier, Johannes Welbl, Sebastian Riedel, and Guillaume Bouchard. Knowledge graph completion via complex tensor factorization. Journal of Machine Learning Research, 18:1-38, 2017.
[49] Zhiqing Sun, Shikhar Vashisht, Soumya Sanyal, Partha Talukdar, and Yiming Yang. A re-evaluation of knowledge graph completion methods. In Annual Meeting of the Association for Computational Linguistics, 2020.