Title: Efficient Hyper-parameter Optimization with Cubic Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: hyper-parameter optimization

cubic regularization
Abstract: As hyper-parameters are ubiquitous and can significantly affect the model performance, hyper-parameter optimization is extremely important in machine learning. In this paper, we consider a sub-class of hyper-parameter optimization problems, where the hyper-gradients are not available. Such problems frequently appear when the performance metric is non-differentiable or the hyper-parameter is not continuous. However, existing algorithms, like Bayesian optimization and reinforcement learning, often get trapped in local optimals with poor performance. To address the above limitations, we propose to use cubic regularization to accelerate convergence and avoid saddle points. First, we adopt stochastic relaxation, which allows obtaining gradient and Hessian information without hyper-gradients. Then, we exploit the rich curvature information by cubic regularization. Theoretically, we prove that the proposed method can converge to approximate second-order stationary points, and the convergence is also guaranteed when the lower-level problem is inexactly solved. Experiments on synthetic and real-world data demonstrate the effectiveness of our proposed method.

Efficient Hyper-parameter Optimization with Cubic Regularization

Abstract

As hyper-parameters are ubiquitous and can significantly affect the model performance, hyper-parameter optimization is extremely important in machine learning. In this paper, we consider a sub-class of hyper-parameter optimization problems, where the hyper-gradients are not available. Such problems frequently appear when the performance metric is non-differentiable or the hyper-parameter is not continuous. However, existing algorithms, like Bayesian optimization and reinforcement learning, often get trapped in local optimals with poor performance. To address the above limitations, we propose to use cubic regularization to accelerate convergence and avoid saddle points. First, we adopt stochastic relaxation, which allows obtaining gradient and Hessian information without hyper-gradients. Then, we exploit the rich curvature information by cubic regularization. Theoretically, we prove that the proposed method can converge to approximate second-order stationary points, and the convergence is also guaranteed when the lower-level problem is inexactly solved. Experiments on synthetic and real-world data demonstrate the effectiveness of our proposed method.

1 Introduction

In a machine learning system, hyper-parameters are parameters that control the learning process. Wellknown examples include learning rate, and choice of loss functions or regularizers. Hyper-parameters play a crucial role in determining the ultimate performance of machine learning models [1; 2; 3], affecting both the convergence of training algorithms and the generalization capabilities of the final models. Therefore, hyper-parameter optimization, as a problem of finding the optimal values for the hyper-parameters in a machine learning system, is fundamental and important.

Since manual hyper-parameter tuning can be time-consuming and often inefficient to find better hyperparameters, there have been many works on hyper-parameter optimization to automatically search for optimal hyper-parameters. Numerous existing hyper-parameter optimization algorithms compute the hyper-gradient with respect to hyper-parameter and utilize it to update current hyper-parameter [4; 5; 6; 7]. However, computing the hyper-gradient requires the objectives to be differentiable, and the hyper-parameters to be optimized must be continuous. As such, hyper-gradients cannot be obtained in many important hyper-parameter optimization problems. For instance, in recommendation system and knowledge graph completion, non-differentiable evaluation metrics such as MRR and NDCG are commonly used [8; 9; 10], which makes it impossible to compute hyper-gradients. Moreover, many

†

hyper-parameters in machine learning procedures are discrete (e.g. number of layers and hidden dimension size), where computing their hyper-gradient is impossible either.

To solve these problems without the use of hyper-gradients, existing methods often use some derivative-free optimization algorithms, e.g., random search [3], Bayesian optimization [11] or genetic programming [12]. However, these methods are often very time-consuming, and cannot find optimal hyper-parameters. Another type of method is based on stochastic relaxation [13; 14; 15], which constructs another differentiable objective from the original upper-level (i.e. the objective to find the hyper-parameters that have the best performance) objective, and use gradient-based optimization algorithms to optimize this new objective. This framework also covers methods based on reinforcement learning [16]. However, existing works under this framework only considers first-order information (i.e., gradient) of the new objective, which is still not sufficient enough as the upper-level objective can be very complex.

To address the limitations of existing works, in this paper, we propose a novel algorithm for hyperparameter optimization that does not depend on hyper-gradients. Based on stochastic relaxation, we use cubic regularization [17] to optimize the relaxed objective and consider inexact solutions for the lower-level problems. Specially, our contributions can be summarized as follows:

We propose to utilize cubic regularization based on stochastic relaxation to accelerate convergence and avoid saddle points in hyper-parameter optimization problems.

Weprovide theoretical analysis of the proposed method, showing that the proposed method converge to approximate second-order stationary points with only inexactly solving lower level objectives.

Experiments on synthetic data and diverse applications demonstrate that the proposed method can help find hyper-parameters that lead to models with good performances.

2 Related Works

2.1 Hyper-parameter Optimization

Hyper-parameter optimization [18; 19], as a subfield of automated machine learning (AutoML) [20], aims to automate the design and tuning of machine learning models. Most existing works on hyperparameter optimization adopt the bi-level formulation [21; 4], where hyper-parameters are optimized in the upper-level objective, and model parameters are trained in the lower-level objective.

Generally, hyper-parameter optimization algorithms can be divided into two types. The first is derivative-free methods, which only depends on the objective value. For example, grid search [22] and random search [3] are two simple search methods to try different hyper-parameter values and choose the best among them based on the upper-level objective. Another example is model-based Bayesian optimization [11; 2], which builds a probabilistic model to map different hyper-parameters to their evaluation performance and updates this model along with the optimization process. Genetic programming [12] starts with a population of randomly sampled hyper-parameters, and evolves new generations of hyper-parameters by selecting top-performing ones and recombining them. Hyperband [23] improves upon random search by stopping training models with bad performances early before convergence. These methods may suffer from huge computation cost when the number of hyper-parameters is more than a few, and usually do not have a theoretical guarantee for convergence.

Another approach uses hyper-gradient to update hyper-parameters by gradient descent [4; 5; 6; 7]. Despite their solid theoretical analysis and good empirical performances, these methods require both continuous hyper-parameter and differentiable upper-level objective to compute the hyper-gradient. In contrast, our method handles the cases where hyper-gradient is not available, such as when the hyper-parameters are discrete or the objectives are non-differentiable. Moreover, exact computation of the hyper-gradient can be time-consuming, and is only feasible for small models.

2.2 Cubic Regularization

Cubic regularization [17] is a novel technique for general unconstrained and smooth optimization problems. The main idea is to construct a cubic polynomial from the original problem, and update the parameters by finding the minimizer of this polynomial in each iteration. Since the pioneering work [17] in this direction, many variants have been proposed to improve its performance. For

example, ARC [24] proposes to use an approximate global minimizer to reduce the computation cost of cubic regularization method. More recently, cubic regularization is utilized to accelerate convergence in stochastic non-convex optimization problems [25; 26; 27].

Compared to first-order optimization methods, cubic regularization can converge faster [17; 25] and escape saddle points [26]. Despite its solid theoretical foundations and good empirical performances, cubic regularization involves computing the full Hessian matrix, which prevents its application when the number of parameters is large (e.g., training a neural network). Nevertheless, since hyperparameters are often low-dimensional, computing the Hessian matrix will not be a bottleneck. Our proposed method is also the first to use cubic regularization in a bi-level optimization problem.

3 Methodology

3.1 Problem Formulation

Mathematically, hyper-parameter optimization with stochastic relaxation [13] can be formulated as the following bi-level optimization problem:

where w denotes the model parameter and z denotes the hyper-parameter. Through stochastic relaxation, the hyper-parameter z is sampled from the probability distribution p θ ( z ) parameterized by θ . H and F are the performance evaluation metrics for the training dataset D tra and the validation dataset D val, respectively.

Different from directly searching for good hyper-parameters, (1) introduces a probability distribution p θ ( z ) on hyper-parameter z , so that we can optimize θ instead of z . The following Lemma shows that optimizing θ is equivalent to finding the best hyper-parameter z on the validation set:

Lemma 3.1. Assume that for any z ∗ , there exists a θ (could be infinite) such that p θ ( z ) = δ ( z -z ∗ ) , where δ ( · ) is the Dirac's delta function. Then, (i) inf θ J ( θ ) = min z H ( w ∗ ( z ) , z ; D val ) ; (ii) The optimal ¯ θ in (1) satisfies p ¯ θ ( z ) = δ ( z -arg min z H ( w ∗ ( z ) , z ; D val )) .

The proof is in Appendix A. The solution of (1) can be easier as J ( θ ) in (1) is always differentiable, even when we cannot compute the hyper-gradient with respect to z . As is introduced in Section 1, hyper-parameter optimization problems where hyper-gradient cannot be computed are really common. Here are two examples:

Example 1: Hyper-parameter optimization for knowledge graph completion [9; 28]. A knowledge graph G is a set of triplets { ( h, r, t ) } that indicates entity h has relation r to another entity t . Knowledge graph completion is to learn a model to predict unknown facts (i.e., triplets with missing components). Here, we mainly focus on embedding-based models, and we consider the task of tuning hyper-parameters for this type of model. Under the formulation of (1), w refers to the embedding model parameters, z refers to the hyper-parameters (e.g. learning rate, regularizer, scoring function). H is a discrete evaluation metric (e.g. MRR, Hit@10). F corresponds to the loss function for training. D tra and D val are formed by training and validation triplets in the knowledge graph.

Example 2: Schedule search for learning with noisy labels [29; 30]. In this application, we have a noisy training dataset (i.e., labels may be incorrect), and we use a proportion schedule to control the proportion of small-loss samples for model training in each epoch, which can avoid over-fitting on incorrect labels. Existing works [29; 30] demonstrate that different schedules have significant impact on the final performance. Therefore, we consider using hyper-parameter z to express this schedule, and we can optimize z to help find good schedules that lead to good models. Under the formulation of (1), we let w be the model parameter, w ∗ ( z ) be the final model parameters trained with schedule parameterized by z . F denotes the training loss and H denotes the negative validation set accuracy. D tra is the training dataset with noisy label, and D val is the clean validation dataset.

3.2 Proposed Method

Since the upper-level objective in (1) can be highly non-convex, common optimization methods such as gradient descent and Newton method may get stuck in saddle points, which leads to poor

performance. As such, we propose to use cubic regularization, which can avoid convergence to saddle points, to optimize the relaxed objective J ( θ ) in (1). We first briefly introduce the main procedure of using cubic regularization to minimize a given objective J ( θ ) . Denote θ m as the value of θ in the m -th iteration, cubic regularization will first construct the following surrogate:

where ρ is a parameter set in advance, We will then update θ m by θ m +1 = θ m +argmin ∆ c m ( ∆ ) , which involves minimizing the above surrogate as a sub-procedure.

To compute gradient ∇ J ( θ m ) and Hessian ∇ 2 J ( θ m ) , we have the following propositions:

Proofs of these two propositions are in Appendix A. Note that we do not require the original upperlevel objective H to be differentiable, as we only need to use the value of upper-level objective H ( w ∗ ( z ) , z ; D val ) to compute the gradient and Hessian of J ( θ ) . Nevertheless, these two propositions still cannot be directly used, as they require infinite number of samples z as well as the corresponding exact lower-level solutions w ∗ ( z ) , and both of them are impossible to obtain in practice. As such, we first define w ′ ( z ) to be an approximate solution of the lower-level objective with hyper-parameter z . Then we define s ( θ ; z ) = H ( w ′ ( z ) , z ; D val ) · ∇ log p θ ( z ) and H ( θ ; z ) = H ( w ′ ( z ) , z ; D val ) · ( ∇ log p θ ( z ) ∇ log p θ ( z ) /latticetop + ∇ 2 log p θ ( z ) ) as approximations of s ∗ ( θ ; z ) and H ∗ ( θ ; z ) . With these notations, the approximated gradient (denoted as g m ) and Hessian (denoted as B m ) in the m -th iteration are computed as:

In other words, we will draw K different hyper-parameters z k from the distribution p θ m ( z ) in m -th iteration, and use them to obtain the corresponding approximate solutions w ′ ( z k ) . Then we can compute the approximate gradient g m and Hessian B m , and an approximate surrogate ˜ c m ( θ ) is then given by:

Compared to c m ( θ ) in (2), we remove the constant term J ( θ m ) , which will not affect the minimizer. θ is then updated as θ m +1 = θ m + ∆ m , where ∆ m = arg min ∆ ˜ c m ( ∆ ) is the minimizer of this new surrogate and can be obtained by existing solvers on cubic regularization (e.g., [24]). The complete algorithm is shown in Algorithm 1.

Since the number of hyper-parameters is often not too large, θ is also a low-dimensional variable in most cases. For example, for the knowledge graph application in Section 4.2.1, the dimension of

θ is only 25 , as we sum up the number of possible values for each hyper-parameter. This is almost negligible compared with the embedding model, which often has millions of parameters. As such, optimizing (5) takes little time compared with solving the lower-level problem, which will also be empirically verified in Section 4.4.

3.3 Theoretical Analysis

Previous works that use cubic regularization for stochastic non-convex optimization [25; 26] do not consider bi-level optimization problems, and assume unbiased estimations of gradient and Hessian. As such, their analysis cannot be directly generalized here as our approximated gradient and Hessian are not unbiased due to inexact lower-level solutions. To analyze the convergence of Algorithm 1, we first introduce the definition for saddle point and approximate second-order stationary point, which is also used in [26] for convergence analysis:

Definition 3.4 ([26]) . For an objective J with ρ -Lipschitz Hessian, 3 θ is an /epsilon1 -second-order stationary point of J if ‖∇ J ( θ ) ‖ 2 ≤ /epsilon1 and λ min ( ∇ 2 J ( θ )) ≥ - √ ρ/epsilon1 .

The definition of /epsilon1 -second-order stationary point can be regarded as a extension of /epsilon1 -first-order stationary point. For /epsilon1 -first-order stationary points, we only require the gradient ∇ J ( θ ) to satisfy ‖∇ J ( θ ) ‖ 2 ≤ /epsilon1 . And for /epsilon1 -second-order stationary points, we also require the eigenvalues of its Hessian to be large enough, so as to avoid getting stuck in saddle points as in Definition 3.5.

Definition 3.5. For an objective J with ρ -Lipschitz Hessian, θ is an /epsilon1 -saddle point of J if ‖∇ J ( θ ) ‖ 2 ≤ /epsilon1 and λ min ( ∇ 2 J ( θ )) < - √ ρ/epsilon1 .

Now we make the following assumptions. Assumption 3.6 (i) can be easily satisfied as the upperlevel objective H is naturally lower-bounded in real-world applications, and the parameter ρ in (2) should match (at least not smaller than) the upper bound ρ in Assumption 3.6 (i). This can be done by gradually increasing ρ in (2) to ensure convergence. Assumption 3.6 (ii) is also satisfied by common probability distributions used in our experiments (e.g. sigmoid function for experiments on synthetic data and softmax liked distribution for hyper-parameter optimization for knowledge graph completion). Assumption 3.6 (i), (iii) and (iv) are popularly used in existing works on cubic regularization [25; 26].

Assumption 3.6. Following assumptions on J are made:

(i) Objective: inf θ J ( θ ) > -∞ and J is second-order differentiable with ρ -Lipschitz Hessian.

(ii) Bounded gradient and Hessian of probability distribution function p θ ( z ) : ‖∇ log p θ ( z ) ‖ 2 ≤ Q 1 and ‖∇ 2 log p θ ( z ) ‖ 2 ≤ Q 2 .

(iii) Bounded variance and bias in gradient estimation: E p θ ( z ) ( ‖ s ( θ ; z ) -E p θ ( z ) ( s ( θ ; z )) ‖ 2 2 ) ≤ σ 2 1 and ‖ s ( θ ; z ) -E p θ ( z ) ( s ( θ ; z )) ‖ 2 ≤ M 1 for all θ and z .

(iv) Bounded error in Hessian estimation: denote the spectral norm of a matrix A (i.e., the maximum singular value of A ) as ‖ A ‖ sp, we have E p θ ( z ) ( ‖ H ( θ ; z ) -E p θ ( z ) ( H ( θ ; z )) ‖ sp ) ≤ σ 2 and ‖ H ( θ ; z ) -E p θ ( z ) ( H ( θ ; z )) ‖ sp ≤ M 2 for all θ and z .

The following theorem guarantees the convergence of Algorithm 1 (proof is in Appendix B).

Theorem 3.7. For any J satisfying Assumption 3.6, and approximate lower-level solution w ′ ( z ) with | H ( w ∗ ( z ) , z ; D val ) -H ( w ′ ( z ) , z ; D val ) | ≤ min( /epsilon1 64 Q 1 , √ ρ/epsilon1 72( Q 2 1 + Q 2 ) ) , there exists K = ¯ O ( 1 /epsilon1 2 log( 1 δ ) ) , with probability 1 -δ for any δ > 0 , such that Algorithm 1 produces an /epsilon1 -second-order stationary point θ M of J , with M = O ( √ ρ ( J ( θ 0 ) -J ( θ ∗ )) /epsilon1 1 . 5 ) .

Theorem 3.7 shows that cubic regularization (Algorithm 1) takes only M = O (1 / /epsilon1 1 . 5 ) iterations and KM = ¯ O (1 / /epsilon1 3 . 5 ) samples to obtain an /epsilon1 -second-order stationary point. This is strictly faster than gradient-based optimization methods [14; 15], which require ¯ O (1 / /epsilon1 4 ) samples only to obtain an /epsilon1 -first-order stationary point. We also note that the convergence rate depends on two parts: inexactness

of lower-level solutions | H ( w ∗ ( z ) , z ; D val ) -H ( w ′ ( z ) , z ; D val ) | , and the number of hyper-parameter samples K to compute the approximate gradient g m and Hessian B m . To better demonstrate their impacts, we consider the following two special cases:

Set K to ∞ : in such case, the convergence is solely controlled by the inexactness of lower-level solutions. To obtain a good θ (hence good hyper-parameter z ), we need to obtain good lowerlevel solutions, which is intuitive as inaccurate lower-level solutions cannot help us identify good hyper-parameters.

Using exact lower-level solutions: in such case, the convergence is solely controlled by the number of hyper-parameter samples K . To obtain a good θ (hence good hyper-parameter z ), we need K to be sufficiently large to accurately compute the approximate gradient g m and Hessian B m . This is also intuitive as inaccurate gradient or Hessian cannot lead to good θ , and matches previous analysis [25; 26] that focus on single-level optimization problems.

The above analysis will also be justified by empirical results in Section 4.3, where we consider the impact of different number of samples K and different training epochs in lower-level, which causes the inexact solution of the lower-level objective.

Comparison with existing works While there have been many works on hyper-parameter optimization [30; 14; 15] based on stochastic relaxation [13], our method has two significant differences. First, to the best of our knowledge, our method is the first method with convergence guarantee to approximate second-order stationary points (second-order convergence guarantees), while previous methods only have first-order convergence guarantees [30; 15]. As such, our method can avoid convergence to bad saddle points. While a recent work [31] also claims that they are able to escape saddle points in bi-level optimization, they require hyper-gradients to be available and more strict assumptions on objectives. Moreover, our method allows the use of inexact lower-level solutions, while existing methods all assume lower-level solutions are exact. Since lower-level problems are often complex non-convex problems, inexact lower-level solutions are more realistic and suitable for convergence analysis. Detailed comparison upon existing works can also be found in Appendix C.

4 Experiments

4.1 Experiment on Synthetic Data

First, we consider a problem of filtering useful features for a linear model [32; 33]. We construct a synthetic dataset on 5-way classification, where the inputs are 50-dimensional vectors. Of the 50 features, 25 have different distributions based on their ground-truth labels, generated from Gaussian distributions with different means for each class and the same variance. The remaining 25 features are filled with i.i.d. Gaussian white noise. The hyper-parameter z ∈ { 0 , 1 } 50 is a 50-dimensional 0/1 vector to specify which dimensions are masked. We use cross entropy loss as the loss function in the lower-level objective and use AUC to measure the performance of the classifier in the upper-level objective. For the mask of i -th dimension z i , we use sigmoid function to represent the probability to mask that dimension, i.e., p θ i ( z i = 1) = 1 / (1+exp( -θ i )) and p θ i ( z i = 0) = 1 -p θ i ( z i = 1) . The complete distribution p θ ( z ) is represented by successive multiplications p θ ( z ) = ∏ 50 i =1 p θ i ( z i ) . Experiments are conducted on a 24GB NVIDIA GeForce RTX 3090 GPU.

We compare the proposed method ( Cubic ) with methods that are also based on stochastic relaxation, including gradient descent ( GD ) [34], natural gradient ( NG ) [35], and Newton's methods ( Newton ) [36]. The validation AUC of different methods are shown in Figure 1(a). We can see that our proposed method clearly outperforms other methods based on stochastic relaxation.

Furthermore, to empirically verify that our proposed method does escape bad saddle points, we compare the eigenvalues of Hessian matrix ∇ 2 J ( θ ) with θ found by different methods. For easier comparison, we divide each method's eigenvalues by their largest, which makes different methods have the same largest eigenvalue as 1. As can be seen from Figure 1(b), only our proposed method obtains an optimized θ that make the negative eigenvalues less significant compared to positive eigenvalues, which means only our proposed method can better escape the saddle points in J ( θ ) .

0.94

global optimum

0.92

Best validation AUC

0.90

0.88

NG

Newton

0.86

BFGS

Cubic

0

100

200

300

400

500

600

700

Running time/s

(a) Validation AUC comparison.

1.00

NG

0.75

Newton

BFGS

0.50

Cubic

eigenvalue

0.25

Relative

0.00

0.25

-0.50

0.75

-1.00

0

10

20

30

40

50

Number of eigenvalues

(b) Eigenvalue comparison.

4.2 Experiments on Real-world Data

In this section, we mainly consider two applications, hyper-parameter optimization for knowledge graph completion and schedule search on learning with noisy training data, which are both introduced as examples in Section 3.1. For baselines, we choose hyper-parameter optimization methods that are commonly used in hyper-parameter optimization literature [19; 30; 28], including random search ( Random ) [3], Bayesian optimization ( BO ) [11], Hyperband [23] and reinforcement learning ( RL ) [37]. These baseline methods are applied to the original hyper-parameter search problem instead of the problem after stochastic relaxation.

4.2.1 Hyper-parameter Optimization for Knowledge Graph Completion

For this application, we need to tune several discrete hyper-parameters for a knowledge graph embedding model. The hyper-parameters to be tuned include negative sampling number, regularizer, loss function, gamma (for margin ranking loss), initializer, learning rate and score function, and more details can be found in Appendix E.2.

As all hyper-parameters considered in this application are discrete, we choose softmax-liked distributions to represent the probability to select a specific value for each hyper-parameter. For example, consider a hyper-parameter z i with possible values { z i 1 , ..., z in } , the probability of z i = z ij is p ( z i = z ij ) = exp( θ ij ) ∑ n k =1 exp( θ ik ) , where θ ij denote the probability function parameter corresponding to hyper-parameter z ij . The dimension of distribution parameter is 25 . Two well-known knowledge graph datasets, FB15k237 [38] and WN18RR [39], are used in experiments and their statistics are in Appendix E.2.

The experimental results are presented in Figure 2, which shows the achieved maximum reciprocal rank (MRR) 4 with respect to the number of trained models for different search algorithms. Our proposed method exhibits rapid convergence, as Theorem 3.7 guarantees its fast convergence rate. Moreover, our proposed method outperforms other search algorithms and is more close to the global optimum in terms of MRR because cubic regularization allows our proposed method to escape from bad saddle points, while other methods may get trapped in such saddle points.

4.2.2 Schedule Search on Learning with Noisy Training Data

For this application, we consider the problem of searching for a schedule R z ( t ) parameterized by hyper-parameter z [30]. The hyper-parameter z is divided into two parts: z ≡ ( α , { β i } ) , and R z ( t ) is parameterized as follows: R z ( t ) ≡ ∑ I i =1 α i r i ( t ; β i ) , where α = ( α 1 , . . . , α I ) can be seen as a set of weights with α i ≥ 0 and ∑ I i =1 α i = 1 . Settings of functions r i ( t ; β i ) parameterized with β i are shown in Table 1. Here the dimension of hyper-parameter is 36.

(a) FB15k237.

(b) WN18RR.

0.475

0.450

validationMRR

0.425

0.400

0.375

Best

0.350

Random

BO

0.325

RL

Hyperband

Cubic

0.300

0

100002000030000400005000060000

70000

80000

Runningtime/s

0.32

0.30

Best validation MRR

0.28

0.26

Random

0.24

BO

RL

Hyperband

0.22

Cubic

0

20000

40000

60000

80000

Running time/s

We use CIFAR-10 dataset for experiments with 50k, 5k, 5k image samples as training, validation, test data, respectively. We consider two different label noise settings: 50% symmetric noise and 45% pair flipping noise, and detailed generation process for the two settings is introduced in Appendix E.3.

Figure 3 shows the validation accuracy with respect to the total number of trained models in the search process. As can be seen, our method with cubic regularization is more efficient than the other algorithms compared. Moreover, our proposed method converges to a more accurate model because of the ability of cubic regularization to escape from bad saddle points.

(a) symmetry flipping (50%).

(b) pair flipping (45%).

50

Bestvalidationaccuracy

45

40

Random

BO

35

RL

Hyperband

Cubic

0

0000900000000000000000000000000090000000007

Running time/s

50.0

47.5

accuracy

45.0

Bestvalidationa

42.5

40.0

37.5

Random

BO

35.0

RL

Hyperband

32.5

Cubic

0

0000900000000000000000000000000009000000007

Running time/s

4.3 Ablation Study

Here we conduct ablation study on two important factors of our proposed method: the number of samples K (which affects the error in estimating the gradient and Hessian of J ( θ ) ) and number of training epochs for each model (which causes the inexact solution of the lower-level objective). The experiments are done on the synthetic data experiment in section 4.1. Results are shown in Figure 4, where in the first row we fix the number of upper-level iterations among different parameter settings, while overall training time is controlled the same in the second row. We fix the epoch number as 40 for experiment on K and set K = 5 for

r

1

(

t

;

β

1

)

e

-

β

1

2

t

β

1

1

+

β

1

3

(

t

T

)

β

1

4

r

2

(

t

;

β

2

)

e

-

β

2

2

t

β

2

1

+

β

2

3

log(1+

t

β

2

4

)

log(1+

T

β

2

4

)

r

3

(

t

;

β

3

)

1

(1+

β

3

2

t

)

β

3

1

+

β

3

3

(

t

T

)

β

3

4

r

4

(

t

;

β

4

)

1

(1+

β

4

2

t

)

β

4

1

+

β

4

3

log(1+

t

β

4

4

)

log(1+

T

β

4

4

)

experiment on epoch number. In the experiment on the number of samples K (Figure 4(a)), we can see that our method performs better when K is larger, which means that when the estimation for gradient and Hessian of J ( θ ) is the more precise, the optimization of J ( θ ) is more effective. However, when the overall training time is limited, excessively large K (e.g. K = 50 ) causes decline

in performance because the number of parameter update for θ is largely reduced. In the experiment for training epoch number (Figure 4(b)), we can see that the performance is better when the epoch number is larger (the solution of lower-level objective is more precise).

(a) number of sample K.

(b) number of training epochs.

0.92

Best validation AUC

0.91

0.90

K=1

0.89-

K=5

K=20

0.88-

0

20

40

60

80

100

Upper loop iterations(m)

0.92

0.91

Best validation AUC

0.90

0.89

K=1

0.88

K=5

0.87

K=20

0

50

100

150

200

250

300

Running time/s

0.92

0.90

0.88

AUC

0.86

BestvalidationA

0.84

0.82

0.80

epoch=5

epoch=10

0.78

epoch=20

epoch=40

0.76-

epoch=80

0

20

40

09

80

100

Upper loop iterations(m)

0.92

0.90

0.88

Best validation AUC

0.86

0.84

0.82

0.80

epoch=5

epoch=10

0.78 -

epoch=20

epoch=40

0.76

epoch=80

0

50

100

150

200

250

300

Running time/s

4.4 Time Cost in Experiments

In this section, we evaluate the time cost on different parts of our method in three experiments above. The total time cost of these hyper-parameter optimization problems can be mainly divided into two parts: (i) the model training with a specific hyper-parameter, and (ii) the update step of θ . Table 2 shows the the clock time in different datasets, where we conduct each experiment for 5 times. As we can see, time cost in model training is consistently far more than time cost in update step of θ , which shows that it is worthwhile to spend more time to exploit the curvature of upper-level optimization objective in (1).

5 Conclusion

In this paper, we address the challenge of hyper-parameter optimization in scenarios where hypergradients are unavailable. To tackle this problem, we introduce a novel algorithm that leverages cubic regularization based on stochastic relaxation. Our proposed method offers several advantages over existing approaches, including accelerated convergence and enhanced capability to escape from undesirable saddle points. Extensive experiments conducted on both synthetic and real-world datasets demonstrate the effectiveness of our proposed method in solving a wide range of hyper-parameter optimization problems.

Acknowledgments

This work was supported in part by The National Key Research and Development Program of China under grant 2020AAA0106000, a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. HKU C7004-22G), and NSF of China (No. 92270106).

References

Ron Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In International Joint Conference on Artificial Intelligence , 1995.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems , 2012.

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research , 13(2):281-305, 2012.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , 2018.

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , 2020.

Valerii Likhosherstov, Xingyou Song, Krzysztof Choromanski, Jared Q Davis, and Adrian Weller. Debiasing a first-order heuristic for approximate bi-level optimization. In International Conference on Machine Learning , 2021.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning , 2021.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning , 2016.

Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering , 29(12):2724-2743, 2017.

Hong-Jian Xue, Xinyu Dai, Jianbing Zhang, Shujian Huang, and Jiajun Chen. Deep matrix factorization models for recommender systems. In International Joint Conference on Artificial Intelligence , 2017.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in Neural Information Processing Systems , 2011.

Lingxi Xie and Alan Yuille. Genetic CNN. In IEEE International Conference on Computer Vision , 2017.

Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence , 6(6):721-741, 1984.

Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In International Conference on Learning Representations , 2019.

Youhei Akimoto, Shinichi Shirakawa, Nozomu Yoshinari, Kento Uchida, Shota Saito, and Kouhei Nishida. Adaptive stochastic natural gradient method for one-shot neural architecture search. In International Conference on Machine Learning , 2019.

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations , 2017.

Yurii Nesterov and Boris T Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming , 108(1):177-205, 2006.

Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges , pages 3-33, 2019.

Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing , 415:295-316, 2020.

Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. Automated Machine Learning: Methods, Systems, Challenges . Springer, 2018.

Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of Operations Research , 153(1):235-256, 2007.

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In International Conference on Machine Learning , 2007.

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research , 18(1):6765-6816, 2017.

Coralia Cartis, Nicholas IM Gould, and Philippe L Toint. Adaptive cubic regularisation methods for unconstrained optimization. part i: motivation, convergence and numerical results. Mathematical Programming , 127(2):245-295, 2011.

Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization. In International Conference on Machine Learning , 2017.

Nilesh Tripuraneni, Mitchell Stern, Chi Jin, Jeffrey Regier, and Michael I Jordan. Stochastic cubic regularization for fast nonconvex optimization. Advances in Neural Information Processing Systems , 2018.

Zhe Wang, Yi Zhou, Yingbin Liang, and Guanghui Lan. Stochastic variance-reduced cubic regularization for nonconvex optimization. In International Conference on Artificial Intelligence and Statistics , 2019.

Yongqi Zhang, Zhanke Zhou, Quanming Yao, and Yong Li. Kgtuner: Efficient hyper-parameter search for knowledge graph learning. arXiv preprint arXiv:2205.02460 , 2022.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems , 2018.

Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Kwok. Searching to exploit memorization effect in learning from corrupted labels. In International Conference on Machine Learning , 2020.

Minhui Huang, Kaiyi Ji, Shiqian Ma, and Lifeng Lai. Efficiently escaping saddle points in bilevel optimization. arXiv preprint arXiv:2202.03684 , 2022.

Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research , 3(Mar):1157-1182, 2003.

Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning , 2020.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations , 2019.

Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation , 10(2):251276, 1998.

Peng Xu, Fred Roosta, and Michael W Mahoney. Newton-type methods for non-convex optimization under inexact hessian information. Mathematical Programming , 184(1-2):35-70, 2020.

Hadi S. Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint arXiv:1906.11527 , 2019.

Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality , 2015.

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018.

Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning , 8(1-2):1-230, 2015.

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems , 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations , 2015.

Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical tensor decomposition for knowledge base completion. In International Conference on Machine Learning , 2018.

Zhanqiu Zhang, Jianyu Cai, and Jie Wang. Duality-induced regularizer for tensor factorization based knowledge graph completion. Advances in Neural Information Processing Systems , 2020.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems , 2013.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations , 2019.

Théo Trouillon, Christopher R Dance, Éric Gaussier, Johannes Welbl, Sebastian Riedel, and Guillaume Bouchard. Knowledge graph completion via complex tensor factorization. Journal of Machine Learning Research , 18:1-38, 2017.

Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, and Yiming Yang. A reevaluation of knowledge graph completion methods. In Annual Meeting of the Association for Computational Linguistics , 2020.

Peer reviews:
comments: SUMMARY:
{Summary}
This paper proposes a stochastic cubic regularization type algorithm for hyperparameter optimization that does not depend on hyper-gradients. Theoretical analysis shows that the proposed method can converge to approximate second order stationary points with lower sample complexity than that of first-order optimization methods, which can only find first-order stationary points. Experiments demonstrate the effectiveness of the proposed method using both synthetic and real data.

STRENGTHS:
{Strengths}
1. This paper is clearly written and provides a new algorithm to solve nonconvex bilevel hyperparameter optimization problems (with stochastic relaxation).  
2. In this setting, solving the cubic subproblem takes little time, as the cubic problem dimension is usually very small.
3. Theoretical analysis shows that the proposed cubic algorithm achieves lower sample complexity than that of first-order methods.

WEAKNESSES:
{Weakness and Questions}
1. The algorithm design is a direct application of cubic regularization, and therefore is not novel.
2. It is not clear what is the technical novelty in the convergence analysis compare to the analysis of inexact cubic regularization. The author mentions that the constructed $g$ and $B$ are not unbiased estimators of the gradient and Hessian. How does this challenge and affect the technical proof?
3. There are existing works on finding second-order stationary points of bi-level optimization, e.g., ``Efficiently Escaping Saddle Points in Bilevel Optimization''. Please discuss and compare with it.
4.  In the experiments, many curves are piece-wise flat (constant). Does that mean the hyperparameters do not change in those iterations?

QUESTIONS:
see the previous section.

LIMITATIONS:
see the previous section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a cubic regularization scheme for the outer loop optimization of bilevel optimization problems. The cubic regularization is very appealing since it can \"avoid saddle points\". Theoretically, authors extend the convergence results of the cubic regularization to inexact gradients to make it work for bilevel optimization. Empirically, authors show that their method achieves better performance on bilevel optimization tasks. Interestingly, they investigate the eigenvalues of the hessian.\n\nSTRENGTHS:\nThe idea is very interesting and tackles the very important and hard problem of finding a proper outer procedure for bilevel optimization. Avoiding saddle points seems to be a very appealing property!\n\nWEAKNESSES:\nHowever, it feels like the writing of the paper could be polished (there are unmerged commit line 264 in the supplementary material.). A lot of statements are not properly backed. I am not sure to properly understand the interest of the proposed method.\n\nQUESTIONS:\n- \"These methods [Bayesian optimization, Reinforcement learning] can easily get trapped in saddle points with poor performance\" Could you provide references for this statement? I am not sure to understand why a Bayesian optimization algorithm would be stuck in a saddle point. From what I understand from the paper, hyper-gradient methods with vanilla first-order methods can be trapped in saddle point, and cubic regularization is here to help.\n- What is the point of Lemma 3.1?\n- Definition of what is called a saddle point could be recalled (i.e. negative eigenvalues of the hessian?)\n- How does the proposed method compare to the use of l-BFGS once the hyper-gradient has been computed? (as done in [1] for instance)\n- Experimental results. In my experience, these kinds of results are usually very sensitive to the number of inner steps to solve the optimization problem. In particular, if the estimation of the inner problem is crude, the value of the hyper-gradient can be poorly estimated. Moreover, to my knowledge,  quasi-second-order methods, combined with approximated gradients, can yield poor results in practice. I did not see this crucial hyper-hyper-parameter reported.\n\n- Figure 1: could you detail how exactly this \"relative eigenvalue\" metric is computed? It seems crucial to make the point of the authors but it is not detailed. I guess this is the ratio of the eigenvalues over the maximal eigenvalue.\n\n- Section 4. Could you write exactly the maths of each bilevel problem which is solved? At least in the appendix?\n\n- I would love to see if the proposed algorithm can improve out-of-distribution generalization in meta-learning settings, like [2])\n\n[1] Deledalle, C.A., Vaiter, S., Fadili, J. and Peyré, G., 2014. Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection. SIAM Journal on Imaging Sciences,\n\n[2] Lee, K., Maji, S., Ravichandran, A. and Soatto, S., 2019. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10657-10665).\n\nTypos:\n- Figure 1 eigen value >> eigenvalue\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper studies the problem of hyper-parameter optimization in the context of machine learning. Through stochastic relaxation, the problem can be formulated as bilevel optimization, and the authors propose to use cubic regularization to solve the optimization problem. It is shown that under some regularization conditions on the loss function and hyper-parameter distribution, the algorithm efficiently converges to an approximate second-order stationary point. Furthermore, experiments are conducted to verify the effectiveness of the proposed method.

STRENGTHS:
1. The paper is well-written and quite easy to read. The authors explain the motivations, method and results in a very clear way.
2. Extensive experiments are conducted to demonstrate the wide applicability of the proposed method. All experiment settings are explained in details.

WEAKNESSES:
While the authors provide theoretical guarantees for the proposed method, it seems that these guarantees can be directly derived from known results on the convergence of cubic regularization methods. At the same time, because the problem considered is bilevel, some assumption seems less natural. For example, the Lipschitz-Hessian assumption on $J(\theta)$ is quite difficult to verify in practice, since the expression of $J$ itself contains a minimization problem. I wonder if it is possible to obtain convergence results under more 'fundamental' assumptions.

QUESTIONS:
1. In Assumption 3.5, the estimation of gradient and Hessian may be biased. How does this affect the final convergence rate?
2. How does the proposed method compared with other methods that are not based on stochastic relaxation?

LIMITATIONS:
The authors have adequately addressed the limitations and potential societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new optimization based technique for hyper-parameter tunings using Adaptive Cubic Regularized Newton method (ARC) based on stochastic relaxation. The author highlights the limitations of the existing hyper-parameter optimization algorithms.They show that their suggested method achieves better convergence guarantees as existing methods only prove convergence to first order methods where their work also achieves second order optimality conditions. The authors also run some numerical experiments to show that the algorithm outperforms existing methods in terms of faster convergence.\n\nSTRENGTHS:\n* A weakness of existing methods is clearly identified and solved.\n* The originality lies in being the first to use the second order method (ARC) in a bi-level optimization problem for hyper-parameter tuning.\n* The paper is well written and the result can serve as the base of future work in  using second order methods for hyper-parameter tunings.\n* Some minimal numerical results are shown\n\nWEAKNESSES:\n* The major contribution of this work should be explicitly stated in a section called “Our contributions”.\n* The authors use ' the curse of dimensionality' as an example of a disadvantage for existing methods; however, when explaining their proposed approach and arguing that even they have to compute the full Hessian, this will not be an expensive operation since hyper-parameters are often low dimensional. This contradicts with the disadvantage mentioned earlier.\n* The authors didn’t compare their results against the global optimal (if exists) set of hyper-parameters in their numerical experiments. \n* In Algorithm.1, the authors didn’t explain how to optimize the lower-level objective in (1)\n* In Algorithm.1, the authors didn’t explain if ARC is being solved using an iterative approach or a factorization based approach when solving the subproblems.\n* In their analysis, the authors are not taking into account the cost of solving the ARC subproblems.\n* For the synthetic data experiments, the authors didn’t use any reference and they didn’t explain why they didn't use real data for the problem of filtering useful features.\n* The details of the experiment parts are not enough so that people can reproduce the results. For example, the authors don't state what machine learning models they used.\n* The proof is a bit tricky and it has some typos and errors.\n\nQUESTIONS:\n* A typo under Table 1 should be fixed.\n* In all figures, the x-axis is confusing. What do you mean by number of trained models? Also does this mean that the AUC on the y-axis is computed by averaging across all the trained models?\n* What machine learning model you used for all the experiments. More details are needed for example for the classification did you use Decision Tree, SVM, Logistic regression …. Or in the image setup, what neural network models you used?\n* In figure 4, you can add (M) next to the upper loop iterations so that it is easier to understand the plot and what parameters are you considering in your ablation study. Also add the unit for the relative training time.\n* The details about the experiment are important as reproducibility of the work is a major concern. Even if you don’t add a new dataset or create a new machine learning model, there is still a need to reproduce your results and validate that. \n* The details of the experiments should include: what ML models are being trained, the environment, how to set up the datasets and access them, which packages are being used, ARC solver package ….\n* The code of your work also should be attached to the submission and later in your final paper, it is recommended to add the github repository link.\n* Also in your code, did you fix the seed so that people can get similar results as what you got.\n\nComments about the proof:\n* I checked the proof carefully and most of the math seems correct, however following the proof is a bit tricky and I would suggest it if you can make it more clear. Also, if you can give more explanation about the intuition behind those lemmas. \n* Some of the lemmas better to be referenced as facts since you are using the results of previous works: Lemma B.1, Lemma B.2, Lemma B.4. \n* In Lemma B.1 if you can refer to the Hessian as $\\rho$-Lipschitz.\n* Prerequisite of Theorem 3.6 should appear on its own as an equation because you are referring to it in multiple places.\n* In the proof of Preposition 3.2, there is a typo. There should be $\\grad \\log$ in the last two lines of the proof.\n* In Lemma B.3, if you can elaborate more the steps as it is not clear how you get rid of the summation term that appears in the definition of the approximate gradient $g^m$. Since the expectation is with respect to the probability $p_\\theta$ and your summation is in term of $1/k$, but the $p_\\theta$ isn’t uniform so to interchange those terms.\n* In Lemma B.3, how did you get rid of the $\\min$ term that appears in the prerequisite of Theorem 3.6 since you only used the first term in from $\\min (term_1, term_2)$.\n* In Lemma B.5, similar comment to Lemma B.3. How you get rid of the summation term that appears in the definition of the approximate hessian. Also, you only used the second term that appears in the $\\min(term_1, term_2)$ from the prerequisite of Theorem 3.6. \n* In Lemma B.5, in your first line of proof you say “By definition, we have“, but you didn’t refer to any definition. \n* In Lemma B.5, I guess you have a typo in the equality that involves $K$. I guess, this should be $b_2 ^ 2$ instead of $c_2 ^ 2$.\n* In Lemma B.5, I guess you have a typo in the inequality bounding the norm of the difference between the approximate hessian and the actual hessian. $\\|| B ^ m - \\grad J(\\theta ^ m)\\||$ should be  $\\|| B ^ m - \\grad ^ 2 J(\\theta ^ m)\\||$. Also in this lemma, you use the l-2 norm and the spectral norm interchangeably but it is not clear if the math is adjusted at all steps. Or you have some typos.\n* Lemma B.6 is well known facts about ARC, so you can refer to some citations and in that case there is no need to do the proof.\n* In Lemma B.7, when you refer to both Lemmas B.3 and B.5 you mention the inequalities from those lemmas, but without taking into consideration that each one satisfies the inequality with a different probability.\n* In lemma B.9, similar comment to the above when referring to lemmas B.3 and B.5.\n* In lemma B.9, the inequality that appears after the line “And from Lemma B.5, we will have” seems wrong since as I understand you are using Cauchy Schwarz inequality here and it is not clear why you have $\\||\\delta ^ m\\|| ^ 2$ and from where the half term appears. I guess the correct term should be $c_2 \\sqrt{\\rho \\epsilon} \\|| \\delta ^ m \\||$. Then the proof needs to be updated.\n* In Lemma B.9, also it is not clear from where you got the result in case 2 that $\\tilde c(\\delta ^ m)$  <= -\\frac{1}{96} \\sqrt {\\epsilon ^ 3 / \\rho}$.\n\nLIMITATIONS:\nExposed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}