Title: Path following algorithms for $\ell_2$-regularized $M$-estimation with approximation guarantee
Venue: NeurIPS 2023
Accepted: yes
Keywords: regularization

optimization

tuning parameter selection
Abstract: Many modern machine learning algorithms are formulated as regularized M-estimation problems, in which a regularization (tuning) parameter controls a trade-off between model fit to the training data and model complexity. To select the ``best'' tuning parameter value that achieves a good trade-off, an approximated solution path needs to be computed. In practice, this is often done through selecting a grid of tuning parameter values and solving the regularized problem at the selected grid points. However, given any desired level of accuracy, it is often not clear how to choose the grid points and also how accurately one should solve the regularized problems at the selected gird points, both of which can greatly impact the overall amount of computation. In the context of  $\ell_2$-regularized $M$-estimation problem, we propose a novel grid point selection scheme and an adaptive stopping criterion for any given optimization algorithm that produces an approximated solution path with approximation error guarantee. Theoretically, we prove that the proposed solution path can approximate the exact solution path to arbitrary level of accuracy, while saving the overall computation as much as possible. Numerical results also corroborate with our theoretical analysis.

Path following algorithms for ℓ 2 -regularized M-estimation with approximation guarantee

Abstract

Many modern machine learning algorithms are formulated as regularized Mestimation problems, in which a regularization (tuning) parameter controls a tradeoff between model fit to the training data and model complexity. To select the 'best' tuning parameter value that achieves a good trade-off, an approximated solution path needs to be computed. In practice, this is often done through selecting a grid of tuning parameter values and solving the regularized problem at the selected grid points. However, given any desired level of accuracy, it is often not clear how to choose the grid points and also how accurately one should solve the regularized problems at the selected gird points, both of which can greatly impact the overall amount of computation. In the context of ℓ 2 -regularized M-estimation problem, we propose a novel grid point selection scheme and an adaptive stopping criterion for any given optimization algorithm that produces an approximated solution path with approximation error guarantee. Theoretically, we prove that the proposed solution path can approximate the exact solution path to arbitrary level of accuracy, while saving the overall computation as much as possible. Numerical results also corroborate our theoretical analysis.

1 Introduction

Modern machine learning algorithms are often formulated as regularized M-minimization problems,

where L n ( θ ) denotes an empirical loss function, p ( θ ) denotes a regularization function, and λ is a tuning parameter that controls the trade-off between model fit and model complexity. Varying the tuning parameter leads to a collection of models, among which one of them may be chosen as the final model. This requires solving a collection of related optimization problems, whose solutions are often referred to as the solution path (We refer readers to Figure S4 and S5 in the supplementary material for some examples of solution paths).

Very often the exact solution path θ ( λ ) can not be computed, and path-following algorithms are usually used to obtain a sequence of solutions at some selected grid points to produce an approximated solution path. Existing path-following algorithms [see, e.g., Hastie et al., 2004, 2007] typically choose a set of equally-spaced grid points (often on a log-scale), and choose certain algorithm to solve the minimization problem at the selected grid points. Warm-start strategy is often used to leverage the fact that the minimizers at two consecutive grid points may be close. However, there is a paucity of literature on how to choose these grid points and how accurately one should solve the optimization

problem at the selected grid points, both of which would influence the overall computational cost and approximation accuracy.

In the context of ℓ 2 -regularized M-estimation where p ( θ ) = (1 / 2) ∥ θ ∥ 2 2 , we propose a novel grid point selection scheme and an associated stopping criterion at each grid point for any optimization method. The proposal is motivated by first proving that the approximation error of a solution path is governed by two important quantities, one depending on how finely the grid points are spaced and the other depending on how accurately one solves the regularized problem at the selected grid points. We refer these two quantities as interpolation error and optimization error. Given any optimization algorithm (that solves a single regularized problem), a stopping criterion can be constructed so that the optimization error is comparable to the interpolation error, which ensures that there is no wasteful computation at each selected grid point. Then a novel grid point selection scheme is constructed to balance the approximation errors over the entire solution path.

Theoretically, we first show that the approximation error of the proposed solution path can be upper bounded by the sum of the interpolation error and the optimization error. Given any optimization algorithm, we show that the proposed stopping criterion ensures that the optimization error is dominated by the interpolation error (see Theorem 1). We then establish a global approximation-error bound for the solution path (see Theorem 2). Optimizing this global approximation-error bound by using the proposed grid point selection scheme, we also prove that the proposed solution path can approximate the exact solution path to an arbitrary level of accuracy (see Theorem 3). Finally, we establish that the total number of grid points required for the proposed algorithm to achieve an ϵ -suboptimality is at most O ( ϵ -1 / 2 ) (see Theorem 4).

Path-following techniques have been used in many optimization algorithms. For instance, it is used in the interior point methods, which is one of most popular algorithm to solve general convex optimization problems [see, e.g., Chapter 1.3 of Nesterov and Nemirovskii, 1993]. However, the focus is often on the limit of the path rather than the entire solution path. In statistics/machine learning literature, Osborne [1992] and Osborne et al. [2000] applied the homotopy techniques to obtain piecewise linear solution paths for quantile regression and LASSO, respectively. Later Efron et al. [2004], Hastie et al. [2004], and Rosset and Zhu [2007] used similar ideas to derive the solution path for a family of regularized M-estimation problems. Subsequent developments include Friedman et al. [2007], Hoefling [2010], Arnold and Tibshirani [2016], among many others. Path following algorithms for more general types of ℓ 1 -reguarlization are also available. For example, Tibshirani and Taylor [2011] used dual path algorithm to compute the solution path of a generalized LASSO problem. In the context of order weight ℓ 1 model, Bao et al. [2019] proposed an efficient approximate solution path algorithm by using primal-dual method. These works often leverage the piecewise linearity of the solution path so that an exact path-following algorithm can be explicitly derived. For situations where the solution paths are not piecewise linear, a path-following algorithm based on Newton method was considered in Rosset [2004]. However, Rosset [2004] used a simple grid point selection scheme and only established pointwise closeness to the solution path. Subsequent research on situations where the solution paths are not piecewise linear includes the work of Garrigues and Ghaoui [2008], which introduced a homotopy algorithm tailored for solving the LASSO problem using online (sequential) observations, and the work of Qu et al. [2022], which approximates the piecewise smooth path in self-paced learning for generalized self-paced regularizer.

To the best of our knowledge, there is a paucity of research on the impact of grid points selection and how to set stopping criterion to save computation. One line of related works use a piecewise constant interpolation scheme to approximate the solution path to an arbitrary level of accuracy [see, e.g., Giesen et al., 2010, 2012, 2014, Ndiaye et al., 2019]. Most relevant to our work here are the works of Giesen et al. [2012] and Ndiaye et al. [2019]. Giesen et al. [2012] proposed a generic algorithm to compute a piecewise constant approximate solution path for a family of optimization problems parameterized concavely 2 in a single hyperparameter. They showed that the number of grid points needed to achieve an ϵ suboptimality is at most O (1 / √ ϵ ) . However, their proposed method relies on the existence of a problem-dependent oracle, which for any chosen hyperparameter produces both an ϵ -approximate solution at that hyperparameter and a concave polynomial that is used to compute the step size. Except for special examples like standard SVM, it not clear how to implement the oracle for general problems.

In view of these limitations, Ndiaye et al. [2019] considered regularized convex optimization problems and proposed general strategies to choose grid points based on primal-dual methods. Again, a piecewise constant path is constructed to approximate the solution path. However, their approach can only be applied to a special class of loss functions that are composition of a 'simple' function (whose conjugate can be analytically derived) and affine functions. Moreover, the choice of the grid points depends on some convexity and smoothness parameters associated with the loss, which either are not available except for some special class of functions or require careful tuning due to their great impact on grid points selection. Ndiaye et al. [2019] also showed that to achieve ϵ suboptimality, the number of grid points required is O (1 / d √ ϵ ) for uniformly convex loss of order d [Bauschke et al., 2011] and O (1 / √ ϵ ) for generalized self-concordant functions [Sun and Tran-Dinh, 2017]. In our paper, we show that the number of grid points needed is O (1 / √ ϵ ) for general differentiable convex loss (see Theorem 4). Compared to Ndiaye et al. [2019], we also demonstrate through simulations that our method produces better approximated solution paths under a fixed computational budget (see Section 4). Moreover, our method can also be applicable to loss functions that are possibly nonconvex (see Section 3).

We also note that Proposition 38 of Ndiaye [2018] shows that under some regularity conditions ϵ -suboptimality along the entire solution path can be achieved by using a piecewise linear interploation. However, it not clear how these conditions can be used to construct an algorithm with approximation guarantee. Our proposed algorithm, on the other hand, provides an easily implementable grid point selection scheme and an associated stopping criterion at each grid point, which together ensures that ϵ -suboptimality can be attained through a piecewise linear interploation.

More recently, in the context of ℓ 2 -regularized convex optimization problems, Zhu and Liu [2021] proposed two path-following algorithms by using Newton method and gradient descent method as the basis algorithm seperately, and discuss the grid point selection scheme for each algorithm based on the piecewise linear interpolation. Our paper differs from Zhu and Liu [2021] in twofold. First, the grid-point selection scheme in Zhu and Liu [2021] depends on the basis algorithm. By contrast, in our paper any basis algorithm can be applied at any chosen grid point as long as the stopping criterion is satisfied by the basis algorithm. Next, the path-following algorithms by Zhu and Liu [2021] can only be applied to convex optimization problems, while in this article we also consider loss functions that are possibly nonconvex (see Section 3).

The rest of the paper is organized as follows. Section 2 introduces the proposed path following algorithm and establish its global approximation-error bound. Section 3 considers an extension to nonconvex empirical loss. In Section 4, we compare the the proposed scheme to a standard path-following scheme through a simulated study using ridge regression and ℓ 2 -regularized logistic regression. We close the article with some remarks in Section 5. All proofs are included in the supplementary material.

2 Path following algorithms

We consider an ℓ 2 -regularized M-estimation problem. More specifically, denote by L n ( θ ) some empirical loss function, where θ ∈ R p denotes a parameter vector. We consider the solution path of an ℓ 2 -regularized M-estimation problem:

Note that as the tuning parameter t varies from 0 to ∞ , the solution θ ( t ) varies from 0 to a minimizer of L n ( θ ) . We also remark that the choice of e t -1 is not essential. In fact, it can be replaced by any function C ( t ) that is a strictly increasing and differentiable function, with C (0) = 0 and lim t →∞ C ( t ) = ∞ . The solution path produced should be independent of C ( t ) . Moreover, we can show that the ℓ 2 norm of the solutions is an increasing function of t , and the solution θ ( t ) converges to the minimum ℓ 2 norm minimizer of L n ( θ ) (if it is finite) as t goes to infinity (see Corollary S1 in the supplementary material).

Suppose that the goal is to approximate the solution path θ ( t ) over a given interval [0 , t max ) for some t max ∈ (0 , ∞ ] , where we allow t max = ∞ . Given a set of grid points 0 < t 1 < · · · < t N < ∞ , and approximated solutions { θ k } N k =1 at these grid points, we construct an approximated solution path over [0 , t max ) through linear interpolation. More specifically, we define a piecewise linear solution

path ˜ θ ( t ) as follows

where t 0 = 0 and θ 0 = 0 . This defines an approximated solution path ˜ θ ( t ) for any t ∈ [0 , t max ] , which interpolates the approximated solutions at the selected grid points. In view of this definition, we may also assume that t N -1 ≤ t max , because we do not need ˜ θ ( t ) over t ∈ [ t N -1 , t N ] if t N -1 > t max .

To assess how well ˜ θ ( t ) approximates θ ( t ) , we use the function-value suboptimality of the solution paths defined by sup 0 ≤ t ≤ t max { f t ( ˜ θ ( t )) -f t ( θ ( t )) } , where f t ( θ ) := (1 -e -t ) L n ( θ ) + e -t ( ∥ θ ∥ 2 2 / 2) is a scaled version of the objective function in (2). In what follows, we call sup 0 ≤ t ≤ t max { f t ( ˜ θ ( t )) -f t ( θ ( t )) } the global approximation error of ˜ θ ( t ) , and sup t k ≤ t ≤ t k +1 { f t ( ˜ θ ( t )) -f t ( θ ( t )) } the local approximation error of ˜ θ ( t ) over [ t k , t k +1 ] .

Intuitively, how well ˜ θ ( t ) approximates θ ( t ) depends on how finely spaced the grid points are and the quality of the solutions at the selected grid points (i.e., θ k 's). Our first main result relates the local approximation error of ˜ θ over [ t k , t k +1 ] to the choice of the grid points and the accuracy of the approximated solutions at the selected grid points measured by the size of the gradients ∥∇ f t k ( θ k ) ∥ 2 , where we assume that L n ( θ ) is differentiable.

Theorem 1. Assume that L n ( θ ) is a differentiable convex function. Let g k := ∇ f t k ( θ k ) = (1 -e -t k ) ∇ L n ( θ k ) + e -t k θ k denote the scaled gradient at θ k . For any 0 < t 1 < t 2 < · · · < t N < ∞ , we have that

for any k = 1 , . . . , N -1 . If we further assume that ∥ θ ( t max ) ∥ 2 < ∞ , then we have that

when t N < t max .

The proof of Theorem 1 starts with relating the local approximation error to the suboptimality of θ k and θ k +1 at some t ∈ [ t k , t k +1 ] . The suboptimality can then further bounded by the square norm of the grandient leveraging the fact that the objective function is e -t -strongly convex. Finally, a triangular inequality is used to bound the norm of the grandient by quantities that depend on ∥ g k ∥ 2 and ∥ θ k ∥ 2 .

We can see from Theorem 1 that the upper bounds for the local approximation error in (4)-(6) consist of two terms, with the first one (depending on g k ) being algorithm dependent and the other term stemming from interpolation. We call them optimization error and interpolation error , respectively. Note that the optimization error is roughly of order e t k ∥ g k ∥ 2 2 , which again depends on how accurate the solutions θ k are at the selected grid points. And the interpolation error is essentially independent of the quality of the solutions θ k 's, as the interpolation error only depends on the spacing of the grid points and the norm of the solutions along the solution path (typically ∥ θ k ∥ 2 = O ( ∥ θ ( t k ) ∥ 2 ) , c.f., Lemma S2 in the supplementary material). In other words, the interpolation error is irreducible once the grid points are chosen, while the optimization error does depend on the algorithm and it can be pushed to be arbitrarily small if we run the algorithm long enough at each grid point. In this sense, if the goal is to minimize the local approximation error, then both the grid points and optimization algorithm should be designed carefully to strike a balance between these two errors to minimize the overall computations. For instance, it would be wasteful to have the optimization error smaller than the interpolation error, because the additional computation would not improve the overall approximation error (at least in order).

Motivated by this observation and the local approximation-error bounds in Theorem 1, we propose a novel stopping criterion scheme at the selected grid points for any general optimization algorithm. In particular, for any given algorithm that takes an initializer and solves the problem at grid point t k +1 , we can run the algorithm initialized at θ k until the optimization error is smaller than the interpolation error for θ k +1 , that is

where the LHS is the optimization error and the RHS is the interpolation error in the bounds in Theorem 1. This condition can be further simplified, under which the upper bounds in Theorem 1 can also be further simplified and combined so that we can bound the global approximation error of ˜ θ ( t ) . This is established in the following theorem.

Theorem 2. Assume that L n ( θ ) is a differentiable and convex. Suppose that 2 -1 α k ≤ α k +1 ≤ 2 α k , α k ≤ α max for some α max ≤ ln(2) , and

for some C 0 ≤ 1 / 4 . Denote

Then

when t N < t max and ∥ θ ( t max ) ∥ 2 < ∞ , and

when t N -1 ≤ t max ≤ t N for some N ≥ 1 .

The above result provides a practical approach to verify whether the optimization error is dominated by the interpolation error by checking (8). In other words, (8) is a sufficient condition for (7) and it can be used as a stopping criterion at each grid point t k for any algorithm that computes a solution at t k . We also remark that, using the above bounds, similar approximation-error bounds for ∥ ˜ θ ( t ) -θ ( t ) ∥ 2 can be derived as well (see Corollary S2 in the supplementary material).

Another important implication of Theorem 2 is that the established global approximation-error bound provides a principled approach to choose the grid points so that for any ϵ > 0 and t max ∈ (0 , ∞ ] we have

for the solution path generated by an algorithm, and at the same time it tries to maximize the speed at which the algorithm explores the solution path from 0 to t max . In particular, to ensure (11), we propose to use the following step sizes

and

Algorithm 1 A general path following algorithm.

Input : ϵ > 0 , C 0 ≤ 1 / 4 , c 1 ≥ 1 , c 2 > 0 , 0 < α max ≤ 5 -1 and t max ∈ (0 , ∞ ] .

Output : grid points { t k } N k =1 and an approximated solution path ˜ θ ( t ) .

1: Initialize: k = 1 .

2: Compute α 1 using (12). Starting from 0 , iteratively calculate θ 1 by minimizing f t 1 ( θ ) until (8) is satisfied for k = 1 .

3: while (14) is not satisfied, do

4: Compute α k +1 using (13), update t k +1 = t k + α k +1 .

5: Starting from θ k , iteratively compute θ k +1 by minimizing f t k +1 ( θ ) until (8) is satisfied.

6: Update k = k +1 .

7: Interpolation: construct a solution path ˜ θ ( t ) through linear interpolation of { θ k } N k =1 using (3).

and we terminate the algorithm at t N when

where c 1 ≥ 1 , c 2 > 0 , and 0 < α max ≤ 1 / 5 are absolute constants. Note that the above step sizes and termination criterion are chosen to ensure that the upper bounds in (9) and (10) are at most O ( ϵ ) . Formally, we show that, given any ϵ > 0 , the above scheme achieves ϵ -suboptimality (11) for ˜ θ ( t ) (up to a multiplicative constant).

Theorem 3. Suppose that L n ( θ ) is differentiable and convex. For any ϵ > 0 and t max ∈ (0 , ∞ ] , assume that either t max < ∞ or ∥ θ ( t max ) ∥ 2 < ∞ . Then, using the step sizes and the termination criterion specified above in (12) , (13) , and (14) , any algorithm that produces iterates satisfying (8) for any k ≥ 1 terminates after finite number of iterations, and when terminated, the solution path ˜ θ ( t ) satisfies

Note that when ϵ is small enough, we have that α 1 = ln (1 + √ ϵ/ ∥∇ L n ( 0 ) ∥ 2 ) , which together with (15) implies that

As a result, the approximation error for the solution path is essentially controlled by the initial step size α 1 . Smaller α 1 leads to better approximation. However, smaller α 1 also leads to a slower exploration of the solution path and a more stringent stopping criterion at individual grid points (c.f. (8)). As such, the initial step size controls the trade-off between computational cost and accuracy. Moreover, given α 1 , we can see from (13) that how fast the algorithm can explore the solution path depends largely on ∥ θ k ∥ 2 . In particular, if ∥ θ k ∥ 2 is bounded (e.g., when θ ⋆ is finite), then the first term in the min function of (13) increases as t k increases. Therefore, aggressive step sizes can be taken in this case until it reaches O (1) , which will likely lead to a fast exploration of the solution path. On the other hand, if ∥ θ k ∥ 2 grows quite quickly to infinity as k increases, then the first term in the min function may go to zero as k →∞ (say, when ∥ θ k ∥ 2 /e t k / 2 →∞ ). This means that the step sizes need to decrease to zero eventually, leading to a slower exploration of the solution path.

We summarize the proposed scheme in Algorithm 1. Again, we emphasize that the above scheme can be applied to any optimization algorithm. Later, we will empirically investigate its performance using Newton method and gradient descent method.

Lastly, we establish an upper bound on the total number of grid points required for Algorithm 1 to achieve ϵ -suboptimality.

Theorem 4. Under the assumptions in Theorem 3, the total number of grid points required for Algorithm 1 to achieve an ϵ -suboptimality (15) is at most O ( ϵ -1 / 2 ) .

3 Extensions to nonconvex loss functions

So far we assume that the empirical loss is convex. Next, we consider a generalization, where we do not assume convexity for L n ( θ ) . Instead, we assume that there exists λ ( t ) > 0 so that

for any t ≥ 0 and θ ∈ dom ( L n ) . In the literature, the condition (16) was often referred to as the Polyak-Łojasiewicz (PL) inequality. Polyak [1964] proved the linear rate of convergence for gradient descent method under this condition. Also see Karimi et al. [2016] for discussions of more recent development. Note that if L n ( θ ) is convex, then (16) holds for λ ( t ) = e -t since f t ( θ ) is e -t -strongly convex. In general, however, L n ( θ ) does not need to be convex for (16) to hold.

For some technical reasons, we need to consider slightly different interpolation scheme

where t 0 = 0 and θ 0 = 0 . As we can see that the approximated solution path is piecewise constant. We use the following termination criterion at each t k

We also define a slightly new step size scheme as follows:

and

where λ k = inf t k ≤ t ≤ t k +1 λ ( t ) , k ≥ 0 . Finally, the stopping criterion for the path-following algorithm is set to be

where c 3 > 0 is an absolute constant.

Moreover, we need to make some additional assumptions on the loss function.

Assumption (A1). Assume that L n ( θ ) is differentiable and inf θ L n ( θ ) > -∞ . Moreover, assume that there exists a positive decreasing function g ( · ) such that λ ( t ) ≥ g ( t ) > 0 for any 0 ≤ t < ∞ .

Theorem 5. Suppose that Assumption (A1) holds. For any ϵ > 0 and t max ∈ (0 , ∞ ] , assume that either t max < ∞ or sup 0 ≤ t ≤ t max ∥ θ ( t ) ∥ 2 < ∞ . Then, using step sizes and termination criterion specified in (19) , (20) , and (21) , any algorithm that produces gradients satisfying (18) terminates after finite number of iterations, and when terminated, the solution path ˜ θ ( t ) defined in (17) satisfies

The above result can be viewed as a generalization of Theorem 3, because when L n ( θ ) is convex, we could set λ ( t ) = exp( -t ) so that both condition (16) and Assumption (A1) are satisfied.

4 Numerical studies

In this section, we study the operating characteristics of Newton method, gradient descent method and a mixed method in ridge regression and ℓ 2 -regularized logistic regression, using both the simulated datasets and a real data example. Here the mixed method applies gradient descent method to minimize

f t k ( θ ) at the beginning and then switch to Newton method when the number of gradient steps n k exceeds min { n, p } . To illustrate the advantages of the proposed grid point selection scheme and stopping criterion, we use a standard implementation of a typical path-following algorithm as a baseline. In particular, the grid points are equally spaced (on a log-scale) for both examples. For ℓ 2 logistic regression, Newton method is used as the basis algorithm with a fixed stopping criterion ∥∇ f t k ( θ k ) ∥ 2 < 10 -5 , while for ridge regression the exact solution can be computed. We refer to this method as the baseline method.

We also compare the proposed methods against the method of Ndiaye et al. [2019]. In particular, we compare our grid points selection scheme (13) with the so-called adaptive unilateral scheme proposed in Ndiaye et al. [2019], where Newton method is used as the basis algorithm at each grid point. Different from the aforementioned methods, adaptive unilateral scheme constructs piecewise constant solution path ˜ θ ( λ ) over chosen grid points to approximate the solution path θ ( λ ) = arg min θ ∈ R p P λ ( θ ) , where P λ ( θ ) = L n ( θ )+( λ/ 2) ·∥ θ ∥ 2 2 , such that for all λ ∈ ( λ min , λ max )

We call this Ndiaye's method throughout this section.

All methods considered in the numerical studies are implemented in R using Rcpp package [Eddelbuettel et al., 2011, Eddelbuettel, 2013].

Example 1: Ridge regression. This example considers ridge regression over a simulated dataset. In this case, the empirical loss function is L n ( θ ) = ∥ Y -Xθ ∥ 2 2 / (2 n ) , where X ∈ R n × p and Y ∈ R n denote the design matrix and the response vector. In the simulation, the data ( X,Y ) are generated from the usual linear regression model Y = Xθ ⋆ + ˜ ϵ , where ˜ ϵ ∼ N (0 , I n × n ) , θ ⋆ = (1 / √ p, . . . , 1 / √ p ) ⊤ , and rows of X are IID samples from N (0 , I p × p ) . Throughout this example, we consider n = 1000 and p = 500 , 10000 ,

Example 2: ℓ 2 -regularized logistic regression. This example considers ℓ 2 -regularized logistic regression over a simulated dataset. Let X ∈ R n and Y ∈ { +1 , -1 } n denote the design matrix and the binary response vector. The empirical loss function for logistic regression is L n ( θ ) = n -1 ∑ n i =1 log(1 + exp( -Y i X ⊤ i θ )) . We simulate the data ( X,Y ) from a linear discriminant analysis (LDA) model. More specifically, we sample the components of Y independently from a Bernoulli distribution with P ( Y i = +1) = 1 / 2 ; i = 1 , 2 , . . . , n . Conditioned on Y i , X i 's are then independently drawn from N ( Y i µ, σ 2 I p × p ) , where µ ∈ R p and σ 2 > 0 . Note that µ and σ 2 controls the Bayes risk, which is Φ( -∥ µ ∥ 2 /σ ) under the 0/1 loss, where Φ( · ) is the cumulative distribution function of a standard normal random variable. Here we choose µ = (1 / √ p, . . . , 1 / √ p ) and σ 2 = 1 so that the Bayes risk is Φ( -1) , which is approximately 15% . Similar to Example 1, two different problem dimension are considered: n = 1000 and p = 500 , 10000 .

Example 3: Real data example. This example fits an ℓ 2 -regularized logistic regression using the a9a dataset from LIBSVM [Chang and Lin, 2011]. This dataset provides information about whether an adult's annual income surpasses $50 , 000 , along with some specific characteristics of the given adult. It contains a total of 32561 examples with 123 features and 1 binary response variable.

For all three examples, we use the global approximation error sup 0 ≤ t ≤ t max { f t ( ˜ θ ( t )) -f t ( θ ( t )) } as an evaluation metric, where ˜ θ ( t ) is the linear interpolation of the iterates θ k generated by each method. Throughout, we let t max = 10 . Since this metric can not be calculated exactly, we approximate the global approximation error by max 1 ≤ i ≤ 100 { f s i ( ˜ θ ( s i )) -f s i ( θ ( s i )) } , where s 1 , . . . , s 100 are uniformly sampled from (0 , t max ) . Here we compute the solutions θ ( s i ) using the CVX solver [Grant and Boyd, 2014, 2008] for ℓ 2 -regularized logistic regression, while calculating θ ( s i ) explicitly for ridge regression.

We compare Newton method, gradient descent method and their mixed version against the baseline method and Ndiaye's method. To make a fair comparison with Ndiaye's method, we set λ min = 1 / ( e 10 -1) to match t max = 10 in log scale and choose λ max = 100 .

Figure 1 presents runtime versus global approximation error for Example 1 and 2 based on 100 replications, as we vary ϵ in (12) for our proposed method and (23) for Ndiaye's method. For the baseline method, we vary the spacing between consecutive grid points instead. Clearly the proposed Newton method outperforms the baseline method and Ndiaye's method in both low dimension and high dimension scenarios, which empirically confirms that the proposed grid point selection scheme

(c.f. (13)) is more efficient than both the standard equally-spaced grid point scheme and the adaptive unilateral scheme proposed by Ndiaye et al. [2019]. Moreover, gradient descent method performs similarly compared with the mixed method in both setups of Example 1, which suggests that switching does not happen much in those cases. However, in both problem dimensions of Example 2, we do see a difference between the mixed method and gradient descent, which is due to the fact that the switching to the Newton update does happen and it can speed up the computation by design. Lastly, as one would expect, Newton method is more efficient than both the gradient descent method and the mixed method when the problem dimension is small, but less so when problem dimension is large.

Figure 2 shows the runtime versus global approximation error tradeoff for our real data example, as we vary ϵ for our proposed methods and Ndiaye's method, and change grid point spacing for the baseline method. Again, the Newton method is more efficient than both the baseline method and the Ndiaye's method, which demonstrates the advantage of our proposed grid point selection scheme over the standard equally-spaced grid point scheme and the scheme by Ndiaye et al. [2019].

We also plot the number of iterations at each grid point for Newton method and gradient descent method in Figure S1-S3 of the supplementary material. Interestingly, for ridge regression (c.f., Figure S1), the number of iterations by gradient method first increases and then stays flat as t k grows. Newton method, however, only takes one iteration at each grid point. Moreover, the level of approximation (i.e., ϵ ) seems to have no impact on the number of iterations at each grid point, which is highly desirable. For the ℓ 2 -regularized logistic regression (c.f., Figure S2 and S3), the number of iterations needed increases as t k increases for gradient descent method, whereas the Newton method always requires just a constant number of iterations at each t k . Again, the level of approximation (i.e., ϵ ) does not seem to influence the number of iterations much.

-8

-6

-4

-2

0.10

1.00

10.00

Runtime ( in seconds )

Suboptimality

Baseline method

Gradient descent

Mixed method

Ndiaye

Newton

p=500

-6

-4

-2

10.00

100.00

1 000.00

Runtime ( in seconds )

Suboptimality

Baseline method

Gradient descent

Mixed method

Ndiaye

Newton

p=10000

-8

-6

-4

1.00

10.00

100.00

Runtime ( in seconds )

Suboptimality

Baseline method

Gradient descent

Mixed method

Ndiaye

Newton

p=500

-6

-4

-2

1.00

10.00

100.00

1 000.00

10 000.00

Runtime ( in seconds )

Suboptimality

Baseline method

Gradient descent

Mixed method

Ndiaye

Newton

p=10000

5 Discussion

In this article, we proposed a novel grid point selection scheme and stopping criterion for any general path-following algorithms. A simple solution path was constructed through linear interpolation

-10

-8

-6

-4

-2

10.00

100.00

1 000.00

Runtime ( in seconds )

Suboptimality

Baseline method

Gradient descent

Mixed method

Ndiaye

Newton

and theoretical approximation-error bound was derived. A generalization to a class of nonconvex empirical loss was also considered. We have not touched on the overall computational complexity analysis for commonly used optimization algorithms. Based on some preliminary analysis, we conjecture that for gradient descent method and Newton method, the number of iterations required are O ( ϵ -1 ln( ϵ -1 )) and O ( ϵ -1 / 2 ) , respectively. Moreover, lower bound results are also worth investigating, that is, what are the minimum number of gradient or Newton steps needed to achieve an ϵ -suboptimality for the entire solution path? Moreover, our current analysis can be readily extended to general differentiable regularization functions. A more challenging future direction is to investigate the case where the empirical loss function or the regularizer are not differentiable.

Acknowledgments and Disclosure of Funding

The authors are supported in part by the US National Science Foundation (NSF) under grant DMS-1712580, DMS-17-21445 and DMS-20-15490. The authors thank the reviewers for valuable comments and suggestions that improved the article.

References

B. Arnold and R. J. Tibshirani. Efficient implementations of the generalized lasso dual path algorithm. Journal of Computational and Graphical Statistics , 25(1):1-27, 2016. doi: 10.1080/10618600.2015.1008638.

Bao, B. Gu, and H. Huang. Efficient approximate solution path algorithm for order weight l_1-norm with accuracy guarantee. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 958-963. IEEE, 2019.

H. Bauschke, P. L. Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces , volume 408. Springer, 2011.

C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

Eddelbuettel. Seamless R and C++ integration with Rcpp . Springer, 2013.

Eddelbuettel, R. François, J. Allaire, K. Ushey, Q. Kou, N. Russel, J. Chambers, and D. Bates. Rcpp: Seamless r and c++ integration. Journal of Statistical Software , 40(8):1-18, 2011.

Efron, T. Hastie, I. Johnstone, and R. Tishirani. Least angle regression. The Annals of Statistics , 32(2):407 499, 2004.

Friedman, T. Hastie, H. Hofling, and R. Tibshirani. Pathwise coordinate optimization. The Annals of Applied Statistics , 1(2):302-332, 2007.

Garrigues and L. Ghaoui. An homotopy algorithm for the lasso with online observations. Advances in neural information processing systems , 21, 2008.

Giesen, M. Jaggi, and S. Laue. Approximating parameterized convex optimization problems. In European Symposium on Algorithms , pages 524-535. Springer, 2010.

Giesen, J. Müller, S. Laue, and S. Swiercy. Approximating concavely parameterized optimization problems. In Advances in neural information processing systems , pages 2105-2113, 2012.

Giesen, S. Laue, and P. Wieschollek. Robust and efficient kernel hyperparameter paths with guarantees. In International Conference on Machine Learning , pages 1296-1304, 2014.

Grant and S. Boyd. Cvx: Matlab software for disciplined convex programming, version 2.1, 2014.

C. Grant and S. P. Boyd. Graph implementations for nonsmooth convex programs. In Recent advances in learning and control , pages 95-110. Springer, 2008.

Hastie, S. Rosset, R. Tishirani, and J. Zhu. The entire regularization path for the support vector machine. Journal of Machine Learning Research , 5:1391 - 1415, 2004.

Hastie, J. Taylor, R. Tibshirani, G. Walther, et al. Forward stagewise regression and the monotone lasso. Electronic Journal of Statistics , 1:1-29, 2007.

Hoefling. A path algorithm for the fused lasso signal approximator. Journal of Computational and Graphical Statistics , 19(4):984-1006, 2010. doi: 10.1198/jcgs.2010.09208.

Karimi, J. Nutini, and M. Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 795-811. Springer, 2016.

Ndiaye. Safe optimization algorithms for variable selection and hyperparameter tuning . PhD thesis, Université Paris-Saclay (ComUE), 2018.

Ndiaye, T. Le, O. Fercoq, J. Salmon, and I. Takeuchi. Safe grid search with optimal complexity. In International Conference on Machine Learning , pages 4771-4780, 2019.

E. Nesterov and A. S. Nemirovskii. Interior Point Polynomial Methods in Convex Programming: Theory and Algorithms . SIAM Publications, 1993.

Osborne. An effective method for computing regression quantiles. IMA Journal of Numerical Analysis , 12: 151 - 166, 1992.

Osborne, B. Presnell, and B. Turlach. A new approach to variable selection in least squares problems. IMA Journal of Numerical Analysis , 20(3):389 - 403, 2000.

T. Polyak. Gradient methods for solving equations and inequalities. USSR Computational Mathematics and Mathematical Physics , 4(6):17-32, 1964.

Qu, D. Li, X. Zhao, and B. Gu. Gaga: Deciphering age-path of generalized self-paced regularizer. Advances in Neural Information Processing Systems , 35:32025-32038, 2022.

Rosset. Following curved regularized optimization solution paths. In Advances in Neural Information Processing Systems , 2004.

Rosset and J. Zhu. Piecewise linear regularized solution paths. Ann. Statist. , 35(3):1012-1030, 2007. doi: 10.1214/009053606000001370.

Sun and Q. Tran-Dinh. Generalized self-concordant functions: a recipe for newton-type methods. Mathematical Programming , pages 1-69, 2017.

J. Tibshirani and J. Taylor. The solution path of the generalized lasso. The Annals of Statistics , pages 1335-1371, 2011.

Zhu and R. Liu. An algorithmic view of ℓ 2 -regularization and some path-following algorithms. The Journal of Machine Learning Research , 22(1):6123-6184, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA novel grid point selection scheme and stopping criterion for any general path-following algorithms are proposed in this paper. However, the presentation of this paper is poor, and the contributions are not clear.\n\nSTRENGTHS:\nA novel grid point selection scheme and stopping criterion for any general path-following algorithms are proposed in this paper.\n\nThis work provides some interesting theoretical results. For example, this work theoretically shows that the approximation error of the proposed solution path can be upper bounded by the sum of the interpolation and optimization error, and establish a global approximation-error bound for the solution path.\n\nWEAKNESSES:\nThis work provides some interesting theoretical results, however, I am dissatisfied with the writing and organization of this paper.\n\n1. The writing of this paper is poor. In order to enhance the readability, I suggest the author discuss the definition of \"regularized M-minimization problems\" in the introduction, maybe with some formulation.\n\n2. The contributions of this paper are not clear. This work shows some interesting theoretical results,  I suggest the author clearly clarify the contributions in the Introduction. \n\n3. The experimental results are all put in the Appendix in the current manuscript. I am unsure why the authors did not put the results in the main text, especially considering that there are many empty spaces on the ninth page of the main text.\n\n4. The format of the appendix may not comply with the requirements of NeurIPS. I recommend that the authors make corrections accordingly.\n\n5. To further improve the readability of this paper, I suggest the author add a \"Related Work\" section to introduce some works related to this work. And the organization of this work is poor, I suggest the author carefully modify the whole paper.\n\nQUESTIONS:\n1. The writing of this paper needs improvement. It is suggested to enhance readability by discussing the definition of \"regularized M-minimization problems\" in the introduction, possibly with some formulation.\n\n2. Lack of clarity regarding the contributions of this paper. Although this work presents some interesting theoretical results, it is recommended that the author explicitly clarify the contributions in the Introduction.\n\n3. Placement of experimental results in the Appendix. Uncertain why the authors chose not to include the results in the main text, especially considering the available space on the ninth page.\n\n4. Non-compliance of the appendix format with NeurIPS requirements. It is advised that the authors make the necessary corrections to adhere to the guidelines.\n\n5. Readability and organization concerns. To enhance the paper's readability, it is suggested to add a \"Related Work\" section to introduce works relevant to this study. Furthermore, the entire paper's structure requires careful modification.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes a new path following algorithm for 2-regularized M-estimation with an approximation guarantee. The algorithm includes a grid point selection scheme and an adaptive stopping criterion to optimize the trade-off between model fit and complexity in machine learning algorithms. The paper also provides a comparison of the proposed scheme to a standard path-following scheme through a simulated study using ridge regression and L2-regularized logistic regression.

STRENGTHS:
1. This paper proposes a new path following algorithm for 2-regularized M-estimation with an approximation guarantee.

2. This work is generally technically sound with rigorous statistical analysis in the work. 

3. The article is well written and mathematical derivations are easy to follow.

WEAKNESSES:
1: The main body does not include the experimental results.

2: The experiments were solely conducted on simulation datasets, and evaluating the method on public datasets could enhance the results.

3: The experiments do not include other baselines on solution path algorithms.

4: Can the proposed method be extended to other regularized M-estimation techniques, such as Lasso or Group Lasso?

5: It is important to discuss relevant existing work on solution path algorithms, including notable studies such as [1], as well as approximate solution path algorithms, like [2].

[1] Ryan Joseph Tibshirani, Jonathan E Taylor, Emmanuel Jean Candes, and Trevor Hastie, "The solution path of the generalized lasso," The Annals of Statistics, 2011.

[2] Runxue Bao, Bin Gu, and Heng Huang, "Efficient Approximate Solution Path Algorithm for Ordered Weighted L_1-Norm with Accuracy Guarantee," ICDM 2019.

QUESTIONS:
see weaknesses

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers this problem of the ($\\ell_2$-regularized) $M$-estimation problem of computing solutions to the regularized objective $(e^t – 1) L(\\Theta) + (1/2) \\| \\Theta \\|_2^2$ for all $t$, denoted $\\theta(t)$. The paper proposes a method for approximating theta(t) for carefully chosen values of t, such that interpolating between these points approximately solves the problem. When $L$ is convex and differentiable the paper proves that (under certain bounds) $1/\\sqrt{\\epsilon}$ values of t suffice to obtain a curve of $\\epsilon$ approximate solution to the regularized objective. The paper discusses how this improves upon prior bounds, how the scheme has desirable properties for implementation, and provides experiments implementing the algorithm. Further, the paper discusses extensions of their work to non-convex optimization.\n\nSTRENGTHS:\nThe $M$-estimation problem that the paper considers is a natural and feels fundamental. This paper provides a seemingly useful set of theoretical (algorithms and bounds) and empirical (experiments) on this fundamental problem. Further, the paper argues how they improve upon prior work on this problem. Additionally, the paper is fairly well-written, suggests an interesting direction or future work, and could invite further study of the problem.\n\nWEAKNESSES:\nThe problem considered and the approach for it feels quite natural. This is not necessarily an issue except that it isn’t clear from the writing if the results are not what one might obtain by following the first approaches for the problem one might think of. For example, if one simply computes the derivative of the value of the regularized objective minimizer with respect to t and then bounds the terms, does that more or less directly suggest the approach of the paper? More broadly, I think the paper could benefit by discussing their proof strategy and any obstacles that had to be overcome in achieving their result.  \n\nAdditionally, I thought it would be helpful if the paper was a little clearer about what assumptions need to be made to achieve, e.g., the claimed $O(1/\\sqrt{\\epsilon})$. What other quantities does this rate depend on exactly? As discussed briefly in the following “questions” section I think it would be helpful to discuss all the quantities that the proposed method depends on.   \n\nFinally, the $M$-estimation problem seems like one that could be studied in part under different names in a number of different areas. Ultimately, it is in some sense asking to approximately follow a certain induced curve which seems like something that could have arisen in a number of contexts. Consequently, more literature review or discussion to raise confidence in the novelty of the proposed method would be helpful.\n\nQUESTIONS:\nMy main question are those raised in the “Weaknesses” category. Additionally, below are some more detailed suggestions, questions, and comments:\n\n* Line 16: “Corroborate with our theoretical” --> “Corroborate our”\n* Line 62 – 99: It would be helpful to know what exactly is hidden in the $O(\\cdot)$ notation.\n* Line 112: why write the $n$ in $L_n$? Does it arise anywhere else in the paper?\n* Theorem 1: some more discussion of the proof strategy would be nice.\n\nLIMITATIONS:\nThe paper doesn’t discuss weaknesses and limitations, but the paper is primarily bout the theoretical and empirical analysis of an algorithm for solving a sequence of regularized optimization problems. Consequently, the societal impact is unclear. Further, the weaknesses and questions discussed reflect limitations for which further discussion might be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission 9422 proposed a novel selection scheme for grid search and stopping criterion for a more general path-following algorithms, while not trying to compute/derive the whole solution spectrum. The analysis of the authors indicates that under certain assumptions, their proposed approximate path-tracking algorithm (with linear interpolation) can approximate the exact solution path and provide theoretical boundaries. The theoretical conclusions have been verified to some extent through some simple numerical simulations.\n\nSTRENGTHS:\n- Solution path (hyperparameter optimization) is an interesting and important topic in both theory and application communities of ML.\n\n- The paper is well written and it clearly state its contributions, notation and results. The most part of their derivations are easy to follow (for me).\n\nWEAKNESSES:\n- The numerical simulations of this work is too weak and might not be convincing. \n  - the scale of the experiment is too small. The authors are suggested to consider more objectives since their analysis can be readily extended to general differentiable functions\n  - lack of experiments on real-world benchmark datasets, e.g., UCI, libsvm\n  - the code (currently) has not been submitted and lacks *sufficient* details for reproducibility\n\n- Lack of some existing research work on the situation where the exact paths are not piecewise linear, e.g.,\n\n  - [1] Pierre Garrigues, and Laurent Ghaoui. *An homotopy algorithm for the Lasso with online observations.*  Advances in neural information processing systems 21 (2008).\n\n  - [2] Xingyu Qu, Diyang Li, Xiaohan Zhao, and Bin Gu. *GAGA: Deciphering Age-path of Generalized Self-paced Regularizer.* Advances in Neural Information Processing Systems 35 (2022).\n\n  - ...\n\nI also suggest that the authors further highlight the intrinsic differences from recent work and the technical challenges faced by their settings.\n\nQUESTIONS:\n- Why not use the vanilla $C(t):=t$ to replace the current $C(t):=e^t-1$ in your analysis?\n\n- I suggest the authors to plot the *path visualization* of $\\tilde{\\theta}(t)$ as well as $\\theta(t)$, to help the readers that may not familiar with the solution path area. also we can observe how the computed path (via algo. 1) approximates the ground truth.\n\nLIMITATIONS:\n- some technical limitations of the current method were mentioned in some parts of the paper, but there was no detailed/thorough discussion.\n\n- there is no need to discuss potential negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}