SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Initialize:  $k = 1$ .
2: Compute  $\alpha_{1}$  using (12). Starting from  $\mathbf{0}$ , iteratively calculate  $\theta_{1}$  by minimizing  $f_{t_1}(\theta)$  until (8) is satisfied for  $k = 1$ .
3: while (14) is not satisfied, do
4: Compute  $\alpha_{k + 1}$  using (13), update  $t_{k + 1} = t_k + \alpha_{k + 1}$ .
5: Starting from  $\theta_{k}$ , iteratively compute  $\theta_{k + 1}$  by minimizing  $f_{t_{k + 1}}(\theta)$  until (8) is satisfied.
6: Update  $k = k + 1$ .
7: Interpolation: construct a solution path  $\tilde{\theta}(t)$  through linear interpolation of  $\{\theta_k\}_{k=1}^N$  using (3).
and we terminate the algorithm at  $t_N$  when
$$
\frac {c _ {2} \left(1 - e ^ {- \max  \left(\alpha_ {N} , t _ {\max } - t _ {N}\right)}\right)}{e ^ {t _ {N}} - 1} \leq \epsilon \text {o r} t _ {N} \geq t _ {\max }, \tag {14}
$$
where  $c_{1} \geq 1$ ,  $c_{2} > 0$ , and  $0 < \alpha_{\max} \leq 1/5$  are absolute constants. Note that the above step sizes and termination criterion are chosen to ensure that the upper bounds in (9) and (10) are at most  $\mathcal{O}(\epsilon)$ . Formally, we show that, given any  $\epsilon > 0$ , the above scheme achieves  $\epsilon$ -suboptimality (11) for  $\tilde{\theta}(t)$  (up to a multiplicative constant).
Theorem 3. Suppose that  $L_{n}(\theta)$  is differentiable and convex. For any  $\epsilon > 0$  and  $t_{\max} \in (0, \infty]$ , assume that either  $t_{\max} < \infty$  or  $\| \theta(t_{\max}) \|_2 < \infty$ . Then, using the step sizes and the termination criterion specified above in (12), (13), and (14), any algorithm that produces iterates satisfying (8) for any  $k \geq 1$  terminates after finite number of iterations, and when terminated, the solution path  $\tilde{\theta}(t)$  satisfies
$$
\sup  _ {0 \leq t \leq t _ {\max }} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \lesssim \epsilon . \tag {15}
$$
Note that when  $\epsilon$  is small enough, we have that  $\alpha_{1} = \ln \left(1 + \sqrt{\epsilon} / \| \nabla L_{n}(\mathbf{0})\|_{2}\right)$ , which together with (15) implies that
$$
\sup_{0\leq t\leq t_{\max}}\left\{f_{t}(\tilde{\theta} (t)) - f_{t}(\theta (t))\right\} \lesssim (e^{\alpha_{1}} - 1)^{2}\| \nabla L_{n}(\mathbf{0})\|_{2}^{2}.
$$
As a result, the approximation error for the solution path is essentially controlled by the initial step size  $\alpha_{1}$ . Smaller  $\alpha_{1}$  leads to better approximation. However, smaller  $\alpha_{1}$  also leads to a slower exploration of the solution path and a more stringent stopping criterion at individual grid points (c.f. (8)). As such, the initial step size controls the trade-off between computational cost and accuracy. Moreover, given  $\alpha_{1}$ , we can see from (13) that how fast the algorithm can explore the solution path depends largely on  $\| \theta_k\| _2$ . In particular, if  $\| \theta_k\| _2$  is bounded (e.g., when  $\theta^{\star}$  is finite), then the first term in the min function of (13) increases as  $t_k$  increases. Therefore, aggressive step sizes can be taken in this case until it reaches  $\mathcal{O}(1)$ , which will likely lead to a fast exploration of the solution path. On the other hand, if  $\| \theta_k\| _2$  grows quite quickly to infinity as  $k$  increases, then the first term in the min function may go to zero as  $k\to \infty$  (say, when  $\| \theta_k\| _2 / e^{t_k / 2}\to \infty$ ). This means that the step sizes need to decrease to zero eventually, leading to a slower exploration of the solution path.
We summarize the proposed scheme in Algorithm 1. Again, we emphasize that the above scheme can be applied to any optimization algorithm. Later, we will empirically investigate its performance using Newton method and gradient descent method.
Lastly, we establish an upper bound on the total number of grid points required for Algorithm 1 to achieve  $\epsilon$ -suboptimality.
Theorem 4. Under the assumptions in Theorem 3, the total number of grid points required for Algorithm 1 to achieve an  $\epsilon$ -suboptimality (15) is at most  $\mathcal{O}(\epsilon^{-1/2})$ .
3 Extensions to nonconvex loss functions
So far we assume that the empirical loss is convex. Next, we consider a generalization, where we do not assume convexity for  $L_{n}(\theta)$ . Instead, we assume that there exists  $\lambda(t) > 0$  so that
$$
f _ {t} (\theta) - f _ {t} (\theta (t)) \leq \frac {1}{2 \lambda (t)} \| \nabla f _ {t} (\theta) \| _ {2} ^ {2}, \tag {16}
$$
for any  $t \geq 0$  and  $\theta \in \mathbf{dom}(L_n)$ . In the literature, the condition (16) was often referred to as the Polyak-Lojasiewicz (PL) inequality. Polyak [1964] proved the linear rate of convergence for gradient descent method under this condition. Also see Karimi et al. [2016] for discussions of more recent development. Note that if  $L_{n}(\theta)$  is convex, then (16) holds for  $\lambda (t) = e^{-t}$  since  $f_{t}(\theta)$  is  $e^{-t}$ -strongly convex. In general, however,  $L_{n}(\theta)$  does not need to be convex for (16) to hold.
For some technical reasons, we need to consider slightly different interpolation schemes
$$
\tilde {\theta} (t) = \theta_ {k} \quad \text {f o r a n y} t \in \left[ t _ {k}, t _ {k + 1} \right], 0 \leq k \leq N - 1;
$$
$$
\tilde {\theta} (t) = \theta_ {N} \quad \text {f o r a n y} t _ {N} <   t \leq t _ {\max } \text {i f} t _ {N} <   t _ {\max }, \tag {17}
$$
where  $t_0 = 0$  and  $\theta_0 = 0$ . As we can see that the approximated solution path is piecewise constant. We use the following termination criterion at each  $t_k$
$$
\left\| g _ {1} \right\| _ {2} \leq \min  \left(C _ {0} \left(e ^ {\alpha_ {1}} - 1\right) \left\| \nabla L _ {n} (\mathbf {0}) \right\| _ {2}, 1\right);
$$
$$
\left\| g _ {k + 1} \right\| _ {2} \leq \min  \left(C _ {0} \frac {e ^ {\alpha_ {k + 1}} - 1}{e ^ {t _ {k}} - 1} \| \theta_ {k} \| _ {2}, 1\right), k \geq 1. \tag {18}
$$
We also define a slightly new step size scheme as follows:
$$
\alpha_ {1} = \min  \left\{\alpha_ {\max }, \ln \left(1 + \frac {\sqrt {\epsilon \operatorname* {m i n} \left(\lambda_ {0} , \lambda_ {1}\right)}}{\| \nabla L _ {n} (\mathbf {0}) \| _ {2}}\right) \right\} \tag {19}
$$
$$
\alpha_ {k + 1} = \min  \left\{\alpha_ {\max }, 2 \alpha_ {k}, \ln \left(1 + \frac {c _ {1} \left(e ^ {\alpha_ {1}} - 1\right) \| \nabla L _ {n} (\mathbf {0}) \| _ {2} \left(e ^ {t _ {k}} - 1\right) \sqrt {\min  \left(\lambda_ {k} , \lambda_ {k + 1}\right)}}{\| \theta_ {k} \| _ {2} \sqrt {\min  \left(\lambda_ {0} , \lambda_ {1}\right)}}\right) \right\}, \tag {20}
$$
where  $\lambda_{k} = \inf_{t_{k}\leq t\leq t_{k + 1}}\lambda (t), k\geq 0$ . Finally, the stopping criterion for the path-following algorithm is set to be
$$
\frac {c _ {3}}{e ^ {t _ {N}} - 1} \leq \epsilon \text {o r} t _ {N} \geq t _ {\max }, \tag {21}
$$
where  $c_{3} > 0$  is an absolute constant.
Moreover, we need to make some additional assumptions on the loss function.
Assumption (A1). Assume that  $L_{n}(\theta)$  is differentiable and  $\inf_{\theta} L_{n}(\theta) > -\infty$ . Moreover, assume that there exists a positive decreasing function  $g(\cdot)$  such that  $\lambda(t) \geq g(t) > 0$  for any  $0 \leq t < \infty$ .
Theorem 5. Suppose that Assumption (A1) holds. For any  $\epsilon >0$  and  $t_{\mathrm{max}}\in (0,\infty ]$  assume that either  $t_\mathrm{max} < \infty$  or  $\sup_{0\leq t\leq t_{\mathrm{max}}}\| \theta (t)\| _2 < \infty$  . Then, using step sizes and termination criterion specified in (19), (20), and (21), any algorithm that produces gradients satisfying (18) terminates after finite number of iterations, and when terminated, the solution path  $\tilde{\theta} (t)$  defined in (17) satisfies
$$
\sup  _ {0 \leq t \leq t _ {\max }} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \lesssim \epsilon . \tag {22}
$$
The above result can be viewed as a generalization of Theorem 3, because when  $L_{n}(\theta)$  is convex, we could set  $\lambda(t) = \exp(-t)$  so that both condition (16) and Assumption (A1) are satisfied.
4 Numerical studies
In this section, we study the operating characteristics of Newton method, gradient descent method and a mixed method in ridge regression and  $\ell_2$ -regularized logistic regression, using both the simulated datasets and a real data example. Here the mixed method applies gradient descent method to minimize
$f_{t_k}(\theta)$  at the beginning and then switch to Newton method when the number of gradient steps  $n_k$  exceeds  $\min\{n, p\}$ . To illustrate the advantages of the proposed grid point selection scheme and stopping criterion, we use a standard implementation of a typical path-following algorithm as a baseline. In particular, the grid points are equally spaced (on a log-scale) for both examples. For  $\ell_2$  logistic regression, Newton method is used as the basis algorithm with a fixed stopping criterion  $\| \nabla f_{t_k}(\theta_k) \|_2 < 10^{-5}$ , while for ridge regression the exact solution can be computed. We refer to this method as the baseline method.
We also compare the proposed methods against the method of Ndiaye et al. [2019]. In particular, we compare our grid points selection scheme (13) with the so-called adaptive unilateral scheme proposed in Ndiaye et al. [2019], where Newton method is used as the basis algorithm at each grid point. Different from the aforementioned methods, adaptive unilateral scheme constructs piecewise constant solution path  $\tilde{\theta} (\lambda)$  over chosen grid points to approximate the solution path  $\theta (\lambda) = \arg \min_{\theta \in \mathbb{R}^p}P_\lambda (\theta)$ , where  $P_{\lambda}(\theta) = L_{n}(\theta) + (\lambda /2)\cdot \| \theta \|_{2}^{2}$ , such that for all  $\lambda \in (\lambda_{\mathrm{min}},\lambda_{\mathrm{max}})$
$$
P _ {\lambda} (\tilde {\theta} (\lambda)) - P _ {\lambda} (\theta (\lambda)) <   \epsilon . \tag {23}
$$
We call this Ndiaye's method throughout this section.
All methods considered in the numerical studies are implemented in R using Rcpp package [Eddelbuettel et al., 2011, Eddelbuettel, 2013].
Example 1: Ridge regression. This example considers ridge regression over a simulated dataset. In this case, the empirical loss function is  $L_{n}(\theta) = \| Y - X\theta \|_{2}^{2} / (2n)$ , where  $X \in \mathbb{R}^{n \times p}$  and  $Y \in \mathbb{R}^{n}$  denote the design matrix and the response vector. In the simulation, the data  $(X,Y)$  are generated from the usual linear regression model  $Y = X\theta^{\star} + \tilde{\epsilon}$ , where  $\tilde{\epsilon} \sim N(0,I_{n\times n})$ ,  $\theta^{\star} = (1 / \sqrt{p},\dots ,1 / \sqrt{p})^{\top}$ , and rows of  $X$  are IID samples from  $N(0,I_{p\times p})$ . Throughout this example, we consider  $n = 1000$  and  $p = 500,10000$ ,
Example 2:  $\ell_2$ -regularized logistic regression. This example considers  $\ell_2$ -regularized logistic regression over a simulated dataset. Let  $X \in \mathbb{R}^n$  and  $Y \in \{+1, -1\}^n$  denote the design matrix and the binary response vector. The empirical loss function for logistic regression is  $L_n(\theta) = n^{-1}\sum_{i=1}^{n}\log(1 + \exp(-Y_iX_i^\top\theta))$ . We simulate the data  $(X,Y)$  from a linear discriminant analysis (LDA) model. More specifically, we sample the components of  $Y$  independently from a Bernoulli distribution with  $\mathbb{P}(Y_i = +1) = 1/2; i = 1,2,\ldots,n$ . Conditioned on  $Y_i, X_i$ 's are then independently drawn from  $N(Y_i\mu, \sigma^2 I_{p\times p})$ , where  $\mu \in \mathbb{R}^p$  and  $\sigma^2 > 0$ . Note that  $\mu$  and  $\sigma^2$  controls the Bayes risk, which is  $\Phi(-\|\mu\|_2/\sigma)$  under the  $0/1$  loss, where  $\Phi(\cdot)$  is the cumulative distribution function of a standard normal random variable. Here we choose  $\mu = (1/\sqrt{p},\dots,1/\sqrt{p})$  and  $\sigma^2 = 1$  so that the Bayes risk is  $\Phi(-1)$ , which is approximately  $15\%$ . Similar to Example 1, two different problem dimension are considered:  $n = 1000$  and  $p = 500$ , 10000.
Example 3: Real data example. This example fits an  $\ell_2$ -regularized logistic regression using the  $a9a$  dataset from LIBSVM [Chang and Lin, 2011]. This dataset provides information about whether an adult's annual income surpasses $50,000, along with some specific characteristics of the given adult. It contains a total of 32561 examples with 123 features and 1 binary response variable.
For all three examples, we use the global approximation error  $\sup_{0\leq t\leq t_{\mathrm{max}}}\{f_t(\tilde{\theta} (t)) - f_t(\theta (t))\}$  as an evaluation metric, where  $\tilde{\theta} (t)$  is the linear interpolation of the iterates  $\theta_{k}$  generated by each method. Throughout, we let  $t_\mathrm{max} = 10$  . Since this metric can not be calculated exactly, we approximate the global approximation error by  $\max_{1\leq i\leq 100}\{f_{s_i}(\tilde{\theta}(s_i)) - f_{s_i}(\theta(s_i))\}$ , where  $s_1,\ldots ,s_{100}$  are uniformly sampled from  $(0,t_{\mathrm{max}})$ . Here we compute the solutions  $\theta (s_i)$  using the CVX solver [Grant and Boyd, 2014, 2008] for  $\ell_2$ -regularized logistic regression, while calculating  $\theta (s_i)$  explicitly for ridge regression.
We compare Newton method, gradient descent method and their mixed version against the baseline method and Ndiaye's method. To make a fair comparison with Ndiaye's method, we set  $\lambda_{\min} = 1 / (e^{10} - 1)$  to match  $t_{\max} = 10$  in log scale and choose  $\lambda_{\max} = 100$ .
Figure 1 presents runtime versus global approximation error for Example 1 and 2 based on 100 replications, as we vary  $\epsilon$  in (12) for our proposed method and (23) for Ndiaye's method. For the baseline method, we vary the spacing between consecutive grid points instead. Clearly the proposed Newton method outperforms the baseline method and Ndiaye's method in both low dimension and high dimension scenarios, which empirically confirms that the proposed grid point selection scheme
(c.f. (13)) is more efficient than both the standard equally-spaced grid point scheme and the adaptive unilateral scheme proposed by Ndiaye et al. [2019]. Moreover, gradient descent method performs similarly compared with the mixed method in both setups of Example 1, which suggests that switching does not happen much in those cases. However, in both problem dimensions of Example 2, we do see a difference between the mixed method and gradient descent, which is due to the fact that the switching to the Newton update does happen and it can speed up the computation by design. Lastly, as one would expect, Newton method is more efficient than both the gradient descent method and the mixed method when the problem dimension is small, but less so when problem dimension is large.
Figure 2 shows the runtime versus global approximation error tradeoff for our real data example, as we vary  $\epsilon$  for our proposed methods and Ndiaye's method, and change grid point spacing for the baseline method. Again, the Newton method is more efficient than both the baseline method and the Ndiaye's method, which demonstrates the advantage of our proposed grid point selection scheme over the standard equally-spaced grid point scheme and the scheme by Ndiaye et al. [2019].
We also plot the number of iterations at each grid point for Newton method and gradient descent method in Figure S1-S3 of the supplementary material. Interestingly, for ridge regression (c.f., Figure S1), the number of iterations by gradient method first increases and then stays flat as  $t_k$  grows. Newton method, however, only takes one iteration at each grid point. Moreover, the level of approximation (i.e.,  $\epsilon$ ) seems to have no impact on the number of iterations at each grid point, which is highly desirable. For the  $\ell_2$ -regularized logistic regression (c.f., Figure S2 and S3), the number of iterations needed increases as  $t_k$  increases for gradient descent method, whereas the Newton method always requires just a constant number of iterations at each  $t_k$ . Again, the level of approximation (i.e.,  $\epsilon$ ) does not seem to influence the number of iterations much.
Figure 1: Runtime v.s. suboptimality for the Newton method, the gradient descent method, the mixed method, the baseline method and Ndiaye's method under two problem dimensions when applied to ridge regression (upper panels) and  $\ell_2$ -regularized logistic regression (lower panels). The vertical and horizontal lines at each point represent the standard errors of suboptimality and runtime.
In this article, we proposed a novel grid point selection scheme and stopping criterion for any general path-following algorithms. A simple solution path was constructed through linear interpolation
Figure 2: Runtime v.s. suboptimality for the Newton method, the gradient descent method, the mixed method, the baseline method and Ndiaye's method when applied to  $\ell_2$ -regularized logistic regression over the  $a9a$  real dataset from LIBSVM [Chang and Lin, 2011].
and theoretical approximation-error bound was derived. A generalization to a class of nonconvex empirical loss was also considered. We have not touched on the overall computational complexity analysis for commonly used optimization algorithms. Based on some preliminary analysis, we conjecture that for gradient descent method and Newton method, the number of iterations required are  $\mathcal{O}(\epsilon^{-1}\ln (\epsilon^{-1}))$  and  $\mathcal{O}(\epsilon^{-1 / 2})$ , respectively. Moreover, lower bound results are also worth investigating, that is, what are the minimum number of gradient or Newton steps needed to achieve an  $\epsilon$ -suboptimality for the entire solution path? Moreover, our current analysis can be readily extended to general differentiable regularization functions. A more challenging future direction is to investigate the case where the empirical loss function or the regularizer are not differentiable.
The authors are supported in part by the US National Science Foundation (NSF) under grant DMS-17-12580, DMS-17-21445 and DMS-20-15490. The authors thank the reviewers for valuable comments and suggestions that improved the article.
T. B. Arnold and R. J. Tibshirani. Efficient implementations of the generalized lasso dual path algorithm. Journal of Computational and Graphical Statistics, 25(1):1-27, 2016. doi: 10.1080/10618600.2015.1008638.
R. Bao, B. Gu, and H. Huang. Efficient approximate solution path algorithm for order weight 1_1-norm with accuracy guarantee. In 2019 IEEE International Conference on Data Mining (ICDM), pages 958-963. IEEE, 2019.
H. H. Bauschke, P. L. Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011.
C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27, 2011.
D. Eddelbuettel. *Seamless R and C++ integration with Rcpp*. Springer, 2013.
D. Eddelbuettel, R. François, J. Allaire, K. Ushey, Q. Kou, N. Russel, J. Chambers, and D. Bates. Rcpp: Seamless r and c++ integration. Journal of Statistical Software, 40(8):1-18, 2011.
B. Efron, T. Hastie, I. Johnstone, and R. Tishirani. Least angle regression. The Annals of Statistics, 32(2):407-499, 2004.
J. Friedman, T. Hastie, H. Hofling, and R. Tibshirani. Pathwise coordinate optimization. The Annals of Applied Statistics, 1(2):302-332, 2007.
P. Garrigues and L. Ghaoui. An homotopy algorithm for the lasso with online observations. Advances in neural information processing systems, 21, 2008.