SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Path following algorithms for  $\ell_2$ -regularized M-estimation with approximation guarantee
Yunzhang Zhu  
Department of Statistics, Ohio State University  
plumbus, OH 43015  
hu.219@osu.edu
Renxiong Liu*  
Statistics and Data Science team  
Nokia Bell Labs  
Murray Hill, NJ 07974  
renxiong.liu@nokia-bell-labs.com
Many modern machine learning algorithms are formulated as regularized M-estimation problems, in which a regularization (tuning) parameter controls a trade-off between model fit to the training data and model complexity. To select the "best" tuning parameter value that achieves a good trade-off, an approximated solution path needs to be computed. In practice, this is often done through selecting a grid of tuning parameter values and solving the regularized problem at the selected grid points. However, given any desired level of accuracy, it is often not clear how to choose the grid points and also how accurately one should solve the regularized problems at the selected gird points, both of which can greatly impact the overall amount of computation. In the context of  $\ell_2$ -regularized M-estimation problem, we propose a novel grid point selection scheme and an adaptive stopping criterion for any given optimization algorithm that produces an approximated solution path with approximation error guarantee. Theoretically, we prove that the proposed solution path can approximate the exact solution path to arbitrary level of accuracy, while saving the overall computation as much as possible. Numerical results also corroborate our theoretical analysis.
Modern machine learning algorithms are often formulated as regularized M-minimization problems,
$$
\theta (\lambda) = \arg \min  _ {\theta} L _ {n} (\theta) + \lambda p (\theta), \tag {1}
$$
where  $L_{n}(\theta)$  denotes an empirical loss function,  $p(\theta)$  denotes a regularization function, and  $\lambda$  is a tuning parameter that controls the trade-off between model fit and model complexity. Varying the tuning parameter leads to a collection of models, among which one of them may be chosen as the final model. This requires solving a collection of related optimization problems, whose solutions are often referred to as the solution path (We refer readers to Figure S4 and S5 in the supplementary material for some examples of solution paths).
Very often the exact solution path  $\theta(\lambda)$  can not be computed, and path-following algorithms are usually used to obtain a sequence of solutions at some selected grid points to produce an approximated solution path. Existing path-following algorithms [see, e.g., Hastie et al., 2004, 2007] typically choose a set of equally-spaced grid points (often on a log-scale), and choose certain algorithms to solve the minimization problem at the selected grid points. Warm-start strategy is often used to leverage the fact that the minimizers at two consecutive grid points may be close. However, there is a paucity of literature on how to choose these grid points and how accurately one should solve the optimization
*Work done when Renxiong Liu is a PhD student at the Ohio State University.
problem at the selected grid points, both of which would influence the overall computational cost and approximation accuracy.
In the context of  $\ell_2$ -regularized M-estimation where  $p(\theta) = (1/2)\|\theta\|_2^2$ , we propose a novel grid point selection scheme and an associated stopping criterion at each grid point for any optimization method. The proposal is motivated by first proving that the approximation error of a solution path is governed by two important quantities, one depending on how finely the grid points are spaced and the other depending on how accurately one solves the regularized problem at the selected grid points. We refer these two quantities as interpolation error and optimization error. Given any optimization algorithm (that solves a single regularized problem), a stopping criterion can be constructed so that the optimization error is comparable to the interpolation error, which ensures that there is no wasteful computation at each selected grid point. Then a novel grid point selection scheme is constructed to balance the approximation errors over the entire solution path.
Theoretically, we first show that the approximation error of the proposed solution path can be upper bounded by the sum of the interpolation error and the optimization error. Given any optimization algorithm, we show that the proposed stopping criterion ensures that the optimization error is dominated by the interpolation error (see Theorem 1). We then establish a global approximation-error bound for the solution path (see Theorem 2). Optimizing this global approximation-error bound by using the proposed grid point selection scheme, we also prove that the proposed solution path can approximate the exact solution path to an arbitrary level of accuracy (see Theorem 3). Finally, we establish that the total number of grid points required for the proposed algorithm to achieve an  $\epsilon$ -suboptimality is at most  $\mathcal{O}(\epsilon^{-1/2})$  (see Theorem 4).
Path-following techniques have been used in many optimization algorithms. For instance, it is used in the interior point methods, which is one of most popular algorithms to solve general convex optimization problems [see, e.g., Chapter 1.3 of Nesterov and Nemirovskii, 1993]. However, the focus is often on the limit of the path rather than the entire solution path. In statistics/machine learning literature, Osborne [1992] and Osborne et al. [2000] applied the homotopy techniques to obtain piecewise linear solution paths for quantile regression and LASSO, respectively. Later Efron et al. [2004], Hastie et al. [2004], and Rosset and Zhu [2007] used similar ideas to derive the solution path for a family of regularized M-estimation problems. Subsequent developments include Friedman et al. [2007], Hoefling [2010], Arnold and Tibshirani [2016], among many others. Path following algorithms for more general types of  $\ell_1$ -regularization are also available. For example, Tibshirani and Taylor [2011] used dual path algorithm to compute the solution path of a generalized LASSO problem. In the context of order weight  $\ell_1$  model, Bao et al. [2019] proposed an efficient approximate solution path algorithm by using primal-dual method. These works often leverage the piecewise linearity of the solution path so that an exact path-following algorithm can be explicitly derived. For situations where the solution paths are not piecewise linear, a path-following algorithm based on Newton method was considered in Rosset [2004]. However, Rosset [2004] used a simple grid point selection scheme and only established pointwise closeness to the solution path. Subsequent research on situations where the solution paths are not piecewise linear includes the work of Garrigues and Ghaoui [2008], which introduced a homotopy algorithm tailored for solving the LASSO problem using online (sequential) observations, and the work of Qu et al. [2022], which approximates the piecewise smooth path in self-paced learning for generalized self-paced regularizer.
To the best of our knowledge, there is a paucity of research on the impact of grid points selection and how to set stopping criterion to save computation. One line of related works use a piecewise constant interpolation scheme to approximate the solution path to an arbitrary level of accuracy [see, e.g., Giesen et al., 2010, 2012, 2014, Ndiaye et al., 2019]. Most relevant to our work here are the works of Giesen et al. [2012] and Ndiaye et al. [2019]. Giesen et al. [2012] proposed a generic algorithm to compute a piecewise constant approximate solution path for a family of optimization problems parameterized concave $^{2}$  in a single hyperparameter. They showed that the number of grid points needed to achieve an  $\epsilon$  suboptimality is at most  $\mathcal{O}(1 / \sqrt{\epsilon})$ . However, their proposed method relies on the existence of a problem-dependent oracle, which for any chosen hyperparameter produces both an  $\epsilon$ -approximate solution at that hyperparameter and a concave polynomial that is used to compute the step size. Except for special examples like standard SVM, it not clear how to implement the oracle for general problems.
It requires that the object function is concave with respect to the hyperparameter for any fixed value of parameters. This includes most regularized machine learning problems as special cases.
In view of these limitations, Ndiaye et al. [2019] considered regularized convex optimization problems and proposed general strategies to choose grid points based on primal-dual methods. Again, a piecewise constant path is constructed to approximate the solution path. However, their approach can only be applied to a special class of loss functions that are composition of a "simple" function (whose conjugate can be analytically derived) and affine functions. Moreover, the choice of the grid points depends on some convexity and smoothness parameters associated with the loss, which either are not available except for some special class of functions or require careful tuning due to their great impact on grid points selection. Ndiaye et al. [2019] also showed that to achieve  $\epsilon$  suboptimality, the number of grid points required is  $\mathcal{O}(1/\sqrt[d]{\epsilon})$  for uniformly convex loss of order  $d$  [Bauschke et al., 2011] and  $\mathcal{O}(1/\sqrt{\epsilon})$  for generalized self-concordant functions [Sun and Tran-Dinh, 2017]. In our paper, we show that the number of grid points needed is  $\mathcal{O}(1/\sqrt{\epsilon})$  for general differentiable convex loss (see Theorem 4). Compared to Ndiaye et al. [2019], we also demonstrate through simulations that our method produces better approximated solution paths under a fixed computational budget (see Section 4). Moreover, our method can also be applicable to loss functions that are possibly nonconvex (see Section 3).
We also note that Proposition 38 of Ndiaye [2018] shows that under some regularity conditions  $\epsilon$ -suboptimality along the entire solution path can be achieved by using a piecewise linear interpolation. However, it not clear how these conditions can be used to construct an algorithm with approximation guarantee. Our proposed algorithm, on the other hand, provides an easily implementable grid point selection scheme and an associated stopping criterion at each grid point, which together ensures that  $\epsilon$ -suboptimality can be attained through a piecewise linear interpolation.
More recently, in the context of  $\ell_2$ -regularized convex optimization problems, Zhu and Liu [2021] proposed two path-following algorithms by using Newton method and gradient descent method as the basis algorithm separately, and discuss the grid point selection scheme for each algorithm based on the piecewise linear interpolation. Our paper differs from Zhu and Liu [2021] in twofold. First, the grid-point selection scheme in Zhu and Liu [2021] depends on the basis algorithm. By contrast, in our paper any basis algorithm can be applied at any chosen grid point as long as the stopping criterion is satisfied by the basis algorithm. Next, the path-following algorithms by Zhu and Liu [2021] can only be applied to convex optimization problems, while in this article we also consider loss functions that are possibly nonconvex (see Section 3).
The rest of the paper is organized as follows. Section 2 introduces the proposed path following algorithm and establish its global approximation-error bound. Section 3 considers an extension to nonconvex empirical loss. In Section 4, we compare the the proposed scheme to a standard path-following scheme through a simulated study using ridge regression and  $\ell_2$ -regularized logistic regression. We close the article with some remarks in Section 5. All proofs are included in the supplementary material.
2 Path following algorithms
We consider an  $\ell_2$ -regularized M-estimation problem. More specifically, denote by  $L_{n}(\theta)$  some empirical loss function, where  $\theta \in \mathbb{R}^p$  denotes a parameter vector. We consider the solution path of an  $\ell_2$ -regularized M-estimation problem:
$$
\theta (t) = \arg \min  _ {\theta \in \mathbb {R} ^ {p}} \left\{\left(e ^ {t} - 1\right) \cdot L _ {n} (\theta) + (1 / 2) \cdot \| \theta \| _ {2} ^ {2} \right\}. \tag {2}
$$
Note that as the tuning parameter  $t$  varies from 0 to  $\infty$ , the solution  $\theta(t)$  varies from 0 to a minimizer of  $L_{n}(\theta)$ . We also remark that the choice of  $e^{t} - 1$  is not essential. In fact, it can be replaced by any function  $C(t)$  that is a strictly increasing and differentiable function, with  $C(0) = 0$  and  $\lim_{t \to \infty} C(t) = \infty$ . The solution path produced should be independent of  $C(t)$ . Moreover, we can show that the  $\ell_{2}$  norm of the solutions is an increasing function of  $t$ , and the solution  $\theta(t)$  converges to the minimum  $\ell_{2}$  norm minimizer of  $L_{n}(\theta)$  (if it is finite) as  $t$  goes to infinity (see Corollary S1 in the supplementary material).
Suppose that the goal is to approximate the solution path  $\theta(t)$  over a given interval  $[0, t_{\max})$  for some  $t_{\max} \in (0, \infty]$ , where we allow  $t_{\max} = \infty$ . Given a set of grid points  $0 < t_1 < \dots < t_N < \infty$ , and approximated solutions  $\{\theta_k\}_{k=1}^N$  at these grid points, we construct an approximated solution path over  $[0, t_{\max})$  through linear interpolation. More specifically, we define a piecewise linear solution
path  $\tilde{\theta} (t)$  as follows
$$
\begin{array}{l} \tilde {\theta} (t) = \frac {t _ {k + 1} - t}{t _ {k + 1} - t _ {k}} \theta_ {k} + \frac {t - t _ {k}}{t _ {k + 1} - t _ {k}} \theta_ {k + 1} \quad \text {f o r a n y} t \in [ t _ {k}, t _ {k + 1} ], k = 0, 1, \dots , N - 1, \tag {3} \\ \theta (t) = \theta_ {N} \quad \text {f o r a n y} t _ {N} <   t \leq t _ {\max } \text {i f} t _ {N} <   t _ {\max }, \\ \end{array}
$$
where  $t_0 = 0$  and  $\theta_0 = 0$ . This defines an approximated solution path  $\tilde{\theta}(t)$  for any  $t \in [0, t_{\max}]$ , which interpolates the approximated solutions at the selected grid points. In view of this definition, we may also assume that  $t_{N-1} \leq t_{\max}$ , because we do not need  $\tilde{\theta}(t)$  over  $t \in [t_{N-1}, t_N]$  if  $t_{N-1} > t_{\max}$ .
To assess how well  $\tilde{\theta}(t)$  approximates  $\theta(t)$ , we use the function-value suboptimality of the solution paths defined by  $\sup_{0 \leq t \leq t_{\max}} \{f_t(\tilde{\theta}(t)) - f_t(\theta(t))\}$ , where  $f_t(\theta) \coloneqq (1 - e^{-t}) L_n(\theta) + e^{-t} (\|\theta\|_2^2 / 2)$  is a scaled version of the objective function in (2). In what follows, we call  $\sup_{0 \leq t \leq t_{\max}} \{f_t(\tilde{\theta}(t)) - f_t(\theta(t))\}$  the global approximation error of  $\tilde{\theta}(t)$ , and  $\sup_{t_k \leq t \leq t_{k+1}} \{f_t(\tilde{\theta}(t)) - f_t(\theta(t))\}$  the local approximation error of  $\tilde{\theta}(t)$  over  $[t_k, t_{k+1}]$ .
Intuitively, how well  $\tilde{\theta}(t)$  approximates  $\theta(t)$  depends on how finely spaced the grid points are and the quality of the solutions at the selected grid points (i.e.,  $\theta_k$ 's). Our first main result relates the local approximation error of  $\tilde{\theta}$  over  $[t_k, t_{k+1}]$  to the choice of the grid points and the accuracy of the approximated solutions at the selected grid points measured by the size of the gradients  $\|\nabla f_{t_k}(\theta_k)\|_2$ , where we assume that  $L_n(\theta)$  is differentiable.
Theorem 1. Assume that  $L_{n}(\theta)$  is a differentiable convex function. Let  $g_{k} \coloneqq \nabla f_{t_{k}}(\theta_{k}) = (1 - e^{-t_{k}})\nabla L_{n}(\theta_{k}) + e^{-t_{k}}\theta_{k}$  denote the scaled gradient at  $\theta_{k}$ . For any  $0 < t_1 < t_2 < \dots < t_N < \infty$ , we have that
$$
\sup  _ {t \in [ 0, t _ {1} ]} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \leq \max  \left(e ^ {t _ {1}} \| g _ {1} \| _ {2} ^ {2}, \| \theta_ {1} \| _ {2} ^ {2}\right) + \frac {e ^ {t _ {1}} \left(1 - e ^ {- t _ {1}}\right) ^ {2}}{2} \| \nabla L _ {n} (\mathbf {0}) \| _ {2} ^ {2}, \tag {4}
$$
$$
\begin{array}{l} \sup  _ {t \in [ t _ {k}, t _ {k + 1} ]} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \leq e ^ {t _ {k + 1}} \max  \left\{\left(\frac {1 - e ^ {- t _ {k + 1}}}{1 - e ^ {- t _ {k}}}\right) ^ {2} \| g _ {k} \| _ {2} ^ {2}, \| g _ {k + 1} \| _ {2} ^ {2} \right\} \\ + \left(e ^ {- t _ {k}} - e ^ {- t _ {k + 1}}\right) ^ {2} \max  \left\{\frac {e ^ {t _ {k + 1}} \| \theta_ {k} \| _ {2} ^ {2}}{(1 - e ^ {- t _ {k}}) ^ {2}}, \frac {e ^ {t _ {k}} \| \theta_ {k + 1} \| _ {2} ^ {2}}{(1 - e ^ {- t _ {k + 1}}) ^ {2}} \right\} \tag {5} \\ \end{array}
$$
for any  $k = 1, \dots, N - 1$ . If we further assume that  $\| \theta(t_{\max}) \|_2 < \infty$ , then we have that
$$
\sup  _ {t _ {N} <   t \leq t _ {\max }} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \leq \frac {e ^ {t _ {N}}}{2 \left(1 - e ^ {- t _ {N}}\right)} \| g _ {N} \| _ {2} ^ {2} + \frac {1}{2 \left(e ^ {t _ {N}} - 1\right)} \| \theta \left(t _ {\max }\right) \| _ {2} ^ {2} \tag {6}
$$
when  $t_N < t_{\mathrm{max}}$
The proof of Theorem 1 starts with relating the local approximation error to the suboptimality of  $\theta_{k}$  and  $\theta_{k + 1}$  at some  $t\in [t_k,t_{k + 1}]$ . The suboptimality can then further bounded by the square norm of the grandient leveraging the fact that the objective function is  $e^{-t}$ -strongly convex. Finally, a triangular inequality is used to bound the norm of the grandient by quantities that depend on  $\| g_k\| _2$  and  $\| \theta_k\| _2$ .
We can see from Theorem 1 that the upper bounds for the local approximation error in (4)-(6) consist of two terms, with the first one (depending on  $g_{k}$ ) being algorithm dependent and the other term stemming from interpolation. We call them optimization error and interpolation error, respectively. Note that the optimization error is roughly of order  $e^{t_k}\| g_k\| _2^2$ , which again depends on how accurate the solutions  $\theta_{k}$  are at the selected grid points. And the interpolation error is essentially independent of the quality of the solutions  $\theta_{k}$ 's, as the interpolation error only depends on the spacing of the grid points and the norm of the solutions along the solution path (typically  $\| \theta_k\| _2 = \mathcal{O}(\| \theta (t_k)\| _2)$ , c.f., Lemma S2 in the supplementary material). In other words, the interpolation error is irreducible once the grid points are chosen, while the optimization error does depend on the algorithm and it can be pushed to be arbitrarily small if we run the algorithm long enough at each grid point. In this sense, if the goal is to minimize the local approximation error, then both the grid points and optimization algorithm should be designed carefully to strike a balance between these two errors to minimize the overall computations. For instance, it would be wasteful to have the optimization error smaller than the interpolation error, because the additional computation would not improve the overall approximation error (at least in order).
Motivated by this observation and the local approximation-error bounds in Theorem 1, we propose a novel stopping criterion scheme at the selected grid points for any general optimization algorithm. In particular, for any given algorithm that takes an initializer and solves the problem at grid point  $t_{k+1}$ , we can run the algorithm initialized at  $\theta_k$  until the optimization error is smaller than the interpolation error for  $\theta_{k+1}$ , that is
$$
\begin{array}{l} \underbrace {e ^ {t _ {k + 1}} \max  \left\{\left(\frac {1 - e ^ {- t _ {k + 1}}}{1 - e ^ {- t _ {k}}}\right) ^ {2} \| \nabla f _ {t _ {k}} (\theta_ {k}) \| _ {2} ^ {2} , \| \nabla f _ {t _ {k}} (\theta_ {k + 1}) \| _ {2} ^ {2} \right\}} _ {\text {o p t i m i z a t i o n e r r o r}} \\ \lesssim \underbrace {\left(e ^ {- t _ {k}} - e ^ {- t _ {k + 1}}\right) ^ {2} \max  \left\{\frac {e ^ {t _ {k + 1}} \| \theta_ {k} \| _ {2} ^ {2}}{(1 - e ^ {- t _ {k}}) ^ {2}}, \frac {e ^ {t _ {k}} \| \theta_ {k + 1} \| _ {2} ^ {2}}{(1 - e ^ {- t _ {k + 1}}) ^ {2}} \right\}} _ {\text {i n t e r p o l a t i o n e r r o r}}, \tag {7} \\ \end{array}
$$
where the LHS is the optimization error and the RHS is the interpolation error in the bounds in Theorem 1. This condition can be further simplified, under which the upper bounds in Theorem 1 can also be further simplified and combined so that we can bound the global approximation error of  $\tilde{\theta}(t)$ . This is established in the following theorem.
Theorem 2. Assume that  $L_{n}(\theta)$  is a differentiable and convex. Suppose that  $2^{-1}\alpha_{k} \leq \alpha_{k + 1} \leq 2\alpha_{k}$ ,  $\alpha_{k} \leq \alpha_{\max}$  for some  $\alpha_{\max} \leq \ln (2)$ , and
$$
\left\| \nabla f _ {t _ {k}} \left(\theta_ {k}\right) \right\| _ {2} \leq C _ {0} \frac {\left(e ^ {\alpha_ {k}} - 1\right)}{\left(e ^ {t _ {k}} - 1\right)} \| \theta_ {k} \| _ {2}; 1 \leq k \leq N \tag {8}
$$
for some  $C_0 \leq 1/4$ . Denote
$$
A = 4 \left(1 + C _ {0} ^ {2} \left(e ^ {\alpha_ {\max }} + e ^ {\alpha_ {\max } / 2} + 1\right) ^ {2}\right), \quad B = \left(e ^ {\alpha_ {1}} - 1\right) ^ {2} \| \nabla L _ {n} (\mathbf {0}) \| _ {2} ^ {2},
$$
$$
C = \max _ {1 \leq k \leq N} \frac {e ^ {- t _ {k}} \| \theta_ {k} \| _ {2} ^ {2} (e ^ {\alpha_ {k + 1}} - 1) ^ {2}}{(1 - e ^ {- t _ {k}}) ^ {2}}, D = \frac {1 - e ^ {- \max (\alpha_ {N} , t _ {\max} - t _ {N})}}{e ^ {t _ {N}} - 1} \max \{\| \theta_ {N} \| _ {2} ^ {2}, \| \theta (t _ {\max}) \| _ {2} ^ {2} \}.
$$
$$
\sup  _ {0 \leq t \leq t _ {\max }} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \leq A \max  \{B, C, D \} \tag {9}
$$
when  $t_N < t_{\mathrm{max}}$  and  $\| \theta (t_{\mathrm{max}})\| _2 < \infty$ , and
$$
\sup  _ {0 \leq t \leq t _ {\max }} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \leq A \max  \{B, C \} \tag {10}
$$
when  $t_{N - 1}\leq t_{\mathrm{max}}\leq t_N$  for some  $N\geq 1$
The above result provides a practical approach to verify whether the optimization error is dominated by the interpolation error by checking (8). In other words, (8) is a sufficient condition for (7) and it can be used as a stopping criterion at each grid point  $t_k$  for any algorithm that computes a solution at  $t_k$ . We also remark that, using the above bounds, similar approximation-error bounds for  $\| \tilde{\theta}(t) - \theta(t) \|_2$  can be derived as well (see Corollary S2 in the supplementary material).
Another important implication of Theorem 2 is that the established global approximation-error bound provides a principled approach to choose the grid points so that for any  $\epsilon >0$  and  $t_{\mathrm{max}}\in (0,\infty ]$  we have
$$
\sup  _ {0 \leq t \leq t _ {\max }} \left\{f _ {t} (\tilde {\theta} (t)) - f _ {t} (\theta (t)) \right\} \lesssim \epsilon \tag {11}
$$
for the solution path generated by an algorithm, and at the same time it tries to maximize the speed at which the algorithm explores the solution path from 0 to  $t_{\mathrm{max}}$ . In particular, to ensure (11), we propose to use the following step sizes
$$
\alpha_ {1} = \min  \left\{\alpha_ {\max }, \ln \left(1 + \frac {\sqrt {\epsilon}}{\| \nabla L _ {n} (\mathbf {0}) \| _ {2}}\right) \right\} \tag {12}
$$
$$
\alpha_ {k + 1} = \min  \left\{\ln \left(1 + \frac {c _ {1} \left(e ^ {\alpha_ {1}} - 1\right) \| \nabla L _ {n} (\mathbf {0}) \| _ {2} e ^ {t _ {k} / 2} \left(1 - e ^ {- t _ {k}}\right)}{\| \theta_ {k} \| _ {2}}\right), \alpha_ {\max }, 2 \alpha_ {k} \right\}, \tag {13}
$$
Algorithm 1 A general path following algorithm.
Input:  $\epsilon > 0$ ,  $C_0 \leq 1/4$ ,  $c_1 \geq 1$ ,  $c_2 > 0$ ,  $0 < \alpha_{\max} \leq 5^{-1}$  and  $t_{\max} \in (0, \infty]$ .
Output: grid points  $\{t_k\}_{k=1}^N$  and an approximated solution path  $\tilde{\theta}(t)$ .