## 3 The SMO-like Algorithm for the Nonsmooth Dual Problem of \(\varepsilon\)-In SVR
### Working Set Selection
The proposed SMO algorithm is specifically designed to optimize dual convex nonsmooth problem (14) but it can also handle the smooth counterpart (\(\mathcal{T}\)) as a special case. To demonstrate the effectiveness of the SMO algorithm for solving this nonsmooth formulation, we compare its performance to the smooth counterpart. As shown in Figure 3, SMO algorithm for solving this nonsmooth version consistently performs better, with iteration ratios close to 1 and training time ratios significantly less than 1 for all datasets. This stems from the smooth version's SMO algorithm handling twice the number of optimization variables, requiring caching a larger kernel matrix of size \(\mathbf{K}\in R^{2L\times 2L}\) rather than \(\mathbf{K}\in R^{L\times L}\). The findings from comparing the proposed method with its FO and smooth counterparts are consistent with previous studies on SMO algorithms for solving piecewise QP problems in \(\varepsilon\)-\(l_{2}\) SVR [21] and \(\varepsilon\)-SVR [22], but with the key distinction that the idea of minimizing an upper bound of the second-order Taylor polynomial approximation in WSS allows for a computationally efficient SMO algorithm even for this complex nonsmooth problem. Figures 2b and 3b demonstrate the efficiency further, presenting ratios for total iterations and times during hyperparameter selection, again highlighting the method's advantages across the different values of the hyperparameters. The proposed SMO-like algorithm generalizes the classical SMO algorithm, designed for solving QP problems, to tackle more general nonsmooth, nonlinear convex dual problems arising in SVR with various loss functions. This generalization allows it to be adapted to both \(\varepsilon\)-\(l_{2}\) SVR and \(\varepsilon\)-\(l_{1}\) SVR. Due to the existence of analytical solutions for each iteration in the SMO-like algorithms for \(\varepsilon\)-\(l_{2}\) SVR and \(\varepsilon\)-\(l_{1}\) SVR, [PERSON]'s method becomes unnecessary. Consequently, only the WSS part, a key innovation of our algorithm, is embedded into the SMO-like algorithms presented in [21; 22]. As shown in Figure 4a,b, the iteration ratios for both \(\varepsilon\)-\(l_{2}\) SVR and \(\varepsilon\)-\(l_{1}\) SVR cases are exactly 1. This indicates that the proposed WSS, which leverages an upper bound based on the second-order Taylor polynomial approximation of consecutive loss function values, performs the same number of total iterations during hyperparameter selection for all datasets. Figure 4 presents the ratios for total times during hyperparameter selection. The proposed, easy-to-compute WSS described in Algorithm 1 demonstrates clear efficiency gains compared to those in [21; 22], since time ratios are below 1 for all datasets. The efficiency comes from incorporating the concept of working set selection, which is associated with the second-order Taylor polynomial approximation,and defining an upper bound that is easy to compute, as described in Proposition 4. This enables the efficient use of SO-like information to effectively tackle SVR problems, even those involving complex nonsmooth nonlinear convex scenarios. ## 5 Discussion

In this study, we introduce the \(\epsilon\)-In SVR model with a flexible Incosh loss function and demonstrate significant advances in its optimization and applicability. We derive a computationally efficient nonsmooth dual formulation of the problem, which addresses the challenges of nonlinearity and nondifferentiability. To overcome these complexities, a novel SMO-like algorithm with an effective WSS procedure is developed. This WSS procedure exploits second-order information by minimizing an upper bound on the Taylor polynomial approximation of consecutive loss function values, resulting in improved computational efficiency compared to its first-order and smooth counterparts. In addition, the single-parameter adjustable Incosh loss function is shown to be optimal in the maximum likelihood sense for the PHS distribution, which includes Laplace, Gaussian and hyperbolic secant distributions.