## 3 The SMO-like Algorithm for the Nonsmooth Dual Problem of \(\varepsilon\)-In SVR
### Working Set Selection
Five-fold cross-validation was used to tune the regularization parameter, kernel parameter and \(\eta_{1}\) within specified ranges \(\left\{10^{-1},10^{0},10^{1},10^{2},10^{3}\right\}\), \(\left\{2^{-3},2^{-2},2^{-1},2^{0},2^{1}\right\}\) and \(\left\{2^{-3},2^{-2},\ldots,2^{3}\right\}\), respectively. The epsilon parameter was then chosen from a separate set \(\left\{0,0.02,0.05,0.1,0.2,0.5,1,1.5,2,2.5,4\right\}\). The choice of the \(\eta_{1}\) parameter range is based on the understanding of its influence on the lncosh loss function. As shown in Figure 1, smaller values of \(\eta_{1}\) result in behavior similar to the \(l_{2}\) loss, while larger values resemble the \(l_{1}\) loss. This is consistent with the theoretical basis given in Equations (20) and (21), where the \(\eta_{1}\) tuning covers the spectrum between Vapnik's loss and the insensitive \(l_{2}\) loss for its extreme values. In addition, Figure 1e shows the relationship between \(\eta_{1}\) and the associated probability density function. In particular, \(\eta_{1}=2^{3}\) and \(\eta_{1}=2^{-3}\) lead to approximations of the Laplacian and Gaussian distributions, respectively. Based on these observations, we chose a range for \(\eta_{1}\) that effectively captures this transition in the behavior of the loss function and the probability density. Root Mean Square Error (RMSE) was used for performance evaluation. Results are averages of four cross-validation repetitions, obtained by running three SVR variants on nine benchmark datasets. Among these datasets, Mpg, Housing, Space ga, Abalone and CpuSmall are accessible on the LIBSVM website ([[https://www.csie.ntu.edu.tw/~clin/libsvmtools/datasets/](https://www.csie.ntu.edu.tw/~clin/libsvmtools/datasets/)]([https://www.csie.ntu.edu.tw/~clin/libsvmtools/datasets/](https://www.csie.ntu.edu.tw/~clin/libsvmtools/datasets/)), accessed on 20 April 2024), while the remaining datasets were from the UCI machine-learning repository ([[https://archive.ics.uci.edu/datasets](https://archive.ics.uci.edu/datasets)]([https://archive.ics.uci.edu/datasets](https://archive.ics.uci.edu/datasets)), accessed on 20 April 2024). While outputs remained unchanged, inputs were normalized to the closed interval \([0,1]\) due to varying dataset ranges. The outputs were also contaminated with additive Gaussian \(N(0,0.04)\) and Cauchy noises \(C(0,0.1)\) to compare the noise and outlier robustness of the proposed and traditional SVR variants. Iteration and training time ratios demonstrated in Figures 2 and 3 are defined as follows:

\[\text{iteration ratio}=\frac{\text{\# iter. by the proposed method}}{\text{\# iter. by other method}}\]

\[\text{training time ratio}=\frac{\text{time by the proposed method}}{\text{time by other method}}\]

Table 1 shows a comparison of \(\varepsilon\)-\(l_{2}\) SVR, \(\varepsilon\)-SVR and the proposed \(\varepsilon\)-ln SVR variants. \"# of SVs\" is the number of support vectors identified by the model during training. \"Test RMSE\" represents the RMSE on the unseen test data, which evaluates the generalization performance of the model. \"Training cpu time\" indicates the computational time requiredto train the model on the training data. \"# of iterations\" indicates the number of iterations required for the SMO-like optimization algorithm described in Algorithm 1 to converge. \"C\" is the regularization parameter, which controls the trade-off between fitting the data and avoiding overfitting. \"\(\sigma\)\" is the kernel parameter, which controls the influence of the data points in the kernel function. \"\(\varepsilon\)\" is the epsilon parameter, which defines the tolerance for errors within the \(\varepsilon\)-insensitive zone of the SVR model. \"\(\eta_{1}\)\" is a hyperparameter specific to the proposed \(\varepsilon\)-In SVR loss function, controlling the shape of the Incosh loss function as shown in Figure 1. The values of the hyperparameters (C, \(\sigma\), \(\varepsilon\), \(\eta_{1}\)) differ between the SVR variants because the employed losses have different characteristics.