## 3 The SMO-like Algorithm for the Nonsmooth Dual Problem of \(\varepsilon\)-In SVR
### Working Set Selection
It should be noted that the proposed WSS procedure, described in (36) and (37), is computationally efficient where \(i\) is chosen as one of the maximum violating pairs and \(j\) is chosen as the argument that minimizes the upper bound of the second-order Taylor polynomial approximation of consecutive loss function values. The proposed WSS is an extension of [21] where \(j\) is chosen in a manner that it is the argument of the minimum of an upper bound of consecutive loss function values. However, by introducing the idea of selecting \(j\) associated with the second-order Taylor polynomial approximation, it allows SO-like information to be used to solve a more complex nonsmooth nonlinear optimization problem. Algorithm 1 presents the pseudocode for the three fundamental parts of the SMO algorithm. To avoid numerical problems, especially for values close to the constraints, a small perturbation value \(\delta\) is added to the relevant sections in Algorithm 1 and chosen such that \(\delta=\hat{C}\times 10^{-5}\). Designing the algorithm also prioritizes computational efficiency. For example, consider the quadratic part of the loss function defined in (22): \(f_{q}(\mathbf{\alpha})=\frac{1}{2}\mathbf{\alpha}^{T}\mathbf{K}\mathbf{\alpha}+\mathbf{p}^{T}\mathbf{\alpha}\). When updating the gradient \(\
abla f_{q}(\mathbf{\alpha})=\mathbf{K}\mathbf{\alpha}+\mathbf{p}\), we avoid calculating the entire kernel matrix \(\mathbf{K}\). Instead, an iterative approach is employed, where the gradient is updated at each step as \(\
abla f_{q}(\mathbf{\alpha}):=\
abla f_{q}(\mathbf{\alpha})-\mathbf{K}_{i}d_{j}^{*}+K_{j }d_{j}^{*}\). This reduces the computational cost significantly. Furthermore, the terms \(\
abla T(\mathbf{\alpha})_{i}\) and \(\
abla T(\mathbf{\alpha})_{j}\) are also updated iteratively as shown in Algorithm 1 to avoid additional redundant nonlinear computations, where \(T(\alpha_{s})=-\frac{\hat{C}}{\eta_{2}}\ln\Big{[}\cosh\big{(}\tanh^{-1}(\frac{ \alpha_{s}}{\hat{C}})\big{)}\Big{]}+\frac{\alpha_{s}}{\eta_{2}}\tanh^{-1}( \frac{\alpha_{s}}{\hat{C}})\). The convergence results from [21] for \(\varepsilon\)-\(l_{2}\) SVR can be extended to \(\varepsilon\)-\(\ln\) SVR in a different manner that relies on the strong convexity of the \(\varepsilon\)-\(\ln\) SVR problem. It should be noted that, when \(\eta_{1}=\eta_{2}^{2}\to 0\), the \(\varepsilon\)-\(\ln\) SVR becomes equivalent to the \(\varepsilon\)-\(l_{2}\) SVR, and Lemma 1 also becomes equivalent to Lemma 2 in [21] derived for the nonsmooth, indeed piecewise quadratic, dual problem of \(\varepsilon\)-\(l_{2}\) SVR. In this context, only the following lemma, which determines an upper bound on the decrease in the consecutive loss function values in each iteration of the SMO algorithm, is presented. **Lemma 1**. : _The decrease in the dual function (22) in an iteration of SMO satisfies_  
\[J\Big{(}\mathbf{\alpha}^{k+1}\Big{)}-J\Big{(}\mathbf{\alpha}^{k}\Big{)}\leq-\frac{ \left\|\mathbf{\alpha}^{k+1}-\mathbf{\alpha}^{k}\right\|^{2}}{2\eta_{2}\hat{C}} \tag{42}\]  
Proof. : \(\hat{f}(d_{j})\) is strongly convex because \(\hat{f}(d_{j})-\frac{1}{\eta_{2}c}d_{j}^{2}\) is convex where its second derivative is \(a_{ij}+\frac{1}{\eta_{2}c\left[1-\frac{(\hat{C}^{k}_{j}+d_{j})^{2}}{\hat{C}^{2 }}\right]}+\frac{1}{\eta_{2}c\left[1-\frac{(\hat{C}^{k}_{j}-d_{j})^{2}}{\hat{ C}^{2}}\right]}-\frac{2}{\hat{C}}\geq a_{ij}\) is greater than zero. Herein, \(a_{ij}=K_{ii}+K_{jj}-2K_{ij}>0\) since the kernel matrix \(\mathbf{K}\) satisfying Mercer's condition is a positive definite matrix. Therefore, \(\hat{f}(d_{j})\) is also strongly convex, since \(\hat{f}(d_{j})=\hat{f}(d_{j})+|\mathbf{a}_{i}^{k}-d_{j}|+|\mathbf{a}_{j}^{k}+d_{j}|\).