## 2 \(\varepsilon\)-In SVR and Its Dual Problem
### The Smooth Dual Problem of \(\varepsilon\)-In SVR  
Benefiting from [40] and representing the primal problem more accurately by eliminating unnecessary constraints and correcting the explicit \(\varepsilon\)-insensitivity definition of the loss function, the primal problem (1) of \(\varepsilon\)-In SVR with inequality constraints is obtained as follows:  
\[\min_{\mathbf{w}\in R^{q},b\in R^{1}} \frac{1}{2}\|\mathbf{w}\|^{2}+C\sum_{s=1}^{L}\left(l\big{(}\xi_{s} \big{)}+l\big{(}\xi_{s}^{\prime}\big{)}\right)\] (1) subject to \[y_{s}-\mathbf{w}^{T}\varphi(\mathbf{x}_{s})-b\leq\xi_{s}+\varepsilon,\] \[\mathbf{w}^{T}\varphi(\mathbf{x}_{s})+b-y_{s}\leq\xi_{s}^{\prime}+\varepsilon,\] \[\forall s\in\{1,\ldots,L\}\]  
where \(l(\xi_{s})=\frac{1}{\eta_{1}}\ln\big{(}\cosh(\eta_{2}\xi_{s})\big{)}\) is the loss function, \(C\) is the penalty parameter, \(x_{s}\in R^{m}\) denotes for the training sample, \(y_{s}\in R^{1}\) is the desired output for \(s\in\{1,\ldots,L\}\), \(\varphi(.)\in R^{m}\to R^{n}\) is a nonlinear function and \(\varepsilon\) determines the insensitivity region. The Lagrangian of this problem (1) is then obtained as follows:  
\[\mathcal{L}(\mathbf{w},b,\mathbf{\xi},\mathbf{\xi}^{{}^{\prime}},\mathbf{\lambda},\mathbf{\lambda}^{{}^{\prime}}) = \frac{1}{2}\|\mathbf{w}\|^{2}+C\sum_{s=1}^{L}\left(l\big{(}\xi_{s} \big{)}+l\big{(}\xi_{s}^{\prime}\big{)}\right)-\sum_{s=1}^{L}\lambda_{s}( \varepsilon+\xi_{s}-y_{s}+\mathbf{w}^{T}\varphi(\mathbf{x}_{s})+b) \tag{2}\] \[-\sum_{s=1}^{L}\lambda_{s}^{{}^{\prime}}(\varepsilon+\xi_{s}^{ \prime}+y_{s}-\mathbf{w}^{T}\varphi(\mathbf{x}_{s})-b)\] subject to \[\lambda_{s}\lambda_{s}^{{}^{\prime}}\geq 0,\ \forall s\in\{1,\ldots,L\}\]  
and the optimality conditions become as follows.  
\[\partial_{b}\mathcal{L}=0 \implies\sum_{s=1}^{L}\lambda_{s}-\lambda_{s}^{{}^{\prime}}=0\] (3) \[\partial_{\mathbf{w}}\mathcal{L}=0 \implies\mathbf{w}=\sum_{s=1}^{L}\alpha_{s}\varphi(\mathbf{x}_{s})\] (4) \[\partial_{\xi_{s}}\mathcal{L}=\lambda_{s}-\frac{C\eta_{2}}{\eta_ {1}}\tanh(\eta_{2}\xi_{s})=0 \implies\xi_{s}=\frac{1}{\eta_{2}}\tanh^{-1}(\frac{\eta_{1}\lambda_{s}}{ \eta_{2}C}),-\frac{\eta_{2}}{\eta_{1}}C\leq\lambda_{s}\leq\frac{\eta_{2}}{\eta_ {1}}C\] (5) \[\partial_{\xi_\[\min_{\lambda,\lambda^{\prime}\in R^{L}} \frac{1}{2}\sum_{s=1}^{L}\sum_{r=1}^{L}(\lambda_{s}-\lambda_{s}^{ \prime})K(\mathbf{x}_{s},\mathbf{x}_{r})(\lambda_{r}-\lambda_{r}^{\prime})+\varepsilon \sum_{s=1}^{L}(\lambda_{s}+\lambda_{s}^{\prime})-\sum_{s=1}^{L}y_{s}(\lambda_{s }-\lambda_{s}^{\prime}) \tag{7}\] \[-\sum_{s=1}^{L}\left(\frac{C}{\eta_{1}}\ln\Big{[}\cosh\big{(}\tanh^ {-1}(\frac{\eta_{1}\lambda_{s}}{\eta_{2}C})\big{)}\Big{]}-\frac{\lambda_{s}}{ \eta_{2}}\tanh^{-1}(\frac{\eta_{1}\lambda_{s}}{\eta_{2}C})\right.\] \[+\frac{C}{\eta_{1}}\ln\Big{[}\cosh\big{(}\tanh^{-1}(\frac{\eta_{ 1}\lambda_{s}^{\prime}}{\eta_{2}C})\big{)}\Big{]}-\frac{\lambda_{s}^{\prime}}{ \eta_{2}}\tanh^{-1}(\frac{\eta_{1}\lambda_{s}^{\prime}}{\eta_{2}C})\Big{)}\] subject to \[\sum_{s=1}^{L}(\lambda_{s}-\lambda_{s}^{\prime})=0,\] \[0\leq\lambda_{s},\lambda_{s}^{\prime}\leq\frac{\eta_{2}}{\eta_{1} }C,\forall s\in\{1,\dots,L\}\]  
where \(K(\mathbf{x}_{s},\mathbf{x}_{r})=\varphi(\mathbf{x}_{s})^{\top}\varphi(\mathbf{x}_{r})\) is the kernel function and \(\mathbf{\lambda}=[\lambda_{1}\cdots\lambda_{L}]\in R^{L}\), \(\mathbf{\lambda}^{\prime}=\big{[}\lambda_{1}^{\prime}\cdots\lambda_{L}^{\prime} \big{]}\in R^{L}\) are the Lagrange multipliers.