## 2 Theory of Support Vector Regression  
A training data set \(S\) can be given by  
\[S=\{\mathbf{x}_{i},\;y_{i}\}_{i=1}^{N}\mathbf{x}_{i}\in\mathbf{X}\subseteq\mathbb{R}^{n}, \;y\in\mathbb{R} \tag{1}\]  
where \(N\) is the number of elements in \(S\) and \(n\) is the dimension of the input. \(S\) can be represented by the regression model [20]:  
\[y_{i}=\langle\mathbf{w},\mathbf{\Phi}(\mathbf{x}_{i})\rangle+b\quad i=1,2,\ldots.N \tag{2}\]  
where \(\mathbf{\Phi}(\mathbf{x}_{i})\) is the image of the input data in feature space, \(b\) is regressor bias, and \(\mathbf{w}\) is a weighting vector. The aim of the optimization is to find the optimal regressor within \(\varepsilon\) tube by minimizing the training error and optimizing the geometric margin. The primal form of the optimization problem can be formulated as [20; 43; 44]:  
\[\underset{(\mathbf{w},\mathbf{b})}{\text{min}}\frac{1}{2}\|\mathbf{w}\|^{2}+C\sum\limits_{i= 1}^{N}(\xi_{i}+\xi_{i}^{*}) \tag{3}\]  
subject to  
\[ \begin{array}{l}y_{i}-\left\langle\mathbf{w},\mathbf{\Phi}(\mathbf{x}_{i})\right\rangle-b \leq\varepsilon+\xi_{i}\\ \left\langle\mathbf{w},\mathbf{\Phi}(\mathbf{x}_{i})+b-y_{i}\leq\varepsilon+\xi_{i}^{*} \right\rangle\\ \xi_{i},\ \ \xi_{i}^{*}\geq 0;\ \ \ \ i=1,\ 2,\ \ldots,\ N\end{array}  \tag{4}\]  
where \(\varepsilon\) is the upper limit for the acceptable error, the slack variables \(\xi_{i}\) and \(\xi_{i}^{*}\) define the deviation from the \(\varepsilon\) tube [43; 44]. The constraints of the optimization problem are constructed by an \(\varepsilon\) insensitive loss function [44; 45]. The primal function and its constraints are used to define a Lagrange function. Lagrangian can be derived by combining the primal function and related constraints by presenting non-negative Lagrange multipliers \(\beta\), \(\beta\) *, \(\eta\), and \(\eta\) * [43; 44]:  
\[\begin{array}{ll}L=\frac{1}{2}\|\mathbf{w}\|^{2}&+C\sum\limits_{i=1}^{N}\big{(} \xi_{i}+\xi_{i}^{*}\big{)}-\sum\limits_{i=1}^{N}\beta  
The training data with non-zero Lagrange multipliers constitute the \"support vectors\" [43; 45; 46]. The regression problem of (2) can be solved by estimation via the support vectors and the corresponding Lagrange multipliers:  
\[\hat{y}(x)=\sum_{i=1}^{N}\lambda_{i}K(\mathbf{x},\ \mathbf{x_{i}})+b,\ \ \ \lambda_{i}=\beta_{i}-\beta_{i}^{*} \tag{12}\]  
Mostly used kernel functions are linear, polynomial, and radial basis kernel (RBF) functions. RBF function can be given as:  
\[K\big{(}\mathbf{X_{i}},\mathbf{X_{j}}\big{)}=exp\Big{(}{-\gamma}\parallel \mathbf{X_{i}}-\mathbf{X_{j}}\parallel{}^{2}\Big{)} \tag{13}\]  
where \(\gamma\) is the Gaussian parameter and \(\mathbf{X}\) is the training vector. Wavelet functions can also be used as kernel functions [47; 48; 49; 50] in SVR, as well as the functions mentioned above. Wavelet analysis has the advantage that it processes signals in both time and frequency domains. The prediction is achieved more accurately via wavelet kernel than other types of kernel functions [35]. The orthonormality of wavelet functions results in an increase in the generalization capability of SVR [47]. A wavelet function is defined as [47]  
\[h_{a,b}(x)=\frac{1}{\sqrt{a}}h\bigg{(}\frac{x-b}{a}\bigg{)} \tag{14}\]  
where \(a\) is the scaling factor and \(b\) is the shifting parameter. Wavelet kernel is derived as [47]  
\[K\Big{(}\mathbf{X},\mathbf{X^{{}^{\prime}}}\Big{)}=\prod_{i=1}^{N}h\bigg{(} \frac{x_{i}-b_{i}}{a}\bigg{)}h\bigg{(}\frac{x_{i}^{\prime}-b_{i}^{\prime}}{a} \bigg{)} \tag{15}\]  
where \(\mathbf{X}\) and \(\mathbf{X^{{}^{\prime}}}\) are \(N\)-dimensional vectors, \(h(x)\) is a wavelet function and \(a,\ b_{i},\ b_{i}^{\prime},\ x_{i},\ x_{i}^{\prime}\in R\) and \(\mathbf{X},\mathbf{X^{{}^{\prime}}}\in R^{N}\).