# A Feature-Weighted SVR Method Based on Kernel Space Feature
## 2 Basic Review of SVR  
The training set is given as \(T=\{(x_{i},y_{i}),i=1,\cdots,l\}\), where each \(x_{i}\in R^{n}\) is the \(i\)-th input sample containing \(n\) features and \(y_{i}\in R\) is the output sample. The model function determined by the SVR method can be regarded as a hyperplane in the kernel space. It is expressed as follows:  
\[f\left(x\right)=<w,\phi\left(x\right)>+b \tag{1}\]  
where \(\phi\left(x\right)\) maps the raw data of input features to a high-dimensional kernel space, \(w\in R^{n}\) is a weight vector of the hyperplane and \(b\) is a bias term.  
An insensitive loss function \(\varepsilon>0\) is introduced to avoid over-fitting, and additional nonnegative slack variables \(\xi_{i},\xi_{i}^{*}\) are adopted to weaken the constraints of some certain sample points. SVR modeling is formulated as a convex quadratic programming problem expressed as follows:  
\[\min_{w,\xi,\xi^{*}} \frac{1}{2}\|w\|^{2}+C\sum_{i=1}^{l}\left(\xi_{i}+\xi_{i}^{*}\right) \tag{2}\]subject to:  
\[\left\{\begin{array}{l}y_{i}-f\left(x_{i}\right)\leq\varepsilon+\xi_{i}\\ f\left(x_{i}\right)-y_{i}\leq\varepsilon+\xi_{i}^{*}\\ \xi_{i},\xi_{i}^{*}\geq 0\end{array}\right. \tag{3}\]  
where \(C>0\) is a penalty parameter. The above convex quadratic programming problem can be solved by constructing a Lagrange function:  
The above convex quadratic programming problem can be reformulated by constructing the Lagrange function:  
\[\begin{array}{l}L=\frac{1}{2}\|w\|^{2}\ +C\sum\limits_{i=1}^{l}\left(\xi_{i}+ \xi_{i}^{*}\right)-\sum\limits_{i=1}^{l}\left(\xi_{i}\eta_{i}+\xi_{i}^{*}\eta_ {i}^{*}\right)\ -\sum\limits_{i=1}^{l}\alpha_{i}\left[\xi_{i}+\varepsilon+y_{i}-<w \cdot\phi\left(x_{i}\right)>-b\right]\\ -\sum\limits_{i=1}^{l}\alpha_{i}^{*}\left[\xi_{i}+\varepsilon-y_{i}+<w\cdot \phi\left(x_{i}\right)>+b\right]\end{array} \tag{4}\]  
where \(\alpha_{i},\alpha_{i}^{*}\geq 0\) and \(\eta_{i},\eta_{i}^{*}\geq 0\) are Lagrange multipliers.  
The kernel function \(K\left(\cdot\cdot,\cdot\right)\), which satisfies the Mercer condition, is introduced to replace the inner product of the high dimensional kernel space in Equation (4). The commonly-used kernel functions are Gaussian kernel, linear kernel, sigmoid kernel, polynomial kernel, and so on [28; 29; 30]. These kernel functions are listed in Table 1.  
The optimized problem can be expressed as follows:  
\[\underset{\alpha,\alpha^{*}}{\text{min}}\ \ \ \frac{1}{2}\sum\limits_{i=1,j=1}^{l} \left(\alpha_{i}-\alpha_{i}^{*}\right)\left(\alpha_{j}-\alpha_{j}^{*}\right)K \left(x_{i},x_{j}\right)+\varepsilon\sum\limits_{i=1}^{l}\left(\alpha_{i}+ \alpha_{i}^{*}\right)-\sum\limits_{i=1}^{l}\left(\alpha_{i}-\alpha_{i}^{*} \right)y_{i} \tag{5}\]  
subject to:  
\[\left\{\begin{array}{l}\sum\limits_{i=1}^{l}\left(\alpha_{i}-\alpha_{i}^{*} \right)=0\\ 0\leq\alpha_{i},\alpha_{i}^{*}\leq C\end{array}\right. \tag{6}\]  
The optimal solution can be obtained as follows:  
\[\tilde{\alpha}=\left(\tilde{\alpha}_{1},\tilde{\alpha}_{1}^{*},...,\tilde{ \alpha}_{l},\tilde{\alpha}_{l}^{*}\right)^{\text{T}} \tag{7}\]  
The model function Equation (1) can be further developed as follows:  
\[f\left(x\right)=\sum\limits_{i=1}^{l}\left(\tilde{\alpha}_{i}^{*}-\tilde{ \alpha}_{i}\right)K\left(x_{i},x\right)+\bar{b} \tag{8}\]