# A Feature-Weighted SVR Method Based on Kernel Space Feature
## 3 Feature-Weighted Support Vector Regression  
### The Necessity of Feature Weighting  
The similarity between sample points \(x_{i}\) and \(x_{j}\) in kernel space can be measured by calculating the distance \(d_{ij}\) between \(\phi\left(x_{i}\right)\) and \(\phi\left(x_{j}\right)\). \begin{table}
\begin{tabular}{c c c} \hline \hline
**Name** & **Definition** & **Parameters** \\ \hline Gaussian kernel & \(K\left(x_{i},x\right)=\exp\left(-\gamma\|x_{i}-x\|^{2}\right)\) & \(\gamma\) \\ Linear kernel & \(K\left(x_{i},x\right)=\left(x_{i}\cdot x\right)\) & - \\ Sigmoid kernel & \(K\left(x_{i},x\right)=\tanh\left(\gamma\left(x_{i}\cdot x\right)+R\right)\) & \(\gamma,R\) \\ Polynomial kernel & \(K\left(x_{i},x\right)=\left(\gamma\left(x_{i}\cdot x\right)+R\right)^{d}\) & \(\gamma,R,d\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Admissible kernel functions. \[ \begin{array}{l}d_{ij}=\left\|\phi\left(x_{i}\right)-\phi\left(x_{j}\right) \right\|^{2}\\ =\left\langle\phi\left(x_{i}\right)-\phi\left(x_{j}\right),\phi\left(x_{i} \right)-\phi\left(x_{j}\right)\right\rangle\\ =\left\langle\phi\left(x_{i}\right),\phi\left(x_{i}\right)\right\rangle-2 \left\langle\phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle\\ =K\left(x_{i},x_{i}\right)-2K\left(x_{i},x_{j}\right)+K\left(x_{j},x_{j} \right)\end{array}  \tag{9}\]  
When the Gauss kernel is adopted, \(d_{ij}\) can be expressed as:  
\[d_{ij}{=}2-2K\left(x_{i},x_{j}\right)=2-2\exp\left(-\gamma\left\|x_{i}-x_{j} \right\|^{2}\right) \tag{10}\]  
We can deduce that the greater similarity of the sample points, the smaller the distance between the mapping in the kernel space from Equation (10). When \(x_{i}{=}x_{j}\), the most similarity is shown, and the distance \(d_{ij}\) is zero. A simple example is given to illustrate that the construction of the kernel space is not reasonable when the value of the feature is the only consideration and its contribution to the model output is neglected. There is a set of sample points \(\left\{x_{1},x_{2},x_{3}\right\}\). Let \(n=2,x_{1},x_{2},x_{3}\in R^{n}\), where \(x_{1}{=}\left(\rho_{1}+\rho_{1}^{\prime},\rho_{2}\right)\), \(x_{2}{=}\left(\rho_{1},\rho_{2}+\rho_{2}^{\prime}\right)\), \(x_{3}{=}\left(\rho_{1},\rho_{2}\right)\), \(\rho_{1},\rho_{2}\in R\), \(\rho_{1}^{\prime},\rho_{2}^{\prime}\in R^{+}\); the first item of the sample point is Feature 1, and the second is Feature 2. Regarding \(\phi\left(x_{1}\right)\) and \(\phi\left(x_{2}\right)\), we can measure which one is more similar to \(\phi\left(x_{3}\right)\) by comparing the value of \(d_{13}\) and \(d_{23}\). \[\left\{ \begin{array}{l}d_{13}{=}2-2\exp\left(-\gamma\left\|x_{1}-x_{3} \right\|^{2}\right){=}2-2\exp\left(-\gamma{\rho_{1}^{\prime}}^{2}\right)\\ d_{23}{=}2-2\exp\left(-\gamma\left\|x_{2}-x_{3}\right\|^{2}\right){=}2-2\exp \left(-\gamma{\rho_{2}^{\prime}}^{2}\right)\end{array} \right. \tag{11}\]  
The difference of the similarities of the two sample groups is decided by \(\rho_{1}^{\prime}\) and \(\rho_{2}^{\prime}\) accordingly. However, the contribution of the two features to the model output is quite different for the actual system in some situations. Assume that Feature 1 has a great contribution to the output and a small change of it can lead to a great change in the output. On the contrary, assume that the contribution of Feature 2 to the output is very small, and a large change can lead to a slight change in the output. When \(\rho_{1}^{\prime}=\rho_{2}^{\prime}\), it is obvious that the impact of the sample point \(x_{2}\) on the output is more similar to that of \(x_{3}\). When \(\rho_{1}^{\prime}<\rho_{2}^{\prime}\), the influence of the sample point \(x_{2}\) on the output may be more similar to that of \(x_{3}\). The similarity to the contrary is deduced without considering the contribution of the features to the output.