# A Feature-Weighted SVR Method Based on Kernel Space Feature
## 4 Simulation Examples
### Synthetic Datasets  
The definitions of these functions are listed in Table 2.  
Where, \(\sigma\) is the added Gaussian noise with a mean of zero and a standard deviation of 0.01. The synthetic dataset \"F1\" is chosen as an example. Feature 1 (\(x_{i1}\)) and Feature 2 (\(x_{i2}\)) in the training data are taken from the sinusoidal signals of 0.01 Hz and 0.05 Hz, respectively. Their corresponding test data are extracted from linear functions and the sinusoidal signal 0.125 Hz, respectively, as shown in Figure 1.  
The expression of F1 shows that the range of the first item affected by Feature 1 is \([-0.2172,1]\) and that of the second item affected by Feature 2 is restricted to \([-0.03,0.03]\). It is deduced that Feature 1, which has a great contribution to the output, is an important feature, while Feature 2 is to the contrary. However, the contribution of Feature 2 to the output of the model is neglected when the kernel matrix is calculated. In order to observe the influence of Feature 1 and Feature 2 on the kernel matrix, a kernel width \(\gamma\) = 0.01 is used to compare the three kernel matrices visualized in 2D heat-maps as follows. According to Equation (12), the kernel matrix can fully reflect the similarity between the sample points in kernel space. The similarity of the sample points in Figure 2c is clearly shown in Figure 2a, while the influence of Feature 1 with a high contribution to the output is obviously weakened. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Name** & **Function Definition** & **Domain of Definition** \\ \hline F1 & \(y_{i}{=}\frac{\sin{(x_{i1})}}{x_{i1}}+\frac{x_{i2}}{1000}+\sigma\) & \(x_{i1}\in[-10,10]\), \(x_{i2}\in[-30,30]\) \\ F2 & \(y_{i}{=}\ 10\cos{(3x_{i1})}+\frac{1}{10}\sin{(x_{i2})}+\sigma\) & \(x_{i1},x_{i2}\in[-2\pi,2\pi]\) \\ F3 & \(y_{i}{=}x_{i1}{}^{2}\sin{(x_{i2})}+\frac{3}{1+\varepsilon^{-4}3}+\sigma\) & \(x_{i1}\in[-3,3]\), \(x_{i2}\in[-2\pi,2\pi]\), \(x_{i3}\in[-200,200]\) \\ F4 & \(y_{i}{=}\frac{x_{i2}}{100}+\frac{x_{i2}}{10000}+\sigma\) & \(x_{i1}\in[-10,10]\), \(x_{i2}\in[-30,30]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Functions used to generate synthetic datasets. Figure 1: The input feature data of the training dataset and the test dataset: (**a**) training dataset; (**b**) test dataset. Figure 2: Three 2D heat-maps of kernel matrices: (**a**) kernel matrix generated by two features; (**b**) kernel matrix generated by Feature 1; (**c**) kernel matrix generated by Feature 2. A grid search method is applied to obtain the corresponding RMSE for each possible combination of \(w_{1},w_{2}\) to verify the necessity of feature weighting [34]. The value of \(w_{1}\) and \(w_{2}\) is searched from the set \(\left\{10^{-4},10^{-3.5},10^{-3},\cdots,10^{0}\right\}\). The model performance is shown as Figure 3 accordingly. According to Figure 3, a better generalization ability occurs when the value of \(w_{1}/w_{2}\) is around 100, and the best is acquired when \(w_{1}=10^{-2}\) and \(w_{2}=10^{-4}\). As a whole, a better generalization ability can be obtained when \(w_{1}>w_{2}\) vs. when \(w_{1}<w_{2}\). The FW-SVR is degraded to the classical SVR when \(w_{1}=w_{2}=1\), and the generalization ability of the model is poor. We compare the feature weighting method with the feature selection and the normalization. In feature weighting, \(w_{1}=10^{-2}\) and \(w_{2}=10^{-4}\). In feature selection, Feature 2 is deleted. In the normalization method, the min-max normalization and the Z-normalization are employed. Min-max normalization converts raw data to [0, 1] by linearization. The \(k\)-th feature of the \(i\)-th sample \(x_{ik}\) is normalized to \(x^{\prime}_{ik}\):  
\[x^{\prime}_{ik}\!\!=\!\frac{x_{ik}-x_{k\min}}{x_{k\max}-x_{k\min}} \tag{19}\]  
where \(x_{k\max},x_{k\min}\) is the maximum and minimum value of the \(k\)-th feature, respectively. The Z-normalization method normalizes the raw data to a dataset with a mean value of zero and a variance of one.