# An Efficient Feature Weighting Method for Support Vector Regression

Authors: Minghua Xie; Lili Xie; Peidong Zhu
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Support vector regression (SVR) is a powerful kernel-based method which has been successfully applied in regression problems. Regarding the feature-weighted SVR algorithms, its contribution to model output has been taken into account. However, the performance of the model is subject to the feature weights and the time consumption on training. In the paper, an efficient feature-weighted SVR is proposed. Firstly, the value constraint of each weight is obtained according to the maximal information coefficient which reveals the relationship between each input feature and output. Then, the constrained particle swarm optimization (PSO) algorithm is employed to optimize the feature weights and the hyperparameters simultaneously. Finally, the optimal weights are used to modify the kernel function. Simulation experiments were conducted on four synthetic datasets and seven real datasets by using the proposed model, classical SVR, and some state-of-the-art feature-weighted SVR models. The results show that the proposed method has the superior generalization ability within acceptable time.


---

## Body

## 1. Introduction

Support vector regression (SVR) is widely applied for regression problems due to its ability of converting the original low-dimensional problem to a high-dimensional kernel space linear problem by introducing kernel functions [1, 2]. The empirical risk and the confidence interval are balanced by adopting the principle of Structural Risk Minimization [3, 4]. SVR showed state-of-the-art performance in system modeling, such as remaining life prediction for lithium-ion batteries, electricity load forecasting, industrial analysis, coal flotation responses prediction, and so on [5–9].Feature weighting is an effective method to solve the problem that all features are treated equally. It makes the feature importance better matched with its influence on the kernel space. Our previous work [10] has analyzed the necessity of feature weighting and verified it by using the grid search (GS) method to select the optimal combination of weights. Grid search is an exhaustive search method because its computational cost is very high. Therefore, an effective identification of weights is needed. The grey correlation degree (GCD) theory was employed to calculate the weights for input features [11, 12]. The main idea of GCD is to evaluate the relationship between the input feature and the output of training samples according to the similarity of the geometric shape of sequence curves. The higher the similarity between the curves, the more important the corresponding input feature. However, the importance of the feature whose geometric shape is not similar to the output may be underestimated. Hou et al. developed a weighted kernel function for SVR based on maximal information coefficient (MIC) which is a distinct correlation statistic [13]. The relationship of both linear and nonlinear between input feature and output was evaluated. MIC can accurately identify the most important features of the input dataset. However, it depends too much on the quality and size of training data. Furthermore, it does not guarantee that the obtained weight combination is optimal, especially when MIC is regarded as an optimal weight. Wen et al. proposed a novel attribute feature weighting method called variable-weighted support vector machine [14]. In this method, the feature weights and optimal model parameters were both tuned by particle swarm optimization (PSO). The similar methods have also appeared in literature [15, 16]. A more flexible rational model can be constructed based on the PSO algorithm. Its further application in feature-weighted SVR is hindered due to the drawbacks of slow convergence and the local optima nevertheless.The paper proposed an efficient feature weighting method for support vector regression. Firstly, the MIC of each feature is calculated to reveal its relationship with output. The larger the value of MIC, the greater its contribution. The value constraint of each weight is obtained according to the MIC. Secondly, an improved constrained PSO is employed to tune the optimal feature weights and model parameters under the abovementioned constraints. Finally, the optimal weights are used to modify the kernel function.The contributions of the work are two folds. Firstly, regarding the feature weighting with limited training samples, the contribution is hard to be calculated accurately. In addition, there is no evidence showing that the contribution can be regarded as an optimal weight. In the paper, the value constraint of weight is obtained according to the MIC. Therefore, the above problems have been avoided. Secondly, both weights and parameters are tuned by a constrained PSO which can converge to the global optima more efficiently to obtain superior generalization ability in the case of acceptable time consumption.The paper is organized as follows: in Section2, the SVR modeling method is briefly described. In Section 3, firstly, the value constraint of each weight is obtained according to the MIC. Then, the constrained PSO algorithm is explicitly illustrated. Simulation examples are given in Section 4. In Section 5, general conclusions are given.


## 2. Basic Review of SVR

The training dataset is given asT=x1,y1,…,xl,yl∈Rm×Yl, where xi∈Rm is an input sample with m features and yi∈Y=R is the output, i=1,…,l. The SVR model function can be regarded as a hyperplane in the kernel space. It is expressed as follows:(1)fx=w,ϕx+b,where ϕx maps the low-dimensional input features to a high-dimensional kernel space, w∈Rn is a weight vector of the hyperplane, and b is a bias term.An insensitive loss functionε>0 is introduced to avoid overfitting, and additional nonnegative slack variables ξi,ξi∗ are adopted to weaken the constraints of some certain sample points. SVR modeling is formulated as a convex quadratic programming problem expressed as follows:(2)minw,ξ,ξ∗12w2+C∑i=1lξi+ξi∗,subject to(3)yi−fxi≤ε+ξi,fxi−yi≤ε+ξi∗,ξi,ξi∗≥0,where C>0 is a penalty parameter. The above convex quadratic programming problem can be solved by constructing a Lagrange function.The above convex quadratic programming problem can be reformulated by constructing the Lagrange function:(4)L=12w2+C∑i=1lξi+ξi∗−∑i=1lξiηi+ξi∗ηi∗−∑i=1lαiξi+ε+yi−w⋅ϕxi−b−∑i=1lαi∗ξi+ε−yi+w⋅ϕxi+b,where αi,αi∗≥0 and ηi,ηi∗≥0 are Lagrange multipliers.The kernel functionK·,·, which satisfies the Mercer condition, is introduced to replace the inner product of the high-dimensional kernel space in equation (4). The commonly used kernel function is Gaussian kernel [17]. It is expressed as follows:(5)Kxi,x=exp−γxi−x2,where γ is the kernel parameter that controls the width of the Gaussian kernel.The optimized problem can be expressed as follows:(6)minα,α∗12∑i=1,j=1lαi−αi∗αj−αj∗Kxi,xj+ε∑i=1lαi+αi∗−∑i=1lαi−αi∗yi,subject to(7)∑i=1lαi−αi∗=0,0≤αi,αi∗≤C.The solution can be obtained asα¯=α¯1,α¯1∗,…,α¯l,α¯l∗T. The model function fx can be represented as follows:(8)fx=∑i=1lα¯i∗−α¯iKxi,x+b¯.When0<α¯j<C is selected, b¯ can be expressed as follows:(9)b¯=yj−∑i=1lα¯i∗−αiKxi,xj+ε.When0<α¯k∗<C is selected, b¯ can be expressed as follows:(10)b¯=yk−∑i=1lα¯i∗−αiKxi,xk−ε.


## 3. Feature-Weighted Support Vector Regression

### 3.1. Method of Weight Constraint

Thejth input feature of the training set with l samples is Fj=x1j,x2j,…,xlj, the output sequence is Y=y1,y2,…,yl,Dj=FjT,YT. A p−by−q grid G is drawn on the scatterplot of Dj that partitions the data to encapsulate that relationship. Both p and q are positive integers.For each pairp,q, the possible largest mutual information achieved by any p−by−q grid and applied to Dj is found as follows:(11)I∗Dj,p,q=maxIDj|G.Then, the mutual information value is normalized to ensure a fair comparison between grids of different dimensions:(12)MDjp,q=I∗D,p,qlogminp,q.M=MDjp,q is defined as a characteristic matrix where MDjp,q is the highest normalized mutual information achieved by any p−by−q grid.Finally, the maximum value inM is defined as MIC as follows:(13)MICDj=maxpq≤BlMDjp,q,where B is a function of samples size. It is usually set to Bl=l0.6.When thejth input feature and output are statistically independent, MICDj tends to 0. When they are entirely correlated, MICDj tends to 1.


### 3.2. Implementation of the Constrained PSO

PSO is a widely used optimization tool based on a population, where each member is dubbed as particle. It simulates bird’s searching food. Each particle is a potential solution to the optimization problem. Particles fly through the problem space according to simple mathematical formulae which are deduced from their position and velocity.Each particle’s movement is influenced by its personal best-known position (pBest), but it is also guided toward the global best position (gBest), which is found by other particles. Assume the solution space isN-dimensional; the position, the velocity, and the pBest of ith particle are represented as pi=pi1,pi2,…,piN, vi=vi1,vi2,…,viN, and bi=bi1,bi2,…,biN, respectively. The gBest is represented as g=g1,g2,…,gN. At each iterative process, ith particle updates its velocity and position according to the following formulae:(14)vinnew=w′×vinold+c1×r1×bin−pin+c2×r2×gn−pin,pinnew=pinold+μ×vinnew,where w′ is a weighting factor to balance the global search and local search, c1 and c2 are both learning factors, r1 and r2 are random values between 0 and 1, and μ is a parameter about flying times.In the paper, PSO is employed to search the optimal hyperparametersC,γ,ε and the optimal combination of feature weights w1,w2,…,wm by minimizing the fitness function. The values of the MIC are used to constrain the weights. With the movement of the particles, both the optimal hyperparameters and the optimal weights will be obtained. The root mean square error (RMSE) is usually employed to evaluate the feasibility of the SVR method. It is selected as fitness function:(15)RMSE=1l∑i=1lyi−fxi2,where yi is the actual output sample and fxi is its corresponding predicted value. The smaller the value of RMSE, the better its generalization ability.


### 3.3. Steps of the Feature-Weighted Support Vector Regression

Steps of the proposed feature weighting support vector regression (MP-FWSVR) modeling method based on the MIC and the PSO are as follows:Step 1. MIC between each feature and output is calculated.Step 2. The values of the MIC are used to constrain the weights as follows:(16)κ1MICDimaxFi−minFi≤wi≤κ2MICDimaxFi−minFi,whereκ1 and κ2 are set to 0.01 and 1, respectively.Step 3. Theith particle is looked as a point in the 2+m-dimensional space and represented as pi=Ci,γi,wi1,wi2,…,wim, and its velocity is represented as vi=vCi,vγi,vwi1,vw2,…,vwim.Step 4. Obtain the optimal hyperparametersC,γ,ε and the optimal combination of feature weights w1,w2,…,wm by using PSO.Step 5. Regarding thekth feature, if a weight value wk is given, the kernel element will be changed as(17)Kwxi,xj=exp−γ∑k=1mwkxik−xjk2.The model functionfx can be further developed as follows:(18)fx=∑i=1lαi∗−αiKwxi,x+b.


## 3.1. Method of Weight Constraint

Thejth input feature of the training set with l samples is Fj=x1j,x2j,…,xlj, the output sequence is Y=y1,y2,…,yl,Dj=FjT,YT. A p−by−q grid G is drawn on the scatterplot of Dj that partitions the data to encapsulate that relationship. Both p and q are positive integers.For each pairp,q, the possible largest mutual information achieved by any p−by−q grid and applied to Dj is found as follows:(11)I∗Dj,p,q=maxIDj|G.Then, the mutual information value is normalized to ensure a fair comparison between grids of different dimensions:(12)MDjp,q=I∗D,p,qlogminp,q.M=MDjp,q is defined as a characteristic matrix where MDjp,q is the highest normalized mutual information achieved by any p−by−q grid.Finally, the maximum value inM is defined as MIC as follows:(13)MICDj=maxpq≤BlMDjp,q,where B is a function of samples size. It is usually set to Bl=l0.6.When thejth input feature and output are statistically independent, MICDj tends to 0. When they are entirely correlated, MICDj tends to 1.


## 3.2. Implementation of the Constrained PSO

PSO is a widely used optimization tool based on a population, where each member is dubbed as particle. It simulates bird’s searching food. Each particle is a potential solution to the optimization problem. Particles fly through the problem space according to simple mathematical formulae which are deduced from their position and velocity.Each particle’s movement is influenced by its personal best-known position (pBest), but it is also guided toward the global best position (gBest), which is found by other particles. Assume the solution space isN-dimensional; the position, the velocity, and the pBest of ith particle are represented as pi=pi1,pi2,…,piN, vi=vi1,vi2,…,viN, and bi=bi1,bi2,…,biN, respectively. The gBest is represented as g=g1,g2,…,gN. At each iterative process, ith particle updates its velocity and position according to the following formulae:(14)vinnew=w′×vinold+c1×r1×bin−pin+c2×r2×gn−pin,pinnew=pinold+μ×vinnew,where w′ is a weighting factor to balance the global search and local search, c1 and c2 are both learning factors, r1 and r2 are random values between 0 and 1, and μ is a parameter about flying times.In the paper, PSO is employed to search the optimal hyperparametersC,γ,ε and the optimal combination of feature weights w1,w2,…,wm by minimizing the fitness function. The values of the MIC are used to constrain the weights. With the movement of the particles, both the optimal hyperparameters and the optimal weights will be obtained. The root mean square error (RMSE) is usually employed to evaluate the feasibility of the SVR method. It is selected as fitness function:(15)RMSE=1l∑i=1lyi−fxi2,where yi is the actual output sample and fxi is its corresponding predicted value. The smaller the value of RMSE, the better its generalization ability.


## 3.3. Steps of the Feature-Weighted Support Vector Regression

Steps of the proposed feature weighting support vector regression (MP-FWSVR) modeling method based on the MIC and the PSO are as follows:Step 1. MIC between each feature and output is calculated.Step 2. The values of the MIC are used to constrain the weights as follows:(16)κ1MICDimaxFi−minFi≤wi≤κ2MICDimaxFi−minFi,whereκ1 and κ2 are set to 0.01 and 1, respectively.Step 3. Theith particle is looked as a point in the 2+m-dimensional space and represented as pi=Ci,γi,wi1,wi2,…,wim, and its velocity is represented as vi=vCi,vγi,vwi1,vw2,…,vwim.Step 4. Obtain the optimal hyperparametersC,γ,ε and the optimal combination of feature weights w1,w2,…,wm by using PSO.Step 5. Regarding thekth feature, if a weight value wk is given, the kernel element will be changed as(17)Kwxi,xj=exp−γ∑k=1mwkxik−xjk2.The model functionfx can be further developed as follows:(18)fx=∑i=1lαi∗−αiKwxi,x+b.


## 4. Simulation Examples

In this section, four synthetic datasets and seven real datasets are employed to evaluate the feasibility of the MP-FWSVR. All the codes are compiled on a Windows 10 PC with Intel Core i5-3470 CPU (3.2 GHz) and 12.0 GB RAM by MATLAB R2013a. The SVR algorithm is implemented via LIBSVM 3.22 [18].


### 4.1. Experiments on Artificial Synthetic Datasets

The definitions of four functions are listed in Table1. σ is the added Gaussian noise with a mean zero and a standard deviation of 0.01.Table 1
Functions used for generating synthetic datasets.
NameFunction definitionDomain of definitionF1yi=sinxi1/xi1+xi2/1000+σxi1∈−10,10,xi2∈−30,30F2yi=10cos3xi1+1/10sinxi2+σxi1,xi2∈−2π,2πF3yi=xi12sinxi2+3/1+e−xi3+σxi1∈−3,3,xi2∈−2π,2π,xi3∈−200,200F4yi=xi1/100+xi2/10000+σxi1∈−10,10,xi2∈−30,30The synthetic dataset “F1” is chosen as an example. The training data of Feature 1xi1 and Feature 2 xi2 are taken from the sinusoidal functions of 0.01 Hz and 0.05 Hz, respectively. Their test datasets are extracted from linear functions and the sinusoidal signal 0.125 Hz, respectively. The training size and test size are both 101×2. The two datasets are shown in Figure 1.Figure 1
The input feature data of (a) training dataset and (b) test dataset.
(a)(b)Firstly, we compare the optimal weights obtained by proposed MP algorithm with GS, MIC, and PSO algorithms. For MP and PSO, the parametersw′=0.6, c1=1.5, c2=1.7,andμ=1. The number of particles is set to 20, and the iteration count is 200. The searching range of hyperparameters C,γ,ε is 2−8,29, 2−8,210, and 10−3,1, respectively. For GS and MIC, the optimal weights w1,w2 of Feature 1 and Feature 2 are 0.01,0.0001 and 0.9496,0.2461, respectively. For MP and PSO, each method is repeatedly implemented 20 times. The optimal weights are shown as the box charts in Figure 2.Figure 2
The optimal weights obtained by MP and PSO.As can be seen from Figure2, the variation in w1 obtained by the PSO method is larger than that of the MP method. Moreover, the values of w2 optimized by PSO are all 0 which will cause the feature to be ignored during model training. According to [10], the weights combination (0.01, 0.0001) searched by GS is effective for model training. The mean weight values (0.0143, 0.0001) and (0.0219, 0.0000) obtained by MP and PSO, respectively, are closer to the GS method than the MIC method.Then, the MP algorithm is compared with MIC and PSO algorithms to observe the differences in the generalization ability. The near-average weights in Figure2 of MP and PSO are selected. The results are shown in Table 2.Table 2
Comparison of performance among MP, MIC, and PSO.
Methodw1w2Amplitude ratio of weighted feature 1 to 2C,γ,εRMSEMP0.01430.000041116.2602(5.2130, 363.0250, 0.001)0.0131MIC0.94960.24611.2862(362.0387, 0.0313, 0.001)0.1025PSO0.02570—(0.3415, 243.6, 0.001)0.0305As shown in Table2, the MP algorithm achieves the best generalization performance compared with the other two feature weighting methods. MIC can reveal the feature’s relationship with output. However, it does not consider the amplitude of the features. The amplitude ratio of MIC is only 1.2862, far less than 116.2602 of MP algorithm, which means that the influence of Feature 2 with a low contribution to the output is not obviously weakened. PSO can obtain both weights and parameters. However, the weight is not constrained to a reasonable interval. In Table 2, w2 is set to 0 which is obviously unreasonable.In order to facilitate the comparison, the predicted outputs for the test set are shown in Figure3.Figure 3
The predicted outputs for test set.Figure3 indicates that the output curve of MP-FWSVR is almost coincident with the true function. The prediction curve of MIC-FWSVR is quite different from the real output. The PSO-FWSVR achieves better results than MIC-FWSVR, but underfitting occurs because Feature 2 is not involved in model training due to w2=0.Finally, we model four synthetic datasets in Table3 to compare the proposed MP-FWSVR with GS-FWSVR [10], classical SVR, MIC-FWSVR [13], and PSO-VWSVR [14]. For classical SVR, the min-max normalization is employed to convert raw data to [0, 1] by linearization. The k-th feature of the i-th sample xik is normalized to xik′:(19)xik′=xik−xkminxkmax−xkmin,where xkmax,xkmin is the maximum and minimum value of the k-th feature, respectively. Each program is repeatedly implemented 10 times. The mean values and standard deviation of the RMSE are shown in Table 3.Table 3
Performance comparison of SVR modeling for the synthetic datasets.
Function nameMP-FWSVR RMSE (mean±std.) (training time (s))GS-FWSVR RMSE (mean±std.) (training time (s))Classical SVR RMSE (mean±std.) (training time (s))MIC-FWSVR RMSE (mean±std.) (training time (s))PSO-FWSVR RMSE (mean±std.) (training time (s))F10.0109±0.00320.0095±0.00300.1006±0.01950.1029±0.01750.0285±0.0059128.5738222.50629.270015.505353.0665F20.0896±0.00470.0864±0.00245.9021±0.00048.2245±0.00120.0892±0.0014382.3847110.87014.55089.4366293.4736F31.7280±0.13841.3450±0.00952.9849±0.00663.1479±0.00712.3882±0.5042116.2665627.05835.11325.989115.2894F40.0027±0.00020.0037±0.00090.0219±0.00310.0115±0.00210.0029±0.001161.6933242.88809.977116.637044.8777According to Table3, MP-FWSVR achieves a competitive generalization performance. Compared with the classical SVR, MIC-FWSVR, and PSO-FWSVR, MP-FWSVR achieves the best generalization performance on almost all four datasets. Compared with the GS-FWSVR, only one optimal result is obtained by MP-FWSVR. However, what needs to be noticed is that other three results are close to the GS-FWSVR ones in the case of much less training time consumption.


### 4.2. Experiments on Benchmark Datasets

We randomly choose seven UCI benchmark datasets [19], and the results are compared as in Table 4.Table 4
Performance comparison of SVR modeling for the UCI datasets.
Datasets (training size, test size)MP-FWSVR RMSE (mean±std.) (training time (s))GS-FWSVR RMSE (mean±std.) (training time (s))Classical SVR RMSE (mean±std.) (training time (s))MIC-FWSVR RMSE (mean±std.) (training time (s))PSO-FWSVR RMSE (mean±std.) (training time (s))CCPP(300×4, 9268×4)4.3690±0.0164165.53364.3739±0.00002.2334e + 0044.4532±0.000037.33704.4220±0.000038.92614.3844±0.0326100.2486Concrete(300×8, 730×8)10.3616±1.2075330.88088.2697±0.00001.4203e + 00711.3164±0.000036.374217.0857±0.000038.222513.9326±3.2440357.7611Airfoil(376×5, 1127×5)2.4946±0.0443691.79912.6239±0.00001.7233e + 0053.0588±0.000056.04226.3633±0.000058.91484.3604±0.1586184.5284Auto-MPG(196×7, 196×7)3.2494±0.192261.98122.5555±0.00001.1709e + 0062.9386±0.000014.86915.6032±0.000014.89083.1244±0.3231147.5384Machine CPU(109×7, 100×7)39.0181±5.834119.663120.9777±0.00003.9267e + 00548.7881±0.00005.1938156.0875±0.00005.250559.4450±15.221116.8121Servo(84×4, 83×4)1.1359±0.02298.84021.0071±0.00003.0122e + 0031.2255±0.00004.55981.2115±0.00004.51161.1439±0.06239.1346Yacht(154×6, 154×6)4.6284±1.10311.5130e + 030.8789±0.00006.4835e + 00515.0084±0.000041.49723.8010±0.000035.51843.2324±1.32731.3726e + 03According to Table4, two optimal results and three suboptimal results are obtained by MP-FWSVR. Compared with the MIC-FWSVR, MP-FWSVR achieves better results in 6 of the 7 datasets. Compared with the PSO-FWSVR, MP-FWSVR achieves better results in 5 of the 7 datasets.To summarize, the Wilcoxon signed rank tests [20] at the 0.05 significance level are implemented to test the differences between the MP-FWSVR, GS-FWSVR, and other SVR methods for all synthetic and UCI datasets to substantiate the indications in Tables 3 and 4. The test results are presented in Table 5.Table 5
Wilcoxon signed rank test for the prediction results.
MethodGS-FWSVRClassical SVRMIC-FWSVRPSO-FWSVRMP-FWSVR0.06740.00980.00980.1016GS-FWSVR—9.7656e − 049.7656e − 040.0020The overall results obtained by the Wilcoxon tests presented in Table5 show that there is significant difference among MP-FWSVR, classical SVR, and MIC-FWSVR. MP-FWSVR achieves the generalization ability which is more approximate to the GS-FWSVR than the PSO-FWSVR method when the time consumption is acceptable.


## 4.1. Experiments on Artificial Synthetic Datasets

The definitions of four functions are listed in Table1. σ is the added Gaussian noise with a mean zero and a standard deviation of 0.01.Table 1
Functions used for generating synthetic datasets.
NameFunction definitionDomain of definitionF1yi=sinxi1/xi1+xi2/1000+σxi1∈−10,10,xi2∈−30,30F2yi=10cos3xi1+1/10sinxi2+σxi1,xi2∈−2π,2πF3yi=xi12sinxi2+3/1+e−xi3+σxi1∈−3,3,xi2∈−2π,2π,xi3∈−200,200F4yi=xi1/100+xi2/10000+σxi1∈−10,10,xi2∈−30,30The synthetic dataset “F1” is chosen as an example. The training data of Feature 1xi1 and Feature 2 xi2 are taken from the sinusoidal functions of 0.01 Hz and 0.05 Hz, respectively. Their test datasets are extracted from linear functions and the sinusoidal signal 0.125 Hz, respectively. The training size and test size are both 101×2. The two datasets are shown in Figure 1.Figure 1
The input feature data of (a) training dataset and (b) test dataset.
(a)(b)Firstly, we compare the optimal weights obtained by proposed MP algorithm with GS, MIC, and PSO algorithms. For MP and PSO, the parametersw′=0.6, c1=1.5, c2=1.7,andμ=1. The number of particles is set to 20, and the iteration count is 200. The searching range of hyperparameters C,γ,ε is 2−8,29, 2−8,210, and 10−3,1, respectively. For GS and MIC, the optimal weights w1,w2 of Feature 1 and Feature 2 are 0.01,0.0001 and 0.9496,0.2461, respectively. For MP and PSO, each method is repeatedly implemented 20 times. The optimal weights are shown as the box charts in Figure 2.Figure 2
The optimal weights obtained by MP and PSO.As can be seen from Figure2, the variation in w1 obtained by the PSO method is larger than that of the MP method. Moreover, the values of w2 optimized by PSO are all 0 which will cause the feature to be ignored during model training. According to [10], the weights combination (0.01, 0.0001) searched by GS is effective for model training. The mean weight values (0.0143, 0.0001) and (0.0219, 0.0000) obtained by MP and PSO, respectively, are closer to the GS method than the MIC method.Then, the MP algorithm is compared with MIC and PSO algorithms to observe the differences in the generalization ability. The near-average weights in Figure2 of MP and PSO are selected. The results are shown in Table 2.Table 2
Comparison of performance among MP, MIC, and PSO.
Methodw1w2Amplitude ratio of weighted feature 1 to 2C,γ,εRMSEMP0.01430.000041116.2602(5.2130, 363.0250, 0.001)0.0131MIC0.94960.24611.2862(362.0387, 0.0313, 0.001)0.1025PSO0.02570—(0.3415, 243.6, 0.001)0.0305As shown in Table2, the MP algorithm achieves the best generalization performance compared with the other two feature weighting methods. MIC can reveal the feature’s relationship with output. However, it does not consider the amplitude of the features. The amplitude ratio of MIC is only 1.2862, far less than 116.2602 of MP algorithm, which means that the influence of Feature 2 with a low contribution to the output is not obviously weakened. PSO can obtain both weights and parameters. However, the weight is not constrained to a reasonable interval. In Table 2, w2 is set to 0 which is obviously unreasonable.In order to facilitate the comparison, the predicted outputs for the test set are shown in Figure3.Figure 3
The predicted outputs for test set.Figure3 indicates that the output curve of MP-FWSVR is almost coincident with the true function. The prediction curve of MIC-FWSVR is quite different from the real output. The PSO-FWSVR achieves better results than MIC-FWSVR, but underfitting occurs because Feature 2 is not involved in model training due to w2=0.Finally, we model four synthetic datasets in Table3 to compare the proposed MP-FWSVR with GS-FWSVR [10], classical SVR, MIC-FWSVR [13], and PSO-VWSVR [14]. For classical SVR, the min-max normalization is employed to convert raw data to [0, 1] by linearization. The k-th feature of the i-th sample xik is normalized to xik′:(19)xik′=xik−xkminxkmax−xkmin,where xkmax,xkmin is the maximum and minimum value of the k-th feature, respectively. Each program is repeatedly implemented 10 times. The mean values and standard deviation of the RMSE are shown in Table 3.Table 3
Performance comparison of SVR modeling for the synthetic datasets.
Function nameMP-FWSVR RMSE (mean±std.) (training time (s))GS-FWSVR RMSE (mean±std.) (training time (s))Classical SVR RMSE (mean±std.) (training time (s))MIC-FWSVR RMSE (mean±std.) (training time (s))PSO-FWSVR RMSE (mean±std.) (training time (s))F10.0109±0.00320.0095±0.00300.1006±0.01950.1029±0.01750.0285±0.0059128.5738222.50629.270015.505353.0665F20.0896±0.00470.0864±0.00245.9021±0.00048.2245±0.00120.0892±0.0014382.3847110.87014.55089.4366293.4736F31.7280±0.13841.3450±0.00952.9849±0.00663.1479±0.00712.3882±0.5042116.2665627.05835.11325.989115.2894F40.0027±0.00020.0037±0.00090.0219±0.00310.0115±0.00210.0029±0.001161.6933242.88809.977116.637044.8777According to Table3, MP-FWSVR achieves a competitive generalization performance. Compared with the classical SVR, MIC-FWSVR, and PSO-FWSVR, MP-FWSVR achieves the best generalization performance on almost all four datasets. Compared with the GS-FWSVR, only one optimal result is obtained by MP-FWSVR. However, what needs to be noticed is that other three results are close to the GS-FWSVR ones in the case of much less training time consumption.


## 4.2. Experiments on Benchmark Datasets

We randomly choose seven UCI benchmark datasets [19], and the results are compared as in Table 4.Table 4
Performance comparison of SVR modeling for the UCI datasets.
Datasets (training size, test size)MP-FWSVR RMSE (mean±std.) (training time (s))GS-FWSVR RMSE (mean±std.) (training time (s))Classical SVR RMSE (mean±std.) (training time (s))MIC-FWSVR RMSE (mean±std.) (training time (s))PSO-FWSVR RMSE (mean±std.) (training time (s))CCPP(300×4, 9268×4)4.3690±0.0164165.53364.3739±0.00002.2334e + 0044.4532±0.000037.33704.4220±0.000038.92614.3844±0.0326100.2486Concrete(300×8, 730×8)10.3616±1.2075330.88088.2697±0.00001.4203e + 00711.3164±0.000036.374217.0857±0.000038.222513.9326±3.2440357.7611Airfoil(376×5, 1127×5)2.4946±0.0443691.79912.6239±0.00001.7233e + 0053.0588±0.000056.04226.3633±0.000058.91484.3604±0.1586184.5284Auto-MPG(196×7, 196×7)3.2494±0.192261.98122.5555±0.00001.1709e + 0062.9386±0.000014.86915.6032±0.000014.89083.1244±0.3231147.5384Machine CPU(109×7, 100×7)39.0181±5.834119.663120.9777±0.00003.9267e + 00548.7881±0.00005.1938156.0875±0.00005.250559.4450±15.221116.8121Servo(84×4, 83×4)1.1359±0.02298.84021.0071±0.00003.0122e + 0031.2255±0.00004.55981.2115±0.00004.51161.1439±0.06239.1346Yacht(154×6, 154×6)4.6284±1.10311.5130e + 030.8789±0.00006.4835e + 00515.0084±0.000041.49723.8010±0.000035.51843.2324±1.32731.3726e + 03According to Table4, two optimal results and three suboptimal results are obtained by MP-FWSVR. Compared with the MIC-FWSVR, MP-FWSVR achieves better results in 6 of the 7 datasets. Compared with the PSO-FWSVR, MP-FWSVR achieves better results in 5 of the 7 datasets.To summarize, the Wilcoxon signed rank tests [20] at the 0.05 significance level are implemented to test the differences between the MP-FWSVR, GS-FWSVR, and other SVR methods for all synthetic and UCI datasets to substantiate the indications in Tables 3 and 4. The test results are presented in Table 5.Table 5
Wilcoxon signed rank test for the prediction results.
MethodGS-FWSVRClassical SVRMIC-FWSVRPSO-FWSVRMP-FWSVR0.06740.00980.00980.1016GS-FWSVR—9.7656e − 049.7656e − 040.0020The overall results obtained by the Wilcoxon tests presented in Table5 show that there is significant difference among MP-FWSVR, classical SVR, and MIC-FWSVR. MP-FWSVR achieves the generalization ability which is more approximate to the GS-FWSVR than the PSO-FWSVR method when the time consumption is acceptable.


## 5. Conclusions

In the paper, we propose a feature-weighted SVR method named MP-FWSVR which integrates the advantages of MIC and PSO algorithms to solve the time-consumption problem of the GS-FWSVR algorithm proposed in our previous work. The contribution of each input feature is expressed as MIC which is used to constrain the value of feature weight in the PSO optimization of the weight combination. Numerical experiments show the effectiveness of the proposed algorithm. Our future work will continue to focus on the feature weighting algorithm to get better generalization ability.



---

*Source: 6675218-2021-03-02.xml*