# Novel Feature Selection Method for Nonlinear Support Vector Regression

Authors: Kejia Xu; Ying Xu; Yafen Ye; Weijie Chen
Journal: Complexity
 (2022)

---

## Abstract

The development of sparse techniques presents a major challenge to complex nonlinear high-dimensional data. In this paper, we propose a novel feature selection method for nonlinear support vector regression, called FS-NSVR, which first attempts to solve the nonlinear feature selection problem in the regression technology field. FS-NSVR preserves the representative features selected in the complex nonlinear system due to its use of a feature selection matrix in the original space. FS-NSVR is a challenging mixed-integer programming problem that is solved efficiently by using an alternate iterative greedy algorithm. Experimental results on three artificial datasets and five real-world datasets confirm that FS-NSVR effectively selects representative features and discards redundant features in a nonlinear system. FS-NSVR outperforms L1-norm support vector regression, L1-norm least squares support vector regression, and Lp-norm support vector regression on both feature selection ability and regression efficiency.


---

## Body

## 1. Introduction

High-dimensional data have commonly emerged in diverse fields, such as finance [1], economics [2], biology [3], and medicine [4]. Complex nonlinear relationships between features may exist in high-dimensional datasets [5]. For example, most economic and financial time series follow nonlinear behavior [6]. Ling et al. explored the nonlinear relationship between globalization, natural resources, financial development, and carbon emissions [6]. Another example of complex nonlinear high-dimensional data is in the medicine field. Medical costs have a sophisticated relation with features [7]. Complex nonlinear high-dimensional data have frequently emerged in the biology field. Nonlinear relations between features can depict biological relationships more precisely and reflect critical patterns in biological systems [8].Complex nonlinear high-dimensional data may include some irrelevant and redundant features, which may reduce the effectiveness of data mining and may detract from the quality of the results [9–11]. Thus, complex nonlinear high-dimensional data need a sparse technique. Feature selection, as a useful sparse technique, selects some useful features upon which to focus its attention and ignores the rest [12–17]. In general, feature selection methods are classified as filter, wrapper, and embedded methods [16–18]. The embedded method is very popular for feature selection since it conducts feature selection and other learning tasks simultaneously [19].Sparse support vector regression, as a branch of sparse support vector machine [20–23], is a computationally powerful feature selection method. Sparse support vector regression always adopts a sparse regularization term to realize feature selection and regression simultaneously. Therefore, sparse support vector regression is an embedded feature selection method [24–26]. L1-norm support vector regression (L1-SVR) [27] and L1-norm least squares support vector regression (L1-LSSVR) [28] use the L1-norm sparse regularization term to shrink some coefficients of the regression estimators towards 0. According to the regression estimators, the contribution of each feature to the final decision function can be judged, and then, the useful features are selected, while the irrelevant and redundant features are discarded. To improve the sparseness of L1-SVR, Zhang et al. [29] proposed Lp-norm support vector regression (Lp-SVR) (0 < p < 1). The Lp-norm regularization term in Lp-SVR shrinks more coefficients of the regression estimators towards 0, and some coefficients are shrunk to exactly 0, leading to more irrelevant and redundant features being discarded. However, L1-SVR, L1-LSSVR, and Lp-SVR only solve the linear feature selection problem, which is not always suitable for complex nonlinear cases.To solve the feature selection problem for complex nonlinear high-dimensional data, we follow the spirit of nonlinear support vector machine-based feature selection [9, 30, 31] and then propose a novel feature selection method for nonlinear support vector regression, which is called FS-NSVR. We bring a feature selection matrix, as a diagonal matrix with elements of either 1 or 0, into nonlinear support vector regression. As a result, FS-NSVR becomes a mixed-integer programming problem (MPP). To solve FS-NSVR efficiently, we employ an alternate iterative greedy algorithm to find a local optimal value [32], in which we iteratively solve the standard SVR problem and a smaller nonconvex feature selection problem. In addition, a feature-ranking strategy is suggested [33], which ranks the features according to their contributions to the objective in the MPP. Compared with L1-SVR, L1-LSSVR, and Lp-SVR, the experimental results show that FS-NSVR selects the most appropriate representative features in the highly complex nonlinear relationships with smaller estimation errors than those produced by L1-SVR, L1-LSSVR, and Lp-SVR. This means that FS-NSVR not only selects the representative features but also has good regression effectiveness. The contributions of this paper are summarized as follows:(1)
Bringing a feature selection matrix into nonlinear support vector regression, we propose a novel feature selection method for nonlinear support vector regression to identify complex nonlinear relationships between features in their original space. The proposed model first attempts to solve the nonlinear feature selection problem in the regression technology field.(2)
The proposed model is a complex mixed-integer programming problem. Ensuring the efficiency of the learning process, we employ an alternate iterative greedy algorithm to find a local optimal value for the proposed model. The alternate iterative greedy algorithm, transferring the complex mixed-integer problem into the min-max optimization problem, effectively reduces the computational complexity.(3)
Experimental results on both artificial and real-world datasets indicate that the proposed model preserves the representative features selected in the complex nonlinear system, outperforming the other three linear feature selection methods, with better feature selection and regression results. The training speed of the proposed method confirms the efficiency of the alternate iterative greedy algorithm.The remainder of this paper is organized as follows: Section2 in this paper briefly focuses on support vector regression. In Section 3, we propose feature selection for nonlinear support vector regression. Section 4 provides artificial and real-world dataset experiments, and Section 5 concludes the paper.


## 2. Background

Starting with the notations, we consider a regression problem inn-dimensional real vector space Rn. Suppose that Y=y1,…,yl′ is the response vector, A is a known l×n design matrix of covariates, and Ai=xi1,…xin′ is the n-dimensional training sample. Next, we briefly review support vector regression (SVR) [26] that are closely related to FS-NSVR.The optimal nonlinear regression function of SVR is constructed as follows:(1)fx=KxT,ATw+b,where w∈Rn, b∈R, and K are an appropriately chosen kernel. Parameters in function (1) are estimated by solving the following optimization problem:(2)minw,b,ξ,ξ∗12w22+CeTξ+ξ∗,subjected.to.Y−KA,ATw−b≤ε+ξ,ξ≥0,KA,ATw+b−Y≤ε+ξ∗,ξ∗≥0.where ξ and ξ∗ are the slack variables and CC>0 is a parameter determining the trade-off between the empirical risk and the regularization term. To derive the dual formulation of SVR, we first introduce the Lagrangian function for the problem (2), which is(3)L=12wTw+CeTξ+ξ∗−αTε+ξ−Y+KA,ATw+b−βTε+ξ∗+Y−KA,ATw−b−ηTξ−γTξ∗,where α,β,η, and γ are the Lagrangian multiplier vectors. The Karush–Kuhn–Tucher (KKT) necessary and sufficient optimality conditions for the problem (2) are given by(4)w−KA,ATTα+KA,ATTβ=0,−eTα+eTβ=0,C−αi−ηi=0,i=1,…,l,C−βi−γi=0,i=1,…,l,Y−KA,ATw−b≤ε+ξ,ξ≥0,KA,ATw+b−Y≤ε+ξ∗,ξ∗≥0,αTε+ξ−Y+KA,ATw+b=0,βTε+ξ∗+Y−KA,ATw−b=0,ηTξ=0,η≥0,γTξ∗=0,γ≥0.According to the previously mentioned KKT conditions, we obtain the dual formulation of problem (2) as follows:(5)maxα,β−12∑i=1l∑j=1lαi−βjKxi,xjαi−βj+∑i=1lαi−βjyi+∑i=1lεαi+βisubjecte.to.∑i=1lαi−βj=00≤αi≤C,i=1,2,…,l,0≤βi≤C,i=1,2,…,l.w can be obtained from the solution α and β of (5) by(6)w=KA,ATTα−β.For any solution to (5), α¯=α¯1,α¯2,…,α¯lT and β¯=β¯1,β¯2,…,β¯lT, if α¯≠0, the solution b¯ to the problem (2) can be obtained in the following way:(1)
For any nonzero componentα¯j∈0,C,(7)b¯=yj−∑i=1lα¯i−β¯iKxi,xj+ε.(2)
For any nonzero componentβ¯k∈0,C,(8)b¯=yk−∑i=1lα¯i−β¯iKxi,xk−ε.The final decision function can be constructed as(9)fx=∑i=1lα¯i−β¯iKxi,x+b¯.


## 3. Feature Selection for Nonlinear Support Vector Regression

### 3.1. Problem Formulation

In this section, we propose feature selection for nonlinear support vector regression.Z=diag1or0 is an n×n feature selection matrix. We consider the following nonlinear regression function:(10)fx=KxTZ,ZATw+b,where w∈Rn, b∈R, and K is an appropriately chosen kernel. w and b are the unknown parameters that need to be estimated. The optimal feature selection matrix Z also needs to be searched simultaneously.The estimator of the regression function (10) can be defined as the solution to the FS-NSVR optimization problem:(11)minw,b,ξ,ξ∗,Z12w22+CeTξ+ξ∗+eTZes.t.Y−KAZ,ZATw−be≤εe+ξ,ξ≥0,KAZ,ZATw+be−Y≤εe+ξ∗,ξ∗≥0,Z=diag1or0,where ξ and ξ∗ are slack variables and C>0 is a parameter determining the trade-off between the empirical risk and the regularization term. In fact, the feature selection matrix Z defines a subspace spanned by the selected features. Minimizing the term eTZe in the objective function (11) has the beneficial effect of suppressing variables to produce a sparse set of nonzero feature weights. Therefore, FS-NSVR has nonlinear feature selection ability.


### 3.2. Problem Solution

Obviously, the FS-NSVR optimization problem is a mixed-integer programming problem. We reformulate problem (11) as follows:(12)minZminw,b,ξ,ξ∗12w22+CeTξ+ξ∗+eTZes.t.Y−KAZ,ZATw−be≤εe+ξ,ξ≥0,KAZ,ZATw+be−Y≤εe+ξ∗,ξ∗≥0,Z=diag1or0.Solving problem (12) to obtain global optimality is highly challenging and impractical [24]. We employ an alternate iterative greedy algorithm to find a local optimal value. First, we fix the integer part Z and then obtain the solution to (12), which leads to solving the problem in the same manner as SVR. Similar to the deduction process of nonlinear SVR in Section 2, we obtain the dual formulation of the inner minimization problem. Then, problem (12) can be rewritten as(13)minZmaxα,β−12∑i=1l∑j=1lαi−βiKxi∘Zii,xj∘Zjjαj−βj+∑i=1lαi−βiyi−ε∑i=1lαi+βi+∑k=1nZkksubjecte.to.∑i=1lαi−βi=0,0≤αi≤C,i=1,2,⋯,l,0≤βi≤C,i=1,2,⋯,l,Zkk∈0,1,k=1,⋯,n.Obviously, problem (13) is a challenging min-max optimization problem. Fixing the optimal solution of the inner maximization problem α,β, we obtain the outer minimization integer problem, which leads to exhaustive computation of the objective for the possible Z.To make the greedy algorithm work, we follow the strategy in [33] to initialize Z to make the algorithm more stable. The value of each feature is computed after solving SVR by(14)Valuefeaturei=∑i=1l∑j=1lαi−βiKxiZii,xjZjjαj−βj.The score of thei th feature that reflects the importance among all the features is computed by (15)Scorefeaturei=Valuefeaturei∑jValuefeaturej.The greedy algorithm starts from an initialZ0 generated by (15). If Scorefeature is less than 1/n, then Zii0=0; otherwise, Zii0=1. We then fix Z0 and solve problem (13) to obtain α0,β0. We calculate Valuefeature and Scorefeatureaccording to (14) and (15), respectively. Z1 is updated by replacing Z0 if objective (13) decreases more than the tolerance. After updating Z1, α1,β1 can be obtained again. The algorithm will be terminated if objective (13) decreases less than the tolerance. We summarize the procedures of the greedy approach in Algorithm 1 to give the feature selection method for nonlinear support vector regression. The proof of convergence of the greedy approach in Algorithm 1 can be obtained from Mangasarian and Kou [32].Algorithm 1: Feature selection for nonlinear support vector regression.
Input: Training set A,Y. The appropriate kernel parameter δ, parameter C;Output:α,β, and Z;BeginStart withZ0=I; set iteration number k=1;whilek<NdoFind the solutionαk,βk to problem (15) with the fixed Zk−1; Compute each feature score by (15).fori = 1 : nIfScorefeaturei<1/nthenZiik=0elseZiik=1endendIfαk+1−βk+12≈αk−βk2thenConverged and outputαk+1,βk+1 and Zk as the final solutions.elsesetk=k+1endendOutputαk+1,βk+1 and Zk as the final solutions;endObtaining the solution of problem (13), w can be obtained by(16)w=KAZ,ZATTα−β.For any solution to (15) α¯=α¯1,α¯2,⋯,α¯lT and β¯=β¯1,β¯2,⋯,β¯lT, if α¯≠0, the solution b¯ to problem (6) can be obtained in the following way:(1)
For any nonzero componentα¯j∈0,C,(17)b¯=yj−∑i=1lα¯i−β¯iKxi∘Zii,xj∘Zjj+ε.(2)
For any nonzero componentβ¯k∈0,C,(18)b¯=yk−∑i=1lα¯i−β¯iKxi∘Zii,xk∘Zkk−ε.The final decision function can be constructed as(19)fx=∑i=1lα¯i−β¯iKxiZii,xTZ+b¯.


### 3.3. Computational Complexity

Concerning the computational complexity of FS-NSVR, we find that FS-NSVR includes two parts: one is repeatedly computing the inner maximization problem of (13), and the other is repeatedly computing (14) and (15). The first part requires solving one quadratic programming problem. The time complexity of this part is approximately O2l3. For the second part, it is easy to compute with the fixed α,β, and the computational complexity of this part is not more than k times.


## 3.1. Problem Formulation

In this section, we propose feature selection for nonlinear support vector regression.Z=diag1or0 is an n×n feature selection matrix. We consider the following nonlinear regression function:(10)fx=KxTZ,ZATw+b,where w∈Rn, b∈R, and K is an appropriately chosen kernel. w and b are the unknown parameters that need to be estimated. The optimal feature selection matrix Z also needs to be searched simultaneously.The estimator of the regression function (10) can be defined as the solution to the FS-NSVR optimization problem:(11)minw,b,ξ,ξ∗,Z12w22+CeTξ+ξ∗+eTZes.t.Y−KAZ,ZATw−be≤εe+ξ,ξ≥0,KAZ,ZATw+be−Y≤εe+ξ∗,ξ∗≥0,Z=diag1or0,where ξ and ξ∗ are slack variables and C>0 is a parameter determining the trade-off between the empirical risk and the regularization term. In fact, the feature selection matrix Z defines a subspace spanned by the selected features. Minimizing the term eTZe in the objective function (11) has the beneficial effect of suppressing variables to produce a sparse set of nonzero feature weights. Therefore, FS-NSVR has nonlinear feature selection ability.


## 3.2. Problem Solution

Obviously, the FS-NSVR optimization problem is a mixed-integer programming problem. We reformulate problem (11) as follows:(12)minZminw,b,ξ,ξ∗12w22+CeTξ+ξ∗+eTZes.t.Y−KAZ,ZATw−be≤εe+ξ,ξ≥0,KAZ,ZATw+be−Y≤εe+ξ∗,ξ∗≥0,Z=diag1or0.Solving problem (12) to obtain global optimality is highly challenging and impractical [24]. We employ an alternate iterative greedy algorithm to find a local optimal value. First, we fix the integer part Z and then obtain the solution to (12), which leads to solving the problem in the same manner as SVR. Similar to the deduction process of nonlinear SVR in Section 2, we obtain the dual formulation of the inner minimization problem. Then, problem (12) can be rewritten as(13)minZmaxα,β−12∑i=1l∑j=1lαi−βiKxi∘Zii,xj∘Zjjαj−βj+∑i=1lαi−βiyi−ε∑i=1lαi+βi+∑k=1nZkksubjecte.to.∑i=1lαi−βi=0,0≤αi≤C,i=1,2,⋯,l,0≤βi≤C,i=1,2,⋯,l,Zkk∈0,1,k=1,⋯,n.Obviously, problem (13) is a challenging min-max optimization problem. Fixing the optimal solution of the inner maximization problem α,β, we obtain the outer minimization integer problem, which leads to exhaustive computation of the objective for the possible Z.To make the greedy algorithm work, we follow the strategy in [33] to initialize Z to make the algorithm more stable. The value of each feature is computed after solving SVR by(14)Valuefeaturei=∑i=1l∑j=1lαi−βiKxiZii,xjZjjαj−βj.The score of thei th feature that reflects the importance among all the features is computed by (15)Scorefeaturei=Valuefeaturei∑jValuefeaturej.The greedy algorithm starts from an initialZ0 generated by (15). If Scorefeature is less than 1/n, then Zii0=0; otherwise, Zii0=1. We then fix Z0 and solve problem (13) to obtain α0,β0. We calculate Valuefeature and Scorefeatureaccording to (14) and (15), respectively. Z1 is updated by replacing Z0 if objective (13) decreases more than the tolerance. After updating Z1, α1,β1 can be obtained again. The algorithm will be terminated if objective (13) decreases less than the tolerance. We summarize the procedures of the greedy approach in Algorithm 1 to give the feature selection method for nonlinear support vector regression. The proof of convergence of the greedy approach in Algorithm 1 can be obtained from Mangasarian and Kou [32].Algorithm 1: Feature selection for nonlinear support vector regression.
Input: Training set A,Y. The appropriate kernel parameter δ, parameter C;Output:α,β, and Z;BeginStart withZ0=I; set iteration number k=1;whilek<NdoFind the solutionαk,βk to problem (15) with the fixed Zk−1; Compute each feature score by (15).fori = 1 : nIfScorefeaturei<1/nthenZiik=0elseZiik=1endendIfαk+1−βk+12≈αk−βk2thenConverged and outputαk+1,βk+1 and Zk as the final solutions.elsesetk=k+1endendOutputαk+1,βk+1 and Zk as the final solutions;endObtaining the solution of problem (13), w can be obtained by(16)w=KAZ,ZATTα−β.For any solution to (15) α¯=α¯1,α¯2,⋯,α¯lT and β¯=β¯1,β¯2,⋯,β¯lT, if α¯≠0, the solution b¯ to problem (6) can be obtained in the following way:(1)
For any nonzero componentα¯j∈0,C,(17)b¯=yj−∑i=1lα¯i−β¯iKxi∘Zii,xj∘Zjj+ε.(2)
For any nonzero componentβ¯k∈0,C,(18)b¯=yk−∑i=1lα¯i−β¯iKxi∘Zii,xk∘Zkk−ε.The final decision function can be constructed as(19)fx=∑i=1lα¯i−β¯iKxiZii,xTZ+b¯.


## 3.3. Computational Complexity

Concerning the computational complexity of FS-NSVR, we find that FS-NSVR includes two parts: one is repeatedly computing the inner maximization problem of (13), and the other is repeatedly computing (14) and (15). The first part requires solving one quadratic programming problem. The time complexity of this part is approximately O2l3. For the second part, it is easy to compute with the fixed α,β, and the computational complexity of this part is not more than k times.


## 4. Experimental Results

To test the feature selection and regression effectiveness of the proposed FS-NSVR, we compare it with L1-SVR [27], L1-LSSVR [28], and Lp-SVR [29] by using three artificial datasets and seven real-world datasets. L1-SVR, Lp-SVR, and L1-LSSVR are embedded linear feature selection methods. All of these methods are implemented in the MATLAB R2019b environment on a PC running the 64-bit Windows XP OS on a 1.6 GHz Intel (R) processor with 16 GB of RAM.For the feature selection of nonlinear support vector regression, we employ a Gaussian kernel, and its kernel parameterp is selected from the set 2−8,…,28. Parameter C is also selected from the set 2−8,…,28. The insensitive parameter ε is fixed at 0.01. The optimal values of the parameters in the experiments are obtained by using the grid search method.Letm be the number of samples, yit is the ith test sample, y^it is the predicted value of yit, and y¯=1/m∑i=1myit is the average value of y1t,…,ymt. We use the following evaluation criteria to evaluate the variable selection and regression results.P1: the proportion of simulation runs with nonzero coefficients are selectedR2: the coefficient of determination R2 is defined as(20)R2=∑i=1my^it−y¯2∑i=1myit−y¯2.NMSE: the normalized mean squared error (NMSE) is defined as(21)NMSE=∑i=1myit−y^it2∑i=1myit−y¯2.RMSE: the root mean square error (RMSE) is defined as(22)RMSE=1m∑i=1myit−y^it2.Thus, the smaller the values of NMSE and RMSE are, the more statistical information is captured from the selected variables.


### 4.1. Artificial Datasets

To test the nonlinear feature selection performance of FS-NSVR, we provide three artificial datasets. Specifically, we setn=200, l=500, and ς∼N0,0.01. The first regression model is generated as follows:(23)Type A:y=sinx502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002x502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002+ς,where xi∈−5π,5π,i∈50,100,150,200,250,300,350,400,450,500.The second regression function is as follows:(24)Type B:y=1−x2+2x2e−0.5x2+ς,x2=x502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002,where xi∈−10,10,i∈50,100,150,200,250,300,350,400,450,500.The third regression function is as follows:(25)Type C:y=6sin0.5π−x+3sin0.5π−xς,x=x502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002,where xi∈−10,10,i∈50,100,150,200,250,300,350,400,450,500. The specifications of these artificial datasets are listed in Table 1.Table 1
Specification of datasets.
Data setsNo. of training samplesNo. of testing samplesNo. of featuresType A200200500Type B200200500Type C200200500Traffic126184048Student10628927CNN50061881Community999994102Pakinson104061226To evaluate the performance of the feature selection results, we adopt the following criteria:(26)Precision=tptp+fp,Recall=tptp+fn,where tp are true positives, fp are false-positives, fn are false negatives, and tn are negative counts. Precision and recall are commonly used to present results for binary decision problems in machine learning since they give a more accurate evaluation of an algorithm’s performance. Here, we use precision and recall to evaluate the feature selection results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR.The best parameters of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for artificial datasets are shown in Table 2. The feature selection and regression results on the previous three artificial datasets are shown in Table 3. From Table 3, we find that FS-NSVR drives larger precision and recall than L1-SVR, Lp-SVR, and L1-LSSVR. Meanwhile, FS-NSVR obtains a larger R2 and smaller NMSE than L1-SVR, Lp-SVR, and L1-LSSVR. It is clear that FS-NSVR has the ability to select the representative features and discard the redundant features in the nonlinear system. Therefore, FS-NSVR is suitable for solving the nonlinear feature selection problem, while L1-SVR, Lp-SVR, and L1-LSSVR are not suitable for solving the feature selection problem of complex nonlinear high-dimensional data. In terms of running times, although the training speed of FS-NSVR is slower than L1-LSSVR, it is significantly faster than L1-SVR and Lp-SVR.Table 2
The optimal parameters of artificial datasets.
Data setsRegressorCδType AFS-NSVR2423L1-SVR2−520Lp-SVR2−32−3L1-LSSVR2−32−3Type BFS-NSVR2322L1-SVR2020Lp-SVR2−32−3L1-LSSVR2−12−1Type CFS-NSVR2121L1-SVR2−22−2Lp-SVR2−42−1L1-LSSVR2−42−4Table 3
Comparison results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for artificial datasets.
Data setsRegressorNMSER2RMSEPrecisionRecallCPU S.Type AFS-NSVR0.7070.1460.14622.221000.148L1-SVR1.0320.0310.17716.67102.107Lp-SVR1.0280.0280.176—02.701L1-LSSVR1.0280.0310.176000.004Type BFS-NSVR0.0320.8840.06934.481000.071L1-SVR1.0680.0740.3961.26101.379Lp-SVR1.0770.0780.397—02.702L1-LSSVR1.0010.0010.383—00.004Type CFS-NSVR0.0060.8850.32418.511000.076L1-SVR1.0030.0094.186001.229Lp-SVR1.0110.0114.201—02.584L1-LSSVR0.9970.0014.174000.005Bold values refer to the best performing regressors under each criterion.


### 4.2. Parameters and Nonlinear Feature Selection Analysis

In this part, we analyze the effects of the parametersp and C on the nonlinear feature selection results. To test the influence of the kernel parameter p on NMSE, R2, precision, and recall, we first fix parameter C as the optimal value used in the experiments on the artificial datasets. Figures 1–3 illustrate the influence of the kernel parameter p on the nonlinear feature selection results for Type A, Type B, and Type C, respectively. From Figures 1–3, we find that when p increases, the NMSE value decreases and then increases. As the kernel parameter p increases, R2 and precision increase and then decrease, which means that the kernel parameter p has a strong influence on the feature selection ability of FS-NSVR. When p is selected as the optimal value, precision and recall reach maximum values, which means that FS-NSVR selects representative features and discards irrelevant features. Although the number of selected features is small, FS-NSVR can select the representative features in the datasets.Figure 1
The influence ofp on NMSE, R2, precision, and recall for the Type A dataset. Parameter C is fixed as the optimal value.Figure 2
The influence ofp on NMSE, R2, precision, and recall for the Type B dataset. Parameter C is fixed as the optimal value.Figure 3
The influence ofp on NMSE, R2, precision, and recall for the Type C dataset. Parameter C is fixed as the optimal value.To further test the influence of parameterC on the FS-NSVR feature selection results, we fix the kernel parameter p as the optimal value used in the experiments on the artificial datasets. Figures 4–6 show the influence of C on NMSE, R2, precision, and recall. From Figures 4–6, we observe that as parameter C increases, NMSE decreases and then remains constant. When C = 2, precision reaches a maximum value and then remains constant, which means that FS-NSVR selects the representative features and discards the irrelevant features.Figure 4
The influence ofC on NMSE, R2, precision, and recall for the Type A dataset. Parameter p is fixed as the optimal value.Figure 5
The influence ofC on the NMSE, R2, precision, and recall for the Type B dataset. Parameter p is fixed as the optimal value.Figure 6
The influence ofC on the NMSE, R2, precision, and recall for the Type C dataset. Parameter p is fixed as the optimal value.


### 4.3. Real-World Datasets

To further test the feature selection and regression performance of FS-NSVR, we consider five real-world datasets from the UC Irvine (UCI) Machine Learning Repository [34]. Table 1 displays the dataset information, including the specific numbers of training samples, test samples, and features. The best parameters of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for real-world datasets are shown in Table 4.Table 4
The optimal parameters of real-world datasets.
Data setsRegressorCδTrafficFS-NSVR2−122L1-SVR2121Lp-SVR2−728L1-LSSVR24`24StudentFS-NSVR2725L1-SVR2−12−1Lp-SVR2−424L1-LSSVR2−32−3CNNFS-NSVR2823L1-SVR2424Lp-SVR2−321L1-LSSVR2525CommunityFS-NSVR2521L1-SVR2020Lp-SVR2−120L1-LSSVR2424PakinsonFS-NSVR2522L1-SVR2−52−5Lp-SVR2−323L1-LSSVR2626Bold values refer to the best performing regressors under each criterion.Table5 lists the feature selection and regression results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR. One can easily observe that FS-NSVR selects fewer features than those of L1-SVR, Lp-SVR, and L1-LSSVR, but FS-NSVR obtains small values for NMSE and RMSE, which are comparable with the other methods. FS-NSVR selects very few useful features and captures the nonlinear statistical information in the test datasets. FS-NSVR realizes feature selection and regression simultaneously due to its inherent feature selection property. Faced with complex nonlinear high-dimensional datasets, L1-SVR, Lp-SVR, and L1-LSSVR present challenges since they can only solve the feature selection problem for the linear version. Regarding the running time, FS-NSVR is significantly faster than L1-SVR and Lp-SVR.Table 5
Comparison results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for real-world datasets.
Data setsRegressorP1 (%)NMSERMSER2CPU S.StudentFS-NSVR6.2500.0940.0370.9130.070L1-SVR72.9170.0620.0300.9561.118Lp-SVR62.5000.0650.0300.9800.400L1-LSSVR52.0830.0620.0250.9600.001TrafficFS-NSVR18.5190.9050.9460.1022.007L1-SVR85.1850.9630.9760.2522.259Lp-SVR88.8891.0831.0350.5300.457L1-LSSVR44.4440.9521.6020.0570.001CNNFS-NSVR7.4070.0010.0351.0000.237L1-SVR86.4200.000.0111.0001.249Lp-SVR9.8770.0010.0320.9810.372L1-LSSVR8.6420.0000.0200.9830.001CommunityFS-NSVR21.5690.3790.1380.9181.108L1-SVR66.6670.3310.1290.6562.047Lp-SVR37.2550.3370.1300.6371.822L1-LSSVR74.5100.3400.1300.7310.001PakinsonFS-NSVR23.0770.8620.4460.4151.176L1-SVR42.3081.0930.5020.3191.276Lp-SVR88.4621.2290.5330.4300.570L1-LSSVR76.9231.0210.3720.1970.001


## 4.1. Artificial Datasets

To test the nonlinear feature selection performance of FS-NSVR, we provide three artificial datasets. Specifically, we setn=200, l=500, and ς∼N0,0.01. The first regression model is generated as follows:(23)Type A:y=sinx502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002x502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002+ς,where xi∈−5π,5π,i∈50,100,150,200,250,300,350,400,450,500.The second regression function is as follows:(24)Type B:y=1−x2+2x2e−0.5x2+ς,x2=x502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002,where xi∈−10,10,i∈50,100,150,200,250,300,350,400,450,500.The third regression function is as follows:(25)Type C:y=6sin0.5π−x+3sin0.5π−xς,x=x502+x1002+x1502+x2002+x2502+x3002+x3502+x4002+x4502+x5002,where xi∈−10,10,i∈50,100,150,200,250,300,350,400,450,500. The specifications of these artificial datasets are listed in Table 1.Table 1
Specification of datasets.
Data setsNo. of training samplesNo. of testing samplesNo. of featuresType A200200500Type B200200500Type C200200500Traffic126184048Student10628927CNN50061881Community999994102Pakinson104061226To evaluate the performance of the feature selection results, we adopt the following criteria:(26)Precision=tptp+fp,Recall=tptp+fn,where tp are true positives, fp are false-positives, fn are false negatives, and tn are negative counts. Precision and recall are commonly used to present results for binary decision problems in machine learning since they give a more accurate evaluation of an algorithm’s performance. Here, we use precision and recall to evaluate the feature selection results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR.The best parameters of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for artificial datasets are shown in Table 2. The feature selection and regression results on the previous three artificial datasets are shown in Table 3. From Table 3, we find that FS-NSVR drives larger precision and recall than L1-SVR, Lp-SVR, and L1-LSSVR. Meanwhile, FS-NSVR obtains a larger R2 and smaller NMSE than L1-SVR, Lp-SVR, and L1-LSSVR. It is clear that FS-NSVR has the ability to select the representative features and discard the redundant features in the nonlinear system. Therefore, FS-NSVR is suitable for solving the nonlinear feature selection problem, while L1-SVR, Lp-SVR, and L1-LSSVR are not suitable for solving the feature selection problem of complex nonlinear high-dimensional data. In terms of running times, although the training speed of FS-NSVR is slower than L1-LSSVR, it is significantly faster than L1-SVR and Lp-SVR.Table 2
The optimal parameters of artificial datasets.
Data setsRegressorCδType AFS-NSVR2423L1-SVR2−520Lp-SVR2−32−3L1-LSSVR2−32−3Type BFS-NSVR2322L1-SVR2020Lp-SVR2−32−3L1-LSSVR2−12−1Type CFS-NSVR2121L1-SVR2−22−2Lp-SVR2−42−1L1-LSSVR2−42−4Table 3
Comparison results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for artificial datasets.
Data setsRegressorNMSER2RMSEPrecisionRecallCPU S.Type AFS-NSVR0.7070.1460.14622.221000.148L1-SVR1.0320.0310.17716.67102.107Lp-SVR1.0280.0280.176—02.701L1-LSSVR1.0280.0310.176000.004Type BFS-NSVR0.0320.8840.06934.481000.071L1-SVR1.0680.0740.3961.26101.379Lp-SVR1.0770.0780.397—02.702L1-LSSVR1.0010.0010.383—00.004Type CFS-NSVR0.0060.8850.32418.511000.076L1-SVR1.0030.0094.186001.229Lp-SVR1.0110.0114.201—02.584L1-LSSVR0.9970.0014.174000.005Bold values refer to the best performing regressors under each criterion.


## 4.2. Parameters and Nonlinear Feature Selection Analysis

In this part, we analyze the effects of the parametersp and C on the nonlinear feature selection results. To test the influence of the kernel parameter p on NMSE, R2, precision, and recall, we first fix parameter C as the optimal value used in the experiments on the artificial datasets. Figures 1–3 illustrate the influence of the kernel parameter p on the nonlinear feature selection results for Type A, Type B, and Type C, respectively. From Figures 1–3, we find that when p increases, the NMSE value decreases and then increases. As the kernel parameter p increases, R2 and precision increase and then decrease, which means that the kernel parameter p has a strong influence on the feature selection ability of FS-NSVR. When p is selected as the optimal value, precision and recall reach maximum values, which means that FS-NSVR selects representative features and discards irrelevant features. Although the number of selected features is small, FS-NSVR can select the representative features in the datasets.Figure 1
The influence ofp on NMSE, R2, precision, and recall for the Type A dataset. Parameter C is fixed as the optimal value.Figure 2
The influence ofp on NMSE, R2, precision, and recall for the Type B dataset. Parameter C is fixed as the optimal value.Figure 3
The influence ofp on NMSE, R2, precision, and recall for the Type C dataset. Parameter C is fixed as the optimal value.To further test the influence of parameterC on the FS-NSVR feature selection results, we fix the kernel parameter p as the optimal value used in the experiments on the artificial datasets. Figures 4–6 show the influence of C on NMSE, R2, precision, and recall. From Figures 4–6, we observe that as parameter C increases, NMSE decreases and then remains constant. When C = 2, precision reaches a maximum value and then remains constant, which means that FS-NSVR selects the representative features and discards the irrelevant features.Figure 4
The influence ofC on NMSE, R2, precision, and recall for the Type A dataset. Parameter p is fixed as the optimal value.Figure 5
The influence ofC on the NMSE, R2, precision, and recall for the Type B dataset. Parameter p is fixed as the optimal value.Figure 6
The influence ofC on the NMSE, R2, precision, and recall for the Type C dataset. Parameter p is fixed as the optimal value.


## 4.3. Real-World Datasets

To further test the feature selection and regression performance of FS-NSVR, we consider five real-world datasets from the UC Irvine (UCI) Machine Learning Repository [34]. Table 1 displays the dataset information, including the specific numbers of training samples, test samples, and features. The best parameters of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for real-world datasets are shown in Table 4.Table 4
The optimal parameters of real-world datasets.
Data setsRegressorCδTrafficFS-NSVR2−122L1-SVR2121Lp-SVR2−728L1-LSSVR24`24StudentFS-NSVR2725L1-SVR2−12−1Lp-SVR2−424L1-LSSVR2−32−3CNNFS-NSVR2823L1-SVR2424Lp-SVR2−321L1-LSSVR2525CommunityFS-NSVR2521L1-SVR2020Lp-SVR2−120L1-LSSVR2424PakinsonFS-NSVR2522L1-SVR2−52−5Lp-SVR2−323L1-LSSVR2626Bold values refer to the best performing regressors under each criterion.Table5 lists the feature selection and regression results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR. One can easily observe that FS-NSVR selects fewer features than those of L1-SVR, Lp-SVR, and L1-LSSVR, but FS-NSVR obtains small values for NMSE and RMSE, which are comparable with the other methods. FS-NSVR selects very few useful features and captures the nonlinear statistical information in the test datasets. FS-NSVR realizes feature selection and regression simultaneously due to its inherent feature selection property. Faced with complex nonlinear high-dimensional datasets, L1-SVR, Lp-SVR, and L1-LSSVR present challenges since they can only solve the feature selection problem for the linear version. Regarding the running time, FS-NSVR is significantly faster than L1-SVR and Lp-SVR.Table 5
Comparison results of FS-NSVR, L1-SVR, Lp-SVR, and L1-LSSVR for real-world datasets.
Data setsRegressorP1 (%)NMSERMSER2CPU S.StudentFS-NSVR6.2500.0940.0370.9130.070L1-SVR72.9170.0620.0300.9561.118Lp-SVR62.5000.0650.0300.9800.400L1-LSSVR52.0830.0620.0250.9600.001TrafficFS-NSVR18.5190.9050.9460.1022.007L1-SVR85.1850.9630.9760.2522.259Lp-SVR88.8891.0831.0350.5300.457L1-LSSVR44.4440.9521.6020.0570.001CNNFS-NSVR7.4070.0010.0351.0000.237L1-SVR86.4200.000.0111.0001.249Lp-SVR9.8770.0010.0320.9810.372L1-LSSVR8.6420.0000.0200.9830.001CommunityFS-NSVR21.5690.3790.1380.9181.108L1-SVR66.6670.3310.1290.6562.047Lp-SVR37.2550.3370.1300.6371.822L1-LSSVR74.5100.3400.1300.7310.001PakinsonFS-NSVR23.0770.8620.4460.4151.176L1-SVR42.3081.0930.5020.3191.276Lp-SVR88.4621.2290.5330.4300.570L1-LSSVR76.9231.0210.3720.1970.001


## 5. Conclusion

Our paper focused on the nonlinear feature selection problem posed by high-dimensional data, especially when nonlinear complex relationships between features exist. To solve this problem, we brought a feature selection matrix in the original space into nonlinear support vector regression and then proposed a novel feature selection method for nonlinear support vector regression (FS-NSVR). FS-NSVR is a mixed-integer programming problem (MPP). To efficiently solve FS-NSVR, we employed an alternate iterative greedy algorithm to find a local optimal value. The feature selection matrix and the supervised selection process of FS-NSVR ensured that the representative features were selected and the redundant features were discarded automatically. The feature selection and regression performance of FS-NSVR in the artificial and real-world datasets confirmed its sparseness and effectiveness.The proposed method also has limitations that should be acknowledged in future studies. First, FS-NSVR is not suitable for the heterogeneity problem of nonlinear high-dimensional data. The spirit of quantile regression [35–37] can be brought into the nonlinear feature selection framework in the future. Second, more efficient methods to solve FS-NSVR are needed since the training speed of the current method is not fast enough with regard to large-scale data sets. Third, forming the application perspective on how to use FS-NSVR to deal with nonlinear feature selection problem in the real world remains a question in future work.



---

*Source: 4740173-2022-11-30.xml*