# Linear Twin Quadratic Surface Support Vector Regression

Authors: Qianru Zhai; Ye Tian; Jingyue Zhou
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Twin support vector regression (TSVR) generates two nonparallel hyperplanes by solving a pair of smaller-sized problems instead of a single larger-sized problem in the standard SVR. Due to its efficiency, TSVR is frequently applied in various areas. In this paper, we propose a totally new version of TSVR named Linear Twin Quadratic Surface Support Vector Regression (LTQSSVR), which directly uses two quadratic surfaces in the original space for regression. It is worth noting that our new approach not only avoids the notoriously difficult and time-consuming task for searching a suitable kernel function and its corresponding parameters in the traditional SVR-based method but also achieves a better generalization performance. Besides, in order to make further improvement on the efficiency and robustness of the model, we introduce the 1-norm to measure the error. The linear programming structure of the new model skips the matrix inverse operation and makes it solvable for those huge-sized problems. As we know, the capability of handling large-sized problem is very important in this big data era. In addition, to verify the effectiveness and efficiency of our model, we compare it with some well-known methods. The numerical experiments on 2 artificial data sets and 12 benchmark data sets demonstrate the validity and applicability of our proposed method.


---

## Body

## 1. Introduction

Support Vector Machine (SVM) was first introduced by Vapnik in [1, 2]. It is a classical classification method based on the principle of structural risk minimization. Due to its great performance, it has attracted much attention in the field of machine learning and has been widely applied in many areas [3, 4]. Similarly, the SVM-based method, Support Vector Regression (SVR), is an efficient model for regression problems, which guarantees a strong generalization ability. In the literature, the traditional regression methods include General Regression Neural network [5], Multiple Linear Regression [6], and Ridge Regression [7]. Compared with these models, SVR achieves better robustness and generalization, since it considers not only the structural risk but also the experience risk. At present, SVM-based regression models are mainly composed of Support Vector Regression (SVR) [8], Least Square Support Vector Regression (LSSVR) [9], and Twin Support Vector Regression (TSVR) [10]. These models have been widely utilized in various areas, such as stock market forecasting [11, 12], image understanding [13, 14], and pattern recognition [15].The classical SVR [8] obtains the final regression function by solving a quadratic programming problem in which the training error and complexity of the model are minimized in the objective function. Similar to SVM, SVR obtains some good statistical properties. Based on this work, LSSVR makes an improvement by introducing a series of equality constraints. In this way, it only needs to solve a linear system of equations which leads to a much lower computational complexity. Inspired by the twin support vector classification machine [16], Peng [10] proposed a method called TSVR. Unlike the traditional SVR with two parallel hyperplanes, this model uses two nonparallel hyperplanes to construct the final regression function. Due to its flexibility, TSVR has a better generalization ability than the traditional SVR. Moreover, another advantage of TSVR is that the final regression function is obtained by solving two small-sized quadratic programming problems, which also leads to a lower computational burden.It is worth pointing out that, in order to deal with the nonlinear structures in most data sets, these three models first have to use the mapping functions to project the original points into some high-dimensional spaces. Since these mapping functions cannot be directly handled, researchers introduce some kernel functions to solve the dual problems. However, there is no universal rule to automatically choose a suitable kernel function for a given data set. Since the kernel function largely affects the final performances of these kernel-based models, the users have to spend a lot of time and effort in selecting a proper kernel function and its corresponding parameters. This is indeed a very tedious and hard work and damages the applicability of those models. Recently, Luo et al. [17] proposed a kernel-free fuzzy quadratic surface support vector machine (FQSSVM) model which directly generates a quadratic surface for the classification. In this way, it can skip the notorious searching process for a proper kernel function in the classical kernel-based SVM. Hence, this new model can save much effort of the user and greatly improve the total efficiency. Based on this leading work, some extensions of kernel-free SVM have been developed and applied [18–20]. The good results of these models demonstrate that the traditional kernel method can be replaced by the novel quadratic surface method.In this paper, we propose a completely new linear twin quadratic surface support vector machine (LTQSSVR) model. The main contribution of our work can be concluded on three aspects. First, inspired by the cutting-edge idea of the FQSSVM model, we apply two nonparallel quadratic surfaces to build the regression model. In this way, we can successfully avoid the searching process of kernel function selection. Hence, the applicability and efficiency of our model get significantly improved. It is worth noting that, to the best of our knowledge, this is the first time to develop a kernel-free SVM-based regression model. Second, in order to increase the robustness of the model, we incorporate the 1-norm regularization into our model to measure the error between the predicted and actual values. The numerical experiment demonstrates the effectiveness and robustness of this extended model. Third, we equivalently transform the above 1-norm regularization regression model into a linear programming problem. Hence, we can further greatly relieve the computational burden and speed up the efficiency. Compared with those benchmark SVM-based regression models, our new model not only beats them with a slightly better performance but also leads far ahead with a much higher efficiency in building and solving the model.The remaining paper is arranged as follows. Section2 briefly introduces those benchmark works. Then, the TQSSVR and LTQSSVR are proposed in Sections 3 and 4, respectively. After that, we conduct a comprehensive numerical experiment to compare the performances of those models in Section 5. Finally, we summarize this paper in Section 6.


## 2. Related Works

### 2.1. Support Vector Regression

First, we show some notations used in this paper.e denotes the vector of ones with an appropriate dimension, ℝ denotes the set of real numbers, ℝn denotes the n-dimensional vector space, and I denotes the identity matrix with an appropriate dimension. For a matrix A∈ℝn×n, Aij denotes the element in the ith row and jth column of A.For a giving data setT=x1T,y1,…,xlT,yl, where xi∈ℝn are input vectors and yi = {−1, 1} are outputs. Let A=A1,A2,…,AlT, where Ai = xi. Let Y=y1,y2,…,ylT denote the output vector. The basic idea of the SVM model is to find a hyperplane f(x) = wTx + b that separates the training points into two classes with a maximum level of separation [2].Comparatively, the elements in the output vectorY∈ℝl are real numbers for the SVR model. Hence, the targets of SVR and SVM are various. Instead of constructing a proper classifier as that in SVM, SVR aims to find a suitable regressor. In other words, SVR wants to let all the training points as close to the regressor plane as possible, while SVM tries to let them as far away from this plane as possible. Specifically, SVR can be transformed into a classification problem by introducing an ϵ-insensitive tube, which allows a small error in fitting the training data. In this way, any error smaller than ϵ is ignored in the classification model. Let ξ1 and ξ2 be the slack vectors to measure the errors of those samples inside the ϵ-tube. Then SVR can be formulated as the following quadratic programming problem:(1)min12wTw+CeTξ1+eTξ2s.t.Y−Aw+eb≤εe+ξ1,ξ1≥0,Aw+eb−Y≤ϵe+ξ2,ξ2≥0.The interested readers can refer to [8] for more details.Specifically, the first term of the objective function maximizes the margin of this classification problem. And the constraints indicate that all training points should be contained in that softϵ-tube. Here, C is a tradeoff parameter between margin and training errors.


### 2.2. Least Squares Support Vector Regression

LSSVR was first introduced by [9, 21]. Like SVR, the main idea of LSSVR is to seek the decision function in the form of f (x) = Aw + b and include all the points in a small region. Then, the regression function is obtained by solving the following quadratic programming problem [9, 22, 23]:(2)min12wTw+12CξTξ,s.t.Y=Aw+be+ξ,where ξ is the slack vector. Similarly, its objective function is to maximize the margin while its constraints indicate all training points should be close to the regression plane. One significant difference between the above model and SVR model is that solving LSSVR is just equivalent to solving a linear equation set. Therefore, the computational process is relatively simple. On the other side, it should be pointed out that almost all training points contribute to the decision function; hence, LSSVR no longer has the sparsity of SVR.


### 2.3. Twin Support Vector Regression

TSVR is similar to TSVM as it also derives the following pair of nonparallel planes around the data points:(3)f1x=Aw1+b1,f2x=Aw2+b2.However, there are some differences between TSVR and TSVM. First, TSVM considers only one class of data points in each quadratic programming problem, while TSVR uses all data points in its both quadratic programming problems. Second, TSVM finds two hyperplanes such that each plane is close to one class and is as far as possible from the other class, whereas TSVR determines the up- or down-bound function by using only one group of constraints in each quadratic programming problem. Specifically, TSVR is obtained by solving the following pair of quadratic programming problems [10]:(4)min12Y−eϵ1−Aw1+eb1TY−eϵ1−Aw1+eb1+C1eTξ,s.t.Y−Aw1+eb1≥eϵ1−ξ,ξ≥0,min12Y+eϵ2−Aw2+eb2TY+eϵ2−Aw2+eb2+C2eTη,s.t.Aw2+eb2−Y≥eϵ2−η,η≥0,where C1 and C2 > 0 are the parameters and ξ and η are the slack vectors. The final regressor is generated by fx=1/2f1x+f2x. TSVR is approximately four times faster than the standard SVR in theory due to its formulation.


### 2.4. Nonlinear Case and Dual Problems

For a nonlinear case, these three models (SVR, LSSVR, and TSVR) first need to project data points into a higher dimensional space via a mapping functionϕ (x): ℝm⟶ℝd, d > m and then conduct the linear regression wTϕ (A) + b = 0 in this new space [24]. Since the mapping function is difficult to calculate directly, researchers have to introduce a kernel function K (xi, xj) = (ϕ (xi) ⋅ ϕ (xj)) into their dual problems [25].Following this way, the dual problem of SVR can be written as follows:(5)min12α−α∗KA,ATα−α∗+ϵeTα+α∗−YTα−α∗,s.t.eTα−α∗=0,α∗≤Ce,α≥0,where α and α∗ are the Lagrange multipliers that satisfy αiαi∗=0,i=1,2,…,l [8]. Then, the decision function of SVR is of the following form:(6)fx=∑i=1lαi∗−αiKxi·x+b.Similarly, we can obtain the dual problems of LSSVR by introducing the following Lagrangian multiplier:(7)KA,AT+1CIeeT0αb=Y0.The decision function of LSSVR is(8)fx=∑i=1lαi∗Kxi,x+b.Moreover, the dual problem of TSVR is in the following forms:(9)min12αTGGTG−1GTα−fTGGTG−1GTα−fTα,s.t.0⩽α⩽C1e,where(10)G=kA,AT,e,f=Y−eϵ1,U1=w1Tb1,min12γTGGTG−1GTγ+hTGGTG−1GTγ+hTγ,s.t.0⩽γ⩽C2e,where(11)h=Y+eϵ2,U2=w2Tb2.The final regression result of TSVR is determined by the average values off1 (x) and f2 (x) as follows:(12)fx=12f1x+f2x=12kA,X,e·U1+kA,X,e·U2.It is worth noting that there are many kinds of kernel functions such as linear kernel, Gaussian kernel, and polynomial kernel. However, a user cannot find a general guideline to help him/her choose a suitable kernel for a given data set. And the choice of kernel function would largely affect the regression performance. Hence, the time-consuming searching process for a proper kernel function and its corresponding parameters significantly drags down the total efficiency.


## 2.1. Support Vector Regression

First, we show some notations used in this paper.e denotes the vector of ones with an appropriate dimension, ℝ denotes the set of real numbers, ℝn denotes the n-dimensional vector space, and I denotes the identity matrix with an appropriate dimension. For a matrix A∈ℝn×n, Aij denotes the element in the ith row and jth column of A.For a giving data setT=x1T,y1,…,xlT,yl, where xi∈ℝn are input vectors and yi = {−1, 1} are outputs. Let A=A1,A2,…,AlT, where Ai = xi. Let Y=y1,y2,…,ylT denote the output vector. The basic idea of the SVM model is to find a hyperplane f(x) = wTx + b that separates the training points into two classes with a maximum level of separation [2].Comparatively, the elements in the output vectorY∈ℝl are real numbers for the SVR model. Hence, the targets of SVR and SVM are various. Instead of constructing a proper classifier as that in SVM, SVR aims to find a suitable regressor. In other words, SVR wants to let all the training points as close to the regressor plane as possible, while SVM tries to let them as far away from this plane as possible. Specifically, SVR can be transformed into a classification problem by introducing an ϵ-insensitive tube, which allows a small error in fitting the training data. In this way, any error smaller than ϵ is ignored in the classification model. Let ξ1 and ξ2 be the slack vectors to measure the errors of those samples inside the ϵ-tube. Then SVR can be formulated as the following quadratic programming problem:(1)min12wTw+CeTξ1+eTξ2s.t.Y−Aw+eb≤εe+ξ1,ξ1≥0,Aw+eb−Y≤ϵe+ξ2,ξ2≥0.The interested readers can refer to [8] for more details.Specifically, the first term of the objective function maximizes the margin of this classification problem. And the constraints indicate that all training points should be contained in that softϵ-tube. Here, C is a tradeoff parameter between margin and training errors.


## 2.2. Least Squares Support Vector Regression

LSSVR was first introduced by [9, 21]. Like SVR, the main idea of LSSVR is to seek the decision function in the form of f (x) = Aw + b and include all the points in a small region. Then, the regression function is obtained by solving the following quadratic programming problem [9, 22, 23]:(2)min12wTw+12CξTξ,s.t.Y=Aw+be+ξ,where ξ is the slack vector. Similarly, its objective function is to maximize the margin while its constraints indicate all training points should be close to the regression plane. One significant difference between the above model and SVR model is that solving LSSVR is just equivalent to solving a linear equation set. Therefore, the computational process is relatively simple. On the other side, it should be pointed out that almost all training points contribute to the decision function; hence, LSSVR no longer has the sparsity of SVR.


## 2.3. Twin Support Vector Regression

TSVR is similar to TSVM as it also derives the following pair of nonparallel planes around the data points:(3)f1x=Aw1+b1,f2x=Aw2+b2.However, there are some differences between TSVR and TSVM. First, TSVM considers only one class of data points in each quadratic programming problem, while TSVR uses all data points in its both quadratic programming problems. Second, TSVM finds two hyperplanes such that each plane is close to one class and is as far as possible from the other class, whereas TSVR determines the up- or down-bound function by using only one group of constraints in each quadratic programming problem. Specifically, TSVR is obtained by solving the following pair of quadratic programming problems [10]:(4)min12Y−eϵ1−Aw1+eb1TY−eϵ1−Aw1+eb1+C1eTξ,s.t.Y−Aw1+eb1≥eϵ1−ξ,ξ≥0,min12Y+eϵ2−Aw2+eb2TY+eϵ2−Aw2+eb2+C2eTη,s.t.Aw2+eb2−Y≥eϵ2−η,η≥0,where C1 and C2 > 0 are the parameters and ξ and η are the slack vectors. The final regressor is generated by fx=1/2f1x+f2x. TSVR is approximately four times faster than the standard SVR in theory due to its formulation.


## 2.4. Nonlinear Case and Dual Problems

For a nonlinear case, these three models (SVR, LSSVR, and TSVR) first need to project data points into a higher dimensional space via a mapping functionϕ (x): ℝm⟶ℝd, d > m and then conduct the linear regression wTϕ (A) + b = 0 in this new space [24]. Since the mapping function is difficult to calculate directly, researchers have to introduce a kernel function K (xi, xj) = (ϕ (xi) ⋅ ϕ (xj)) into their dual problems [25].Following this way, the dual problem of SVR can be written as follows:(5)min12α−α∗KA,ATα−α∗+ϵeTα+α∗−YTα−α∗,s.t.eTα−α∗=0,α∗≤Ce,α≥0,where α and α∗ are the Lagrange multipliers that satisfy αiαi∗=0,i=1,2,…,l [8]. Then, the decision function of SVR is of the following form:(6)fx=∑i=1lαi∗−αiKxi·x+b.Similarly, we can obtain the dual problems of LSSVR by introducing the following Lagrangian multiplier:(7)KA,AT+1CIeeT0αb=Y0.The decision function of LSSVR is(8)fx=∑i=1lαi∗Kxi,x+b.Moreover, the dual problem of TSVR is in the following forms:(9)min12αTGGTG−1GTα−fTGGTG−1GTα−fTα,s.t.0⩽α⩽C1e,where(10)G=kA,AT,e,f=Y−eϵ1,U1=w1Tb1,min12γTGGTG−1GTγ+hTGGTG−1GTγ+hTγ,s.t.0⩽γ⩽C2e,where(11)h=Y+eϵ2,U2=w2Tb2.The final regression result of TSVR is determined by the average values off1 (x) and f2 (x) as follows:(12)fx=12f1x+f2x=12kA,X,e·U1+kA,X,e·U2.It is worth noting that there are many kinds of kernel functions such as linear kernel, Gaussian kernel, and polynomial kernel. However, a user cannot find a general guideline to help him/her choose a suitable kernel for a given data set. And the choice of kernel function would largely affect the regression performance. Hence, the time-consuming searching process for a proper kernel function and its corresponding parameters significantly drags down the total efficiency.


## 3. Twin Quadratic Surface Support Vector Regression

In this paper, we propose a totally new kernel-free TQSSVR model, which directly generates a quadratic surface for regression in the original space instead of projecting data points into a higher dimensional space. Our new model can overcome the main drawback of those traditional SVM-based regression models by avoiding the selection process. Hence, it has a much higher efficiency and applicability.For a given training data sets, we want to find two quadratic surfaces:(13)g1x=12xTW1x+b1Tx+c1=0,g2x=12xTW2x+b2Tx+c2=0,which determine the ϵ-insensitive down- and up-bound regressors. Following the basic scheme of TSVR, TQSSVR can be reformulated as the following two quadratic programming problems:(14)min12Y−eϵ1−12xTW1x+b1Tx+c1TY−eϵ1−12xTW1x+b1Tx+c1+v1eTξ,s.t.Y−12xTW1x+b1Tx+c1≥eϵ1−ξ,ξ≥0,(15)min12Y+eε2−12xTW2x+b2Tx+c2TY+eε2−12xTW2x+b2Tx+c2+v2eTη,s.t.12xTW2x+b2Tx+c2−Y≥eε2−η,η≥0,where v1andv2 > 0 are the parameters chosen as a priori and ξ and η are the slack vectors. g1 (x) generates the ɛ1-insensitive down-bound regressor, while g2 (x) generates the ɛ2-insensitive up-bound regressor.Note that the first term in the objective function of (14) is the sum of squared distance from the training points to the quadratic surface g1 (x). Therefore, minimizing it equals to minimizing the regression error. Moreover, the constraint indicate that the estimated function g1 (x) is at least ϵ1 away from training points. The second term of the objective function is to minimize the sum of errors. As noted, the loss is zero if and only if g1 (x) is exactly the same as y. Moreover, like SVR, we also assume that we can tolerate at most ϵ1 deviation between g1 (x) and y. The similar explanations apply for problem (15).It is worth pointing out that the above formulations can be further simplified by the following steps. Note that matricesW1 and W2 are symmetric. First, let W¯1 and W¯2 be the vectors formed by taking the m2+m/2 elements of the upper triangle part of matrices W1 and W2, respectively:(16)W¯1=W111,…,W11m,W122,…,W12m,…,W1mmT∈ℝm2+m/2,W¯2=W211,…,W21m,W222,…,W22m,…,W2mmT∈ℝm2+m/2.Then, for each pointxi=x1i,x2i,…,xmiT∈ℝm, we construct an m2+3m/2 vector si as follows:(17)si=12x1ix1i,…,x1ixmi,12x2ix2i,…,x2ixmi,…,12xm−1ixm−1i,xm−1ixmi,12xmixmi,x1i,x2i,…,xmiT∈ℝm2+3m/2,LetS=s1,s2,…,sl∈ℝm2+3m/2×l. Finally, we can define two vectors of variables in the following form:(18)z1=W1¯b1∈ℝm2+3m/2,z2=W2¯b2∈ℝm2+3m/2.It is easy to check that problem (14) and problem (15) can be equivalently reformulated as follows:(19)min12Y−eϵ1−STz1+c1TY−eϵ1−STz1+c1+v1eTξ,s.t.yi−siTz1+c1≥eϵ1−ξi,i=1,2,…,l,ξi≥0,i=1,2,…,l,(20)min12Y+eϵ2−STz2+c2TY+eϵ2−STz2+c+v2eTη,s.t.sjTz2+c2−yj≥eϵ2−ηj,j=1,2,…,l,ηj≥0,j=1,2,…,l.For a pointx, the final regression result of TQSSVR is the mean value of g1 (x) and g2 (x):(21)gx=12g1x+g2x=12sTz1+c1+sTz2+c2.


## 4. Linear Twin Quadratic Surface Support Vector Regression

In the above model, the error between the predicted value and the actual value is measured by the 2-norm. It is noting that 2-norm is sensitive to those points which are far away from the regressor and may amplify the influence of those error points, especially in the case with outliers or mislabeled information. Compared with 2-norm, 1-norm is more robust and insensitive to the errors [26, 27].In this section, in order to increase the robustness, we will introduce the 1-norm regularization condition and transform problem (19) and problem (20) into two linear programming problems. This not only enhances the performance but also greatly improves the computational efficiency. Hence, we extended our model by incorporating the 1-norm regularization as follows:(22)min12Y−eϵ1−STz1+c1T1+v1eTξ,s.t.yi−siTz1+c1≥eϵ1−ξi,i=1,2,…,l,ξi≥0,i=1,2,…,l,min12Y+eϵ2−STz2+c2T1+v2eTη,s.t.sjTz2+c2−yj≥eϵ2−ηj,j=1,2,…,l,ηj≥0,j=1,2,…,l.For a vectora∈ℝn, ‖a‖1 is the sum of the absolute values of all its elements, and the sum of the absolute values of each element is greater than or equal to the absolute value of the sum. Let s1 = ∥Y − eϵ1 − (sTz1 + c1)∥1, s2 = ∥Y + eϵ2 − (sTz2 + c2)∥1. Then, we have(23)−s1≤∑i=1lyi−ϵ1−siTz1+c1≤s1,−s2≤∑j=1lyj+ϵ2−sjTz2+c2≤s2.Based on these inequalities, we can relax and reformulate those two twin regression problems as the following two linear programming problems:(24)min12s1+v1eTξ,s.t.yi−siTz1+c1≥eϵ1−ξi,i=1,2,…,l,−s1≤∑i=1lyi−ϵ1−siTz1+c1≤s1,ξi≥0,i=1,2,…,l.min12s2+v2eTη,s.t.sjTz2+c2−yj≥eϵ2−ηj,j=1,2,…,l,−s2≤∑j=1lyj+ϵ2−sjTz2+c2≤s2,ηj≥0,j=1,2,…,l.It is worth pointing out that, the linear structure of these two reformulations greatly improves the computational efficiency. Moreover, the existence of the 1-norm distance opposed to the 2-norm distance in TQSSVR makes the new model less sensitive to noise or large errors.To further illustrate the robustness of 1-norm, we give an example here. The points are generated byyi=2xiTxi+5xi+ξi, where xi uniformly distributed between [0, 1] and ξi belong to a Gaussian noise N (0, 0.152). For simplicity, we set ϵ1 = ϵ2 = 0. The final regressors are derived by LTQSSVR and TQSSVR. Then, to test the robustness of TQSSVR and LTQSSVR, we added some outliers to the data. As shown in Figure 1, it can be easily seen from the left graph that two regressors are almost overlapping for the data without some noises. However, for the data with outliers, TQSSVR is obviously biased while LTQSSVR is quite robust.Figure 1
Regressors of LTQSSVR and TQSSVR withc1 = c2 = 2−2. The graph (a) shows the regressors of the data without large noises; the graph (b) shows the regressors calculated on the data with outliers.
(a)
(b)


## 5. Numerical Experiment and Discussion

To investigate the performance of our proposed LTQSSVR and compare it with other benchmark SVM-based regression methods, we conducted a comprehensive numerical experiment. The comparison list includes Ridge Regression, Linear Regression, nu-SVR,ϵ-SVR, LSSVR, TSVR, LTSVR, and our LTQSSVR. LTSVR was proposed by Xu in [28] which adds 1-norm to TSVR. Besides, two different types of artificial data sets and twelve various benchmark real data sets were used in the experiment.All computational tests were executed via MATLAB (R2016a) on a personal laptop with a system configuration as Intel p4 processor (1.8 GHZ) and 4 GB usable RAM. All the corresponding models were solved by the function “quadprog.m” or “linprog.m” in the MATLAB toolbox (available fromhttps://pan.baidu.com/s/1QRX40tVcnO–f4bz8c0–58Q.password:itiz).


### 5.1. Kernel Function and Parameters Selection

For the kernel-based models, since Gaussian kernel functionK (xi, xj) = exp(−∥xi − xj∥2)/σ2 is the most commonly used one, it was applied in this paper. The corresponding optimal kernel parameter σ in nu-SVR, ϵ-SVR, LSSVR, TSVR, and LTSVR was selected from the set {2i∣i = −4, −3, …, 4, 5}. It is worth pointing out that we only considered the Gaussian kernel for those kernel-based models in the experiment. If we compare other kernel functions in the searching process, the total efficiencies of those models would be tremendously reduced.Moreover, we setc1 = c2 = c and ϵ1 = ϵ2 = ϵ in our experiment. Specifically, the optimal penalty parameter c for these models was selected from the set {2i∣i = −4, −3, …, 4, 5}, and the insensitive parameter ϵ was chosen from the set {0.001, 0.01, 0.1, 0.2}. Note that all these parameters were determined by the fivefold cross validation. That is, the data set was randomly split into five subsets. In each time, one of those subsets was reserved as a testing set while other four subsets were used together as a training set. For every data set, this process was repeated ten times. And all the results are the average values of these ten tests.In each cross validation, we use the following four indicators in [10, 28] as the criteria for selecting the optimal parameters and verify the performance of eight models:(1)
RMSE: Root Mean Squared Error, defined asRMSE=1/N∑i=1Ny^i−yi2. It is used to measure the deviation between the observed value and the true value. Because RMSE uses the average error, it is sensitive to abnormal points. If the regression value of a certain point is not reasonable, its error is relatively large, which has a great impact on the value of RMSE. In general, the smaller the RMSE is, the more accurate the prediction results will be.(2)
MAE: Mean Absolute Error, defined asMAE=1/Nyi−y^i. It is the mean of absolute error which can reflect the actual situation of predicted error.(3)
SSE/SST: SSE (Sum of Squares due to Error) measures the difference between the actual value and the predict value, which represents the part interpreted by the regression equation. SST (Sum of squares for total) measures the deviation ofy, which represents the total change of data. In most cases, a smaller SSE/SST means a better consistency between estimates and actual values.(4)
SSR/SST: SSR (Sum of Squares due to Regression) measures the difference between estimate values and the mean value ofy, which represents the part not explained by the regression equation. The larger the SSR, the more statistic information it captures from the test sample. In order to obtain a small SSE/SST, it is usually accompanied by an increase in SSR/SST.


### 5.2. Artificial Data Sets

First, we followed the traditional way in [10, 28] to generate a classical type of artificial data sets in the experiment. This is a kind of 2-D artificial data sets which is obtained by the equation y=sincx=sinπx/πx,x∈−5,5. In reality, data sets always contain noise points. In order to check the performance of our model in this situation, two kinds of noises including the Gaussian noise and the uniformly distributed noise were added. Specially, four types of training points were generated as follows:(25)yi=sinπxiπxi+ξi,ξi∼U0,0.15,yi=sinπxiπxi+ξi,ξi∼U0,0.3,yi=sinπxiπxi+ξi,ξi∼N0,0.152,yi=sinπxiπxi+ξi,ξi∼N0,0.32,where U[a, b] represents the uniform distribution in the range of [a, b] and N[0, d2] represents the normal distribution with a mean of 0 and a variance of d2. To avoid bias comparison, we randomly generated 10 groups of independent noise samples for each type of noise. Following the traditional way in [10, 28], each group contains 600 training samples and 400 test samples. And the test points were generated by function sin c (x) without noise. The corresponding results of eight algorithms Ridge Regression, Linear Regression, nu-SVR, ϵ-SVR, LSSVR, TSVR, LTSVR, and LTQSSVR are shown in Table 1.Table 1
Results of Ridge Regression (RR), Linear Regression (LR), nu-SVR,ϵ-SVR, LSSVR, TSVR, LTSVR, and LTQSSVR on benchmark data sets with RBF kernel.
Noise
Alg
RMSE
MAE
SSE/SST
SSR/SST
Time (s)
c
ϵ
σ
Type 1
RR
0.3102 ± 0.0115
0.1804 ± 0.0066
1.1385 ± 0.0163
0.1389 ± 0.0162
0.0434
LR
0.3106 ± 0.0115
0.1805 ± 0.0066
1.1410 ± 0.0170
0.1410 ± 0.0170
0.0004
nu-SVR
635.0782 ± 2.0081e + 03
257.3113 ± 813.5255
4.7260e + 07 ± 1.4945e + 08
4.7260e + 07 ± 1.4945e + 08
47.0139
22
0.001
2–1
ϵ-SVR
158.5021 ± 350.9712
91.1181 ± 197.1021
1.6476e + 06 ± 3.9456e + 06
1.6476e + 06 ± 3.9456e + 06
28.7549
22
0.001
2–1
LSSVR
0.6531 ± 0.0155
0.5141 ± 0.0144
4.5383 ± 0.4865
4.3325 ± 0.5131
830.3177
25
2–1
TSVR
0.6544 ± 0.0270
0.5314 ± 0.0176
4.3852 ± 0.4677
4.2048 ± 0.4878
2024.7431
2–4
0.001
2–1
LTSVR
0.5870 ± 0.0372
0.4839 ± 0.0247
3.7994 ± 0.2871
3.8503 ± 0.1351
26531.2
22
0.01
22
LTQSSVR
0.3961 ± 0.0187
0.2851 ± 0.0239
1.7631 ± 0.1997
0.3917 ± 0.1451
96.3907
2–1
0.01
Type 2
RR
0.3432 ± 0.0102
0.2034 ± 0.0065
1.1693 ± 0.0168
0.1695 ± 0.0172
0.0027
LR
0.3434 ± 0.0110
0.2034 ± 0.0075
1.1705 ± 0.0222
0.1705 ± 0.0222
0.0024
nu-SVR
588.4 ± 186.1
140.7 ± 444.7
3.4195e + 07 ± 1.0814e + 08
3.4196e + 07 ± 1.0814e + 08
53.9877
2–2
0.001
20
ϵ-SVR
107.6 ± 238.9
87.9 ± 193.4
6.4387e + 05 ± 1.6125e + 06
6.4345e + 05 ± 1.6114e + 06
43.9494
2–2
0.001
20
LSSVR
0.6406 ± 0.0257
0.5193 ± 0.0233
4.3594 ± 0.4286
4.0586 ± 0.3830
959.2185
20
2–2
TSVR
0.6338 ± 0.0397
0.5123 ± 0.0321
4.1191 ± 0.6138
3.7309 ± 0.5828
5657.1416
2–4
0.2
20
LTSVR
0.6988 ± 0.1573
0.5541 ± 0.1162
4.8817 ± 1.5382
5.0229 ± 1.9708
4583.9875
2–2
0.001
24
LTQSSVR
0.3979 ± 0.0321
0.2817 ± 0.0225
1.7730 ± 0.2054
0.3920 ± 0.1574
93.6575
2–1
0.01
Type 3
RR
0.3431 ± 0.0102
0.2033 ± 0.0068
1.1688 ± 0.0171
0.1690 ± 0.0175
0.0038
LR
0.3431 ± 0.0107
0.2033 ± 0.0072
1.1688 ± 0.0190
0.1688 ± 0.0190
0.0003
nu-SVR
165.9 ± 524.4
93.1 ± 293.7
2.8467e + 06 ± 9.0020e + 06
2.8464e + 06 ± 9.0012e + 06
26531.2
23
0.001
2–1
ϵ-SVR
0.1545 ± 0.0015
0.1536 ± 0.0017
0.2373 ± 0.0130
1.2355 ± 0.0176
30.5638
20
0.001
20
LSSVR
0.6531 ± 0.0155
0.5141 ± 0.0144
4.5383 ± 0.4865
4.3325 ± 0.5131
1200.9825
25
2–1
TSVR
0.6273 ± 0.0062
0.4944 ± 0.0094
4.0209 ± 0.0178
3.6071 ± 0.0231
2887.8195
2–4
0.1
20
LTSVR
0.6852 ± 0.0327
0.5311 ± 0.0301
5.4208 ± 1.1646
5.4496 ± 1.3123
4735.0025
22
0.001
22
LTQSSVR
0.3935 ± 0.0160
0.3310 ± 0.0148
1.7354 ± 0.1279
0.4470 ± 0.0608
100.1174
2–2
0.01
Type 4
RR
0.3431 ± 0.0103
0.2033 ± 0.0071
1.1687 ± 0.0174
0.1687 ± 0.0175
0.0037
LR
0.3431 ± 0.0104
0.2033 ± 0.0070
1.1687 ± 0.0183
0.1687 ± 0.0183
0.0003
nu-SVR
542.5 ± 1429.7
235.1 ± 571.1
2.1116e + 07 ± 6.3937e + 07
2.1115e + 07 ± 6.3936e + 07
26531.2
23
0.001
2–1
ϵ-SVR
0.0811 ± 0.0070
0.0769 ± 0.0070
0.0658 ± 0.0119
0.9687 ± 0.0149
9.4324
23
0.001
2–1
LSSVR
0.6531 ± 0.0155
0.5141 ± 0.0144
4.5383 ± 0.4865
4.3325 ± 0.5131
1200.9825
25
2–1
LSSVR
0.6442 ± 0.0173
0.5121 ± 0.0142
4.4135 ± 0.4501
4.2044 ± 0.4833
12.1205
23
2–2
TSVR
0.6064 ± 0.0099
0.4784 ± 0.0060
3.7604 ± 0.2137
3.2961 ± 0.2944
737.7178
2–4
0.2
20
LTSVR
0.6426 ± 0.0481
0.5 ± 0.0140
4.1282 ± 0.3409
4.0077 ± 0.3712
9533.4572
22
0.01
2–2
LTQSSVR
0.3781 ± 0.0257
0.2980 ± 0.0298
1.6028 ± 0.1709
0.3151 ± 0.1171
113.2787
2–1
0.01From Table1, we can see that Ridge Regression achieves the best results by obtaining the minimum RMSE and MAE values for noises of Type 1 and Type 2, and ϵ-SVR obtains the best for noises of Type 3 and Type 4. Besides, LTQSSVR take a stable performance for most data sets which has a better robustness for noises. And it can also be seen from Table 1 that Linear Regression is the fastest one among all algorithms. However, Linear Regression is only suitable for the cases with linear separable data.In order to verify the universal ability of eight models, we used another type of artificial data sets in [20] to test the performances of eight models. Now, we show how to obtain this type of artificial data sets. First, we used different matrices U and vectors c to get various quadratic surfaces 1/2XTUX+cTX=0, where each element of U and c was randomly selected from the domain [−10, 10]. Then, we randomly generated points on both sides of the quadratic surface. Similarly, we generated 10 independent groups of samples. Each group includes 60% training samples with noise and 40% test points without noise. The corresponding results are shown in Table 2.Table 2
Results of Ridge Regression (RR), Linear Regression (LR), nu-SVR,ϵ-SVR, LSSVR, TSVR, LTSVR, and LTQSSVR on quadratic artificial data sets.
Data sets num × fea
Algorithm
RMSE
MAE
SSE/SST
SSR/SST
Time (s)
c
ϵ
σ
AR130 × 10
RR
556.9348 ± 64.6680
472.5440 ± 68.1756
0.5760 ± 0.3548
0.7606 ± 0.3899
0.0037
LR
399.4814 ± 100.0622
328.3254 ± 88.2998
0.3022 ± 0.1929
1.1167 ± 0.4799
2.0278
nu-SVR
864.5692 ± 197.0818
673.1263 ± 149.9371
1.1580 ± 0.2129
0.1580 ± 0.2129
0.1191
25
0.001
20
ϵ-SVR
865.8422 ± 198.0437
675.8055 ± 151.8056
1.1606 ± 0.2109
0.1606 ± 0.2109
0.1090
25
0.001
2–4
LSSVR
634.8957 ± 256.3804
520.7602 ± 208.0142
0.5531 ± 0.4678
0.3691 ± 0.1874
1.511
22
23
TSVR
509.4136 ± 229.8878
416.2661 ± 194.7960
0.7154 ± 0.3129
0.3126 ± 0.4475
14.91
2–4
0.1
23
LTSVR
875.5967 ± 178.9845
724.9856 ± 141.0931
0.9595 ± 0.3107
0.2126 ± 0.1810
4.2708
2–3
0.2
23
LTQSSVR
233.7883 ± 74.5269
184.9046 ± 58.4456
0.0698 ± 0.0375
0.9108 ± 0.1189
3.397
2–1
0.01
AR250 × 20
RR
772.9401 ± 94.6686
644.1068 ± 80.0471
0.7422 ± 0.2818
1.2858 ± 0.3805
0.0002
LR
804.7534 ± 205.3123
660.0124 ± 152.2902
0.7986 ± 0.3627
1.4378 ± 0.5091
22.6734
nu-SVR
1019.9 ± 164.4628
828.6489 ± 138.9097
1.2486 ± 0.4260
0.2486 ± 0.4260
0.1761
2–4
0.001
2–2
ϵ-SVR
1013.4 ± 147.9270
832.0854 ± 131.6954
1.2332 ± 0.4159
0.2332 ± 0.4159
0.1979
25
0.001
2–4
LSSVR
876.2712 ± 35.8862
656.5610 ± 33.6003
0.3968 ± 0.0538
0.5012 ± 0.0699
0.7977
25
25
TSVR
825.4982 ± 192.4043
662.9940 ± 167.3309
0.2658 ± 0.1259
0.7697 ± 0.2515
18.8629
2–1
0.01
25
LTSVR
2370 ± 767.3638
2016.2 ± 758.3574
2.1851 ± 1.4099
1.3458 ± 1.2179
9.1719
2–3
0.2
24
LTQSSVR
503.2282 ± 111.1342
391.0349 ± 93.5383
0.1362 ± 0.0736
0.9079 ± 0.0424
20.7707
20
0.2
AR3100 × 30
RR
951.3805 ± 112.4494
750.8384 ± 88.4491
0.1356 ± 0.0377
0.8898 ± 0.0631
0.0072
LR
922.5558 ± 132.01183
741.5420 ± 92.6491
0.1268 ± 0.0358
0.9371 ± 0.0548
0.5144
nu-SVR
2685.5 ± 229.4038
2179.9 ± 162.6985
1.0437 ± 0.0377
0.0437 ± 0.0377
3.7895
25
0.001
2–4
ϵ-SVR
2701.6 ± 228.9889
2197.2 ± 166.9867
1.0571 ± 0.0578
0.0571 ± 0.0578
0.5144
25
0.001
2–4
LSSVR
1098 ± 121.5927
889.7450 ± 74.5102
0.0923 ± 0.0197
0.7229 ± 0.0703
2.2872
25
25
TSVR
758.5976 ± 149.9849
601.7016 ± 128.2513
0.1026 ± 0.0611
0.8481 ± 0.0975
48.9992
25
0.2
25
LTSVR
3841.9 ± 2813.9
3325.2 ± 2771.0
1.7216 ± 2.7258
1.4169 ± 2.4844
64.7086
2–4
0.2
25
LTQSSVR
863.4386 ± 114.7082
664.7501 ± 96.2545
0.1590 ± 0.0540
0.8574 ± 0.1567
144.6868
24
0.001
AR4500 × 30
RR
804.0858 ± 29.61234
620.5212 ± 29.0568
0.1167 ± 0.0122
0.8591 ± 0.0527
3.5505
LR
741.3826 ± 28.4248
589.9683 ± 27.1126
0.0992 ± 0.0108
0.9021 ± 0.0496
6.5738
nu-SVR
2367.5 ± 67.0977
1864.1 ± 58.4901
1.0067 ± 0.0074
0.0067 ± 0.0074
6.3427
25
0.001
2–4
ϵ-SVR
2370.9 ± 68.4916
1867.5 ± 60.7537
1.0096 ± 0.0128
0.0096 ± 0.0128
9.1899
22
0.001
2–4
LSSVR
579.1923 ± 69.7749
460.4036 ± 43.4087
0.0656 ± 0.0110
0.8806 ± 0.0369
46.9875
25
25
TSVR
489.2266 ± 23.5683
391.9541 ± 24.6263
0.0612 ± 0.0012
0.9893 ± 0.0012
519.6265
25
0.01
25
LTSVR
7321 ± 1491
6689.5 ± 859.5
−
−
2714.643
2–4
0.2
23
LTQSSVR
529.4326 ± 59.2533
473.1416 ± 2.9623
0.0607 ± 0.0122
1.001 ± 0.0011
442.1526
25
0.1
AR51000 × 10
RR
355.4752 ± 13.4051
283.0450 ± 10.6422
0.1897 ± 0.0117
0.6229 ± 0.0495
0.0018
LR
231.9386 ± 8.2504
181.3980 ± 6.9325
0.0809 ± 0.0064
0.9459 ± 0.0603
1.8952
nu-SVR
817.6871 ± 22.6287
651.5018 ± 13.5489
1.0028 ± 0.0031
0.0028 ± 0.0031
22.8603
25
0.001
2–4
ϵ-SVR
817.6872 ± 22.6285
651.5021 ± 13.5487
1.0028 ± 0.0031
0.0028 ± 0.0031
21.8329
25
0.001
2–4
LSSVR
43.43 ± 1.26
31.09 ± 0.79
0.002 ± 0.0001
0.9815 ± 0.0039
115.985
25
24
TSVR
8.25 ± 1.17
5.65 ± 0.83
0.0001 ± 2.33E − 05
0.9967 ± 0.0033
32081.806
2–4
0.01
25
LTSVR
435.58 ± 431.67
347.57 ± 344.48
0.5135 ± 0.5134
0.5134 ± 0.4865
5105.084
2–3
0.01
25
LTQSSVR
0.0758 ± 0.0191
0.0731 ± 0.0212
1.34E − 08 ± 6.24E − 09
1.000 ± 1.98E − 05
1546.984
22
0.2
AR62000 × 10
RR
348.4587 ± 5.2498
277.5702 ± 4.6563
0.1884 ± 0.0089
0.6374 ± 0.0228
0.0029
LR
232.5254 ± 5.7657
180.2640 ± 4.9633
0.0840 ± 0.0066
0.9209 ± 0.0215
8.6790
nu-SVR
804.4539 ± 15.5199
644.4631 ± 13.4341
1.0033 ± 0.0058
0.0033 ± 0.0058
68.0419
25
0.001
2–4
ϵ-SVR
804.4538 ± 15.5199
644.4633 ± 13.4341
1.0033 ± 0.0058
0.0033 ± 0.0058
95.3783
25
0.001
2–4
LSSVR
29.2831 ± 0.1239
19.5835 ± 0.5175
0.0016 ± 3.1304e − 05
0.9765 ± 6.0976e − 04
920.0551
25
24
TSVR
10.3615 ± 1.5314
7.2374 ± 1.4115
0.0001 ± 1.96E − 05
0.9963 ± 0.0017
10673.192
2–2
0.01
25
LTSVR
775.3161 ± 133.3652
625.1344 ± 119.2564
1.2355 ± 0.2456
0.3495 ± 0.1316
16739.2
2–4
0.01
2–4
LTQSSVR
0.1074 ± 0.0410
0.1032 ± 0.0416
3.06e − 08 ± 2.23e − 08
1.000 ± 1.3591e − 05
7536.641
25
0.01
AR73000 × 5
RR
294.1609 ± 3.7125
237.5383 ± 2.6317
0.1974 ± 0.0039
0.4435 ± 0.0082
0.1503
LR
143.8636 ± 2.7915
110.1801 ± 1.7634
0.0472 ± 0.0012
0.9656 ± 0.0308
6.5705
nu-SVR
560.7090 ± 10.0120
442.1160 ± 6.1240
0.7170 ± 0.0127
0.0643 ± 0.0074
1134.1517
25
0.001
2–4
ϵ-SVR
560.7196 ± 10.0115
442.1315 ± 6.1242
0.7170 ± 0.0127
0.0643 ± 0.0074
170.9971
25
0.001
2–4
LSSVR
10.4242 ± 0.5163
6.6254 ± 0.2775
0.0002 ± 2.23083e − 05
0.9964 ± 0.0003
2959.8848
25
24
TSVR
−
−
−
−
−
2–4
0.01
25
LTSVR
−
−
−
−
−
2–3
0.01
25
LTQSSVR
0.0667 ± 0.0049
0.0664 ± 0.0051
1E − 08 ± 1.39E − 09
1.000 ± 1.98E − 05
1546.984
22
0.2
AR85000 × 3
RR
121.4335 ± 1.9984
98.4936 ± 1.4719
0.2308 ± 0.0034
0.5152 ± 0.0083
0.0044
LR
104.7501 ± 2.3667
78.1530 ± 1.0814
0.1717 ± 0.0049
0.8325 ± 0.0107
8.9268
nu-SVR
7.7250 ± 0.8782
1.3397 ± 0.1542
0.0009 ± 2.0694e − 04
0.9892 ± 0.0018
1219.7795
25
0.001
2–4
ϵ-SVR
7.7357 ± 0.8790
1.3571 ± 0.1540
0.0009 ± 2.0743e − 04
0.9888 ± 0.0018
438.3024
25
0.001
2–4
LSSVR
2.0616 ± 0.3076
0.8998 ± 0.3241
6.7308e − 05 ± 1.9711e − 05
0.9967 ± 0.0016
150667.6
24
24
TSVR
−
−
−
−
−
2–4
0.01
25
LTSVR
−
−
−
−
−
2–3
0.01
25
LTQSSVR
0.0716 ± 0.0240
0.0710 ± 0.0245
8.4658E − 08 ± 6.31E − 09
1.000 ± 1.86311E − 06
22332.86
25
0.2From Table2, we can see that LTQSSVR still achieves the best performance in seven cases among total eight cases by obtaining the smallest RMSE, MAE, and SSE/SST values. Compared with TSVR, LTSVR improves the training results by introducing 1-norm instead of the original 2-norm. But for large-scaled data sets, its advantages are not significant. Among these methods, Ridge Regression has the shortest processing time, followed by Linear Regression, nu-SVR, and ϵ-SVR, which lack robustness to noises. Moreover, the experimental results with different scaled data sets show that LTQSSVR has better performance with large-scaled but small-dimension data sets. Besides, we also show the box plot of RMSE values in Figure 2. It is easy to check that LTQSSVR are concentrated and obtain a smallest average, Hence LTQSSVR is robust in most cases.Figure 2
Boxplots of RMSE for artificial data sets: (a) artificial data set 1; (b) artificial data set 3; (c) artificial data set 5.
(a)
(b)
(c)


### 5.3. Benchmark Data Sets

In this part, we use 12 benchmark data sets from the UCI Repository to test these four models (available fromhttps://archive.ics.uci.edu/ml/datasets.php). This list includes Auto MPG, Oring, Wis, Slump, Haberman, and so on. The detailed information of these data sets is summarized in Table 3.Table 3
Detailed information about UCI data sets.
Data set
Number of points
Number of features
Auto mpg
392
8
Oring
23
5
Container-CC
15
3
Daily demand
60
13
Haber man
306
4
Wis
194
30
Hayes roth
132
6
Real estate valuation
414
7
Slump
103
8
Ozone
517
11
Body fat
252
15
Computer hardware
209
8To eliminate the overwhelming dominance between features, we normalized the features of all data to [0,1] before the training. The experimental results of eight models are summarized in Table4. For each result, the first term denotes the average value of ten times while the second term denotes the standard deviation. From Table 4, we can see that the LTQSSVR obtains good results in most of the cases. Compared with LSSVR and TSVR, the new model greatly improves the computational efficiency on the basis of ensuring accuracy.Table 4
Results of Ridge Regression (I), Linear Regression (II), nu-SVR (III),ϵ-SVR (IV), LSSVR (V), TSVR (VI), LTSVR (VII), and LTQSSVR (VIII) on benchmark data sets with RBF kernel.
Data sets
Alg
RMSE
MAE
SSE/SST
SSR/SST
Time(s)
c
ϵ
σ
Hayes roth132 × 6
I
0.6253 ± 0.0601
0.5191 ± 0.0515
0.6914 ± 0.1324
1.0377 ± 0.1817
0.005
II
0.5495 ± 0.0570
0.4486 ± 0.0487
0.5335 ± 0.1090
0.5345 ± 0.1049
0.0042
III
1.2221 ± 1.1811
0.7785 ± 0.6059
4.2785 ± 7.4431
4.4979 ± 7.2225
5.8376
23
0.001
2–4
IV
2.5118 ± 3.8211
1.2269 ± 1.5683
33.4225 ± 69.4087
33.8838 ± 70.3782
0.8978
22
0.001
2–4
V
0.5787 ± 0.0870
0.4394 ± 0.0832
0.4919 ± 0.1487
0.6280 ± 0.1470
4.0312
25
22
VI
0.6157 ± 0.1650
0.4700 ± 0.0966
0.6656 ± 0.4210
1.0015 ± 0.7287
74.9323
2–4
0.2
25
VII
0.5999 ± 0.1008
0.4450 ± 0.0773
0.5866 ± 0.1730
0.9682 ± 0.3809
208.5302
2–3
0.01
25
VIII
0.5665 ± 0.0715
0.4394 ± 0.0696
0.5219 ± 0.1174
0.7902 ± 0.2385
6.7313
22
0.2
Oring23 × 5
I
3.7095 ± 1.5179
2.9936 ± 1.2695
0.5474 ± 0.7138
0.9454 ± 0.5311
0.0026
II
3.7096 ± 1.5179
2.9936 ± 1.2695
0.5474 ± 0.7138
0.9455 ± 0.5311
0.0086
III
40.3245 ± 65.7156
26.3581 ± 43.9478
166.4272 ± 367.8244
166.9181 ± 374.4771
0.7148
24
0.001
20
IV
511.2011 ± 1.4669e + 03
245.0591 ± 680.3442
7.0421e + 04 ± 2.2234e + 05
7.0542e + 04 ± 2.2272e + 05
0.3195
25
0.001
2–3
V
4.7860 ± 1.8800
3.8188 ± 1.3995
0.8257 ± 1.0693
0.8409 ± 0.7116
1.6519
21
2–2
VI
22.4245 ± 44.2676
13.8426 ± 25.2836
46.6614 ± 135.2396
49.9042 ± 143.4053
15.3404
23
0.2
2–2
VII
28.9964 ± 74.4129
20.1901 ± 50.8615
92.7013 ± 289.5774
90.7100 ± 282.2787
28.7152
2–3
0.01
23
VIII
5.3305 ± 1.9512
4.1148 ± 1.6144
1.0038 ± 1.3476
1.4496 ± 0.7396
1.5079
21
0.2
Container15 × 3
I
0.3012 ± 0.0677
0.2792 ± 0.0522
9.2105 ± 5.1356
6.3104 ± 5.0631
0.0027
II
0.1787 ± 0.0254
0.1574 ± 0.0279
3.1271 ± 1.3823
1.8975 ± 1.6807
0.0068
III
0.2468 ± 0.3123
0.2143 ± 0.2597
1.8268 ± 0.8391
0.8461 ± 0.8428
0.1491
24
0.001
2–4
IV
0.1541 ± 0.0395
0.1359 ± 0.0370
2.5484 ± 1.2829
1.2351 ± 1.0699
0.2179
2–4
0.001
25
V
0.1458 ± 0.0542
0.1343 ± 0.0514
2.1108 ± 1.1380
1.6074 ± 0.9613
1.4901
2–4
25
VI
0.3147 ± 0.2579
0.2645 ± 0.1928
8.2127 ± 12.0510
6.3811 ± 11.9280
14.4593
2–3
0.2
21
VII
0.5611 ± 1.0092
0.4895 ± 0.8664
44.1796 ± 114.0210
44.5070 ± 119.0015
16.4264
2–4
0.001
22
VIII
0.2567 ± 0.1222
0.2277 ± 0.1040
6.2074 ± 4.6990
3.5947 ± 3.4960
0.2904
2–2
0.1
Daily de60 × 13
I
28.1321 ± 4.1695
22.2524 ± 3.1676
0.2260 ± 0.1701
1.2616 ± 0.3249
0.0033
II
9.7112e − 14 ± 2.0289e − 14
8.1476e − 14 ± 1.9473e − 14
2.8124e − 30 ± 2.2297e − 30
1 ± 8.4956e − 16
0.0022
III
46.5750 ± 18.2514
26.3692 ± 9.0850
0.2679 ± 0.1062
0.5115 ± 0.2421
0.2335
25
0.001
2–1
IV
42.4564 ± 13.7516
23.9045 ± 7.3832
0.2352 ± 0.0667
0.4670 ± 0.1249
0.2396
25
0.001
2–1
V
14.4057 ± 7.2369
9.2370 ± 4.5731
0.0292 ± 0.0176
0.8718 ± 0.1066
2.3495
25
21
VI
2.4664 ± 2.1341
1.5724 ± 0.9919
9.5111e − 04 ± 0.0013
0.9875 ± 0.0239
40.7319
2–4
0.1
22
VII
0.0782 ± 0.0778
0.0486 ± 0.0355
9.4084e − 07 ± 1.5290e − 06
0.9999 ± 5.0720e − 04
32.5313
2–4
0.2
25
VIII
10.8529 ± 7.5422
6.9757 ± 3.9298
0.0232 ± 0.0274
1.0648 ± 0.1351
9.7285
23
0.001
Haber man306 × 4
I
0.5854 ± 0.0482
0.4515 ± 0.0431
1.8093 ± 0.3646
1.1981 ± 0.2128
0.0249
II
0.4259 ± 0.0210
0.3564 ± 0.0123
0.9495 ± 0.1062
0.0889 ± 0.0590
0.0169
III
0.4877 ± 0.0236
0.4818 ± 0.0195
1.1689 ± 0.1149
0.1572 ± 0.0958
13.646
2–1
0.001
25
IV
0.4592 ± 0.0262
0.3539 ± 0.0491
1.1258 ± 0.1258
0.2377 ± 0.1436
7.5302
2–1
0.001
23
V
0.4176 ± 0.0236
0.3422 ± 0.0181
0.9156 ± 0.0695
0.1285 ± 0.0547
33.9049
2–1
2–4
VI
0.4655 ± 0.1198
0.3290 ± 0.0410
1.1776 ± 0.6418
0.4589 ± 0.8390
301.7816
2–2
0.1
22
VII
0.3273 ± 0.2326
0.3129 ± 0.0410
1.2283 ± 0.2454
0.7573 ± 0.3473
6716.377
2–3
0.01
24
VIII
0.4446 ± 0.0454
0.3077 ± 0.0310
1.0407 ± 0.1664
0.3038 ± 0.1428
112.8566
22
0.1
Wis194 × 30
I
0.0088 ± 0.0015
0.0067 ± 7.9729e − 04
0.2061 ± 0.0867
1.2493 ± 0.1829
0.0098
II
0.0062 ± 0.0017
0.0043 ± 8.0280e − 04
0.1050 ± 0.0512
1.0385 ± 0.1263
0.0029
III
0.0091 ± 0.0029
0.0057 ± 0.0012
0.1792 ± 0.0688
0.6678 ± 0.1592
8.7957
22
0.001
2–4
IV
0.0484 ± 0.0048
0.0453 ± 0.0052
6.8505 ± 3.3307
5.8505 ± 3.3307
0.1361
2–4
0.001
2–4
V
0.0415 ± 0.0123
0.0277 ± 0.0061
0.1043 ± 0.0622
0.8787 ± 0.2353
15.4689
25
22
VI
0.0057 ± 0.0013
0.0041 ± 6.9250e − 04
0.0768 ± 0.0250
0.9603 ± 0.1223
151.3218
2–4
0.2
23
VII
0.0060 ± 0.0013
0.0044 ± 6.0337e − 04
0.0858 ± 0.0274
1.0015 ± 0.1607
877.5655
2–4
0.1
24
VIII
0.0087 ± 0.0016
0.0057 ± 9.3220e − 04
0.1977 ± 0.1165
1.1763 ± 0.2253
83.3716
21
0.001
Auto mpg392 × 8
I
0.6124 ± 0.0654
0.4813 ± 0.0568
0.5958 ± 0.1036
0.5781 ± 0.1547
0.0067
II
0.5898 ± 0.0655
0.4632 ± 0.0588
0.5520 ± 0.0947
0.5221 ± 0.1290
0.0018
III
2.1847 ± 2.5549
1.1657 ± 1.2357
65.4009 ± 126.7810
65.9644 ± 128.8542
36.0491
20
0.001
25
IV
1.0737 ± 2.2710
0.6030 ± 1.1399
35.8631 ± 110.4902
36.6985 ± 112.1304
7.6174
23
0.001
20
V
0.5598 ± 0.0647
0.3760 ± 0.0577
0.5070 ± 0.0846
0.5995 ± 0.1304
67.7641
24
2–2
VI
0.5545 ± 0.0367
0.3910 ± 0.0373
0.4989 ± 0.0681
0.6271 ± 0.1919
636.534
2–4
0.2
21
VII
0.5746 ± 0.0513
0.3942 ± 0.0512
0.5373 ± 0.0993
0.5703 ± 0.1899
10958.5977
20
0.001
24
VIII
0.5516 ± 0.0311
0.3774 ± 0.0304
0.4948 ± 0.0715
0.6022 ± 0.1469
90.1874
24
0.1
Real estate414 × 7
I
9.2684 ± 1.9122
6.7511 ± 0.7482
0.5151 ± 0.1388
0.6653 ± 0.1543
0.003
II
8.5631 ± 1.8673
6.1966 ± 0.6894
0.4393 ± 0.1278
0.5793 ± 0.1340
0.0019
III
22.8803 ± 17.2117
13.6317 ± 9.2799
3.9393 ± 6.2263
3.9543 ± 5.7283
65.1059
25
0.001
21
IV
0.2448 ± 0.1747
0.1351 ± 0.0448
6.4868 ± 11.0001
6.7479 ± 9.9919
5.6065
24
0.001
2–1
V
8.3322 ± 2.0481
5.4289 ± 0.5484
0.3697 ± 0.1331
0.5995 ± 0.1285
41.1297
22
20
VI
8.2777 ± 2.1514
5.4656 ± 0.6343
0.3653 ± 0.1393
0.6563 ± 0.1276
629.299
2–4
0.01
22
VII
8.3537 ± 1.9192
5.4032 ± 0.5786
0.3707 ± 0.1209
0.1209 ± 0.1497
10178.2897
25
0.001
23
VIII
8.3202 ± 2.0455
5.5315 ± 0.6919
0.3685 ± 0.1295
0.6650 ± 0.1121
52.4031
22
0.001
Slump103 × 8
I
2.5807 ± 0.5759
2.0662 ± 0.4485
0.1365 ± 0.0793
0.8919 ± 0.2365
0.003
II
2.5680 ± 0.4819
2.0623 ± 0.3818
0.1324 ± 0.0650
0.8981 ± 0.1997
0.0021
III
0.0195 ± 0.0058
0.0146 ± 0.0041
0.0161 ± 0.0134
0.9754 ± 0.0510
3.7786
25
0.001
2–2
IV
0.0678 ± 0.0168
0.0533 ± 0.0113
0.1505 ± 0.0494
0.5995 ± 0.1064
0.6752
25
0.001
2–4
V
1.4282 ± 0.4010
1.0612 ± 0.3002
0.0380 ± 0.0290
0.8541 ± 0.0565
2.7774
25
20
VI
0.7027 ± 0.2838
0.5388 ± 0.1588
0.0097 ± 0.0111
0.9834 ± 0.0584
90.5473
2–4
0.01
21
VII
0.8015 ± 0.2867
0.5805 ± 0.1746
0.0123 ± 0.0112
0.9997 ± 0.0808
116.1572
25
0.1
22
VIII
0.7169 ± 0.1260
0.5188 ± 0.0829
0.0087 ± 0.0031
1.0151 ± 0.0682
8.1602
24
0.01
Ozone517 × 11
I
4.5653 ± 1.2146
3.5805 ± 0.6836
2.3044 ± 1.2403
1.5985 ± 1.2093
0.0026
II
2.6971 ± 0.5073
2.1701 ± 0.4575
0.7660 ± 0.1714
0.6772 ± 0.3418
0.0018
III
2.6364 ± 7.7191
1.7112 ± 4.9129
1.8606e + 03 ± 5.8801e + 03
1.8509e + 03 ± 5.8503e + 03
2.9356
23
0.001
2–4
IV
80.2195 ± 253.1354
56.6820 ± 178.8180
2.0582e + 06 ± 6.5086e + 06
2.0578e + 06 ± 6.5074e + 06
1.6641
25
0.001
2–4
V
2.9800 ± 0.4769
2.3844 ± 0.4648
0.8916 ± 0.2495
0.3908 ± 0.2312
5.8676
25
2–1
VI
2.6906 ± 0.3238
2.1287 ± 0.3460
0.7354 ± 0.2126
0.4839 ± 0.1758
90.8887
2–4
0.01
21
VII
3.0426 ± 0.6280
2.3599 ± 0.4222
0.9596 ± 0.4674
0.6488 ± 0.3200
118.4467
24
0.01
22
VIII
2.8883 ± 0.3950
2.3200 ± 0.3918
0.8525 ± 0.2914
0.5106 ± 0.2615
3.7969
22
0.1
Body fat252 × 15
I
0.0671 ± 0.0287
0.0446 ± 0.0063
13.3892 ± 11.8766
12.7669 ± 13.0222
0.006
II
0.0033 ± 0.0022
0.0011 ± 2.3305e − 04
0.0359 ± 0.0342
0.9539 ± 0.0750
0.0044
III
0.0050 ± 0.0016
0.0025 ± 9.1624e − 04
0.0773 ± 0.0398
0.9214 ± 0.0724
9.4223
24
0.001
2–3
IV
0.0196 ± 0.0014
0.0160 ± 0.0010
1.1480 ± 0.1444
0.1480 ± 0.1444
0.6526
2–4
0.001
2–4
V
0.0027 ± 0.0014
0.0013 ± 3.6086e − 04
0.0246 ± 0.0289
0.9323 ± 0.0683
25.3603
25
21
VI
0.0319 ± 0.0013
0.0316 ± 0.0011
2.7861 ± 0.3020
3.6795 ± 0.3559
2206530
2–4
0.001
21
VII
0.0026 ± 0.0021
9.7435e − 04 ± 2.8288e − 04
0.0273 ± 0.0388
0.9893 ± 0.0613
1041.3736
22
0.1
24
VIII
0.0031 ± 0.0030
8.3023e − 04 ± 4.7873e − 04
0.0507 ± 0.1011
1.0414 ± 0.2066
513.7258
21
0.001
Computer209 × 8
I
44.1936 ± 14.5749
22.2690 ± 5.1415
0.0788 ± 0.0440
0.9084 ± 0.3800
0.0033
II
41.8599 ± 13.6174
23.2696 ± 3.8787
0.0725 ± 0.0424
0.9786 ± 0.4010
0.0027
III
112.3888 ± 133.1836
50.4208 ± 83.6521
5.3675 ± 15.9797
5.7851 ± 15.5227
10.5498
25
0.001
2–1
IV
1.5487e + 04 ± 8.5220e + 03
5767 ± 3212
1.5393e + 04 ± 1.1253e + 04
1.5350e + 04 ± 1.1198e + 04
0.6358
23
0.001
2–4
V
51.2335 ± 60.1664
15.0893 ± 15.4565
0.1271 ± 0.1760
0.8189 ± 0.3210
18.4958
25
20
VI
28.3684 ± 46.4711
7.4601 ± 9.2279
0.0490 ± 0.1161
0.9014 ± 0.1963
251.1328
2–4
0.01
20
VII
14.2389 ± 18.4680
4.8588 ± 4.3811
0.0137 ± 0.0253
0.9908 ± 0.1313
750.8074
25
0.01
21
VIII
11.1153 ± 17.0344
4.3034 ± 4.3046
0.0097 ± 0.0213
0.9701 ± 0.1138
20.4763
22
0.001The performances of eight methods on 12 data sets are summarized in Table4. We can see that the introduction of 1-norm in LTSVR improves the efficiency in some cases, but this advantage is not valid for large-sized data sets. On the contrary, LTQSSVR greatly improves the training efficiency in most of the cases. It can be seen from Figures 3 and 4 that the processing time of LTQSSVR is much shorter than those of other methods; hence, LTQSSVR has a large advantage on the aspect of efficiency. This is because our kernel-free model avoids the time-consuming task of selecting a proper kernel and parameters. Besides, the structure of two small-scaled linear programming problems further accelerates the computational speed [29–32].Figure 3
CPU time on the general scale public benchmark data sets of LSSVR, TSVR, LTSVR, and LTQSSVR.Figure 4
CPU time on the large scale public benchmark data sets of LSSVR, TSVR, LTSVR, and LTQSSVR.From Table4, we can easily check out that our proposed method generates stable regressions on most of the data sets. It achieves the best accuracy on two data sets (Auto mpg, Computer hardware), the second best accuracy on one data sets (Hayes roth), the third best accuracy on two data sets (Body fat, Haberman), and the fourth best accuracy on five data sets (Oring, Wis, Real estate valuation, Slump, Ozone). We summarize the average rank information about these eight methods on 12 data sets in Table 5.Table 5
Average rank on RMSE of eight slgorithms.
Data set
RR
LR
nu-SVR
ϵ-SVR
LSSVR
TSVR
LTSVR
LTQSSVR
Hayes roth
6
1
7
8
3
5
4
2
Oring
1
2
7
8
3
5
6
4
Container-CC
6
3
4
2
1
7
8
5
Daily demand
7
8
1
3
6
4
2
5
Haberman
5
8
7
6
4
2
1
3
Wis
5
3
6
8
7
1
2
4
Auto mpg
6
5
8
7
3
2
4
1
Real estate valuation
8
7
1
2
5
3
6
4
Slump
8
7
1
2
6
3
5
4
Ozone
7
3
1
8
5
2
6
4
Body fat
8
4
5
6
2
7
1
3
Computer hardware
5
4
7
8
6
3
2
1
Average
6.08
4.17
5.08
5.83
3.92
3.83
3.83
3.25Then, we employ the Friedman test to check the difference of all methods under the null hypothesis that all the algorithms are no significantly different from the mean rank:(26)χF2=12Nkk+1∑jRj2−kk+124,where Rj=1/N∑irij and rij denotes the jth of k methods on the ith of N data sets. Then, the Friedmans FF is obtained by(27)FF=N−1χF2Nk−1−χF2,With eight methods and 12 data sets, FF is distributed according to the F distribution with (7, 77) degrees of freedom. According to (26), (27), and Table 5, we can obtain χF2=14.8364 and FF = 2.3593. The critical value of F (7, 77) for α = 0.05 is 2.131 and similarly is 1.796 for α = 0.1, so we reject both levels of the null hypothesis. We can identify that there is a significantly different between the eight methods. Note that LTQSSVR obtains the smallest average rank which is 3.25. It means that the accuracy of our approach is also very close to the best one.As for the processing time, Ridge Regression and Linear Regression take the shortest time among these methods, which lack robustness. In addition, LTQSSVR takes much shorter computational times than TSVR and LTSVR on all data sets. For the large data sets, TSVR and LTSVR require a lot of time for solving. This is because they have to tuning those parameters in the kernel function. In particular, we only used the Gaussian kernel function in this paper. Hence, if we add the selecting process of kernel function to those kernel-based models, their efficiencies will be further dragged down a lot. Therefore, these methods are not suitable for dealing with those huge-sized cases and their applicabilities would be strictly limited in this big data era. In contrast, due to the kernel-free and linear programming structure, our new approach has a strong ability to handling these huge-sized data sets quickly. Above all, our proposed method not only has a good generalization ability but also achieves a great efficiency.


## 5.1. Kernel Function and Parameters Selection

For the kernel-based models, since Gaussian kernel functionK (xi, xj) = exp(−∥xi − xj∥2)/σ2 is the most commonly used one, it was applied in this paper. The corresponding optimal kernel parameter σ in nu-SVR, ϵ-SVR, LSSVR, TSVR, and LTSVR was selected from the set {2i∣i = −4, −3, …, 4, 5}. It is worth pointing out that we only considered the Gaussian kernel for those kernel-based models in the experiment. If we compare other kernel functions in the searching process, the total efficiencies of those models would be tremendously reduced.Moreover, we setc1 = c2 = c and ϵ1 = ϵ2 = ϵ in our experiment. Specifically, the optimal penalty parameter c for these models was selected from the set {2i∣i = −4, −3, …, 4, 5}, and the insensitive parameter ϵ was chosen from the set {0.001, 0.01, 0.1, 0.2}. Note that all these parameters were determined by the fivefold cross validation. That is, the data set was randomly split into five subsets. In each time, one of those subsets was reserved as a testing set while other four subsets were used together as a training set. For every data set, this process was repeated ten times. And all the results are the average values of these ten tests.In each cross validation, we use the following four indicators in [10, 28] as the criteria for selecting the optimal parameters and verify the performance of eight models:(1)
RMSE: Root Mean Squared Error, defined asRMSE=1/N∑i=1Ny^i−yi2. It is used to measure the deviation between the observed value and the true value. Because RMSE uses the average error, it is sensitive to abnormal points. If the regression value of a certain point is not reasonable, its error is relatively large, which has a great impact on the value of RMSE. In general, the smaller the RMSE is, the more accurate the prediction results will be.(2)
MAE: Mean Absolute Error, defined asMAE=1/Nyi−y^i. It is the mean of absolute error which can reflect the actual situation of predicted error.(3)
SSE/SST: SSE (Sum of Squares due to Error) measures the difference between the actual value and the predict value, which represents the part interpreted by the regression equation. SST (Sum of squares for total) measures the deviation ofy, which represents the total change of data. In most cases, a smaller SSE/SST means a better consistency between estimates and actual values.(4)
SSR/SST: SSR (Sum of Squares due to Regression) measures the difference between estimate values and the mean value ofy, which represents the part not explained by the regression equation. The larger the SSR, the more statistic information it captures from the test sample. In order to obtain a small SSE/SST, it is usually accompanied by an increase in SSR/SST.


## 5.2. Artificial Data Sets

First, we followed the traditional way in [10, 28] to generate a classical type of artificial data sets in the experiment. This is a kind of 2-D artificial data sets which is obtained by the equation y=sincx=sinπx/πx,x∈−5,5. In reality, data sets always contain noise points. In order to check the performance of our model in this situation, two kinds of noises including the Gaussian noise and the uniformly distributed noise were added. Specially, four types of training points were generated as follows:(25)yi=sinπxiπxi+ξi,ξi∼U0,0.15,yi=sinπxiπxi+ξi,ξi∼U0,0.3,yi=sinπxiπxi+ξi,ξi∼N0,0.152,yi=sinπxiπxi+ξi,ξi∼N0,0.32,where U[a, b] represents the uniform distribution in the range of [a, b] and N[0, d2] represents the normal distribution with a mean of 0 and a variance of d2. To avoid bias comparison, we randomly generated 10 groups of independent noise samples for each type of noise. Following the traditional way in [10, 28], each group contains 600 training samples and 400 test samples. And the test points were generated by function sin c (x) without noise. The corresponding results of eight algorithms Ridge Regression, Linear Regression, nu-SVR, ϵ-SVR, LSSVR, TSVR, LTSVR, and LTQSSVR are shown in Table 1.Table 1
Results of Ridge Regression (RR), Linear Regression (LR), nu-SVR,ϵ-SVR, LSSVR, TSVR, LTSVR, and LTQSSVR on benchmark data sets with RBF kernel.
Noise
Alg
RMSE
MAE
SSE/SST
SSR/SST
Time (s)
c
ϵ
σ
Type 1
RR
0.3102 ± 0.0115
0.1804 ± 0.0066
1.1385 ± 0.0163
0.1389 ± 0.0162
0.0434
LR
0.3106 ± 0.0115
0.1805 ± 0.0066
1.1410 ± 0.0170
0.1410 ± 0.0170
0.0004
nu-SVR
635.0782 ± 2.0081e + 03
257.3113 ± 813.5255
4.7260e + 07 ± 1.4945e + 08
4.7260e + 07 ± 1.4945e + 08
47.0139
22
0.001
2–1
ϵ-SVR
158.5021 ± 350.9712
91.1181 ± 197.1021
1.6476e + 06 ± 3.9456e + 06
1.6476e + 06 ± 3.9456e + 06
28.7549
22
0.001
2–1
LSSVR
0.6531 ± 0.0155
0.5141 ± 0.0144
4.5383 ± 0.4865
4.3325 ± 0.5131
830.3177
25
2–1
TSVR
0.6544 ± 0.0270
0.5314 ± 0.0176
4.3852 ± 0.4677
4.2048 ± 0.4878
2024.7431
2–4
0.001
2–1
LTSVR
0.5870 ± 0.0372
0.4839 ± 0.0247
3.7994 ± 0.2871
3.8503 ± 0.1351
26531.2
22
0.01
22
LTQSSVR
0.3961 ± 0.0187
0.2851 ± 0.0239
1.7631 ± 0.1997
0.3917 ± 0.1451
96.3907
2–1
0.01
Type 2
RR
0.3432 ± 0.0102
0.2034 ± 0.0065
1.1693 ± 0.0168
0.1695 ± 0.0172
0.0027
LR
0.3434 ± 0.0110
0.2034 ± 0.0075
1.1705 ± 0.0222
0.1705 ± 0.0222
0.0024
nu-SVR
588.4 ± 186.1
140.7 ± 444.7
3.4195e + 07 ± 1.0814e + 08
3.4196e + 07 ± 1.0814e + 08
53.9877
2–2
0.001
20
ϵ-SVR
107.6 ± 238.9
87.9 ± 193.4
6.4387e + 05 ± 1.6125e + 06
6.4345e + 05 ± 1.6114e + 06
43.9494
2–2
0.001
20
LSSVR
0.6406 ± 0.0257
0.5193 ± 0.0233
4.3594 ± 0.4286
4.0586 ± 0.3830
959.2185
20
2–2
TSVR
0.6338 ± 0.0397
0.5123 ± 0.0321
4.1191 ± 0.6138
3.7309 ± 0.5828
5657.1416
2–4
0.2
20
LTSVR
0.6988 ± 0.1573
0.5541 ± 0.1162
4.8817 ± 1.5382
5.0229 ± 1.9708
4583.9875
2–2
0.001
24
LTQSSVR
0.3979 ± 0.0321
0.2817 ± 0.0225
1.7730 ± 0.2054
0.3920 ± 0.1574
93.6575
2–1
0.01
Type 3
RR
0.3431 ± 0.0102
0.2033 ± 0.0068
1.1688 ± 0.0171
0.1690 ± 0.0175
0.0038
LR
0.3431 ± 0.0107
0.2033 ± 0.0072
1.1688 ± 0.0190
0.1688 ± 0.0190
0.0003
nu-SVR
165.9 ± 524.4
93.1 ± 293.7
2.8467e + 06 ± 9.0020e + 06
2.8464e + 06 ± 9.0012e + 06
26531.2
23
0.001
2–1
ϵ-SVR
0.1545 ± 0.0015
0.1536 ± 0.0017
0.2373 ± 0.0130
1.2355 ± 0.0176
30.5638
20
0.001
20
LSSVR
0.6531 ± 0.0155
0.5141 ± 0.0144
4.5383 ± 0.4865
4.3325 ± 0.5131
1200.9825
25
2–1
TSVR
0.6273 ± 0.0062
0.4944 ± 0.0094
4.0209 ± 0.0178
3.6071 ± 0.0231
2887.8195
2–4
0.1
20
LTSVR
0.6852 ± 0.0327
0.5311 ± 0.0301
5.4208 ± 1.1646
5.4496 ± 1.3123
4735.0025
22
0.001
22
LTQSSVR
0.3935 ± 0.0160
0.3310 ± 0.0148
1.7354 ± 0.1279
0.4470 ± 0.0608
100.1174
2–2
0.01
Type 4
RR
0.3431 ± 0.0103
0.2033 ± 0.0071
1.1687 ± 0.0174
0.1687 ± 0.0175
0.0037
LR
0.3431 ± 0.0104
0.2033 ± 0.0070
1.1687 ± 0.0183
0.1687 ± 0.0183
0.0003
nu-SVR
542.5 ± 1429.7
235.1 ± 571.1
2.1116e + 07 ± 6.3937e + 07
2.1115e + 07 ± 6.3936e + 07
26531.2
23
0.001
2–1
ϵ-SVR
0.0811 ± 0.0070
0.0769 ± 0.0070
0.0658 ± 0.0119
0.9687 ± 0.0149
9.4324
23
0.001
2–1
LSSVR
0.6531 ± 0.0155
0.5141 ± 0.0144
4.5383 ± 0.4865
4.3325 ± 0.5131
1200.9825
25
2–1
LSSVR
0.6442 ± 0.0173
0.5121 ± 0.0142
4.4135 ± 0.4501
4.2044 ± 0.4833
12.1205
23
2–2
TSVR
0.6064 ± 0.0099
0.4784 ± 0.0060
3.7604 ± 0.2137
3.2961 ± 0.2944
737.7178
2–4
0.2
20
LTSVR
0.6426 ± 0.0481
0.5 ± 0.0140
4.1282 ± 0.3409
4.0077 ± 0.3712
9533.4572
22
0.01
2–2
LTQSSVR
0.3781 ± 0.0257
0.2980 ± 0.0298
1.6028 ± 0.1709
0.3151 ± 0.1171
113.2787
2–1
0.01From Table1, we can see that Ridge Regression achieves the best results by obtaining the minimum RMSE and MAE values for noises of Type 1 and Type 2, and ϵ-SVR obtains the best for noises of Type 3 and Type 4. Besides, LTQSSVR take a stable performance for most data sets which has a better robustness for noises. And it can also be seen from Table 1 that Linear Regression is the fastest one among all algorithms. However, Linear Regression is only suitable for the cases with linear separable data.In order to verify the universal ability of eight models, we used another type of artificial data sets in [20] to test the performances of eight models. Now, we show how to obtain this type of artificial data sets. First, we used different matrices U and vectors c to get various quadratic surfaces 1/2XTUX+cTX=0, where each element of U and c was randomly selected from the domain [−10, 10]. Then, we randomly generated points on both sides of the quadratic surface. Similarly, we generated 10 independent groups of samples. Each group includes 60% training samples with noise and 40% test points without noise. The corresponding results are shown in Table 2.Table 2
Results of Ridge Regression (RR), Linear Regression (LR), nu-SVR,ϵ-SVR, LSSVR, TSVR, LTSVR, and LTQSSVR on quadratic artificial data sets.
Data sets num × fea
Algorithm
RMSE
MAE
SSE/SST
SSR/SST
Time (s)
c
ϵ
σ
AR130 × 10
RR
556.9348 ± 64.6680
472.5440 ± 68.1756
0.5760 ± 0.3548
0.7606 ± 0.3899
0.0037
LR
399.4814 ± 100.0622
328.3254 ± 88.2998
0.3022 ± 0.1929
1.1167 ± 0.4799
2.0278
nu-SVR
864.5692 ± 197.0818
673.1263 ± 149.9371
1.1580 ± 0.2129
0.1580 ± 0.2129
0.1191
25
0.001
20
ϵ-SVR
865.8422 ± 198.0437
675.8055 ± 151.8056
1.1606 ± 0.2109
0.1606 ± 0.2109
0.1090
25
0.001
2–4
LSSVR
634.8957 ± 256.3804
520.7602 ± 208.0142
0.5531 ± 0.4678
0.3691 ± 0.1874
1.511
22
23
TSVR
509.4136 ± 229.8878
416.2661 ± 194.7960
0.7154 ± 0.3129
0.3126 ± 0.4475
14.91
2–4
0.1
23
LTSVR
875.5967 ± 178.9845
724.9856 ± 141.0931
0.9595 ± 0.3107
0.2126 ± 0.1810
4.2708
2–3
0.2
23
LTQSSVR
233.7883 ± 74.5269
184.9046 ± 58.4456
0.0698 ± 0.0375
0.9108 ± 0.1189
3.397
2–1
0.01
AR250 × 20
RR
772.9401 ± 94.6686
644.1068 ± 80.0471
0.7422 ± 0.2818
1.2858 ± 0.3805
0.0002
LR
804.7534 ± 205.3123
660.0124 ± 152.2902
0.7986 ± 0.3627
1.4378 ± 0.5091
22.6734
nu-SVR
1019.9 ± 164.4628
828.6489 ± 138.9097
1.2486 ± 0.4260
0.2486 ± 0.4260
0.1761
2–4
0.001
2–2
ϵ-SVR
1013.4 ± 147.9270
832.0854 ± 131.6954
1.2332 ± 0.4159
0.2332 ± 0.4159
0.1979
25
0.001
2–4
LSSVR
876.2712 ± 35.8862
656.5610 ± 33.6003
0.3968 ± 0.0538
0.5012 ± 0.0699
0.7977
25
25
TSVR
825.4982 ± 192.4043
662.9940 ± 167.3309
0.2658 ± 0.1259
0.7697 ± 0.2515
18.8629
2–1
0.01
25
LTSVR
2370 ± 767.3638
2016.2 ± 758.3574
2.1851 ± 1.4099
1.3458 ± 1.2179
9.1719
2–3
0.2
24
LTQSSVR
503.2282 ± 111.1342
391.0349 ± 93.5383
0.1362 ± 0.0736
0.9079 ± 0.0424
20.7707
20
0.2
AR3100 × 30
RR
951.3805 ± 112.4494
750.8384 ± 88.4491
0.1356 ± 0.0377
0.8898 ± 0.0631
0.0072
LR
922.5558 ± 132.01183
741.5420 ± 92.6491
0.1268 ± 0.0358
0.9371 ± 0.0548
0.5144
nu-SVR
2685.5 ± 229.4038
2179.9 ± 162.6985
1.0437 ± 0.0377
0.0437 ± 0.0377
3.7895
25
0.001
2–4
ϵ-SVR
2701.6 ± 228.9889
2197.2 ± 166.9867
1.0571 ± 0.0578
0.0571 ± 0.0578
0.5144
25
0.001
2–4
LSSVR
1098 ± 121.5927
889.7450 ± 74.5102
0.0923 ± 0.0197
0.7229 ± 0.0703
2.2872
25
25
TSVR
758.5976 ± 149.9849
601.7016 ± 128.2513
0.1026 ± 0.0611
0.8481 ± 0.0975
48.9992
25
0.2
25
LTSVR
3841.9 ± 2813.9
3325.2 ± 2771.0
1.7216 ± 2.7258
1.4169 ± 2.4844
64.7086
2–4
0.2
25
LTQSSVR
863.4386 ± 114.7082
664.7501 ± 96.2545
0.1590 ± 0.0540
0.8574 ± 0.1567
144.6868
24
0.001
AR4500 × 30
RR
804.0858 ± 29.61234
620.5212 ± 29.0568
0.1167 ± 0.0122
0.8591 ± 0.0527
3.5505
LR
741.3826 ± 28.4248
589.9683 ± 27.1126
0.0992 ± 0.0108
0.9021 ± 0.0496
6.5738
nu-SVR
2367.5 ± 67.0977
1864.1 ± 58.4901
1.0067 ± 0.0074
0.0067 ± 0.0074
6.3427
25
0.001
2–4
ϵ-SVR
2370.9 ± 68.4916
1867.5 ± 60.7537
1.0096 ± 0.0128
0.0096 ± 0.0128
9.1899
22
0.001
2–4
LSSVR
579.1923 ± 69.7749
460.4036 ± 43.4087
0.0656 ± 0.0110
0.8806 ± 0.0369
46.9875
25
25
TSVR
489.2266 ± 23.5683
391.9541 ± 24.6263
0.0612 ± 0.0012
0.9893 ± 0.0012
519.6265
25
0.01
25
LTSVR
7321 ± 1491
6689.5 ± 859.5
−
−
2714.643
2–4
0.2
23
LTQSSVR
529.4326 ± 59.2533
473.1416 ± 2.9623
0.0607 ± 0.0122
1.001 ± 0.0011
442.1526
25
0.1
AR51000 × 10
RR
355.4752 ± 13.4051
283.0450 ± 10.6422
0.1897 ± 0.0117
0.6229 ± 0.0495
0.0018
LR
231.9386 ± 8.2504
181.3980 ± 6.9325
0.0809 ± 0.0064
0.9459 ± 0.0603
1.8952
nu-SVR
817.6871 ± 22.6287
651.5018 ± 13.5489
1.0028 ± 0.0031
0.0028 ± 0.0031
22.8603
25
0.001
2–4
ϵ-SVR
817.6872 ± 22.6285
651.5021 ± 13.5487
1.0028 ± 0.0031
0.0028 ± 0.0031
21.8329
25
0.001
2–4
LSSVR
43.43 ± 1.26
31.09 ± 0.79
0.002 ± 0.0001
0.9815 ± 0.0039
115.985
25
24
TSVR
8.25 ± 1.17
5.65 ± 0.83
0.0001 ± 2.33E − 05
0.9967 ± 0.0033
32081.806
2–4
0.01
25
LTSVR
435.58 ± 431.67
347.57 ± 344.48
0.5135 ± 0.5134
0.5134 ± 0.4865
5105.084
2–3
0.01
25
LTQSSVR
0.0758 ± 0.0191
0.0731 ± 0.0212
1.34E − 08 ± 6.24E − 09
1.000 ± 1.98E − 05
1546.984
22
0.2
AR62000 × 10
RR
348.4587 ± 5.2498
277.5702 ± 4.6563
0.1884 ± 0.0089
0.6374 ± 0.0228
0.0029
LR
232.5254 ± 5.7657
180.2640 ± 4.9633
0.0840 ± 0.0066
0.9209 ± 0.0215
8.6790
nu-SVR
804.4539 ± 15.5199
644.4631 ± 13.4341
1.0033 ± 0.0058
0.0033 ± 0.0058
68.0419
25
0.001
2–4
ϵ-SVR
804.4538 ± 15.5199
644.4633 ± 13.4341
1.0033 ± 0.0058
0.0033 ± 0.0058
95.3783
25
0.001
2–4
LSSVR
29.2831 ± 0.1239
19.5835 ± 0.5175
0.0016 ± 3.1304e − 05
0.9765 ± 6.0976e − 04
920.0551
25
24
TSVR
10.3615 ± 1.5314
7.2374 ± 1.4115
0.0001 ± 1.96E − 05
0.9963 ± 0.0017
10673.192
2–2
0.01
25
LTSVR
775.3161 ± 133.3652
625.1344 ± 119.2564
1.2355 ± 0.2456
0.3495 ± 0.1316
16739.2
2–4
0.01
2–4
LTQSSVR
0.1074 ± 0.0410
0.1032 ± 0.0416
3.06e − 08 ± 2.23e − 08
1.000 ± 1.3591e − 05
7536.641
25
0.01
AR73000 × 5
RR
294.1609 ± 3.7125
237.5383 ± 2.6317
0.1974 ± 0.0039
0.4435 ± 0.0082
0.1503
LR
143.8636 ± 2.7915
110.1801 ± 1.7634
0.0472 ± 0.0012
0.9656 ± 0.0308
6.5705
nu-SVR
560.7090 ± 10.0120
442.1160 ± 6.1240
0.7170 ± 0.0127
0.0643 ± 0.0074
1134.1517
25
0.001
2–4
ϵ-SVR
560.7196 ± 10.0115
442.1315 ± 6.1242
0.7170 ± 0.0127
0.0643 ± 0.0074
170.9971
25
0.001
2–4
LSSVR
10.4242 ± 0.5163
6.6254 ± 0.2775
0.0002 ± 2.23083e − 05
0.9964 ± 0.0003
2959.8848
25
24
TSVR
−
−
−
−
−
2–4
0.01
25
LTSVR
−
−
−
−
−
2–3
0.01
25
LTQSSVR
0.0667 ± 0.0049
0.0664 ± 0.0051
1E − 08 ± 1.39E − 09
1.000 ± 1.98E − 05
1546.984
22
0.2
AR85000 × 3
RR
121.4335 ± 1.9984
98.4936 ± 1.4719
0.2308 ± 0.0034
0.5152 ± 0.0083
0.0044
LR
104.7501 ± 2.3667
78.1530 ± 1.0814
0.1717 ± 0.0049
0.8325 ± 0.0107
8.9268
nu-SVR
7.7250 ± 0.8782
1.3397 ± 0.1542
0.0009 ± 2.0694e − 04
0.9892 ± 0.0018
1219.7795
25
0.001
2–4
ϵ-SVR
7.7357 ± 0.8790
1.3571 ± 0.1540
0.0009 ± 2.0743e − 04
0.9888 ± 0.0018
438.3024
25
0.001
2–4
LSSVR
2.0616 ± 0.3076
0.8998 ± 0.3241
6.7308e − 05 ± 1.9711e − 05
0.9967 ± 0.0016
150667.6
24
24
TSVR
−
−
−
−
−
2–4
0.01
25
LTSVR
−
−
−
−
−
2–3
0.01
25
LTQSSVR
0.0716 ± 0.0240
0.0710 ± 0.0245
8.4658E − 08 ± 6.31E − 09
1.000 ± 1.86311E − 06
22332.86
25
0.2From Table2, we can see that LTQSSVR still achieves the best performance in seven cases among total eight cases by obtaining the smallest RMSE, MAE, and SSE/SST values. Compared with TSVR, LTSVR improves the training results by introducing 1-norm instead of the original 2-norm. But for large-scaled data sets, its advantages are not significant. Among these methods, Ridge Regression has the shortest processing time, followed by Linear Regression, nu-SVR, and ϵ-SVR, which lack robustness to noises. Moreover, the experimental results with different scaled data sets show that LTQSSVR has better performance with large-scaled but small-dimension data sets. Besides, we also show the box plot of RMSE values in Figure 2. It is easy to check that LTQSSVR are concentrated and obtain a smallest average, Hence LTQSSVR is robust in most cases.Figure 2
Boxplots of RMSE for artificial data sets: (a) artificial data set 1; (b) artificial data set 3; (c) artificial data set 5.
(a)
(b)
(c)


## 5.3. Benchmark Data Sets

In this part, we use 12 benchmark data sets from the UCI Repository to test these four models (available fromhttps://archive.ics.uci.edu/ml/datasets.php). This list includes Auto MPG, Oring, Wis, Slump, Haberman, and so on. The detailed information of these data sets is summarized in Table 3.Table 3
Detailed information about UCI data sets.
Data set
Number of points
Number of features
Auto mpg
392
8
Oring
23
5
Container-CC
15
3
Daily demand
60
13
Haber man
306
4
Wis
194
30
Hayes roth
132
6
Real estate valuation
414
7
Slump
103
8
Ozone
517
11
Body fat
252
15
Computer hardware
209
8To eliminate the overwhelming dominance between features, we normalized the features of all data to [0,1] before the training. The experimental results of eight models are summarized in Table4. For each result, the first term denotes the average value of ten times while the second term denotes the standard deviation. From Table 4, we can see that the LTQSSVR obtains good results in most of the cases. Compared with LSSVR and TSVR, the new model greatly improves the computational efficiency on the basis of ensuring accuracy.Table 4
Results of Ridge Regression (I), Linear Regression (II), nu-SVR (III),ϵ-SVR (IV), LSSVR (V), TSVR (VI), LTSVR (VII), and LTQSSVR (VIII) on benchmark data sets with RBF kernel.
Data sets
Alg
RMSE
MAE
SSE/SST
SSR/SST
Time(s)
c
ϵ
σ
Hayes roth132 × 6
I
0.6253 ± 0.0601
0.5191 ± 0.0515
0.6914 ± 0.1324
1.0377 ± 0.1817
0.005
II
0.5495 ± 0.0570
0.4486 ± 0.0487
0.5335 ± 0.1090
0.5345 ± 0.1049
0.0042
III
1.2221 ± 1.1811
0.7785 ± 0.6059
4.2785 ± 7.4431
4.4979 ± 7.2225
5.8376
23
0.001
2–4
IV
2.5118 ± 3.8211
1.2269 ± 1.5683
33.4225 ± 69.4087
33.8838 ± 70.3782
0.8978
22
0.001
2–4
V
0.5787 ± 0.0870
0.4394 ± 0.0832
0.4919 ± 0.1487
0.6280 ± 0.1470
4.0312
25
22
VI
0.6157 ± 0.1650
0.4700 ± 0.0966
0.6656 ± 0.4210
1.0015 ± 0.7287
74.9323
2–4
0.2
25
VII
0.5999 ± 0.1008
0.4450 ± 0.0773
0.5866 ± 0.1730
0.9682 ± 0.3809
208.5302
2–3
0.01
25
VIII
0.5665 ± 0.0715
0.4394 ± 0.0696
0.5219 ± 0.1174
0.7902 ± 0.2385
6.7313
22
0.2
Oring23 × 5
I
3.7095 ± 1.5179
2.9936 ± 1.2695
0.5474 ± 0.7138
0.9454 ± 0.5311
0.0026
II
3.7096 ± 1.5179
2.9936 ± 1.2695
0.5474 ± 0.7138
0.9455 ± 0.5311
0.0086
III
40.3245 ± 65.7156
26.3581 ± 43.9478
166.4272 ± 367.8244
166.9181 ± 374.4771
0.7148
24
0.001
20
IV
511.2011 ± 1.4669e + 03
245.0591 ± 680.3442
7.0421e + 04 ± 2.2234e + 05
7.0542e + 04 ± 2.2272e + 05
0.3195
25
0.001
2–3
V
4.7860 ± 1.8800
3.8188 ± 1.3995
0.8257 ± 1.0693
0.8409 ± 0.7116
1.6519
21
2–2
VI
22.4245 ± 44.2676
13.8426 ± 25.2836
46.6614 ± 135.2396
49.9042 ± 143.4053
15.3404
23
0.2
2–2
VII
28.9964 ± 74.4129
20.1901 ± 50.8615
92.7013 ± 289.5774
90.7100 ± 282.2787
28.7152
2–3
0.01
23
VIII
5.3305 ± 1.9512
4.1148 ± 1.6144
1.0038 ± 1.3476
1.4496 ± 0.7396
1.5079
21
0.2
Container15 × 3
I
0.3012 ± 0.0677
0.2792 ± 0.0522
9.2105 ± 5.1356
6.3104 ± 5.0631
0.0027
II
0.1787 ± 0.0254
0.1574 ± 0.0279
3.1271 ± 1.3823
1.8975 ± 1.6807
0.0068
III
0.2468 ± 0.3123
0.2143 ± 0.2597
1.8268 ± 0.8391
0.8461 ± 0.8428
0.1491
24
0.001
2–4
IV
0.1541 ± 0.0395
0.1359 ± 0.0370
2.5484 ± 1.2829
1.2351 ± 1.0699
0.2179
2–4
0.001
25
V
0.1458 ± 0.0542
0.1343 ± 0.0514
2.1108 ± 1.1380
1.6074 ± 0.9613
1.4901
2–4
25
VI
0.3147 ± 0.2579
0.2645 ± 0.1928
8.2127 ± 12.0510
6.3811 ± 11.9280
14.4593
2–3
0.2
21
VII
0.5611 ± 1.0092
0.4895 ± 0.8664
44.1796 ± 114.0210
44.5070 ± 119.0015
16.4264
2–4
0.001
22
VIII
0.2567 ± 0.1222
0.2277 ± 0.1040
6.2074 ± 4.6990
3.5947 ± 3.4960
0.2904
2–2
0.1
Daily de60 × 13
I
28.1321 ± 4.1695
22.2524 ± 3.1676
0.2260 ± 0.1701
1.2616 ± 0.3249
0.0033
II
9.7112e − 14 ± 2.0289e − 14
8.1476e − 14 ± 1.9473e − 14
2.8124e − 30 ± 2.2297e − 30
1 ± 8.4956e − 16
0.0022
III
46.5750 ± 18.2514
26.3692 ± 9.0850
0.2679 ± 0.1062
0.5115 ± 0.2421
0.2335
25
0.001
2–1
IV
42.4564 ± 13.7516
23.9045 ± 7.3832
0.2352 ± 0.0667
0.4670 ± 0.1249
0.2396
25
0.001
2–1
V
14.4057 ± 7.2369
9.2370 ± 4.5731
0.0292 ± 0.0176
0.8718 ± 0.1066
2.3495
25
21
VI
2.4664 ± 2.1341
1.5724 ± 0.9919
9.5111e − 04 ± 0.0013
0.9875 ± 0.0239
40.7319
2–4
0.1
22
VII
0.0782 ± 0.0778
0.0486 ± 0.0355
9.4084e − 07 ± 1.5290e − 06
0.9999 ± 5.0720e − 04
32.5313
2–4
0.2
25
VIII
10.8529 ± 7.5422
6.9757 ± 3.9298
0.0232 ± 0.0274
1.0648 ± 0.1351
9.7285
23
0.001
Haber man306 × 4
I
0.5854 ± 0.0482
0.4515 ± 0.0431
1.8093 ± 0.3646
1.1981 ± 0.2128
0.0249
II
0.4259 ± 0.0210
0.3564 ± 0.0123
0.9495 ± 0.1062
0.0889 ± 0.0590
0.0169
III
0.4877 ± 0.0236
0.4818 ± 0.0195
1.1689 ± 0.1149
0.1572 ± 0.0958
13.646
2–1
0.001
25
IV
0.4592 ± 0.0262
0.3539 ± 0.0491
1.1258 ± 0.1258
0.2377 ± 0.1436
7.5302
2–1
0.001
23
V
0.4176 ± 0.0236
0.3422 ± 0.0181
0.9156 ± 0.0695
0.1285 ± 0.0547
33.9049
2–1
2–4
VI
0.4655 ± 0.1198
0.3290 ± 0.0410
1.1776 ± 0.6418
0.4589 ± 0.8390
301.7816
2–2
0.1
22
VII
0.3273 ± 0.2326
0.3129 ± 0.0410
1.2283 ± 0.2454
0.7573 ± 0.3473
6716.377
2–3
0.01
24
VIII
0.4446 ± 0.0454
0.3077 ± 0.0310
1.0407 ± 0.1664
0.3038 ± 0.1428
112.8566
22
0.1
Wis194 × 30
I
0.0088 ± 0.0015
0.0067 ± 7.9729e − 04
0.2061 ± 0.0867
1.2493 ± 0.1829
0.0098
II
0.0062 ± 0.0017
0.0043 ± 8.0280e − 04
0.1050 ± 0.0512
1.0385 ± 0.1263
0.0029
III
0.0091 ± 0.0029
0.0057 ± 0.0012
0.1792 ± 0.0688
0.6678 ± 0.1592
8.7957
22
0.001
2–4
IV
0.0484 ± 0.0048
0.0453 ± 0.0052
6.8505 ± 3.3307
5.8505 ± 3.3307
0.1361
2–4
0.001
2–4
V
0.0415 ± 0.0123
0.0277 ± 0.0061
0.1043 ± 0.0622
0.8787 ± 0.2353
15.4689
25
22
VI
0.0057 ± 0.0013
0.0041 ± 6.9250e − 04
0.0768 ± 0.0250
0.9603 ± 0.1223
151.3218
2–4
0.2
23
VII
0.0060 ± 0.0013
0.0044 ± 6.0337e − 04
0.0858 ± 0.0274
1.0015 ± 0.1607
877.5655
2–4
0.1
24
VIII
0.0087 ± 0.0016
0.0057 ± 9.3220e − 04
0.1977 ± 0.1165
1.1763 ± 0.2253
83.3716
21
0.001
Auto mpg392 × 8
I
0.6124 ± 0.0654
0.4813 ± 0.0568
0.5958 ± 0.1036
0.5781 ± 0.1547
0.0067
II
0.5898 ± 0.0655
0.4632 ± 0.0588
0.5520 ± 0.0947
0.5221 ± 0.1290
0.0018
III
2.1847 ± 2.5549
1.1657 ± 1.2357
65.4009 ± 126.7810
65.9644 ± 128.8542
36.0491
20
0.001
25
IV
1.0737 ± 2.2710
0.6030 ± 1.1399
35.8631 ± 110.4902
36.6985 ± 112.1304
7.6174
23
0.001
20
V
0.5598 ± 0.0647
0.3760 ± 0.0577
0.5070 ± 0.0846
0.5995 ± 0.1304
67.7641
24
2–2
VI
0.5545 ± 0.0367
0.3910 ± 0.0373
0.4989 ± 0.0681
0.6271 ± 0.1919
636.534
2–4
0.2
21
VII
0.5746 ± 0.0513
0.3942 ± 0.0512
0.5373 ± 0.0993
0.5703 ± 0.1899
10958.5977
20
0.001
24
VIII
0.5516 ± 0.0311
0.3774 ± 0.0304
0.4948 ± 0.0715
0.6022 ± 0.1469
90.1874
24
0.1
Real estate414 × 7
I
9.2684 ± 1.9122
6.7511 ± 0.7482
0.5151 ± 0.1388
0.6653 ± 0.1543
0.003
II
8.5631 ± 1.8673
6.1966 ± 0.6894
0.4393 ± 0.1278
0.5793 ± 0.1340
0.0019
III
22.8803 ± 17.2117
13.6317 ± 9.2799
3.9393 ± 6.2263
3.9543 ± 5.7283
65.1059
25
0.001
21
IV
0.2448 ± 0.1747
0.1351 ± 0.0448
6.4868 ± 11.0001
6.7479 ± 9.9919
5.6065
24
0.001
2–1
V
8.3322 ± 2.0481
5.4289 ± 0.5484
0.3697 ± 0.1331
0.5995 ± 0.1285
41.1297
22
20
VI
8.2777 ± 2.1514
5.4656 ± 0.6343
0.3653 ± 0.1393
0.6563 ± 0.1276
629.299
2–4
0.01
22
VII
8.3537 ± 1.9192
5.4032 ± 0.5786
0.3707 ± 0.1209
0.1209 ± 0.1497
10178.2897
25
0.001
23
VIII
8.3202 ± 2.0455
5.5315 ± 0.6919
0.3685 ± 0.1295
0.6650 ± 0.1121
52.4031
22
0.001
Slump103 × 8
I
2.5807 ± 0.5759
2.0662 ± 0.4485
0.1365 ± 0.0793
0.8919 ± 0.2365
0.003
II
2.5680 ± 0.4819
2.0623 ± 0.3818
0.1324 ± 0.0650
0.8981 ± 0.1997
0.0021
III
0.0195 ± 0.0058
0.0146 ± 0.0041
0.0161 ± 0.0134
0.9754 ± 0.0510
3.7786
25
0.001
2–2
IV
0.0678 ± 0.0168
0.0533 ± 0.0113
0.1505 ± 0.0494
0.5995 ± 0.1064
0.6752
25
0.001
2–4
V
1.4282 ± 0.4010
1.0612 ± 0.3002
0.0380 ± 0.0290
0.8541 ± 0.0565
2.7774
25
20
VI
0.7027 ± 0.2838
0.5388 ± 0.1588
0.0097 ± 0.0111
0.9834 ± 0.0584
90.5473
2–4
0.01
21
VII
0.8015 ± 0.2867
0.5805 ± 0.1746
0.0123 ± 0.0112
0.9997 ± 0.0808
116.1572
25
0.1
22
VIII
0.7169 ± 0.1260
0.5188 ± 0.0829
0.0087 ± 0.0031
1.0151 ± 0.0682
8.1602
24
0.01
Ozone517 × 11
I
4.5653 ± 1.2146
3.5805 ± 0.6836
2.3044 ± 1.2403
1.5985 ± 1.2093
0.0026
II
2.6971 ± 0.5073
2.1701 ± 0.4575
0.7660 ± 0.1714
0.6772 ± 0.3418
0.0018
III
2.6364 ± 7.7191
1.7112 ± 4.9129
1.8606e + 03 ± 5.8801e + 03
1.8509e + 03 ± 5.8503e + 03
2.9356
23
0.001
2–4
IV
80.2195 ± 253.1354
56.6820 ± 178.8180
2.0582e + 06 ± 6.5086e + 06
2.0578e + 06 ± 6.5074e + 06
1.6641
25
0.001
2–4
V
2.9800 ± 0.4769
2.3844 ± 0.4648
0.8916 ± 0.2495
0.3908 ± 0.2312
5.8676
25
2–1
VI
2.6906 ± 0.3238
2.1287 ± 0.3460
0.7354 ± 0.2126
0.4839 ± 0.1758
90.8887
2–4
0.01
21
VII
3.0426 ± 0.6280
2.3599 ± 0.4222
0.9596 ± 0.4674
0.6488 ± 0.3200
118.4467
24
0.01
22
VIII
2.8883 ± 0.3950
2.3200 ± 0.3918
0.8525 ± 0.2914
0.5106 ± 0.2615
3.7969
22
0.1
Body fat252 × 15
I
0.0671 ± 0.0287
0.0446 ± 0.0063
13.3892 ± 11.8766
12.7669 ± 13.0222
0.006
II
0.0033 ± 0.0022
0.0011 ± 2.3305e − 04
0.0359 ± 0.0342
0.9539 ± 0.0750
0.0044
III
0.0050 ± 0.0016
0.0025 ± 9.1624e − 04
0.0773 ± 0.0398
0.9214 ± 0.0724
9.4223
24
0.001
2–3
IV
0.0196 ± 0.0014
0.0160 ± 0.0010
1.1480 ± 0.1444
0.1480 ± 0.1444
0.6526
2–4
0.001
2–4
V
0.0027 ± 0.0014
0.0013 ± 3.6086e − 04
0.0246 ± 0.0289
0.9323 ± 0.0683
25.3603
25
21
VI
0.0319 ± 0.0013
0.0316 ± 0.0011
2.7861 ± 0.3020
3.6795 ± 0.3559
2206530
2–4
0.001
21
VII
0.0026 ± 0.0021
9.7435e − 04 ± 2.8288e − 04
0.0273 ± 0.0388
0.9893 ± 0.0613
1041.3736
22
0.1
24
VIII
0.0031 ± 0.0030
8.3023e − 04 ± 4.7873e − 04
0.0507 ± 0.1011
1.0414 ± 0.2066
513.7258
21
0.001
Computer209 × 8
I
44.1936 ± 14.5749
22.2690 ± 5.1415
0.0788 ± 0.0440
0.9084 ± 0.3800
0.0033
II
41.8599 ± 13.6174
23.2696 ± 3.8787
0.0725 ± 0.0424
0.9786 ± 0.4010
0.0027
III
112.3888 ± 133.1836
50.4208 ± 83.6521
5.3675 ± 15.9797
5.7851 ± 15.5227
10.5498
25
0.001
2–1
IV
1.5487e + 04 ± 8.5220e + 03
5767 ± 3212
1.5393e + 04 ± 1.1253e + 04
1.5350e + 04 ± 1.1198e + 04
0.6358
23
0.001
2–4
V
51.2335 ± 60.1664
15.0893 ± 15.4565
0.1271 ± 0.1760
0.8189 ± 0.3210
18.4958
25
20
VI
28.3684 ± 46.4711
7.4601 ± 9.2279
0.0490 ± 0.1161
0.9014 ± 0.1963
251.1328
2–4
0.01
20
VII
14.2389 ± 18.4680
4.8588 ± 4.3811
0.0137 ± 0.0253
0.9908 ± 0.1313
750.8074
25
0.01
21
VIII
11.1153 ± 17.0344
4.3034 ± 4.3046
0.0097 ± 0.0213
0.9701 ± 0.1138
20.4763
22
0.001The performances of eight methods on 12 data sets are summarized in Table4. We can see that the introduction of 1-norm in LTSVR improves the efficiency in some cases, but this advantage is not valid for large-sized data sets. On the contrary, LTQSSVR greatly improves the training efficiency in most of the cases. It can be seen from Figures 3 and 4 that the processing time of LTQSSVR is much shorter than those of other methods; hence, LTQSSVR has a large advantage on the aspect of efficiency. This is because our kernel-free model avoids the time-consuming task of selecting a proper kernel and parameters. Besides, the structure of two small-scaled linear programming problems further accelerates the computational speed [29–32].Figure 3
CPU time on the general scale public benchmark data sets of LSSVR, TSVR, LTSVR, and LTQSSVR.Figure 4
CPU time on the large scale public benchmark data sets of LSSVR, TSVR, LTSVR, and LTQSSVR.From Table4, we can easily check out that our proposed method generates stable regressions on most of the data sets. It achieves the best accuracy on two data sets (Auto mpg, Computer hardware), the second best accuracy on one data sets (Hayes roth), the third best accuracy on two data sets (Body fat, Haberman), and the fourth best accuracy on five data sets (Oring, Wis, Real estate valuation, Slump, Ozone). We summarize the average rank information about these eight methods on 12 data sets in Table 5.Table 5
Average rank on RMSE of eight slgorithms.
Data set
RR
LR
nu-SVR
ϵ-SVR
LSSVR
TSVR
LTSVR
LTQSSVR
Hayes roth
6
1
7
8
3
5
4
2
Oring
1
2
7
8
3
5
6
4
Container-CC
6
3
4
2
1
7
8
5
Daily demand
7
8
1
3
6
4
2
5
Haberman
5
8
7
6
4
2
1
3
Wis
5
3
6
8
7
1
2
4
Auto mpg
6
5
8
7
3
2
4
1
Real estate valuation
8
7
1
2
5
3
6
4
Slump
8
7
1
2
6
3
5
4
Ozone
7
3
1
8
5
2
6
4
Body fat
8
4
5
6
2
7
1
3
Computer hardware
5
4
7
8
6
3
2
1
Average
6.08
4.17
5.08
5.83
3.92
3.83
3.83
3.25Then, we employ the Friedman test to check the difference of all methods under the null hypothesis that all the algorithms are no significantly different from the mean rank:(26)χF2=12Nkk+1∑jRj2−kk+124,where Rj=1/N∑irij and rij denotes the jth of k methods on the ith of N data sets. Then, the Friedmans FF is obtained by(27)FF=N−1χF2Nk−1−χF2,With eight methods and 12 data sets, FF is distributed according to the F distribution with (7, 77) degrees of freedom. According to (26), (27), and Table 5, we can obtain χF2=14.8364 and FF = 2.3593. The critical value of F (7, 77) for α = 0.05 is 2.131 and similarly is 1.796 for α = 0.1, so we reject both levels of the null hypothesis. We can identify that there is a significantly different between the eight methods. Note that LTQSSVR obtains the smallest average rank which is 3.25. It means that the accuracy of our approach is also very close to the best one.As for the processing time, Ridge Regression and Linear Regression take the shortest time among these methods, which lack robustness. In addition, LTQSSVR takes much shorter computational times than TSVR and LTSVR on all data sets. For the large data sets, TSVR and LTSVR require a lot of time for solving. This is because they have to tuning those parameters in the kernel function. In particular, we only used the Gaussian kernel function in this paper. Hence, if we add the selecting process of kernel function to those kernel-based models, their efficiencies will be further dragged down a lot. Therefore, these methods are not suitable for dealing with those huge-sized cases and their applicabilities would be strictly limited in this big data era. In contrast, due to the kernel-free and linear programming structure, our new approach has a strong ability to handling these huge-sized data sets quickly. Above all, our proposed method not only has a good generalization ability but also achieves a great efficiency.


## 6. Conclusion

In this paper, we proposed a new approach of TSVR, which directly uses two quadratic surfaces in the original space to regress the data sets. It is worth noting that our new approach avoids the difficult and time-consuming task for searching a suitable kernel function and its corresponding parameters in the traditional kernel-based SVR methods. Besides, in order to make further improvement of the efficiency and robustness, our model incorporates the 1-norm to measure the regression error. The corresponding linear programming structure avoids the matrix inverse operation and leads to a high efficiency even for those huge-sized problems. Finally, the experimental results on different data sets demonstrate the validity and applicability of our method. Specifically, compared with those benchmark nonlinear regression models, our model is superior in both accuracy and efficiency.For the future work, since the advantage of our method is not obvious on efficiency for those high-dimensional data sets, we hope to incorporate a feature selection method to improve it.



---

*Source: 3238129-2020-04-04.xml*