# Robust Matching Pursuit Extreme Learning Machines

Authors: Zejian Yuan; Xin Wang; Jiuwen Cao; Haiquan Zhao; Badong Chen
Journal: Scientific Programming
 (2018)

---

## Abstract

Extreme learning machine (ELM) is a popular learning algorithm for single hidden layer feedforward networks (SLFNs). It was originally proposed with the inspiration from biological learning and has attracted massive attentions due to its adaptability to various tasks with a fast learning ability and efficient computation cost. As an effective sparse representation method, orthogonal matching pursuit (OMP) method can be embedded into ELM to overcome the singularity problem and improve the stability. Usually OMP recovers a sparse vector by minimizing a least squares (LS) loss, which is efficient for Gaussian distributed data, but may suffer performance deterioration in presence of non-Gaussian data. To address this problem, a robust matching pursuit method based on a novel kernel risk-sensitive loss (in short KRSLMP) is first proposed in this paper. The KRSLMP is then applied to ELM to solve the sparse output weight vector, and the new method named the KRSLMP-ELM is developed for SLFN learning. Experimental results on synthetic and real-world data sets confirm the effectiveness and superiority of the proposed method.


---

## Body

## 1. Introduction

Extreme learning machine [1] is a kind of single hidden layer feedforward network (SLFN) [2]. In the past decade, ELM became popular and attractive in the machine learning and pattern recognition communities for its fast adaptability and good generalization performance [3]. In general, ELM has the following advantages: (i) It not only has the ability of estimating the unknown mathematical model embedded in a mass of training samples but also possesses parallel schemes to be efficiently implemented in parallel for training and testing; (ii) it uses randomly generated input weights and hidden biases without tuning during the training phase, and therefore, the output weights can be analytically obtained by solving the standard least squares (LS) problem. Thus, extremely fast learning ability and efficient computation cost can be achieved, especially for big data applications. In view of these remarkable superiorities, ELM has been widely applied in many applications, such as face recognition [4], series compensated transmission line protection [5], time series analysis [6], and nonlinear model identification [7].However, ELM still has several drawbacks. First, ELM encounters the problem of irrelevant variables when handling real-world data sets [8]. Second, choosing a proper hidden nodes number is an open problem for all ELM algorithms. An ELM network with too few hidden nodes may not be accurate for modeling the input data, whereas a network with too many hidden nodes tends to generate an overfitting model [9]. Moreover, when the number of hidden nodes is more than the input data, ELM might have the singularity problem [4]. Third, the original ELM learns the model with an L2-norm based loss function, which is very vulnerable to noise. It is well known that the L2-norm can magnify the bad effects of outliers associated with large deviations [10]. The presence of non-Gaussian noises or outliers in the training data may thus lead to an unreliable model with degraded performance.To overcome the first and second limitations, several methods have been proposed in the regularization framework [9, 11–13]. Furthermore orthogonal matching pursuit (OMP) is a plain and efficient iterative algorithm which chooses an atom in the dictionary with the best correlation to the remaining elements at each iteration [14]. As such, OMP has been embedded to ELM (OMP-ELM) to overcome the singularity problem and led to more stable solution than the original ELM [15]. Most of the existing methods learn the model with an L2-norm based loss function, which may perform poorly in the presence of non-Gaussian noises (which exist in many real-world situations) or outliers [16–18]. To combat non-Gaussian noises or outliers and improve the generalization ability, the regularized correntropy criterion is used to replace the L2-norm based loss function in original ELM model to develop the ELM-RCC [16]. In [19], ELM with L1-norm based loss function (ORELM) was proposed to achieve robust performance.The kernel risk-sensitive loss (KRSL) is a nonlinear similarity measure firstly proposed in [20], which can reach a more satisfying robust performance. The KRSL is based on the original structure of risk-sensitive loss and is defined in the reproducing kernel Hilbert space (RKHS) [21, 22]:(1)VX,Y=1λEexpλ1-κσX-Y,where E[·] denotes the mathematical expectation, κσ(·) is the Gaussian kernel with bandwidth σ, and λ is the risk-sensitive parameter. In this paper, we propose a KRSL based matching pursuit (KRSLMP) method. The KRSLMP is then embedded to ELM to construct a robust and sparse ELM model.The rest of the paper is structured as follows. In Section2, we sketch the related work, including similarity measures in kernel space, kernel risk-sensitive loss, ELM model, and orthogonal matching pursuit algorithm. In Section 3, we develop the KRSLMP-ELM. In Section 4, experiments on regression problem with synthetic and real-world data sets are conducted to verify the effectiveness of the proposed algorithm. The sensitivity of the KRSLMP-ELM to free parameters is also analyzed. Finally, conclusion is given in Section 5.


## 2. Preliminaries and Related Works

For convenience of presentation, the following notations used in this paper are introduced. Vectors and matrices are represented with boldface lowercase letters and boldface capital letters, respectively. For any vectorx, we use x(i) to denote its ith entry. The notation x|I denotes the subvector of x∈Rn with entries indexed by the set I⊂Ω={1,2,…,n}. The complementary set of I is denoted as Ic=Ω-I.


### 2.1. Similarity Measures in Kernel Space

LetX and Y be two random variables; the correntropy between X and Y is defined by [17, 23](2)VX,Y=EκσX-Y=∫κσx-ydFXYx,y,where FXY(x,y) is the joint distribution function of (X,Y). The Gaussian kernel with bandwidth σ is given by(3)κσx-y=exp-x-y22σ2.CorrentropyV(X,Y) is a local correlation measure in the kernel space H. According to Mercers theorem [24], it can be expressed in terms of the inner product as(4)VX,Y=EΦX,ΦYH.It applies a kernel trick that nonlinearly maps the original space to a higher dimensional feature space. It can be shown that correntropy is directly related to the probability of how similar two random variables are in a neighborhood of the joint space controlled by the kernel bandwidthσ [17, 25, 26].


### 2.2. Kernel Risk-Sensitive Loss

Similarity measures in kernel space have the ability to extract higher-order statistics of data, which can significantly improve the learning performance in non-Gaussian environments [21]. The optimization problem can be determined by maximizing the correntropy criterion (MCC) or equivalently minimizing the correntropic loss (C-Loss) [27, 28] between the output estimation and the target response. However, highly nonconvex problem may happen in C-Loss performance surface which has steep slopes around the optimal solution but is extremely flat far from the solution. This may lead to slow convergence and poor performance. Choosing a large kernel bandwidth may overcome the above problem. But the robustness will decrease significantly when outliers occur with kernel bandwidth increasing [29]. To achieve a satisfying performance surface, the KRSL was proposed in [20].The KRSL is defined by(5)LλX,Y=1λEexpλ1-κσX-Y=1λ∫expλ1-κσx-ydFXYx,ywhich can also be expressed in a traditional risk-sensitive loss form as [30](6)LλX,Y=1λEexpλ12ΦX-ΦYH2,where λ is the risk-sensitive parameter that controls the shape of performance surface.In practice, the joint distribution function ofX and Y is usually unknown and only a finite number of samples {(xj,yj)}j=1M are available. The KRSL can thus be estimated by(7)L^λX,Y=1Mλ∑j=1Mexpλ1-κσxj-yj.As one can see, (6) defines a distance between the vectors X=[x1,x2,…,xM]T and Y=[y1,y2,…,yM]T.


### 2.3. Extreme Learning Machine

Extreme learning machine (ELM) was proposed by Huang et al. for training single hidden layer feedforward neural networks (SLFNs) [2, 31]. The input weights and biases are initialized randomly in ELM and remain unchanged during training. The network learning thus becomes optimizing the output weights, which can be formulated as solving a linear equation. Let {(xj,yj)}j=1M be given by M training samples, where input xj∈Rn and corresponding desired output yj∈R; the relationship between xj and yj can be represented under the assumption of the model. The network model of ELM with L hidden neurons can be modeled and expressed as(8)∑i=1Lβifai·xj+bi=yj,j=1,2,…,M,where L is hidden nodes number, βi is the weight connecting the ith hidden node and output nodes, f is the activation function (in this work, f is a sigmoid function without explicit mention), ai denotes the weight that connects the ith hidden node and input nodes, and bi represents the randomly chosen bias of the ith hidden node. Equation (7) can be compactly written as a matrix notation(9)y=Hβ,where(10)H=fa1·x1+b1⋯faL·x1+bL⋮⋱⋮fa1·xM+b1⋯faL·xM+bLand β is the minimal norm least squares solution of (8). The parameter β can be obtained by(11)β=H†y,where H† is the Moore Penrose generalized inverse of the hidden layer output matrix H.


### 2.4. Orthogonal Matching Pursuit

Matching pursuit method is one of the effective methods for sparse representation [14, 32, 33]. In general, a sparse representation problem can be formulated as(12)y=Ax+ρ,where A∈Rm×n(m<n) denotes the measurement matrix, x is the sparse vector, and ρ∈Rm represents the noise vector. The main purpose is to recover the sparse vector x from the observation y and the measurement matrix A. The OMP uses the L0-norm constrained least squares model(13)minxy-Ax22s.t.x0≤K,where x0 counts the number of nonzero coordinates of x.In the following, we briefly describe the OMP method. First, we initialize the residualr0=y, the index set Λ0=∅, and the iteration t=1. At each iteration, OMP algorithm selects a column of the measurement matrix A which is most correlated to the residual as(14)αt=argmaxi=1,…,nrt-1,φi,where rt-1 denotes the residual in t-1th iteration and φi is the ith column of A. Then collect αt to index set Λ(15)Λt=Λt-1∪αt.We can solve an LS problem to obtain a new estimationxt supported in Λt:(16)xt=argminx∈Rn,suppx⊂Λty-Ax2,where supp(x) denotes the support set of x. If the stopping criterion is satisfied, we output xt as the estimate of x.Then one can update the residual(17)rt=y-Axt.From (8) and (11), we can find that ELM has a similar network model for sparse representation problem. Thus, one can take advantage of the OMP algorithm for selecting the best hidden nodes of the ELM network. The OMP estimates the sparse vector by using the L2-norm based criterion, which performs well with the Gaussian error distribution. However, the presence of non-Gaussian noise may give rise to performance degradation.


## 2.1. Similarity Measures in Kernel Space

LetX and Y be two random variables; the correntropy between X and Y is defined by [17, 23](2)VX,Y=EκσX-Y=∫κσx-ydFXYx,y,where FXY(x,y) is the joint distribution function of (X,Y). The Gaussian kernel with bandwidth σ is given by(3)κσx-y=exp-x-y22σ2.CorrentropyV(X,Y) is a local correlation measure in the kernel space H. According to Mercers theorem [24], it can be expressed in terms of the inner product as(4)VX,Y=EΦX,ΦYH.It applies a kernel trick that nonlinearly maps the original space to a higher dimensional feature space. It can be shown that correntropy is directly related to the probability of how similar two random variables are in a neighborhood of the joint space controlled by the kernel bandwidthσ [17, 25, 26].


## 2.2. Kernel Risk-Sensitive Loss

Similarity measures in kernel space have the ability to extract higher-order statistics of data, which can significantly improve the learning performance in non-Gaussian environments [21]. The optimization problem can be determined by maximizing the correntropy criterion (MCC) or equivalently minimizing the correntropic loss (C-Loss) [27, 28] between the output estimation and the target response. However, highly nonconvex problem may happen in C-Loss performance surface which has steep slopes around the optimal solution but is extremely flat far from the solution. This may lead to slow convergence and poor performance. Choosing a large kernel bandwidth may overcome the above problem. But the robustness will decrease significantly when outliers occur with kernel bandwidth increasing [29]. To achieve a satisfying performance surface, the KRSL was proposed in [20].The KRSL is defined by(5)LλX,Y=1λEexpλ1-κσX-Y=1λ∫expλ1-κσx-ydFXYx,ywhich can also be expressed in a traditional risk-sensitive loss form as [30](6)LλX,Y=1λEexpλ12ΦX-ΦYH2,where λ is the risk-sensitive parameter that controls the shape of performance surface.In practice, the joint distribution function ofX and Y is usually unknown and only a finite number of samples {(xj,yj)}j=1M are available. The KRSL can thus be estimated by(7)L^λX,Y=1Mλ∑j=1Mexpλ1-κσxj-yj.As one can see, (6) defines a distance between the vectors X=[x1,x2,…,xM]T and Y=[y1,y2,…,yM]T.


## 2.3. Extreme Learning Machine

Extreme learning machine (ELM) was proposed by Huang et al. for training single hidden layer feedforward neural networks (SLFNs) [2, 31]. The input weights and biases are initialized randomly in ELM and remain unchanged during training. The network learning thus becomes optimizing the output weights, which can be formulated as solving a linear equation. Let {(xj,yj)}j=1M be given by M training samples, where input xj∈Rn and corresponding desired output yj∈R; the relationship between xj and yj can be represented under the assumption of the model. The network model of ELM with L hidden neurons can be modeled and expressed as(8)∑i=1Lβifai·xj+bi=yj,j=1,2,…,M,where L is hidden nodes number, βi is the weight connecting the ith hidden node and output nodes, f is the activation function (in this work, f is a sigmoid function without explicit mention), ai denotes the weight that connects the ith hidden node and input nodes, and bi represents the randomly chosen bias of the ith hidden node. Equation (7) can be compactly written as a matrix notation(9)y=Hβ,where(10)H=fa1·x1+b1⋯faL·x1+bL⋮⋱⋮fa1·xM+b1⋯faL·xM+bLand β is the minimal norm least squares solution of (8). The parameter β can be obtained by(11)β=H†y,where H† is the Moore Penrose generalized inverse of the hidden layer output matrix H.


## 2.4. Orthogonal Matching Pursuit

Matching pursuit method is one of the effective methods for sparse representation [14, 32, 33]. In general, a sparse representation problem can be formulated as(12)y=Ax+ρ,where A∈Rm×n(m<n) denotes the measurement matrix, x is the sparse vector, and ρ∈Rm represents the noise vector. The main purpose is to recover the sparse vector x from the observation y and the measurement matrix A. The OMP uses the L0-norm constrained least squares model(13)minxy-Ax22s.t.x0≤K,where x0 counts the number of nonzero coordinates of x.In the following, we briefly describe the OMP method. First, we initialize the residualr0=y, the index set Λ0=∅, and the iteration t=1. At each iteration, OMP algorithm selects a column of the measurement matrix A which is most correlated to the residual as(14)αt=argmaxi=1,…,nrt-1,φi,where rt-1 denotes the residual in t-1th iteration and φi is the ith column of A. Then collect αt to index set Λ(15)Λt=Λt-1∪αt.We can solve an LS problem to obtain a new estimationxt supported in Λt:(16)xt=argminx∈Rn,suppx⊂Λty-Ax2,where supp(x) denotes the support set of x. If the stopping criterion is satisfied, we output xt as the estimate of x.Then one can update the residual(17)rt=y-Axt.From (8) and (11), we can find that ELM has a similar network model for sparse representation problem. Thus, one can take advantage of the OMP algorithm for selecting the best hidden nodes of the ELM network. The OMP estimates the sparse vector by using the L2-norm based criterion, which performs well with the Gaussian error distribution. However, the presence of non-Gaussian noise may give rise to performance degradation.


## 3. Kernel Risk-Sensitive Loss Based Matching Pursuit Extreme Learning Machine

To address the aforementioned issue, we propose a robust kernel risk-sensitive loss based orthogonal matching pursuit extreme learning machine algorithm (KRSLMP-ELM) in this section. In the KRSLMP-ELM, we initialize the residualr0 as y and the initial index set as Λ0=∅. Then, similar to OMP, a column of H most correlated with the residual is selected and the index set is augmented at each iteration. Then we obtain a new estimation βt by solving the following KRSL minimization problem:(18)βt=argminβ∈RL,suppβ⊂Λtϕσ,λy-Hβ+Cβ22.We utilize the half-quadratic (HQ) theory [34] to construct the optimization algorithm. Considering that the measurements may include both large and small noise, we can use HQ optimization to estimate the importance of different samples. The samples severely corrupted will be assigned small weight values in learning procedure to decrease the impact of large noise. Thus, the performance of KRSLMP-ELM can be significantly further improved.According to the convex optimization theory [35], the dual function for ϕσ,λ(x)=1/λexp(λ(1-exp(-x2/2σ2)))(0<λ<1) is convex and defined as(19)ψs=supt∈R-sx2+ϕσ,λxand then(20)ϕσ,λx=infs∈Rsx2+ψs,where the infimum is reached at s=ϕσ,λ(x). We point out here that when the parameter λ>1, the KRSLMP-ELM can also work well in our simulations. Substituting (18) for (20), the KRSLMP-ELM objective function can be reformulated as(21)minβ∈RL,w∈R+Msuppβ⊂Λk⁡Jβ,w=diagwy-Hβ22+Cβ22+∑i=1Mψwi,where diag(w) represents a diagonal matrix with its primary diagonal element w(i) and C is the regularization parameter. Inspired by the HQ theory, (21) can be solved by the following alternate technique:(22)wt+1i=1λexpλ1-κσyi-Hβti,βt+1=argminβ∈RL,suppβ⊂Λkdiagwt+1y-Hβ22+Cβ22,where t denotes the iteration number. In the proposed algorithm, the bandwidth is adaptively chosen during the iteration. In order to make the scheme robust to outliers, we calculate the value of σ as follows.Denote the training error ase(i)=y(i)-Hβ(i)22, i=1,2,…,M. We can then reorder the error in an ascending order, and we get the reordered as eσ. Let k=⌊τM⌋, where scalar τ∈(0,1] and ⌊τM⌋ outputs the largest integer smaller than τM. We can select eσ(k) as the bandwidth in accordance with the proportion of outlier. Discussions on the detailed experimental results by choosing different bandwidths are given in the experiment section. A solution for the optimization problem in (21) can be derived as follows:(23)βt+1Λt=HTdiagwt+1H+1CI-1HTdiagwt+1y,where β(t+1)|Λtc=0 and I denotes the identity matrix.Since the importance degree of the measurements is employed to adaptively update the output weight vector in the KRSLMP-ELM, we update the residual(24)rt=diagwty-Hβt.It is noted that the sparsity levelK has to be assigned in advance in the KRSLMP-ELM. The sparsity K directly determines the number of the active hidden nodes used in ELM due to the fact that more hidden nodes than necessary are generated. To obtain the best sparsity level K, namely, the best number of hidden nodes used in ELM, we utilize the root mean square error (RMSE) as the criterion(25)RMSE=∑i=1Myi-y^i2M,where yi denotes the target response and y^i the corresponding output estimated by the KRSLMP-ELM.For different sparsity levelK, the corresponding RMSE is first calculated. Then the best β coefficients associated with the minimum RMSE value are selected.The iteration is repeated until achieving the stopping criterion. The KRSLMP-ELM is summarized in Algorithm1.Algorithm 1:KRSLMP-ELM.
Input: samples {xi,yi}i=1M
Output: weight vector β
Parameters setting: number of hidden nodes L~, regularization parameter C and sparsity level K.
Initialization: randomly initialize ELM parameters: input weights ai and biases bi(i=1,…,L) in measurement matrix H.
Set the index setΛ0=∅, the residual r0=y, the iteration counter t=0 and diag(w0)=I.
(
1
)  for    t=1,2,…,K  do
(
2
)    t=t+1
(
3
)    Find a column of H most correlated with the residual
αt=argmaxj=1,2,…,Lrt-1,diagwt-1⋅hj
(
4
)    Augment the index set
Λt=Λt-1∪αt
(
5
)    Solve the KRSLMP minimization problem by the following iterations
wi(t+1)=1λexpλ1-κσyi-Hβti
β(t+1)=argminβ∈RL,supp(β)⊂Λkdiag(w(t+1))(y-Hβ)22+Cβ22
The solution is denoted as(w(t),β(t))
(
6
)    Update residual rt=diag(w(t))(y-Hβ(t))
(
7
) end   for


## 4. Experimental Results

To validate the effectiveness of the proposed KRSLMP-ELM algorithm, experiments on two synthetic data sets and seven benchmark data sets are conducted in this section. The performance of the new method is compared to five state-of-the-art algorithms, namely, ELM, RELM, ELM-RCC, OMP-ELM, and ORELM. Sigmoid functionf(x)=1/1+e-x is used as the activation function for all methods.


### 4.1. Synthetic Data Sets

In this subsection, experiments on two synthetic regression data sets for nonlinear function approximation problem are carried out. Descriptions of the two data sets are as follows.Sinc.The synthetic data set is generated by yi=c·Sinc(xi)+ρi, where c=8 and(26)Sincx=sinxxx≠01x=0and ρi contains two mutually independent noises that are inner noise Bi and outliers noise Oi. Specifically, ρi is defined as ρi=(1-gi)Bi+giOi, where gi is binary distributed with the probability masses Pr{gi=1}=p and Pr{gi=0}=1-p(0≤p≤1). Bi and Oi are independent of gi. In this experiment, p is set at 0.1. The outlier Oi is generated by using a zero-mean Gaussian distributed noise with standard deviation 4.0. For the inner noise Bi, two different noises are tested, which are (a) uniform distribution over [-1.0,1.0] and (b) Sine wave noise sin(α), with α uniformly distributed over [0,2π]. We uniformly generate the input data xi from [-10.0,10.0], where 200 data points are used for training and another 200 clean data points which are not contaminated by any noise are used for testing.Func. This synthetic data set is generated by(27)yx1,x2=x1exp-x12+x22+ρ,where ρ is a zero-mean Gaussian distributed noise vector with standard deviation 0.4. The input data vectors x1 and x2 are uniformly generated from [-2.0,2.0]. Similar to the previous experiments, 200 data samples are used for training and another 200 data samples without noise are used for testing.Parameters used in the six methods for experiments of the two synthetic data sets are summarized in Table1, where L, C, K, and λ represent the number of hidden layer nodes, regularization parameter, sparsity level, and risk-sensitive parameter in KRSLMP-ELM. We set τ=0.9 in Sinc synthetic data set experiment and τ=1 in Func synthetic data set experiment. For the convenient distinguishment of the proposed method with other methods in Sinc function approximation problem, only the estimation results of the original ELM, ORELM, ELM-RCC, and KRSLMP-ELM are illustrated in Figure 1. In Figure 2, we plot the squared training errors obtained by the KRSLMP-ELM, ELM-RCC, ORELM, and the original ELM, respectively. As shown in these figures, the KRSLMP-ELM wins the best approximation performance. The testing RMSEs of six algorithms are presented in Table 2. It is indicated that the KRSLMP-ELM is more robust than the other five methods.Table 1
Parameter settings of four algorithms in function fitting.
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
L
L
C
L
K
L
C
L
σ
C
L
K
C
λ
Sinc-Uniform
10
80
1
0
-
4
200
10
80
1
0
-
4
100
1
1
0
-
4
200
50
2
×
1
0
-
5
0.01
Sinc-Sine wave
10
40
2
×
1
0
-
5
200
10
90
1
0
-
4
50
1.1
2
×
1
0
-
5
200
50
1
0
-
5
0.05
Func
35
35
1
0
-
6
200
20
100
1
0
-
9
70
3
1
0
-
4
200
70
1
0
-
5
0.001Table 2
Testing RMSEs of six methods.
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
Sinc-Uniform
0.4737
0.2871
0.3038
0.2369
0.1948
0.1485
Sinc-Sine wave
0.4406
0.2935
0.2711
0.2798
0.2155
0.1562
Func
0.0652
0.0638
0.0637
0.0591
0.0582
0.0434Figure 1
Sinc function regression results with different inner noises: (a) Uniform; (b) Sine wave.
(a)
(b)Figure 2
Squared training errors of Func function regression: (a) KRSLMP-ELM; (b) ELM-RCC; (C) ORELM; (D) original ELM.
(a)
(b)
(c)
(d)Further, we perform another experiment to compare the performance of KRSLMP-ELM to that of the original ELM with different outliers. We consider the Sinc function approximation problem and set the inner noise as a zero-mean Gaussian distributed noise with standard deviation 0.1, and the outliers noise is zero-mean Gaussian with standard deviation ranging between 0.1 and 10. We run 100 trials for different outliers noises and show the RMSE results in Figure3. One can see that the original ELM’s performance degrades severely when the outliers get enhanced while the KRSLMP-ELM’s performance is much less influenced by outliers.Figure 3
Sinc function regression results with different outliers noises.


### 4.2. Benchmark Data Sets

In this subsection, seven benchmark regression data sets from UCI machine learning repository [36] are tested to support the superiority of the proposed method. Specifications of the data sets are shown detailedly in Table 3. It should be pointed out that the training and testing data samples are randomly chosen in each data set and all the features are normalized into [0,1]. The parameters of each method are all chosen by the fivefold cross-validation and are given in Table 4. For all algorithms, 100 independent trials are conducted and the average results are reported. The training and testing RMSEs and their standard deviation of all algorithms are listed in Table 5. As highlighted in boldface, the ELM-KRSLMP achieves the best performance in most regression data sets.Table 3
Specification of the data sets.
Data sets
Features
Observations
Training
Testing
Servo
5
83
83
Auto MPG
7
192
200
Body fat
14
126
126
Concrete
9
515
515
Housing
14
253
253
Yacht
6
154
154
Airfoil
5
751
751Table 4
Parameter settings of six methods.
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
L
L
C
L
K
L
C
L
σ
C
L
K
C
λ
Servo
25
90
1
0
-
5
100
20
120
1
0
-
6
65
0.8
1
0
-
4
200
40
5
×
1
0
-
5
1.5
Auto MPG
20
40
1
0
-
4
50
15
100
1
0
-
4
100
0.3
1
0
-
2
200
80
1
0
-
3
0.8
Body fat
20
100
1
0
-
2
50
15
100
1
0
-
3
160
0.1
1
0
-
1
100
25
1
0
-
2
1.1
Concrete
120
185
2
×
1
0
-
4
200
80
200
1
0
-
7
200
0.6
5
×
1
0
-
6
500
140
1
0
-
5
0.6
Housing
40
180
2
×
1
0
-
4
200
30
200
1
0
-
5
200
0.8
1
0
-
3
500
150
1
0
-
3
0.5
Yacht
90
185
2
×
1
0
-
5
200
60
200
1
0
-
9
195
0.4
1
0
-
7
500
145
1
0
-
8
0.3
Airfoil
130
200
2
×
1
0
-
4
200
85
180
1
0
-
9
150
0.4
1
0
-
7
500
140
1
0
-
8
1.0Table 5
Training and testing RMSEs for different data sets.
Data sets
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
Training
Testing
Training
Testing
Training
Testing
Training
Testing
Training
Testing
Training
Testing
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
Servo
0.0745
0.1189
0.0583
0.1044
0.0727
0.1134
0.0849
0.1036
0.0749
0.1034
0.0841
0.1017
±
±
±
±
±
±
±
±
±
±
±
±
0.0133
0.0228
0.0105
0.0189
0.0123
0.0181
0.0174
0.0211
0.0117
0.0171
0.0115
0.0176
Auto MPG
0.0689
0.0785
0.0627
0.0765
0.0676
0.0777
0.0705
0.0764
0.0683
0.0757
0.0632
0.0749
±
±
±
±
±
±
±
±
±
±
±
±
0.0049
0.0053
0.0044
0.0043
0.0044
0.0045
0.0047
0.0052
0.0042
0.0045
0.0043
0.0045
Body fat
0.0237
0.0340
0.0196
0.0278
0.0218
0.0313
0.0253
0.0233
0.0240
0.0236
0.0252
0.0239
±
±
±
±
±
±
±
±
±
±
±
±
0.0073
0.0060
0.0076
0.0062
0.0076
0.0059
0.0118
0.0119
0.0093
0.0087
0.0097
0.0095
Concrete
0.0615
0.1001
0.0732
0.0925
0.0654
0.0981
0.0668
0.0929
0.0557
0.0882
0.0542
0.0864
±
±
±
±
±
±
±
±
±
±
±
±
0.0025
0.0125
0.0022
0.0041
0.0026
0.0101
0.0026
0.0069
0.0018
0.0077
0.0018
0.0067
Housing
0.0736
0.0990
0.0443
0.0897
0.0644
0.0935
0.0576
0.0899
0.0453
0.0874
0.0503
0.0849
±
±
±
±
±
±
±
±
±
±
±
±
0.0053
0.0094
0.0041
0.0136
0.0048
0.0108
0.0061
0.0160
0.0039
0.0129
0.0040
0.0114
Yacht
0.0041
0.0583
0.0300
0.0483
0.0154
0.0437
0.0189
0.0373
0.0126
0.0320
0.0060
0.0250
±
±
±
±
±
±
±
±
±
±
±
±
0.0004
0.1374
0.0002
0.0064
0.0014
0.0134
0.0013
0.0073
0.0008
0.0063
0.0005
0.0099
Airfoil
0.0664
0.0967
0.0925
0.0991
0.0731
0.0968
0.0806
0.0951
0.0740
0.0907
0.0635
0.0882
±
±
±
±
±
±
±
±
±
±
±
±
0.0020
0.0130
0.0021
0.0023
0.0024
0.0105
0.0023
0.0052
0.0021
0.0061
0.0021
0.0068


### 4.3. Sensitivity of Parameters

We analyze the sensitivity of the parametersL, K, C, and λ of KRSLMP-ELM in this subsection. For illustration, we use the regression results obtained by the Servo data set as an example. For each parameter, its sensitivity is tested by fixing the remaining parameters as the ones used in Table 4. Then, the testing RMSEs are recorded as criteria for performance comparison. The results of the regression performance are demonstrated in Figure 4.Figure 4
Regression results with different parameters: (a)L; (b) 1/C; (c) K; (d) λ.
(a)
(b)
(c)
(d)


## 4.1. Synthetic Data Sets

In this subsection, experiments on two synthetic regression data sets for nonlinear function approximation problem are carried out. Descriptions of the two data sets are as follows.Sinc.The synthetic data set is generated by yi=c·Sinc(xi)+ρi, where c=8 and(26)Sincx=sinxxx≠01x=0and ρi contains two mutually independent noises that are inner noise Bi and outliers noise Oi. Specifically, ρi is defined as ρi=(1-gi)Bi+giOi, where gi is binary distributed with the probability masses Pr{gi=1}=p and Pr{gi=0}=1-p(0≤p≤1). Bi and Oi are independent of gi. In this experiment, p is set at 0.1. The outlier Oi is generated by using a zero-mean Gaussian distributed noise with standard deviation 4.0. For the inner noise Bi, two different noises are tested, which are (a) uniform distribution over [-1.0,1.0] and (b) Sine wave noise sin(α), with α uniformly distributed over [0,2π]. We uniformly generate the input data xi from [-10.0,10.0], where 200 data points are used for training and another 200 clean data points which are not contaminated by any noise are used for testing.Func. This synthetic data set is generated by(27)yx1,x2=x1exp-x12+x22+ρ,where ρ is a zero-mean Gaussian distributed noise vector with standard deviation 0.4. The input data vectors x1 and x2 are uniformly generated from [-2.0,2.0]. Similar to the previous experiments, 200 data samples are used for training and another 200 data samples without noise are used for testing.Parameters used in the six methods for experiments of the two synthetic data sets are summarized in Table1, where L, C, K, and λ represent the number of hidden layer nodes, regularization parameter, sparsity level, and risk-sensitive parameter in KRSLMP-ELM. We set τ=0.9 in Sinc synthetic data set experiment and τ=1 in Func synthetic data set experiment. For the convenient distinguishment of the proposed method with other methods in Sinc function approximation problem, only the estimation results of the original ELM, ORELM, ELM-RCC, and KRSLMP-ELM are illustrated in Figure 1. In Figure 2, we plot the squared training errors obtained by the KRSLMP-ELM, ELM-RCC, ORELM, and the original ELM, respectively. As shown in these figures, the KRSLMP-ELM wins the best approximation performance. The testing RMSEs of six algorithms are presented in Table 2. It is indicated that the KRSLMP-ELM is more robust than the other five methods.Table 1
Parameter settings of four algorithms in function fitting.
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
L
L
C
L
K
L
C
L
σ
C
L
K
C
λ
Sinc-Uniform
10
80
1
0
-
4
200
10
80
1
0
-
4
100
1
1
0
-
4
200
50
2
×
1
0
-
5
0.01
Sinc-Sine wave
10
40
2
×
1
0
-
5
200
10
90
1
0
-
4
50
1.1
2
×
1
0
-
5
200
50
1
0
-
5
0.05
Func
35
35
1
0
-
6
200
20
100
1
0
-
9
70
3
1
0
-
4
200
70
1
0
-
5
0.001Table 2
Testing RMSEs of six methods.
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
Sinc-Uniform
0.4737
0.2871
0.3038
0.2369
0.1948
0.1485
Sinc-Sine wave
0.4406
0.2935
0.2711
0.2798
0.2155
0.1562
Func
0.0652
0.0638
0.0637
0.0591
0.0582
0.0434Figure 1
Sinc function regression results with different inner noises: (a) Uniform; (b) Sine wave.
(a)
(b)Figure 2
Squared training errors of Func function regression: (a) KRSLMP-ELM; (b) ELM-RCC; (C) ORELM; (D) original ELM.
(a)
(b)
(c)
(d)Further, we perform another experiment to compare the performance of KRSLMP-ELM to that of the original ELM with different outliers. We consider the Sinc function approximation problem and set the inner noise as a zero-mean Gaussian distributed noise with standard deviation 0.1, and the outliers noise is zero-mean Gaussian with standard deviation ranging between 0.1 and 10. We run 100 trials for different outliers noises and show the RMSE results in Figure3. One can see that the original ELM’s performance degrades severely when the outliers get enhanced while the KRSLMP-ELM’s performance is much less influenced by outliers.Figure 3
Sinc function regression results with different outliers noises.


## 4.2. Benchmark Data Sets

In this subsection, seven benchmark regression data sets from UCI machine learning repository [36] are tested to support the superiority of the proposed method. Specifications of the data sets are shown detailedly in Table 3. It should be pointed out that the training and testing data samples are randomly chosen in each data set and all the features are normalized into [0,1]. The parameters of each method are all chosen by the fivefold cross-validation and are given in Table 4. For all algorithms, 100 independent trials are conducted and the average results are reported. The training and testing RMSEs and their standard deviation of all algorithms are listed in Table 5. As highlighted in boldface, the ELM-KRSLMP achieves the best performance in most regression data sets.Table 3
Specification of the data sets.
Data sets
Features
Observations
Training
Testing
Servo
5
83
83
Auto MPG
7
192
200
Body fat
14
126
126
Concrete
9
515
515
Housing
14
253
253
Yacht
6
154
154
Airfoil
5
751
751Table 4
Parameter settings of six methods.
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
L
L
C
L
K
L
C
L
σ
C
L
K
C
λ
Servo
25
90
1
0
-
5
100
20
120
1
0
-
6
65
0.8
1
0
-
4
200
40
5
×
1
0
-
5
1.5
Auto MPG
20
40
1
0
-
4
50
15
100
1
0
-
4
100
0.3
1
0
-
2
200
80
1
0
-
3
0.8
Body fat
20
100
1
0
-
2
50
15
100
1
0
-
3
160
0.1
1
0
-
1
100
25
1
0
-
2
1.1
Concrete
120
185
2
×
1
0
-
4
200
80
200
1
0
-
7
200
0.6
5
×
1
0
-
6
500
140
1
0
-
5
0.6
Housing
40
180
2
×
1
0
-
4
200
30
200
1
0
-
5
200
0.8
1
0
-
3
500
150
1
0
-
3
0.5
Yacht
90
185
2
×
1
0
-
5
200
60
200
1
0
-
9
195
0.4
1
0
-
7
500
145
1
0
-
8
0.3
Airfoil
130
200
2
×
1
0
-
4
200
85
180
1
0
-
9
150
0.4
1
0
-
7
500
140
1
0
-
8
1.0Table 5
Training and testing RMSEs for different data sets.
Data sets
ELM
RELM
OMP-ELM
ORELM
ELM-RCC
KRSLMP-ELM
Training
Testing
Training
Testing
Training
Testing
Training
Testing
Training
Testing
Training
Testing
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
Servo
0.0745
0.1189
0.0583
0.1044
0.0727
0.1134
0.0849
0.1036
0.0749
0.1034
0.0841
0.1017
±
±
±
±
±
±
±
±
±
±
±
±
0.0133
0.0228
0.0105
0.0189
0.0123
0.0181
0.0174
0.0211
0.0117
0.0171
0.0115
0.0176
Auto MPG
0.0689
0.0785
0.0627
0.0765
0.0676
0.0777
0.0705
0.0764
0.0683
0.0757
0.0632
0.0749
±
±
±
±
±
±
±
±
±
±
±
±
0.0049
0.0053
0.0044
0.0043
0.0044
0.0045
0.0047
0.0052
0.0042
0.0045
0.0043
0.0045
Body fat
0.0237
0.0340
0.0196
0.0278
0.0218
0.0313
0.0253
0.0233
0.0240
0.0236
0.0252
0.0239
±
±
±
±
±
±
±
±
±
±
±
±
0.0073
0.0060
0.0076
0.0062
0.0076
0.0059
0.0118
0.0119
0.0093
0.0087
0.0097
0.0095
Concrete
0.0615
0.1001
0.0732
0.0925
0.0654
0.0981
0.0668
0.0929
0.0557
0.0882
0.0542
0.0864
±
±
±
±
±
±
±
±
±
±
±
±
0.0025
0.0125
0.0022
0.0041
0.0026
0.0101
0.0026
0.0069
0.0018
0.0077
0.0018
0.0067
Housing
0.0736
0.0990
0.0443
0.0897
0.0644
0.0935
0.0576
0.0899
0.0453
0.0874
0.0503
0.0849
±
±
±
±
±
±
±
±
±
±
±
±
0.0053
0.0094
0.0041
0.0136
0.0048
0.0108
0.0061
0.0160
0.0039
0.0129
0.0040
0.0114
Yacht
0.0041
0.0583
0.0300
0.0483
0.0154
0.0437
0.0189
0.0373
0.0126
0.0320
0.0060
0.0250
±
±
±
±
±
±
±
±
±
±
±
±
0.0004
0.1374
0.0002
0.0064
0.0014
0.0134
0.0013
0.0073
0.0008
0.0063
0.0005
0.0099
Airfoil
0.0664
0.0967
0.0925
0.0991
0.0731
0.0968
0.0806
0.0951
0.0740
0.0907
0.0635
0.0882
±
±
±
±
±
±
±
±
±
±
±
±
0.0020
0.0130
0.0021
0.0023
0.0024
0.0105
0.0023
0.0052
0.0021
0.0061
0.0021
0.0068


## 4.3. Sensitivity of Parameters

We analyze the sensitivity of the parametersL, K, C, and λ of KRSLMP-ELM in this subsection. For illustration, we use the regression results obtained by the Servo data set as an example. For each parameter, its sensitivity is tested by fixing the remaining parameters as the ones used in Table 4. Then, the testing RMSEs are recorded as criteria for performance comparison. The results of the regression performance are demonstrated in Figure 4.Figure 4
Regression results with different parameters: (a)L; (b) 1/C; (c) K; (d) λ.
(a)
(b)
(c)
(d)


## 5. Conclusion

In this paper, a robust matching pursuit based ELM algorithm, called the kernel risk-sensitive loss based matching pursuit extreme learning machine (KRSLMP-ELM), has been developed. Kernel risk-sensitive loss (KRSL) is a nonlinear similarity measure defined in kernel space, and it can achieve better performance than the conventional MSE criterion when dealing with non-Gaussian and nonlinear problems. Incorporating the KRSL into the existing orthogonal matching pursuit algorithm, we developed an improved KRSLMP-ELM algorithm, which is more robust than the OMP-ELM method. Comparisons with several existing state-of-the-art algorithms have also been provided to validate the superiority of the proposed KRSLMP-ELM algorithm.



---

*Source: 4563040-2018-02-01.xml*