# An Extreme Learning Machine Based on the Mixed Kernel Function of Triangular Kernel and Generalized Hermite Dirichlet Kernel

Authors: Senyue Zhang; Wenan Tan
Journal: Discrete Dynamics in Nature and Society
 (2016)

---

## Abstract

According to the characteristics that the kernel function of extreme learning machine (ELM) and its performance have a strong correlation, a novel extreme learning machine based on a generalized triangle Hermitian kernel function was proposed in this paper. First, the generalized triangle Hermitian kernel function was constructed by using the product of triangular kernel and generalized Hermite Dirichlet kernel, and the proposed kernel function was proved as a valid kernel function of extreme learning machine. Then, the learning methodology of the extreme learning machine based on the proposed kernel function was presented. The biggest advantage of the proposed kernel is its kernel parameter values only chosen in the natural numbers, which thus can greatly shorten the computational time of parameter optimization and retain more of its sample data structure information. Experiments were performed on a number of binary classification, multiclassification, and regression datasets from the UCI benchmark repository. The experiment results demonstrated that the robustness and generalization performance of the proposed method are outperformed compared to other extreme learning machines with different kernels. Furthermore, the learning speed of proposed method is faster than support vector machine (SVM) methods.


---

## Body

## 1. Introduction

The kernel extreme learning machine (KELM) is proposed by Huang et al. in 2010 by applying the kernel functions to ELM algorithm [1, 2] and where the random hidden layer feature mapping in ELM is substituted by the kernel mapping [3]. It effectively improves the undesirable generalization performance and stability caused by the stochastic nature of hidden layer output matrix and greatly reduces computational complexity. In KELM, optimization of the number of hidden layer nodes is avoided and the least square optimal solution can be obtained. Compared with SVM [4] and basic ELM [5, 6], it can provide more stable and better generalization performance. Hence, KELM has been widely applied in classification and regression problems [7, 8].It is well known that the learning ability and generalization performance of ELM mainly depend on the kernel function, and different kernel functions or same kernel function with different parameters has different influence on the generalization performance. Besides, the time required to search for the optimal kernel parameters is different using various kernel functions. Normally, the selection and optimization of kernel parameters are much tedious and time-consuming [9, 10]. Liu et al. [11] pointed out that the common Gauss kernel function and polynomial kernel function are very sensitive to the changes of kernel parameters, so the selection range of kernel parameters is large with small stepsize leading to high computation complex issue. In order to solve this issue, a series of SVM kernel functions based on generalized orthogonal polynomials have been constructed in recent years [12–16]. Ozer et al. [12] introduced a set of Chebyshev kernel functions derived from the generalized Chebyshev polynomials. The test results show that the generalized Chebyshev kernel approaches the minimum support vector number for classification in general. Furthermore, Zhang et al. constructed a series of novel SVM kernel functions, such as Laguerre kernel function from generalized Laguerre polynomial and Hermite kernel function from generalized Hermite polynomial [13–15]. Those algorithms can shorten the time of parameter optimization; however, the processing to the parameters of weight function is so simple that the influence of structure information of sample data on the generalization performance is neglected. Tian and Wang [17] verified that Gaussian Hermite kernel function can achieve the highest classification accuracy in the binary classification problem compared to the rest of above orthogonal polynomial kernel functions; however, the efficiency of its training and the robustness are relatively lower.According to above analysis, based on Hermite orthogonal polynomials, a mixed kernel function called Generalized Triangular Hermite kernel function is constructed by using the product of triangular kernel and generalized Hermite Dirichlet kernel. This kernel function has only one parameter chosen from a small range of integer numbers, thus the parameter optimization is facilitated greatly, and more structure information of sample data is retained. It is proved that generalized triangle Hermite kernel can be used as an allowed kernel function of extreme learning machine in theory. The effectiveness of the proposed method for regression and binary, multiclass classification problems is demonstrated by performing numerical experiments on a number of real-world datasets from the UCI benchmark repository and comparing their results with SVM and other extreme learning machines with different kernels.


## 2. Kernel Extreme Learning Machine

### 2.1. Introduction to Kernel Extreme Learning Machine

A training setℵ=xi,ti∣xi∈Rd,ti∈Rm,i=1,…,N is given, the output function of hidden node is chosen as G(ai,bi,x) with L hidden nodes, and ELM algorithm can be written as follows.Step 1.
Randomly generate hidden node parameters(ai,bi),i=1,…,L.Step 2.
Calculate the hidden layer output matrixH (ensure H to be full rank):(1)H=hx1⋮hxN=Ga1,b1,x1⋯GaL,bL,x1⋮⋯⋮Ga1,b1,xN⋯GaL,bL,xNN×L.Step 3.
Calculate the output weight vectorβ:(2)β=HTIλ+HHT-1T,where λ is the regularization coefficient and T=t1⋯tNm×N.The output function of ELM is(3)fx=hxHTIλ+HHT-1T.If the hidden layer feature mappingh(x) is unknown, a kernel matrix can be defined to replace HHT by using Mercer’s condition. Thus, KELM algorithm is generated as follows:(4)ΩELM=HHT:ΩELMi,j=hxi·hxj=Kxi,xj,HHT=ΩELM=Kx1,x1⋯Kx1,xN⋮⋱⋮KxN,x1⋯KxN,xN,hxHT=Kx,x1⋮Kx,xNT.Finally, the output function of KELM is defined as(5)fx=Kx,x1⋮Kx,xNTIλ+ΩELM-1T.


### 2.2. Introduction to the Property of KELM Kernel Function

Some of the well-known common KELM kernel functions are(1)
polynomial kernel:K(x,z)=(xTz+1)n;(2)
Gaussian kernel:K(x,z)=exp⁡(-x-z2/2σ2);(3)
Laplacian kernel:K(x,z)=exp⁡(-x-z/σ).In addition to the above kernel functions, a new kernel function also can be constructed according to the property of the kernel function.Property 1.
Assume thatK1(x,z) and K2(x,z) are valid kernel functions on X×X; then kernel functions K(x,z)=K1(x,z)+K2(x,z) and K(x,z)=K1(x,z)×K2(x,z) also are valid on X×X.Theorem 1.
Letf:X→R be an integrable bounded continuous function. Then the necessary and sufficient condition for the translation invariant function K(x,z)=f(x-z) to be a kernel function is f(0)>0 and Fourier transform is(6)FKω=2π-n/2∫Rnexp⁡-iωxKxdx≥0.
Like SVM, a function is an allowed KELM kernel function as long as it satisfies Mercer’s condition.Mercer Theorem [18]. Assume that, for X⊂Rn, K(x,z) is a continuous symmetric real value function on X×X such that the following integration should always be nonnegative for every f⊂L2(X):∬x×xK(x,z)f(x)f(z)dxdz≥0. Then K(x,z) must be a valid kernel function.


## 2.1. Introduction to Kernel Extreme Learning Machine

A training setℵ=xi,ti∣xi∈Rd,ti∈Rm,i=1,…,N is given, the output function of hidden node is chosen as G(ai,bi,x) with L hidden nodes, and ELM algorithm can be written as follows.Step 1.
Randomly generate hidden node parameters(ai,bi),i=1,…,L.Step 2.
Calculate the hidden layer output matrixH (ensure H to be full rank):(1)H=hx1⋮hxN=Ga1,b1,x1⋯GaL,bL,x1⋮⋯⋮Ga1,b1,xN⋯GaL,bL,xNN×L.Step 3.
Calculate the output weight vectorβ:(2)β=HTIλ+HHT-1T,where λ is the regularization coefficient and T=t1⋯tNm×N.The output function of ELM is(3)fx=hxHTIλ+HHT-1T.If the hidden layer feature mappingh(x) is unknown, a kernel matrix can be defined to replace HHT by using Mercer’s condition. Thus, KELM algorithm is generated as follows:(4)ΩELM=HHT:ΩELMi,j=hxi·hxj=Kxi,xj,HHT=ΩELM=Kx1,x1⋯Kx1,xN⋮⋱⋮KxN,x1⋯KxN,xN,hxHT=Kx,x1⋮Kx,xNT.Finally, the output function of KELM is defined as(5)fx=Kx,x1⋮Kx,xNTIλ+ΩELM-1T.


## 2.2. Introduction to the Property of KELM Kernel Function

Some of the well-known common KELM kernel functions are(1)
polynomial kernel:K(x,z)=(xTz+1)n;(2)
Gaussian kernel:K(x,z)=exp⁡(-x-z2/2σ2);(3)
Laplacian kernel:K(x,z)=exp⁡(-x-z/σ).In addition to the above kernel functions, a new kernel function also can be constructed according to the property of the kernel function.Property 1.
Assume thatK1(x,z) and K2(x,z) are valid kernel functions on X×X; then kernel functions K(x,z)=K1(x,z)+K2(x,z) and K(x,z)=K1(x,z)×K2(x,z) also are valid on X×X.Theorem 1.
Letf:X→R be an integrable bounded continuous function. Then the necessary and sufficient condition for the translation invariant function K(x,z)=f(x-z) to be a kernel function is f(0)>0 and Fourier transform is(6)FKω=2π-n/2∫Rnexp⁡-iωxKxdx≥0.
Like SVM, a function is an allowed KELM kernel function as long as it satisfies Mercer’s condition.Mercer Theorem [18]. Assume that, for X⊂Rn, K(x,z) is a continuous symmetric real value function on X×X such that the following integration should always be nonnegative for every f⊂L2(X):∬x×xK(x,z)f(x)f(z)dxdz≥0. Then K(x,z) must be a valid kernel function.


## 3. Triangular Hermite Kernel Extreme Learning Machine

### 3.1. Construction of Triangular Kernel Function

Laplace kernel functionK(x,z)=exp⁡(-x-z/σ) also is a radial basis function. Its classification performance is nearly equivalent to the Gaussian kernel KRBF(x,z)=exp⁡(-x-z2/2σ2), but it is less sensitive for the changes of parameterσ. The Laplace kernel can be used as an alternative when using the Gaussian becomes too expensive.Whilex→0, using the Taylor formula, Laplace kernel can be simplified as(7)Kmacx,z=1-x-zσ×ε1-x-zσ.Figure1 shows the function curve of (7) and KRBF(x,z), when σ=1,2, where x∈[-7,7] and z=0.Figure 1
The graph ofKRBF(x,0) and Kmac(x,0), whereσ=1,2.Seen from Figure1, two types of kernel functions are quite different, and Kmac(x,z)is relatively less sensitive to changes of the parameter σ. A typical triangle is presented on its function curve; thus, it is called triangular kernel function [19].Given a sample setX=x¯ii=1N, where x¯ is the sample mean and N is the number of training samples, in order to further simplify (7), let σ0 [16] be twice as long as the maximum distance of all sample points to sample mean such that formula holds with probability one. Thus, the simplified triangular kernel function is obtained:(8)KTrix,z=1-x-zσ0,where σ0=2×arg⁡max⁡x-x¯≤r,x∈X,x¯=1/N∑i=1N‍xi.Equation (8) is a translation invariant function; according to Theorem 1, KTri(x,z) is strictly proved to be a valid KELM kernel function in the following proof.Proof.
K ( x ) = f ( x ) = 1 - x / σ 0; it is well known that f(x) is an integrable bounded continuous function on Rn by function analysis and f(0)=1>0 such that the Fourier transform of K(x) is(9)FKω=2π-n/2∫Rn‎exp⁡-iωxKxdx=2π-n/2∫Rn‎exp⁡-iωx1-xσ0dx=2π-n/2∫-∞+∞exp⁡-iωx1-xσ0dx=2π-n/2∫-∞+∞exp⁡-iωx×1dx-1σ0∫-∞+∞exp⁡-iωx×xdx=2π-n/22πδω+2σ0ω2≥0.
The proof is completed.References [12–16] constructed the weighting function as KGau(x,z)=exp⁡(-x-z2/d), whereddenotes the dimension of vector x,z, in which since the parameter σ of Gaussian kernel KRBF is directly set with d/2, the data structure information is lost although the parameter optimization is simplified. However, the setting of parameter σ in triangular kernel function proposed in this paper just makes up for the shortcoming of it.In order to reflect the difference more intuitively betweenKGau(x,z) andKTri(x,z), Figures 2 and 3 show the graph of the above two kernel functions, where x∈[-0.5,0.5],[-1,1],[-1.5,1.5], and [-2,2], z=0.2,0.4.Figure 2
The graph ofKGau(x,z) and KTri(x,z), where z=0.2.Figure 3
The graph ofKGau(x,z) and KTri(x,z), where z=0.4.As is shown in Figures2 and 3, both the vectors x and z are one-dimensional (d=1). When z takes a fixed value, the graph of kernel function is invariably constant, but the graph of kernel function changes as the value interval of x changes. It is well known that the choice of different kernel functions is to select different criteria to measure the similarity and the degree of similarity [18].Consequently, for the same point(x,z), the similarity in different intervals should be different, and the value of KTri(x,z) increases as the expansion of the interval, while the value of KGau(x,z) in the four intervals always remains changeless. Therefore, it can be said that the parameter σ0 of kernel function KTri(x,z) is set to retain more distance similarity information of sample data; in addition, its computational cost is very low.


### 3.2. Construction of Generalized Hermite Dirichlet Kernel

Hermite polynomial [16] is a kind of orthogonal polynomials with respect to the weighting function e-x2 between the intervals (-∞,+∞), which is defined as(10)Hnx=-1nex2dndxne-x2=∑k=0n/2-1kn!k!n-2k!2xn-2k,n=0,1,2,….It satisfies the orthogonal relationship:(11)∫-∞+∞e-x2HmxHnxdx=0,m≠n,2nn!π,m=n.It has a recursive relationship:(12)H0x=1,H1x=2x,Hn+1x=2xHnx-2nHn-1x,n=1,2,….Owing to the orthogonality, variability, and universal approximation function capability of Hermite polynomial, a general Hermite kernel function can be constructed as a good alternative to other common kernel functions (Gaussian kernel, polynomial kernel, etc.). For this purpose, let the scalar variablex be instead of row vector x and xn+1 be substituted as follows correspondingly:(13)xn+1⟶xxTn+1/2,n=2k+1,xxTn/2x,n=2k,k=0,1,2,…,where xT is the transpose of xn+1.Therefore, for vector input, it can define the generalized Hermite polynomial as(14)H0x=1,H1x=2x,Hn+1x=2xHnTx-2nHn-1x,n=1,2,….By using generalized Hermite polynomial, this paper defines generalizednth order Hermite Dirichlet kernel [12] as(15)KHemx,z=∑i=1nHixHiTz.It can evaluate and verify the Mercer Theorem forKHem(x,z) as follows by assuming that each element is independent from others:(16)
∬
X
×
X
K
X
,
Z
f
x
f
z
d
x
d
z
=
∬
X
×
X
∑
i
=
1
n
H
i
x
H
i
T
z
f
x
f
z
d
x
d
z
=
∑
i
=
1
n
∬
X
×
X
H
i
x
H
i
T
z
f
x
f
z
d
x
d
z
=
∑
i
=
1
n
∫
X
H
i
x
f
x
d
x
∫
X
H
i
T
z
f
z
d
z
=
∑
i
=
1
n
∫
X
H
i
x
f
x
d
x
2
≥
0
.Therefore,KHem(x,z) is a valid KELM kernel, and its kernel parameters can only be natural numbers, which greatly simplifies the selection and optimization of kernel parameters.


### 3.3. Generalized Triangular Hermite Kernel Extreme Learning Machine

According to Property1, a new KELM kernel called Generalized Triangular Hermite kernel is constructed, which is the multiplication of triangular kernel and generalized Hermite Dirichlet kernel, which is defined as(17)KTri-Hx,z=1-x-zσ0∑i=1nHixHiTz.Generalized Triangular Hermite kernel combines the advantages of triangular kernel and generalized Hermite Dirichlet kernel, which not only retains more distance similarity information of sample data, but also just chooses natural numbers to its parameter. Accordingly, it can greatly shorten the time of parameter optimization. Although it has two kernel parameters, that can be determined quickly and easily, which greatly reduces parameter optimization cost. The Generalized Triangular Hermite kernel function up to the 3rd order is listed as in Table1.Table 1
List of the Generalized Triangular Hermite kernel function up to the 3rd order.
n K T r i ( x , z ) 0 1 ⋅ 1 - x - z / σ 0 1 1 + 4 x z T ⋅ 1 - x - z / σ 0 2 1 + 4 ⋅ 2 x 2 - 1 2 z 2 - 1 + 4 x z T ⋅ 1 - x - z / σ 0 3 1 + 4 2 x 2 - 1 2 z 2 - 1 + 4 x z T + 16 2 x 2 - 3 2 z 2 - 3 ⋅ x z T ⋅ 1 - x - z / σ 0Figures4 and 5 show the Generalized Triangular Hermite kernel output up to the 3rd order at two different coordinate scales of vertical axis, where changes are within the range of [-2,2] and are fixed at a constant value. Figure 4 shows the kernel function, while Figure 5 shows the value, where the 0th and 1st orders correspond to the left vertical axis, and the 2nd and 3rd orders correspond to the right vertical axis.Figure 4
The Triangular Hermite kernel outputKTri-H(x,z) up to the 3rd order: x∈[-0.5,0.5] and z=0.2.Figure 5
The Triangular Hermite kernel outputKTri-H(x,z) up to the 3rd order: x∈[-0.5,0.5] and z=0.4.Finally, the kernelKTri-H(x,z) to KELM algorithm is introduced; as a result, the Generalized Triangular Hermite kernel extreme learning machine algorithm is obtained as follows.Given a training setℵ=xi,ti∣xi∈Rd,ti∈Rm,i=1,…,N, the output function is(18)fx=Kx,x1⋮Kx,xNTIλ+ΩELM-1T,where Kxi,xj=(1-xi-xj/σ0)∑k=1nHk(xi)HkT(xj).


## 3.1. Construction of Triangular Kernel Function

Laplace kernel functionK(x,z)=exp⁡(-x-z/σ) also is a radial basis function. Its classification performance is nearly equivalent to the Gaussian kernel KRBF(x,z)=exp⁡(-x-z2/2σ2), but it is less sensitive for the changes of parameterσ. The Laplace kernel can be used as an alternative when using the Gaussian becomes too expensive.Whilex→0, using the Taylor formula, Laplace kernel can be simplified as(7)Kmacx,z=1-x-zσ×ε1-x-zσ.Figure1 shows the function curve of (7) and KRBF(x,z), when σ=1,2, where x∈[-7,7] and z=0.Figure 1
The graph ofKRBF(x,0) and Kmac(x,0), whereσ=1,2.Seen from Figure1, two types of kernel functions are quite different, and Kmac(x,z)is relatively less sensitive to changes of the parameter σ. A typical triangle is presented on its function curve; thus, it is called triangular kernel function [19].Given a sample setX=x¯ii=1N, where x¯ is the sample mean and N is the number of training samples, in order to further simplify (7), let σ0 [16] be twice as long as the maximum distance of all sample points to sample mean such that formula holds with probability one. Thus, the simplified triangular kernel function is obtained:(8)KTrix,z=1-x-zσ0,where σ0=2×arg⁡max⁡x-x¯≤r,x∈X,x¯=1/N∑i=1N‍xi.Equation (8) is a translation invariant function; according to Theorem 1, KTri(x,z) is strictly proved to be a valid KELM kernel function in the following proof.Proof.
K ( x ) = f ( x ) = 1 - x / σ 0; it is well known that f(x) is an integrable bounded continuous function on Rn by function analysis and f(0)=1>0 such that the Fourier transform of K(x) is(9)FKω=2π-n/2∫Rn‎exp⁡-iωxKxdx=2π-n/2∫Rn‎exp⁡-iωx1-xσ0dx=2π-n/2∫-∞+∞exp⁡-iωx1-xσ0dx=2π-n/2∫-∞+∞exp⁡-iωx×1dx-1σ0∫-∞+∞exp⁡-iωx×xdx=2π-n/22πδω+2σ0ω2≥0.
The proof is completed.References [12–16] constructed the weighting function as KGau(x,z)=exp⁡(-x-z2/d), whereddenotes the dimension of vector x,z, in which since the parameter σ of Gaussian kernel KRBF is directly set with d/2, the data structure information is lost although the parameter optimization is simplified. However, the setting of parameter σ in triangular kernel function proposed in this paper just makes up for the shortcoming of it.In order to reflect the difference more intuitively betweenKGau(x,z) andKTri(x,z), Figures 2 and 3 show the graph of the above two kernel functions, where x∈[-0.5,0.5],[-1,1],[-1.5,1.5], and [-2,2], z=0.2,0.4.Figure 2
The graph ofKGau(x,z) and KTri(x,z), where z=0.2.Figure 3
The graph ofKGau(x,z) and KTri(x,z), where z=0.4.As is shown in Figures2 and 3, both the vectors x and z are one-dimensional (d=1). When z takes a fixed value, the graph of kernel function is invariably constant, but the graph of kernel function changes as the value interval of x changes. It is well known that the choice of different kernel functions is to select different criteria to measure the similarity and the degree of similarity [18].Consequently, for the same point(x,z), the similarity in different intervals should be different, and the value of KTri(x,z) increases as the expansion of the interval, while the value of KGau(x,z) in the four intervals always remains changeless. Therefore, it can be said that the parameter σ0 of kernel function KTri(x,z) is set to retain more distance similarity information of sample data; in addition, its computational cost is very low.


## 3.2. Construction of Generalized Hermite Dirichlet Kernel

Hermite polynomial [16] is a kind of orthogonal polynomials with respect to the weighting function e-x2 between the intervals (-∞,+∞), which is defined as(10)Hnx=-1nex2dndxne-x2=∑k=0n/2-1kn!k!n-2k!2xn-2k,n=0,1,2,….It satisfies the orthogonal relationship:(11)∫-∞+∞e-x2HmxHnxdx=0,m≠n,2nn!π,m=n.It has a recursive relationship:(12)H0x=1,H1x=2x,Hn+1x=2xHnx-2nHn-1x,n=1,2,….Owing to the orthogonality, variability, and universal approximation function capability of Hermite polynomial, a general Hermite kernel function can be constructed as a good alternative to other common kernel functions (Gaussian kernel, polynomial kernel, etc.). For this purpose, let the scalar variablex be instead of row vector x and xn+1 be substituted as follows correspondingly:(13)xn+1⟶xxTn+1/2,n=2k+1,xxTn/2x,n=2k,k=0,1,2,…,where xT is the transpose of xn+1.Therefore, for vector input, it can define the generalized Hermite polynomial as(14)H0x=1,H1x=2x,Hn+1x=2xHnTx-2nHn-1x,n=1,2,….By using generalized Hermite polynomial, this paper defines generalizednth order Hermite Dirichlet kernel [12] as(15)KHemx,z=∑i=1nHixHiTz.It can evaluate and verify the Mercer Theorem forKHem(x,z) as follows by assuming that each element is independent from others:(16)
∬
X
×
X
K
X
,
Z
f
x
f
z
d
x
d
z
=
∬
X
×
X
∑
i
=
1
n
H
i
x
H
i
T
z
f
x
f
z
d
x
d
z
=
∑
i
=
1
n
∬
X
×
X
H
i
x
H
i
T
z
f
x
f
z
d
x
d
z
=
∑
i
=
1
n
∫
X
H
i
x
f
x
d
x
∫
X
H
i
T
z
f
z
d
z
=
∑
i
=
1
n
∫
X
H
i
x
f
x
d
x
2
≥
0
.Therefore,KHem(x,z) is a valid KELM kernel, and its kernel parameters can only be natural numbers, which greatly simplifies the selection and optimization of kernel parameters.


## 3.3. Generalized Triangular Hermite Kernel Extreme Learning Machine

According to Property1, a new KELM kernel called Generalized Triangular Hermite kernel is constructed, which is the multiplication of triangular kernel and generalized Hermite Dirichlet kernel, which is defined as(17)KTri-Hx,z=1-x-zσ0∑i=1nHixHiTz.Generalized Triangular Hermite kernel combines the advantages of triangular kernel and generalized Hermite Dirichlet kernel, which not only retains more distance similarity information of sample data, but also just chooses natural numbers to its parameter. Accordingly, it can greatly shorten the time of parameter optimization. Although it has two kernel parameters, that can be determined quickly and easily, which greatly reduces parameter optimization cost. The Generalized Triangular Hermite kernel function up to the 3rd order is listed as in Table1.Table 1
List of the Generalized Triangular Hermite kernel function up to the 3rd order.
n K T r i ( x , z ) 0 1 ⋅ 1 - x - z / σ 0 1 1 + 4 x z T ⋅ 1 - x - z / σ 0 2 1 + 4 ⋅ 2 x 2 - 1 2 z 2 - 1 + 4 x z T ⋅ 1 - x - z / σ 0 3 1 + 4 2 x 2 - 1 2 z 2 - 1 + 4 x z T + 16 2 x 2 - 3 2 z 2 - 3 ⋅ x z T ⋅ 1 - x - z / σ 0Figures4 and 5 show the Generalized Triangular Hermite kernel output up to the 3rd order at two different coordinate scales of vertical axis, where changes are within the range of [-2,2] and are fixed at a constant value. Figure 4 shows the kernel function, while Figure 5 shows the value, where the 0th and 1st orders correspond to the left vertical axis, and the 2nd and 3rd orders correspond to the right vertical axis.Figure 4
The Triangular Hermite kernel outputKTri-H(x,z) up to the 3rd order: x∈[-0.5,0.5] and z=0.2.Figure 5
The Triangular Hermite kernel outputKTri-H(x,z) up to the 3rd order: x∈[-0.5,0.5] and z=0.4.Finally, the kernelKTri-H(x,z) to KELM algorithm is introduced; as a result, the Generalized Triangular Hermite kernel extreme learning machine algorithm is obtained as follows.Given a training setℵ=xi,ti∣xi∈Rd,ti∈Rm,i=1,…,N, the output function is(18)fx=Kx,x1⋮Kx,xNTIλ+ΩELM-1T,where Kxi,xj=(1-xi-xj/σ0)∑k=1nHk(xi)HkT(xj).


## 4. Experiments and Analysis

In order to test the performance of Generalized Triangular Hermite kernel extreme learning machine (Tri-H KELM) algorithms, this section compares it concerning testing accuracy, training time, and regression determination coefficient with other various algorithms in bispiral dataset and real-world benchmark regression, binary, multiclass classification datasets. Table2 lists the various algorithms used in the experiments and the range of corresponding kernel parameter value, which also includes Gaussian kernel (Gauss), polynomial kernel (Poly), Gaussian Hermite kernel (Gau-H) [15] extreme learning machine algorithms, and triangular Hermite kernel support vector machine algorithm (Tri-H SVM). In multiclass simulations, we use the LIBSVM toolbox available and train SVM for each class separately as one versus all. The SVM cost parameter value is 100 in each corresponding experiment. Better test results are given in boldface in Tables 2–9.Table 2
The various algorithms and the range of their kernel parameters.
Algorithm Kernel parameter Range Stepsize Gauss KELM σ 1–10 0.2 Poly KELM n 0–3 1 Gau-H KELM n 0–3 1 Tri-H KELM n 0–3 1 Tri-H SVM n 0–3 1Table 3
The maximum testing rate (%) and the corresponding kernel parameter value.
KELM algorithm Maximum rate and optimization time Gauss KELM 92.5 (T=7.2384 s) Poly KELM 91.75 (T=2.6988 s) Gau-H KELM 92.25 (T=0.9360 s) Tri-H KELM 93 (T=0.8424 s)Table 4
Specifications of binary classification cases.
Datasets Training data Testing data Attributes Liver 230 115 6 Australian 460 230 6 Breast-cancer 200 77 9 Wpbc 128 70 33 Sonar 150 58 60Table 5
Specifications of multiclass classification cases.
Datasets Training data Testing data Attributes Classes Iris 100 50 4 3 Glass 142 72 9 6 Wine 118 60 13 8 Vehicle 564 282 18 4 Segment 1540 770 18 7 Waveform 800 400 21 3Table 6
Specifications of multiclass classification cases.
Datasets Algorithms Gauss KELM Poly KELM Gau-H KELM Tri-H, KELM Tri-H SVM Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Liver (C=1) 60(σ=7.4) 0.0037 72.17 (n=3) 0.0062 59.13 (n=0) 0.0036 73.91 (n=0) 0.0027 73.65(n=0) 0.5714 Breast cancer (C=0.1) 81.82 (σ=4.2) 0.0016 76.62 (n=1) 0.0058 79.22 (n=0) 0.0053 81.82 (n=0) 0.0061 81.03 (n=0) 0.6413 Wpbc (C=0.1) 77.14 (σ=0.4) 0.0126 77.14 (n=2) 0.003 80 (n=1) 0.0026 78.57 (n=1) 0.0026 78.03 (n=1) 0.4638 Sonar (C=1) 84.48 (σ=0.6) 0.001 84.48 (n=3) 0.0024 81.03 (n=2) 0.002 84.48 (n=1) 0.0084 83.12 (n=1) 0.8127Table 7
Performance comparison of deferent algorithms in multiclass datasets.
Datasets Algorithms Gauss KELM Poly KELM Gau-H KELM Tri-H, KELM Tri-H SVM Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Iris (C=0.1) 98 (σ=2) 0.0005 97 (n=4) 0.0012 98 (n=0) 0.0038 98 (n=0) 0.0007 96 (n=0) 0.073 Glass (C=1) 77.78 (σ=0.4) 0.0026 72.22 (n=3) 0.004 73.61 (n=0) 0.0036 76.37 (n=0) 0.0031 74.58 (n=0) 4.54 Wine (C=100) 80 (σ=4) 0.0012 96.67 (n=1) 0.0046 97 (n=0) 0.0048 97.48 (n=0) 0.0025 96.44 (n=0) 1.41 Vehicle (C=10) 80.5 (σ=1.4) 0.0163 81.21 (n=2) 0.0076 82.62 (n=2) 0.0206 82.62 (n=1) 0.064 81.58 (n=1) 1.82 Segment (C=10) 87.4 (σ=7) 0.2002 88.44 (n=2) 0.0834 87.27 (n=0) 0.4636 88.68 (n=0) 0.5265 87.51 (n=0) 14.65 Waveform (C=1) 86.75 (σ=10) 0.0361 84.25 (n=1) 0.0254 86.25 (n=0) 0.0438 87.04 (n=0) 0.1266 85.33 (n=0) 2.87Table 8
Specifications of binary classification cases.
Datasets Training data Testing data Attributes Cloud 682 342 9 Autoprice 106 53 15 Housing 337 169 13 Body fat 168 84 14 NIR 50 10 401Table 9
Performance comparison of deferent algorithms in regression datasets.
Datasets Algorithms Gauss KELM Poly KELM Gau-H KELM Tri-H, KELM Tri-H SVM Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Testing rate (%) Training time (s) Cloud (C=1) 0.4163 (σ=2) 0.0221 0.9646 (n=1) 0.0184 0.9658 (n=2) 0.0379 0.9709 (n=2) 0.0364 0.9642 (n=2) 0.0502 Autoprice (C=1) 0.8251 (σ=10) 0.0016 0.8661 (n=2) 0.0018 0.9183 (n=1) 0.0012 0.9194 (n=1) 0.0011 0.9057 (n=1) 0.0616 Housing (C=1) 0.8661 (σ=3) 0.0051 0.8994 (n=3) 0.0134 0.9141 (n=1) 0.007 0.9102 (n=1) 0.0071 0.8969 (n=1) 0.1914 Body fat (C=0.1) 0.8936 (σ=5.6) 0.0011 0.9791 (n=3) 0.0034 0.9643 (n=2) 0.0014 0.9865 (n=1) 0.0013 0.9725 (n=1) 0.0455 NIR (C=1000) 0.2784 (σ=10) 0.0034 0.9371 (n=1) 0.0049 0.9452 (n=0) 0.0077 0.9614 (n=0) 0.0022 0.9678 (n=0) 0.0638


### 4.1. Classification Performance Comparison on Banana Dataset

The Banana dataset is a well-known binary class dataset from the UCI benchmark repository used in many pattern recognition tests. Each vector of the dataset has 2 features. The training set consists of 800 randomly selected samples (400 for (+1) class and 400 for (−1) class); meanwhile, 400 samples are randomly chosen as a testing set, which contains 200 samples of each class. At each trial of simulation, regularization coefficient is assumed. The simulation results, the maximum testing accuracy, and the corresponding kernel parameter optimization time are given in Table3.Figures6–9 show the boundaries of different KELM algorithms on benchmark Banana dataset.Figure 6
The boundaries of Gauss KELM algorithms in Banana dataset.Figure 7
The boundaries of Poly KELM algorithms in Banana dataset.Figure 8
The boundaries of Gau-H KELM algorithms in Banana dataset.Figure 9
The boundaries of Tri-H KELM algorithms in Banana dataset.Seen from Table3 and Figures 6–9, Tri-H KELM achieves the highest testing accuracy in Banana dataset compared with Gauss, Poly, and Gau-H KELM. Furthermore, for the process of parameter optimization, the kernel parameter of two types of Hermite KELM algorithms is only taken from 0 to 3, which greatly shortens the time.


### 4.2. Classification Performance Comparison on UCI Benchmark Datasets

In order to extensively verify the performance of different algorithms, wide types of datasets have been tested in simulations, which include 5 binary classification cases and 6 multiclass classification cases. Most of the datasets are taken from UCI Machine Learning Repository. The specifications of binary classification and multiclass classification cases are listed in Tables4 and 5. Comparing Tri-H KELM with Gauss, Poly, and Gau-H KELM and Tri-H SVM, experiment results, including the maximum testing accuracy and corresponding kernel parameter value, and the training time are given in Tables 6 and 7.(1  ) Classification Performance Comparison on Binary Classification Cases. As demonstrated by the simulation results, Tri-H KELM achieves much better generalization performance in binary classification cases than other KELM algorithms (Gauss, Poly, and Gau-H KELM). More importantly, Tri-H KELM has achieved the highest classification accuracy just within a smaller range of integer numbers of kernel parameter, which greatly reduces the parameter optimization time. Besides, it is observed that Tri-H KELM can always achieve comparable performance with Tri-H SVM at much faster learning speed.(2  ) Classification Performance Comparison on Multiclass Classification Cases. The experiment results infer that Tri-H KELM performs better than other KELM algorithms and Tri-H SVM on average for multiclass classification cases. Its biggest advantage is that the parameter selection time is short, because the best classification performance is achieved with given kernel parameters. A further study reveals that all KELM algorithms have better scalability and run at much faster learning speed than traditional SVM.


### 4.3. Regression Performance Comparison on UCI Benchmark Datasets

This subsection selects 5 regression cases from UCI Machine Learning Repository; data is described in  Table8. The simulations of different algorithms on all the regression datasets were carried out. The regression performance of ELM was evaluated by the coefficient of determination R2 which is defined within the interval of [0,1] and is closer to 1. The simulations results (R2 and the training time) are given in Table 9.As Table9 shows, in comparison to other several KELM algorithms, Tri-H KELM obtains the maximum coefficient of determination in most regression datasets, and it has all achieved the maximum with respect to kernel parameter. It has better generalization performance for regression problem. Note that Tri-H KELM achieves similar regression performance as SVM at much faster learning speeds.


## 4.1. Classification Performance Comparison on Banana Dataset

The Banana dataset is a well-known binary class dataset from the UCI benchmark repository used in many pattern recognition tests. Each vector of the dataset has 2 features. The training set consists of 800 randomly selected samples (400 for (+1) class and 400 for (−1) class); meanwhile, 400 samples are randomly chosen as a testing set, which contains 200 samples of each class. At each trial of simulation, regularization coefficient is assumed. The simulation results, the maximum testing accuracy, and the corresponding kernel parameter optimization time are given in Table3.Figures6–9 show the boundaries of different KELM algorithms on benchmark Banana dataset.Figure 6
The boundaries of Gauss KELM algorithms in Banana dataset.Figure 7
The boundaries of Poly KELM algorithms in Banana dataset.Figure 8
The boundaries of Gau-H KELM algorithms in Banana dataset.Figure 9
The boundaries of Tri-H KELM algorithms in Banana dataset.Seen from Table3 and Figures 6–9, Tri-H KELM achieves the highest testing accuracy in Banana dataset compared with Gauss, Poly, and Gau-H KELM. Furthermore, for the process of parameter optimization, the kernel parameter of two types of Hermite KELM algorithms is only taken from 0 to 3, which greatly shortens the time.


## 4.2. Classification Performance Comparison on UCI Benchmark Datasets

In order to extensively verify the performance of different algorithms, wide types of datasets have been tested in simulations, which include 5 binary classification cases and 6 multiclass classification cases. Most of the datasets are taken from UCI Machine Learning Repository. The specifications of binary classification and multiclass classification cases are listed in Tables4 and 5. Comparing Tri-H KELM with Gauss, Poly, and Gau-H KELM and Tri-H SVM, experiment results, including the maximum testing accuracy and corresponding kernel parameter value, and the training time are given in Tables 6 and 7.(1  ) Classification Performance Comparison on Binary Classification Cases. As demonstrated by the simulation results, Tri-H KELM achieves much better generalization performance in binary classification cases than other KELM algorithms (Gauss, Poly, and Gau-H KELM). More importantly, Tri-H KELM has achieved the highest classification accuracy just within a smaller range of integer numbers of kernel parameter, which greatly reduces the parameter optimization time. Besides, it is observed that Tri-H KELM can always achieve comparable performance with Tri-H SVM at much faster learning speed.(2  ) Classification Performance Comparison on Multiclass Classification Cases. The experiment results infer that Tri-H KELM performs better than other KELM algorithms and Tri-H SVM on average for multiclass classification cases. Its biggest advantage is that the parameter selection time is short, because the best classification performance is achieved with given kernel parameters. A further study reveals that all KELM algorithms have better scalability and run at much faster learning speed than traditional SVM.


## 4.3. Regression Performance Comparison on UCI Benchmark Datasets

This subsection selects 5 regression cases from UCI Machine Learning Repository; data is described in  Table8. The simulations of different algorithms on all the regression datasets were carried out. The regression performance of ELM was evaluated by the coefficient of determination R2 which is defined within the interval of [0,1] and is closer to 1. The simulations results (R2 and the training time) are given in Table 9.As Table9 shows, in comparison to other several KELM algorithms, Tri-H KELM obtains the maximum coefficient of determination in most regression datasets, and it has all achieved the maximum with respect to kernel parameter. It has better generalization performance for regression problem. Note that Tri-H KELM achieves similar regression performance as SVM at much faster learning speeds.


## 5. Conclusion

In this work, a novel extreme learning machine based on the mixed kernel function of triangular kernel and generalized Hermite Dirichlet kernel (Tri-H KELM) has been put forward, which introduces the triangular Hermite kernel function to kernel extreme learning machine algorithm. Because the presented kernel has only one parameter chosen from a small set of integers, the parameter optimization is facilitated greatly. Besides, more structure information of sample data is retained in the proposed kernel. Numerical experiments have been performed with different algorithms (Tri-H SVM, Gauss, Poly, Gau-H, and Tri-H KELM) on bispiral benchmark dataset and a number of real-world benchmark datasets and their results have been compared with Tri-H SVM and Gauss, Poly, and Gau-KELM for regression and binary, multiclass classification. Comparable generalization and robustness performance of the proposed approach with the rest of the methods considered at a much faster learning speed than Tri-H SVM indicate its usefulness and effectiveness. Future work will be on the study of Tri-H ELM in its practical applications.



---

*Source: 7293278-2016-10-04.xml*