# A Multiple Hidden Layers Extreme Learning Machine Method and Its Application

Authors: Dong Xiao; Beijing Li; Yachun Mao
Journal: Mathematical Problems in Engineering
 (2017)

---

## Abstract

Extreme learning machine (ELM) is a rapid learning algorithm of the single-hidden-layer feedforward neural network, which randomly initializes the weights between the input layer and the hidden layer and the bias of hidden layer neurons and finally uses the least-squares method to calculate the weights between the hidden layer and the output layer. This paper proposes a multiple hidden layers ELM (MELM for short) which inherits the characteristics of parameters of the first hidden layer. The parameters of the remaining hidden layers are obtained by introducing a method (make the actual output zero error approach the expected hidden layer output). Based on the MELM algorithm, many experiments on regression and classification show that the MELM can achieve the satisfactory results based on average precision and good generalization performance compared to the two-hidden-layer ELM (TELM), the ELM, and some other multilayer ELM.


---

## Body

## 1. Introduction

At present, artificial neural network has been widely applied in many research fields, such as pattern recognition, signal processing, and short-term prediction. Among them, the single-hidden-layer feedforward neural network (SLFN) is the most widely used type of the artificial neural network [1, 2]. Because the parameters of traditional feedforward neural network are usually determined by gradient-based error backpropagation algorithms, the network bears the time-expensive training and testing process and easily falls into the local optimum. Now, many algorithms have been proposed to improve the SLFN operation rate and precision such as the backpropagation algorithm (BP) and its improved algorithms [3, 4]. With limitations of BP algorithms, generalization ability of networks is unsatisfactory and the over learning easily occurs. In 1989, Lowe proposed the RBF neural network [5] which indicated that the parameters of the SLFNs can also be randomly selected in his articles. In 1992, Pao Y. H. et al. proposed the theory of the random vector functional link network (RVFL) [6, 7], and they presented that only one parameter of the output weights should be calculated during the training process.In 2004, Huang G. B. proposed the extreme learning machine (ELM) reducing the training time of network and improving the generalization performance [8–10]. Traditional neural network learning algorithms (such as BP) need to randomly set all the training parameters and use iterative algorithm to update the parameters. Also it is easy to generate local optimal solution. But ELM only needs to randomly set the weights and bias of the hidden neurons, and the output weights are determined by using the Moore-Penrose pseudoinverse under the criterion of least-squares method. In recent years, various ELM variants have been proposed aiming to achieve better achievements, such as the deep ELM with kernel based on Multilayer Extreme Learning Machine (DELM) algorithm [11]; two-hidden-layer extreme learning machine (TELM) [12]; a Four-Layered Feedforward Neural Network [13]; online sequential extreme learning machine [14, 15]; multiple kernel extreme learning machine (MK-ELM) [16]; two-stage extreme learning machine [17], using noise detection and improving the classifier accuracy [18, 19].First, consider the DELM with kernel based on ELM-AE algorithm (DELM) presented in [11], which quotes the ELM autoencoder (ELM-AE) [20–22] as the learning algorithm in each layer. The DELM also has multilayer network structure divided into two parts: the first part uses the ELM-AE to deep learn the original data aiming at obtaining the most representative new data; the second part calculates the network parameters by using the Kernel ELM algorithm with a three-layer structure (the output of the first part, hidden layer, and output layer). But for the MELM we do not need the data processing (such as extract the representative data from the original data) but make the actual output of the hidden layers more closer to the expected output of the hidden layers by calculating step by step in the multiple hidden layers.Next, a two-hidden-layer feedforward network (TLFN) was proposed by Huang in 2003 [23]. This article demonstrates that the TLFNs could learn arbitrary N training samples with a very small training error by employing 2m+3N hidden neurons [24]. But the changing process of the TLFNs structure is very complicated. First the TLFN has a three-layer network with L output neurons, then adds 2L neurons (two parts) to the hidden layer aiming at making the original output layer transform into the second hidden layer with L hidden neurons, and finally adds a output layer to the structure. Eventually the final network structure has one input layer, two hidden layers, and one output layer. But the MELM has a relatively simple stable network structure and the simple calculation process, and the MELM is a time-saving algorithm compared with TLFNs.In the paper, we propose a multiple hidden layers extreme learning machine algorithm (MELM) that the MELM adds some hidden layers to the original ELM network structure, randomly initializes the weights between the input layer and the first hidden layer as well as the bias of the first hidden layer, utilizes the method (make the actual each hidden layer output approach the expected each hidden layer output) to calculate the parameters of the hidden layers (except the first hidden layer), and finally uses the least square method to calculate the output weights of the network. In the following chapters, we have carried out many experiments with the ideas proposed. The MELM experimental results on regression problems and some popular classification problems have shown satisfactory advantages in terms of average accuracy compared to other ELM variants. Our experiments also study the effect of different numbers of the hidden layer neurons, the compatible activation function, and the different numbers of the hidden layers on the same problems.The rest of this paper is organized as follows. Section2 reviews the original ELM; Section 3 presents the method and framework structure of two-hidden-layer ELM; Section 4 presents the proposed the MELM technique: multihidden-layer ELM; Section 5 reports and analyzes experimental results; Section 6 presents the conclusions.


## 2. Extreme Learning Machine

The extreme learning machine (ELM) proposed by Huang G.B. aims at avoiding time-costing iterative training process and improving the generalization performance [8–10, 25]. As a single-hidden-layer feedforward neural networks (SLFNs), the ELM structure includes input layer, hidden layer, and output layer. Different from the traditional neural network learning algorithms (such as BP algorithm) randomly setting all the network training parameters and easily generating local optimal solution, the ELM only sets the number of hidden neurons of the network, randomizes the weights between the input layer and the hidden layer as well as the bias of the hidden neurons in the algorithm execution process, calculates the hidden layer output matrix, and finally obtains the weight between the hidden layer and the output layer by using the Moore-Penrose pseudoinverse under the criterion of least-squares method. Because the ELM has the simple network structure and the concise parameters computation processes, so the ELM has the advantages of fast learning speed. The original structure of ELM is expressed in Figure 1.Figure 1
The structure of the ELM.Figure1 is the extreme learning machine network structure which includes n input layer neurons, l hidden layer neurons, and m output layer neurons. First, consider the training sample {X,Y}={xi,yi}(i=1,2,…,Q), and there is an input feature X=xi1xi2⋯xiQ and a desired matrix Y=yj1yj2⋯yjQ comprised of the training samples, where the matrix X and the matrix Y can be expressed as follows:(1)X=x11x12⋯x1Qx21x22⋯x2Q⋮⋮⋱⋮xn1xn2⋯xnQ,Y=y11y12⋯ymQy21y22⋯ymQ⋮⋮⋱⋮ym1ym2⋯ymQ,where the parameters n and m are the dimension of input matrix and output matrix.Then the ELM randomly sets the weights between the input layer and the hidden layer:(2)w=w11w12⋯w1nw21w22⋯w2n⋮⋮⋱⋮wl1wl2⋯wln,where wij represents the weights between the jth input layer neuron and ith hidden layer neuron.Third, the ELM assumes the weights between the hidden layer and the output layer that can be expressed as follows:(3)β=β11β12⋯β1mβ21β22⋯β2m⋮⋮⋱⋮βl1βl2⋯βlm,where βjk   represents the weights between the jth hidden layer neuron and kth output layer neuron.Fourth, the ELM randomly sets the bias of the hidden layer neurons:(4)B=b1b2⋯bnT. Fifth, the ELM chooses the network activation function g(x). According to Figure 1, the output matrix  T can be expressed as follows:(5)T=t1,t2,…,tQm×Q.Each column vector of the output matrix T is as follows:(6)tj=t1jt2j⋮tmj=∑i=1lβi1gwixj+bi∑i=1lβi2gwixj+bi⋮∑i=1lβimgwixj+bij=1,2,3,…,Q.Sixth, consider formulae (5) and (6), and we can get (7)Hβ=T′,where T′ is the transpose of T and H is the output of the hidden layer. In order to obtain the unique solution with minimum-error, we use least square method to calculate the weight matrix values of β [8, 9].(8)β=H+T′.To improve the generalization ability of network and make the results more stable, we add a regularization term to theβ [26]. When the number of hidden layer neurons is less than the number of training samples, β can be expressed as(9)β=Iλ+HTH-1HTT′.When the number of hidden layer nodes is more than the number of training samples,β can be expressed as(10)β=HTIλ+HHT-1T′.


## 3. Two-Hidden-Layer ELM

In 2016, B. Y. Qu and B. F. Lang proposed the two-hidden-layer extreme learning machine (TELM) in [11]. The TELM tries to make the actual hidden layers output approach expected hidden layer outputs by adding parameter setting step for the second hidden layer. Finally, the TELM finds a better way to map the relationship between input and output signals, which is the two-hidden-layer ELM. The network structure of TELM includes one input layer, two hidden layers, one output layer, and each hidden layer with l hidden neurons. The activation function of the network is selected for g(x).The focus of the TELM algorithm is the process of calculating and updating the weights between the first hidden layer and the second hidden layer as well as the bias of the hidden layer and the output weights between the second hidden layer and the output layer. The workflow of the TELM architecture is depicted in Figure2.Figure 2
The workflow of the TELM.Consider the training sample datasets{X,T}=xi,tii=1,2,3,…,Q, where the matrix X is input samples and T is the labeled samples.The TELM first puts the two hidden layers as one hidden layer, so the output of the hidden layer can be expressed asH=g(WX+B) with the parameters of the weight W and bias B of the first hidden layer randomly initialized. Next, the output weight matrix  β between the second hidden layer and the output layer can be obtained by using(11)β=H+T.Now the TELM separates the two hidden layers merged previously, so the network has two hidden layers. According to the workflow of Figure2, the output of the second hidden layer can be obtained as follows:(12)gW1H+B1=H1,where W1 is the weight matrixes between the first hidden layer and the second hidden layer, H is the output matrix of the first hidden layer, B1 is the bias of the second hidden layer, and H1 is the expected output of the second hidden layer.However the expected output of the second hidden layer can be obtained by calculating(13)H1=Tβ+,where β+ is the generalized inverse of the matrix β.Now the TELM defines the matrixWHE=B1W1, so the parameters of the second hidden layer can be easily obtained by using formula (12) and the inverse function of the activation function.(14)WHE=g-1H1HE+,where HE+ is the generalized inverse of HE=1HT,1 denotes a one-column vector of size Q, and its elements are the scalar unit 1. The notation g-1(x) indicates the inverse of the activation function g(x).With selecting the appropriate activation functiong(x), the TELM calculates (14), so the actual output of the second hidden layer is updated as follows:(15)H2=gWHEHE.So the weights matrixβ between the second hidden layer and the output layer is updated as follows: (16)βnew=H2+T,where H2+ is the generalized inverse of H2, so the actual output of the TELM network can be expressed as (17)fx=H2βnew.To sum up, the TELM algorithm process can be expressed as follows.Algorithm 1.
(
1
) Assume the training sample dataset is X,T=xi,tii=1,2,3,…,Q, where the matrix X is input samples and the matrix T is the labeled samples; each hidden layer with l hidden neurons; the activation function g(x).
(
2
) Randomly generate the weights W between the input layer and the first hidden layer and bias B of the first hidden neurons WIE=BW,XE=1XT.
(
3
) Calculate the equation H=g(WIEXE).
(
4
) Obtain the weights between the second hidden layer and output layer β=H+T.
(
5
) Calculate the expected output of the second hidden layer H1=Tβ+.
(
6
) According to formulae (12)–(14) and the algorithm steps (4, 5), calculate the weights W1 between the first hidden layer and the second hidden layer and the bias B1 of the second hidden neurons WHE=g-1(H1)HE+.
(
7
) Obtain and update the actual output of the second hidden layer H2=g(WHEHE).
(
8
) Update the weights matrix β between the second hidden layer and the output layer βnew=H2+T.
(
9
) Calculate the output of the network fx=g{WHgWX+B+B1}βnew.


## 4. Multihidden-Layer ELM

At present, some scholars have proposed many improvements on ELM algorithm and structure and obtained some achievements and advantages. Such advantages of neural network motivate us to explore the better ideas behind the ELM. Based on the above articles, we adjust the structure of ELM neural network. Thus we propose an algorithm named multiple hidden layers extreme learning machine (MELM). The structure of the MELM (select the three-hidden-layer ELM for example) is illustrated in Figure3. The workflow of the three-hidden-layer ELM is illustrated in Figure 4.Figure 3
The structure of the three-hidden-layer ELM.Figure 4
The workflow of the three-hidden-layer ELM.Here we take a three-hidden-layer ELM, for example, and analyze the MELM algorithm below. First give the training samplesX,T=xi,tii=1,2,3,…,Q and the three-hidden-layer network structure (each of the three-hidden-layer has l hidden neurons) with the activation function g(x). The structure of the three-hidden-layer ELM has input layer, three hidden layers, and output layer. According to the theory of the TELM algorithm, now we put the three hidden layers as two hidden layers (the first hidden layer is still the first hidden layer, but put the second hidden layer and the third hidden layer together as one hidden layer), so that the structure of the network is the same as the TELM network mentioned above. So we can obtain the weights matrix βnew between the second hidden layer and the output layer. According to the number of the actual samples, we can use formula (8) or formula (9) to calculate the weights β, which can improve the generalization ability of the network.Then the MELM separates the three hidden layers merged previously, so the MELM structure has three hidden layers. So the expected output of the third hidden layer can be expressed as follows:(18)H3=Tβnew+.βnew+ is generalized inverse of the weights matrix βnew.Third the MELM defines the matrixWHE1=B2W2, so the parameters of the third hidden layer can be easily obtained by calculating formula (18) and the formula H3=g(H2W2+B2)=g(WHE1HE1).(19)WHE1=g-1H3HE1+,where H2 is the actual output of the second hidden layer, W2 is the weights between the second hidden layer and the third hidden layer, B2 is the bias of the third hidden neurons, HE1+ is the generational inverse of HE1=1H2T,1 denotes a one-column vector of size Q, and its elements are the scalar unit 1. The notation g-1(x) indicates the inverse of the activation function g(x).In order to test the performance of the proposed MELM algorithm, we adopt different activation functions for regression and classification problems to experiment. Generally, we adopt the logistic sigmoid functiong(x)=1/(1+e-x). The actual output of the third hidden layer is calculated as follows:(20)H4=gWHE1HE1.Finally, the output weights matrixβnew between the third hidden layer and the output layer is calculated as follows: when the number of hidden layer neurons is less than the number of training samples, β can be expressed as follows:(21)βnew=Iλ+H4TH4-1H4TT.When the number of hidden layer neurons is more than the number of training samples,β can be expressed as follows:(22)βnew=H4TIλ+H4H4T-1T.The actual output of the three-hidden-layer ELM network can be expressed as follows:(23)fx=H4βnew.To ensure that the actual final hidden output more approaches the expected hidden output during the training process, the operation process is the optimization of the network structure parameters starting from the second hidden layer.The above is the parameter calculation process of three-hidden-layer ELM network, but the purpose of this paper is to calculate the parameter of the multiple hidden layers ELM network and the final output of the MELM network structure. We can use cycle calculation theory to illustrate the calculating process of the MELM. When the four-hidden-layer ELM network occurs, we can recalculate formula (18) to formula (22) in the calculation process of the network, obtain and record the parameters of each hidden layer, and finally calculate the final output of the MELM network. If the number of hidden layers increased, the calculation process can be recycled and executed in the same way. The calculation process of MELM network can be described as follows.Algorithm 2.
(
1
) Assume the training sample dataset is {X,T}={xi,ti}(i=1,2,3,…,Q), where the matrix X is the input samples and the matrix T is the labeled samples. Each hidden layer has l hidden neurons with the activation function g(x).
(
2
) Randomly initialize the weights W between the input layer and the first hidden layer as well as the bias B of the first hidden neurons WIE=BW,XE=1XT.
(
3
) Calculate the equation H=g(WIEXE).
(
4
) Calculate the weights between the hidden layers and the output layer β=I/λ+HTH-1HTT or β=HTI/λ+HHT-1T.
(
5
) Calculate the expected output of the second hidden layer H1=Tβ+.
6 According to formulae (12)–(14) and the algorithm steps (4, 5), calculate the weights W1 between the first hidden layer and the second hidden layer and the bias B1 of the second hidden neurons WHE=g-1H1HE+.
(
7
) Obtain and update the actual output of the second hidden layer H2=gWHEHE.
(
8
) Update the weights matrix β between the hidden layer and the output layer βnew=I/λ+H2TH2-1H2TT or βnew=H2TI/λ+H2H2T-1T.
(
9
) If the number of the hidden layer is three, we can calculate the parameters by recycle executing the above operation from step (5) to step (9). Now βnew is expressed as follows: βnew=β,HE=1H2T.
(
10
) Calculate the output, f(x)=H2βnew.
If the numberK of the hidden layer is more than three, recycle is executing step (5) to step (9) for (K-1) times. All the H matrix (H1,H2) must be normalized between the range of −0.9 and 0.9, when the max of the matrix is more than 1 and the min of the matrix is less than −1.


## 5. Application

In order to verify the actual effect of the algorithm proposed in this paper, we have done the following experiments which are divided into three parts: regression problems, classification problems, and the application of mineral selection industry. All the experiments are conducted in the MATLAB 2010b computational environment running on a computer with a 2.302 GHZ in i3 CPU.


### 5.1. Regression Problems

To test the performance of the regression problems, several widely used functions are listed below [27]. We use these functions to generate a dataset which includes random selection of sufficient training samples and the remaining is used as a testing samples, and the activation function is selected as the hyperbolic tangent function g(x)=(1-e-x)/(1+e-x).(1)
f1x=∑i=1Dxi.(2)
f2(x)=∑i=1D-1100xi2-xi+12+xi-12.(3)
f3(x)=-20e-0.2∑i=1nxi2/D-e∑i=1ncos⁡(2πxi)/D+20.The symbolD that is set as a positive integer represents the dimensions of the function we are using. The function f2(x) and f3(x) are the complex multimodal function. Each function-experiment has 900 training samples and 100 testing samples. The evaluation criterion of the experiments is the average accuracy, namely, the root mean square error (RMSE) in the regression problems. We first need to conduct some experiments to verify the average accuracy of the neural network structure in different number of hidden neurons and hidden layers. Then we can observe the optimal neural network structure with specific hidden layers number and hidden neurons number compared with the results of the TELM on regression problems.In Table1, we give the average RMSE of the above three functions in the case of the two-hidden-layer structure, the MLELM, and the three-hidden-layer structure (we do a series of experiments, including the four-hidden-layer structure, five-hidden-layer structure, and eight-hidden-layer structure. But the experiments proved that the three-hidden-layer structure for the three functions f1(x),f2(x),f3(x) can achieve the best results, so we only give the value of the RMSE of the three-hidden-layer structure) in regression problems. The table includes the testing RMSE and the training RMSE of the three algorithm structure. We can see that the MELM (three-hidden-layer) has the better results.Table 1
The RMSE of the three function examples.
Algorithm
Training RMSE
Testing RMSE
f
1
(
x
)
TELM
8.6233
E
-
9
1.0797
E
-
8
MELM (three-hidden-layer)
8.722
E
-
15
1.3055
E
-
14
MLELM
1.8428
E
-
6
1.5204
E
-
5
f
2
(
x
)
TELM
0.2867
0.2775
MELM (three-hidden-layer)
0.0011
0.0019
MELM
0.1060
0.1098
f
3
(
x
)
TELM
0.3528
0.6204
MELM (three-hidden-layer)
0.2110
0.4177
MLELM
0.5538
0.4591


### 5.2. Classification Problems

To test the performance of the MELM algorithm proposed on classification datasets, so we quote some datasets (such as Mice Protein, svmguide4, vowel, AutoUnix, and Iris) that are collected from the University of California [28] and the LIBSVM website [29]. The training data and testing data of each experiment are randomly selected from the original datasets. This information of the datasets introduced is given in Table 2.Table 2
The three datasets for the classification.
Datasets
Training samples
Testing samples
Attributes
Class
Mice Protein
750
330
82
8
svmguide4
300
312
10
6
vowel
568
422
10
11
AutoUnix
380
120
101
4
Iris
100
50
4
3The classification performance criteria of the problems are the average classification accuracy of the testing data. In the Figure5, the average testing classification correct percentage for the ELM, TELM, MLELM, and MELM algorithm is shown clearly by using (a) Mice Protein, (b) svmguide4, (c) Vowel, (d) AutoUnix, and (e) Iris, and different datasets (a, b, c, d, and e) have an important effect on the classification accuracy for the three algorithms (ELM, TELM, MLELM, and MELM). But from the changing trend of the average testing correct percentage for the three algorithms with the same datasets, we can see that the MELM algorithm can select the optimal number of hidden layers for the network structure to adapt to the specific datasets, aiming at obtaining the better results. So the datasets of the Mice Protein and the Forest type mapping use the three-hidden-layer ELM algorithm to experiment and the datasets of the svmguide4 use the five-hidden-layer ELM algorithm to experiment.Figure 5
The average testing classification correct percentage for the ELM, TELM, and MELM using (a) Mice Protein, (b) svmguide4, (c) Vowel, (d) AutoUnix, and (e) Iris.
(a)
(b)
(c)
(d)
(e)


### 5.3. The Application of Ores Selection Industry

In this part, we use the actual datasets to analyze and experiment with the model we have established and test the performance of the MELM algorithm. First, our datasets come from AnQian Mining Group, namely, the mining area of Ya-ba-ling and Xi-da-bei. And the samples we selected are the hematite (50 samples), magnetite (90 samples), granite (57 samples), phyllite (15 samples), and chlorite (32 samples). On the precise classification of the ores and the prediction of total iron content of iron ores, people usually use the chemical analysis method. But the analysis process of total iron content is a complex process, a long process, and the color of the solution sometimes without the significantly change in the end. Due to the extensive using of near infrared spectroscopy for chemical composition analysis, we can infer the structure of the unknown substance according to the position and shape of peaks in the absorption spectra. So we can use the theory to obtain the correct information of the ores with the low influence of environment on data acquisition. HR1024 SVC spectrometer is used to test the spectral data of each sample, and the total iron content of hematite and magnetite is obtained from the chemical analysis center of Northeastern University of China. Our experiments’ datasets are the near infrared spectrum information of iron ores which are the absorption rate of different iron ores under the condition of the near infrared spectrum. The wavelength of near infrared spectrum is 300 nm–2500 nm, so the datasets of the iron ores we obtained have the very high dimensions (973 dimensions). We use the theory of the principal component analysis to reduce the dimensions of the datasets.(A) Here we use the algorithm proposed in this paper to classify the five kinds of ores and compare with the results of the ELM algorithm and TELM algorithm. The activation function of the algorithm in classification problems is selected as the sigmoid functiong(x)=1/(1+e-x).Figure6 expresses the average classification accuracy of the ELM, TELM, MLELM, and MELM (three-hidden-layer ELM) algorithm in different hidden layers. After we conduct a series of experiments with different hidden layers of the MELM structure, finally we select the three-hidden-layer ELM structure which can obtain the optimal classification results in the MELM algorithm. Figure 6(a) is the average training correct percentage and the MELM has the high result compared with others in the same hidden neurons. Figure 6(b) is the average testing correct percentage and the MELM has the high result in the hidden neurons between 20 and 60. In the actual situation, we can reasonably choose the number of each hidden neurons to model.Figure 6
Average classification accuracy of different hidden layers.
(a)
(b)(B) Here we use the method proposed in this paper to test the total iron content of the hematite and magnetite and compare with the results of the ELM algorithm and TELM algorithm. Here is the prediction of total iron content of hematite and magnetite with the optimal hidden layers and optimal hidden layer neurons. The activation function of the algorithm in regression problems is selected as the hyperbolic tangent functiong(x)=(1-e-x)/(1+e-x).Figures7 and 8, respectively, expresses the total iron content of the hematite and magnetite by using the ELM, TELM, MLELM, and MELM algorithm (the MELM includes the eight-hidden-layer structure which is the optimal structure in Figure 7; the six-hidden-layer structure which is the optimal structure in Figure 8). The optimal results of each algorithm are obtained by constantly experimenting with the different number of hidden layer neurons. In the prediction process of the hematite, we can see that the MELM has the better results and each hidden layer of this structure has 100 hidden neurons. In the prediction process of the magnetite, we can see that the MELM has the better results and each hidden layer of this structure has 20 hidden neurons. So the MELM algorithm has strong performance to the ores selection problems, which can achieve the best adaptability to the problems by adjusting the number of hidden layers and the number of hidden layer neurons to improve the ability of system regression analysis.Figure 7
The total iron content of the hematite.
(a)
(b)Figure 8
The total iron content of the magnetite.
(a)
(b)


## 5.1. Regression Problems

To test the performance of the regression problems, several widely used functions are listed below [27]. We use these functions to generate a dataset which includes random selection of sufficient training samples and the remaining is used as a testing samples, and the activation function is selected as the hyperbolic tangent function g(x)=(1-e-x)/(1+e-x).(1)
f1x=∑i=1Dxi.(2)
f2(x)=∑i=1D-1100xi2-xi+12+xi-12.(3)
f3(x)=-20e-0.2∑i=1nxi2/D-e∑i=1ncos⁡(2πxi)/D+20.The symbolD that is set as a positive integer represents the dimensions of the function we are using. The function f2(x) and f3(x) are the complex multimodal function. Each function-experiment has 900 training samples and 100 testing samples. The evaluation criterion of the experiments is the average accuracy, namely, the root mean square error (RMSE) in the regression problems. We first need to conduct some experiments to verify the average accuracy of the neural network structure in different number of hidden neurons and hidden layers. Then we can observe the optimal neural network structure with specific hidden layers number and hidden neurons number compared with the results of the TELM on regression problems.In Table1, we give the average RMSE of the above three functions in the case of the two-hidden-layer structure, the MLELM, and the three-hidden-layer structure (we do a series of experiments, including the four-hidden-layer structure, five-hidden-layer structure, and eight-hidden-layer structure. But the experiments proved that the three-hidden-layer structure for the three functions f1(x),f2(x),f3(x) can achieve the best results, so we only give the value of the RMSE of the three-hidden-layer structure) in regression problems. The table includes the testing RMSE and the training RMSE of the three algorithm structure. We can see that the MELM (three-hidden-layer) has the better results.Table 1
The RMSE of the three function examples.
Algorithm
Training RMSE
Testing RMSE
f
1
(
x
)
TELM
8.6233
E
-
9
1.0797
E
-
8
MELM (three-hidden-layer)
8.722
E
-
15
1.3055
E
-
14
MLELM
1.8428
E
-
6
1.5204
E
-
5
f
2
(
x
)
TELM
0.2867
0.2775
MELM (three-hidden-layer)
0.0011
0.0019
MELM
0.1060
0.1098
f
3
(
x
)
TELM
0.3528
0.6204
MELM (three-hidden-layer)
0.2110
0.4177
MLELM
0.5538
0.4591


## 5.2. Classification Problems

To test the performance of the MELM algorithm proposed on classification datasets, so we quote some datasets (such as Mice Protein, svmguide4, vowel, AutoUnix, and Iris) that are collected from the University of California [28] and the LIBSVM website [29]. The training data and testing data of each experiment are randomly selected from the original datasets. This information of the datasets introduced is given in Table 2.Table 2
The three datasets for the classification.
Datasets
Training samples
Testing samples
Attributes
Class
Mice Protein
750
330
82
8
svmguide4
300
312
10
6
vowel
568
422
10
11
AutoUnix
380
120
101
4
Iris
100
50
4
3The classification performance criteria of the problems are the average classification accuracy of the testing data. In the Figure5, the average testing classification correct percentage for the ELM, TELM, MLELM, and MELM algorithm is shown clearly by using (a) Mice Protein, (b) svmguide4, (c) Vowel, (d) AutoUnix, and (e) Iris, and different datasets (a, b, c, d, and e) have an important effect on the classification accuracy for the three algorithms (ELM, TELM, MLELM, and MELM). But from the changing trend of the average testing correct percentage for the three algorithms with the same datasets, we can see that the MELM algorithm can select the optimal number of hidden layers for the network structure to adapt to the specific datasets, aiming at obtaining the better results. So the datasets of the Mice Protein and the Forest type mapping use the three-hidden-layer ELM algorithm to experiment and the datasets of the svmguide4 use the five-hidden-layer ELM algorithm to experiment.Figure 5
The average testing classification correct percentage for the ELM, TELM, and MELM using (a) Mice Protein, (b) svmguide4, (c) Vowel, (d) AutoUnix, and (e) Iris.
(a)
(b)
(c)
(d)
(e)


## 5.3. The Application of Ores Selection Industry

In this part, we use the actual datasets to analyze and experiment with the model we have established and test the performance of the MELM algorithm. First, our datasets come from AnQian Mining Group, namely, the mining area of Ya-ba-ling and Xi-da-bei. And the samples we selected are the hematite (50 samples), magnetite (90 samples), granite (57 samples), phyllite (15 samples), and chlorite (32 samples). On the precise classification of the ores and the prediction of total iron content of iron ores, people usually use the chemical analysis method. But the analysis process of total iron content is a complex process, a long process, and the color of the solution sometimes without the significantly change in the end. Due to the extensive using of near infrared spectroscopy for chemical composition analysis, we can infer the structure of the unknown substance according to the position and shape of peaks in the absorption spectra. So we can use the theory to obtain the correct information of the ores with the low influence of environment on data acquisition. HR1024 SVC spectrometer is used to test the spectral data of each sample, and the total iron content of hematite and magnetite is obtained from the chemical analysis center of Northeastern University of China. Our experiments’ datasets are the near infrared spectrum information of iron ores which are the absorption rate of different iron ores under the condition of the near infrared spectrum. The wavelength of near infrared spectrum is 300 nm–2500 nm, so the datasets of the iron ores we obtained have the very high dimensions (973 dimensions). We use the theory of the principal component analysis to reduce the dimensions of the datasets.(A) Here we use the algorithm proposed in this paper to classify the five kinds of ores and compare with the results of the ELM algorithm and TELM algorithm. The activation function of the algorithm in classification problems is selected as the sigmoid functiong(x)=1/(1+e-x).Figure6 expresses the average classification accuracy of the ELM, TELM, MLELM, and MELM (three-hidden-layer ELM) algorithm in different hidden layers. After we conduct a series of experiments with different hidden layers of the MELM structure, finally we select the three-hidden-layer ELM structure which can obtain the optimal classification results in the MELM algorithm. Figure 6(a) is the average training correct percentage and the MELM has the high result compared with others in the same hidden neurons. Figure 6(b) is the average testing correct percentage and the MELM has the high result in the hidden neurons between 20 and 60. In the actual situation, we can reasonably choose the number of each hidden neurons to model.Figure 6
Average classification accuracy of different hidden layers.
(a)
(b)(B) Here we use the method proposed in this paper to test the total iron content of the hematite and magnetite and compare with the results of the ELM algorithm and TELM algorithm. Here is the prediction of total iron content of hematite and magnetite with the optimal hidden layers and optimal hidden layer neurons. The activation function of the algorithm in regression problems is selected as the hyperbolic tangent functiong(x)=(1-e-x)/(1+e-x).Figures7 and 8, respectively, expresses the total iron content of the hematite and magnetite by using the ELM, TELM, MLELM, and MELM algorithm (the MELM includes the eight-hidden-layer structure which is the optimal structure in Figure 7; the six-hidden-layer structure which is the optimal structure in Figure 8). The optimal results of each algorithm are obtained by constantly experimenting with the different number of hidden layer neurons. In the prediction process of the hematite, we can see that the MELM has the better results and each hidden layer of this structure has 100 hidden neurons. In the prediction process of the magnetite, we can see that the MELM has the better results and each hidden layer of this structure has 20 hidden neurons. So the MELM algorithm has strong performance to the ores selection problems, which can achieve the best adaptability to the problems by adjusting the number of hidden layers and the number of hidden layer neurons to improve the ability of system regression analysis.Figure 7
The total iron content of the hematite.
(a)
(b)Figure 8
The total iron content of the magnetite.
(a)
(b)


## 6. Conclusion

The MELM we proposed in this paper solves the two problems which exist in the training process of the original ELM. The first is the stability of single network that the disadvantage will also influence the generalization performance. The second is that the output weights are calculated by the product of the generalized inverse of hidden layer output and the system actual output. And the parameters randomly selected of hidden layer neurons which can lead to the singular matrix or morbid matrix. At the same time, the MELM network structure also improves the average accuracy of training and testing performance compared to the ELM and TELM network structure. The MELM algorithm inherits the characteristics of traditional ELM that randomly initializes the weights and bias (between the input layer and the first hidden layer), also adopts a part of the TELM algorithm, and uses the inverse activation function to calculate the weights and bias of hidden layers (except the first hidden layer). Then we make the actual hidden layer output approximate to the expected hidden layer output and use the parameters obtained above to calculate the actual output. In the function regression problems, this algorithm reduces the least mean square error. In the datasets classification problems, the average accuracy of the multiple classifications is significantly higher than that of the ELM and TELM network structure. In such cases, the MELM is able to improve the performance of the network structure.



---

*Source: 4670187-2017-12-13.xml*