# SmoothingL0 Regularization for Extreme Learning Machine

Authors: Qinwei Fan; Ting Liu
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Extreme learning machine (ELM) has been put forward for single hidden layer feedforward networks. Because of its powerful modeling ability and it needs less human intervention, the ELM algorithm has been used widely in both regression and classification experiments. However, in order to achieve required accuracy, it needs many more hidden nodes than is typically needed by the conventional neural networks. This paper considers a new efficient learning algorithm for ELM with smoothingL0 regularization. A novel algorithm updates weights in the direction along which the overall square error is reduced the most and then this new algorithm can sparse network structure very efficiently. The numerical experiments show that the ELM algorithm with smoothing L0 regularization has less hidden nodes but better generalization performance than original ELM and ELM with L1 regularization algorithms.


---

## Body

## 1. Introduction

Recently, the studies and applications of artificial intelligence have raised a new high tide. Artificial neural networks, also known as neural networks, are an usual artificial intelligence learning algorithm, which can autonomously learn the characteristics of data, hence avoiding the course of artificial selection.Among them, the most common network type is feedforward neural networks (SLFNs), such as perceptron, back-propagation (BP), and radial basis function (RBF) networks. Generally, it includes an input layer, several hidden layers, and an output layer. Each neuron is arranged in layers, only connected to the antecedent layer. It takes the output of the previous layer and delivers it to the next layer. There is no feedback between the layers. As a result of its good characteristics, it has been widely used in various fields [1–4]. However, the drawback of the feedforward neural network is inefficiency, especially when it deals with complex data.Lately, ELM was proposed by Huang et al. [5–7]. Other than the conventional neural network, ELM is a new type of SLFNs, which input weights and the thresholds of the hidden layer can be discretionarily assigned if the activation function of the hidden layer is immortally differentiable. The output weights can be decided by Moore–Penrose generalized inverse. In some simulations, the learning speed of the ELM algorithm can be completed in seconds [8]. It is extremely fast and has better generalization performance than other primitive gradient-based learning algorithms.However, in order to guarantee the veracity of regression and classification experiments, the ELM method needs a huge number of hidden nodes. This leads to a particularly complex network structure. It inevitably increases the model size and testing time. Therefore, in order to ameliorate the predictive power and generalization performance of ELM, choosing the appropriate quantity of hidden layer nodes is a hot topic in the research of ELM.Many scholars have done a lot of research on the hidden layer structure optimization of ELM and achieved many remarkable results. Rong et al. [9] proposed a fast pruned ELM (P-ELM) and successfully applied it to the classification problem. This learning mechanism mainly achieves the effect of “pruning” the network structure by deleting hidden layer nodes irrelevant to class labels. Then, Miche et al. [10] proposed an optimal extreme learning machine (OP-ELM) for optimal pruning and extended the pruning method. Incremental extreme learning machine (I-ELM) was put forward by Huang et al. [11], and it can increase the quantity of hidden layer nodes adaptively. The efficiency of learning is improved, and the network performance is optimized by increasing nodes in the training process. In [12], Yu and Deng proposed a series of new efficient algorithms for training ELM, which exploit both the structure of SLFNs and the gradient information over all training epochs.In conclusion, the number of neurons is an important factor that determines the structure of the network. There are two methods to adjust the size of the network, one is the growing method, and the other is the pruning method. For the growing method, a common strategy is to begin with a smaller network and then add new hidden nodes during the process of training the network [13, 14]. The pruning method starts with a large network structure and then prunes unimportant weights or nodes [15–19]. There are some network structure optimization methods, such as regularization method, cross validation, sensitivity analysis, mutual information-based, and magnitude-based methods [20–23].In this paper, based on the regularization method, we propose a new efficient algorithm to train ELM. This novel algorithm updates the weights in the direction along which the overall square error is reduced the most and then to the direction which enforces sparsity of the weights effectively. Our strategy is to combine theL0 regularization term with the standard error function. On the basis of the regularization theory, when p⟶0, the Lp regularization method tends to produce more sparse results. However, it is not differentiable at the origin of coordinates and leads to NP-hard problem [24]. So, L0 regularization cannot make use of the optimization algorithm directly [25]. Also, there are some other sparse strategies for optimized structure of neural networks [26, 27]. Inspired by the work on smoothing L1/2 regularization for feedforward neural networks [28, 29] and other optimizing strategies, this paper draws on some smoothing techniques to overcome the shortcomings of the origin is not differentiable and the phenomenon of oscillation, and we will show the related details in the next section. The main contributions are as follows:(1)
It is shown how the smoothingL0 regularization is proposed to train ELM, which can discriminate important weights from unnecessary weights and drives the unnecessary weights to zeros, thus effectively simplified the structure of the network.(2)
The ideal approximate solution is obtained by using the smoothing approximation techniques. The drawbacks of nonsmooth and nondifferentiable of the normalL0 regularization term can be addressed. And this effectively prevents the oscillatory phenomenon of the training.(3)
Numerical simulations shown that the sparsity effect and the generalization performance of the ELMSL0 are better than those of the original ELM and ELML1. For example, from Tables1–3, the results clearly show that the novel algorithm ELMSL0 uses fewer neurons but has higher testing accuracy for most of the data sets.Table 1
Comparison of RMSE in ELM, ELML1, and ELMSL0 algorithms on the SinC data set.
Learning algorithmsTraining (RMSE)Testing (RMSE)Average nodesELM0.11570.009650ELML10.11580.008333.90ELMSL00.11580.008031.30Table 2
Comparison of RMSE in ELM, ELML1, and ELMSL0 algorithms on Gabor function.
Learning algorithmsTraining (RMSE)Testing (RMSE)Average nodesELM0.22620.0408100ELML10.22790.036964.60ELMSL00.22720.034660.45Table 3
Comparison of success rate in ELM, ELML1, and ELMSL0 algorithms on classification data sets.
Data setTraining accuracyTesting accuracyAverage nodesELM (%)ELML1 (%)ELMSL0 (%)ELM (%)ELML1 (%)ELMSL0 (%)ELMELML1ELMSL0Diabetes83.4081.6281.1373.9775.5076.6810063.9060.20Australiancredit90.7989.9189.1784.8885.5786.5010063.4059.50Heart89.7287.9288.1178.1780.2882.895031.8029.05Sonar100.00100.00100.0077.9478.5382.50500315.90310.10Wiscousin98.5297.5697.8394.8396.4596.6210063.5059.70SPECT96.5696.0096.5672.0870.6769.36300189.60186.30SPECTF100.00100.00100.0060.3560.4059.01300192.15187.95Iris98.9099.2598.8091.8092.4096.005030.7027.60Wine100.00100.0099.9292.7693.2896.557041.8039.70Glass98.4796.5396.6784.9280.1788.206035.8033.75Olitos99.3596.7996.0071.8676.9480.755029.8027.20E.coli92.7891.2790.8981.3584.6086.106035.5532.60Seeds99.3598.5798.2094.0594.2995.005030.3527.65Wholesale90.2183.3682.3749.6258.7560.57200125.85121.20The remainder of this paper is organized as follows. Section2 describes the network structure of the traditional ELM algorithm. Section 3 shows the ELM algorithm with smoothing L0 regularization (ELMSL0). Section 4 shows that how the smoothing L0 regularization helps the gradient-based method to bring forth sparse results. The performance of the ELMSL0 algorithm is compared with ELM and ELM with L1 regularization (ELML1) algorithms by regression and classification applications in Section 5. Finally, conclusion is given in Section 6.


## 2. Extreme Learning Machine (ELM)

Next, we will do a description of the traditional ELM. The neuron quantities of the input, hidden, and output layers aren, L, and m, respectively (see Figure 1). The ELM algorithm randomly generates the connection weights of the input layer and hidden layer, and the thresholds of the hidden layer neurons need not be adjusted in the process of training the network. As long as the number of hidden layer neurons is set, the unique optimal solution can be acquired. The input matrix X and output matrix Y of the training set with Q samples are, respectively,(1)X=x11x12⋯x1Qx21x22⋯x2Q⋮⋮⋯⋮xn1xn2⋯xnQn×Q,Y=y11y12⋯y1Qy21y22⋯y2Q⋮⋮⋯⋮ym1ym2⋯ymQm×Q.Figure 1
The network structure of extreme learning machine.On the basis of the structure of ELM, the actual outputT of the network is as follows:(2)T=t11t12⋯t1Qt21t22⋯t2Q⋮⋮⋯⋮tm1tm2⋯tmQm×Q.The normative ELM withL hidden nodes and activation function gx can be expressed as(3)∑i=1Lβigixj=∑i=1Lβigwi⋅xj+bi=tj,where tj=t1j,t2j,…,tmjT, and the specific expression of tj is as follows:(4)tj=∑i=1Lβi1gwi⋅xj+bi∑i=1Lβi2gwi⋅xj+bi⋮∑i=1Lβimgwi⋅xj+bim×1,where j=1,2,…,Q,xj=x1j,x2j,…,xnjT,wi=wi1,wi2,…,win is the weight vector connecting the ith hidden node and input nodes, βi=βi1,βi2,…,βimT is the weight vector connecting the ith hidden node and output nodes, bi is the threshold of the ith hidden node, and wi⋅xj denotes the inner product of wi and xj.Equation (4) can be written as Hβ=TT, and H is called the hidden layer output matrix of the ELM, where(5)Hw1,…,wL,b1,…,bL,x1,…,xQ=gw1⋅x1+b1⋯gwL⋅x1+bLgw1⋅x2+b1⋯gwL⋅x2+bL⋮⋯⋮gw1⋅xQ+b1⋯gwL⋅xQ+bLQ×L.By using the Moore–Penrose generalized inverse ofH, the least-squares solutions of Hβ=TT can be written as(6)β^=H†TT.


## 3. Extreme Learning Machine withL0 Regularization (ELMSL0)

In the general case, the weights with large absolute value play a more important role in training. In order to prune the network availably, we need to distinguish the unimportant weights in the first place and then remove them. Therefore, the aim of the network training is to find the appropriate weightsβ^ that can minimize Hβ−YT as well as βp:(7)β^=argminHβ−YT22+λβpp,where λ is the regularization coefficient to balance the training accuracy and the complexity of the network. βpp is called Lp regularization, and it shows different properties for different values of p. Moreover, L2 regularization can prevent the weight from increasing too large effectively, but it is not sparse. L1 regularization can generate sparse solutions. L0 regularization produces more sparse solutions than L1 regularization. So, we focus on the L0 regularization in this paper. Here, β00 is the L0 regularization with the L0 norm defined by(8)β00=limp⟶0β1p+β2p+⋯+βnp1/pp=limp⟶0β1p+β2p+⋯+βnp,for β=β1,β2,…,βnT. We know that the quantity of nonzero elements of the vector β equals to β00. However, according to combinatorial optimization theory, we know that minimizing Hβ−YT22+λβ00 is a NP-hard problem. In order to overcome this drawback, the following continuous function Hρβ is employed to approximate the L0 regularization at the origin:(9)Hρβ=∑i=1nhρβi.Here,hρ⋅ is continuously differentiable on ℝ and(10)limp⟶0hρβ=1,ifβ≠0,0,ifβ=0,where ρ is a positive number, and it is used to control the degree of how Hρβ approximates the L0 regularization. A representative choice for hρβ is(11)hρβ=1−exp−β22ρ2.To sum up, the error function withL0 regularization has the following form:(12)Eβ=∑k=1QHβ−YTk∗THβ−YTk∗+λHρβ,where Hβ−YTk∗ represents the kth row of the matrix Hβ−YT. We use the gradient descent method to minimize the error function, and the gradient of the error function is given by(13)Eβiβ=∑k=1Q2HkiHβ−YTki+λβiρ2exp−βi22ρ2,where i=1,2,…,L. For any initial value β0, the batch gradient method with the smoothing L0 regularization term updates the weights by(14)βn+1=βn−ηEβiβ,n=0,1,2,…,where η>0 is the learning rate.


## 4. Description of Sparsity

Regularized sparse model plays an increasingly important role in machine learning and image processing. It removes a large number of redundant variables and only retains the explanatory variables that are most relevant to the response variables, which simplifies the model while retaining the most important information in the data sets and effectively solves many practical problems.Next, we show that the ELMSL0 algorithm differentiates between important weights and unimportant weights. Among them, the weights with large absolute value are more important. The curves ofhρβ and hρ′β for different ρ are shown in Figures 2 and 3. Notice that the only minimum point of hρβ is β=0. On the grounds of (10), for adequately small ρ, there exists a positive number β0, such that hρ′β≈0 when β>β0 and hρ′β will be quite large when β<β0. Therefore, when using the ELMSL0 algorithm to train the network, the absolute value is greater than the weight of β0, which is not easily affected by the regularization term. The unimportant weights are absolutely less than β0, they will be driven to zero during the training. This makes clear why the smoothing L0 regularization term can help the gradient-based method achieve sparse results.Figure 2
The curve ofhρβ for different ρ.Figure 3
The curve ofhρ′β for different ρ.In the light of the above discussion, we look forward to more weights fall into the interval−β0,β0 to achieve the more sparse results. To make this true, one choice is to set the very small initial weights, but it leads to slow convergence at the beginning of the training procedure on account of the gradient of the square error function will be very small [30]. The weight decay method is another choice, and the size of the network weight is reduced compulsively during the training process. As shown in Figure 3, if the parameter ρ is set too small, it will be unstable and affect the performance of the algorithm. Therefore, the parameter ρ should be set to a decreasing sequence with lower bounds.


## 5. Simulation Results

In order to substantiate the reliability of the introduced ELMSL0 algorithm, we conduct some experiments which are both regression and classification applications. In Section5.1, ELM, ELML1, and ELMSL0 algorithms are used to approximate the SinC and multidimensional Gabor function. There are a few real-word classification data to test the performance of three algorithms in Section 5.2.


### 5.1. Function Regression Problem

The SinC function is defined by(15)yx=sinx/x,x≠0,1,x=0,and it has a training set xi,yi and testing set xi,yi with 5000 data, where xi is uniformly distributed on the interval (−10, 10). To make the regression problem more real, adding the uniform noise distributed in [−0.2, 0.2] to all the training samples, the testing samples remain noise-free. There are 50 hidden nodes, and the activation function is RBF for ELM, ELML1, and ELMSL0 algorithms. We choose the learning rate η=0.2, the regularization coefficient λ=0.05, and hρβ=1−exp−β2/2ρ2, where ρ is set to be a decreasing sequence. As demonstrated by Figure 3, hρ′β may be absolutely too large when parameter ρ is too little, which may cause instability during the training procedure. Thus, we set a lower bound for ρ=0.06. Figures 4–6 exhibit the prediction results. Figure 4 shows the prediction values of the ELM algorithm for SinC function. Figure 5 shows the prediction values of the ELML1 algorithm. Figure 6 shows the prediction values of the ELMSL0 algorithm. It is obvious that the ELMSL0 algorithm has better prediction performance than other two algorithms.Figure 4
Prediction of the ELM algorithm for SinC function.Figure 5
Prediction of the ELML1 algorithm for SinC function.Figure 6
Prediction of the ELMSL0 algorithm for SinC function.The root mean square error (RMSE) is usually used as error function for evaluating the performance of the algorithm. The smaller value of the RMSE shows that the algorithm is more accurate to the described experimental data. By the following equation, we can calculate the RMSE:(16)RMSE=∑i=1nfxi−yi2n,where fxi denotes the predicted data and yi denotes the actual data. We run 50 experiments on each of the three algorithms and then give the average training and testing RMSE values in Table 1. By comparing the number of hidden nodes of three algorithms, the ELMSL0 algorithm needs less hidden nodes but the testing sets have the highest accuracy.Next, we consider using three algorithms to approximate a multidimensional Gabor function individually:(17)Zx,y=12π0.52exp−x2+y220.52cos2πx+y.We select 2601 samples from an equably spaced51×51 grid on −0.5≤x≤0.5 and −0.5≤y≤0.5 for training samples, and 2601 testing samples are selected similarly. In order to prevent overfitting in the training process, the noise which is evenly distributed in [−0.4, 0.4] has been added to training samples, and there is no noise in the testing samples. The original ELM algorithm has 100 hidden nodes, and RBF is selected for the activation function. Then, we use ELML1 and ELMSL0 algorithms to prune the network, respectively, choosing the learning rate η=0.2, the regularization coefficient λ=0.05, and ρ=0.06 to approximate Gabor function (Figure 7). We perform 50 experiments on each of the three algorithms. As shown in Figures 8–10, it is so clear that the ELMSL0 algorithm has better prediction performance than conventional ELM and ELML1 algorithms. Table 2 gives the average training and testing RMSE values and the number of hidden nodes required by three algorithms. The accuracy of the ELMSL0 algorithm in testing sets is higher than other algorithms.Figure 7
Gabor function.Figure 8
Prediction of the ELM algorithm for Gabor function.Figure 9
Prediction of the ELML1 algorithm for Gabor function.Figure 10
Prediction of the ELMSL0 algorithm for Gabor function.


### 5.2. Real-Word Classification Problems

In this section, we compare the generalization performance of ELM, ELML1, and ELMSL0 algorithms in some real-word classification problems, which include seven binary classification problems and seven multiclass classification problems. Tables4 and 5 describe these classified data clearly, including the number of training and testing data and the attributes of each classification data set.Table 4
Information of the binary classification data sets.
Data setTrainingTestingAttributesDiabetes5302388Australiancredit48021014Heart1809013Sonar1406860Wiscousin4802039SPECT8018722SPECTF8018744Table 5
Information of the multiclass classification data sets.
Data setTrainingTestingAttributesClassesIris1005043Wine12058133Glass1506496Olitos8436254E.coli23610088Seeds1476373Wholesale30813273To explain Tables4 and 5 clearly, we take the Diabetes and Iris data sets for example. Diabetes data belongs to either positive or negative class. The data from “Pima Indians Diabetes Database” are created by Applied Physics Laboratory. It consists of 768 women over the age of 21 who come from Phoenix, Arizona. Iris data include four features: calyx length, calyx width, petal length, and petal width. It contains three classes: Setosa, Versicolor, and Virginica.There are different hidden nodes of each data, and the activation function is sigmoid function for three algorithms. We choose the learning rateη=0.02, the regularization coefficient λ=0.03, and hρβ=1−exp−β2/2ρ2, where ρ is a decreasing sequence with a lower bound of 0.08. We run 50 experiments with each data set, and Table 3 shows the average training and testing accuracy. It can be found that the ELMSL0 algorithm requires fewer hidden nodes without affecting the testing accuracy of the algorithm. So, the ELMSL0 algorithm not only produces sparse results to prune the network but also has better generalization performance than other two algorithms in most classified data sets.It is well known that regularization coefficient and the number of hidden nodes affect the accuracy of algorithms. Therefore, we need several experiments to select the appropriate regularization coefficients. A binary Sonar data set and a multiclassification Iris data set are selected here. We take the Sonar signal classification data set from the UCI machine learning repository for example. It is a typical benchmark problem in the field of neural networks. All samples are divided into two categories, one is sonar signals bounced off a metal cylinder, the other is those bounced off a roughly cylindrical rock. Here, we consider the number of hidden nodes is 500 in ELM, pruned by ELML1 and ELMSL0 algorithms. Figure11 shows the testing accuracy of the two algorithms with different regularization coefficients. Figure 12 shows the number of hidden nodes required by the two algorithms as the regularization coefficient increases. From Figures 11 and 12, the ELMSL0 algorithm requires fewer hidden nodes but better generalization performance than the ELML1 algorithm with different regularization coefficients.Figure 11
Testing classification performance of (a) Sonar and (b) Iris with different regularization coefficients.
(a)(b)Figure 12
The number of hidden nodes required by ELML1 and ELMSL0 algorithms with different regularization coefficients.
(a)(b)


## 5.1. Function Regression Problem

The SinC function is defined by(15)yx=sinx/x,x≠0,1,x=0,and it has a training set xi,yi and testing set xi,yi with 5000 data, where xi is uniformly distributed on the interval (−10, 10). To make the regression problem more real, adding the uniform noise distributed in [−0.2, 0.2] to all the training samples, the testing samples remain noise-free. There are 50 hidden nodes, and the activation function is RBF for ELM, ELML1, and ELMSL0 algorithms. We choose the learning rate η=0.2, the regularization coefficient λ=0.05, and hρβ=1−exp−β2/2ρ2, where ρ is set to be a decreasing sequence. As demonstrated by Figure 3, hρ′β may be absolutely too large when parameter ρ is too little, which may cause instability during the training procedure. Thus, we set a lower bound for ρ=0.06. Figures 4–6 exhibit the prediction results. Figure 4 shows the prediction values of the ELM algorithm for SinC function. Figure 5 shows the prediction values of the ELML1 algorithm. Figure 6 shows the prediction values of the ELMSL0 algorithm. It is obvious that the ELMSL0 algorithm has better prediction performance than other two algorithms.Figure 4
Prediction of the ELM algorithm for SinC function.Figure 5
Prediction of the ELML1 algorithm for SinC function.Figure 6
Prediction of the ELMSL0 algorithm for SinC function.The root mean square error (RMSE) is usually used as error function for evaluating the performance of the algorithm. The smaller value of the RMSE shows that the algorithm is more accurate to the described experimental data. By the following equation, we can calculate the RMSE:(16)RMSE=∑i=1nfxi−yi2n,where fxi denotes the predicted data and yi denotes the actual data. We run 50 experiments on each of the three algorithms and then give the average training and testing RMSE values in Table 1. By comparing the number of hidden nodes of three algorithms, the ELMSL0 algorithm needs less hidden nodes but the testing sets have the highest accuracy.Next, we consider using three algorithms to approximate a multidimensional Gabor function individually:(17)Zx,y=12π0.52exp−x2+y220.52cos2πx+y.We select 2601 samples from an equably spaced51×51 grid on −0.5≤x≤0.5 and −0.5≤y≤0.5 for training samples, and 2601 testing samples are selected similarly. In order to prevent overfitting in the training process, the noise which is evenly distributed in [−0.4, 0.4] has been added to training samples, and there is no noise in the testing samples. The original ELM algorithm has 100 hidden nodes, and RBF is selected for the activation function. Then, we use ELML1 and ELMSL0 algorithms to prune the network, respectively, choosing the learning rate η=0.2, the regularization coefficient λ=0.05, and ρ=0.06 to approximate Gabor function (Figure 7). We perform 50 experiments on each of the three algorithms. As shown in Figures 8–10, it is so clear that the ELMSL0 algorithm has better prediction performance than conventional ELM and ELML1 algorithms. Table 2 gives the average training and testing RMSE values and the number of hidden nodes required by three algorithms. The accuracy of the ELMSL0 algorithm in testing sets is higher than other algorithms.Figure 7
Gabor function.Figure 8
Prediction of the ELM algorithm for Gabor function.Figure 9
Prediction of the ELML1 algorithm for Gabor function.Figure 10
Prediction of the ELMSL0 algorithm for Gabor function.


## 5.2. Real-Word Classification Problems

In this section, we compare the generalization performance of ELM, ELML1, and ELMSL0 algorithms in some real-word classification problems, which include seven binary classification problems and seven multiclass classification problems. Tables4 and 5 describe these classified data clearly, including the number of training and testing data and the attributes of each classification data set.Table 4
Information of the binary classification data sets.
Data setTrainingTestingAttributesDiabetes5302388Australiancredit48021014Heart1809013Sonar1406860Wiscousin4802039SPECT8018722SPECTF8018744Table 5
Information of the multiclass classification data sets.
Data setTrainingTestingAttributesClassesIris1005043Wine12058133Glass1506496Olitos8436254E.coli23610088Seeds1476373Wholesale30813273To explain Tables4 and 5 clearly, we take the Diabetes and Iris data sets for example. Diabetes data belongs to either positive or negative class. The data from “Pima Indians Diabetes Database” are created by Applied Physics Laboratory. It consists of 768 women over the age of 21 who come from Phoenix, Arizona. Iris data include four features: calyx length, calyx width, petal length, and petal width. It contains three classes: Setosa, Versicolor, and Virginica.There are different hidden nodes of each data, and the activation function is sigmoid function for three algorithms. We choose the learning rateη=0.02, the regularization coefficient λ=0.03, and hρβ=1−exp−β2/2ρ2, where ρ is a decreasing sequence with a lower bound of 0.08. We run 50 experiments with each data set, and Table 3 shows the average training and testing accuracy. It can be found that the ELMSL0 algorithm requires fewer hidden nodes without affecting the testing accuracy of the algorithm. So, the ELMSL0 algorithm not only produces sparse results to prune the network but also has better generalization performance than other two algorithms in most classified data sets.It is well known that regularization coefficient and the number of hidden nodes affect the accuracy of algorithms. Therefore, we need several experiments to select the appropriate regularization coefficients. A binary Sonar data set and a multiclassification Iris data set are selected here. We take the Sonar signal classification data set from the UCI machine learning repository for example. It is a typical benchmark problem in the field of neural networks. All samples are divided into two categories, one is sonar signals bounced off a metal cylinder, the other is those bounced off a roughly cylindrical rock. Here, we consider the number of hidden nodes is 500 in ELM, pruned by ELML1 and ELMSL0 algorithms. Figure11 shows the testing accuracy of the two algorithms with different regularization coefficients. Figure 12 shows the number of hidden nodes required by the two algorithms as the regularization coefficient increases. From Figures 11 and 12, the ELMSL0 algorithm requires fewer hidden nodes but better generalization performance than the ELML1 algorithm with different regularization coefficients.Figure 11
Testing classification performance of (a) Sonar and (b) Iris with different regularization coefficients.
(a)(b)Figure 12
The number of hidden nodes required by ELML1 and ELMSL0 algorithms with different regularization coefficients.
(a)(b)


## 6. Conclusions

In this paper, we use a smoothing function to approximateL0 regularization, proposing a pruning method with smoothing L0 regularization (ELMSL0) for training and pruning extreme learning machine. And also it is shown that the ELMSL0 algorithm can produce sparse results to prune networks availably. Both regression and classification problems show that the ELMSL0 algorithm has better generalization performance and simpler network structure. In future, we consider applying the intelligent algorithm to the ELM algorithm to find the most appropriate weights and thresholds.



---

*Source: 9175106-2020-07-06.xml*