# SGB-ELM: An Advanced Stochastic Gradient Boosting-Based Ensemble Scheme for Extreme Learning Machine

Authors: Hua Guo; Jikui Wang; Wei Ao; Yulin He
Journal: Computational Intelligence and Neuroscience
 (2018)

---

## Abstract

A novel ensemble scheme for extreme learning machine (ELM), named Stochastic Gradient Boosting-based Extreme Learning Machine (SGB-ELM), is proposed in this paper. Instead of incorporating the stochastic gradient boosting method into ELM ensemble procedure primitively, SGB-ELM constructs a sequence of weak ELMs where each individual ELM is trained additively by optimizing the regularized objective. Specifically, we design an objective function based on the boosting mechanism where a regularization item is introduced simultaneously to alleviate overfitting. Then the derivation formula aimed at solving the output-layer weights of each weak ELM is determined using the second-order optimization. As the derivation formula is hard to be analytically calculated and the regularized objective tends to employ simple functions, we take the output-layer weights learned by the current pseudo residuals as an initial heuristic item and thus obtain the optimal output-layer weights by using the derivation formula to update the heuristic item iteratively. In comparison with several typical ELM ensemble methods, SGB-ELM achieves better generalization performance and predicted robustness, which demonstrates the feasibility and effectiveness of SGB-ELM.


---

## Body

## 1. Introduction

Extreme learning machine (ELM) was proposed as a promising learning algorithm for single-hidden-layer feedforward neural networks (SLFN) by Huang [1–3], which randomly chooses weights and biases for hidden nodes and analytically determines the output-layer weights by using Moore-Penrose (MP) generalized inverse [4]. Due to avoiding the iterative parameter adjustment and time-consuming weight updating, ELM obtains an extremely fast learning speed and thus attracts a lot of attention. However, random initialization of input-layer weights and hidden biases might generate some suboptimal parameters, which have negative impact on its generalization performance and predicted robustness.To alleviate such weakness, many works have been proposed to further improve the generalization capability and stability of ELM, where ELM ensemble algorithms are the representative ones. Three representative ELM ensemble algorithms are summarized as follows. The earliest ensemble based ELM (EN-ELM) method was presented by Liu and Wang in [5]. EN-ELM introduced the cross-validation scheme into its training phase, where the original training dataset was partitioned into R subsets and then R pairs of training and validation sets were obtained so that each training set consists of R-1 subsets. Additionally, with updated input weights and hidden biases, K individual ELMs were trained based on each pair of the training and validation set. There were totally K×R ELMs that were constructed for decision-making in EN-ELM algorithm. Cao et al. [6] proposed a voting-based ELM (V-ELM) ensemble algorithm, which made the final decision based on the majority voting mechanism in classification applications. All the individual ELMs in V-ELM were trained on the same training dataset and the learning parameters of each basic ELM were randomly initialized independently. Moreover, a genetic ensemble of ELM (GE-ELM) method was designed by Xue et al. in [7], which used the genetic algorithm to produce optimal input weights as well as hidden biases for individual ELMs and selected ELMs equipped with not only higher fitness values but also smaller norm of output weights from the candidate networks. In GE-ELM, the fitness value of each individual ELM was evaluated based on the validation set which was randomly selected from the entire training dataset. There are still several other types of ELM ensemble algorithms which can be found in literatures [8–13].As for ensemble of the traditional neural networks, the most prevailing approaches are Bagging and Boosting. In Bagging scheme [14], it generates several training datasets from the original training dataset and then trains a component neural network from each of those training datasets. Boosting mechanism [15] generates a series of component neural networks whose training datasets are determined by the performance of former ones. There are also many other approaches for training the component neural networks. Hampshire [16] utilizes different object functions to train distinct component neural networks. Xu et al. [17] introduce the stochastic gradient boosting ensemble scheme to bioinformatics applications. Yao et al. [18] regard all the individuals in an evolved population of neural networks as component networks.In this paper, a new ELM ensemble scheme called Stochastic Gradient Boosting-based Extreme Learning Machine (SGB-ELM) which makes use of the mechanism of stochastic gradient boosting [19, 20] is proposed. SGB-ELM constructs an ensemble model by training a sequence of ELMs where the output weights of each individual ELM is learned by optimizing the regularized objective in an additive manner. More specifically, we design an objective based on the training mechanism of boosting method. In order to alleviate overfitting, we introduce a regularization item which controls the complexity of our ensemble model to the objective function concurrently. Then the derivation formula aimed at solving output weights of the newly added ELM is determined by optimizing the objective using second-order approximation. As the output weights of the newly added ELM at each iteration are hard to be analytically calculated based on the derivation formula, we take the output weights learned by the pseudo-residuals-based training dataset as an initial heuristic item and thus obtain the optimal output weights by using the derivation formula to update the heuristic item iteratively. Because the regularized objective tends to employ not only predictive but also simple functions and meanwhile a randomly selected subset rather than the whole training set is used to minimize training residuals at each iteration, SGB-ELM can continually improve the generalization capability of ELM while effectively avoiding overfitting. The experimental results in comparison with Bagging ELM, Boosting ELM, EN-ELM, and V-ELM show that SGB-ELM obtains better classification and regression performances, which demonstrates the feasibility and effectiveness of SGB-ELM algorithm.The rest of this paper is organized as follows. In Section2, we briefly summarize the basic ELM model as well as the stochastic gradient boosting method. Section 3 introduces our proposed SGB-ELM algorithm. Experimental results are presented in Section 4. Finally, we conclude this paper and make some discussions in Section 5.


## 2. Preliminaries

In this section, we briefly review the principles of basic ELM model and the stochastic gradient boosting method to provide necessary backgrounds for the development of SGB-ELM algorithm in Section3.


### 2.1. Extreme Learning Machine

ELM is a special learning algorithm for SLFN, which randomly selects weights (linking the input layer to the hidden layer) and biases for hidden nodes and analytically determines the output weights (linking the hidden layer to the output layer) by using MP generalized inverse. Suppose we have a training dataset withN instances D=xi,yii=1N, where xi=xi1,xi2,⋯,xid∈Rd and yi=yi1,yi2,⋯,yim∈Rm. It is known that m=1 for regression and m>1 for classification. In ELM, the input weights and hidden biases can be randomly chosen according toany continuous probability distribution [2]. Namely, we randomly select the learning parameters within the range of -1,1 as(1)W=w1w2⋮wLT=w11w12⋯w1Lw21w22⋯w2L⋮⋮⋱⋮wd1wd2⋯wdLd×Land(2)B=b1,b2,⋯,bLT,where L is the number of hidden-layer nodes in SLFN. Depending on the theory proved in [2], the output-layer weights in ELM model can be analytically calculated by(3)β=H†Y.Here, H† is the MP generalized inverse of the hidden-layer output matrix(4)H=gxiwl+blN×L,where i=1,2,⋯,N,  l=1,2,⋯,L, and gu=1/1+exp⁡-u is the sigmoid activation function, and(5)Y=y1y2⋮yN=y11y12⋯y1my21y22⋯y2m⋮⋮⋱⋮yN1yN2⋯yNmN×mis the target matrix. Generally, for an unseen instance x^=x^1,x^2,⋯,x^d, ELM predicts its output y^ as follows:(6)y^=hx^β,where hx^=gx^w1+b1,⋯,gx^wL+bL is the hidden-layer output vector of x^.Due to avoiding the iterative adjustment to input-layer weights and hidden biases, ELM’s training speed can be thousands of times faster than those of traditional gradient-based learning algorithms [2]. At the meantime, ELM also produces good generalization performance. It has been verified that ELM can achieve the equal generalization performance with the typical Support Vector Machine algorithm [3].


### 2.2. Stochastic Gradient Boosting

Stochastic gradient boosting scheme was proposed by Friedman in [20], and it is a variant of the gradient boosting method presented in [19]. Given a training set xi,yii=1N, the goal is to learn a hypothesis FKx that maps x to y and minimizes the training loss as follows:(7)FKx=argminFKx∑i=1NLyi,FKxi,where L·,· is the loss function which evaluates the difference between the predicted value and the target and K denotes the number of iterations. In boosting mechanism, K additive individual learners are trained sequentially by(8)fkx=argminfkx∑i=1NLyi,Fk-1xi+fkxiand(9)Fkx=Fk-1x+fkx,where k=1,2,⋯,K. It is shown that the optimization problem depends much on the loss function and becomes unsolvable when L·,· is complex. Creatively, gradient boosting constructs the weak individuals based on the pseudo residuals, which are the gradient of loss function with respect to the model values predicted at the current learning step. For instance, let ϵik be the pseudo residual of the ith sample at the kth iteration written as(10)ϵik=-∂Lyi,y^∂y^y^=Fk-1xi,and thus the kth weak learner Ekx is trained by(11)Ekx=argminEkx∑i=1NLϵik,Ekxi.As gradient boosting constructs additive ensemble model by sequentially fitting a weak individual learner to the current pseudo-residuals of whole training dataset at each iteration, it costs much training time and may suffer from overfitting problem. In view of that, a minor modification named stochastic gradient boosting is proposed to incorporate some randomization to the procedure. Specifically, at each iteration a randomly selected subset instead of the full training dataset is used to fit the individual learner and compute the model update for the current iteration. Namely, letπi1N be a random permutation of the integers 1,2,⋯,N, and the subset with size N~<N of the entire training dataset can be given by xπi,yπii=1N~. Furthermore, the kth weak learner using the stochastic gradient boosting ensemble scheme is trained by solving the following optimization problem as(12)Ek∗x=argminEk∗x∑i=1N~Lϵπik,Ek∗xπi.Given the base learner F0x which is trained by the initial training dataset, the final ensemble learning model constructed by stochastic gradient boosting scheme predicts an unknown testing instance x^ as follows:(13)FKx^=F0x^+∑k=1KEk∗x^.Stochastic gradient boosting is also considered as a special linear search optimization algorithm, which makes the newly added individual learner fit the fastest descent direction of partial training loss at each learning step.


## 2.1. Extreme Learning Machine

ELM is a special learning algorithm for SLFN, which randomly selects weights (linking the input layer to the hidden layer) and biases for hidden nodes and analytically determines the output weights (linking the hidden layer to the output layer) by using MP generalized inverse. Suppose we have a training dataset withN instances D=xi,yii=1N, where xi=xi1,xi2,⋯,xid∈Rd and yi=yi1,yi2,⋯,yim∈Rm. It is known that m=1 for regression and m>1 for classification. In ELM, the input weights and hidden biases can be randomly chosen according toany continuous probability distribution [2]. Namely, we randomly select the learning parameters within the range of -1,1 as(1)W=w1w2⋮wLT=w11w12⋯w1Lw21w22⋯w2L⋮⋮⋱⋮wd1wd2⋯wdLd×Land(2)B=b1,b2,⋯,bLT,where L is the number of hidden-layer nodes in SLFN. Depending on the theory proved in [2], the output-layer weights in ELM model can be analytically calculated by(3)β=H†Y.Here, H† is the MP generalized inverse of the hidden-layer output matrix(4)H=gxiwl+blN×L,where i=1,2,⋯,N,  l=1,2,⋯,L, and gu=1/1+exp⁡-u is the sigmoid activation function, and(5)Y=y1y2⋮yN=y11y12⋯y1my21y22⋯y2m⋮⋮⋱⋮yN1yN2⋯yNmN×mis the target matrix. Generally, for an unseen instance x^=x^1,x^2,⋯,x^d, ELM predicts its output y^ as follows:(6)y^=hx^β,where hx^=gx^w1+b1,⋯,gx^wL+bL is the hidden-layer output vector of x^.Due to avoiding the iterative adjustment to input-layer weights and hidden biases, ELM’s training speed can be thousands of times faster than those of traditional gradient-based learning algorithms [2]. At the meantime, ELM also produces good generalization performance. It has been verified that ELM can achieve the equal generalization performance with the typical Support Vector Machine algorithm [3].


## 2.2. Stochastic Gradient Boosting

Stochastic gradient boosting scheme was proposed by Friedman in [20], and it is a variant of the gradient boosting method presented in [19]. Given a training set xi,yii=1N, the goal is to learn a hypothesis FKx that maps x to y and minimizes the training loss as follows:(7)FKx=argminFKx∑i=1NLyi,FKxi,where L·,· is the loss function which evaluates the difference between the predicted value and the target and K denotes the number of iterations. In boosting mechanism, K additive individual learners are trained sequentially by(8)fkx=argminfkx∑i=1NLyi,Fk-1xi+fkxiand(9)Fkx=Fk-1x+fkx,where k=1,2,⋯,K. It is shown that the optimization problem depends much on the loss function and becomes unsolvable when L·,· is complex. Creatively, gradient boosting constructs the weak individuals based on the pseudo residuals, which are the gradient of loss function with respect to the model values predicted at the current learning step. For instance, let ϵik be the pseudo residual of the ith sample at the kth iteration written as(10)ϵik=-∂Lyi,y^∂y^y^=Fk-1xi,and thus the kth weak learner Ekx is trained by(11)Ekx=argminEkx∑i=1NLϵik,Ekxi.As gradient boosting constructs additive ensemble model by sequentially fitting a weak individual learner to the current pseudo-residuals of whole training dataset at each iteration, it costs much training time and may suffer from overfitting problem. In view of that, a minor modification named stochastic gradient boosting is proposed to incorporate some randomization to the procedure. Specifically, at each iteration a randomly selected subset instead of the full training dataset is used to fit the individual learner and compute the model update for the current iteration. Namely, letπi1N be a random permutation of the integers 1,2,⋯,N, and the subset with size N~<N of the entire training dataset can be given by xπi,yπii=1N~. Furthermore, the kth weak learner using the stochastic gradient boosting ensemble scheme is trained by solving the following optimization problem as(12)Ek∗x=argminEk∗x∑i=1N~Lϵπik,Ek∗xπi.Given the base learner F0x which is trained by the initial training dataset, the final ensemble learning model constructed by stochastic gradient boosting scheme predicts an unknown testing instance x^ as follows:(13)FKx^=F0x^+∑k=1KEk∗x^.Stochastic gradient boosting is also considered as a special linear search optimization algorithm, which makes the newly added individual learner fit the fastest descent direction of partial training loss at each learning step.


## 3. Stochastic Gradient Boosting-Based Extreme Learning Machine (SGB-ELM)

SGB-ELM is a novel hybrid learning algorithm, which introduces the stochastic gradient boosting method into ELM ensemble procedure. As boosting mechanism focuses on gradually reducing the training residuals at each iteration and ELM is a special multiparameters network (for classification tasks particularly), instead of combining the ELM and stochastic gradient boosting primitively, we design an enhanced training scheme to alleviate possible overfitting in our proposed SGB-ELM algorithm. The detailed implementation of SGB-ELM is presented in Algorithm2, where the determination of optimal output weights for each individual ELM learner is illustrated in Algorithm 1 accordingly.Algorithm 1:The determination of βk-Optimal.
Input: βk –– Heuristic item for the optimal output weights; Hk –– The hidden-layer output matrix; λ –– Regularization factor, λ>0; uπik∣i=1,2,⋯,N~ –– First order gradient statistics; vπik∣i=1,2,⋯,N~ –– Second order gradient statistics; ( 1 ) Let Uk=uπ1k,uπ2k,⋯,uπN~kT=uijkN~×m; ( 2 ) Let Vk=vπ1k,vπ2k,⋯,vπN~kT=vijkN~×m; ( 3 ) for j = 1 to mdo ( 4 )  forl=1 to Ldo ( 5 )   β^ljk=-∑i=1N~‍uijkhilk+vijkhilk∑s=1,s≠lL‍hiskβsjk∑i=1N~‍vijkhilk2+λN~ ( 6 )   βljk=β^ljk ( 7 )  end for ( 8 ) end for Output: The optimal output weightsβk-Optimal=β^kAlgorithm 2:SGB-ELM.
Input: D0 –– Initial training dataset xi,yii=1N; N~ –– Size of random subset at each iteration; L –– Number of hidden nodes; K –– Number of iterations; L⋅,⋅ ––Loss function which is twice differentiable; λ –– Regularization factor, λ>0. ( 1 ) Use D0 to train the initial ELM0 written as F0x, where the input weightsW0 and hidden biases B0 are randomly selected within the range of-1,1 and the output-layer weights β0 are determined analytically byβ0=H0†Y, and record the initial base learnerF0=W0;B0;β0; ( 2 ) for k = 1 ; k ≤ K ; k + + do ( 3 )  Randomly generate a permutation πi1N of the integers 1,2,⋯,N, and then a stochastic subset of the whole training dataset is defined asxπi,yπii=1N~; ( 4 )  Calculate the first order gradient statistics on the loss function with regard to the predicted output of the current ensemble model for each training instance in the subset as uπik=∂Lyπi,y^∂y^y^=Fk-1xπi,i=1,2,⋯,N~; ( 5 )  Calculate the second order gradient statistics on the loss function with regard to the predicted output of current ensemble model for each training instance in the subset as vπik=∂L2yπi,y^∂y^2y^=Fk-1xπi,i=1,2,⋯,N~; ( 6 )  For the training instances in the subset, compute the current pseudo residualsEk=ϵπ1k,ϵπ2k,⋯,ϵπN~kT, where ϵπik=-∂Lyπi,y^∂y^y^=Fk-1xπi,i=1,2,⋯,N~; ( 7 )  Determine the output weights βk used as a heuristic item for the derivation formula based on the modified training datasetDk=xπi,επiki=1N~ as follows βk=Hk†Ek,    (∗) whereHk is calculated according to the randomly selected input weightsWk and hidden biases Bk; ( 8 )  Use the derivation formula in Algorithm 1 to obtain the optimal output-layer weightsβk∗ of ELMk; ( 9 )  Add the k-th individual learner ELMk (Ek=Wk;Bk;βk∗) to the current ensemble learning model as Fkx=Fk-1x+Ekx; ( 10 ) end for output: The final ensemble modelFKx=F0x+∑k=1K‍Ekx.There are many existing second-order approximation methods including sequential quadratic programming (SQP) [21] and majorization-minimization algorithm (MM) [22]. SQP is an effective method for nonlinearly constrained optimization by solving quadratic subproblems. MM aims to optimize the local alternative objective which is easier to solve in comparison with the original cost function. Instead of using second-order approximation directly, SGB-ELM designs an optimization criterion for the output-layer weights of each individual ELM. In view of that, quadratic approximation is merely employed as an optimization tool in SGB-ELM.In SGB-ELM, the key issue is to determine the optimal output-layer weights of each weak individual ELM, which is expected to further decrease the training loss and meanwhile keep a simple network structure. Consequently, we design a learning objective considering not only the fitting ability for training instances but also the complexity of our ensemble model as follows:(14)QK=∑i=1NLyi,y^i+λ∑k=1KΩEk,where L·,· is a differentiable loss function that measures the difference between the predicted output y^i and the target value yi. The second term Ω represents the complexity of the ensemble model consisting of K weak individual learners. Moreover, λ is a regularization factor that makes a balance between training loss and architectural risk. It is obvious that the objective falls back to the traditional gradient booting method when the regularization factor λ is set to zero.As for boosting training mechanism, each individual ELM is greedily added to the current ensemble model sequentially so that it can most improve our model according to (8). Specifically, let y^ik-1 be the predicted value of the ith instance at the k-1th iteration and Ekx be the kth weak ELM learner that needs to be incorporated into the ensemble model, then the prediction of the ith instance at the kth iteration y^ik can be written as(15)y^ik=y^ik-1+Ekxi,k=1,2,⋯,K.In order to obtain the newly added individual ELM, we first introduce Ekx to the existing learned ensemble model and then minimize the following objective:(16)Qk=∑i=1NLyi,y^ik+λ∑j=1kΩEj=∑i=1NLyi,y^ik-1+Ekxi+λ∑j=1k-1ΩEj+ΩEk,where E1,E2,⋯,Ek-1 is already obtained at the previous iterations. As a consequence, the complexity of the learned ensemble model ∑j=1k-1ΩEj is a constant, and we only need to take ΩEk into consideration. Removing the constant item, the objective Qk is simplified as(17)Qk=∑i=1NLyi,y^ik-1+Ekxi+λΩEk.Stochastic gradient boosting selects a random subset with size N~<N of the whole training set to fit the individual learner at each iteration. Namely, let πi1N be a random permutation of the integers 1,2,⋯,N, then we can define a stochastic subset as xπi,yπii=1N~. Accordingly, the objective using stochastic gradient boosting is transformed as(18)Qk=∑i=1N~Lyπi,y^πik-1+Ekxπi+λΩEk.We use second-order approximation to optimize the above learning objective, where the lose function is derived by Taylor expansion as follows:(19)Lk≃Lyj,y^jk-1+ujkTEkxj+12vjkTEk2xj,where j=πi is the new index for xπi,yπi in the randomly generated subset,(20)ujk=∂Lyπi,y^∂y^y^=Fk-1xπi,i=1,2,⋯,N~is the first-order gradient statistics on the loss function with respect to the current predicted output Fk-1xπi, and(21)vjk=∂L2yπi,y^∂y^2y^=Fk-1xπi,i=1,2,⋯,N~is the second-order gradient statistics on the loss function with respect to the current predicted output Fk-1xπi. Due to the approximation for training loss, we can provide a general solution scheme regardless of the specific type of loss function. In addition, second-order optimization tends to achieve better convergence in comparison with the traditional gradient method [23]. Obviously, Lyj,y^jk-1 is a fixed value, and thus the objective can be further expressed as(22)Qk=∑j=1N~ujkTEkxj+12vjkTEk2xj+λΩEk.Let X;Y=xπi,yπii=1N~, and the objective can be rewritten in a matrix form as(23)Qk=TrUkTEkX+12TrVkTEk2X+λΩEk,where(24)Uk=uπ1kuπ2k⋮uπN~k=u11ku12k⋯u1mku21ku22k⋯u2mk⋮⋮⋱⋮uN~1kuN~2k⋯uN~mkN~×mand(25)Vk=vπ1kvπ2k⋮vπN~k=v11kv12k⋯v1mkv21kv22k⋯v2mk⋮⋮⋱⋮vN~1kvN~2k⋯vN~mkN~×m.The kth individual learner EkX is a basic ELM model, which randomly selects input-layer weights Wk and hidden biases Bk. Given the hidden-layer output matrix(26)Hk=gXWk+Bk=h11kh12k⋯h1Lkh21kh22k⋯h2Lk⋮⋮⋱⋮hN~1khN~2k⋯hN~LkN~×L,EkX can be expressed as(27)EkX=Hkβk,where(28)βk=β11kβ12k⋯β1mkβ21kβ22k⋯β2mk⋮⋮⋱⋮βL1kβL2k⋯βLmkL×mis the output-layer weight matrix that needs to be determined. As Bartlett [24] pointed out that networks tend to perform better generalization with not only small training error but also small norm of weights (βk), we use L2-norm to evaluate the complexity of a basic ELM model as(29)ΩEk=12βk2.Accordingly, the conclusive objective can be written as(30)Qk=TrUkTHkβk+12TrVkTHkβk2+λ2βk2. From (30), we can find that the objective is only sensitive to βk at the kth iteration. For single-variable optimization, solving partial derivative is conducted as(31)∂Qk∂βk=0,where each element in βk conducted a partial derivative, respectively. Thus we obtain the derivation formula as follows:(32)βljk=-∑i=1N~uijkhilk+vijkhilk∑s=1,s≠lLhiskβsjk∑i=1N~vijkhilk2+λN~,where l=1,2,⋯,L and j=1,2,⋯,m. It is shown that βk is difficult to be calculated analytically. Since our designed regularized objective tends to generate an ensemble model employing predictive as well as simple hypotheses, (32) derived by the objective can be used as an optimization criterion. Specifically, we take the output-layer weights determined by pseudo-residuals dataset as an initial heuristic item and thus obtain the optimal output-layer weights by using the derivation formula to update the heuristic item iteratively. Algorithm 1 illustrates how the optimal output weight matrix is determined and the detailed implementation of SGB-ELM is presented in Algorithm 2.In Algorithm2, all the input weights and hidden biases of individual ELMs are randomly chosen within the range of -1,1. For boosting-based ensemble methods, the initial base learner is expected to be enhanced by adding weak individual learners to the current ensemble model step by step. In view of that, high-precision initial base learner might affect the effectiveness of ensemble negatively. In order to control the fitting ability of the initial base learner F0x and meanwhile reduce the instability brought by random determination of the input weights and hidden biases, SGB-ELM conducts multiple random initializations for parameters in ELM0 and takes the average at last. For instance, we take the average of 100 random initializations as(33)wil=∑t=1100wilt100,bl=∑t=1100blt100,where i=1,2,⋯,d and l=1,2,⋯,L. For the weak individual ELM, which plays a smaller role in the whole ensemble model, random initialization of parameters exactly increases the diversity between weak individual learners.


## 4. Performance Validation

In this section, a series of experiments are conducted to validate the feasibility and effectiveness of our proposed SGB-ELM algorithm, and meanwhile we compare the generalization performance and predicted stability of several typical ensemble learning methods (EN-ELM [5], V-ELM [6], Bagging [14], and Adaboost [15]) on 4 KEEL [25] regression and 5 UCI [26] classification datasets. Among all the above-mentioned ensemble methods, the basic ELM model proposed in [2] is used as the individual learner, where the sigmoid function gx=1/1+exp⁡-x is selected as the activation function. All the experiments are carried out on a desktop computer with Win10 operating system, Intel (R) i5-4590 3.30GHz CPU, and 12GB memory and implemented with Matlab 9.0 version. Meanwhile, all the experimental results are the average of 50 repeated trials. The experiments are generally divided into two parts: one part is to evaluate the performance of SGB-ELM, and the other part is to measure the effect of learning parameters on training SGB-ELM algorithm.


### 4.1. Performance Evaluation of SGB-ELM

For regression problem, the performances of SGB-ELM and other comparative algorithms are both measured by Root Mean Square Error (RMSE), which reveals the difference between the predicted value and the target. Additionally, in this paper, we take the squared loss as our loss function in SGB-ELM algorithm for regression task. Supposey^i and yi are the predicted value and the target of the ith instance, respectively, and the loss function L·,· is given by(34)Lyi,y^i=12y^i-yi2.Since V-ELM and EN-ELM are designed for classification applications, we compare the generalization capability of SGB-ELM with the basic ELM, simple ensemble ELM, Bagging ELM, and Adaboost ELM in regression tasks. Among them, simple ensemble ELM can be considered as a variant of the V-ELM method, which trains a number of individual ELMs independently and takes the simple average of all the predictions at last. Adaboost ELM is implemented by Adaboost.R2 method [27], which applies the primitive Adaboost algorithm designed for classification tasks [15] to the regression field. Furthermore, we adopt resampling the original training dataset rather than assigning a weight to every instance to train each individual learner in Adaboost.R2 ELM.The performances of the traditional ELM, simple ensemble ELM, Bagging ELM, Adaboost ELM, and our proposed SGB-ELM are compared on 4 representative regression datasets, which are selected from the KEEL [25] repository. Experimentally, all the inputs of each dataset are normalized into the range of 0,1. The characteristics of these datasets are summarized in Table 1, where each original dataset is divided into two groups including a training set (70%) and a testing set (30%). In our regression experiments, for each dataset, the number of hidden nodes L is selected from 10,20,⋯,200. The parameters in SGB-ELM are set as λ=0.001,0.0015,⋯,0.01,  N~/N=0.1,0.2,⋯,1, and K=50. The settings of other comparative algorithms can be found in Table 2. Figure 1 shows the training and testing RMSE of different learning methods during 50 trials on Friedman dataset. The detailed comparison results between SGB-ELM and other learning algorithms on 4 regression benchmark datasets are shown in Table 2. Furthermore, we compare the training and testing performances of SGB-ELM with those of Adaboost.R2 with regard to the number of iterations on Mortgage dataset, which is presented in Figure 3(a).Table 1
Details of 4 KEEL regression datasets.
No. Datasets Condition attributes Training samples Testing samples 1 Laser 4 695 298 2 Friedman 5 840 360 3 Mortgage 15 734 315 4 Wizmir 9 1023 438Table 2
The comparison results between SGB-ELM and other representative algorithms on 4 regression datasets.
Dataset Algorithm Training time Training RMSE (Dev) Testing RMSE (Dev) Hidden nodes Iterations Laser(N~/N=0.4) ELM 0.0097 12.3791± 0.6067 12.7783± 1.5863 80 N/A Simple ensemble 0.1547 12.1216± 0.4585 12.8794± 1.5415 80 10 Bagging 0.7991 12.1707± 0.5094 13.4085± 1.5998 80 50 Adaboost 0.1591 11.4460± 0.4875 12.0666± 1.2080 50 Max = 50 SGB-ELM 3.4853 7.6354± 0.3919 8.4170 ± 1.0788 50 (λ=0.001) 50  Friedman(N~/N=0.5) ELM 0.0141 1.4220± 0.0532 1.5124± 0.0906 100 N/A Simple ensemble 0.2081 1.4005± 0.0240 1.4791± 0.0747 100 10 Bagging 1.0144 1.4111± 0.0197 1.4906± 0.0701 100 50 Adaboost 0.5219 1.2551± 0.0304 1.3342± 0.0587 60 Max = 50 SGB-ELM 4.8853 1.0627± 0.0136 1.1581 ± 0.0346 60 (λ=0.002) 50  Mortgage(N~/N=0.4) ELM 0.0200 0.0855± 0.0022 0.0961± 0.0078 150 N/A Simple ensemble 0.3044 0.0843± 0.0021 0.0947± 0.0085 150 10 Bagging 1.4544 0.0834± 0.0018 0.0937± 0.0077 150 50 Adaboost 0.4778 0.0785± 0.0020 0.0885± 0.0058 80 Max = 50 SGB-ELM 6.2434 0.0607± 0.0015 0.0759 ± 0.0056 80 (λ=0.004) 50  Wizmir(N~/N=0.5) ELM 0.0128 1.0906± 0.0277 1.1263± 0.0667 100 N/A Simple ensemble 0.2066 1.0869± 0.0269 1.1203± 0.0629 100 10 Bagging 1.0366 1.0859± 0.0259 1.1165± 0.0620 100 50 Adaboost 0.4331 1.0622± 0.0519 1.1091± 0.0857 60 Max = 50 SGB-ELM 5.6525 1.0148± 0.0258 1.1032 ± 0.0615 60 (λ=0.002) 50Figure 1
The training and testing performance of ELM, simple ensemble, Bagging, Adaboost.R2, and SGB-ELM during 50 trials on the Friedman dataset.As for classification problem, like other typical feedforward neural networks (for instance, BP neural networks [28]), SGB-ELM evaluates the predicted output by calculating the sum of squared errors. Specifically, let y^i be the predicted output vector and yi be the target encoded by One-Hot scheme [29] of the ith sample, respectively, and we define the loss function L·,· in SGB-ELM for classification as follows:(35)Lyi,y^i=12y^i-yi2=12∑j=1my^ij-yij2.It is shown that SGB-ELM aims at reducing the training RMSE inch by inch for classification problem. Accordingly, we compare SGB-ELM with several representative ensemble learning methods including V-ELM, EN-ELM, Bagging ELM, and Adaboost ELM. Among them, V-ELM and EN-ELM have been briefly summarized inSection1, and Adaboost ELM is implemented by Adaboost.SAMME method [30], which extends the original Adaboost designed for binary classification to multiclassification problem.Similarly, we select 5 popular classification datasets from the UCI Machine Learning Repository [26] to verify the performance of our proposed SGB-ELM algorithm. For each dataset, all the decision attributes are encoded by One-Hot scheme [29]. The characteristics of these datasets are described in Table 3, where each original data set is equally divided into two groups including a training set (50%) and a testing set (50%). The number of hidden nodes is also selected from L=10,20,⋯,200 for each dataset. The parameters in SGB-ELM are set as λ=0.001,0.002,⋯,0.03,  N~/N=0.1,0.2,⋯,1, and K=100. The cross-validation is tenfold (R=10) in EN-ELM. The number of individual ELMs for ensemble is 7 (K=7) in V-ELM. Other settings can be found in Table 4. Figure 2 shows the training and testing accuracy of different algorithms during 50 trials on Segmentation dataset. The detailed performances of SGB-ELM in comparison with other learning algorithms on 5 classification benchmark datasets are summarized in Table 4. Lastly, the training and testing accuracy of SGB-ELM and Adaboost.SAMME with regard to the number of iterations on the Segmentation dataset are presented in Figure 3(b).Table 3
Details of 5 UCI classification datasets.
No. Datasets Condition attributes Decision attributes Training samples Testing samples 1 Image segmentation 19 7 1155 1155 2 Texture 40 11 2750 2750 3 Spambase 57 2 2295 2294 4 Banana 2 2 2650 2650 5 Ring 20 2 3700 3700Table 4
The comparison results between SGB-ELM and other representative algorithms on 5 classification datasets.
Dataset Algorithm Training time Training accuracy (Dev) Testing accuracy (Dev) Hidden nodes Iterations Segmentation(N~/N=0.5) ELM 0.0431 0.9465± 0.0055 0.9351± 0.0064 180 N/A V-ELM 0.4487 0.9463± 0.0061 0.9374± 0.0060 180 7 EN-ELM 43.9234 0.9472± 0.0048 0.9353± 0.0063 180 (R=10) 50 Bagging 3.1853 0.9474± 0.0035 0.9353± 0.0050 180 50 Adaboost 3.5372 0.9853± 0.0052 0.9466± 0.0067 100 Max = 100 SGB-ELM 134.2969 0.9761± 0.0030 0.9558 ± 0.0049 100 (λ=0.004) 100  Texture(N~/N=0.7) ELM 0.0338 0.9954± 0.0011 0.9945± 0.0016 100 N/A V-ELM 0.4275 0.9965± 0.0007 0.9950± 0.0013 100 7 EN-ELM 44.4969 0.9963± 0.0008 0.9946± 0.0011 100 (R=10) 50 Bagging 3.0959 0.9965± 0.0007 0.9957± 0.0013 100 50 Adaboost 10.3628 0.9996± 0.0017 0.9972± 0.0024 60 Max = 100 SGB-ELM 193.2019 0.9992± 0.0005 0.9982 ± 0.0008 60 (λ=0.002) 100  Spambase(N~/N=0.7) ELM 0.0459 0.9174± 0.0044 0.9080± 0.0061 150 N/A V-ELM 0.4213 0.9192± 0.0042 0.9115± 0.0053 150 7 EN-ELM 62.6000 0.9183± 0.0054 0.9071± 0.0060 150 (R=10) 50 Bagging 2.9869 0.9219± 0.0039 0.9145± 0.0051 150 50 Adaboost 7.6875 0.9620± 0.0046 0.9234 ± 0.0072 100 Max = 100 SGB-ELM 129.0922 0.9522± 0.0033 0.9222±0.0043 100 (λ=0.006) 100  Banana(N~/N=0.6) ELM 0.0550 0.6838± 0.0253 0.6787± 0.0263 180 N/A V-ELM 0.4906 0.6860± 0.0261 0.6848± 0.0264 180 7 EN-ELM 67.5578 0.6821± 0.0227 0.6780± 0.0250 180 (R=10) 50 Bagging 3.3253 0.6808± 0.0164 0.6777± 0.0178 180 50 Adaboost 7.5100 0.7457± 0.0288 0.7448± 0.0320 100 Max = 100 SGB-ELM 133.0791 0.7610± 0.0085 0.7563 ± 0.0082 100 (λ=0.004) 100  Ring(N~/N=0.8) ELM 0.0897 0.9492± 0.0024 0.9418± 0.0035 200 N/A V-ELM 0.7609 0.9532± 0.0024 0.9466± 0.0032 200 7 EN-ELM 114.0641 0.9517± 0.0028 0.9418± 0.0031 200 (R=10) 50 Bagging 5.5241 0.9539± 0.0027 0.9468± 0.0030 200 50 Adaboost 17.2109 0.9940± 0.0023 0.9524± 0.0038 150 Max = 100 SGB-ELM 363.7976 0.9750± 0.0021 0.9567 ± 0.0027 150 (λ=0.008) 100Figure 2
The training and testing performance of V-ELM, EN-ELM, Bagging, Adaboost.SAMME, and SGB-ELM during 50 trials on the Segmentation dataset.Figure 3
The training and testing RMSE/accuracy of SGB-ELM with regard to the number of iterations in comparison with Adaboost method.
(a)
On the Mortgage dataset (regression) (b)
On the Segmentation dataset (classification)Tables2 and 4 present the comparison results including training time, training RMSE/accuracy, and testing RMSE/accuracy for regression and classification tasks, respectively. It is shown that SGB-ELM obtains the better generalization capability in most cases without significantly increasing the training time. At the same time, SGB-ELM tends to have smaller training Dev and testing Dev than those of the comparative learning algorithms, which exactly validates the robustness and stability of our proposed SGB-ELM Algorithm. In particular, since SGB-ELM adopts the similar training mechanism with Adaboost which integrates multiple weak individual learners sequentially, the number of hidden nodes is set as a smaller value in both SGB-ELM and Adaboost method. It is worth noting that SGB-ELM can achieve better performance than the existing methods with less hidden nodes and outperforms Adaboost with the same number of hidden nodes.From Figures1 and 2, we can find that SGB-ELM is more stable than the traditional ELM, simple ensemble, Bagging, and Adaboost.R2 in regression problem and also produces better robustness than V-ELM, EN-ELM, Bagging, and Adaboost.SAMME in classification problem. It is shown that SGB-ELM not only focuses on reducing the predicted bias as other boosting like methods, but also generates a robust ensemble model with a low variance. As observed in Figure 2 although Adaboost.SAMME generates higher training accuracy than SGB-ELM during the most of 50 trials, SGB-ELM obtains the better generalization capability (testing accuracy). It can be explained by two reasons as (1) we introduce a regularization item (L2-norm) to the learning objective to control the complexity of our ensemble learning model; (2) a randomly selected subset rather than the whole training dataset is used to minimize the training loss at each iteration in our proposed SGB-ELM algorithm.Figure3 shows the training RMSE/accuracy and testing RMSE/accuracy of Adaboost (Adaboost.R2 for regression and Adaboost.SAMME for classification) and SGB-ELM with regard to the number of iterations. The fixed reference line denotes the training and testing performance of a traditional ELM, which is equipped with much more hidden nodes. As shown in Figure 3, SGB-ELM obviously improves the generalization capability of the initial base ELM in both regression and classification tasks. From Figure 3(a), we can find that the training and testing RMSE is declining gradually as the number of iterations increases. Similarly, both the training and testing accuracy curve show an increasing trend in Figure 3(b). Because we conduct multiple random initializations for parameters in the initial base learner F0 and take the average at last, the fitting ability of F0 is artificially weakened to some extent. As a result, the initial training and testing RMSE/accuracy of SGB-ELM are much lower than the initial Adaboost. It is shown that both SGB-ELM and Adaboost outperform the traditional ELM equipped more hidden nodes after a small number of learning steps. Furthermore, we can find that SGB-ELM produces better performance than Adaboost after only 5 iterations in regression tasks and 10 iterations in classification tasks. It verifies the significant convergence of second-order optimization method, which is incorporated into the procedure of SGB-ELM.From the experimental results of both regression and classification problems, we can conclude that our proposed SGB-ELM algorithm can not only achieve better generalization capability (low predicted bias) than the typical existing variants of ELM, but also obtain an enough robust ELM ensemble learning model (low predicted variance).


### 4.2. Impact of Learning Parameters on Training SGB-ELM

To achieve good generalization performance, three learning parameters of SGB-ELM including the number of hidden nodesL, the regularization factor λ, and the size of subset N~ need to be chosen appropriately. In this section, we attempt to evaluate the impact of learning parameters on training SGB-ELM algorithm and provide some empirical references of choosing these parameters.For the basic ELM model, the number of hidden nodesL decides the model's capacity. In other words, an ELM with more hidden nodes is more complex and can deal with more training instances. However, it tends to obtain an overfitting model when L is set as a value too large. The regularization factor λ makes a balance between the training loss and the complexity of model. It means that λ can control the capacity or the complexity of our model. The size of subset N~ represents the number of training instances at each iteration and it introduces some randomization to the training procedure of SGB-ELM. Firstly, we use grid-search method to observe the training and testing performance of SGB-ELM with different L and λ. Specifically, we set L=10,15,⋯,160,  λ=0.001,0.002,⋯,0.03, and a fixed N~=0.8. The training and testing performance of SGB-ELM with regard to the combination of L,λ on the Spambase dataset is shown in Figure 4. Secondly, as we empirically find that the optimal N~ depends much on the size of training dataset, we conduct two experiments (including a small dataset and a large dataset) to measure the impact of N~ on training SGB-ELM. We choose the optimal value of L,λ according to the grid-search results and set N~/N=0.1,0.2,⋯,1. Figure 5 shows the training and testing performance of SGB-ELM with different sampling fraction (N~/N) on the Wizmir and Spambase datasets.Figure 4
The training and testing performance of SGB-ELM with different combinations ofL,λ on the Spambase dataset.Figure 5
The training and testing performance of SGB-ELM with different sampling fraction (N~/N) on the Wizmir and Spambase datasets.
(a)
On the Wizmir dataset(N=1023) (b)
On the Spambase dataset(N=2295)As shown in Figure4, changing the value of L,λ has a significant effect on the training and testing accuracy of SGB-ELM algorithm. It is obvious that SGB-ELM with excess hidden nodes is more likely to produce overfitting when the regularization factor λ is set as a small value. It also demonstrates that SGB-ELM can effectively reduce overfitting when λ is assigned a proper value. In addition, from Figure 4 we can find that SGB-ELM achieves better performance with enough hidden nodes and a proper λ. It can be explained by the rule that although SGB-ELM with a small number of hidden nodes can avoid overfitting intuitively, meanwhile it produces a barrier to fit the current training residuals appropriately.From Figure5, it is obvious that randomization improves the performance of SGB-ELM substantially. As each weak individual ELM is learned based on randomly selected subset of the whole training dataset, it exactly increases the diversity between all the individuals. On the other hand, randomization introduces a noisy estimate of the total training loss. As a result, it slows down the convergence and even makes the learning curve fluctuate (higher variance) if N~ is too small. It is shown that the best value of the sampling fraction is approximately 50% on the Wizmir dataset and 70% on the Spambase dataset, where there are a typical improvement in testing performance comparing to no sampling at all. Since the optimal values of N~/N are different on the Wizmir and Spambase datasets, it indicates that the sampling fraction (N~/N) is expected to be determined based on the specific learning tasks and assigned a bigger value on the training dataset containing more instances.


## 4.1. Performance Evaluation of SGB-ELM

For regression problem, the performances of SGB-ELM and other comparative algorithms are both measured by Root Mean Square Error (RMSE), which reveals the difference between the predicted value and the target. Additionally, in this paper, we take the squared loss as our loss function in SGB-ELM algorithm for regression task. Supposey^i and yi are the predicted value and the target of the ith instance, respectively, and the loss function L·,· is given by(34)Lyi,y^i=12y^i-yi2.Since V-ELM and EN-ELM are designed for classification applications, we compare the generalization capability of SGB-ELM with the basic ELM, simple ensemble ELM, Bagging ELM, and Adaboost ELM in regression tasks. Among them, simple ensemble ELM can be considered as a variant of the V-ELM method, which trains a number of individual ELMs independently and takes the simple average of all the predictions at last. Adaboost ELM is implemented by Adaboost.R2 method [27], which applies the primitive Adaboost algorithm designed for classification tasks [15] to the regression field. Furthermore, we adopt resampling the original training dataset rather than assigning a weight to every instance to train each individual learner in Adaboost.R2 ELM.The performances of the traditional ELM, simple ensemble ELM, Bagging ELM, Adaboost ELM, and our proposed SGB-ELM are compared on 4 representative regression datasets, which are selected from the KEEL [25] repository. Experimentally, all the inputs of each dataset are normalized into the range of 0,1. The characteristics of these datasets are summarized in Table 1, where each original dataset is divided into two groups including a training set (70%) and a testing set (30%). In our regression experiments, for each dataset, the number of hidden nodes L is selected from 10,20,⋯,200. The parameters in SGB-ELM are set as λ=0.001,0.0015,⋯,0.01,  N~/N=0.1,0.2,⋯,1, and K=50. The settings of other comparative algorithms can be found in Table 2. Figure 1 shows the training and testing RMSE of different learning methods during 50 trials on Friedman dataset. The detailed comparison results between SGB-ELM and other learning algorithms on 4 regression benchmark datasets are shown in Table 2. Furthermore, we compare the training and testing performances of SGB-ELM with those of Adaboost.R2 with regard to the number of iterations on Mortgage dataset, which is presented in Figure 3(a).Table 1
Details of 4 KEEL regression datasets.
No. Datasets Condition attributes Training samples Testing samples 1 Laser 4 695 298 2 Friedman 5 840 360 3 Mortgage 15 734 315 4 Wizmir 9 1023 438Table 2
The comparison results between SGB-ELM and other representative algorithms on 4 regression datasets.
Dataset Algorithm Training time Training RMSE (Dev) Testing RMSE (Dev) Hidden nodes Iterations Laser(N~/N=0.4) ELM 0.0097 12.3791± 0.6067 12.7783± 1.5863 80 N/A Simple ensemble 0.1547 12.1216± 0.4585 12.8794± 1.5415 80 10 Bagging 0.7991 12.1707± 0.5094 13.4085± 1.5998 80 50 Adaboost 0.1591 11.4460± 0.4875 12.0666± 1.2080 50 Max = 50 SGB-ELM 3.4853 7.6354± 0.3919 8.4170 ± 1.0788 50 (λ=0.001) 50  Friedman(N~/N=0.5) ELM 0.0141 1.4220± 0.0532 1.5124± 0.0906 100 N/A Simple ensemble 0.2081 1.4005± 0.0240 1.4791± 0.0747 100 10 Bagging 1.0144 1.4111± 0.0197 1.4906± 0.0701 100 50 Adaboost 0.5219 1.2551± 0.0304 1.3342± 0.0587 60 Max = 50 SGB-ELM 4.8853 1.0627± 0.0136 1.1581 ± 0.0346 60 (λ=0.002) 50  Mortgage(N~/N=0.4) ELM 0.0200 0.0855± 0.0022 0.0961± 0.0078 150 N/A Simple ensemble 0.3044 0.0843± 0.0021 0.0947± 0.0085 150 10 Bagging 1.4544 0.0834± 0.0018 0.0937± 0.0077 150 50 Adaboost 0.4778 0.0785± 0.0020 0.0885± 0.0058 80 Max = 50 SGB-ELM 6.2434 0.0607± 0.0015 0.0759 ± 0.0056 80 (λ=0.004) 50  Wizmir(N~/N=0.5) ELM 0.0128 1.0906± 0.0277 1.1263± 0.0667 100 N/A Simple ensemble 0.2066 1.0869± 0.0269 1.1203± 0.0629 100 10 Bagging 1.0366 1.0859± 0.0259 1.1165± 0.0620 100 50 Adaboost 0.4331 1.0622± 0.0519 1.1091± 0.0857 60 Max = 50 SGB-ELM 5.6525 1.0148± 0.0258 1.1032 ± 0.0615 60 (λ=0.002) 50Figure 1
The training and testing performance of ELM, simple ensemble, Bagging, Adaboost.R2, and SGB-ELM during 50 trials on the Friedman dataset.As for classification problem, like other typical feedforward neural networks (for instance, BP neural networks [28]), SGB-ELM evaluates the predicted output by calculating the sum of squared errors. Specifically, let y^i be the predicted output vector and yi be the target encoded by One-Hot scheme [29] of the ith sample, respectively, and we define the loss function L·,· in SGB-ELM for classification as follows:(35)Lyi,y^i=12y^i-yi2=12∑j=1my^ij-yij2.It is shown that SGB-ELM aims at reducing the training RMSE inch by inch for classification problem. Accordingly, we compare SGB-ELM with several representative ensemble learning methods including V-ELM, EN-ELM, Bagging ELM, and Adaboost ELM. Among them, V-ELM and EN-ELM have been briefly summarized inSection1, and Adaboost ELM is implemented by Adaboost.SAMME method [30], which extends the original Adaboost designed for binary classification to multiclassification problem.Similarly, we select 5 popular classification datasets from the UCI Machine Learning Repository [26] to verify the performance of our proposed SGB-ELM algorithm. For each dataset, all the decision attributes are encoded by One-Hot scheme [29]. The characteristics of these datasets are described in Table 3, where each original data set is equally divided into two groups including a training set (50%) and a testing set (50%). The number of hidden nodes is also selected from L=10,20,⋯,200 for each dataset. The parameters in SGB-ELM are set as λ=0.001,0.002,⋯,0.03,  N~/N=0.1,0.2,⋯,1, and K=100. The cross-validation is tenfold (R=10) in EN-ELM. The number of individual ELMs for ensemble is 7 (K=7) in V-ELM. Other settings can be found in Table 4. Figure 2 shows the training and testing accuracy of different algorithms during 50 trials on Segmentation dataset. The detailed performances of SGB-ELM in comparison with other learning algorithms on 5 classification benchmark datasets are summarized in Table 4. Lastly, the training and testing accuracy of SGB-ELM and Adaboost.SAMME with regard to the number of iterations on the Segmentation dataset are presented in Figure 3(b).Table 3
Details of 5 UCI classification datasets.
No. Datasets Condition attributes Decision attributes Training samples Testing samples 1 Image segmentation 19 7 1155 1155 2 Texture 40 11 2750 2750 3 Spambase 57 2 2295 2294 4 Banana 2 2 2650 2650 5 Ring 20 2 3700 3700Table 4
The comparison results between SGB-ELM and other representative algorithms on 5 classification datasets.
Dataset Algorithm Training time Training accuracy (Dev) Testing accuracy (Dev) Hidden nodes Iterations Segmentation(N~/N=0.5) ELM 0.0431 0.9465± 0.0055 0.9351± 0.0064 180 N/A V-ELM 0.4487 0.9463± 0.0061 0.9374± 0.0060 180 7 EN-ELM 43.9234 0.9472± 0.0048 0.9353± 0.0063 180 (R=10) 50 Bagging 3.1853 0.9474± 0.0035 0.9353± 0.0050 180 50 Adaboost 3.5372 0.9853± 0.0052 0.9466± 0.0067 100 Max = 100 SGB-ELM 134.2969 0.9761± 0.0030 0.9558 ± 0.0049 100 (λ=0.004) 100  Texture(N~/N=0.7) ELM 0.0338 0.9954± 0.0011 0.9945± 0.0016 100 N/A V-ELM 0.4275 0.9965± 0.0007 0.9950± 0.0013 100 7 EN-ELM 44.4969 0.9963± 0.0008 0.9946± 0.0011 100 (R=10) 50 Bagging 3.0959 0.9965± 0.0007 0.9957± 0.0013 100 50 Adaboost 10.3628 0.9996± 0.0017 0.9972± 0.0024 60 Max = 100 SGB-ELM 193.2019 0.9992± 0.0005 0.9982 ± 0.0008 60 (λ=0.002) 100  Spambase(N~/N=0.7) ELM 0.0459 0.9174± 0.0044 0.9080± 0.0061 150 N/A V-ELM 0.4213 0.9192± 0.0042 0.9115± 0.0053 150 7 EN-ELM 62.6000 0.9183± 0.0054 0.9071± 0.0060 150 (R=10) 50 Bagging 2.9869 0.9219± 0.0039 0.9145± 0.0051 150 50 Adaboost 7.6875 0.9620± 0.0046 0.9234 ± 0.0072 100 Max = 100 SGB-ELM 129.0922 0.9522± 0.0033 0.9222±0.0043 100 (λ=0.006) 100  Banana(N~/N=0.6) ELM 0.0550 0.6838± 0.0253 0.6787± 0.0263 180 N/A V-ELM 0.4906 0.6860± 0.0261 0.6848± 0.0264 180 7 EN-ELM 67.5578 0.6821± 0.0227 0.6780± 0.0250 180 (R=10) 50 Bagging 3.3253 0.6808± 0.0164 0.6777± 0.0178 180 50 Adaboost 7.5100 0.7457± 0.0288 0.7448± 0.0320 100 Max = 100 SGB-ELM 133.0791 0.7610± 0.0085 0.7563 ± 0.0082 100 (λ=0.004) 100  Ring(N~/N=0.8) ELM 0.0897 0.9492± 0.0024 0.9418± 0.0035 200 N/A V-ELM 0.7609 0.9532± 0.0024 0.9466± 0.0032 200 7 EN-ELM 114.0641 0.9517± 0.0028 0.9418± 0.0031 200 (R=10) 50 Bagging 5.5241 0.9539± 0.0027 0.9468± 0.0030 200 50 Adaboost 17.2109 0.9940± 0.0023 0.9524± 0.0038 150 Max = 100 SGB-ELM 363.7976 0.9750± 0.0021 0.9567 ± 0.0027 150 (λ=0.008) 100Figure 2
The training and testing performance of V-ELM, EN-ELM, Bagging, Adaboost.SAMME, and SGB-ELM during 50 trials on the Segmentation dataset.Figure 3
The training and testing RMSE/accuracy of SGB-ELM with regard to the number of iterations in comparison with Adaboost method.
(a)
On the Mortgage dataset (regression) (b)
On the Segmentation dataset (classification)Tables2 and 4 present the comparison results including training time, training RMSE/accuracy, and testing RMSE/accuracy for regression and classification tasks, respectively. It is shown that SGB-ELM obtains the better generalization capability in most cases without significantly increasing the training time. At the same time, SGB-ELM tends to have smaller training Dev and testing Dev than those of the comparative learning algorithms, which exactly validates the robustness and stability of our proposed SGB-ELM Algorithm. In particular, since SGB-ELM adopts the similar training mechanism with Adaboost which integrates multiple weak individual learners sequentially, the number of hidden nodes is set as a smaller value in both SGB-ELM and Adaboost method. It is worth noting that SGB-ELM can achieve better performance than the existing methods with less hidden nodes and outperforms Adaboost with the same number of hidden nodes.From Figures1 and 2, we can find that SGB-ELM is more stable than the traditional ELM, simple ensemble, Bagging, and Adaboost.R2 in regression problem and also produces better robustness than V-ELM, EN-ELM, Bagging, and Adaboost.SAMME in classification problem. It is shown that SGB-ELM not only focuses on reducing the predicted bias as other boosting like methods, but also generates a robust ensemble model with a low variance. As observed in Figure 2 although Adaboost.SAMME generates higher training accuracy than SGB-ELM during the most of 50 trials, SGB-ELM obtains the better generalization capability (testing accuracy). It can be explained by two reasons as (1) we introduce a regularization item (L2-norm) to the learning objective to control the complexity of our ensemble learning model; (2) a randomly selected subset rather than the whole training dataset is used to minimize the training loss at each iteration in our proposed SGB-ELM algorithm.Figure3 shows the training RMSE/accuracy and testing RMSE/accuracy of Adaboost (Adaboost.R2 for regression and Adaboost.SAMME for classification) and SGB-ELM with regard to the number of iterations. The fixed reference line denotes the training and testing performance of a traditional ELM, which is equipped with much more hidden nodes. As shown in Figure 3, SGB-ELM obviously improves the generalization capability of the initial base ELM in both regression and classification tasks. From Figure 3(a), we can find that the training and testing RMSE is declining gradually as the number of iterations increases. Similarly, both the training and testing accuracy curve show an increasing trend in Figure 3(b). Because we conduct multiple random initializations for parameters in the initial base learner F0 and take the average at last, the fitting ability of F0 is artificially weakened to some extent. As a result, the initial training and testing RMSE/accuracy of SGB-ELM are much lower than the initial Adaboost. It is shown that both SGB-ELM and Adaboost outperform the traditional ELM equipped more hidden nodes after a small number of learning steps. Furthermore, we can find that SGB-ELM produces better performance than Adaboost after only 5 iterations in regression tasks and 10 iterations in classification tasks. It verifies the significant convergence of second-order optimization method, which is incorporated into the procedure of SGB-ELM.From the experimental results of both regression and classification problems, we can conclude that our proposed SGB-ELM algorithm can not only achieve better generalization capability (low predicted bias) than the typical existing variants of ELM, but also obtain an enough robust ELM ensemble learning model (low predicted variance).


## 4.2. Impact of Learning Parameters on Training SGB-ELM

To achieve good generalization performance, three learning parameters of SGB-ELM including the number of hidden nodesL, the regularization factor λ, and the size of subset N~ need to be chosen appropriately. In this section, we attempt to evaluate the impact of learning parameters on training SGB-ELM algorithm and provide some empirical references of choosing these parameters.For the basic ELM model, the number of hidden nodesL decides the model's capacity. In other words, an ELM with more hidden nodes is more complex and can deal with more training instances. However, it tends to obtain an overfitting model when L is set as a value too large. The regularization factor λ makes a balance between the training loss and the complexity of model. It means that λ can control the capacity or the complexity of our model. The size of subset N~ represents the number of training instances at each iteration and it introduces some randomization to the training procedure of SGB-ELM. Firstly, we use grid-search method to observe the training and testing performance of SGB-ELM with different L and λ. Specifically, we set L=10,15,⋯,160,  λ=0.001,0.002,⋯,0.03, and a fixed N~=0.8. The training and testing performance of SGB-ELM with regard to the combination of L,λ on the Spambase dataset is shown in Figure 4. Secondly, as we empirically find that the optimal N~ depends much on the size of training dataset, we conduct two experiments (including a small dataset and a large dataset) to measure the impact of N~ on training SGB-ELM. We choose the optimal value of L,λ according to the grid-search results and set N~/N=0.1,0.2,⋯,1. Figure 5 shows the training and testing performance of SGB-ELM with different sampling fraction (N~/N) on the Wizmir and Spambase datasets.Figure 4
The training and testing performance of SGB-ELM with different combinations ofL,λ on the Spambase dataset.Figure 5
The training and testing performance of SGB-ELM with different sampling fraction (N~/N) on the Wizmir and Spambase datasets.
(a)
On the Wizmir dataset(N=1023) (b)
On the Spambase dataset(N=2295)As shown in Figure4, changing the value of L,λ has a significant effect on the training and testing accuracy of SGB-ELM algorithm. It is obvious that SGB-ELM with excess hidden nodes is more likely to produce overfitting when the regularization factor λ is set as a small value. It also demonstrates that SGB-ELM can effectively reduce overfitting when λ is assigned a proper value. In addition, from Figure 4 we can find that SGB-ELM achieves better performance with enough hidden nodes and a proper λ. It can be explained by the rule that although SGB-ELM with a small number of hidden nodes can avoid overfitting intuitively, meanwhile it produces a barrier to fit the current training residuals appropriately.From Figure5, it is obvious that randomization improves the performance of SGB-ELM substantially. As each weak individual ELM is learned based on randomly selected subset of the whole training dataset, it exactly increases the diversity between all the individuals. On the other hand, randomization introduces a noisy estimate of the total training loss. As a result, it slows down the convergence and even makes the learning curve fluctuate (higher variance) if N~ is too small. It is shown that the best value of the sampling fraction is approximately 50% on the Wizmir dataset and 70% on the Spambase dataset, where there are a typical improvement in testing performance comparing to no sampling at all. Since the optimal values of N~/N are different on the Wizmir and Spambase datasets, it indicates that the sampling fraction (N~/N) is expected to be determined based on the specific learning tasks and assigned a bigger value on the training dataset containing more instances.


## 5. Conclusions

In this paper, we proposed a novel ensemble model named Stochastic Gradient Boosting-based Extreme Learning Machine (SGB-ELM). Instead of combining ELM and stochastic gradient boosting primitively, we construct an ELM flow or ELM sequence where the output-layer weights of each weak ELM are determined by optimizing the regularized objective additively. Firstly, by minimizing the objective using second-order approximation, the derivation formula aimed at solving the output-layer weights of each individual ELM is determined. Then we take the output-layer weights learned by the current pseudo residuals as a heuristic item and thus obtain the optimal output-layer weights by updating the heuristic item iteratively. The performance of SGB-ELM was evaluated on 4 regression and 5 classification datasets. In comparison with several typical ELM ensemble methods, SGB-ELM obtained better performance and robustness, which demonstrated the feasibility and effectiveness of SGB-ELM algorithm.



---

*Source: 4058403-2018-06-26.xml*