# Hybrid Ensemble Pruning Using Coevolution Binary Glowworm Swarm Optimization and Reduce-Error

Authors: Xuhui Zhu; Zhiwei Ni; Pingfan Xia; Liping Ni
Journal: Complexity
 (2020)

---

## Abstract

Ensemble pruning has been widely applied to improve the capacity of multiple learner system. Both diversity and classification accuracy of learners are considered as two key factors for achieving an ensemble with competitive classification ability. Considering that extreme learning machine (ELM) is characterized by excellent training rate and generalization capability, it is employed as the base classifier. For a multiple ELM system, when we increase its constituents’ diversity, the mean accuracy of the whole members must be decreased. Therefore, a compromise between them can ensure that the ELMs remain good diversity and high precision, but finding the compromise brings a heavy computational burden. It is hard to look for the exact result via the searching of intelligent algorithms or pruning of diversity measures. On the basis, we propose a hybrid ensemble pruning approach employing coevolution binary glowworm swarm optimization and reduce-error (HEPCBR). Considering the good performance of reduce-error (RE) in selecting ELMs with high diversity and precision, we try to employ RE to choose the satisfactory ELMs from the generated ELMs. In addition, the constituents are further selected via the proposed coevolution binary glowworm swarm optimization, which are utilized to construct the promising ensemble. Experimental results indicate that, compared to other frequently used methods, the proposed HEPCBR achieves significantly superior performance in classification.


---

## Body

## 1. Introduction

Ensemble learning is widely used to improve classification ability in the areas such as image recognition [1, 2], intelligent detection [3, 4], and data mining [5, 6]. Its main idea is to aggregate the predictions of multiple learners so that a better predictive results can be attained [7, 8]. Considering that its remarkable enhancement in classification in contrast to single ones [9, 10], it has been widely used to several distinct applications, e.g., image processing [11, 12], medical diagnosis [13, 14], age prediction [15, 16], gene expression data classification [17, 18], and intrusion detection [19], but there is no such thing as a free lunch. When we employ massive learners for attaining an ensemble with good generalization ability, it needs to consume a large number of computing resources in the application of practical problems [20]. To reduce the consumption of computing resources, ensemble pruning has been studied to cope with it [9]. It aims to aggregate a fraction of learners for increasing prediction accuracy via consuming less resource [10]. It is widely accepted that the same results are achieved by aggregating multiple identical learners, and the classification capacity cannot be enhanced [20]. Namely, the learners should be more diverse on data samples, so as to acquire the ensemble with higher predictive ability.Nevertheless, combining diverse learners may result in a better or worse performance, because the ensemble composed of the members with great diversity and low precision may win low classification capacity [21]. Hence, the performance of ensemble is dramatically influenced by the two fundamental factors, i.e., diversity and precision of learners [22, 23]. As a consequence, we should select the learners with large diversity and high precision, and it can be converted to a combinatory optimization question [24]. Assume that there are n learners in an ensemble, and its size of nonempty subsets is very large [9, 25]. Hence, most algorithms cannot do an exhaustive search in such a large subset [9]. Many ensemble pruning methods are proposed for coping with the selection of learners, and the existing works are presented as below.As a rule, we can classify these techniques into several categories [10, 20]. Firstly, some scholars attempt to adopt some diversity measures to evaluate the learners, and those ones who satisfy some stability conditions can be chosen to build the ensemble via different pruning strategies [26, 27], e.g., kappa [28], reduce-error [9], complementarity [26], contribution [29], margin-based criteria [30], margin and diversity-based measure [31], and relevancy and complementary measures [32]. Secondly, the pruning of classifiers is a combinatory optimization question. Some researchers try to directly employ heuristic search algorithms to solve it [33, 34], for instance, a well-known pruning method, named GASEN [33], and GSOEP [34]. Thirdly, some clustering techniques are used to select the learners with good diversity for integration, and those ones are a series of representative learners [8]. Widely used techniques include k-means [35] and deterministic annealing [36]. Finally, many scholars study other different pruning methods to acquire the selection of learners, for example, frequent pattern [7], a randomized greedy selective strategy and ballot [37], greedy randomized dynamic pruning [38], confidence interval based on double-fault measure [39], graph coloring way [40], cost-sensitive rotation forest [17], induction of decision tree [41], and simple coalitional games [20].As to the analysis above, it is easy to find that those existing works via diversity measures only remove a part of learners with bad performance, and there still exist learners with low precision, which may impact on the performance of the ensemble. To tackle the aforementioned issue, we combine diversity measures and heuristic algorithms to filter out redundant learners. The integration of them can seek for the optimal combination of learners. Where diversity measure is used to preprune a collection of learners with poor comprehensive capability [9]. They notably reduce its computational requirements. Then, we employ heuristic algorithms to select the learners with good comprehensive performance from the remaining learners to build the ensemble [33].Considering the advantages of ELM [42, 43], we use it as the base learner in this work [39]. Reduce-error (RE) can perform better than other diversity measures in filtering out learners with bad performance [9]. Hence, RE can select some complementary learners with high classification capability, memorably downsize ELMs, and reduce the computing requirements. Therefore, RE denotes an appropriate preselection measure. In addition, glowworm swarm optimization (GSO) shows the superiorities [44–48] of strong robustness, easy to realize, and good capability of global search. Hence, it provides an efficient search strategy for further pruning redundant ELMs.Therefore, we propose a novel hybrid pruning method called HEPCBR utilizing the integration of the proposed CBGSO and RE. Existing pruning methods generally find the subensemble with best capability employing GSO or RE separately, which cannot exactly find it, so the combination of CBGSO and RE is employed to search for it. HEPCBR is an integrated method for dealing with ensemble pruning of multiple ELMs, which performs well in terms of searching for the optimal combination of ELMs. We remove some redundant ELMs using RE for reducing the size of ELMs and alleviating the computational complexity, and the presented CBGSO is also adopted to further seek for the optimal subensemble of ELMs. Therefore, the combination of CBGSO and RE can cope with the selection of ELMs.This work’s contributions have been presented as follows:(1)
An ensemble pruning algorithm, named HEPCBR, which utilizes the integration of RE and the proposed CBGSO(2)
We modify the basic GSO, named CBGSO, which has good convergence accuracy and high evolution velocity(3)
RE takes full advantage of the diversity among ELMs and filters out a fraction of ELMs with poor comprehensive performance(4)
Experimental results show the proposed approach can obtain a significant enhancement in classification capacity


## 2. CBGSO

The presented GSO was inspired via glowworms’ luminous behavior [44, 45]. In GSO’s searching process, an initial population composed of many glowworms is generated randomly, and then each glowworm moves forwards the one with larger luciferin. In addition, many individuals can come together the locations of the ones with high luciferin values [49]. Therefore, the optimal one has been gained. For an optimization question in a discrete space, the basic GSO cannot be directly used because of its fixed step.To deal with the above question in a discrete space, we propose the CBGSO by improving the searching processes of the basic GSO. In CBGSO, its fixed step is modified, so as to make the algorithm efficiently seek for global optimum solution in the binary space. Crossover operation and collaborative operation are introduced into the basic GSO, which can create some new individuals and maintain the population diversity, so as to avoid getting in the local optimum and improve the convergence rate. Escaping operation is used in this work, which make those individuals who equal to the locally optimal solution have a chance to escape from the current area and avoid falling into a local optimum. The above operations are introduced into the basic GSO, so as to make it efficiently find the optimum solution. CBGSO has been displayed in the following part.


### 2.1. Improvement of Moving Way

Given a binary combinatorial optimization problem, GSO cannot directly do a search using a fixed step, and we need to modify its movement. To make sure that GSO can move in binary space, we try to update the position of a glowworm with a certain probability [34, 48]. Its updating process can be formulated as(1)xikt+1=r,if rand≤p3,xikt,ifp3<rand≤2p3,xijt,if rand>p3,where Xit=xi1,xi2,…,xin is the current glowworm; xikt is the k-th dimension of Xit at the t-th iteration; r is random integer 0 or 1; p is a predefined probability; and rand is a random number.


### 2.2. Crossover Operation

To enhance the searching rate of GSO, a crossover operation is introduced in GSO, which is inspired by GA [33]. In the population of glowworms, two selected glowworms perform a crossover, and two new offspring individuals can be created by exchanging part of elements of their parent individuals. That is, it has a chance to create new offspring individuals that perform better than before, and those individuals are not maintained in the population. If all the glowworms in the population perform the crossover, then the population performs diversely with a large probability. After that, those glowworms with bad performance will be eliminated. Namely, if the new generated glowworms can achieve better results, then the current individuals would be replaced. The crossover operation is described as(2)Xi′t=xi1,xi2,…,xjk,…,xinXj′t=xj1,xj2,…,xik,…,xjn,ifrk<0.5,Xit=Xi′t,ifYi′t>YitandYi′t>Yj′t,Xit=Xj′t,ifYj′t>YitandYj′t>Yi′t,where Xi′t,Xj′t represent the new generated glowworms who are created using the crossover operation; Xi,Xj express two different glowworms, respectively; and rk declares a random number, rk∈0,1, 1≤k≤n.


### 2.3. Collaborative Operation

To avoid the remaining local optima, a collaborative operation is employed. Each glowworm in GSO can keep strong links with those glowworms inside its dynamic local-decision domain but has no link with that outside the dynamic local-decision domain. So, each glowworm should keep link with those individuals outside the dynamic local-decision domain. In this work, the population of glowworms can be partitioned into several subgroups. Each subgroup can keep strong links with other subpopulations. The optimal glowworm in one subgroup performs a crossover with the individual in another one, so as to gain new individuals with better performance via the exchange of their elements.Assume that the population can be partitioned intoS subgroups, Xis1t=xi1s1,xi2s1,…,xins1 is the optimal glowworm in the s1-th subgroup, Xjs2t=xj1s2,xj2s2,…,xjns2 is the optimal glowworm in the s2-th one, and 1≤s1≠s2≤S. The collaborative operation can be expressed as follows:(3)Xis1s2t=xi1s1,xi2s1,…,xip′−1s1,xjps2,…,xins2,Xjs1s2t=xj1s2,xj2s2,…,xjp′−1s2,xips1,…,xins1,Xis1t=Xis1s2t,ifYis1s2t>Yis1tandYis1s2t>Yjs1s2t,Xis1t=Xjs1s2t,ifYjs1s2t>Yis1tandYjs1s2t>Yis1s2t,where Xis1s2t,Xjs1s2t are the new generated glowworms, respectively; Yis1t,Yis1s2t,Yjs1s2t are the fitness values of Xis1t,Xis1s2t,Xjs1s2t, respectively.


### 2.4. Escaping Operation

To increase the probability of escaping from the local optima, escaping operation [50] is adopted. If the glowworm wins the same with the locally optimal solution, several random elements of the glowworm will be changed, so as to move forward other position for escaping from the local optima.


## 2.1. Improvement of Moving Way

Given a binary combinatorial optimization problem, GSO cannot directly do a search using a fixed step, and we need to modify its movement. To make sure that GSO can move in binary space, we try to update the position of a glowworm with a certain probability [34, 48]. Its updating process can be formulated as(1)xikt+1=r,if rand≤p3,xikt,ifp3<rand≤2p3,xijt,if rand>p3,where Xit=xi1,xi2,…,xin is the current glowworm; xikt is the k-th dimension of Xit at the t-th iteration; r is random integer 0 or 1; p is a predefined probability; and rand is a random number.


## 2.2. Crossover Operation

To enhance the searching rate of GSO, a crossover operation is introduced in GSO, which is inspired by GA [33]. In the population of glowworms, two selected glowworms perform a crossover, and two new offspring individuals can be created by exchanging part of elements of their parent individuals. That is, it has a chance to create new offspring individuals that perform better than before, and those individuals are not maintained in the population. If all the glowworms in the population perform the crossover, then the population performs diversely with a large probability. After that, those glowworms with bad performance will be eliminated. Namely, if the new generated glowworms can achieve better results, then the current individuals would be replaced. The crossover operation is described as(2)Xi′t=xi1,xi2,…,xjk,…,xinXj′t=xj1,xj2,…,xik,…,xjn,ifrk<0.5,Xit=Xi′t,ifYi′t>YitandYi′t>Yj′t,Xit=Xj′t,ifYj′t>YitandYj′t>Yi′t,where Xi′t,Xj′t represent the new generated glowworms who are created using the crossover operation; Xi,Xj express two different glowworms, respectively; and rk declares a random number, rk∈0,1, 1≤k≤n.


## 2.3. Collaborative Operation

To avoid the remaining local optima, a collaborative operation is employed. Each glowworm in GSO can keep strong links with those glowworms inside its dynamic local-decision domain but has no link with that outside the dynamic local-decision domain. So, each glowworm should keep link with those individuals outside the dynamic local-decision domain. In this work, the population of glowworms can be partitioned into several subgroups. Each subgroup can keep strong links with other subpopulations. The optimal glowworm in one subgroup performs a crossover with the individual in another one, so as to gain new individuals with better performance via the exchange of their elements.Assume that the population can be partitioned intoS subgroups, Xis1t=xi1s1,xi2s1,…,xins1 is the optimal glowworm in the s1-th subgroup, Xjs2t=xj1s2,xj2s2,…,xjns2 is the optimal glowworm in the s2-th one, and 1≤s1≠s2≤S. The collaborative operation can be expressed as follows:(3)Xis1s2t=xi1s1,xi2s1,…,xip′−1s1,xjps2,…,xins2,Xjs1s2t=xj1s2,xj2s2,…,xjp′−1s2,xips1,…,xins1,Xis1t=Xis1s2t,ifYis1s2t>Yis1tandYis1s2t>Yjs1s2t,Xis1t=Xjs1s2t,ifYjs1s2t>Yis1tandYjs1s2t>Yis1s2t,where Xis1s2t,Xjs1s2t are the new generated glowworms, respectively; Yis1t,Yis1s2t,Yjs1s2t are the fitness values of Xis1t,Xis1s2t,Xjs1s2t, respectively.


## 2.4. Escaping Operation

To increase the probability of escaping from the local optima, escaping operation [50] is adopted. If the glowworm wins the same with the locally optimal solution, several random elements of the glowworm will be changed, so as to move forward other position for escaping from the local optima.


## 3. HEPCBR

In this section, HEPCBR is proposed using the combination of CBGSO and RE. First, the ELMs with poor comprehensive performance are prepruned using RE for downsizing ELMs and notably reducing the computational overheads of the selection of ELMs; second, the subensemble of ELMs are chosen from the remaining ELMs using the proposed CBGSO. The proposed HEPCBR is described as follows.


### 3.1. Initial Pool

To maintain the good ability in classification by grouping multiple learners, these members should be diverse. When we try to gain the diverse learners, the weak one can be employed [40]. ELM is a weak learner [39] with fast training speed. We produce multiple diverse ELMs with a high efficiency. Therefore, ELM can be utilized to generate M ELMs via the bootstrap sampling method in bagging [51], and the initial pool of ELMs is constructed.Huang et al. [42, 43] first published the first version of ELM for dealing with classification and regression problems [52]. When the hidden node of ELM is a certain value, the hidden weights and biases are generated randomly, so as to gain the unique solutions with a high efficiency.For a training setZtrain=zi,li,i=1,2,…,Ntrain, then the model of ELM is expressed as follows:(4)∑i=1Lβigizj=∑i=1Lβigωizj+bi=li,where zi=zi1,zi2,…,zinT∈Rn,li=li1,li2,…,limT∈Rm,1≤i,j≤Ntrain; L declares the hidden nodes’ size; gz exhibits an activate function; ωi denotes the weight matrix; βi discloses the weight matrix of the output layer; and bi reveals a threshold. Equation (4) is recharacterized in another way:(5)H∗β=L∗,where H∗=gω1z1+b1⋯gωLz1+bL⋮⋱⋮gω1zN+b1⋯gωLzN+bLN×L, L∗=l1T⋯lNTN×mT, and β=β1T⋯βLTL×mT, and H∗ manifests the hidden output matrix. Then, we can get(6)β=H+L∗.where H+ intimates the Moore–Penrose generalized inverse of H∗.


### 3.2. Preselection of ELMs

The pruning of learners is an NP-complete problem. To make an ensemble perform at its best, we should balance the accuracy and the diversity of ELMs, which will bring heavy computational burden. Therefore, it is almost impossible to directly perform the search for the optimal subensemble. To efficiently find the optimal subensemble, we need to downsize the redundant ELMs and reduce the computational burden of the selection of ELMs. Finally, the final ensemble can be obtained using heuristic algorithms with a high searching efficiency.Reduce-error (RE) [9, 26, 28] selects the ELMs with good diversity and high precision via evaluating its contribution to the ensemble, and it should be utilized for removing some poorly performing ELMs. We introduce its basic concept in the following part.For a training setZtrain=zi,li,i=1,2,…,Ntrain. Its samples are composed of a vector zi and a label li∈1,2,…,c. The predictive results are attained by classifying the samples in the training set Ztrain, and hx shows an ELM that can achieve the predictive label li in terms of zi. The predictive result EHz≡hmzm=1M of the original ensemble is obtained using the unweighted majority voting method, which is shown as follows [9, 26, 28]:(7)EHz=argmax∑i=1MIhiz=l,l∈C,where I⋅ represents an indicator function (Itrue=1 and Ifalse=0); hmz expresses the class label classified by the i-th ELM; and C=1,2,…,c represents the set of sample labels.The ELMs who can reduce the error rate of the candidate subensemble in the pool of unselected ELMs can be used to construct the final ensemble [9, 26, 28]. The first selected ELM is that with the highest classification ability on the training set. Assume u ELMs have been selected to constitute the ensemble Su, and then the u+1th ELM is packaged into the ensemble Su:(8)Su+1=argmaxk∑z,l∈ZtrainIESu∪hkz=l,where hk∈F\Su,hk∈F∩hk∉Su.In this section, we use RE to reduce a part of redundant ELMs for a goal of significantly alleviating the computation complexity. The ELM with the best classification accuracy is selected according to the prepruning method utilizing RE, which is the first ELM of the ensemble, and it grows by joining new ELMs in terms of equation (8). We attempt to determine the size of the preselection through the use of RE. Suppose that we retain the first M′ ELMs acquired by RE to construct the prepruned ensemble, and then the M′ members are used to be further selected.Theorem 1.
The ensemble composed of the firstM′ ELMs extracted from the initial pool using RE, and its ensemble accuracy can be improved.Proof.
M training subsets are generated employing a bootstrap sampling way, and multiple ELMs can be independently trained on the different training subsets. Hence, M ELMs are achieved H=h1,h2,…,hm. The process of preselection based on RE is as follows.
First, the ELMh1′ in H who does the best in classification is selected to add into the new subensemble H′=h1′, and its accuracy is en1. Second, the ELM h2′ is extracted from the unselected ELMs H\H′, so as to attain that new subensemble H′=h1′,h2′ wins the highest precision according to equation (8), and its ensemble accuracy is en2. Finally, h3′,h4′,…,hm′ in the unselected ELMs H\H′ can be added into the new subensemble H′ one by one according to equation (8) in a similar fashion, and their ensemble accuracies en3,en4,…,enm are achieved. Then, the new sequence of ELMs H′=h1′,h2′,…,hm′ and its sequence of ensemble accuracies en1,en2,…,enm are attained.
LetM′=argmaxien1,en2,…,eni,…,enm, namely, enM′ is the best. In addition, h1′,h2′,…,hM′′ should be retained, because the ensemble composed of h1′,h2′,…,hM′′ can achieve the highest accuracy. In a word, the first M′ ELMs are utilized to form the prepruned ensemble using RE, and then its ensemble performance can be improved.
To verify the above theorem, the average ensemble accuracies are achieved by ordered bagging in terms of RE with 50, 150, and 250 ELMs on different datasets (Bupa, Vehicle, and CMC), respectively, as shown in Figure1. We can observe from Figure 1 that the curves of the accuracies achieved by RE first rise up and then move down, with the ELMs’ size growing. It can attain a peak when the number of ELMs is moderated. After that, as the size of ELMs increases, its precision gradually decreases. Namely, the ensemble ability can be enhanced after preselection based on RE. At the beginning, the ELMs have low diversity, so their accuracies go up with the size of ELMs rising, whereas as the number gets larger, the poor-performing ELMs have been added into the subensemble, which leads to the decline of the precision. From Figure 1, we can know that the ordered ensemble using RE achieves a maximal value before the ELMs’ size reaches a certain value. Yang et al. [21] found that the heuristic algorithms can perform better in the selection of learners than other techniques when the size of learners is less than 25. As a consequence, we select the first 25 well-performing ELMs, which are used for postselection via CBGSO.Figure 1
Mean precision of ordered ensemble using reduce-error on the different datasets. On Bupa: (a) size of 50, (b) size of 150, and (c) size of 250. On Vehicle: (d) size of 50, (e) size of 150, and (f) size of 250. On CMC: (g) size of 50, (h) size of 150, and (i) size of 250.
(a)(b)(c)(d)(e)(f)(g)(h)(i)


### 3.3. Postselection of ELMs

Ensemble pruning is a combinatorial optimization problem. To calculate its fitness value, we take the classification accuracyf as its fitness function [33, 34], where f shows the classification precision between the predictive outputs and the actual labels, f=1/m∑j=1mAccy^j,yj, where if y^j=yj, then Accy^j,yj=1; otherwise, Accy^j,yj=0, m expresses the size of testing samples, y^j,yj represent the predictive result and its actual label on the j-th testing samples, respectively. In addition, H′=h1′,h2′,…,h25′ shows the selected 25 ELMs gained by RE. In this work, each glowworm is characterized as a bit sequence. In the bit sequence, 1 discloses the ELM is selected, and 0 points to the ELM is not selected. So, we can cope with the pruning question of well employing CBGSO.


### 3.4. The Pseudocode of HEPCBR

The pseudocode of HEPCBR has been exhibited as follows (Algorithm1).Algorithm 1: HEPCBR.
Inputs: HEPCBR’s parameters.Outputs: the subensemble Xopt and its precision fopt.(1)
A pool composed ofM ELMs is built, H⟵h1,h2,…,hM.(2)
H′⟵∅.(3)
The optimal ELMh1′ with the best classification ability is chosen.(4)
H′⟵h1′.(5)
H⟵H\h1′.(6)
fori⟵2 to M′do(7)
The ELMhi′ is chosen via equation (8).(8)
H′⟵H′∪hi′.(9)
H⟵H\hi′.(10)
end for(11)
N individuals can be produced randomly, and evaluate their fitness values f.(12)
Xopt⟵maxfitnessX1,…,XN, fopt⟵maxf1,…,fN.(13)
t⟵1.(14)
whilet≤tmaxdo(15)
fors⟵1 to Sdo(16)
fori⟵1 to N/Sdo(17)
iffi=foptthen(18)
Perform escaping operation to generate a new glowworm.(19)
end if(20)
Find the objective individualXj.(21)
Update the position ofXi via equation (1).(22)
Update parameters ofXi.(23)
Perform crossover operation to produce new individuals.(24)
end for(25)
Perform collaborative operation produce new individuals.(26)
end for(27)
Xopt⟵maxfitnessX1,…,XN, fopt⟵maxf1,…,fN.(28)
end while(29)
returnXopt and fopt


## 3.1. Initial Pool

To maintain the good ability in classification by grouping multiple learners, these members should be diverse. When we try to gain the diverse learners, the weak one can be employed [40]. ELM is a weak learner [39] with fast training speed. We produce multiple diverse ELMs with a high efficiency. Therefore, ELM can be utilized to generate M ELMs via the bootstrap sampling method in bagging [51], and the initial pool of ELMs is constructed.Huang et al. [42, 43] first published the first version of ELM for dealing with classification and regression problems [52]. When the hidden node of ELM is a certain value, the hidden weights and biases are generated randomly, so as to gain the unique solutions with a high efficiency.For a training setZtrain=zi,li,i=1,2,…,Ntrain, then the model of ELM is expressed as follows:(4)∑i=1Lβigizj=∑i=1Lβigωizj+bi=li,where zi=zi1,zi2,…,zinT∈Rn,li=li1,li2,…,limT∈Rm,1≤i,j≤Ntrain; L declares the hidden nodes’ size; gz exhibits an activate function; ωi denotes the weight matrix; βi discloses the weight matrix of the output layer; and bi reveals a threshold. Equation (4) is recharacterized in another way:(5)H∗β=L∗,where H∗=gω1z1+b1⋯gωLz1+bL⋮⋱⋮gω1zN+b1⋯gωLzN+bLN×L, L∗=l1T⋯lNTN×mT, and β=β1T⋯βLTL×mT, and H∗ manifests the hidden output matrix. Then, we can get(6)β=H+L∗.where H+ intimates the Moore–Penrose generalized inverse of H∗.


## 3.2. Preselection of ELMs

The pruning of learners is an NP-complete problem. To make an ensemble perform at its best, we should balance the accuracy and the diversity of ELMs, which will bring heavy computational burden. Therefore, it is almost impossible to directly perform the search for the optimal subensemble. To efficiently find the optimal subensemble, we need to downsize the redundant ELMs and reduce the computational burden of the selection of ELMs. Finally, the final ensemble can be obtained using heuristic algorithms with a high searching efficiency.Reduce-error (RE) [9, 26, 28] selects the ELMs with good diversity and high precision via evaluating its contribution to the ensemble, and it should be utilized for removing some poorly performing ELMs. We introduce its basic concept in the following part.For a training setZtrain=zi,li,i=1,2,…,Ntrain. Its samples are composed of a vector zi and a label li∈1,2,…,c. The predictive results are attained by classifying the samples in the training set Ztrain, and hx shows an ELM that can achieve the predictive label li in terms of zi. The predictive result EHz≡hmzm=1M of the original ensemble is obtained using the unweighted majority voting method, which is shown as follows [9, 26, 28]:(7)EHz=argmax∑i=1MIhiz=l,l∈C,where I⋅ represents an indicator function (Itrue=1 and Ifalse=0); hmz expresses the class label classified by the i-th ELM; and C=1,2,…,c represents the set of sample labels.The ELMs who can reduce the error rate of the candidate subensemble in the pool of unselected ELMs can be used to construct the final ensemble [9, 26, 28]. The first selected ELM is that with the highest classification ability on the training set. Assume u ELMs have been selected to constitute the ensemble Su, and then the u+1th ELM is packaged into the ensemble Su:(8)Su+1=argmaxk∑z,l∈ZtrainIESu∪hkz=l,where hk∈F\Su,hk∈F∩hk∉Su.In this section, we use RE to reduce a part of redundant ELMs for a goal of significantly alleviating the computation complexity. The ELM with the best classification accuracy is selected according to the prepruning method utilizing RE, which is the first ELM of the ensemble, and it grows by joining new ELMs in terms of equation (8). We attempt to determine the size of the preselection through the use of RE. Suppose that we retain the first M′ ELMs acquired by RE to construct the prepruned ensemble, and then the M′ members are used to be further selected.Theorem 1.
The ensemble composed of the firstM′ ELMs extracted from the initial pool using RE, and its ensemble accuracy can be improved.Proof.
M training subsets are generated employing a bootstrap sampling way, and multiple ELMs can be independently trained on the different training subsets. Hence, M ELMs are achieved H=h1,h2,…,hm. The process of preselection based on RE is as follows.
First, the ELMh1′ in H who does the best in classification is selected to add into the new subensemble H′=h1′, and its accuracy is en1. Second, the ELM h2′ is extracted from the unselected ELMs H\H′, so as to attain that new subensemble H′=h1′,h2′ wins the highest precision according to equation (8), and its ensemble accuracy is en2. Finally, h3′,h4′,…,hm′ in the unselected ELMs H\H′ can be added into the new subensemble H′ one by one according to equation (8) in a similar fashion, and their ensemble accuracies en3,en4,…,enm are achieved. Then, the new sequence of ELMs H′=h1′,h2′,…,hm′ and its sequence of ensemble accuracies en1,en2,…,enm are attained.
LetM′=argmaxien1,en2,…,eni,…,enm, namely, enM′ is the best. In addition, h1′,h2′,…,hM′′ should be retained, because the ensemble composed of h1′,h2′,…,hM′′ can achieve the highest accuracy. In a word, the first M′ ELMs are utilized to form the prepruned ensemble using RE, and then its ensemble performance can be improved.
To verify the above theorem, the average ensemble accuracies are achieved by ordered bagging in terms of RE with 50, 150, and 250 ELMs on different datasets (Bupa, Vehicle, and CMC), respectively, as shown in Figure1. We can observe from Figure 1 that the curves of the accuracies achieved by RE first rise up and then move down, with the ELMs’ size growing. It can attain a peak when the number of ELMs is moderated. After that, as the size of ELMs increases, its precision gradually decreases. Namely, the ensemble ability can be enhanced after preselection based on RE. At the beginning, the ELMs have low diversity, so their accuracies go up with the size of ELMs rising, whereas as the number gets larger, the poor-performing ELMs have been added into the subensemble, which leads to the decline of the precision. From Figure 1, we can know that the ordered ensemble using RE achieves a maximal value before the ELMs’ size reaches a certain value. Yang et al. [21] found that the heuristic algorithms can perform better in the selection of learners than other techniques when the size of learners is less than 25. As a consequence, we select the first 25 well-performing ELMs, which are used for postselection via CBGSO.Figure 1
Mean precision of ordered ensemble using reduce-error on the different datasets. On Bupa: (a) size of 50, (b) size of 150, and (c) size of 250. On Vehicle: (d) size of 50, (e) size of 150, and (f) size of 250. On CMC: (g) size of 50, (h) size of 150, and (i) size of 250.
(a)(b)(c)(d)(e)(f)(g)(h)(i)


## 3.3. Postselection of ELMs

Ensemble pruning is a combinatorial optimization problem. To calculate its fitness value, we take the classification accuracyf as its fitness function [33, 34], where f shows the classification precision between the predictive outputs and the actual labels, f=1/m∑j=1mAccy^j,yj, where if y^j=yj, then Accy^j,yj=1; otherwise, Accy^j,yj=0, m expresses the size of testing samples, y^j,yj represent the predictive result and its actual label on the j-th testing samples, respectively. In addition, H′=h1′,h2′,…,h25′ shows the selected 25 ELMs gained by RE. In this work, each glowworm is characterized as a bit sequence. In the bit sequence, 1 discloses the ELM is selected, and 0 points to the ELM is not selected. So, we can cope with the pruning question of well employing CBGSO.


## 3.4. The Pseudocode of HEPCBR

The pseudocode of HEPCBR has been exhibited as follows (Algorithm1).Algorithm 1: HEPCBR.
Inputs: HEPCBR’s parameters.Outputs: the subensemble Xopt and its precision fopt.(1)
A pool composed ofM ELMs is built, H⟵h1,h2,…,hM.(2)
H′⟵∅.(3)
The optimal ELMh1′ with the best classification ability is chosen.(4)
H′⟵h1′.(5)
H⟵H\h1′.(6)
fori⟵2 to M′do(7)
The ELMhi′ is chosen via equation (8).(8)
H′⟵H′∪hi′.(9)
H⟵H\hi′.(10)
end for(11)
N individuals can be produced randomly, and evaluate their fitness values f.(12)
Xopt⟵maxfitnessX1,…,XN, fopt⟵maxf1,…,fN.(13)
t⟵1.(14)
whilet≤tmaxdo(15)
fors⟵1 to Sdo(16)
fori⟵1 to N/Sdo(17)
iffi=foptthen(18)
Perform escaping operation to generate a new glowworm.(19)
end if(20)
Find the objective individualXj.(21)
Update the position ofXi via equation (1).(22)
Update parameters ofXi.(23)
Perform crossover operation to produce new individuals.(24)
end for(25)
Perform collaborative operation produce new individuals.(26)
end for(27)
Xopt⟵maxfitnessX1,…,XN, fopt⟵maxf1,…,fN.(28)
end while(29)
returnXopt and fopt


## 4. Experiments

To evaluate the classification ability of HEPCBR, we select 25 UCI machine learning benchmark datasets to implement the massive experiments, and these datasets are presented in Table1. Specifically, each experiment is repeated 30 times using different random seeds, and then the final classification accuracies are attained. The parameters of CBGSO are given in the following part [44, 45]: the ρ=0.4, the γ=0.08, the nt=5, the p=0.45, the S=5, and the tmax=400.We randomly divide the dataset into five equal parts, three of them as training, one of them as validation, and the other one as testing, where the ELMs were trained on the training datasets, the pruning process of ELMs is implemented on the validation datasets, and the final results are achieved on the testing datasets.Table 1
UCI datasets.
DatasetsInstancesAttributesClassesHeart270132Cleveland303135Column31062Ecoli33688Bupa34562Wholesale44072Forest523274Balance62543Diabetes76882Vehicle846184Tic-tac-toe95892German1000202QSAR1055412Diabetic-r1151192Phishing135393CMC147393Yeast1484910Wineq-r1599116Car172864CTG2126223Segment2310197Abalone417783Spambase4601572Wineq-w4898117Waveform5000213


### 4.1. Experimental Results

The proposed HEPCBR achieves the final ensemble using the combination of RE and the proposed CBGSO, and we attempt to verify that whether HEPCBR can perform better than any one of them. So, the predictive results of the proposed HEPCBR are compared to both RE and CBGSO in different sizes of ELMs, as shown in Tables2 and 3. It is easy to see from Tables 2 and 3 that HEPCBR, via the integration of RE and CBGSO, wins a higher precision than any one of them (CBGSO or RE) on the most datasets using less size of ELMs. We can also observe from Tables 2 and 3 that, as the pool size increases, the prediction results achieved by HEPCBR and RE are getting better, but those attained by CBGSO are getting worse. We diagnose the causes, and we find that the computation complexity of the selection of ELMs exponentially increases as the size gets larger. For CBGSO, it is difficult to do an exhaustive search to find the exact subensemble. As a result, the preselection of ELMs can significantly lower the calculation complexity of the selection of ELMs. RE can select some well-performing ELMs, and the postselection of CBGSO can make the ensemble gain better results. Moreover, the classification ability of HEPCBR can attain a very small improvement as the initial size exceeds 150. Hence, 150 ELMs are used in this work.Table 2
Precisions of HEPCBR in comparison with RE and CB.
Datasets50100150REnCBnHEnREnCBnHEnREnCBnHEnHeart74.292576.831978.071277.592574.384480.491381.352575.317183.1213Cleveland56.822556.262559.341058.622554.444760.251057.042553.097557.9515Column86.432587.592090.541188.222586.034690.511490.602586.707292.3814Ecoli87.632588.232388.431288.382588.254988.921389.272588.137589.8812Bupa74.412574.972277.911378.882578.184481.051479.222575.987081.7214Wholesale89.332590.841891.371391.822589.484393.191390.982588.767191.6814Forest92.472593.012393.961192.522591.164693.441494.642591.506994.9415Balance92.722593.122193.041291.842591.365092.481391.842591.767392.8011Diabetes69.412572.271972.551071.782570.674573.621174.332569.686875.1714Vehicle65.852567.261969.951369.432568.504271.251669.772566.146571.5915Tic-tac-toe93.952594.882295.621394.942595.004396.251396.292593.537696.6615German78.602578.102181.201180.902577.504482.801380.902574.207382.2012QSAR87.392586.892189.961289.082587.054791.551190.352584.526992.6013Diabetic-r77.022577.591881.011378.962577.184681.471480.502575.847183.1012Phishing86.392586.532488.541187.602585.824888.781188.532585.547689.3513CMC61.142562.102063.791262.332561.294164.301263.402559.866364.7514Yeast58.742560.532060.561262.572561.984663.991362.232561.397163.2312Wineq-r57.302561.561960.341161.862559.134163.951364.072558.746766.2613Car92.642594.271993.801094.722592.714695.181294.542592.067295.3011CTG81.702584.391685.28985.092580.884287.82985.142579.547787.5410Segment86.122587.971988.341188.272586.884389.441489.192585.976990.4414Abalone57.112559.991958.581160.602557.214161.641258.742559.977159.9612Spambase80.212581.661782.891181.822579.754084.271283.372579.046185.1712Wineq-w47.322551.201550.121051.052548.484153.141351.182548.606953.7612Waveform88.242589.382089.711389.282588.354490.171390.122588.567090.9414HE: HEPCBR; RE: reduce-error; CB: CBGSO.Table 3
Precisions of HEPCBR in comparison with RE and CB.
Datasets200250300REnCBnHEnREnCBnHEnREnCBnHEnHeart79.922572.809282.161380.622571.1412182.291482.302571.1214483.1916Cleveland56.752555.2110057.661160.062554.0212661.381157.892554.4014958.9013Column91.272584.819693.301691.522585.2712192.881592.102584.0914793.7214Ecoli89.802587.479890.531190.172587.7912590.621389.352587.6515090.0510Bupa82.242574.629483.961582.032574.0611984.291482.452575.3114984.0215Wholesale92.902589.209193.641493.392588.0311793.851392.552588.2115093.5914Forest94.382591.669795.041394.632590.5812295.291395.412590.5914595.9515Balance91.442591.049791.601393.122591.1212493.521291.682591.6015392.0812Diabetes72.792569.019874.341374.262569.2012275.751375.252568.9114378.8415Vehicle70.812566.969172.461671.772565.7211673.471573.932564.9713874.5218Tic-tac-toe96.602593.379697.281396.192592.4811996.761397.502592.4314697.6516German82.002576.409783.101281.702576.9012183.101480.502572.2015281.9013QSAR90.582587.089592.581389.212583.3911990.911190.472583.7014191.5215Diabetic-r80.422573.859582.591383.422575.7812185.331681.992575.6614083.4616Phishing90.532585.4610091.201390.622586.2712391.291390.522586.8615090.8214CMC64.292559.349065.651365.922558.0811767.071564.792558.6514466.0114Yeast64.802559.239765.541262.632559.3312363.181363.542558.1014864.0813Wineq-r62.582556.929665.281362.232556.4711763.991363.682559.2114965.4914Car93.812591.609694.391294.142591.7412394.611294.612592.2914695.1312CTG86.452580.209887.911188.142580.3912689.231488.402579.9414989.7613Segment89.302585.439790.301289.772586.0811890.581589.532585.5714390.1815Abalone60.042559.829560.761260.552558.2111961.161460.872557.8114562.0112Spambase83.732578.949085.341484.702579.1711085.911483.852576.4814085.0615Wineq-w52.422546.549654.391349.122548.2412851.681151.432546.8614953.2814Waveform90.042587.789990.781590.092587.3712290.661589.462587.8614890.0616HE: HEPCBR; RE: reduce-error; CB: CBGSO.


### 4.2. Comparison with Other Methods

To test the classification ability of HEPCBR further, massive experiments were implemented by comparing it with other techniques: bagging [51], kappa [28], AGOB [26], POBE [27], DREP [22], DEELM [39], GASEN [33], GSOEP [34], MOAG [30], RREP [10], DMEP [40], EPSCG [20], DASEP [24], MDOEP [31], RCOA [32], and PEAD [41]. Bagging extracts the training samples with equal probability, and it can construct an initial pool composed of multiple learners with a good diversity. Kappa pruning achieves the ensemble by grouping the learners with their measures under a certain value. AGOB, POBE, and MOAG find the ordering of those learners in the ensemble works well. DREP first selects learner with the best classification ability and gradually increases the size of the ensemble which can make the ensemble achieve better precisions. DEELM chooses the learners with their measures falling into an interval. GASEN can assign a random weight to each learner. It adopts GA to evolve their parameters, and the learners whose weights exceed a predefined value can be utilized for combination. GSOEP employs GSO to directly filter out the redundant learners. RREP attained the ensemble by utilizing the retained learners and the removed ones gained by RE. DMEP employs GA to optimize five pairwise diversity matrices for a combining diversity measure, and select a part of learners in the light of the graph coloring theory. EPSCG evaluates the learners’ contribution to the ensemble in terms of the Banzhaf index of power. DASEP finds new measures by simultaneously considering diversity and accuracy of learners, and acquires a good result in the selection of learners. MDOEP attains the final ensemble by the aggregation using margin and diversity-based measure. RCOA ranks all the learners based on the relevancy and complementary measures in a descending order, and the final ensemble is achieved. PEAD selects a collection of learners extracted from a pool using the induction of decision tree, which are used to form the final ensemble.We gain the classification precisions of the proposed HEPCBR, and other methods with 150 ELMs in the pool are displayed in Tables4 and 5. “+/ = /−,” respectively, indicates that HEPCBR achieves higher/neutral/lower precisions than other methods. It is clearly shown in Tables 4 and 5 that the proposed HEPCBR outperforms other state-of-the-art techniques on the most classification tasks, which shows its enhancements in classification. Table 6 displays the sizes of the final ensemble attained by all the pruning approaches. As shown in Table 6, HEPCBR utilizes less the size of ELMs to build the ensemble, but it uses more ELMs than DMEP. Additionally, HEPCBR gains better results than DMEP. As a whole, the proposed HEPCBR has acquired the expected results in classification.Table 4
Classification precisions won by other methods with 150 ELMs.
DatasetsHEPCBRBaggingKappaAGOBPOBEDREPDEELMGASENGSOEPHeart83.1268.5770.0775.9075.5172.6272.8974.8774.97Cleveland57.9553.5252.4554.8853.8753.2653.5254.5852.77Column92.3880.4084.3187.6786.9984.6183.3685.6285.10Ecoli89.8887.1687.9588.8489.0387.7288.2588.4287.98Bupa81.7268.2768.4375.0375.3072.5571.9373.2274.34Wholesale91.6885.7888.9290.1390.1688.3288.5987.9288.72Forest94.9489.0591.3393.3892.8290.9391.2291.1591.52Balance92.8090.4891.3692.7292.6491.9292.2891.6891.52Diabetes75.1766.1269.5772.4072.3071.0670.3570.3970.13Vehicle71.5957.8460.4466.8364.3864.5864.0263.4065.35Tic-tac-toe96.6690.9794.6596.1194.8493.1493.4092.4992.59German82.2072.3075.2579.5078.5076.8576.7576.9076.70QSAR92.6083.2184.8287.5587.0885.2885.2884.8285.59Diabetic-r83.1069.0473.0776.6275.8974.1573.5874.0975.14Phishing89.3584.3985.6687.2186.8485.7085.7088.0787.18CMC64.7554.3153.5358.5957.7456.5556.2459.6160.28Yeast63.2358.0357.8560.5460.1659.1159.1859.7759.22Wineq-r66.2655.8955.9359.4959.3957.8957.0958.0257.63Car95.3091.2992.7493.5293.4992.4892.5493.6592.89CTG87.5478.6282.1085.7283.5181.7582.0884.7880.09Segment90.4484.3585.5387.2486.8685.6685.7086.3686.71Abalone59.9655.4056.5258.5358.4657.4057.2657.9757.25Spambase85.1773.6178.8781.7779.7078.4977.5576.8377.40Wineq-w53.7646.5146.8948.5948.0247.8546.9349.0848.16Waveform90.9486.3587.6688.5688.5887.8287.5789.0289.41+/ = /−17/3/50/0/250/0/250/0/250/0/250/0/250/0/250/0/250/0/25Table 5
Classification precisions won by other methods with 150 ELMs.
DatasetsMOAGRREPDMEPEPSCGDASEPMDOEPRCOAPEADHeart71.6381.4078.5176.1278.8577.5277.3876.54Cleveland53.2656.2456.6054.8155.8455.7656.9257.88Column83.4391.4388.7686.9486.2785.8286.3985.98Ecoli87.5989.6089.8387.1489.9686.9788.4689.22Bupa70.4579.6177.3075.1576.6477.9776.1175.26Wholesale87.6992.0591.2487.4889.8287.7688.5287.50Forest90.5092.8092.8390.4594.9494.9494.8293.75Balance91.9693.0093.3291.7293.2091.4091.6892.44Diabetes69.3775.8974.5672.5272.6671.3871.6270.27Vehicle60.3271.0867.5766.1666.5967.6368.8665.27Tic-tac-toe92.3896.6695.4196.6694.2996.6695.1794.92German75.8581.5081.1577.3078.6078.7078.0077.15QSAR84.4890.2988.9786.5887.5088.3687.3386.52Diabetic-r72.4681.4878.7977.2477.8079.9978.0876.34Phishing85.7487.9189.0685.1188.0386.5286.7787.65CMC55.8763.3161.4458.9559.1558.1258.0657.60Yeast59.4662.1162.3258.6060.9957.9958.9859.63Wineq-r57.7463.8062.8057.0860.8659.6859.0957.82Car92.4294.6594.7192.1393.9691.4793.7993.13CTG81.2187.5486.9082.5485.2187.5486.3987.54Segment85.1389.0387.4984.9286.3086.1085.6384.13Abalone58.1960.3360.1357.3358.9359.2060.5059.96Spambase76.3184.1582.2478.8379.4581.3479.1079.91Wineq-w47.8951.5751.6548.9249.9449.4850.4049.52Waveform87.4390.2689.3687.2888.1789.3488.6788.29+/ = /−0/0/252/2/201/0/240/1/241/1/230/3/221/0/240/1/24Table 6
Sizes of the selected ELMs via other methods with 150 ELMs.
DatasetsHEBaKaAGPODRDEGAGSMORRDMEPDAMDMRPEHeart1315030173075775671301853129302637Cleveland1515030212975613776231721526302522Column1415030233075814471352141713303247Ecoli1215030202975765279261621915302532Bupa1415030253275827170262431923302452Wholesale1415030203175815465331731219302932Forest1515030183075836971311892730302623Balance111503020307577617281621724302322Diabetes1415030202975762869351665014302427Vehicle1515030223175826772321986930302341Tic-tac-toe1515030153175827369311783629301939German1215030153075793973361653117301829QSAR1315030193075816772272191315302822Diabetic-r1215030223275815062341771514302947Phishing1315030193175782973241742134303035CMC1415030263475815860241883813302323Yeast1215030203175793674311861118302623Wineq-r1315030203275802969351841733302324Car1115030183075794073261653821302626CTG1015030172575673155321645117302323Segment1415030233275826670302094519302423Abalone1215030263275791465282151631302328Spambase1215030162875724471331869816301924Wineq-w1215030263275712768292044017302432Waveform1415030243175817161362152924302529HE: HEPCBR; Ba: bagging; Ka: kappa; AG: AGOB; PO: POBE; DR: DREP; DE: DEELM; GA: GASEN; GS: GSOEP; MO: MOAG; RR: RREP; DM: DMEP; EP: EPSCG; DA: DASEP; MD: MDOEP; RC: RCOA; PE: PEAD.In this section, we try to estimate the difference significance between the predictive accuracies gained by HEPCBR and that attained by other comparative methods, and we used the Wilcoxon rank sum test [40] with its level of 0.05. When the p value of two techniques is lower than 0.05, it expresses that there exists significant difference between them. Table 7 exhibits the p values acquired in the test between HEPCBR and other comparative approaches. Table 7 indicates that the p values have been below 0.05. It manifests the significance and the effectiveness of HEPCBR.Table 7
Thep values of the Wilcoxon rank sum test.
Comparisonp valueSignificant or notHEPCBR vs bagging0.00123YesHEPCBR vs kappa0.00123YesHEPCBR vs AGOB0.00123YesHEPCBR vs POBE0.00123YesHEPCBR vs DREP0.00123YesHEPCBR vs DEELM0.00123YesHEPCBR vs GASEN0.00123YesHEPCBR vs GSOEP0.00123YesHEPCBR vs MOAG0.00123YesHEPCBR vs RREP0.02327YesHEPCBR vs DMEP0.00286YesHEPCBR vs EPSCG0.00182YesHEPCBR vs DASEP0.00267YesHEPCBR vs MDOEP0.00401YesHEPCBR vs RCOA0.00157YesHEPCBR vs PEAD0.00270YesThe execution time of the proposed HEPCBR and other comparative techniques is reported in Table8. The observations from Table 8 flag up that HEPCBR uses much less time than GASEN, GSOEP, and DMEP, but utilizes more time than the retaining comparative methods. In HEPCBR, CBGSO is employed to extract the final ELMs from the remaining ones after pruning, which needs to calculate the classification accuracy of each candidate subensemble at each iteration. During the search of CBGSO, each candidate subensemble is expressed by a glowworm, and each glowworm needs to calculate its fitness value at least once per iteration. Therefore, it requires much more time. Those techniques with less running time achieve the selection of ELMs without repetitive iteration process and assessment. Nevertheless, they cannot attain good results in classification. HEPCBR cost less time than the mentioned three methods, it is the reason that those approaches directly selected the ELMs with good comprehensive performance without preselection, and it might not be enough to seek for the exact solution due to the enormous amount candidate subensembles. Moreover, the preselection process can notably alleviate the computational burden of the selection of ELMs. It is concluded that HEPCBR consumes much more time, but the remarkable improvements in classification are obtained.Table 8
Execution time of different techniques.
DatasetsExecution time (s)HEBagKaAGPODRDFGAGSOMORREDivPSCSDMDMRPEHeart6.320.470.504.090.491.270.5116.2216.710.504.2318.120.500.650.561.390.49Cleveland25.290.420.448.090.453.480.4539.8039.340.598.1942.480.590.820.551.670.42Column6.730.510.544.700.531.420.5518.0418.340.554.7823.520.550.700.631.470.51Ecoli34.640.450.4712.530.485.090.4957.5757.970.7212.5965.550.721.040.741.940.54Bupa7.720.480.515.170.491.530.5220.0719.470.535.8222.430.530.700.601.670.57Wholesale9.560.690.736.590.691.920.7425.2624.820.746.6726.170.740.950.871.830.81Forest24.430.810.847.730.823.670.8540.5239.360.988.2145.870.971.190.992.230.85Balance21.460.870.897.310.903.540.9038.7538.761.017.5147.921.011.221.052.171.01Diabetes10.540.690.736.760.711.980.7425.8525.510.756.8333.710.750.970.851.840.71Vehicle37.701.111.1612.501.235.941.1764.7063.031.3912.9769.211.381.751.333.701.48Tic-tac-toe12.481.501.559.351.603.161.5632.1732.371.589.5143.531.571.851.753.582.27German6.651.271.295.291.292.101.3019.0619.031.305.8225.281.301.451.372.021.07QSAR7.221.651.685.951.652.581.6920.0320.561.696.0526.671.691.861.792.361.38Diabetic-r7.591.151.195.941.162.211.2021.3821.791.205.9423.171.201.381.282.271.17Phishing23.252.292.329.942.355.482.3349.9347.892.4710.2159.352.472.722.502.911.66CMC26.572.222.2610.832.335.842.2752.9953.622.4210.9867.292.422.702.392.931.27Yeast59.841.061.0919.421.168.111.1087.8688.761.4720.3998.211.471.951.363.040.95Wineq-r45.942.002.0416.532.207.802.0577.2376.342.3316.9186.122.332.742.243.861.80Car36.043.083.1214.123.387.533.1364.0462.533.3414.5369.183.333.663.394.452.86CTG36.071.531.5712.421.586.031.5864.7566.941.7812.6765.621.772.111.823.261.45Segment81.083.203.2527.393.1912.773.26115.91115.463.7628.15135.403.754.423.747.424.10Abalone47.833.813.8718.403.919.873.8890.2588.634.1518.4999.824.144.604.117.485.03Spambase19.305.495.5817.895.558.215.5957.6957.975.6118.1168.335.616.055.857.875.73Wineq-w85.034.914.9931.504.9615.525.00140.39138.235.5232.32165.145.516.265.337.994.60Waveform58.346.366.4423.666.6113.636.45110.49107.946.7625.01131.156.767.306.859.076.26HE: HEPCBR; Ba: bagging; Ka: kappa; AG: AGOB; PO: POBE; DR: DREP; DE: DEELM; GA: GASEN; GS: GSOEP; MO: MOAG; RR: RREP; DM: DMEP; EP: EPSCG; DA: DASEP; MD: MDOEP; RC: RCOA; PE: PEAD.The aforementioned comparative methods are based on bagging, which are attained using diversity measures with different strategies or heuristic algorithms. In addition, we further compare the proposed HEPCBR with the more state-of-the-art approaches by considering other factors, which are not the same category with the aforementioned comparative methods: C-RoF [17], CS-D-ELM [18], KPCA-RoF [4], and DyPReVNsGraspEnS [38]. C-RoF introduces misclassification, test, and rejection costs for overcoming that some existing works neglect the classification costs, and it achieves a good result. CS-D-ELM extends the D-ELM employing misclassification cost of the learner and embeds rejection cost into it to increase the classification stability. KPCA-RoF integrates the KPCA and RoF for linearly inseparable dataset classification problems, and it enhances the classification capacity of RoF on the nonlinear fractional datasets. DyPReVNsGraspEnS realizes random multistart search, and it can avoid falling into local optimal solutions and possesses a high probability to exactly find the ensemble with better performance. The classification results achieved by different approaches are presented in Table 9. We can see from Table 9 that the accuracies gained by HEPCBR are higher than other four state-of-the-art methods. Then, we verify the significance between HEPCBR and other four approaches using the Wilcoxon rank sum test, which is displayed in Table 10. From Table 10, all the p values are less than 0.05. So, we can conclude that the proposed HEPCBR has powerful advantages over other four techniques in classification, and it is significant and effective.Table 9
Comparison of classification performance with other state-of-the-art methods.
DatasetsHEC-RoFCS-DKPCADyPHeart83.1278.6277.5579.2180.84Cleveland57.9557.0557.6656.0557.86Column92.3889.0788.4888.7389.43Ecoli89.8890.3689.8289.6590.00Bupa81.7277.5277.6978.2280.46Wholesale91.6889.3891.3691.5892.84Forest94.9492.5893.7094.0893.28Balance92.8092.8092.4892.4892.80Diabetes75.1774.7174.6075.4174.81Vehicle71.5968.1868.9168.4370.56Tic-tac-toe96.6694.3295.5195.5796.57German82.2078.0079.2078.4080.80QSAR92.6089.9489.3290.3290.81Diabetic-r83.1075.2177.6476.6180.76Phishing89.3588.3589.8389.2489.71CMC64.7561.0261.2860.8862.28Yeast63.2361.6562.7361.6562.41Wineq-r66.2663.6361.6362.2561.50Car95.3094.4494.9193.8594.37CTG87.5484.1885.8784.7486.53Segment90.4487.3087.9188.0189.72Abalone59.9657.9357.7258.7858.36Spambase85.1780.5481.4481.2283.28Wineq-w53.7650.2351.3052.5851.21Waveform90.9488.4588.7088.6888.84HE: HEPCBR; CS-D: CS-D-ELM; KPCA: KPCARoF; DyP: DyPReVNsGraspEnS.Table 10
Thep values of the Wilcoxon rank sum test.
Comparisonp valueSignificant or notHEPCBR vs C-RoF0.00235YesHEPCBR vs CS-D0.00254YesHEPCBR vs KPCA0.00200YesHEPCBR vs DyP0.01716YesCS-D: CS-D-ELM; KPCA: KPCA-RoF; DyP: DyPReVNsGraspEnS.


### 4.3. Parameter Analysis

In HEPCBR, we employ CBGSO to select the ELMs with good diversity and high accuracy from the remaining ones after prepruning. To make the proposed CBGSO perform well, we try to analyze its parameters. In addition, to assess its convergence speed and precision, the comparative analyses have been presented as follows: MGSO [48], MBGSO [47], MDGSO [34], IBFS [50], MBFS [53], MFS [54], and GA [33]. We implemented the experiments on two datasets, and we adopt different methods in search of the optimal subensemble from the remaining ELMs using RE with the pool size of 150.It is clearly observed in Figure2 that, as a whole, CBGSO attains better convergence accuracy and higher evolution velocity than other seven comparative algorithms, i.e., CBGSO performs better than others in terms of the speed and the precision of convergence. It is also easy to find that the accuracies acquired by CBGSO rise up and stay stable. The results achieved by CBGSO just can attain a slight gain in classification, when the number of iterations is above 400. So, it is 400. In addition, when CBGSO is utilized to search for the final ensemble in HEPCBR, it costs less time than other seven heuristic algorithms, which is displayed in Table 11. Table 11 demonstrates that CBGSO performs higher efficiency of searching than other heuristic algorithms. Hence, CBGSO can work well when it comes to searching speed and efficiency. The accuracies mount and level off, as the population size increases, which is rendered in Figure 3(a). We can see from Figure 3(a) that, when the population size of glowworms is above 25, it will greatly increase the computation complexity with less improvements in classification. Hence, the size is 25. Figure 3(b) analyzes the affection of the initial range on the performance of CBGSO. We can observe from Figure 3(b) that it performs at its best when the initial range is 11. Hence, the initial range is 11. Figure 3(c) reveals the affection of the maximal one on the performance of CBGSO. It is more than the initial one. The observations from Figure 3(c) disclose that the performance of CBGSO peaks at 15. So, we advise it is set as 15.Figure 2
The affection of iteration on the convergence rate of algorithms. (a) Vehicle. (b) CMC.
(a)(b)Table 11
Comparison of execution time with different heuristic algorithms.
MethodsVehicle (s)CMC (s)CBGSO33.1625.09MGSO36.4125.89MBGSO36.6425.62MDGSO36.8526.18IBFS87.2656.07MBFS57.7040.03MFS93.1560.66GA33.5525.37Figure 3
The affection of other parameters on the performance of CBGSO. (a) Population. (b) Initial range. (c) Maximal range.
(a)(b)(c)


## 4.1. Experimental Results

The proposed HEPCBR achieves the final ensemble using the combination of RE and the proposed CBGSO, and we attempt to verify that whether HEPCBR can perform better than any one of them. So, the predictive results of the proposed HEPCBR are compared to both RE and CBGSO in different sizes of ELMs, as shown in Tables2 and 3. It is easy to see from Tables 2 and 3 that HEPCBR, via the integration of RE and CBGSO, wins a higher precision than any one of them (CBGSO or RE) on the most datasets using less size of ELMs. We can also observe from Tables 2 and 3 that, as the pool size increases, the prediction results achieved by HEPCBR and RE are getting better, but those attained by CBGSO are getting worse. We diagnose the causes, and we find that the computation complexity of the selection of ELMs exponentially increases as the size gets larger. For CBGSO, it is difficult to do an exhaustive search to find the exact subensemble. As a result, the preselection of ELMs can significantly lower the calculation complexity of the selection of ELMs. RE can select some well-performing ELMs, and the postselection of CBGSO can make the ensemble gain better results. Moreover, the classification ability of HEPCBR can attain a very small improvement as the initial size exceeds 150. Hence, 150 ELMs are used in this work.Table 2
Precisions of HEPCBR in comparison with RE and CB.
Datasets50100150REnCBnHEnREnCBnHEnREnCBnHEnHeart74.292576.831978.071277.592574.384480.491381.352575.317183.1213Cleveland56.822556.262559.341058.622554.444760.251057.042553.097557.9515Column86.432587.592090.541188.222586.034690.511490.602586.707292.3814Ecoli87.632588.232388.431288.382588.254988.921389.272588.137589.8812Bupa74.412574.972277.911378.882578.184481.051479.222575.987081.7214Wholesale89.332590.841891.371391.822589.484393.191390.982588.767191.6814Forest92.472593.012393.961192.522591.164693.441494.642591.506994.9415Balance92.722593.122193.041291.842591.365092.481391.842591.767392.8011Diabetes69.412572.271972.551071.782570.674573.621174.332569.686875.1714Vehicle65.852567.261969.951369.432568.504271.251669.772566.146571.5915Tic-tac-toe93.952594.882295.621394.942595.004396.251396.292593.537696.6615German78.602578.102181.201180.902577.504482.801380.902574.207382.2012QSAR87.392586.892189.961289.082587.054791.551190.352584.526992.6013Diabetic-r77.022577.591881.011378.962577.184681.471480.502575.847183.1012Phishing86.392586.532488.541187.602585.824888.781188.532585.547689.3513CMC61.142562.102063.791262.332561.294164.301263.402559.866364.7514Yeast58.742560.532060.561262.572561.984663.991362.232561.397163.2312Wineq-r57.302561.561960.341161.862559.134163.951364.072558.746766.2613Car92.642594.271993.801094.722592.714695.181294.542592.067295.3011CTG81.702584.391685.28985.092580.884287.82985.142579.547787.5410Segment86.122587.971988.341188.272586.884389.441489.192585.976990.4414Abalone57.112559.991958.581160.602557.214161.641258.742559.977159.9612Spambase80.212581.661782.891181.822579.754084.271283.372579.046185.1712Wineq-w47.322551.201550.121051.052548.484153.141351.182548.606953.7612Waveform88.242589.382089.711389.282588.354490.171390.122588.567090.9414HE: HEPCBR; RE: reduce-error; CB: CBGSO.Table 3
Precisions of HEPCBR in comparison with RE and CB.
Datasets200250300REnCBnHEnREnCBnHEnREnCBnHEnHeart79.922572.809282.161380.622571.1412182.291482.302571.1214483.1916Cleveland56.752555.2110057.661160.062554.0212661.381157.892554.4014958.9013Column91.272584.819693.301691.522585.2712192.881592.102584.0914793.7214Ecoli89.802587.479890.531190.172587.7912590.621389.352587.6515090.0510Bupa82.242574.629483.961582.032574.0611984.291482.452575.3114984.0215Wholesale92.902589.209193.641493.392588.0311793.851392.552588.2115093.5914Forest94.382591.669795.041394.632590.5812295.291395.412590.5914595.9515Balance91.442591.049791.601393.122591.1212493.521291.682591.6015392.0812Diabetes72.792569.019874.341374.262569.2012275.751375.252568.9114378.8415Vehicle70.812566.969172.461671.772565.7211673.471573.932564.9713874.5218Tic-tac-toe96.602593.379697.281396.192592.4811996.761397.502592.4314697.6516German82.002576.409783.101281.702576.9012183.101480.502572.2015281.9013QSAR90.582587.089592.581389.212583.3911990.911190.472583.7014191.5215Diabetic-r80.422573.859582.591383.422575.7812185.331681.992575.6614083.4616Phishing90.532585.4610091.201390.622586.2712391.291390.522586.8615090.8214CMC64.292559.349065.651365.922558.0811767.071564.792558.6514466.0114Yeast64.802559.239765.541262.632559.3312363.181363.542558.1014864.0813Wineq-r62.582556.929665.281362.232556.4711763.991363.682559.2114965.4914Car93.812591.609694.391294.142591.7412394.611294.612592.2914695.1312CTG86.452580.209887.911188.142580.3912689.231488.402579.9414989.7613Segment89.302585.439790.301289.772586.0811890.581589.532585.5714390.1815Abalone60.042559.829560.761260.552558.2111961.161460.872557.8114562.0112Spambase83.732578.949085.341484.702579.1711085.911483.852576.4814085.0615Wineq-w52.422546.549654.391349.122548.2412851.681151.432546.8614953.2814Waveform90.042587.789990.781590.092587.3712290.661589.462587.8614890.0616HE: HEPCBR; RE: reduce-error; CB: CBGSO.


## 4.2. Comparison with Other Methods

To test the classification ability of HEPCBR further, massive experiments were implemented by comparing it with other techniques: bagging [51], kappa [28], AGOB [26], POBE [27], DREP [22], DEELM [39], GASEN [33], GSOEP [34], MOAG [30], RREP [10], DMEP [40], EPSCG [20], DASEP [24], MDOEP [31], RCOA [32], and PEAD [41]. Bagging extracts the training samples with equal probability, and it can construct an initial pool composed of multiple learners with a good diversity. Kappa pruning achieves the ensemble by grouping the learners with their measures under a certain value. AGOB, POBE, and MOAG find the ordering of those learners in the ensemble works well. DREP first selects learner with the best classification ability and gradually increases the size of the ensemble which can make the ensemble achieve better precisions. DEELM chooses the learners with their measures falling into an interval. GASEN can assign a random weight to each learner. It adopts GA to evolve their parameters, and the learners whose weights exceed a predefined value can be utilized for combination. GSOEP employs GSO to directly filter out the redundant learners. RREP attained the ensemble by utilizing the retained learners and the removed ones gained by RE. DMEP employs GA to optimize five pairwise diversity matrices for a combining diversity measure, and select a part of learners in the light of the graph coloring theory. EPSCG evaluates the learners’ contribution to the ensemble in terms of the Banzhaf index of power. DASEP finds new measures by simultaneously considering diversity and accuracy of learners, and acquires a good result in the selection of learners. MDOEP attains the final ensemble by the aggregation using margin and diversity-based measure. RCOA ranks all the learners based on the relevancy and complementary measures in a descending order, and the final ensemble is achieved. PEAD selects a collection of learners extracted from a pool using the induction of decision tree, which are used to form the final ensemble.We gain the classification precisions of the proposed HEPCBR, and other methods with 150 ELMs in the pool are displayed in Tables4 and 5. “+/ = /−,” respectively, indicates that HEPCBR achieves higher/neutral/lower precisions than other methods. It is clearly shown in Tables 4 and 5 that the proposed HEPCBR outperforms other state-of-the-art techniques on the most classification tasks, which shows its enhancements in classification. Table 6 displays the sizes of the final ensemble attained by all the pruning approaches. As shown in Table 6, HEPCBR utilizes less the size of ELMs to build the ensemble, but it uses more ELMs than DMEP. Additionally, HEPCBR gains better results than DMEP. As a whole, the proposed HEPCBR has acquired the expected results in classification.Table 4
Classification precisions won by other methods with 150 ELMs.
DatasetsHEPCBRBaggingKappaAGOBPOBEDREPDEELMGASENGSOEPHeart83.1268.5770.0775.9075.5172.6272.8974.8774.97Cleveland57.9553.5252.4554.8853.8753.2653.5254.5852.77Column92.3880.4084.3187.6786.9984.6183.3685.6285.10Ecoli89.8887.1687.9588.8489.0387.7288.2588.4287.98Bupa81.7268.2768.4375.0375.3072.5571.9373.2274.34Wholesale91.6885.7888.9290.1390.1688.3288.5987.9288.72Forest94.9489.0591.3393.3892.8290.9391.2291.1591.52Balance92.8090.4891.3692.7292.6491.9292.2891.6891.52Diabetes75.1766.1269.5772.4072.3071.0670.3570.3970.13Vehicle71.5957.8460.4466.8364.3864.5864.0263.4065.35Tic-tac-toe96.6690.9794.6596.1194.8493.1493.4092.4992.59German82.2072.3075.2579.5078.5076.8576.7576.9076.70QSAR92.6083.2184.8287.5587.0885.2885.2884.8285.59Diabetic-r83.1069.0473.0776.6275.8974.1573.5874.0975.14Phishing89.3584.3985.6687.2186.8485.7085.7088.0787.18CMC64.7554.3153.5358.5957.7456.5556.2459.6160.28Yeast63.2358.0357.8560.5460.1659.1159.1859.7759.22Wineq-r66.2655.8955.9359.4959.3957.8957.0958.0257.63Car95.3091.2992.7493.5293.4992.4892.5493.6592.89CTG87.5478.6282.1085.7283.5181.7582.0884.7880.09Segment90.4484.3585.5387.2486.8685.6685.7086.3686.71Abalone59.9655.4056.5258.5358.4657.4057.2657.9757.25Spambase85.1773.6178.8781.7779.7078.4977.5576.8377.40Wineq-w53.7646.5146.8948.5948.0247.8546.9349.0848.16Waveform90.9486.3587.6688.5688.5887.8287.5789.0289.41+/ = /−17/3/50/0/250/0/250/0/250/0/250/0/250/0/250/0/250/0/25Table 5
Classification precisions won by other methods with 150 ELMs.
DatasetsMOAGRREPDMEPEPSCGDASEPMDOEPRCOAPEADHeart71.6381.4078.5176.1278.8577.5277.3876.54Cleveland53.2656.2456.6054.8155.8455.7656.9257.88Column83.4391.4388.7686.9486.2785.8286.3985.98Ecoli87.5989.6089.8387.1489.9686.9788.4689.22Bupa70.4579.6177.3075.1576.6477.9776.1175.26Wholesale87.6992.0591.2487.4889.8287.7688.5287.50Forest90.5092.8092.8390.4594.9494.9494.8293.75Balance91.9693.0093.3291.7293.2091.4091.6892.44Diabetes69.3775.8974.5672.5272.6671.3871.6270.27Vehicle60.3271.0867.5766.1666.5967.6368.8665.27Tic-tac-toe92.3896.6695.4196.6694.2996.6695.1794.92German75.8581.5081.1577.3078.6078.7078.0077.15QSAR84.4890.2988.9786.5887.5088.3687.3386.52Diabetic-r72.4681.4878.7977.2477.8079.9978.0876.34Phishing85.7487.9189.0685.1188.0386.5286.7787.65CMC55.8763.3161.4458.9559.1558.1258.0657.60Yeast59.4662.1162.3258.6060.9957.9958.9859.63Wineq-r57.7463.8062.8057.0860.8659.6859.0957.82Car92.4294.6594.7192.1393.9691.4793.7993.13CTG81.2187.5486.9082.5485.2187.5486.3987.54Segment85.1389.0387.4984.9286.3086.1085.6384.13Abalone58.1960.3360.1357.3358.9359.2060.5059.96Spambase76.3184.1582.2478.8379.4581.3479.1079.91Wineq-w47.8951.5751.6548.9249.9449.4850.4049.52Waveform87.4390.2689.3687.2888.1789.3488.6788.29+/ = /−0/0/252/2/201/0/240/1/241/1/230/3/221/0/240/1/24Table 6
Sizes of the selected ELMs via other methods with 150 ELMs.
DatasetsHEBaKaAGPODRDEGAGSMORRDMEPDAMDMRPEHeart1315030173075775671301853129302637Cleveland1515030212975613776231721526302522Column1415030233075814471352141713303247Ecoli1215030202975765279261621915302532Bupa1415030253275827170262431923302452Wholesale1415030203175815465331731219302932Forest1515030183075836971311892730302623Balance111503020307577617281621724302322Diabetes1415030202975762869351665014302427Vehicle1515030223175826772321986930302341Tic-tac-toe1515030153175827369311783629301939German1215030153075793973361653117301829QSAR1315030193075816772272191315302822Diabetic-r1215030223275815062341771514302947Phishing1315030193175782973241742134303035CMC1415030263475815860241883813302323Yeast1215030203175793674311861118302623Wineq-r1315030203275802969351841733302324Car1115030183075794073261653821302626CTG1015030172575673155321645117302323Segment1415030233275826670302094519302423Abalone1215030263275791465282151631302328Spambase1215030162875724471331869816301924Wineq-w1215030263275712768292044017302432Waveform1415030243175817161362152924302529HE: HEPCBR; Ba: bagging; Ka: kappa; AG: AGOB; PO: POBE; DR: DREP; DE: DEELM; GA: GASEN; GS: GSOEP; MO: MOAG; RR: RREP; DM: DMEP; EP: EPSCG; DA: DASEP; MD: MDOEP; RC: RCOA; PE: PEAD.In this section, we try to estimate the difference significance between the predictive accuracies gained by HEPCBR and that attained by other comparative methods, and we used the Wilcoxon rank sum test [40] with its level of 0.05. When the p value of two techniques is lower than 0.05, it expresses that there exists significant difference between them. Table 7 exhibits the p values acquired in the test between HEPCBR and other comparative approaches. Table 7 indicates that the p values have been below 0.05. It manifests the significance and the effectiveness of HEPCBR.Table 7
Thep values of the Wilcoxon rank sum test.
Comparisonp valueSignificant or notHEPCBR vs bagging0.00123YesHEPCBR vs kappa0.00123YesHEPCBR vs AGOB0.00123YesHEPCBR vs POBE0.00123YesHEPCBR vs DREP0.00123YesHEPCBR vs DEELM0.00123YesHEPCBR vs GASEN0.00123YesHEPCBR vs GSOEP0.00123YesHEPCBR vs MOAG0.00123YesHEPCBR vs RREP0.02327YesHEPCBR vs DMEP0.00286YesHEPCBR vs EPSCG0.00182YesHEPCBR vs DASEP0.00267YesHEPCBR vs MDOEP0.00401YesHEPCBR vs RCOA0.00157YesHEPCBR vs PEAD0.00270YesThe execution time of the proposed HEPCBR and other comparative techniques is reported in Table8. The observations from Table 8 flag up that HEPCBR uses much less time than GASEN, GSOEP, and DMEP, but utilizes more time than the retaining comparative methods. In HEPCBR, CBGSO is employed to extract the final ELMs from the remaining ones after pruning, which needs to calculate the classification accuracy of each candidate subensemble at each iteration. During the search of CBGSO, each candidate subensemble is expressed by a glowworm, and each glowworm needs to calculate its fitness value at least once per iteration. Therefore, it requires much more time. Those techniques with less running time achieve the selection of ELMs without repetitive iteration process and assessment. Nevertheless, they cannot attain good results in classification. HEPCBR cost less time than the mentioned three methods, it is the reason that those approaches directly selected the ELMs with good comprehensive performance without preselection, and it might not be enough to seek for the exact solution due to the enormous amount candidate subensembles. Moreover, the preselection process can notably alleviate the computational burden of the selection of ELMs. It is concluded that HEPCBR consumes much more time, but the remarkable improvements in classification are obtained.Table 8
Execution time of different techniques.
DatasetsExecution time (s)HEBagKaAGPODRDFGAGSOMORREDivPSCSDMDMRPEHeart6.320.470.504.090.491.270.5116.2216.710.504.2318.120.500.650.561.390.49Cleveland25.290.420.448.090.453.480.4539.8039.340.598.1942.480.590.820.551.670.42Column6.730.510.544.700.531.420.5518.0418.340.554.7823.520.550.700.631.470.51Ecoli34.640.450.4712.530.485.090.4957.5757.970.7212.5965.550.721.040.741.940.54Bupa7.720.480.515.170.491.530.5220.0719.470.535.8222.430.530.700.601.670.57Wholesale9.560.690.736.590.691.920.7425.2624.820.746.6726.170.740.950.871.830.81Forest24.430.810.847.730.823.670.8540.5239.360.988.2145.870.971.190.992.230.85Balance21.460.870.897.310.903.540.9038.7538.761.017.5147.921.011.221.052.171.01Diabetes10.540.690.736.760.711.980.7425.8525.510.756.8333.710.750.970.851.840.71Vehicle37.701.111.1612.501.235.941.1764.7063.031.3912.9769.211.381.751.333.701.48Tic-tac-toe12.481.501.559.351.603.161.5632.1732.371.589.5143.531.571.851.753.582.27German6.651.271.295.291.292.101.3019.0619.031.305.8225.281.301.451.372.021.07QSAR7.221.651.685.951.652.581.6920.0320.561.696.0526.671.691.861.792.361.38Diabetic-r7.591.151.195.941.162.211.2021.3821.791.205.9423.171.201.381.282.271.17Phishing23.252.292.329.942.355.482.3349.9347.892.4710.2159.352.472.722.502.911.66CMC26.572.222.2610.832.335.842.2752.9953.622.4210.9867.292.422.702.392.931.27Yeast59.841.061.0919.421.168.111.1087.8688.761.4720.3998.211.471.951.363.040.95Wineq-r45.942.002.0416.532.207.802.0577.2376.342.3316.9186.122.332.742.243.861.80Car36.043.083.1214.123.387.533.1364.0462.533.3414.5369.183.333.663.394.452.86CTG36.071.531.5712.421.586.031.5864.7566.941.7812.6765.621.772.111.823.261.45Segment81.083.203.2527.393.1912.773.26115.91115.463.7628.15135.403.754.423.747.424.10Abalone47.833.813.8718.403.919.873.8890.2588.634.1518.4999.824.144.604.117.485.03Spambase19.305.495.5817.895.558.215.5957.6957.975.6118.1168.335.616.055.857.875.73Wineq-w85.034.914.9931.504.9615.525.00140.39138.235.5232.32165.145.516.265.337.994.60Waveform58.346.366.4423.666.6113.636.45110.49107.946.7625.01131.156.767.306.859.076.26HE: HEPCBR; Ba: bagging; Ka: kappa; AG: AGOB; PO: POBE; DR: DREP; DE: DEELM; GA: GASEN; GS: GSOEP; MO: MOAG; RR: RREP; DM: DMEP; EP: EPSCG; DA: DASEP; MD: MDOEP; RC: RCOA; PE: PEAD.The aforementioned comparative methods are based on bagging, which are attained using diversity measures with different strategies or heuristic algorithms. In addition, we further compare the proposed HEPCBR with the more state-of-the-art approaches by considering other factors, which are not the same category with the aforementioned comparative methods: C-RoF [17], CS-D-ELM [18], KPCA-RoF [4], and DyPReVNsGraspEnS [38]. C-RoF introduces misclassification, test, and rejection costs for overcoming that some existing works neglect the classification costs, and it achieves a good result. CS-D-ELM extends the D-ELM employing misclassification cost of the learner and embeds rejection cost into it to increase the classification stability. KPCA-RoF integrates the KPCA and RoF for linearly inseparable dataset classification problems, and it enhances the classification capacity of RoF on the nonlinear fractional datasets. DyPReVNsGraspEnS realizes random multistart search, and it can avoid falling into local optimal solutions and possesses a high probability to exactly find the ensemble with better performance. The classification results achieved by different approaches are presented in Table 9. We can see from Table 9 that the accuracies gained by HEPCBR are higher than other four state-of-the-art methods. Then, we verify the significance between HEPCBR and other four approaches using the Wilcoxon rank sum test, which is displayed in Table 10. From Table 10, all the p values are less than 0.05. So, we can conclude that the proposed HEPCBR has powerful advantages over other four techniques in classification, and it is significant and effective.Table 9
Comparison of classification performance with other state-of-the-art methods.
DatasetsHEC-RoFCS-DKPCADyPHeart83.1278.6277.5579.2180.84Cleveland57.9557.0557.6656.0557.86Column92.3889.0788.4888.7389.43Ecoli89.8890.3689.8289.6590.00Bupa81.7277.5277.6978.2280.46Wholesale91.6889.3891.3691.5892.84Forest94.9492.5893.7094.0893.28Balance92.8092.8092.4892.4892.80Diabetes75.1774.7174.6075.4174.81Vehicle71.5968.1868.9168.4370.56Tic-tac-toe96.6694.3295.5195.5796.57German82.2078.0079.2078.4080.80QSAR92.6089.9489.3290.3290.81Diabetic-r83.1075.2177.6476.6180.76Phishing89.3588.3589.8389.2489.71CMC64.7561.0261.2860.8862.28Yeast63.2361.6562.7361.6562.41Wineq-r66.2663.6361.6362.2561.50Car95.3094.4494.9193.8594.37CTG87.5484.1885.8784.7486.53Segment90.4487.3087.9188.0189.72Abalone59.9657.9357.7258.7858.36Spambase85.1780.5481.4481.2283.28Wineq-w53.7650.2351.3052.5851.21Waveform90.9488.4588.7088.6888.84HE: HEPCBR; CS-D: CS-D-ELM; KPCA: KPCARoF; DyP: DyPReVNsGraspEnS.Table 10
Thep values of the Wilcoxon rank sum test.
Comparisonp valueSignificant or notHEPCBR vs C-RoF0.00235YesHEPCBR vs CS-D0.00254YesHEPCBR vs KPCA0.00200YesHEPCBR vs DyP0.01716YesCS-D: CS-D-ELM; KPCA: KPCA-RoF; DyP: DyPReVNsGraspEnS.


## 4.3. Parameter Analysis

In HEPCBR, we employ CBGSO to select the ELMs with good diversity and high accuracy from the remaining ones after prepruning. To make the proposed CBGSO perform well, we try to analyze its parameters. In addition, to assess its convergence speed and precision, the comparative analyses have been presented as follows: MGSO [48], MBGSO [47], MDGSO [34], IBFS [50], MBFS [53], MFS [54], and GA [33]. We implemented the experiments on two datasets, and we adopt different methods in search of the optimal subensemble from the remaining ELMs using RE with the pool size of 150.It is clearly observed in Figure2 that, as a whole, CBGSO attains better convergence accuracy and higher evolution velocity than other seven comparative algorithms, i.e., CBGSO performs better than others in terms of the speed and the precision of convergence. It is also easy to find that the accuracies acquired by CBGSO rise up and stay stable. The results achieved by CBGSO just can attain a slight gain in classification, when the number of iterations is above 400. So, it is 400. In addition, when CBGSO is utilized to search for the final ensemble in HEPCBR, it costs less time than other seven heuristic algorithms, which is displayed in Table 11. Table 11 demonstrates that CBGSO performs higher efficiency of searching than other heuristic algorithms. Hence, CBGSO can work well when it comes to searching speed and efficiency. The accuracies mount and level off, as the population size increases, which is rendered in Figure 3(a). We can see from Figure 3(a) that, when the population size of glowworms is above 25, it will greatly increase the computation complexity with less improvements in classification. Hence, the size is 25. Figure 3(b) analyzes the affection of the initial range on the performance of CBGSO. We can observe from Figure 3(b) that it performs at its best when the initial range is 11. Hence, the initial range is 11. Figure 3(c) reveals the affection of the maximal one on the performance of CBGSO. It is more than the initial one. The observations from Figure 3(c) disclose that the performance of CBGSO peaks at 15. So, we advise it is set as 15.Figure 2
The affection of iteration on the convergence rate of algorithms. (a) Vehicle. (b) CMC.
(a)(b)Table 11
Comparison of execution time with different heuristic algorithms.
MethodsVehicle (s)CMC (s)CBGSO33.1625.09MGSO36.4125.89MBGSO36.6425.62MDGSO36.8526.18IBFS87.2656.07MBFS57.7040.03MFS93.1560.66GA33.5525.37Figure 3
The affection of other parameters on the performance of CBGSO. (a) Population. (b) Initial range. (c) Maximal range.
(a)(b)(c)


## 5. Conclusions

Ensemble pruning extracts a subset of ELMs extracted from a constructed pool of ELMs for achieving better predictive results and efficiency. To make an ensemble perform well, it must consist of accurate and diverse ELMs. Attaining that is a combinatory optimization question, and it has high complexity. To address this issue, HEPCBR is presented via the fusion of RE and CBGSO, the prepruning strategy using RE can avoid heavy computing burdens of the selection of ELMs, and then the ELMs with good comprehensive performance can be efficiently selected by CBGSO. Experimental study utilizing 25 UCI datasets has demonstrated that the proposed HEPCBR outperforms bagging and other algorithms, and it also indicates the effectiveness and significance. Furthermore, the presented CBGSO can look for the better results than other methods. The proposed HEPCBR brings forward a new study way for the pruning question of ELMs.With respect to future research directions, we aim to design a new combining diversity measure, which can minimize the redundant ELMs. Therefore, it can provide these well-performing ELMs for further pruning.



---

*Source: 1329692-2020-10-31.xml*