# A Novel SMOTE-Based Classification Approach to Online Data Imbalance Problem

Authors: Chunlin Gong; Liangxian Gu
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

In many practical engineering applications, data are usually collected in online pattern. However, if the classes of these data are severely imbalanced, the classification performance will be restricted. In this paper, a novel classification approach is proposed to solve the online data imbalance problem by integrating a fast and efficient learning algorithm, that is, Extreme Learning Machine (ELM), and a typical sampling strategy, that is, the synthetic minority oversampling technique (SMOTE). To reduce the severe imbalance, the granulation division for major-class samples is made according to the samples’ distribution characteristic, and the original samples are replaced by the obtained granule core to prepare a balanced sample set. In online stage, we firstly make granulation division for minor-class and then conduct oversampling using SMOTE in the region around granule core and granule border. Therefore, the training sample set is gradually balanced and the online ELM model is dynamically updated. We also theoretically introduce fuzzy information entropy to prove that the proposed approach has the lower bound of model reliability after undersampling. Numerical experiments are conducted on two different kinds of datasets, and the results demonstrate that the proposed approach outperforms some state-of-the-art methods in terms of the generalization performance and numerical stability.


---

## Body

## 1. Introduction

With more and more successful real applications, machine learning acting as an efficient technique of data analysis and modeling is now becoming an important research field in the area of aeronautics science and mechanical engineering, for example, building surrogate model dynamically to support system design or identifying the characteristics of system by giving samplings. Among many concrete topics, online learning and imbalanced classification are both receiving a lot of attentions, and there also are many researches in the last decade for these two issues separately. However, to the best of our knowledge, there are few studies about the incorporation of these two topics, that is, data imbalance problem in online learning procedure. We also name it as online data imbalance problem which can be widely found in many real engineering applications such as fault diagnosis and damage detection. Therefore, studying this problem is of great significance.In this paper, we try to provide an efficient solution from the perspectives of sampling strategy and learning algorithm. Considering the whole distribution characteristic of dataset and the feature of online learning, we present a novel SMOTE-based classification approach to the online data imbalance problem. This approach borrows the idea of granulation division to conduct oversampling and undersampling simultaneously. For major class, we use granule cores to replace the original samples for undersampling, while, for minor class, we firstly make granulation division and then conduct oversampling using SMOTE in the region around granule core and granule border. The above strategy is capable of following the data distribution of sample set, so it can be conducted repeatedly in the offline and online stages to balance the total sample set easily. Moreover, we introduce an efficient online learning algorithm, online sequential extreme learning machine (OS-ELM), which is combined with the proposed sampling strategy to achieve the fast and robust online learning for imbalanced data. To testify the effectiveness of the proposed approach, we first prove theoretically by means of the fuzzy information entropy that the proposed approach has the lower bound of model reliability after undersampling. And the experimental results on two different kinds of datasets also demonstrate the comparative performance of the proposed approach.


## 2. Related Works

Nowadays, the researches for the traditional data imbalance problem focus on two strategies [1]. One is data-based strategy which aims to make the original dataset balanced using mainly undersampling or oversampling techniques. The key idea of this kind of method is how to explore and obey the inner distribution characteristic of sample set in the sampling procedure. The synthetic minority oversampling technique (SMOTE) [2] is a widely used technique due to its simple form and straight idea, but it also suffers from the relatively low accuracy because SMOTE is incapable of exploiting the data distribution of sample set, especially in online case. Therefore, many SMOTE-based methods were developed with integrating other techniques. To improve the quality of synthetic samples, Verbiest et al. [3] introduced fuzzy-rough selection algorithm to reduce the noise generated by SMOTE after the balance stage. Gao et al. [4] utilized particle swarm optimization to optimize the undersampling procedure of SMOTE and then introduced RBF classification to reduce the misclassified cases. To reduce the imbalance level between classes, Zeng et al. [5] integrated the kernel trick and SMOTE into a new support vector machine (SVM) algorithm for data imbalance problem. As a combination with learning algorithm, Jeatrakul et al. [6] introduced SMOTE to neural networks in order to improve the generalization performance. Granular computing, known as an abstract idea for data processing, has instinct capacity to effectively remodel the original data according to data distribution, and then a high value of keeping the raw information of sample set is put. Therefore, granular computing has also been introduced to solve the data imbalance problem in SVM. For example, Wang et al. [7] utilized granulation division to hierarchically suppress the samples before SVM training. It is worth noticing that although the methods discussed above can balance sample set to some extent and thus improve the classification accuracy, they easily cause severe information loss if the distribution characteristic and feature are not considered well.The other strategy is algorithm-based strategy which tries to improve the classification efficiency by developing the algorithms structure. For example, Hwang et al. [8] added weight factors to Lagrange multiplayer to improve the effectiveness of SVM upon facing imbalanced data. To lessen the misclassification rate, Yu et al. [9] calculated the moving distance of hyperplane by adjusting the decision threshold of SVM. Many other algorithms such as price-sensitive learning [10], weighted support vector machine [11], and weighted boost learning [12] were devoted to solve data imbalance problem. Although this strategy has been researched thoroughly, most of the algorithms can not apply to the online case directly due to lack of online structure. Besides, upon facing a large amount of data, it is generally hard for these algorithms to get results quickly. As extension form of single-hidden layer feedforward neural network (SLFN), extreme learning machines (ELMs), introduced by Huang et al. [13], have been recognized by their high learning speed and good generalization capacity for solving many problems of regression estimate and pattern recognition. As a sequential extension of ELM, online sequential ELM (OS-ELM) proposed by Liang et al. [14] can learn data one by one or chunk by chunk with fixed varied chunk size at very high speed. Although ELM has also been developed for data imbalance problem [15], it seems that OS-ELM has not been widely applied to data imbalance problems.According to our literature survey, there are not too many researches about online data imbalance problem. By introducing prior duplication strategy, Vong et al. [16] firstly generated synthetic minority class samples and then utilized OS-ELM to establish an online sequential prediction model. Focusing on the modeling of data distribution in online pattern, Mao et al. [17] introduced the principal curve to exploit the inner structure of online data and then applied SMOTE to conduct oversampling by means of the distance from sample to principal curve. However, although this method could overcome many shortages of traditional methods, the principal curve is not well applicable to tackle the dataset with no apparent distribution feature. We noticed another recent work [18] for this problem which tried to adopt granulation division to remodel the distribution characteristic with a theoretical analysis about concrete information loss. Although it neglects some potential shortage of synthetic samples and the theoretical analysis needs to be improved largely, it is still an interesting attempt at this problem with reference value.


## 3. Background

### 3.1. SMOTE

SMOTE (synthetic minority oversampling technique) is a common oversampling method proposed by Chawla et al. [2]. In the SMOTE, instead of mere data oriented duplicating, the minority class is oversampled by creating synthetic instances in the feature space formed by the instance and its K-nearest neighbors, which effectively avoid the overfitting problem.This method is described as follows. Choose two samples,x1 and x2, from the given minority sample set randomly, where each sample has n attributes. For x1 and x2, calculate the difference on the ith attribute; that is, diffi=x2i-x1i. Then, we obtain the ith attribute value of the new target sample according to(1)x12i=rand⁡0,1∗diffi,where rand⁡[0,1] means a random number between 0 and 1. So the final synthetic sample of x1 and x2 is(2)x12=rand⁡0,1∗diff,where diff=(diff1,diff2,…,diffn).According to the sampling rate we set execution times and repeat the above process. Incorporating the synthetic samples and the original samples, the final minority sample set is obtained.


### 3.2. Review of ELM and OS-ELM

As originally proposed for solving the single-hidden layer feedforward neural network (SLFN), it has been proved that, with at mostN hidden neurons, ELM can learn N distinct samples with zero errors by adopting any bounded nonlinear activation function [19]. Then, based on this approximation ability, ELM received wide attentions and has been developed into various forms, for example, multioutput regression [20]. The most important feature of ELM is its fast speed, owing to its single-hidden layer structure requiring no iterative process. In ELM, all the hidden node parameters are randomly generated without tuning. As an extension version of ELM, online sequential extreme learning machine (OS-ELM) is a faster and more accurate algorithm, which has been widely used in many fields, such as pattern recognition and data mining. The process of OS-ELM is divided into two steps: initialization phase and sequential learning phase and the detailed algorithm is described as follows [14].Step 1 (initialization phase).
Choose a small chunkM0={(xi,ti),i=1,2,…,N0} of initial training data, where N0≥N~. Consider the following:(1)
Randomly generate the input weightwi and bias bi,  i=1,2,…,N~. Calculate the initial hidden layer output matrix H0.(2)
Calculate the output weight vector:(3)β0=D0H0TT0,whereD0=(H0TH0)-1 and T0=[t1,t2,…,tN0]T.(3)
Setk=0 24.Step 2 (sequential learning phase).
Consider the following:(1)
Learn thek+1th training data: dk+1=(xN0+k+1,tN0+k+1).(2)
Calculate the partial hidden layer output matrix:(4)Hk+1=gw1·xN0+k+1+b1⋯gwL·xN0+k+1+bL1×L.SetTk+1=[tN0+k+1]T.(3)
Calculate the output weight vector(5)Dk+1=Dk-DkHk+1TI+Hk+1DkHk+1T-1Hk+1Dk,βk+1=βk+Dk+1Hk+1TTk+1-Hk+1βk.(4)
Setk=k+1. Go to Step  2(1).


## 3.1. SMOTE

SMOTE (synthetic minority oversampling technique) is a common oversampling method proposed by Chawla et al. [2]. In the SMOTE, instead of mere data oriented duplicating, the minority class is oversampled by creating synthetic instances in the feature space formed by the instance and its K-nearest neighbors, which effectively avoid the overfitting problem.This method is described as follows. Choose two samples,x1 and x2, from the given minority sample set randomly, where each sample has n attributes. For x1 and x2, calculate the difference on the ith attribute; that is, diffi=x2i-x1i. Then, we obtain the ith attribute value of the new target sample according to(1)x12i=rand⁡0,1∗diffi,where rand⁡[0,1] means a random number between 0 and 1. So the final synthetic sample of x1 and x2 is(2)x12=rand⁡0,1∗diff,where diff=(diff1,diff2,…,diffn).According to the sampling rate we set execution times and repeat the above process. Incorporating the synthetic samples and the original samples, the final minority sample set is obtained.


## 3.2. Review of ELM and OS-ELM

As originally proposed for solving the single-hidden layer feedforward neural network (SLFN), it has been proved that, with at mostN hidden neurons, ELM can learn N distinct samples with zero errors by adopting any bounded nonlinear activation function [19]. Then, based on this approximation ability, ELM received wide attentions and has been developed into various forms, for example, multioutput regression [20]. The most important feature of ELM is its fast speed, owing to its single-hidden layer structure requiring no iterative process. In ELM, all the hidden node parameters are randomly generated without tuning. As an extension version of ELM, online sequential extreme learning machine (OS-ELM) is a faster and more accurate algorithm, which has been widely used in many fields, such as pattern recognition and data mining. The process of OS-ELM is divided into two steps: initialization phase and sequential learning phase and the detailed algorithm is described as follows [14].Step 1 (initialization phase).
Choose a small chunkM0={(xi,ti),i=1,2,…,N0} of initial training data, where N0≥N~. Consider the following:(1)
Randomly generate the input weightwi and bias bi,  i=1,2,…,N~. Calculate the initial hidden layer output matrix H0.(2)
Calculate the output weight vector:(3)β0=D0H0TT0,whereD0=(H0TH0)-1 and T0=[t1,t2,…,tN0]T.(3)
Setk=0 24.Step 2 (sequential learning phase).
Consider the following:(1)
Learn thek+1th training data: dk+1=(xN0+k+1,tN0+k+1).(2)
Calculate the partial hidden layer output matrix:(4)Hk+1=gw1·xN0+k+1+b1⋯gwL·xN0+k+1+bL1×L.SetTk+1=[tN0+k+1]T.(3)
Calculate the output weight vector(5)Dk+1=Dk-DkHk+1TI+Hk+1DkHk+1T-1Hk+1Dk,βk+1=βk+Dk+1Hk+1TTk+1-Hk+1βk.(4)
Setk=k+1. Go to Step  2(1).


## 4. Online Sequential Extreme Learning Machine Based on Granulation Division and SMOTE

To improve the classification accuracy of minority class, we proposed a new algorithm based on granulation division and SMOTE using extreme learning machine. The main idea is improving the accuracy of minority class and reducing the information loss of majority class.For the convenience of description of the algorithm, we give some definitions in the beginning. Suppose thatD={(xi,ti),i=1,2,…,N1} and S={(yi,ti),i=1,2,…,N2} represent majority sample set and minority sample set, respectively, where xi and yi mean m-dimensional vector. Dimension indicates the number of features. ti=1 means the corresponding sample is the majority and ti=0 means minority sample.Definition 1 (maximum radius of granule).
In thejth granule, the distance between the granule core and the furthest point is called maximum radius of granule of the jth granule:(6)Rj=maxxij∈XC-X,where C=(c1,c2,…,ck1), j=1,2,…,k1, and cj is the coordinate of granule core.Definition 2 (granule dispersion).
Granule dispersion represents the discrete degree in a granule. Obviously, the granule dispersion is inversely proportional to the number of samples in a granule and is directly proportional to the maximum radius of granule:(7)1Separatecj=Rj∗∑i=1kpcj-xijkp3=Rj∗∑i=1kpcj2-2xijcj+xij2kp3,where xij is the sample in the granule with cj as the granule core. It is easy to know that the bigger the granule dispersion is, the more sparse and scattered the samples in the granule are and thus the higher the information loss is upon using granule core instead of the whole granule.Definition 3 (sample weight).
For each sample in the granule except the samples farthest from the granule core, the oversampling is conducted using SMOTE. Obviously, the sample weight is inversely proportional to the distance between the granule core and the virtual samples:(8)wi=1-cj2-2xijcj+xij2Rj.


### 4.1. Offline Stage

Firstly, we refactor the imbalanced sample set using the proposed method and get the balanced sample setA={(xi,ti)∣i=1,2,…,N0}. Then establish the initial model. The main idea is undersampling for majority class by choosing all the granule cores, which can reduce the number of majority samples and ensure that the samples’ distribution trend is consistent with the trend before undersampling.For initial majority sample setD={(xi,ti),i=1,2,…,N1}, the first granulation division is conducted. Then, we obtain the new majority sample set Dn={(xi,ti),i=1,2,…,N11}.Clustering algorithm [21] is adopted to simulate the process of granulation division. We set the clustering algorithm k1 in the first granulation division according to the overall distribution of original samples. The up and down threshold values of maximum radius of granule are set to [η1,η2], which can guarantee that the sample distribution trend keeps unchanged before and after the first undersampling. Then, we choose k1 clustering center as the granule core and replace the original majority samples. Merging the new majority sample set Dn={(xi,ti),i=1,2,…,N11} and the minority sample set S={(yi,ti),i=1,2,…,N2}, we obtain the new training sample set A={(xi,ti)∣i=1,2,…,N0}.Given the hidden active functiong(x) and the number of hidden nodes L, choose input weight wi and bias bi, i=1,2,…,N0 randomly and calculate the hidden layer output matrix H0:(9)H0=hx1hx2⋮hxN0=gw1·x1+b1⋯gwL·x1+bLgw1·x2+b1⋯gwL·x2+bL⋮gw1·xN0+b1⋯gwL·xN0+bL.The output vector is T0=t1,t2,…,tN0T and the output weight is(10)β0=H0†T0,where H0†=(H0TH0)-1H0T. Set P0=(H0TH0)-1, and then we have H0†=P0H0T.Setk=k+1.


### 4.2. Online Stage

Suppose that the sequential samples chunk isΩk+1={(xi,ti)∣i=N0+1+k,…,N0+1+k+Block} in (k+1)th step, where Block means the number of samples in Ωk+1. Ωk+1 is divided into majority class Ωd and minority class Ωs according to the value of label ti.(1) Granulation division for majority classisΩd is conducted. We choose k1 points as the initial granule core uniformly according to the samples distribution trend, where k1 is set as about three times the number of minority samples. We can obtain k1 clustering center in each iteration, namely, granule core, by equation of clustering algorithm μi(j+1)=1/ni∑i=1nipit, until the distance between each sample and the clustering center meets the condition of the following equation. Finally, we can obtain k1 clustering center C={c1,c2,…,ck1}, namely, k1 granule core. Now, the majority sample set is Ωdn and the imbalance rate reduces to 3 : 1:(11)min∑∑dist⁡ci,x2.(2) Granulation computing for minority class: the value ofk2 is set to half the number of original minority samples:(12)k2=12sizeΩs.If there are other samples except the granule core, we add the virtual samples within the granule and granule boundary using SMOTE. Then, we obtain the new minority sample set by merging the virtual samples and original minority samples. The detailed description is as follows.Step 1.
Choose the granule core as the center of a circle. We add virtual samples between the granule core and all the other samples using SMOTE, as shown in the following equation:(13)xjim′=cj+diffi×rand⁡0,1=cj+cj-xji×rand⁡0,1.The virtual samples are generated between the granule core and other samples in the granule. Every time there are m virtual samples generated. The value of m1 is set according to the actual situation.Step 2.
According to the following equation:(14)xjim+1′=cj+diffi×rand⁡1,n1=cj+cj-xji×rand⁡1,n1,
m
2 virtual samples will be generated between the granule core and most of the samples in the granule, except the sample farthest from the granule core, which can ensure that the new virtual samples are not too far from the granule core and thus maintain credibility. Usually we set m2≤1/3m1, where m1 means the total sample numbers in the granule. The new virtual samples expand the distribution range of the granule and do not affect the overall credibility simultaneously. Besides, the random number is between 1 and n1, 1<n1≤1.5, which can ensure that the distance between new virtual samples and the granule core is farther than the distance between raw samples and the granule core, and most of the virtual samples are still in the granule.Step 3.
Set the sample weight of virtual samples according to Definition3 and then update the virtual samples. Merging the virtual samples and the original minority samples, we can get the new minority sample set Ωsn.(3) The final balanced sample set isnewΩk+1={(xi,ti)∣i=N0+1+k,…,N0+1+k+newBlock}, where newBlock is the number of samples in (k+1)th step. Now the rate of majority class and minority class is probably between 1.5 : 1 and 1.1 : 1.The corresponding hidden layer matrix ofnewΩk+1 is HΩ=hk+N0+1hk+N0+2⋯hk+N0+newBlock, and now the hidden layer matrix becomes Hk+1=[HkTHΦT]T. Update the network weight according to the following equation:(15)βk+1=Hk+1†Tk+1,where Tk+1=[TkTTΩT]T is the output vector and Hk+1† is (Hk+1THk+1)-1Hk+1T. Let Pk+1=(Hk+1THk+1)-1. We have(16)Hk+1†=Pk+1Hk+1T,because(17)Hk+1THk+1=HkTHΦTHkTHΦTT=HkTHk+HΦTHΦ;namely,(18)Pk+1-1=Pk-1+HΦTHΦ.Calculate the inversion of both ends of the equation according to Sherman-Morrision matrix inversion lemma. We obtain the recursive expression ofPk+1:(19)Pk+1=Pk-1+HΦTHΦ-1=Pk-PkHΦTHΦPkI+HΦPkHΦT.SoPk+1 can be calculated based on Pk, which reduces calculation and greatly improves the computational efficiency. We can obtain Hk+1† by substituting (15) into (14) and then update the network weight βk+1.


## 4.1. Offline Stage

Firstly, we refactor the imbalanced sample set using the proposed method and get the balanced sample setA={(xi,ti)∣i=1,2,…,N0}. Then establish the initial model. The main idea is undersampling for majority class by choosing all the granule cores, which can reduce the number of majority samples and ensure that the samples’ distribution trend is consistent with the trend before undersampling.For initial majority sample setD={(xi,ti),i=1,2,…,N1}, the first granulation division is conducted. Then, we obtain the new majority sample set Dn={(xi,ti),i=1,2,…,N11}.Clustering algorithm [21] is adopted to simulate the process of granulation division. We set the clustering algorithm k1 in the first granulation division according to the overall distribution of original samples. The up and down threshold values of maximum radius of granule are set to [η1,η2], which can guarantee that the sample distribution trend keeps unchanged before and after the first undersampling. Then, we choose k1 clustering center as the granule core and replace the original majority samples. Merging the new majority sample set Dn={(xi,ti),i=1,2,…,N11} and the minority sample set S={(yi,ti),i=1,2,…,N2}, we obtain the new training sample set A={(xi,ti)∣i=1,2,…,N0}.Given the hidden active functiong(x) and the number of hidden nodes L, choose input weight wi and bias bi, i=1,2,…,N0 randomly and calculate the hidden layer output matrix H0:(9)H0=hx1hx2⋮hxN0=gw1·x1+b1⋯gwL·x1+bLgw1·x2+b1⋯gwL·x2+bL⋮gw1·xN0+b1⋯gwL·xN0+bL.The output vector is T0=t1,t2,…,tN0T and the output weight is(10)β0=H0†T0,where H0†=(H0TH0)-1H0T. Set P0=(H0TH0)-1, and then we have H0†=P0H0T.Setk=k+1.


## 4.2. Online Stage

Suppose that the sequential samples chunk isΩk+1={(xi,ti)∣i=N0+1+k,…,N0+1+k+Block} in (k+1)th step, where Block means the number of samples in Ωk+1. Ωk+1 is divided into majority class Ωd and minority class Ωs according to the value of label ti.(1) Granulation division for majority classisΩd is conducted. We choose k1 points as the initial granule core uniformly according to the samples distribution trend, where k1 is set as about three times the number of minority samples. We can obtain k1 clustering center in each iteration, namely, granule core, by equation of clustering algorithm μi(j+1)=1/ni∑i=1nipit, until the distance between each sample and the clustering center meets the condition of the following equation. Finally, we can obtain k1 clustering center C={c1,c2,…,ck1}, namely, k1 granule core. Now, the majority sample set is Ωdn and the imbalance rate reduces to 3 : 1:(11)min∑∑dist⁡ci,x2.(2) Granulation computing for minority class: the value ofk2 is set to half the number of original minority samples:(12)k2=12sizeΩs.If there are other samples except the granule core, we add the virtual samples within the granule and granule boundary using SMOTE. Then, we obtain the new minority sample set by merging the virtual samples and original minority samples. The detailed description is as follows.Step 1.
Choose the granule core as the center of a circle. We add virtual samples between the granule core and all the other samples using SMOTE, as shown in the following equation:(13)xjim′=cj+diffi×rand⁡0,1=cj+cj-xji×rand⁡0,1.The virtual samples are generated between the granule core and other samples in the granule. Every time there are m virtual samples generated. The value of m1 is set according to the actual situation.Step 2.
According to the following equation:(14)xjim+1′=cj+diffi×rand⁡1,n1=cj+cj-xji×rand⁡1,n1,
m
2 virtual samples will be generated between the granule core and most of the samples in the granule, except the sample farthest from the granule core, which can ensure that the new virtual samples are not too far from the granule core and thus maintain credibility. Usually we set m2≤1/3m1, where m1 means the total sample numbers in the granule. The new virtual samples expand the distribution range of the granule and do not affect the overall credibility simultaneously. Besides, the random number is between 1 and n1, 1<n1≤1.5, which can ensure that the distance between new virtual samples and the granule core is farther than the distance between raw samples and the granule core, and most of the virtual samples are still in the granule.Step 3.
Set the sample weight of virtual samples according to Definition3 and then update the virtual samples. Merging the virtual samples and the original minority samples, we can get the new minority sample set Ωsn.(3) The final balanced sample set isnewΩk+1={(xi,ti)∣i=N0+1+k,…,N0+1+k+newBlock}, where newBlock is the number of samples in (k+1)th step. Now the rate of majority class and minority class is probably between 1.5 : 1 and 1.1 : 1.The corresponding hidden layer matrix ofnewΩk+1 is HΩ=hk+N0+1hk+N0+2⋯hk+N0+newBlock, and now the hidden layer matrix becomes Hk+1=[HkTHΦT]T. Update the network weight according to the following equation:(15)βk+1=Hk+1†Tk+1,where Tk+1=[TkTTΩT]T is the output vector and Hk+1† is (Hk+1THk+1)-1Hk+1T. Let Pk+1=(Hk+1THk+1)-1. We have(16)Hk+1†=Pk+1Hk+1T,because(17)Hk+1THk+1=HkTHΦTHkTHΦTT=HkTHk+HΦTHΦ;namely,(18)Pk+1-1=Pk-1+HΦTHΦ.Calculate the inversion of both ends of the equation according to Sherman-Morrision matrix inversion lemma. We obtain the recursive expression ofPk+1:(19)Pk+1=Pk-1+HΦTHΦ-1=Pk-PkHΦTHΦPkI+HΦPkHΦT.SoPk+1 can be calculated based on Pk, which reduces calculation and greatly improves the computational efficiency. We can obtain Hk+1† by substituting (15) into (14) and then update the network weight βk+1.


## 5. The Reliability Analysis

According to the discussion as above, we reduce the majority samples using granulation division both in offline stage and in online stage. For the original majority sample setθd={(xi,ti),i=1,2,…,M}, we only choose m most representative samples and get the new balanced majority sample set θd′. Although the imbalanced phenomenon could be reduced to some extent, there is a loss of information in the undersampling because of abandoning some samples. To illustrate the rationality of the proposed method, we give the lower bound of the model reliability after undersampling based on information entropy [22], which can indicate indirectly that there is upper bound of the information loss in undersampling.Suppose that the loss sample set isA={(xij,ti),j=1,2,…,m} in every online undersampling, where xij is a sample in the granule centered with the granule core cj. As discussed in Section 4.2, we reject all samples in this granule except cj. The sample weight of cj is defined as follows:(20)ωcj=1Separatecj=kp3Rj∗∑i=1kpcj-xij.Then the missed classed probability is(21)Pf=∑j=1mwk=∑j=1m1-kp3Rj∗∑i=1kpcj-xij,where kp means the number of samples and ∑i=1kp|cj-xij| means the sum of Euclidean distance between each sample and the granule core.In offline stage, the loss sample set isA1={(yij1,ti),j=1,2,…,m0}, where yij1 means the sample in the jth granule. The granule core cj1 will join the final balanced sample set representing the whole granule. So the misclassification rate is(22)Pf1=∑l=1m0wk1=∑l=1m01-kp13Rj1∗∑i=1kp1cj1-xij1.Theorem 4.
At present,N is the sample number of majority set D and L is the number of misclassification majority samples. Let RL represent the lower bound of model reliability. Because binary classification result obeys the binomial distribution, the lower bound of model reliability can be obtained when the confidence coefficient is determined:(23)∑r=0LNrRLN-r1-RLr=∑r=0N∗PfNrRLN-r1-RLr=∑r=0N∗∑j=1m1-kp3/Rj∗∑i=1kpcj-xijNrRLN-r1-RLr=1-α,where L is negatively correlated with RL. It can be seen that the fuzzy reliability is only related to the discrete degree.Proof.
According to the definition of fuzzy reliability∑r=0FN/rRLN-r(1-RL)r=1-α, RL reaches the maximum when F is the minimum with definite α, because(24)F≤L=N∗Pf=N∗∑j=1m1-kp3Rj∗∑i=1kpcj-xij.As can be seen from the above equation, RL is only related to kp3/Rj∗∑i=1kpcj-xij. We know kp3/Rj∗∑i=1kpcj-xij=1/Separate(cj). So the smaller the maximum radius of granule is, the smaller the distance sum of samples is, the higher the number of samples in granule is, the smaller the dispersion is, and the bigger the value of kp3/Rj∗∑i=1kpcj-xij is, which will cause smaller lower bound of reliability and more reliable model.Theorem 5.
The sample size of majority setD0 is N0 in offline stage and the number of misclassification samples is L0. The reliability can be obtained according to the following equation. And the reliability is only related to the value of kp13/Rj1∗∑i=1kp1cj1-xij1: (25)∑r=0L0N0rRL0N0-r1-RL0r=∑r=0N0∗Pf1N0rRL0N0-r1-RL0r=∑r=0N0∗∑l=1m01-kp13/Rj1∗∑i=1kp1cj1-xij1NrRL0N0-r1-RL0r=1-α.Proof.
According to the definition of fuzzy reliability, we have(26)F≤L0=N0∗Pf1=N0∗∑l=1m01-kp13Rj1∗∑i=1kp1cj1-xij1.
According to the equation, the value ofRL0 is related to the sum of distance between the granule core and other samples. The smaller the distance sum is, the smaller the dispersion is and the more compact the samples in granule are, which will cause the bigger fuzzy reliability and more reliable model.Theorems4 and 5 prove the reasonability of the proposed algorithm from the point of information entropy. Considering the extreme case, if the granule dispersion is 0, namely, not undersampling by granulation division, the misclassification rate of majority class is almost 0; that is, limPs→1⁡H(Φ)→0 means that it does not provide the information entropy and the information loss is 0, which is accordant with the practical situation.


## 6. Simulation Experiment

In order to demonstrate the effectiveness and the superiority of the proposed algorithm, we conduct the simulation experiments on the chessboard-shaped dataset with uneven density distribution and the imbalanced distribution meteorological data of Macao in 2010 and 2012. At the same time, we compare the experimental result of our algorithm with that of SVM (support vector machine) [23], OS-ELM (online sequential ELM) [14], and MCOS-ELM (Metacognitive OS-ELM) [16]. Among them, MCOS-ELM is an online extreme learning algorithm presented by Vong et al. [16] for the online data imbalance problem. For better demonstration, we call the proposed method DGSMOTE (Division of Granulation and SMOTE OS-ELM). Before the training, we apply the normalization procedure to the dataset. We take the average value of 30 trials as the final experimental result.


### 6.1. Construct the Chessboard-Shaped Dataset

In the chessboard-shaped dataset, both the majority and the minority samples take up eight cells in the chessboard. According to the respective class in each cell, some data chunks are randomly generated. Ultimately, the quantity of majority and minority samples is1000∗8 and 100∗8, respectively; that is to say, the ratio of the classes is 10 : 1. The testing data are generated with the same method.


#### 6.1.1. Experimental Results Analysis of Chessboard-Shaped Data

In the offline stage, we conduct the undersampling on the majority samples first. Then the changes in the distribution of the chessboard-shaped data are shown as Figure1(b) after the first granulation division. In the online stage, after conducting granulation division, we process the SMOTE algorithm to realize the oversampling for the minority samples. As a result, the changes of the dataset are shown as Figure 1(c).Figure 1
The distribution of the offline chessboard-shaped data (a) before and (b) after the first granulation and (c) the result after using SMOTE.
(a)
(b)
(c)It can be seen from Figure1 that, compared with the original samples, now the classifications of dataset at the moment are nearly balanced. For the different classes of the chessboard-shaped dataset, Table 1 presents the changes in their numbers before and after the process of this proposed algorithm.Table 1
Changes in the numbers before and after balancing the offline data.
Dataset
Before
After
Majority
Minority
Majority
Minority
Chessboard-shaped data
7997
807
2500
2449In the simulation experiment, the activation function of the hidden layer is set as “sig” and the numbers of hidden nodes are set as 140. We take the mean value of 30 trials as the experimental result. Finally, the performance comparison of the four models is shown as Table2.Table 2
Comparative results on chessboard-shaped dataset.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0874
0.1976
0.0531
132.476
Minority training accuracy
0.9167
0.6329
0.7438
0.6385
Majority training accuracy
0.9414
0.8988
0.8051
0.9792
Whole testing accuracy
0.9363
0.8439
0.8612
0.9606
G-mean
0.9325
0.7334
0.7445
0.7822From Table2, though the DGSMOTE’s whole testing accuracy is not the highest among the models, its testing accuracy and testing times do not appear to be much different from the other three algorithms. In addition, the minority training accuracy of DGSMOTE is much higher than that of others. Compared with LS-SVM, DGSMOTE shows superior performance in both testing training speed and accuracy. This demonstrates the instantaneity of the proposed algorithm. Furthermore, the new DGSMOTE presents good performance upon using G-mean to evaluate specialty and the sensitivity of the algorithm. It can highly improve the classification accuracy of minority with less decrease of amplitude of majority accuracy. At the same time, it can eliminate the bias which is generated by applying traditional algorithm to handle the imbalanced data.To strengthen the reliability and observability of our algorithm, the classification results and the minority accuracy variation with different numbers of hidden nodes on chessboard-shaped dataset are shown in Figures2 and 3, where the dark spot means the misclassified samples. They also reflect the good generalization and learning performance of DGSMOTE.Figure 2
Classification results of on chessboard-shaped dataset. (a) OS-ELM and (b) DGSMOTE.
(a)
(b)Figure 3
Classification accuracy on minority class with different number of hidden nodes for chessboard-shaped dataset.From Figure2, classification accuracy of minority samples is significantly less than that of majority samples upon using OS-ELM to classify the dataset. Namely, the model of OS-ELM possesses obvious bias in the classification. Compared with the improved OS-ELM algorithms, the DGSMOTE has a better overall classification performance with little effect on the majority classification accuracy. During the procedure of undersampling and oversampling in our algorithm, both the whole distribution characteristics and the original feature of the samples are fully considered. So loss information value of the balanced samples is low and stable. We transform the number of the hidden nodes for the three online extreme learning algorithms. Then the accuracy of each node for the corresponding algorithm is obtained by taking the mean value of 20 trials. Hence, we get Figure 3. By observing the changes of minority accuracy, we know that the whole performance of DGSMOTE algorithm is not only high but also stable.After synthesizing all the above indicators, it is obvious that the DGSMOTE possesses effective generalization performance and outstanding learning ability. In order to display the sensitivity and specificity of our algorithm, we employ ROC curve to reveal the excellent performance. The ROC curves of the four models on the chessboard-shaped dataset are shown in Figure4.Figure 4
Comparison of the ROC on chessboard-shaped dataset for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)AUC denotes the area under the ROC curve. The larger the value of AUC is, the better the classification is. From Figures1–3, we can know that the proposed DGSMOTE algorithm significantly outperforms the other three models with better overall performance and lower minority misclassification rate. Besides, it reduces the loss cost generated by misclassification because of its strong recognition capability and lower classification bias.


### 6.2. Experimental Results Analysis of Macao Forecasting Data

Macao forecasting dataset is obtained from the website of the Macao Meteorological Bureau [24]. Compared with chessboard-shaped dataset, it has less samples but more attributions and is a kind of flow distribution data. According to its own features, we choose PM10 and SO2 as the two main characteristics from all the six features for illustration. And the changes after the first granulation division are shown in Figure 5(b).Figure 5
The distribution of the forecasting data (a) before and (b) after the first granulation division and (c) the result after using SMOTE.
(a)
(b)
(c)After the first granulation division, the imbalanced ratio of the new sample set is markedly decreased compared to that of the original dataset.In the online stage, we first conduct granular computing. Next, we apply SMOTE algorithm to process the oversampling for the minority. Figure5(c) displays how the minority samples of Macao forecasting data change after the procedure of DGSMOTE.It is obvious form Figure5 that the sample data are nearly balanced after the oversampling and undersampling. Table 3 shows the changes in the number of the observations before and after being handled by our algorithm.Table 3
Changes in the numbers before and after balancing the offline data.
Dataset
Before
After
Majority
Minority
Majority
Minority
Forecasting data in 2010
334
31
110
97
Forecasting data in 2011
330
35
110
104The next step is to use the new balanced sample set to establish the initial model of the online extreme learning machine. Similarly, the activation function of the hidden layer is set as “sig.” According to features of the forecasting data, the numbers of hidden nodes were assigned as 30. The four models established by the four algorithms conduct the learning on the two different forecasting datasets, respectively. Finally, the comparative performances of the models are presented in Tables4 and 5.Table 4
Comparative results on Macao forecasting data in 2010.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0399
0.0468
0.0468
1.6224
Minority training accuracy
0.9655
0.5775
0.7037
0.7009
Majority training accuracy
0.9481
0.9771
0.9746
0.9286
Whole testing accuracy
0.9546
0.9432
0.9368
0.9025
G-mean
0.9472
0.7403
0.8213
0.7655Table 5
Comparative results on Macao forecasting data in 2011.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0321
0.0412
0.0425
1.638
Minority training accuracy
0.9469
0.5723
0.7206
0.7457
Majority training accuracy
0.9456
0.9756
0.9673
0.9492
Whole testing accuracy
0.9495
0.9421
0.9662
0.9263
G-mean
0.9347
0.7387
0.8399
0.7948As can be seen from Tables4 and 5, compared with the other three algorithms, DGSMOTE can effectively increase the classification accuracy and recognition capability of the minority samples upon dealing with the practical imbalanced problem. However, the overall testing accuracy and the value of G-mean suffer no decline. This case declares that most original features of the minority samples are reserved after the undersampling. It also reflects the low information loss and the validity of the granulation division. Meanwhile, the shorter testing time reveals the fast learning speed, strong recognition ability, and good instantaneity of the DGSMOTE algorithm.In order to make the validity and stability of our proposed algorithm more clear, the classification results and the minority accuracy variation with different numbers of hidden nodes on the two datasets are shown in Figures6, 7, and 8, respectively. In Figures 6 and 7, the dark spots mean the misclassified samples.Figure 6
Classification results of (a) OS-ELM and (b) DGSMOTE on forecasting data in 2010 year.
(a)
(b)Figure 7
Classification results of (a) OS-ELM and (b) DGSMOTE on forecasting data in 2011 year.
(a)
(b)Figure 8
Classification accuracy on minority class with different number of hidden nodes on Macao forecasting data in (a) 2010 year and (b) 2011 year.
(a)
(b)From Figures6 and 7, our proposed algorithm has superior recognition ability to the other three algorithms as well as avoiding significant decline of majority class accuracy. That is to say, the DGSMOTE effectively eliminates the bias generated by applying the original OS-ELM algorithms to handle the imbalanced problems. In Figure 8, the curves of DGSMOTE are much smoother without erratic fluctuation along with the variation of the number of hidden nodes. And they further testify the favorable generalization and stability of our proposed algorithm.We still use ROC curves to exhibit the outstanding overall effect and superior performance of the proposed algorithm. Figures9 and 10 indicate the ROC curves of the four models on the Macao forecasting data in 2010 year and 2011 year, respectively.Figure 9
Comparison of the ROC on Macao forecasting data in 2010 year for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)Figure 10
Comparison of the ROC on Macao forecasting data in 2011 year for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)According to ROC curves in Figures9 and 10, it is obvious that the DGSMOTE algorithm has an advantage over the other three algorithms and possesses better overall performance and recognition ability upon dealing with the flow distribution imbalanced data. This shows more research and application value for the practical problems.


## 6.1. Construct the Chessboard-Shaped Dataset

In the chessboard-shaped dataset, both the majority and the minority samples take up eight cells in the chessboard. According to the respective class in each cell, some data chunks are randomly generated. Ultimately, the quantity of majority and minority samples is1000∗8 and 100∗8, respectively; that is to say, the ratio of the classes is 10 : 1. The testing data are generated with the same method.


### 6.1.1. Experimental Results Analysis of Chessboard-Shaped Data

In the offline stage, we conduct the undersampling on the majority samples first. Then the changes in the distribution of the chessboard-shaped data are shown as Figure1(b) after the first granulation division. In the online stage, after conducting granulation division, we process the SMOTE algorithm to realize the oversampling for the minority samples. As a result, the changes of the dataset are shown as Figure 1(c).Figure 1
The distribution of the offline chessboard-shaped data (a) before and (b) after the first granulation and (c) the result after using SMOTE.
(a)
(b)
(c)It can be seen from Figure1 that, compared with the original samples, now the classifications of dataset at the moment are nearly balanced. For the different classes of the chessboard-shaped dataset, Table 1 presents the changes in their numbers before and after the process of this proposed algorithm.Table 1
Changes in the numbers before and after balancing the offline data.
Dataset
Before
After
Majority
Minority
Majority
Minority
Chessboard-shaped data
7997
807
2500
2449In the simulation experiment, the activation function of the hidden layer is set as “sig” and the numbers of hidden nodes are set as 140. We take the mean value of 30 trials as the experimental result. Finally, the performance comparison of the four models is shown as Table2.Table 2
Comparative results on chessboard-shaped dataset.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0874
0.1976
0.0531
132.476
Minority training accuracy
0.9167
0.6329
0.7438
0.6385
Majority training accuracy
0.9414
0.8988
0.8051
0.9792
Whole testing accuracy
0.9363
0.8439
0.8612
0.9606
G-mean
0.9325
0.7334
0.7445
0.7822From Table2, though the DGSMOTE’s whole testing accuracy is not the highest among the models, its testing accuracy and testing times do not appear to be much different from the other three algorithms. In addition, the minority training accuracy of DGSMOTE is much higher than that of others. Compared with LS-SVM, DGSMOTE shows superior performance in both testing training speed and accuracy. This demonstrates the instantaneity of the proposed algorithm. Furthermore, the new DGSMOTE presents good performance upon using G-mean to evaluate specialty and the sensitivity of the algorithm. It can highly improve the classification accuracy of minority with less decrease of amplitude of majority accuracy. At the same time, it can eliminate the bias which is generated by applying traditional algorithm to handle the imbalanced data.To strengthen the reliability and observability of our algorithm, the classification results and the minority accuracy variation with different numbers of hidden nodes on chessboard-shaped dataset are shown in Figures2 and 3, where the dark spot means the misclassified samples. They also reflect the good generalization and learning performance of DGSMOTE.Figure 2
Classification results of on chessboard-shaped dataset. (a) OS-ELM and (b) DGSMOTE.
(a)
(b)Figure 3
Classification accuracy on minority class with different number of hidden nodes for chessboard-shaped dataset.From Figure2, classification accuracy of minority samples is significantly less than that of majority samples upon using OS-ELM to classify the dataset. Namely, the model of OS-ELM possesses obvious bias in the classification. Compared with the improved OS-ELM algorithms, the DGSMOTE has a better overall classification performance with little effect on the majority classification accuracy. During the procedure of undersampling and oversampling in our algorithm, both the whole distribution characteristics and the original feature of the samples are fully considered. So loss information value of the balanced samples is low and stable. We transform the number of the hidden nodes for the three online extreme learning algorithms. Then the accuracy of each node for the corresponding algorithm is obtained by taking the mean value of 20 trials. Hence, we get Figure 3. By observing the changes of minority accuracy, we know that the whole performance of DGSMOTE algorithm is not only high but also stable.After synthesizing all the above indicators, it is obvious that the DGSMOTE possesses effective generalization performance and outstanding learning ability. In order to display the sensitivity and specificity of our algorithm, we employ ROC curve to reveal the excellent performance. The ROC curves of the four models on the chessboard-shaped dataset are shown in Figure4.Figure 4
Comparison of the ROC on chessboard-shaped dataset for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)AUC denotes the area under the ROC curve. The larger the value of AUC is, the better the classification is. From Figures1–3, we can know that the proposed DGSMOTE algorithm significantly outperforms the other three models with better overall performance and lower minority misclassification rate. Besides, it reduces the loss cost generated by misclassification because of its strong recognition capability and lower classification bias.


## 6.1.1. Experimental Results Analysis of Chessboard-Shaped Data

In the offline stage, we conduct the undersampling on the majority samples first. Then the changes in the distribution of the chessboard-shaped data are shown as Figure1(b) after the first granulation division. In the online stage, after conducting granulation division, we process the SMOTE algorithm to realize the oversampling for the minority samples. As a result, the changes of the dataset are shown as Figure 1(c).Figure 1
The distribution of the offline chessboard-shaped data (a) before and (b) after the first granulation and (c) the result after using SMOTE.
(a)
(b)
(c)It can be seen from Figure1 that, compared with the original samples, now the classifications of dataset at the moment are nearly balanced. For the different classes of the chessboard-shaped dataset, Table 1 presents the changes in their numbers before and after the process of this proposed algorithm.Table 1
Changes in the numbers before and after balancing the offline data.
Dataset
Before
After
Majority
Minority
Majority
Minority
Chessboard-shaped data
7997
807
2500
2449In the simulation experiment, the activation function of the hidden layer is set as “sig” and the numbers of hidden nodes are set as 140. We take the mean value of 30 trials as the experimental result. Finally, the performance comparison of the four models is shown as Table2.Table 2
Comparative results on chessboard-shaped dataset.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0874
0.1976
0.0531
132.476
Minority training accuracy
0.9167
0.6329
0.7438
0.6385
Majority training accuracy
0.9414
0.8988
0.8051
0.9792
Whole testing accuracy
0.9363
0.8439
0.8612
0.9606
G-mean
0.9325
0.7334
0.7445
0.7822From Table2, though the DGSMOTE’s whole testing accuracy is not the highest among the models, its testing accuracy and testing times do not appear to be much different from the other three algorithms. In addition, the minority training accuracy of DGSMOTE is much higher than that of others. Compared with LS-SVM, DGSMOTE shows superior performance in both testing training speed and accuracy. This demonstrates the instantaneity of the proposed algorithm. Furthermore, the new DGSMOTE presents good performance upon using G-mean to evaluate specialty and the sensitivity of the algorithm. It can highly improve the classification accuracy of minority with less decrease of amplitude of majority accuracy. At the same time, it can eliminate the bias which is generated by applying traditional algorithm to handle the imbalanced data.To strengthen the reliability and observability of our algorithm, the classification results and the minority accuracy variation with different numbers of hidden nodes on chessboard-shaped dataset are shown in Figures2 and 3, where the dark spot means the misclassified samples. They also reflect the good generalization and learning performance of DGSMOTE.Figure 2
Classification results of on chessboard-shaped dataset. (a) OS-ELM and (b) DGSMOTE.
(a)
(b)Figure 3
Classification accuracy on minority class with different number of hidden nodes for chessboard-shaped dataset.From Figure2, classification accuracy of minority samples is significantly less than that of majority samples upon using OS-ELM to classify the dataset. Namely, the model of OS-ELM possesses obvious bias in the classification. Compared with the improved OS-ELM algorithms, the DGSMOTE has a better overall classification performance with little effect on the majority classification accuracy. During the procedure of undersampling and oversampling in our algorithm, both the whole distribution characteristics and the original feature of the samples are fully considered. So loss information value of the balanced samples is low and stable. We transform the number of the hidden nodes for the three online extreme learning algorithms. Then the accuracy of each node for the corresponding algorithm is obtained by taking the mean value of 20 trials. Hence, we get Figure 3. By observing the changes of minority accuracy, we know that the whole performance of DGSMOTE algorithm is not only high but also stable.After synthesizing all the above indicators, it is obvious that the DGSMOTE possesses effective generalization performance and outstanding learning ability. In order to display the sensitivity and specificity of our algorithm, we employ ROC curve to reveal the excellent performance. The ROC curves of the four models on the chessboard-shaped dataset are shown in Figure4.Figure 4
Comparison of the ROC on chessboard-shaped dataset for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)AUC denotes the area under the ROC curve. The larger the value of AUC is, the better the classification is. From Figures1–3, we can know that the proposed DGSMOTE algorithm significantly outperforms the other three models with better overall performance and lower minority misclassification rate. Besides, it reduces the loss cost generated by misclassification because of its strong recognition capability and lower classification bias.


## 6.2. Experimental Results Analysis of Macao Forecasting Data

Macao forecasting dataset is obtained from the website of the Macao Meteorological Bureau [24]. Compared with chessboard-shaped dataset, it has less samples but more attributions and is a kind of flow distribution data. According to its own features, we choose PM10 and SO2 as the two main characteristics from all the six features for illustration. And the changes after the first granulation division are shown in Figure 5(b).Figure 5
The distribution of the forecasting data (a) before and (b) after the first granulation division and (c) the result after using SMOTE.
(a)
(b)
(c)After the first granulation division, the imbalanced ratio of the new sample set is markedly decreased compared to that of the original dataset.In the online stage, we first conduct granular computing. Next, we apply SMOTE algorithm to process the oversampling for the minority. Figure5(c) displays how the minority samples of Macao forecasting data change after the procedure of DGSMOTE.It is obvious form Figure5 that the sample data are nearly balanced after the oversampling and undersampling. Table 3 shows the changes in the number of the observations before and after being handled by our algorithm.Table 3
Changes in the numbers before and after balancing the offline data.
Dataset
Before
After
Majority
Minority
Majority
Minority
Forecasting data in 2010
334
31
110
97
Forecasting data in 2011
330
35
110
104The next step is to use the new balanced sample set to establish the initial model of the online extreme learning machine. Similarly, the activation function of the hidden layer is set as “sig.” According to features of the forecasting data, the numbers of hidden nodes were assigned as 30. The four models established by the four algorithms conduct the learning on the two different forecasting datasets, respectively. Finally, the comparative performances of the models are presented in Tables4 and 5.Table 4
Comparative results on Macao forecasting data in 2010.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0399
0.0468
0.0468
1.6224
Minority training accuracy
0.9655
0.5775
0.7037
0.7009
Majority training accuracy
0.9481
0.9771
0.9746
0.9286
Whole testing accuracy
0.9546
0.9432
0.9368
0.9025
G-mean
0.9472
0.7403
0.8213
0.7655Table 5
Comparative results on Macao forecasting data in 2011.
Algorithms
DGSMOTE
OS-ELM
MCOS-ELM
LS-SVM
Testing time (s)
0.0321
0.0412
0.0425
1.638
Minority training accuracy
0.9469
0.5723
0.7206
0.7457
Majority training accuracy
0.9456
0.9756
0.9673
0.9492
Whole testing accuracy
0.9495
0.9421
0.9662
0.9263
G-mean
0.9347
0.7387
0.8399
0.7948As can be seen from Tables4 and 5, compared with the other three algorithms, DGSMOTE can effectively increase the classification accuracy and recognition capability of the minority samples upon dealing with the practical imbalanced problem. However, the overall testing accuracy and the value of G-mean suffer no decline. This case declares that most original features of the minority samples are reserved after the undersampling. It also reflects the low information loss and the validity of the granulation division. Meanwhile, the shorter testing time reveals the fast learning speed, strong recognition ability, and good instantaneity of the DGSMOTE algorithm.In order to make the validity and stability of our proposed algorithm more clear, the classification results and the minority accuracy variation with different numbers of hidden nodes on the two datasets are shown in Figures6, 7, and 8, respectively. In Figures 6 and 7, the dark spots mean the misclassified samples.Figure 6
Classification results of (a) OS-ELM and (b) DGSMOTE on forecasting data in 2010 year.
(a)
(b)Figure 7
Classification results of (a) OS-ELM and (b) DGSMOTE on forecasting data in 2011 year.
(a)
(b)Figure 8
Classification accuracy on minority class with different number of hidden nodes on Macao forecasting data in (a) 2010 year and (b) 2011 year.
(a)
(b)From Figures6 and 7, our proposed algorithm has superior recognition ability to the other three algorithms as well as avoiding significant decline of majority class accuracy. That is to say, the DGSMOTE effectively eliminates the bias generated by applying the original OS-ELM algorithms to handle the imbalanced problems. In Figure 8, the curves of DGSMOTE are much smoother without erratic fluctuation along with the variation of the number of hidden nodes. And they further testify the favorable generalization and stability of our proposed algorithm.We still use ROC curves to exhibit the outstanding overall effect and superior performance of the proposed algorithm. Figures9 and 10 indicate the ROC curves of the four models on the Macao forecasting data in 2010 year and 2011 year, respectively.Figure 9
Comparison of the ROC on Macao forecasting data in 2010 year for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)Figure 10
Comparison of the ROC on Macao forecasting data in 2011 year for the four algorithms (a) SVM, (b) OS-ELM, (c) MCOS-ELM, and (d) DGSMOTE.
(a)
(b)
(c)
(d)According to ROC curves in Figures9 and 10, it is obvious that the DGSMOTE algorithm has an advantage over the other three algorithms and possesses better overall performance and recognition ability upon dealing with the flow distribution imbalanced data. This shows more research and application value for the practical problems.


## 7. Conclusion

In this paper, a novel classification approach based on SMOTE is proposed from the application in actual engineering. In the offline stage, we conduct the granulation division according to the distribution and the clustering characteristics of the majority samples. The central sample in each granule is used to replace the granule itself. Finally, the balanced offline dataset is obtained. In the online stage, we first process the granulation division for the minority class on the basis of the offline stage and then conduct the SMOTE to realize the oversampling of the minority samples. Our algorithm effectively increases the classification accuracy of the minority class under the premise that the overall distribution was unchanged and the information loss of majority samples reduced.Furthermore, entropy theorem is used to testify the rationality of the proposed algorithm. The final experimental results demonstrate that the overall generalization performance, classification efficiency, and classification accuracy of the online imbalanced samples can get improved by applying granulation division to make the dataset balanced. For the online imbalanced small sample set and the large scale data, our research is of both great theoretical significance and practical value.



---

*Source: 5685970-2016-05-25.xml*