# High-Performance Machine Learning for Large-Scale Data Classification considering Class Imbalance

Authors: Yang Liu; Xiang Li; Xianbang Chen; Xi Wang; Huaqiang Li
Journal: Scientific Programming
 (2020)

---

## Abstract

Currently, data classification is one of the most important ways to analysis data. However, along with the development of data collection, transmission, and storage technologies, the scale of the data has been sharply increased. Additionally, due to multiple classes and imbalanced data distribution in the dataset, the class imbalance issue is also gradually highlighted. The traditional machine learning algorithms lack of abilities for handling the aforementioned issues so that the classification efficiency and precision may be significantly impacted. Therefore, this paper presents an improved artificial neural network in enabling the high-performance classification for the imbalanced large volume data. Firstly, the Borderline-SMOTE (synthetic minority oversampling technique) algorithm is employed to balance the training dataset, which potentially aims at improving the training of the back propagation neural network (BPNN), and then, zero-mean, batch-normalization, and rectified linear unit (ReLU) are further employed to optimize the input layer and hidden layers of BPNN. At last, the ensemble learning-based parallelization of the improved BPNN is implemented using the Hadoop framework. Positive conclusions can be summarized according to the experimental results. Benefitting from Borderline-SMOTE, the imbalanced training dataset can be balanced, which improves the training performance and the classification accuracy. The improvements for the input layer and hidden layer also enhance the training performances in terms of convergence. The parallelization and the ensemble learning techniques enable BPNN to implement the high-performance large-scale data classification. The experimental results show the effectiveness of the presented classification algorithm.


---

## Body

## 1. Introduction

Classification is one of the most effective approaches in enabling the analysis of the digital data in quite a number of academia and research fields, for example, the medical researches [1–6] and the power system researches [7–12]. Multiple applications including image processing, pattern recognition, and pattern matching benefit from the accurate and efficient classification algorithms [1–6]. Li et al. [13] implemented the medical image classification using convolutional neural network (CNN). Their classification strategy can automatically and efficiently learn the graphical characteristics of the interstitial lung disease. As a result, the strategy is able to supply accurate and efficient classification performances. Jiao et al. [14] also employed and improved CNN to classify the mental load data. The experimental results show the effectiveness of their classification algorithm. Yang and Shen [15] reviewed the load classification researches of the electrical power system. The paper pointed out that classification is an effective way of researching the complicated load behaviors, which significantly affects the power production and consumption. In order to implement the natural language processing (NLP), Zhang et al. [16] employed deep learning as the underlying methodology to execute the text classification. Their work is able to identify the semantic labels of the texts successfully.In the recent years, a number of classification algorithm researches have been presented, which are mainly based on k-means [17], fuzzy c-means (FCM) [18], neural networks (NNs) [19], support vector machine (SVM) [20], and so on. For example, Peng et al. [7] employed a number of clustering algorithms containing k-means, k-medoids, self-organizing maps (SOM) [21], and FCM to recognize the patterns of the electrical load data. Xu et al. [8] improved and optimized k-means algorithm using canopy algorithm. The authors claimed that the clustering accuracy and efficiency of their algorithm have been significantly enhanced. Niazmardi et al. [22] presented an improved FCM algorithm which can achieve accurate results for clustering the hyperspectral data. However, quite a number of researches [23–25] pointed out that the flaws, for example, the sensitivity of outliers, difficulty of clustering nonlinear-separable classes, and empirical values-based parameters, existing in the abovementioned unsupervised machine learning algorithms prevent them from being effectively used. As a result, the supervised machine learning algorithms, for example, the neural networks, have been widely employed in the classification researches [26]. Gu et al. [27] optimized the learning rate and inertial factor for the traditional BPNN. Their improved self-adaptive BPNN algorithm shows great performances in terms of data modeling. Li et al. [9] combined BPNN and FCM to implement the electrical load forecasting. The authors reported that the load forecasting accuracy can be significantly improved. Based on the conclusions of the researches [10,11,23–25], BPNN has been proved that it is quite suitable for the classification tasks. Although BPNN has a number of advantages, it still has flaws, for example, the slow convergence issue in its training caused by the sensitivity of initial weights, sensitivity of learning rate, gradient exploding, and gradient vanishing. However, several researches [28–33] suggested that batch-normalization [28] and ReLU [34] have great potentials to solve these issues.Additionally, current digital data collection benefits from the developments of the smart meters, data communication systems, and data storage technologies, which results in sharply increasing of the data scale and data dimension. Tang et al. [35] pointed out that to overcome the issue, machine learning algorithms need focus on dimension reduction and sample selection techniques to improve the algorithm efficiencies. Farrell et al. [36] also proved that their improvements of the maximum entropy principle, random forest, and SVM can achieve satisfied performances for processing the high-dimensional data. Xu et al. [37] claimed that the data labeling and training efficiency are two main challenges for the large-scale data classification. Therefore, they presented a k-means and SVM-based strategy to implement the large-scale data classification. Their experimental results show that based on their strategy, the size of the training dataset can be reduced with maintaining the classification accuracy. Liu et al. [23] further pointed out that BPNN encounters extremely low efficiency when it classifies the large-scale and high-dimension data. The research also stated that the distributed computing could be a suitable solution to overcome the low-efficiency issue. Su et al. [12] employed the Hadoop framework [38] to implement the efficiency improvement for BPNN. The authors reported that their solution achieves satisfied precision for the data forecasting. Liu et al. [25] also presented a Spark-based distributed BPNN algorithm [39] which shows remarkable efficiency for classifying large-scale data. However, Liu et al. [23] pointed out that the algorithm decoupling-based parallelization of BPNN may generate a great number of iterations, which deteriorates the processing efficiency. The authors also claimed that the data separation-based parallelization may reduce the final classification accuracy. Therefore, to figure out a parallelized BPNN with high efficiency and accuracy is a valuable work.It also should be noticed that due to multiple classes and uneven data distribution, the class imbalance issue [40] frequently exists in the training data. The issue could significantly affect the training effect, which further impacts the final classification accuracy. Zhang et al. [41] reported that the image recognition ability of the deep CNN declines in the case of unbalanced training data. Therefore, the authors presented a classification method which recognizes a query image by comparing distances between category centers of CNN features of the whole training dataset and the corresponding CNN feature of the query image. Li et al. [42] also pointed out that although CNN supplies very high performances, it is still suffering from the problem of class imbalance. By adding an extra class-imbalance aware regularization, the authors presented a new loss function enabling CNN more sensitive to the samples of the minority classes. Zhang et al. [43] also claimed that the imbalanced class issue leads to significant misclassification of deep belief network (DBN). To address the issue, the authors unequalize the misclassification costs between classes, and then the costs are applied to DBN to achieve the accurate classification. Although the efforts contributed by the abovementioned researches are able to solve the class imbalance issue, Han et al. [40] suggested that the sampling technique should be an effective way of rebuilding the samples for the minority class and further solves the class imbalance issue efficiently.As a result, this paper presents a BPNN-based high-performance large-scale data classification method considering class imbalance. The paper firstly improves the Borderline-SMOTE [40] algorithm using Fréchet distance [44] to solve the potential class imbalance issue in the training data. Secondly, in order to solve the slow convergence issue in the training phase of BPNN, zero-mean [45], batch-normalization, and ReLU are employed to improve the input layer, hidden layer, and activation function. Thirdly, this paper also presents a MapReduce-based parallelization method for the improved BPNN. Based on the data separation and ensemble learning techniques, the parallelization of the improved BPNN can be implemented.The rest of the paper is organized as follows: Section2 presents the details of the methodologies for the BPNN improvements; Section 3 shows the experimental results; Section 4 concludes the paper.


## 2. Class Balance-Based Improved BPNN in Enabling Large-Scale Classification

This section firstly presents the Fréchet distance-based Borderline-SMOTE which is able to solve the class imbalance issue in the training dataset, and then, the section presents the details of BPNN improvement using zero-mean, batch-normalization, and ReLU. Finally, the section presents the ensemble learning-based parallelization for the improved BPNN using the Hadoop framework.


### 2.1. Fréchet Distance and Borderline-SMOTE-Based Class Balance

Class imbalance issue may significantly impact the training of BPNN, which finally leads to the misclassification. Borderline-SMOTE [40] is proved to be an effective solution for balancing the classes in the training dataset. However, Borderline-SMOTE measures the similarity between data instances using Euclidean distance which lacks the ability for representing the shape features and sequential characteristics of the data instances. Therefore, this paper employs Fréchet distance [44] instead of Euclidean distance to measure the similarity between the data instances.


#### 2.1.1. Fréchet Distance Computation for Two Data Instances

LetR2 denote the metric space; A, B :  [0, 1] ⟶ R2 represent two continuous curves in the Fréchet space [46]; and α, β :  [0, 1] ⟶ [0, 1] denote two continuous nondecreasing functions with independent variable t in the unit interval. Therefore, the Fréchet distance δdF(A, B) is defined as the following equation:(1)δdFA,B=infmaxt∈0,1Aαt−BβtP,where ||.||P represents the Euclidean norm; inf represents the infimum of the set; α(0) = β(0) = 0, α(1) = β(1) = 1 [46].The parametert is continuous which cannot adapt to the computation for the discrete parameters. Therefore, the discrete Fréchet distance is presented by researches [47,48]. Let P = (p1, p2, ..., pu) and Q=q1,q2,…,qv denote two discrete curves, and k denote the total number of the Fréchet permutations [47,48]; therefore, for one Fréchet permutation Wj = {(Pi, Qi)}, 1 ≤I ≤u, 1 ≤j≤k, the max value of the max distances in Wj can be represented by dFWjP,Q=maximaxp,q∈Pi×Qidisp,q. As a result, the discrete Fréchet distance for P and Q can be represented by the following equation:(2)dFP,Q=minWdFWP,Q.Fréchet distance has better ability to represent the shape features and sequential characteristics of the data instances [44]. Therefore, it is employed by this paper to compute the nearest neighbors for the Borderline-SMOTE algorithm.


#### 2.1.2. Borderline-SMOTE in Enabling Class Balance

Borderline-SMOTE is able to balance the classes for the imbalanced dataset [40]. It balances the dataset based on two remarkable advantages including the identification of the border between the major and minority classes, and the sample synthesis around the border.Step 1.
In one datasetT, for each point pii=1,...,pnum in the minority class P, compute a set of m nearest neighbors using Fréchet distance (equation (2)). Inside the set, the number of the points belonged to the major class is m′ (0 ≤ m′ ≤ m).Step 2.
Ifm′ equals m which indicates the m nearest neighbors are majority examples, pi is regarded as noise and neglected. Otherwise, if 0 ≤m′ ≤m/2, pi has less chance to be misclassified, which does not need to be further processed. If m/2 ≤m′ ≤m, pi has higher chance to be misclassified. Therefore, for all pi in the minority class, a borderline set E=p1′,p2′,...,pdnum′, 0≤dnum≤pnum can be achieved, where pi′ denotes a nearest neighbor of pi.Step 3.
For eachpi′ in E, compute a number of k nearest neighbors from the minority class P, and then a number of s points are randomly selected from the k neighbors. As a result, let rjj=1,2...,s denote a random value between 0 and 1, and pj′ denote one of the s neighbors. Therefore, a new instance syntheticj=pi′+rj×pi′−pj′ can be ultimately synthesized.
A demonstration performed by Fréchet distance-based Borderline-SMOTE is shown in Figure1. Firstly, the algorithm is able to identify the border between two imbalance classes. Secondly, new instances can be synthesized to balance the classes and further highlight the border to distinguish two classes. Therefore, if class imbalance issue exists in the training dataset of BPNN, the Fréchet distance-based Borderline-SMOTE has great potential to improve the training performance. However, it also should be noted that the values of two parameters k and m affect the performance of Borderline-SMOTE. Therefore, in the later algorithm evaluation sections, the optimal values of k and m are selected from a series of preexecuted experiments.Figure 1
Borderline-SMOTE-based data instance synthesis ( majority class;  minority class;  synthetic data).


### 2.2. Improved BPNN Using Zero-Mean, Batch-Normalization, and ReLU

As an effective classification algorithm, BPNN has been successfully employed in quite a number of researches [10,11,26,49]. However, issues, for example, the sensitivity of initial weights, sensitivity of learning rate, gradient exploding, and gradient vanishing, are still needed to be handled. Therefore, this paper firstly employs zero-mean to normalize the input data instances. Secondly, batch-normalization and ReLU activation function are employed to overcome the convergence issues in the training phase.


#### 2.2.1. Brief Introduction of the Standard BPNN

Figure2 shows the architecture of a typical BPNN, which contains the input layer, several hidden layers, and the output layer. In the hidden and output layers, a number of neurons exist, respectively. In Figure 2, x1, …, xn represent the input of BPNN; wij, bi, and g represent the weight, bias, and activation function of a neuron; ai represents the output of a neuron in the hidden layer; y represents the output of BPNN (a = y in the output layer). To facilitate the presentation, the aforementioned parameters are represented in the compact form of matrices X, W, b, A, and Y. In terms of the network training, two phases including feed forward and back propagation are employed. Basically, feed forward computes the input X using each layer of the network to achieve the output Y according to Y=gWTX+b. Back propagation employs the loss function loss to compute the variance J between the output Y and the actual value Y_ according to J = loss(Y, Y_). The neural network chooses a proper optimization algorithm, for example, the stochastic gradient descent (SGD) [29] algorithm, to update W and b using W=W−α∂J/∂W, b=b−α∂J/∂b, where α is the learning rate [50].Figure 2
The architecture of a typical BPNN.In terms of classification using BPNN, firstly the training phase should be carried out. Let instancei = [a1, a2, …, an] indicate the ith data instance in the training dataset; a denote one feature of instancei; and cj denote the class that instancei is belonged to. Firstly, each feature of instancei is normalized. Secondly, BPNN inputs instancei to run the feed forward to compute the output Y, and then, the coded cj is regarded as the actual value Y_ to run the back propagation to update the weights and biases. As long as all the training instances in the training dataset have been processed with several epochs and iterations, the training phase terminates. In the classification phase, let instancek denote the kth testing instance in the testing dataset. BPNN inputs instancek and computes the output using only feed forward. The output is the class which instancek is belonged to. As long as all the testing instances in the testing dataset have been processed, the classification phase terminates.Figure3 shows the BPNN improvements using zero-mean, batch-normalization, and ReLU. The details of the improvements are presented in the following sections.Figure 3
The improvements for the input, hidden, and output layers.


#### 2.2.2. Zero-Mean-Based Input Layer Improvement

Zero-mean is a normalization technique which is able to improve the data distribution to accelerate the gradient descent [45]. Let the input matrix X denote a number of batch-size input data instances in each iteration. Processed by the zero-mean layer shown in Figure 3, the zero-centered data instances become the actual input. Let the matrix Xmean denote the average value of the number of batch-size input data instances, and the matrix Xzero-centered represent the zero-centered input; therefore, equations (3) and (4) indicate the computations for Xmean and Xzero-centered:(3)Xmean=x¯1⋮x¯batch−size,(4)Xzero−centered=X−Xmean.


#### 2.2.3. Batch-Normalization-Based Hidden Layer Improvement

Briefly, in the training phase of a multiple-hidden-layers neural network, the data distributions of different layers may vary, which is no longer independent and identically distributed (IID). Therefore, the internal covariate shift (ICS) occurs [28], which causes two main issues impacting the training. The first one is that the variation of the parameters in the current layer may cause the distribution variation of the input data in the last layer. As a result, the last layer has to tune itself to adapt to the distribution variation so that the learning performance is deteriorated. The second one is that the values of W and b may keep enlarging, which leads to larger value of W×x+b in each layer. Consequently, the gradient saturation of the activation function may occur. Therefore, the value of the updated gradient may be extremely small in the back propagation, which leads to the deterioration of the network convergence.However, batch-normalization [28] employs the normalization operations to recover the input data distribution for each neuron in the hidden layers to the standard normal distribution N(0, 1). Therefore, the activation function is able to work sensitively. Generally, in batch-normalization, the parameters mean μ and variance σ of the input data enable the normalization. Additionally, the tiny variations of μ and σ in each mini-batch can also improve the generalization of the neural network. The initial scale factor γ and the displacement factor β are adopted to implement the linear transformation. Both of the factors can be trained in batch-normalization. As a result, batch-normalization is capable of dealing with larger initial weights, larger learning rates, gradient issues, and overfitting issue, which significantly improves the training performance of the neural network [28].This paper employs batch-normalization to improve the hidden layer of BPNN due to its remarkable advantages. It normalizes the linear output of the hidden layer, and then the output of batch-normalization is input into the nonlinear ReLU. The details of batch-normalization are shown in Algorithm1.where D represents the output of the linear computation of the hidden layer; yi denotes the output of batch-normalization, which is the input of the nonlinear ReLU; ε is a constant value to stabilize the training; μD and σx2 represent the mean and variance of the input data instances; d^i represents the standardized intermediate data. Similar to the weight and bias of BPNN, γ and β are also updated in the back propagation phase according to γ=γ−α∂L/∂yi∂yi/∂γ and β=β−α∂L/∂yi∂yi/∂β, where L represents the back propagated loss function of the hidden layer.Algorithm 1: Batch-normalization.
Input: a batch of inputD=d1,d2,...,dbatch−sizeInitial scale and displacement factors:γ, βOutput:yi=BNγ,βdiμD=1/batch−size∑i=1batch−sizediσD2=1/batch−size∑i=1batch−sizedi−μD2d^i=di−μD/σd2+εyi=γd^i+β≡BNγ,βdi,


#### 2.2.4. ReLU Activation Function

To overcome the saturation of the commonly usedSigmoid and Tanh activation functions, as well as to improve the convergence of the network, this paper employs ReLU shown by equation (5) as the activation function for the neural network, which is able to handle the gradient issues [29]:(5)ReLUx=0,x<0,x,x≥0,where x presents the input of ReLU.


### 2.3. Ensemble Learning-Based Parallelization of BPNN

In order to implement the large-scale data classification, the Hadoop framework based on MapReduce computing model [51] is employed to parallelize the improved BPNN. This paper firstly separates the entire training dataset into a number of data chunks which are saved in HDFS (Hadoop Distributed File System), and then, each participated mapper initializes one sub-BPNN and inputs one data chunk, respectively. Each mapper trains its own classifier individually so that a number of different classifiers can be finally achieved in parallel. Therefore, for one testing data instance, these classifiers may generate different decisions. Finally, this paper also presents a weighted voting strategy to decide the ultimate classification result for the testing data instance.


#### 2.3.1. Data Separation-Based Parallelization for the Improved BPNN

Although BPNN can be directly decoupled and parallelized in Hadoop, the research [23] suggested that the efficiency of the parallelized algorithm is extremely low due to the imperfect iteration support of the Hadoop framework. Therefore, the parallelization in this paper is based on the data separation. Firstly, the entire training dataset is separated into a number of data chunks using the random sampling. The data chunks are saved in HDFS. However, in each data chunk, the class imbalance issue may be intensified or alleviated due to the random sampling and the separation. Therefore, if the class imbalance exists in one data chunk, Borderline-SMOTE balances the classes, and then, a number of mappers start, each of which individually initializes one improved BPNN as a sub-BPNN and inputs one data chunk to train the parameters of the sub-BPNN. The workflow of each sub-BPNN in a mapper is shown in Figure 4.Figure 4
The data balancing and training processes for one sub-BPNN in a mapper.Finally, as long as the training for each mapper terminates, multiple different classifiers can be achieved in parallel. In the classification phase, one testing instance is input into all the classifiers. As a result, different classifiers may generate different classification results. One reducer collects all the results from the mappers to form an aggregation, in which the weighted voting is carried out to achieve the final classification result for the testing instance.


#### 2.3.2. Weighted Voting

In order to ensemble the multiple classification results into one final result in the reducer, this paper presents a classification reliability-based weighted voting, which can achieve the final classification result according to the reliabilities of the results from the multiple classifiers.It is known that for different classes, each classifier has different classification accuracies. Therefore, letC represent the reliability matrix; n represent the number of classes in the training dataset; and r=r1,r2,...,rn represent the classification accuracy for each class of the classifier. Consequently, softmax function [52] can be adopted to compute the reliability ci for the ith class of the classifier using the following equation:(6)ci=eri∑j=1nerj.Therefore, for a number ofm classifiers, the reliability matrix C can be represented by the following equation:(7)Cm×n=c11⋯c1n⋮⋱⋮cm1⋯cmn.The output coding of each classifier is based on one-hot encoding [44]. So, the output p of the classifier is the probability distribution of each class pi (i = 1, 2, …, n) indicated by the following equation:(8)p=p1,p2,...,pn.Therefore, the outputs ofm classifiers form a probability distribution matrix P shown by the following equation:(9)Pm×n=p11⋯p1n⋮⋱⋮pm1⋯pmn.An intermediate matrixQ shown by equation (10) can be achieved by multiplying the reliability matrix C and the probability distribution matrix P:(10)Qm×n=c11p11⋯c1np1n⋮⋱⋮cm1pm1⋯cmnpmn.The sumri′ of the elements in the ith column in Q finally forms a weight matrix R denoted by the following equation:(11)R1×n=r1′,r2′,...,rn′.Ultimately, the final classification result class_label for the input data instance can be identified according to the following equation:(12)class_label=argmaxiR.Figure5 shows the logical flow of the BPNN parallelization using Hadoop.Figure 5
Parallelization of the improved BPNN using MapReduce.


## 2.1. Fréchet Distance and Borderline-SMOTE-Based Class Balance

Class imbalance issue may significantly impact the training of BPNN, which finally leads to the misclassification. Borderline-SMOTE [40] is proved to be an effective solution for balancing the classes in the training dataset. However, Borderline-SMOTE measures the similarity between data instances using Euclidean distance which lacks the ability for representing the shape features and sequential characteristics of the data instances. Therefore, this paper employs Fréchet distance [44] instead of Euclidean distance to measure the similarity between the data instances.


### 2.1.1. Fréchet Distance Computation for Two Data Instances

LetR2 denote the metric space; A, B :  [0, 1] ⟶ R2 represent two continuous curves in the Fréchet space [46]; and α, β :  [0, 1] ⟶ [0, 1] denote two continuous nondecreasing functions with independent variable t in the unit interval. Therefore, the Fréchet distance δdF(A, B) is defined as the following equation:(1)δdFA,B=infmaxt∈0,1Aαt−BβtP,where ||.||P represents the Euclidean norm; inf represents the infimum of the set; α(0) = β(0) = 0, α(1) = β(1) = 1 [46].The parametert is continuous which cannot adapt to the computation for the discrete parameters. Therefore, the discrete Fréchet distance is presented by researches [47,48]. Let P = (p1, p2, ..., pu) and Q=q1,q2,…,qv denote two discrete curves, and k denote the total number of the Fréchet permutations [47,48]; therefore, for one Fréchet permutation Wj = {(Pi, Qi)}, 1 ≤I ≤u, 1 ≤j≤k, the max value of the max distances in Wj can be represented by dFWjP,Q=maximaxp,q∈Pi×Qidisp,q. As a result, the discrete Fréchet distance for P and Q can be represented by the following equation:(2)dFP,Q=minWdFWP,Q.Fréchet distance has better ability to represent the shape features and sequential characteristics of the data instances [44]. Therefore, it is employed by this paper to compute the nearest neighbors for the Borderline-SMOTE algorithm.


### 2.1.2. Borderline-SMOTE in Enabling Class Balance

Borderline-SMOTE is able to balance the classes for the imbalanced dataset [40]. It balances the dataset based on two remarkable advantages including the identification of the border between the major and minority classes, and the sample synthesis around the border.Step 1.
In one datasetT, for each point pii=1,...,pnum in the minority class P, compute a set of m nearest neighbors using Fréchet distance (equation (2)). Inside the set, the number of the points belonged to the major class is m′ (0 ≤ m′ ≤ m).Step 2.
Ifm′ equals m which indicates the m nearest neighbors are majority examples, pi is regarded as noise and neglected. Otherwise, if 0 ≤m′ ≤m/2, pi has less chance to be misclassified, which does not need to be further processed. If m/2 ≤m′ ≤m, pi has higher chance to be misclassified. Therefore, for all pi in the minority class, a borderline set E=p1′,p2′,...,pdnum′, 0≤dnum≤pnum can be achieved, where pi′ denotes a nearest neighbor of pi.Step 3.
For eachpi′ in E, compute a number of k nearest neighbors from the minority class P, and then a number of s points are randomly selected from the k neighbors. As a result, let rjj=1,2...,s denote a random value between 0 and 1, and pj′ denote one of the s neighbors. Therefore, a new instance syntheticj=pi′+rj×pi′−pj′ can be ultimately synthesized.
A demonstration performed by Fréchet distance-based Borderline-SMOTE is shown in Figure1. Firstly, the algorithm is able to identify the border between two imbalance classes. Secondly, new instances can be synthesized to balance the classes and further highlight the border to distinguish two classes. Therefore, if class imbalance issue exists in the training dataset of BPNN, the Fréchet distance-based Borderline-SMOTE has great potential to improve the training performance. However, it also should be noted that the values of two parameters k and m affect the performance of Borderline-SMOTE. Therefore, in the later algorithm evaluation sections, the optimal values of k and m are selected from a series of preexecuted experiments.Figure 1
Borderline-SMOTE-based data instance synthesis ( majority class;  minority class;  synthetic data).


## 2.1.1. Fréchet Distance Computation for Two Data Instances

LetR2 denote the metric space; A, B :  [0, 1] ⟶ R2 represent two continuous curves in the Fréchet space [46]; and α, β :  [0, 1] ⟶ [0, 1] denote two continuous nondecreasing functions with independent variable t in the unit interval. Therefore, the Fréchet distance δdF(A, B) is defined as the following equation:(1)δdFA,B=infmaxt∈0,1Aαt−BβtP,where ||.||P represents the Euclidean norm; inf represents the infimum of the set; α(0) = β(0) = 0, α(1) = β(1) = 1 [46].The parametert is continuous which cannot adapt to the computation for the discrete parameters. Therefore, the discrete Fréchet distance is presented by researches [47,48]. Let P = (p1, p2, ..., pu) and Q=q1,q2,…,qv denote two discrete curves, and k denote the total number of the Fréchet permutations [47,48]; therefore, for one Fréchet permutation Wj = {(Pi, Qi)}, 1 ≤I ≤u, 1 ≤j≤k, the max value of the max distances in Wj can be represented by dFWjP,Q=maximaxp,q∈Pi×Qidisp,q. As a result, the discrete Fréchet distance for P and Q can be represented by the following equation:(2)dFP,Q=minWdFWP,Q.Fréchet distance has better ability to represent the shape features and sequential characteristics of the data instances [44]. Therefore, it is employed by this paper to compute the nearest neighbors for the Borderline-SMOTE algorithm.


## 2.1.2. Borderline-SMOTE in Enabling Class Balance

Borderline-SMOTE is able to balance the classes for the imbalanced dataset [40]. It balances the dataset based on two remarkable advantages including the identification of the border between the major and minority classes, and the sample synthesis around the border.Step 1.
In one datasetT, for each point pii=1,...,pnum in the minority class P, compute a set of m nearest neighbors using Fréchet distance (equation (2)). Inside the set, the number of the points belonged to the major class is m′ (0 ≤ m′ ≤ m).Step 2.
Ifm′ equals m which indicates the m nearest neighbors are majority examples, pi is regarded as noise and neglected. Otherwise, if 0 ≤m′ ≤m/2, pi has less chance to be misclassified, which does not need to be further processed. If m/2 ≤m′ ≤m, pi has higher chance to be misclassified. Therefore, for all pi in the minority class, a borderline set E=p1′,p2′,...,pdnum′, 0≤dnum≤pnum can be achieved, where pi′ denotes a nearest neighbor of pi.Step 3.
For eachpi′ in E, compute a number of k nearest neighbors from the minority class P, and then a number of s points are randomly selected from the k neighbors. As a result, let rjj=1,2...,s denote a random value between 0 and 1, and pj′ denote one of the s neighbors. Therefore, a new instance syntheticj=pi′+rj×pi′−pj′ can be ultimately synthesized.
A demonstration performed by Fréchet distance-based Borderline-SMOTE is shown in Figure1. Firstly, the algorithm is able to identify the border between two imbalance classes. Secondly, new instances can be synthesized to balance the classes and further highlight the border to distinguish two classes. Therefore, if class imbalance issue exists in the training dataset of BPNN, the Fréchet distance-based Borderline-SMOTE has great potential to improve the training performance. However, it also should be noted that the values of two parameters k and m affect the performance of Borderline-SMOTE. Therefore, in the later algorithm evaluation sections, the optimal values of k and m are selected from a series of preexecuted experiments.Figure 1
Borderline-SMOTE-based data instance synthesis ( majority class;  minority class;  synthetic data).


## 2.2. Improved BPNN Using Zero-Mean, Batch-Normalization, and ReLU

As an effective classification algorithm, BPNN has been successfully employed in quite a number of researches [10,11,26,49]. However, issues, for example, the sensitivity of initial weights, sensitivity of learning rate, gradient exploding, and gradient vanishing, are still needed to be handled. Therefore, this paper firstly employs zero-mean to normalize the input data instances. Secondly, batch-normalization and ReLU activation function are employed to overcome the convergence issues in the training phase.


### 2.2.1. Brief Introduction of the Standard BPNN

Figure2 shows the architecture of a typical BPNN, which contains the input layer, several hidden layers, and the output layer. In the hidden and output layers, a number of neurons exist, respectively. In Figure 2, x1, …, xn represent the input of BPNN; wij, bi, and g represent the weight, bias, and activation function of a neuron; ai represents the output of a neuron in the hidden layer; y represents the output of BPNN (a = y in the output layer). To facilitate the presentation, the aforementioned parameters are represented in the compact form of matrices X, W, b, A, and Y. In terms of the network training, two phases including feed forward and back propagation are employed. Basically, feed forward computes the input X using each layer of the network to achieve the output Y according to Y=gWTX+b. Back propagation employs the loss function loss to compute the variance J between the output Y and the actual value Y_ according to J = loss(Y, Y_). The neural network chooses a proper optimization algorithm, for example, the stochastic gradient descent (SGD) [29] algorithm, to update W and b using W=W−α∂J/∂W, b=b−α∂J/∂b, where α is the learning rate [50].Figure 2
The architecture of a typical BPNN.In terms of classification using BPNN, firstly the training phase should be carried out. Let instancei = [a1, a2, …, an] indicate the ith data instance in the training dataset; a denote one feature of instancei; and cj denote the class that instancei is belonged to. Firstly, each feature of instancei is normalized. Secondly, BPNN inputs instancei to run the feed forward to compute the output Y, and then, the coded cj is regarded as the actual value Y_ to run the back propagation to update the weights and biases. As long as all the training instances in the training dataset have been processed with several epochs and iterations, the training phase terminates. In the classification phase, let instancek denote the kth testing instance in the testing dataset. BPNN inputs instancek and computes the output using only feed forward. The output is the class which instancek is belonged to. As long as all the testing instances in the testing dataset have been processed, the classification phase terminates.Figure3 shows the BPNN improvements using zero-mean, batch-normalization, and ReLU. The details of the improvements are presented in the following sections.Figure 3
The improvements for the input, hidden, and output layers.


### 2.2.2. Zero-Mean-Based Input Layer Improvement

Zero-mean is a normalization technique which is able to improve the data distribution to accelerate the gradient descent [45]. Let the input matrix X denote a number of batch-size input data instances in each iteration. Processed by the zero-mean layer shown in Figure 3, the zero-centered data instances become the actual input. Let the matrix Xmean denote the average value of the number of batch-size input data instances, and the matrix Xzero-centered represent the zero-centered input; therefore, equations (3) and (4) indicate the computations for Xmean and Xzero-centered:(3)Xmean=x¯1⋮x¯batch−size,(4)Xzero−centered=X−Xmean.


### 2.2.3. Batch-Normalization-Based Hidden Layer Improvement

Briefly, in the training phase of a multiple-hidden-layers neural network, the data distributions of different layers may vary, which is no longer independent and identically distributed (IID). Therefore, the internal covariate shift (ICS) occurs [28], which causes two main issues impacting the training. The first one is that the variation of the parameters in the current layer may cause the distribution variation of the input data in the last layer. As a result, the last layer has to tune itself to adapt to the distribution variation so that the learning performance is deteriorated. The second one is that the values of W and b may keep enlarging, which leads to larger value of W×x+b in each layer. Consequently, the gradient saturation of the activation function may occur. Therefore, the value of the updated gradient may be extremely small in the back propagation, which leads to the deterioration of the network convergence.However, batch-normalization [28] employs the normalization operations to recover the input data distribution for each neuron in the hidden layers to the standard normal distribution N(0, 1). Therefore, the activation function is able to work sensitively. Generally, in batch-normalization, the parameters mean μ and variance σ of the input data enable the normalization. Additionally, the tiny variations of μ and σ in each mini-batch can also improve the generalization of the neural network. The initial scale factor γ and the displacement factor β are adopted to implement the linear transformation. Both of the factors can be trained in batch-normalization. As a result, batch-normalization is capable of dealing with larger initial weights, larger learning rates, gradient issues, and overfitting issue, which significantly improves the training performance of the neural network [28].This paper employs batch-normalization to improve the hidden layer of BPNN due to its remarkable advantages. It normalizes the linear output of the hidden layer, and then the output of batch-normalization is input into the nonlinear ReLU. The details of batch-normalization are shown in Algorithm1.where D represents the output of the linear computation of the hidden layer; yi denotes the output of batch-normalization, which is the input of the nonlinear ReLU; ε is a constant value to stabilize the training; μD and σx2 represent the mean and variance of the input data instances; d^i represents the standardized intermediate data. Similar to the weight and bias of BPNN, γ and β are also updated in the back propagation phase according to γ=γ−α∂L/∂yi∂yi/∂γ and β=β−α∂L/∂yi∂yi/∂β, where L represents the back propagated loss function of the hidden layer.Algorithm 1: Batch-normalization.
Input: a batch of inputD=d1,d2,...,dbatch−sizeInitial scale and displacement factors:γ, βOutput:yi=BNγ,βdiμD=1/batch−size∑i=1batch−sizediσD2=1/batch−size∑i=1batch−sizedi−μD2d^i=di−μD/σd2+εyi=γd^i+β≡BNγ,βdi,


### 2.2.4. ReLU Activation Function

To overcome the saturation of the commonly usedSigmoid and Tanh activation functions, as well as to improve the convergence of the network, this paper employs ReLU shown by equation (5) as the activation function for the neural network, which is able to handle the gradient issues [29]:(5)ReLUx=0,x<0,x,x≥0,where x presents the input of ReLU.


## 2.2.1. Brief Introduction of the Standard BPNN

Figure2 shows the architecture of a typical BPNN, which contains the input layer, several hidden layers, and the output layer. In the hidden and output layers, a number of neurons exist, respectively. In Figure 2, x1, …, xn represent the input of BPNN; wij, bi, and g represent the weight, bias, and activation function of a neuron; ai represents the output of a neuron in the hidden layer; y represents the output of BPNN (a = y in the output layer). To facilitate the presentation, the aforementioned parameters are represented in the compact form of matrices X, W, b, A, and Y. In terms of the network training, two phases including feed forward and back propagation are employed. Basically, feed forward computes the input X using each layer of the network to achieve the output Y according to Y=gWTX+b. Back propagation employs the loss function loss to compute the variance J between the output Y and the actual value Y_ according to J = loss(Y, Y_). The neural network chooses a proper optimization algorithm, for example, the stochastic gradient descent (SGD) [29] algorithm, to update W and b using W=W−α∂J/∂W, b=b−α∂J/∂b, where α is the learning rate [50].Figure 2
The architecture of a typical BPNN.In terms of classification using BPNN, firstly the training phase should be carried out. Let instancei = [a1, a2, …, an] indicate the ith data instance in the training dataset; a denote one feature of instancei; and cj denote the class that instancei is belonged to. Firstly, each feature of instancei is normalized. Secondly, BPNN inputs instancei to run the feed forward to compute the output Y, and then, the coded cj is regarded as the actual value Y_ to run the back propagation to update the weights and biases. As long as all the training instances in the training dataset have been processed with several epochs and iterations, the training phase terminates. In the classification phase, let instancek denote the kth testing instance in the testing dataset. BPNN inputs instancek and computes the output using only feed forward. The output is the class which instancek is belonged to. As long as all the testing instances in the testing dataset have been processed, the classification phase terminates.Figure3 shows the BPNN improvements using zero-mean, batch-normalization, and ReLU. The details of the improvements are presented in the following sections.Figure 3
The improvements for the input, hidden, and output layers.


## 2.2.2. Zero-Mean-Based Input Layer Improvement

Zero-mean is a normalization technique which is able to improve the data distribution to accelerate the gradient descent [45]. Let the input matrix X denote a number of batch-size input data instances in each iteration. Processed by the zero-mean layer shown in Figure 3, the zero-centered data instances become the actual input. Let the matrix Xmean denote the average value of the number of batch-size input data instances, and the matrix Xzero-centered represent the zero-centered input; therefore, equations (3) and (4) indicate the computations for Xmean and Xzero-centered:(3)Xmean=x¯1⋮x¯batch−size,(4)Xzero−centered=X−Xmean.


## 2.2.3. Batch-Normalization-Based Hidden Layer Improvement

Briefly, in the training phase of a multiple-hidden-layers neural network, the data distributions of different layers may vary, which is no longer independent and identically distributed (IID). Therefore, the internal covariate shift (ICS) occurs [28], which causes two main issues impacting the training. The first one is that the variation of the parameters in the current layer may cause the distribution variation of the input data in the last layer. As a result, the last layer has to tune itself to adapt to the distribution variation so that the learning performance is deteriorated. The second one is that the values of W and b may keep enlarging, which leads to larger value of W×x+b in each layer. Consequently, the gradient saturation of the activation function may occur. Therefore, the value of the updated gradient may be extremely small in the back propagation, which leads to the deterioration of the network convergence.However, batch-normalization [28] employs the normalization operations to recover the input data distribution for each neuron in the hidden layers to the standard normal distribution N(0, 1). Therefore, the activation function is able to work sensitively. Generally, in batch-normalization, the parameters mean μ and variance σ of the input data enable the normalization. Additionally, the tiny variations of μ and σ in each mini-batch can also improve the generalization of the neural network. The initial scale factor γ and the displacement factor β are adopted to implement the linear transformation. Both of the factors can be trained in batch-normalization. As a result, batch-normalization is capable of dealing with larger initial weights, larger learning rates, gradient issues, and overfitting issue, which significantly improves the training performance of the neural network [28].This paper employs batch-normalization to improve the hidden layer of BPNN due to its remarkable advantages. It normalizes the linear output of the hidden layer, and then the output of batch-normalization is input into the nonlinear ReLU. The details of batch-normalization are shown in Algorithm1.where D represents the output of the linear computation of the hidden layer; yi denotes the output of batch-normalization, which is the input of the nonlinear ReLU; ε is a constant value to stabilize the training; μD and σx2 represent the mean and variance of the input data instances; d^i represents the standardized intermediate data. Similar to the weight and bias of BPNN, γ and β are also updated in the back propagation phase according to γ=γ−α∂L/∂yi∂yi/∂γ and β=β−α∂L/∂yi∂yi/∂β, where L represents the back propagated loss function of the hidden layer.Algorithm 1: Batch-normalization.
Input: a batch of inputD=d1,d2,...,dbatch−sizeInitial scale and displacement factors:γ, βOutput:yi=BNγ,βdiμD=1/batch−size∑i=1batch−sizediσD2=1/batch−size∑i=1batch−sizedi−μD2d^i=di−μD/σd2+εyi=γd^i+β≡BNγ,βdi,


## 2.2.4. ReLU Activation Function

To overcome the saturation of the commonly usedSigmoid and Tanh activation functions, as well as to improve the convergence of the network, this paper employs ReLU shown by equation (5) as the activation function for the neural network, which is able to handle the gradient issues [29]:(5)ReLUx=0,x<0,x,x≥0,where x presents the input of ReLU.


## 2.3. Ensemble Learning-Based Parallelization of BPNN

In order to implement the large-scale data classification, the Hadoop framework based on MapReduce computing model [51] is employed to parallelize the improved BPNN. This paper firstly separates the entire training dataset into a number of data chunks which are saved in HDFS (Hadoop Distributed File System), and then, each participated mapper initializes one sub-BPNN and inputs one data chunk, respectively. Each mapper trains its own classifier individually so that a number of different classifiers can be finally achieved in parallel. Therefore, for one testing data instance, these classifiers may generate different decisions. Finally, this paper also presents a weighted voting strategy to decide the ultimate classification result for the testing data instance.


### 2.3.1. Data Separation-Based Parallelization for the Improved BPNN

Although BPNN can be directly decoupled and parallelized in Hadoop, the research [23] suggested that the efficiency of the parallelized algorithm is extremely low due to the imperfect iteration support of the Hadoop framework. Therefore, the parallelization in this paper is based on the data separation. Firstly, the entire training dataset is separated into a number of data chunks using the random sampling. The data chunks are saved in HDFS. However, in each data chunk, the class imbalance issue may be intensified or alleviated due to the random sampling and the separation. Therefore, if the class imbalance exists in one data chunk, Borderline-SMOTE balances the classes, and then, a number of mappers start, each of which individually initializes one improved BPNN as a sub-BPNN and inputs one data chunk to train the parameters of the sub-BPNN. The workflow of each sub-BPNN in a mapper is shown in Figure 4.Figure 4
The data balancing and training processes for one sub-BPNN in a mapper.Finally, as long as the training for each mapper terminates, multiple different classifiers can be achieved in parallel. In the classification phase, one testing instance is input into all the classifiers. As a result, different classifiers may generate different classification results. One reducer collects all the results from the mappers to form an aggregation, in which the weighted voting is carried out to achieve the final classification result for the testing instance.


### 2.3.2. Weighted Voting

In order to ensemble the multiple classification results into one final result in the reducer, this paper presents a classification reliability-based weighted voting, which can achieve the final classification result according to the reliabilities of the results from the multiple classifiers.It is known that for different classes, each classifier has different classification accuracies. Therefore, letC represent the reliability matrix; n represent the number of classes in the training dataset; and r=r1,r2,...,rn represent the classification accuracy for each class of the classifier. Consequently, softmax function [52] can be adopted to compute the reliability ci for the ith class of the classifier using the following equation:(6)ci=eri∑j=1nerj.Therefore, for a number ofm classifiers, the reliability matrix C can be represented by the following equation:(7)Cm×n=c11⋯c1n⋮⋱⋮cm1⋯cmn.The output coding of each classifier is based on one-hot encoding [44]. So, the output p of the classifier is the probability distribution of each class pi (i = 1, 2, …, n) indicated by the following equation:(8)p=p1,p2,...,pn.Therefore, the outputs ofm classifiers form a probability distribution matrix P shown by the following equation:(9)Pm×n=p11⋯p1n⋮⋱⋮pm1⋯pmn.An intermediate matrixQ shown by equation (10) can be achieved by multiplying the reliability matrix C and the probability distribution matrix P:(10)Qm×n=c11p11⋯c1np1n⋮⋱⋮cm1pm1⋯cmnpmn.The sumri′ of the elements in the ith column in Q finally forms a weight matrix R denoted by the following equation:(11)R1×n=r1′,r2′,...,rn′.Ultimately, the final classification result class_label for the input data instance can be identified according to the following equation:(12)class_label=argmaxiR.Figure5 shows the logical flow of the BPNN parallelization using Hadoop.Figure 5
Parallelization of the improved BPNN using MapReduce.


## 2.3.1. Data Separation-Based Parallelization for the Improved BPNN

Although BPNN can be directly decoupled and parallelized in Hadoop, the research [23] suggested that the efficiency of the parallelized algorithm is extremely low due to the imperfect iteration support of the Hadoop framework. Therefore, the parallelization in this paper is based on the data separation. Firstly, the entire training dataset is separated into a number of data chunks using the random sampling. The data chunks are saved in HDFS. However, in each data chunk, the class imbalance issue may be intensified or alleviated due to the random sampling and the separation. Therefore, if the class imbalance exists in one data chunk, Borderline-SMOTE balances the classes, and then, a number of mappers start, each of which individually initializes one improved BPNN as a sub-BPNN and inputs one data chunk to train the parameters of the sub-BPNN. The workflow of each sub-BPNN in a mapper is shown in Figure 4.Figure 4
The data balancing and training processes for one sub-BPNN in a mapper.Finally, as long as the training for each mapper terminates, multiple different classifiers can be achieved in parallel. In the classification phase, one testing instance is input into all the classifiers. As a result, different classifiers may generate different classification results. One reducer collects all the results from the mappers to form an aggregation, in which the weighted voting is carried out to achieve the final classification result for the testing instance.


## 2.3.2. Weighted Voting

In order to ensemble the multiple classification results into one final result in the reducer, this paper presents a classification reliability-based weighted voting, which can achieve the final classification result according to the reliabilities of the results from the multiple classifiers.It is known that for different classes, each classifier has different classification accuracies. Therefore, letC represent the reliability matrix; n represent the number of classes in the training dataset; and r=r1,r2,...,rn represent the classification accuracy for each class of the classifier. Consequently, softmax function [52] can be adopted to compute the reliability ci for the ith class of the classifier using the following equation:(6)ci=eri∑j=1nerj.Therefore, for a number ofm classifiers, the reliability matrix C can be represented by the following equation:(7)Cm×n=c11⋯c1n⋮⋱⋮cm1⋯cmn.The output coding of each classifier is based on one-hot encoding [44]. So, the output p of the classifier is the probability distribution of each class pi (i = 1, 2, …, n) indicated by the following equation:(8)p=p1,p2,...,pn.Therefore, the outputs ofm classifiers form a probability distribution matrix P shown by the following equation:(9)Pm×n=p11⋯p1n⋮⋱⋮pm1⋯pmn.An intermediate matrixQ shown by equation (10) can be achieved by multiplying the reliability matrix C and the probability distribution matrix P:(10)Qm×n=c11p11⋯c1np1n⋮⋱⋮cm1pm1⋯cmnpmn.The sumri′ of the elements in the ith column in Q finally forms a weight matrix R denoted by the following equation:(11)R1×n=r1′,r2′,...,rn′.Ultimately, the final classification result class_label for the input data instance can be identified according to the following equation:(12)class_label=argmaxiR.Figure5 shows the logical flow of the BPNN parallelization using Hadoop.Figure 5
Parallelization of the improved BPNN using MapReduce.


## 3. Experimental Result

The experiments are organized into three parts. The first part evaluates the performances of the Fréchet distance-based Borderline-SMOTE using the randomly generated linearly inseparable two-dimensional semiannular dataset. The second part evaluates the performances of the improved BPNN in the standalone environment using the Iris dataset [53], Wine dataset [52], and Vehicle Silhouettes dataset [54]. The third part evaluates the performances of the parallelized improved BPNN in the Hadoop distributed computing environment. The details of the experimental environments are listed in Table 1.Table 1
Details of the experimental environment.
Standalone environmentDistributed environmentCPU:  Intel Core i5 2.5 GHzCPU: Intel Core i5-8250U 1.6 GHzMEM: 16 GBMEM: 8 GBOS: Ubuntu 16.04 64 bitOS: Win 10 64 bitHadoop: 2.9.1Python: 3.6.6Python: 3.5.2Node no.: 4


### 3.1. Evaluation for Fréchet Distance-Based Borderline-SMOTE

A randomly generated dataset is employed to evaluate the Fréchet distance-based Borderline-SMOTE. The dataset has two classes, each of which contains 150 data instances, respectively. Based on the original dataset shown in Figure6, an imbalanced dataset is generated, which is shown in Figure 7. Four class balance algorithms are implemented including Fréchet distance-based Borderline-SMOTE (k = 5, m = 5), random oversampling, random undersampling, and SMOTE. After the class balance for the imbalanced dataset, the improved BPNN carries out the classification using a number of 100 randomly selected training instances and a number of 90 testing instances. The classification results are shown in Table 2.Figure 6
The randomly generated linearly inseparable two-dimensional semiannular dataset with 300 instances.Figure 7
The imbalanced dataset based on the original dataset.Table 2
The comparisons of the classification accuracy for different class balance algorithms.
Class balancing algorithmClassification accuracy (%)Borderline-SMOTE95.556Random oversampling78.889Random undersampling74.444SMOTE86.667Without balancing66.667Table2 indicates that without the class balance, the classification accuracy is only 66.67%. However, all the class balance algorithms can help to improve the classification accuracy, among which the Fréchet distance-based Borderline-SMOTE significantly outperforms the other algorithms.To further evaluate the potential of the Fréchet distance-based Borderline-SMOTE, a number of 20000 instances which averagely belonged to two classes are generated using the random sampling. Firstly, a number of 5000 instances are randomly selected from class 1, and then a number of 1000, 500, 50, and 10 instances are randomly selected from class 2, respectively, to compose 4 imbalanced training datasets. Balanced by the Fréchet distance-based Borderline-SMOTE (k = 5, m = 5), the improved BPNN is trained, and then carries out the classification. The number of the testing instances is 10000. The classification accuracies are listed in Table 3.Table 3
The comparisons of the classification accuracy using Borderline-SMOTE.
Imbalance class ratioBalanced dataset accuracy (%)Imbalanced dataset accuracy (%)5000 : 100098.0973.155000 : 50095.1968.325000 : 5080.5356.205000 : 1067.4553.13Table3 shows that the imbalanced training data significantly impacts the classification accuracy. However, benefitting from the Fréchet distance-based Borderline-SMOTE, the classification accuracy can be improved substantially. Table 3 also indicates that the imbalance ratio can impact the classification accuracy. The slightly imbalanced training dataset achieves better classification performance as long as it is balanced by the Fréchet distance-based Borderline-SMOTE. Contrarily, the extremely imbalanced training dataset leads to low classification accuracy even if it is balanced.


### 3.2. Evaluation for the Improved BPNN

This section evaluates the performances of the improved BPNN in the standalone environment. Iris dataset, Wine dataset, and Vehicle Silhouettes dataset are employed. The details of the training and testing instances are listed in Table4. In terms of comparison, SVM, traditional BPNN, and self-adaptive BPNN [55] are also implemented.Table 4
The training instances and the testing instances of the datasets.
DatasetTraining instance numberTesting instance numberIris120 (class1: 40; class2: 40; class3: 40)30 (class1: 10; class2: 10; class3: 10)Wine124 (class1: 36; class2: 52; class3: 36)54 (class1: 23; class2: 19; class3: 12)Vehicle592 (Class1: 137; Class2: 155; Class3: 159; Class4: 141)254 (Class1: 62 Class2: 62; Class3: 59; Class4: 71)Table5 shows the average classification accuracy based on 50 times experiments for each algorithm using the Iris dataset. The traditional BPNN performs the lowest classification accuracy with the largest epochs. The presented improved BPNN and the self-adaptive BPNN perform the similar classification accuracy. However, in terms of the number of epochs, self-adaptive BPNN slightly outperforms the improved BPNN.Table 5
The accuracy comparisons using the Iris dataset.
AlgorithmTesting dataset accuracyEpochsImproved BPNN96.667188Self-adaptive BPNN96.667172Traditional BPNN93.333342SVM96.667/Tables6 and 7 show the average classification accuracy based on 50 times experiments for each algorithm using the Wine dataset and Vehicle Silhouettes dataset. Firstly, the traditional BPNN also shows the worst performance. Secondly, in the Wine dataset-based experiments, although the self-adaptive BPNN slightly outperforms the improved BPNN in terms of epochs, the classification accuracy of the improved BPNN is higher than that of the self-adaptive BPNN. Thirdly, in the Vehicle Silhouettes dataset-based experiments, the improved BPNN performs the best in terms of accuracy and epochs.Table 6
The accuracy comparisons using the Wine dataset.
AlgorithmTesting dataset accuracyEpochsImproved BPNN94.370723Self-adaptive BPNN91.444537Traditional BPNN67.3171531SVM96.296/Table 7
The accuracy comparisons using the Vehicle Silhouettes dataset.
AlgorithmTesting dataset accuracy (%)EpochsImproved BPNN83.465245Self-adaptive BPNN80.709465Traditional BPNN74.803993SVM44.488/Figures8(a)–8(c) show the convergences of the traditional BPNN, the improved BPNN, and the self-adaptive BPNN using the Iris dataset, Wine dataset, and Vehicle Silhouettes dataset, respectively. For the simple Iris dataset, three algorithms perform relatively close convergences. However, for the complicated Wine dataset, both the improved BPNN and the self-adaptive BPNN significantly outperform the traditional BPNN. Until the error becomes small, the self-adaptive BPNN converges slightly faster than the improved BPNN. However, Figure 8(c) indicates that the convergence of the self-adaptive BPNN shows a number of fluctuations. The algorithm converges slowly and unstably. Contrarily, the convergence of the improved BPNN is fast and stable.Figure 8
(a) Comparison of the convergence using the Iris dataset. (b) Comparison of the convergence using the Wine dataset. (c) Comparison of the convergence using the Vehicle Silhouettes dataset.
(a)(b)(c)Figures9(a)–9(c) show the max and min number of epochs in the 50 times experiments using the Iris dataset, Wine dataset, and Vehicle Silhouettes dataset, respectively. The improved BPNN outperforms the traditional BPNN and performs the similar performances to those of the self-adaptive BPNN based on Iris and Wine datasets. In terms of the Vehicle Silhouettes dataset, the improved BPNN performs the best.Figure 9
(a) The max and min number of epochs in 50 times experiments using the Iris dataset. (b) The max and min number of epochs in 50 times experiments using the Wine dataset. (c) The max and min number of epochs in 50 times experiments using the Vehicle Silhouettes dataset.
(a)(b)(c)Figures10(a)–10(c) show the max and min classification accuracies in the 50 times experiments using the Iris dataset, Wine dataset, and Vehicle Silhouettes dataset, respectively. For the simple Iris dataset, the three algorithms perform closely. The presented improved BPNN slightly outperforms the other two algorithms. However, in terms of the Wine dataset, the traditional BPNN performs the worst. Although the classification accuracy of the self-adaptive BPNN can reach to 98%, its minimal accuracy is only 55%, which indicates the unstable performance of the algorithm. Contrarily, the improved BPNN performs the highest accuracy and the most stable performances. In terms of the Vehicle Silhouettes dataset, the improved BPNN also supplies the best accuracies.Figure 10
(a) The max and min classification accuracies in 50 times experiments using the Iris dataset. (b) The max and min classification accuracies in 50 times experiments using the Wine dataset. (c) The max and min classification accuracies in 50 times experiments using the Vehicle Silhouettes dataset.
(a)(b)(c)To further indicate the performance of the improved BPNN, the statistical indices of evaluating the epoch and accuracy using three datasets for 50 times experiments are listed in Tables8–10.Table 8
The statistical indices of evaluating the epoch and accuracy using the Iris dataset for 50 times experiments.
AlgorithmEpochAccuracyMeanVarianceMean (%)VarianceImproved BPNN188.822557.6296.79.23Traditional BPNN3424096.4393.363.27Self-adaptive BPNN172.31093.9696.711.73Table 9
The statistical indices of evaluating the epoch and accuracy using the Wine dataset for 50 times experiments.
AlgorithmEpochAccuracyMeanVarianceMean (%)VarianceImproved BPNN723.266266.6994.428.31Traditional BPNN536.85974.3767.3372.87Self-adaptive BPNN153131924.1291.455.2Table 10
The statistical indices of evaluating the epoch and accuracy using the Vehicle Silhouettes dataset for 50 times experiments.
AlgorithmEpochAccuracyMeanVarianceMean (%)VarianceImproved BPNN2453243.1983.521.06Traditional BPNN99323542.0874.8187.21Self-adaptive BPNN4652806.7480.735.19In the Iris dataset-based experiments, in terms of the epoch and accuracy tests, the improved BPNN and the self-adaptive BPNN perform similarly, though the improved BPNN shows slightly higher variance of the epoch, and also, both of them outperform the traditional BPNN.In the Wine dataset-based experiments, although the traditional BPNN performs the least mean and variance of the epoch, it shows the lowest mean accuracy with the highest variance. Contrarily, the improved BPNN significantly outperforms the self-adaptive BPNN in terms of both the epoch and accuracy.In the Vehicle Silhouettes dataset-based experiments, the traditional BPNN shows the worst performance. The improved BPNN outperforms the self-adaptive BPNN in terms of accuracy. For the epoch, the improved BPNN shows the least mean, and its variance is slightly higher than that of the self-adaptive BPNN.Figures11(a)–11(c) indicate that for the improved BPNN, the classification accuracy is impacted by batch size, and also, in terms of different datasets, the optimal batch size which leads to the highest accuracy is different. Therefore, a proper batch size is able to improve the classification accuracy for the presented improved BPNN.Figure 11
(a) Impact of batch sizes on classification accuracy of the improved BPNN using the Iris dataset. (b) Impact of batch sizes on classification accuracy of the improved BPNN using the Wine dataset. (c) Impact of batch sizes on classification accuracy of the improved BPNN using the Vehicle Silhouettes dataset.
(a)(b)(c)Figures12(a) and 12(b) indicate the numbers of epochs, nonconvergence, and overfitting of the traditional BPNN and the improved BPNN with varying learning rates (lr = 0.1, 0.01, 0.001, and 0.0001; 50 times experiments for each lr).Figure 12
(a) The convergence of the traditional BPNN with various learning rates (: nonconvergence, : overfitting). (b) The convergence of the improved BPNN with various learning rates (: nonconvergence, : overfitting).
(a)(b)Firstly, Figure12(a) shows that if the learning rate is small (lr = 0.0001), the traditional BPNN cannot converge totally, which results in the loss of the experimental result. However, Figure 12(b) shows that even if the learning rate is 0.0001, the improved BPNN can still converge. Secondly, for each learning rate, respectively, the number of exceptions (nonconvergence and overfitting) of the improved BPNN is less than that of the traditional BPNN. Thirdly, for each learning rate, respectively, the improved BPNN performs a smaller number of epochs than that of the traditional BPNN. The experimental results prove that the improvements done by this paper can significantly improve the convergence of BPNN.In next experiments, imbalanced training datasets are generated based on the original Iris dataset, Wine dataset, and Vehicle Silhouettes dataset. The details of the training and testing instances are listed in Table11. The imbalanced training datasets are firstly processed by the Fréchet distance-based Borderline-SMOTE (k = 5, m = 10), and then the improved BPNN carries out the classifications based on both the balanced and imbalanced training data for 50 times. The average classification accuracies are shown in Figures 13(a)–13(c).Table 11
The details of the training instances and the testing instances.
DatasetTraining instance numberTesting instance numberIris80 (class1: 36; class2: 7; class3: 37)45 (class1: 14; class2: 18; class3: 13)Wine79 (class1: 36; class2: 7; class3: 36)54 (class1: 23; class2: 19; class3: 12)Vehicle450 (Class1: 30; Class2: 140; Class3: 140; Class4: 140)254 (Class1: 62; Class2: 62; Class3: 59; Class4: 71)Figure 13
(a) Comparisons of the classification results using the Iris dataset. (b) Comparisons of the classification results using the Wine dataset. (c) Comparisons of the classification results using the Vehicle Silhouettes dataset.
(a)(b)(c)Both Figures13(a) and 13(b) indicate that the imbalanced training datasets significantly impact the training of the improved BPNN, which finally leads to severe misclassifications. However, balanced by the Fréchet distance-based Borderline-SMOTE, the network can be correctly and sufficiently trained. Therefore, the classification accuracies can be greatly improved. However, due to the complicated attributes of the instances of the Vehicle Silhouettes dataset, although the classification based on the balanced dataset also outperforms the imbalanced dataset, its average accuracy is only 79.53%. This point indicates that the dimension of the data may severely impact the performance of the class balance algorithm.


### 3.3. Evaluation for the Parallelized Improved BPNN in Hadoop Cluster

Firstly, the original Iris dataset is employed to evaluate the classification accuracy of the parallelized improved BPNN. A number of 105 instances are the training instances which will be separated for the parallelized training. The other 45 instances are the testing instances. A number of three mappers start in parallel. Each mapper initializes one sub-BPNN. The standalone BPNN and the parallelized long short-term memory network (LSTM) are also implemented in terms of comparison. Especially, the configuration of the parallelized LSTM is as the same as that of the parallelized BPNN. The number of the training instances is 105 and the number of the testing instances is 45 for the standalone BPNN.Figure14 indicates that the parallelized BPNN outperforms the standalone BPNN and the parallelized LSTM. Benefitting from the weighted voting, the potential accuracy loss issue in the data separation is properly handled. Actually, the classification accuracies for the 3 sub-BPNNs are 96.88%, 94.44%, and 91.89%, respectively. However, the accuracy for the aggregated result can reach to 100%. It further proves that the weighted voting can help to aggregate the weak classifiers to a strong classifier. Additionally, the data separation may intensify the class imbalance issue of the training dataset in each sub-BPNN. However, the Fréchet distance-based Borderline-SMOTE can effectively solve the issue, which guarantees the classification accuracy. Moreover, the parallelized LSTM performs slightly worse than the parallelized BPNN. It is well known that LSTM is suitable for processing the time-series data. However, the Iris dataset employed for the experiments is not related to time series, which indicates that the type of the dataset may affect the performance of the parallelized LSTM in terms of accuracy.Figure 14
The accuracy comparison for the standalone BPNN, the parallelized BPNN, and the parallelized LSTM.In terms of the classification efficiency evaluation, this paper duplicates the size of the Iris dataset from 1 MB to 1024 MB. The cluster starts 16 mappers in parallel. The processing times of the standalone BPNN, the parallelized BPNN, and the parallelized LSTM are listed in Figure15.Figure 15
Efficiency comparison for the standalone BPNN, the parallelized BPNN, and the parallelized LSTM.Figure15 indicates that the performances of three algorithms are quite close with smaller data sizes. Due to the overhead of the Hadoop cluster, the standalone BPNN even outperforms the parallelized BPNN and the parallelized LSTM. However, along with the increasing data sizes, because of the computing resource limitations of the standalone environment, the processing time of the standalone BPNN increases sharply. Contrarily, due to less computations, the parallelized BPNN outperforms the parallelized LSTM slightly and processes the large volume of data relatively efficiently.


## 3.1. Evaluation for Fréchet Distance-Based Borderline-SMOTE

A randomly generated dataset is employed to evaluate the Fréchet distance-based Borderline-SMOTE. The dataset has two classes, each of which contains 150 data instances, respectively. Based on the original dataset shown in Figure6, an imbalanced dataset is generated, which is shown in Figure 7. Four class balance algorithms are implemented including Fréchet distance-based Borderline-SMOTE (k = 5, m = 5), random oversampling, random undersampling, and SMOTE. After the class balance for the imbalanced dataset, the improved BPNN carries out the classification using a number of 100 randomly selected training instances and a number of 90 testing instances. The classification results are shown in Table 2.Figure 6
The randomly generated linearly inseparable two-dimensional semiannular dataset with 300 instances.Figure 7
The imbalanced dataset based on the original dataset.Table 2
The comparisons of the classification accuracy for different class balance algorithms.
Class balancing algorithmClassification accuracy (%)Borderline-SMOTE95.556Random oversampling78.889Random undersampling74.444SMOTE86.667Without balancing66.667Table2 indicates that without the class balance, the classification accuracy is only 66.67%. However, all the class balance algorithms can help to improve the classification accuracy, among which the Fréchet distance-based Borderline-SMOTE significantly outperforms the other algorithms.To further evaluate the potential of the Fréchet distance-based Borderline-SMOTE, a number of 20000 instances which averagely belonged to two classes are generated using the random sampling. Firstly, a number of 5000 instances are randomly selected from class 1, and then a number of 1000, 500, 50, and 10 instances are randomly selected from class 2, respectively, to compose 4 imbalanced training datasets. Balanced by the Fréchet distance-based Borderline-SMOTE (k = 5, m = 5), the improved BPNN is trained, and then carries out the classification. The number of the testing instances is 10000. The classification accuracies are listed in Table 3.Table 3
The comparisons of the classification accuracy using Borderline-SMOTE.
Imbalance class ratioBalanced dataset accuracy (%)Imbalanced dataset accuracy (%)5000 : 100098.0973.155000 : 50095.1968.325000 : 5080.5356.205000 : 1067.4553.13Table3 shows that the imbalanced training data significantly impacts the classification accuracy. However, benefitting from the Fréchet distance-based Borderline-SMOTE, the classification accuracy can be improved substantially. Table 3 also indicates that the imbalance ratio can impact the classification accuracy. The slightly imbalanced training dataset achieves better classification performance as long as it is balanced by the Fréchet distance-based Borderline-SMOTE. Contrarily, the extremely imbalanced training dataset leads to low classification accuracy even if it is balanced.


## 3.2. Evaluation for the Improved BPNN

This section evaluates the performances of the improved BPNN in the standalone environment. Iris dataset, Wine dataset, and Vehicle Silhouettes dataset are employed. The details of the training and testing instances are listed in Table4. In terms of comparison, SVM, traditional BPNN, and self-adaptive BPNN [55] are also implemented.Table 4
The training instances and the testing instances of the datasets.
DatasetTraining instance numberTesting instance numberIris120 (class1: 40; class2: 40; class3: 40)30 (class1: 10; class2: 10; class3: 10)Wine124 (class1: 36; class2: 52; class3: 36)54 (class1: 23; class2: 19; class3: 12)Vehicle592 (Class1: 137; Class2: 155; Class3: 159; Class4: 141)254 (Class1: 62 Class2: 62; Class3: 59; Class4: 71)Table5 shows the average classification accuracy based on 50 times experiments for each algorithm using the Iris dataset. The traditional BPNN performs the lowest classification accuracy with the largest epochs. The presented improved BPNN and the self-adaptive BPNN perform the similar classification accuracy. However, in terms of the number of epochs, self-adaptive BPNN slightly outperforms the improved BPNN.Table 5
The accuracy comparisons using the Iris dataset.
AlgorithmTesting dataset accuracyEpochsImproved BPNN96.667188Self-adaptive BPNN96.667172Traditional BPNN93.333342SVM96.667/Tables6 and 7 show the average classification accuracy based on 50 times experiments for each algorithm using the Wine dataset and Vehicle Silhouettes dataset. Firstly, the traditional BPNN also shows the worst performance. Secondly, in the Wine dataset-based experiments, although the self-adaptive BPNN slightly outperforms the improved BPNN in terms of epochs, the classification accuracy of the improved BPNN is higher than that of the self-adaptive BPNN. Thirdly, in the Vehicle Silhouettes dataset-based experiments, the improved BPNN performs the best in terms of accuracy and epochs.Table 6
The accuracy comparisons using the Wine dataset.
AlgorithmTesting dataset accuracyEpochsImproved BPNN94.370723Self-adaptive BPNN91.444537Traditional BPNN67.3171531SVM96.296/Table 7
The accuracy comparisons using the Vehicle Silhouettes dataset.
AlgorithmTesting dataset accuracy (%)EpochsImproved BPNN83.465245Self-adaptive BPNN80.709465Traditional BPNN74.803993SVM44.488/Figures8(a)–8(c) show the convergences of the traditional BPNN, the improved BPNN, and the self-adaptive BPNN using the Iris dataset, Wine dataset, and Vehicle Silhouettes dataset, respectively. For the simple Iris dataset, three algorithms perform relatively close convergences. However, for the complicated Wine dataset, both the improved BPNN and the self-adaptive BPNN significantly outperform the traditional BPNN. Until the error becomes small, the self-adaptive BPNN converges slightly faster than the improved BPNN. However, Figure 8(c) indicates that the convergence of the self-adaptive BPNN shows a number of fluctuations. The algorithm converges slowly and unstably. Contrarily, the convergence of the improved BPNN is fast and stable.Figure 8
(a) Comparison of the convergence using the Iris dataset. (b) Comparison of the convergence using the Wine dataset. (c) Comparison of the convergence using the Vehicle Silhouettes dataset.
(a)(b)(c)Figures9(a)–9(c) show the max and min number of epochs in the 50 times experiments using the Iris dataset, Wine dataset, and Vehicle Silhouettes dataset, respectively. The improved BPNN outperforms the traditional BPNN and performs the similar performances to those of the self-adaptive BPNN based on Iris and Wine datasets. In terms of the Vehicle Silhouettes dataset, the improved BPNN performs the best.Figure 9
(a) The max and min number of epochs in 50 times experiments using the Iris dataset. (b) The max and min number of epochs in 50 times experiments using the Wine dataset. (c) The max and min number of epochs in 50 times experiments using the Vehicle Silhouettes dataset.
(a)(b)(c)Figures10(a)–10(c) show the max and min classification accuracies in the 50 times experiments using the Iris dataset, Wine dataset, and Vehicle Silhouettes dataset, respectively. For the simple Iris dataset, the three algorithms perform closely. The presented improved BPNN slightly outperforms the other two algorithms. However, in terms of the Wine dataset, the traditional BPNN performs the worst. Although the classification accuracy of the self-adaptive BPNN can reach to 98%, its minimal accuracy is only 55%, which indicates the unstable performance of the algorithm. Contrarily, the improved BPNN performs the highest accuracy and the most stable performances. In terms of the Vehicle Silhouettes dataset, the improved BPNN also supplies the best accuracies.Figure 10
(a) The max and min classification accuracies in 50 times experiments using the Iris dataset. (b) The max and min classification accuracies in 50 times experiments using the Wine dataset. (c) The max and min classification accuracies in 50 times experiments using the Vehicle Silhouettes dataset.
(a)(b)(c)To further indicate the performance of the improved BPNN, the statistical indices of evaluating the epoch and accuracy using three datasets for 50 times experiments are listed in Tables8–10.Table 8
The statistical indices of evaluating the epoch and accuracy using the Iris dataset for 50 times experiments.
AlgorithmEpochAccuracyMeanVarianceMean (%)VarianceImproved BPNN188.822557.6296.79.23Traditional BPNN3424096.4393.363.27Self-adaptive BPNN172.31093.9696.711.73Table 9
The statistical indices of evaluating the epoch and accuracy using the Wine dataset for 50 times experiments.
AlgorithmEpochAccuracyMeanVarianceMean (%)VarianceImproved BPNN723.266266.6994.428.31Traditional BPNN536.85974.3767.3372.87Self-adaptive BPNN153131924.1291.455.2Table 10
The statistical indices of evaluating the epoch and accuracy using the Vehicle Silhouettes dataset for 50 times experiments.
AlgorithmEpochAccuracyMeanVarianceMean (%)VarianceImproved BPNN2453243.1983.521.06Traditional BPNN99323542.0874.8187.21Self-adaptive BPNN4652806.7480.735.19In the Iris dataset-based experiments, in terms of the epoch and accuracy tests, the improved BPNN and the self-adaptive BPNN perform similarly, though the improved BPNN shows slightly higher variance of the epoch, and also, both of them outperform the traditional BPNN.In the Wine dataset-based experiments, although the traditional BPNN performs the least mean and variance of the epoch, it shows the lowest mean accuracy with the highest variance. Contrarily, the improved BPNN significantly outperforms the self-adaptive BPNN in terms of both the epoch and accuracy.In the Vehicle Silhouettes dataset-based experiments, the traditional BPNN shows the worst performance. The improved BPNN outperforms the self-adaptive BPNN in terms of accuracy. For the epoch, the improved BPNN shows the least mean, and its variance is slightly higher than that of the self-adaptive BPNN.Figures11(a)–11(c) indicate that for the improved BPNN, the classification accuracy is impacted by batch size, and also, in terms of different datasets, the optimal batch size which leads to the highest accuracy is different. Therefore, a proper batch size is able to improve the classification accuracy for the presented improved BPNN.Figure 11
(a) Impact of batch sizes on classification accuracy of the improved BPNN using the Iris dataset. (b) Impact of batch sizes on classification accuracy of the improved BPNN using the Wine dataset. (c) Impact of batch sizes on classification accuracy of the improved BPNN using the Vehicle Silhouettes dataset.
(a)(b)(c)Figures12(a) and 12(b) indicate the numbers of epochs, nonconvergence, and overfitting of the traditional BPNN and the improved BPNN with varying learning rates (lr = 0.1, 0.01, 0.001, and 0.0001; 50 times experiments for each lr).Figure 12
(a) The convergence of the traditional BPNN with various learning rates (: nonconvergence, : overfitting). (b) The convergence of the improved BPNN with various learning rates (: nonconvergence, : overfitting).
(a)(b)Firstly, Figure12(a) shows that if the learning rate is small (lr = 0.0001), the traditional BPNN cannot converge totally, which results in the loss of the experimental result. However, Figure 12(b) shows that even if the learning rate is 0.0001, the improved BPNN can still converge. Secondly, for each learning rate, respectively, the number of exceptions (nonconvergence and overfitting) of the improved BPNN is less than that of the traditional BPNN. Thirdly, for each learning rate, respectively, the improved BPNN performs a smaller number of epochs than that of the traditional BPNN. The experimental results prove that the improvements done by this paper can significantly improve the convergence of BPNN.In next experiments, imbalanced training datasets are generated based on the original Iris dataset, Wine dataset, and Vehicle Silhouettes dataset. The details of the training and testing instances are listed in Table11. The imbalanced training datasets are firstly processed by the Fréchet distance-based Borderline-SMOTE (k = 5, m = 10), and then the improved BPNN carries out the classifications based on both the balanced and imbalanced training data for 50 times. The average classification accuracies are shown in Figures 13(a)–13(c).Table 11
The details of the training instances and the testing instances.
DatasetTraining instance numberTesting instance numberIris80 (class1: 36; class2: 7; class3: 37)45 (class1: 14; class2: 18; class3: 13)Wine79 (class1: 36; class2: 7; class3: 36)54 (class1: 23; class2: 19; class3: 12)Vehicle450 (Class1: 30; Class2: 140; Class3: 140; Class4: 140)254 (Class1: 62; Class2: 62; Class3: 59; Class4: 71)Figure 13
(a) Comparisons of the classification results using the Iris dataset. (b) Comparisons of the classification results using the Wine dataset. (c) Comparisons of the classification results using the Vehicle Silhouettes dataset.
(a)(b)(c)Both Figures13(a) and 13(b) indicate that the imbalanced training datasets significantly impact the training of the improved BPNN, which finally leads to severe misclassifications. However, balanced by the Fréchet distance-based Borderline-SMOTE, the network can be correctly and sufficiently trained. Therefore, the classification accuracies can be greatly improved. However, due to the complicated attributes of the instances of the Vehicle Silhouettes dataset, although the classification based on the balanced dataset also outperforms the imbalanced dataset, its average accuracy is only 79.53%. This point indicates that the dimension of the data may severely impact the performance of the class balance algorithm.


## 3.3. Evaluation for the Parallelized Improved BPNN in Hadoop Cluster

Firstly, the original Iris dataset is employed to evaluate the classification accuracy of the parallelized improved BPNN. A number of 105 instances are the training instances which will be separated for the parallelized training. The other 45 instances are the testing instances. A number of three mappers start in parallel. Each mapper initializes one sub-BPNN. The standalone BPNN and the parallelized long short-term memory network (LSTM) are also implemented in terms of comparison. Especially, the configuration of the parallelized LSTM is as the same as that of the parallelized BPNN. The number of the training instances is 105 and the number of the testing instances is 45 for the standalone BPNN.Figure14 indicates that the parallelized BPNN outperforms the standalone BPNN and the parallelized LSTM. Benefitting from the weighted voting, the potential accuracy loss issue in the data separation is properly handled. Actually, the classification accuracies for the 3 sub-BPNNs are 96.88%, 94.44%, and 91.89%, respectively. However, the accuracy for the aggregated result can reach to 100%. It further proves that the weighted voting can help to aggregate the weak classifiers to a strong classifier. Additionally, the data separation may intensify the class imbalance issue of the training dataset in each sub-BPNN. However, the Fréchet distance-based Borderline-SMOTE can effectively solve the issue, which guarantees the classification accuracy. Moreover, the parallelized LSTM performs slightly worse than the parallelized BPNN. It is well known that LSTM is suitable for processing the time-series data. However, the Iris dataset employed for the experiments is not related to time series, which indicates that the type of the dataset may affect the performance of the parallelized LSTM in terms of accuracy.Figure 14
The accuracy comparison for the standalone BPNN, the parallelized BPNN, and the parallelized LSTM.In terms of the classification efficiency evaluation, this paper duplicates the size of the Iris dataset from 1 MB to 1024 MB. The cluster starts 16 mappers in parallel. The processing times of the standalone BPNN, the parallelized BPNN, and the parallelized LSTM are listed in Figure15.Figure 15
Efficiency comparison for the standalone BPNN, the parallelized BPNN, and the parallelized LSTM.Figure15 indicates that the performances of three algorithms are quite close with smaller data sizes. Due to the overhead of the Hadoop cluster, the standalone BPNN even outperforms the parallelized BPNN and the parallelized LSTM. However, along with the increasing data sizes, because of the computing resource limitations of the standalone environment, the processing time of the standalone BPNN increases sharply. Contrarily, due to less computations, the parallelized BPNN outperforms the parallelized LSTM slightly and processes the large volume of data relatively efficiently.


## 4. Conclusion

In order to serve the classifications for large-scale data, this paper presents a parallelized improved BPNN algorithm. The parallelization is based on the data separation, and the parallelization is implemented using the Hadoop framework. To overcome the classification accuracy loss issue caused by the separation, the weighted voting is presented to improve the classification accuracy. Based on the experimental results, the parallelization shows effectiveness for dealing with the large-scale data. However, there are other two issues existing. The first is that the class imbalance issue in the training dataset significantly impacts the training effect of BPNN, which finally leads to the deterioration of the classification accuracy. Therefore, this paper presents the Fréchet distance-based Borderline-SMOTE algorithm in enabling the class balance. According to the experimental results, the balanced training dataset can tremendously improve the classification accuracy. The second is that the convergence issue may exist in BPNN. Therefore, this paper improves the input layer, hidden layer, and activation function of BPNN by employing zero-mean, batch-normalization, and ReLU, respectively. Based on the comparisons to the traditional BPNN and the self-adaptive BPNN, the presented improved BPNN has great potential to serve the classification tasks accurately and efficiently.



---

*Source: 1953461-2020-05-18.xml*