# Improved Pre-miRNA Classification by Reducing the Effect of Class Imbalance

Authors: Yingli Zhong; Ping Xuan; Ke Han; Weiping Zhang; Jianzhong Li
Journal: BioMed Research International
 (2015)

---

## Abstract

MicroRNAs (miRNAs) play important roles in the diverse biological processes of animals and plants. Although the prediction methods based on machine learning can identify nonhomologous and species-specific miRNAs, they suffered from severe class imbalance on real and pseudo pre-miRNAs. We propose a pre-miRNA classification method based on cost-sensitive ensemble learning and refer to it as MiRNAClassify. Through a series of iterations, the information of all the positive and negative samples is completely exploited. In each iteration, a new classification instance is trained by the equal number of positive and negative samples. In this way, the negative effect of class imbalance is efficiently relieved. The new instance primarily focuses on those samples that are easy to be misclassified. In addition, the positive samples are assigned higher cost weight than the negative samples. MiRNAClassify is compared with several state-of-the-art methods and some well-known classification models by testing the datasets about human, animal, and plant. The result of cross validation indicates that MiRNAClassify significantly outperforms other methods and models. In addition, the newly added pre-miRNAs are used to further evaluate the ability of these methods to discover novel pre-miRNAs. MiRNAClassify still achieves consistently superior performance and can discover more pre-miRNAs.


---

## Body

## 1. Introduction

MicroRNAs (miRNAs) are a set of short noncoding RNAs (~22 nt) which bind with the mRNAs to regulate their expression and result in their cleavage or translational repression [1, 2]. The miRNAs usually participate in plenty of biological processes of animals and plants, including the developmental process, hematopoietic process, organogenesis, cell apoptosis, and cell proliferation in animals as well as growth and signal transduction in plants [3, 4]. An effective route for identifying miRNA is to predict the candidates by using computational methods, following by purposeful validation with the biological experiments.For the computational prediction of miRNAs, the methods based on homologous search, comparative genomics, and machine learning are three major categories. Although the first two kinds of methods can accurately identify miRNAs, they cannot identify those nonhomologous and species-specific miRNAs. The main reason is that they depend on sequence homology and sequence conservation among the multiple species. The prediction based on machine learning is ade novo method which can identify nonhomologous and species-specific miRNAs. A prediction method based on machine learning uses real pre-miRNAs and pseudo pre-miRNAs as positive samples and negative samples, respectively. Many features are extracted from these samples according to the sequence and structure characteristics of pre-miRNAs. At last, a classification model is trained by using the samples and their features. So far, researchers have established models based on naive Bayes [5, 6], support vector machine (SVM) [7–9], Random Forest [10], and probabilistic colearning [11] and used these models to discriminate whether a given new sequence is a real pre-miRNA. Those sequences classified as real pre-miRNAs can be regarded as candidates. The position prediction method [12–14] is used to further determine the locations of miRNAs in the candidates. All pre-miRNA candidates and the location information of their miRNAs contribute to the subsequent biological validation.When constructing a classification model, the experimental validated pre-miRNAs (real pre-miRNAs) are used as positive samples. As nearly all reported miRNAs located in the untranslated regions or intergenic regions, the sequences with stem-loop structure (pseudo pre-miRNAs) are collected from the protein-coding regions. These pseudo pre-miRNAs are regarded as negative samples. In reality, the number of negative samples is far greater than positive samples, which forms the severe class imbalance problem. It is well studied that a classification model trained with these samples would tend to determine a new sequence as a pseudo pre-miRNA (the majority class) [15]. Simultaneously, it would result in poor classification with respect to the minority class (real pre-miRNAs).Recently, several pre-miRNA classification methods have taken the imbalance problem into account. Triple-SVM [8] adopted random undersampling method to select the same number of negative samples with that of positive samples. PlantMiRNAPred [16] clustered the positive and negative samples according to their distribution. The equal number of representative positive and negative samples was selected as the training data. HuntMi [17] selected the partial negative samples based on the ROC score. Actually, there are a lot of various pseudo pre-miRNAs (negative samples) in the genomes. However, the methods with undersampling strategies ignored so much information of negative samples that the classification model has poor discriminative ability on them. In addition, miRNApre [18] only extracted the negative samples which are similar to the positive samples. As a result, the corresponding classification model has low robustness especially for the new pre-miRNAs that are not similar to the previous pre-miRNAs. Furthermore, this method also suffered from the negative effect of ignoring plenty of negative samples. On the other hand, microPred [19] generated the simulative positive samples based on SMOTE [20] to balance the numbers of positive and negative samples. Nonetheless, the process of generating simulative samples introduced the noisy data, which decrease the classification accuracy especially for positive samples. Therefore, it is essential to develop a new method to efficiently classify the imbalanced real and pseudo pre-miRNAs.In addition, the correct classification of real pre-miRNAs (positive samples) has greater value than the correct classification of pseudo pre-miRNAs (negative samples). It is helpful for the biologists to discover more novel pre-miRNAs. Hence a better classification model should provide a higher discrimination ability on the positive samples. In order to construct this type of model, cost-sensitive and ensemble learning [21, 22] has taken the various misclassification costs of different categories of samples and the class imbalance into account and achieved superior performance. Therefore, we proposed a classification method based on cost-sensitive and ensemble learning, which completely exploits the class imbalance factor and the cost factor of different categories. Through a series of iterations, the samples that are easy to be classified correctly and those that are easy to be misclassified are determined gradually and are assigned different weights. During each iteration, a new classification instance is trained by using the equal numbers of positive and negative samples. The weights of samples are updated according to the classification result of the new instance. After the iteration process is finished, all the instances are integrated into an ensemble classifier.First, each classification instance is trained by the same amount of positive and negative samples, which can efficiently relieve the negative effect of class imbalance. Second, the cost of misclassifying a positive sample is set higher than misclassifying a negative one. In this way, the classification accuracy on the positive samples is further improved. Third, the information of all the negative samples is exploited during the training process. At last, when a new classification instance is constructed, it focuses on the samples that are easy to be misclassified, which contributes to the improvement of the global classification performance. In addition, the data about human, animal, and plant were used to test our method, several state-of-the-art methods, and some classification models. Our method achieved superior classification performance not only for the cross validation but also for the testing on the newly added pre-miRNAs.The rest of this paper is organized as follows. Section2 describes the features of real/pseudo pre-miRNAs and the process of constructing the ensemble classification model. In Section 3, we discuss the evaluation metrics of classification performance and compare and analyze the results of our method, other methods, and other classification models.


## 2. Methods

### 2.1. Features of Pre-miRNAs

It has been well studied that the pre-miRNAs have some characteristics about their sequences, secondary structures, and energy [23, 24]. The sequences and secondary structure of pre-miRNAs are usually conserved. The secondary structures often contain stem and loop regions. Furthermore, they have lower free energy. The previous pre-miRNA classification methods usually extracted different feature sets, as they focused on different characteristics of pre-miRNAs. So far, miPred [10], microPred [19], triplet-SVM [8], and miRNApre [18] have extracted the sequence-related, structure-related, and energy-related features. All the features are summarized as follows.First, there are 81 sequence-related features. (1) 16 features are related to the two adjacent nucleotides, and they represent the frequencies of two adjacent nucleotides, denoted as XY%. X and Y can be A (adenine), G (guanine), C (cytosine), or U (uracil). (2) The G + C% is used to describe the total content of G and C in the sequences. (3) 64 features are related to the frequencies of three adjacent nucleotides and denoted as XYZ%, where X, Y, and Z∈{A,G,C,U}.The following 49 features are related to the secondary structures of pre-miRNAs which can be obtained by using the structure-related prediction software such as the Vienna software package RNAfold [25]. (1) There are 8 features about the structure topology, including the structural diversity propertyDiversity, the structural frequency propertyFreq, the structural entropy-related propertiesdS anddS/L, the structural enthalpy-related propertiesdH anddH/L, the Shannon entropy valuedQ, and the compactness of topologydF. (2) There are 9 features about base pairs, including the average amounts of base pairs |A-U|/L, |G-C|/L, and |G-U|/L, the percentages of base pairs within the stem region |A-U|%/n_stems, |G-C|%/n_stems, and |G-U|%/n_stems, the average number of base pairs in the stem regionAvg_BP_Stem, the distance between a pair of basesdD, and the base-pairing tendencydP. (3) There are 32 features related to the characteristic of three adjacent nucleotides. “(” and “.” represent paired and unpaired bases, respectively. The nucleotide at the middle is recorded as A, G, C, or U. Thus, the percentages of the 32 combinations can be obtained.Finally, there are 9 features related to the energy of secondary structure: (1) the minimal free energy of the secondary structuredG, the minimal free energy-related featuresMFEI1,MFEI2,MFEI3, andMFEI4, and the overall free energyNEFE, the energy combination features Diff=|MFE-EFE|/L, and (2) the energy required for dissolving the secondary structure Tm and Tm/L.These features have been successfully used to classify the real and pseudo pre-miRNAs. Therefore, the total of 139 features were merged to form our feature set. During the process of our feature extraction, the codes of miPred, microPred, triplet-SVM, and miRNApre were executed to obtain the respective features. All the extracted features were merged to represent each of real and pseudo pre-miRNAs.


### 2.2. Constructing the Cost-Sensitive Ensemble Model

A pre-miRNA classification method based on cost-sensitive ensemble learning is proposed and referred to as MiRNAClassify. We constructed multiple classification instances through a series of iterations and integrated these instances into an ensemble classifier. In this way, the information of all the positive and negative samples can be exploited. In the process of each iteration, a new classification instance is trained by the equal number of positive and negative samples, which can protect it from the negative effect of class imbalance. At the same time, each sample has its own weight which reflects the degree that it is easy to be misclassified. The training samples are selected in proportion to their weights, which make the new classification instance focus on the samples that are easy to be misclassified. Moreover, the weights of samples are adjusted according to the classification result of each instance. Hence when constructing a new instance, the new weight distribution of samples can be integrated to obtain a more accurate instance. In addition, as the cost of misclassifying each positive sample is higher than each negative sample, the positive samples are assigned greater cost weights. As shown in Figure1, constructing the ensemble classification model includes the following 4 steps.Figure 1
Constructing the integrated model to classify the real/pseudo pre-miRNAs.(1) The sample setV contains all the positive samples (real pre-miRNAs) and the negative samples (pseudo pre-miRNAs). n is the number of the features. We extracted n features from each positive sample and each negative one. Since the purpose of our cost-sensitive learning is to improve the discrimination ability on the positive samples (small class), the cost weights of positive samples should be set higher than those of negative samples. CP and CN represent the misclassification cost of positive and negative samples, respectively. The weight of each positive sample is initially set to CP/|V| and that of each negative one is set to CN/|V| where |V| is the size of V. As CP should be higher than CN, CP was set as 1 and CN varied from 0.1 to 0.9 with step length of 0.1. The best classification performance was achieved when CP=1 and CN=0.6.During the process of the first iteration, in then-dimensional sample space, the distance between any two of negative samples is calculated. Next, all the negative samples in V are gathered into k clusters based on K-means, and k equals the number of positive samples. For each cluster, the negative sample that is closest to the center is obtained to be a training sample. Thus the initial training dataset A1 is composed of k positive samples and k negative samples. The first classification instance based on the support vector machine (SVM) is trained by A1 and denoted as M1. The purpose of constructing M1 according to above steps is to quickly determine the positive and negative samples that are easy to be misclassified. As M1 is trained by the same number of positive and negative samples, M1 was not affected by the class imbalance.It has been confirmed that Rogers and Tanimoto measurement [26] could successfully measure the similarity between pre-miRNAs [16]. On the basis of the similarity measurement, the distance between two negative samples, such as x and y, is defined as follows:(1)disvx,vy=1-vxt·vyvxt·vx+vyt·vy-vxt·vy,where vx and vy are the feature vectors of x and y, respectively, and vxt and vyt represent their transpose.(2) During the process of each subsequent iteration, the classification instanceMt (1≤t≤T) classifies all the positive and negative samples in V. Gmt represents the global classification performance of Mt and it is described in detail in Section 3.2. The weight of each positive and negative sample is adjusted according to the classification results of Mt. The weight of a sample that was classified correctly is reduced and that of a sample that was misclassified remains unchanged. After a series of iterations were completed, a greater weight represents that a sample has been misclassified for more times.AsGmt is Mt’s classification performance, the error rate of Mt is φt=1-Gmt. θt is a parameter for adjusting the weight of each sample and θt=φt/(1-φt). Thus, the weight of the ith sample xi at the tth iteration is updated as wit-1θt1-pt(xi)-yi. Here, wit-1 is the weight of xi at the (t-1)th iteration, and yi is the true label of xi. If xi is a real pre-miRNA, yi is 1. Otherwise, xi is a pseudo pre-miRNA and yi is 0. pt(xi) represents the classification result of Mt over xi, and 1 and 0 represent that xi is classified as positive sample and negative one, respectively. If xi is classified correctly, the value of pt(xi) is the same as its true label yi, we have |pt(xi)-yi|=0 and 1-|pt(xi)-yi|=1. The weights of the correctly classified samples are multiplied by θt which is smaller than 1. Moreover, if the classification error rate is lower, the weights of the samples will become smaller. In terms of the misclassified samples, we have |pt(xi)-yi|=1 and 1-|pt(xi)-yi|=0. Thus, their weights remain unchanged.(3) Assume the number of positive samples is k. To make the balance of positive and negative samples, we also selected k negative samples. A negative sample with greater weight means that it has been misclassified for more times in the previous iterations. Therefore, we selected the negative samples in proportion to their weights. A new classification instance Mt+1 is trained by k negative samples and k positive samples. At the same time, the weights of the training samples are integrated.(4) Steps (2) and (3) are repeatedly performed until the termination condition is satisfied.At last,T classification instances are constructed and denoted as M1,M2,…,MT. The results of all instances are integrated based on the voting mechanism to give the final classification result. If the classification error rate φt is lower, θt is smaller and the determination weight log⁡(1/θt) is greater. In this way, the result given by a instance with higher classification performance accounts for the greater proportion for the final result during the voting process. The algorithm of classification of the real/pseudo pre-miRNAs based on cost-sensitive ensemble learning is illustrated in Algorithm 1.Algorithm 1:Algorithm of classifying the real/pseudo pre-miRNAs based on cost-sensitive ensemble learning.
Input: a dataset, V, including all the positive and negative samples, and the negative samples are more than the positive samples.
Output: ensemble classifier based on integrating multiple classification instances.
(1) Fori=1 to V
(2)If xi is a positive sample
(3)wi1=CP/V, wi1 is the weight of xi
(4)else
(5)wi1=CN/V
(6)End If
(7) End For
(8)t is used to record the current iteration number, and its initial value is set as 1
(9) While (t≤T)
(10)Dt = Null, the negative training set Dt is emptied
(11) Ift equals 1
(12)  All negative samples are gathered intok clusters based on the K-means method. Assume set P is composed of all the
positive samples and the parameter k=P
(13)  For each cluster, the sample locating closest to the center is selected and added intoDt. Furthermore, the number of
negative samples Dt is equal to that of positive samples P
(14) else
(15)  According to the weights of negative samples,k negative samples are selected in proportion to their weights.
These samples are added into Dt and Dt=P
(16) End If
(17) The training dataset is composed ofDt and P. A new classification instance Mt based on SVM is constructed by using
the training dataset and integrating their weight distribution
(18)Mt is used to classify all the samples in V, evaluate its classification performance Gm,
and compute its classification error rate φt
(19) The adjustment weightθt=φt/1-φt is calculated, and the weight of each positive and negative sample
is updated by using the rule wit=wit-1θt1-ptxi-yi
(20)t=t+1
(21) End While
(22) An integrated classifier is constructed by integratingT classification instances based on the voting mechanism.
The final classification result is obtained as follows.
pf(x)=1if∑t=1Tlog1θtpt(x)≥12∑t=1Tlog1θt0otherwiseThe iterative process is terminated after the while loop is performed forT times. When the value of T is large enough, the error rate of the ensemble classifier can reach values as small as possible. In this study, the value of T is set as 300.


## 2.1. Features of Pre-miRNAs

It has been well studied that the pre-miRNAs have some characteristics about their sequences, secondary structures, and energy [23, 24]. The sequences and secondary structure of pre-miRNAs are usually conserved. The secondary structures often contain stem and loop regions. Furthermore, they have lower free energy. The previous pre-miRNA classification methods usually extracted different feature sets, as they focused on different characteristics of pre-miRNAs. So far, miPred [10], microPred [19], triplet-SVM [8], and miRNApre [18] have extracted the sequence-related, structure-related, and energy-related features. All the features are summarized as follows.First, there are 81 sequence-related features. (1) 16 features are related to the two adjacent nucleotides, and they represent the frequencies of two adjacent nucleotides, denoted as XY%. X and Y can be A (adenine), G (guanine), C (cytosine), or U (uracil). (2) The G + C% is used to describe the total content of G and C in the sequences. (3) 64 features are related to the frequencies of three adjacent nucleotides and denoted as XYZ%, where X, Y, and Z∈{A,G,C,U}.The following 49 features are related to the secondary structures of pre-miRNAs which can be obtained by using the structure-related prediction software such as the Vienna software package RNAfold [25]. (1) There are 8 features about the structure topology, including the structural diversity propertyDiversity, the structural frequency propertyFreq, the structural entropy-related propertiesdS anddS/L, the structural enthalpy-related propertiesdH anddH/L, the Shannon entropy valuedQ, and the compactness of topologydF. (2) There are 9 features about base pairs, including the average amounts of base pairs |A-U|/L, |G-C|/L, and |G-U|/L, the percentages of base pairs within the stem region |A-U|%/n_stems, |G-C|%/n_stems, and |G-U|%/n_stems, the average number of base pairs in the stem regionAvg_BP_Stem, the distance between a pair of basesdD, and the base-pairing tendencydP. (3) There are 32 features related to the characteristic of three adjacent nucleotides. “(” and “.” represent paired and unpaired bases, respectively. The nucleotide at the middle is recorded as A, G, C, or U. Thus, the percentages of the 32 combinations can be obtained.Finally, there are 9 features related to the energy of secondary structure: (1) the minimal free energy of the secondary structuredG, the minimal free energy-related featuresMFEI1,MFEI2,MFEI3, andMFEI4, and the overall free energyNEFE, the energy combination features Diff=|MFE-EFE|/L, and (2) the energy required for dissolving the secondary structure Tm and Tm/L.These features have been successfully used to classify the real and pseudo pre-miRNAs. Therefore, the total of 139 features were merged to form our feature set. During the process of our feature extraction, the codes of miPred, microPred, triplet-SVM, and miRNApre were executed to obtain the respective features. All the extracted features were merged to represent each of real and pseudo pre-miRNAs.


## 2.2. Constructing the Cost-Sensitive Ensemble Model

A pre-miRNA classification method based on cost-sensitive ensemble learning is proposed and referred to as MiRNAClassify. We constructed multiple classification instances through a series of iterations and integrated these instances into an ensemble classifier. In this way, the information of all the positive and negative samples can be exploited. In the process of each iteration, a new classification instance is trained by the equal number of positive and negative samples, which can protect it from the negative effect of class imbalance. At the same time, each sample has its own weight which reflects the degree that it is easy to be misclassified. The training samples are selected in proportion to their weights, which make the new classification instance focus on the samples that are easy to be misclassified. Moreover, the weights of samples are adjusted according to the classification result of each instance. Hence when constructing a new instance, the new weight distribution of samples can be integrated to obtain a more accurate instance. In addition, as the cost of misclassifying each positive sample is higher than each negative sample, the positive samples are assigned greater cost weights. As shown in Figure1, constructing the ensemble classification model includes the following 4 steps.Figure 1
Constructing the integrated model to classify the real/pseudo pre-miRNAs.(1) The sample setV contains all the positive samples (real pre-miRNAs) and the negative samples (pseudo pre-miRNAs). n is the number of the features. We extracted n features from each positive sample and each negative one. Since the purpose of our cost-sensitive learning is to improve the discrimination ability on the positive samples (small class), the cost weights of positive samples should be set higher than those of negative samples. CP and CN represent the misclassification cost of positive and negative samples, respectively. The weight of each positive sample is initially set to CP/|V| and that of each negative one is set to CN/|V| where |V| is the size of V. As CP should be higher than CN, CP was set as 1 and CN varied from 0.1 to 0.9 with step length of 0.1. The best classification performance was achieved when CP=1 and CN=0.6.During the process of the first iteration, in then-dimensional sample space, the distance between any two of negative samples is calculated. Next, all the negative samples in V are gathered into k clusters based on K-means, and k equals the number of positive samples. For each cluster, the negative sample that is closest to the center is obtained to be a training sample. Thus the initial training dataset A1 is composed of k positive samples and k negative samples. The first classification instance based on the support vector machine (SVM) is trained by A1 and denoted as M1. The purpose of constructing M1 according to above steps is to quickly determine the positive and negative samples that are easy to be misclassified. As M1 is trained by the same number of positive and negative samples, M1 was not affected by the class imbalance.It has been confirmed that Rogers and Tanimoto measurement [26] could successfully measure the similarity between pre-miRNAs [16]. On the basis of the similarity measurement, the distance between two negative samples, such as x and y, is defined as follows:(1)disvx,vy=1-vxt·vyvxt·vx+vyt·vy-vxt·vy,where vx and vy are the feature vectors of x and y, respectively, and vxt and vyt represent their transpose.(2) During the process of each subsequent iteration, the classification instanceMt (1≤t≤T) classifies all the positive and negative samples in V. Gmt represents the global classification performance of Mt and it is described in detail in Section 3.2. The weight of each positive and negative sample is adjusted according to the classification results of Mt. The weight of a sample that was classified correctly is reduced and that of a sample that was misclassified remains unchanged. After a series of iterations were completed, a greater weight represents that a sample has been misclassified for more times.AsGmt is Mt’s classification performance, the error rate of Mt is φt=1-Gmt. θt is a parameter for adjusting the weight of each sample and θt=φt/(1-φt). Thus, the weight of the ith sample xi at the tth iteration is updated as wit-1θt1-pt(xi)-yi. Here, wit-1 is the weight of xi at the (t-1)th iteration, and yi is the true label of xi. If xi is a real pre-miRNA, yi is 1. Otherwise, xi is a pseudo pre-miRNA and yi is 0. pt(xi) represents the classification result of Mt over xi, and 1 and 0 represent that xi is classified as positive sample and negative one, respectively. If xi is classified correctly, the value of pt(xi) is the same as its true label yi, we have |pt(xi)-yi|=0 and 1-|pt(xi)-yi|=1. The weights of the correctly classified samples are multiplied by θt which is smaller than 1. Moreover, if the classification error rate is lower, the weights of the samples will become smaller. In terms of the misclassified samples, we have |pt(xi)-yi|=1 and 1-|pt(xi)-yi|=0. Thus, their weights remain unchanged.(3) Assume the number of positive samples is k. To make the balance of positive and negative samples, we also selected k negative samples. A negative sample with greater weight means that it has been misclassified for more times in the previous iterations. Therefore, we selected the negative samples in proportion to their weights. A new classification instance Mt+1 is trained by k negative samples and k positive samples. At the same time, the weights of the training samples are integrated.(4) Steps (2) and (3) are repeatedly performed until the termination condition is satisfied.At last,T classification instances are constructed and denoted as M1,M2,…,MT. The results of all instances are integrated based on the voting mechanism to give the final classification result. If the classification error rate φt is lower, θt is smaller and the determination weight log⁡(1/θt) is greater. In this way, the result given by a instance with higher classification performance accounts for the greater proportion for the final result during the voting process. The algorithm of classification of the real/pseudo pre-miRNAs based on cost-sensitive ensemble learning is illustrated in Algorithm 1.Algorithm 1:Algorithm of classifying the real/pseudo pre-miRNAs based on cost-sensitive ensemble learning.
Input: a dataset, V, including all the positive and negative samples, and the negative samples are more than the positive samples.
Output: ensemble classifier based on integrating multiple classification instances.
(1) Fori=1 to V
(2)If xi is a positive sample
(3)wi1=CP/V, wi1 is the weight of xi
(4)else
(5)wi1=CN/V
(6)End If
(7) End For
(8)t is used to record the current iteration number, and its initial value is set as 1
(9) While (t≤T)
(10)Dt = Null, the negative training set Dt is emptied
(11) Ift equals 1
(12)  All negative samples are gathered intok clusters based on the K-means method. Assume set P is composed of all the
positive samples and the parameter k=P
(13)  For each cluster, the sample locating closest to the center is selected and added intoDt. Furthermore, the number of
negative samples Dt is equal to that of positive samples P
(14) else
(15)  According to the weights of negative samples,k negative samples are selected in proportion to their weights.
These samples are added into Dt and Dt=P
(16) End If
(17) The training dataset is composed ofDt and P. A new classification instance Mt based on SVM is constructed by using
the training dataset and integrating their weight distribution
(18)Mt is used to classify all the samples in V, evaluate its classification performance Gm,
and compute its classification error rate φt
(19) The adjustment weightθt=φt/1-φt is calculated, and the weight of each positive and negative sample
is updated by using the rule wit=wit-1θt1-ptxi-yi
(20)t=t+1
(21) End While
(22) An integrated classifier is constructed by integratingT classification instances based on the voting mechanism.
The final classification result is obtained as follows.
pf(x)=1if∑t=1Tlog1θtpt(x)≥12∑t=1Tlog1θt0otherwiseThe iterative process is terminated after the while loop is performed forT times. When the value of T is large enough, the error rate of the ensemble classifier can reach values as small as possible. In this study, the value of T is set as 300.


## 3. Results and Discussion

### 3.1. Data Preparation

Both the sample set and the feature set are important factors influencing the pre-miRNA classification. Furthermore, the previous methods extracted various features because they focused on the different characteristics of pre-miRNAs. Therefore, in order to fairly compare with each of previous methods, our classification method was also trained by the sample set and feature set which were the same as its ones. The information about the sample and feature of each compared method is listed as follows.MicroPred collected 691 real pre-miRNAs from the early version of miRNA database miRBase [27] (12.0 version, abbreviated as miRBase12.0) as the positive samples. The 660 pre-miRNAs have the stem-loop secondary structures and 31 pre-miRNAs have multiple stem-loops. These positive samples formed a positive dataset which is referred to as “MP_positive_set.” MicroPred obtained the 8494 pseudo hairpins as the negative samples and these hairpins were extracted from the human protein-coding regions. In addition, microPred collected 754 other types of noncoding RNAs, such as tRNA and rRNA, as the negative samples. The total 9248 negative samples formed a negative dataset named “MP_negative_set.” In terms of feature set, microPred obtained 21 features to train its classification model.PlantMiRNAPred was a classical method used to classify the plant pre-miRNAs. It collected the 2043 plant pre-miRNAs in the miRBase14.0 as the positive samples and extracted the 2122 pseudo hairpins from the protein-coding regions ofArabidopsis thaliana andGlycine max as the negative samples. The positive samples and negative samples of PlantMiRNAPred formed its positive dataset “PMP_positive_set” and negative one “PMP_negative_set.” PlantMiRNAPred extracted 68 features from plant pre-miRNAs.HuntMi obtained 1406 human pre-miRNAs from miRBase17.0 and extracted 81228 human pseudo hairpins to form the human positive dataset “HM_hsa_positive_set” and negative one “HM_hsa_negative_set.” It also obtained 7053 real animal pre-miRNAs and 218154 animal pseudo hairpins to construct the animal positive dataset “HM_animal_positive_set” and negative one “HM_animal_negative_set,” respectively. In addition, it obtained 2172 plant pre-miRNAs and 114929 plant pseudo hairpins to form the plant positive dataset “HM_plant_positive_set” and negative one “HM_plant_negative_set.” HuntMi used 28 features to train its model.MiRNApre obtained 1496 human pre-miRNAs from miRBase17.0 as the positive samples and extracted 1446 pseudo hairpins similar with the positive samples. These pseudo hairpins are used as the negative samples. These positive samples and negative samples formed the positive and negative datasets “MP_positive_set” and “MP_negative_set.” MiRNApre extracted 98 features from each positive and negative sample.Moreover, the new pre-miRNAs about human,Arabidopsis lyrata,Oryza sativa, andGlycine max were reported by miRBase when our work was almost completed. These pre-miRNAs were also used to test the classification methods, which can further validate their ability of discovering the new pre-miRNAs.In addition, to compare our method with other methods simultaneously, all the positive and negative samples about human, animal, and plant were merged, respectively. The 1496 human pre-miRNAs and 81982 pseudo hairpins formed the human-related positive and negative datasets which are referred to as “Merged_hsa_positive_set” and “Merged_hsa_negative_set.” The animal-related positive and negative datasets contain 7053 real animal pre-miRNAs and 218154 pseudo hairpins and they are named “Merged_animal_positive_set” and “Merged_animal_negative_set.” The plant-related datasets are “Merged_plant_positive_set” and “Merged_plant_negative_set” which are composed of 2172 plant real pre-miRNAs and 117051 pseudo hairpins. All the classification methods were tested by performing 5-fold cross validation on the three groups of datasets. The 355 new human pre-miRNAs (updated human dataset), 68Arabidopsis thaliana pre-miRNAs (updated ath dataset), 169Oryza sativa pre-miRNAs (updated osa dataset), and 302Glycine max pre-miRNAs (updated gma dataset) were also used for testing. In terms of features, the 139 features described in Section 2.1 are used to represent each positive sample and negative one.


### 3.2. Performance Evaluation Metrics

Suppose that TP and TN represent the number of the correctly classified positive samples (real pre-miRNAs) and that of the correctly classified negative samples (pseudo pre-miRNAs), respectively. FP and FN represent the numbers of the misclassified positive and negative samples, respectively. Sensitivity (SE) represents the proportion of the positive samples that are classified successfully accounting for the total positive samples. Specificity (SP) represents the proportion of the successfully classified negative samples accounting for the total negative samples. Consider(2)SE=TPTP+FN,SP=TNTN+FP.For the pre-miRNA classification, if a classifier has higher SE and lower SP, it has poorer ability to identify the pseudo pre-miRNAs. Thus, many pseudo pre-miRNAs will be misclassified as the real pre-miRNAs, which reduces the possibility that the biological experiments can successfully identify pre-miRNAs. On the contrary, the classifier has poorer ability to identify the real pre-miRNAs, and many real pre-miRNAs will be misclassified as the pseudo pre-miRNAs, which reduces the possibility that the real pre-miRNAs are discovered by the experimental study. Therefore, the classifier should have both high SE and high SP.The global classification accuracy based on machine learning is usually evaluated by the parameter Acc. However, the number of pseudo pre-miRNAs is usually much greater than that of real pre-miRNAs, which causes TN and FP to be much higher than TP and FN. Then we have(3)Acc=TP+TNTP+TN+FP+FN≈TNTN+FP=SP.Therefore, besides SE and SP, we also compute the geometric mean of SE and SP, denoted as Gm, to evaluate the global classification performance:(4)Gm=SE×SP.


### 3.3. Comparison with Other Methods and Classification Models

In order to compare MiRNAClassify with the state-of-the-art methods microPred [19], PlantMiRNAPred [16], HuntMi [17], and miRNApre [18], the 5-fold cross validation is performed. During the process of the cross validation, the positive and negative samples are randomly divided into 5 parts. 4 parts are used as the training samples, and the remaining part is used for testing. The testing dataset does not intersect with the training dataset. So it can objectively test the classification performances of the methods. Besides cross validation, the newly added human, animal, and plant pre-miRNAs are used to test the ability of discovering the new pre-miRNAs.MiRNAClassify is compared with other methods in two different ways. For one thing, in terms of each compared method, we choose its sample set and feature set to train MiRNAClassify. In this way, we compare MiRNAClassify with other methods, respectively. For another, MiRNAClassify is compared with all the other methods simultaneously by using the merged datasets and our feature set. In addition, MiRNAClassify is compared with other classification models, including SVM, naive Bayes, and Random Forest.


#### 3.3.1. Comparison Using the Same Dataset and Feature Set

MicroPred focused on classification of the human pre-miRNAs, and it generated new simulated positive samples based on SMOTE. In order to compare with microPred in a fair way, we used the sample set and feature set of microPred to train the model of MiRNAClassify. As shown in Figure2 (details in Table 1), the first 3 columns are the results based on 5-fold cross validation, and the last column is the result for testing the newly added human pre-miRNAs. MicroPred obtained the lower sensitivity (SE=90.65%) and specificity (SP=92.16%) in the cross validation. The possible reason is that generating samples based on SMOTE introduced the noise data. MiRNAClassify is 5.31% better than microPred in overall accuracy Gm. SE increased by 5.75% and SP increased by 4.87%. For the dataset that composed of 1186 newly added human pre-miRNAs, MP_updated_set, the SEs of MiRNAClassify and microPred are 90.56% and 85.16%, respectively. These SEs are not as good as those in the cross validation. The main reason is that MicroPred used the pre-miRNAs in the early version of miRBase and the new pre-miRNAs for testing are even more than the pre-miRNAs used to train its classification model.Table 1
Datasets and detailed classification results of MiRNAClassify and microPred.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
MP_positive_set
Real
691
96.40
97.03
96.71
microPred
MP_negative_set
Pseudo
9248
90.65
92.16
91.40
MiRNAClassify
MP_updated_set
Real
1186
90.56
microPred
85.16Figure 2
Comparison of the performances of MiRNAClassify and microPred.PlantMiRNAPred mainly studied the classification of plant pre-miRNAs. The model of MiRNAClassify was constructed by using the same sample and feature sets with PlantMiRNAPred. As shown in Figure3 (details in Table 2), the first 3 columns are the comparison results of cross validation. MiRNAClassify achieved slightly better performance than PlantMiRNAPred. PlantMiRNAPred only extracted a small number of pseudo hairpins. The number of negative samples is close to that of positive samples, and there is no obvious class imbalance. It confirms that MiRNAClassify is still effective for this type of data. Actually, PlantMiRNAPred only selected partial positive and negative samples to train their model, which results in some of the sample information was lost. On the contrary, MiRNAClassify fully exploited the information from all the positive and negative samples by adjusting the sample weights and constructing multiple classification instances. In Figure 3, the last 3 columns are the results on 138 newly addedArabidopsis thaliana pre-miRNAs, 178Oryza sativa pre-miRNAs, and 488Glycine max pre-miRNAs. MiRNAClassify obtained consistently better classification accuracies.Table 2
Datasets and detailed classification results of MiRNAClassify and PlantMiRNAPred.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Plant
PMP_positive_set
Real
2043
96.57
98.35
97.46
PlantMiRNAPred
PMP_negative_set
Pseudo
2122
95.10
97.17
96.13
MiRNAClassify
Arabidopsis thaliana
PMP_ath_updated_set
Real
138
92.75
PlantMiRNAPred
90.58
MiRNAClassify
Oryza sativa
PMP_osa_updated_set
Real
178
77.53
PlantMiRNAPred
67.42
MiRNAClassify
Glycine max
PMP_gma_updated_set
Real
488
90.78
PlantMiRNAPred
86.07Figure 3
Comparison of the performances of MiRNAClassify and PlantMiRNAPred.HuntMi was constructed to distinguish the real and pseudo pre-miRNAs about human, animal, and plant. The model of MiRNAClassify was also trained by using its samples and features. As Figure4 shows (details in Table 3), MiRNAClassify and HuntMi performed the cross validation on three groups of datasets, respectively. It is obvious that the number of negative samples is far more than that of positive samples. There is a severe imbalance between the positive samples and negative samples. Although there are quite a lot of negative samples, HuntMi selected a small number of negative samples based on the ROC values to train its model. Apparently, it lost a large amount of information of negative samples. MiRNAClassify completely presented its advantage in processing the class imbalance and achieved superior performance. In addition, we performed further testing using the newly added human,Arabidopsis thaliana,Oryza sativa, andGlycine max pre-miRNAs. The result was demonstrated in Figure 5, which confirms that MiRNAClassify can discover more new pre-miRNAs. We found that the classification performances of these two methods on newly added plant pre-miRNAs were worse than the performances on the new human pre-miRNAs. One of the important reasons is that the sequences and structures of plant pre-miRNAs are more complicated than those of human pre-miRNAs.Table 3
Datasets and detailed classification results of MiRNAClassify and HuntMi.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
HM_hsa_positive_set
Real
1406
97.15
98.35
97.75
HuntMi
HM_hsa_negative_set
Pseudo
81228
95.02
96.94
95.98
MiRNAClassify
Animal
HM_animal_positive_set
Real
7053
95.74
97.55
96.64
HuntMi
HM_animal_negative_set
Pseudo
218154
94.11
95.95
95.03
MiRNAClassify
Plant
HM_plant_positive_set
Real
2172
93.32
97.82
95.54
HuntMi
HM_plant_negative_set
Pseudo
114929
91.71
95.87
93.77
MiRNAClassify
Homo sapiens
HM_hsa_updated_set
Real
445
93.93
HuntMi
92.14
MiRNAClassify
Arabidopsis thaliana
HM_ath_updated_set
Real
68
92.65
HuntMi
91.18
MiRNAClassify
Oryza sativa
HM_osa_updated_set
Real
169
75.74
HuntMi
69.82
MiRNAClassify
Glycine max
HM_gma_updated_set
Real
302
92.72
HuntMi
88.41Figure 4
Comparison of the performances of MiRNAClassify and HuntMi based on 5-fold cross validation.Figure 5
Comparison of the performances of MiRNAClassify and HuntMi on the newly added data.Although miRNApre was tested on human, animal, and plant, only the training and testing samples about human could be obtained. Therefore, the human-related dataset was used to train the model of MiRNAClassify. Since miRNApre only selected the negative samples which are similar to the positive ones, the negative dataset is only composed of 1446 samples. The positive dataset contains 1496 samples (Table4). There is no class imbalance because the number of positive samples nearly equals that of negative samples. In this case, the cross validation performance of MiRNAClassify is still better than miRNApre, as shown in the first 3 columns of Figure 6. In addition, we obtained 355 new human pre-miRNAs for further testing. The SEs of MiRNAClassify and miRNApre are 91.2% and 90.1%, respectively. The SEs are not as good as those obtained in the cross validation. The reason may be that the extracted negative samples were very similar to the positive samples which resulted in the low robustness of the classification models.Table 4
Datasets and detailed classification results of MiRNAClassify and miRNApre.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
MP_positive_set
Real
1496
98.33
98.27
98.29
miRNApre
MP_negative_set
Pseudo
1446
97.66
97.23
97.44
MiRNAClassify
MP_updated_set
Real
355
91.27
miRNApre
90.14Figure 6
Comparison of the performances of MiRNAClassify and miRNApre.


#### 3.3.2. Comparison over the Merged Datasets

Besides comparison with each method, respectively, we tested MiRNAClassify and other methods simultaneously by using the merged datasets about human, animal, and plant and the newly updated datasets and extracting a set of same features. It is worth noting that the number of negative samples is much greater than that of positive samples in each merged dataset. The classification results are demonstrated in Table5.Table 5
The classification results of MiRNAClassify and other methods over the merged datasets and the updated datasets.
Method
Accuracy (%)
Merged human dataset
Merged animal dataset
Merged plant dataset
Updated human dataset
Updated ath dataset
Updated osa dataset
Updated gma dataset
MiRNAClassify
SE
97.93
95.85
93.37
SP
98.30
97.62
97.91
94.08
92.65
79.29
93.05
G
m
98.11
96.73
95.61
MicroPred
SE
92.25
91.61
89.50
SP
95.70
94.85
93.10
91.27
89.71
66.86
85.76
G
m
93.96
93.21
91.28
PlantMiRNAPred
SE
93.58
92.70
91.39
SP
91.20
88.60
87.10
92.11
89.71
70.42
88.41
G
m
92.38
90.63
89.22
HuntMi
SE
95.32
94.14
91.76
SP
97.11
96.07
95.94
92.68
91.18
72.78
89.07
G
m
96.21
95.10
93.83
miRNApre
SE
97.39
93.49
91.71
SP
90.90
89.80
88.10
90.14
89.71
71.01
86.09
G
m
94.09
91.63
89.89MiRNAClassify performed the best not only over the merged datasets but also over the updated datasets. HuntMiRNA also achieved decent prediction performance. MicroPred, PlantMiRNAPred, and miRNApre obtained the inferior results, especially over the negative samples. In terms of MicroPred, more simulated positive samples have to be generated because there are so many negative samples, which also introduced more noisy data. The possible reason for the worse performance of PlantMiRNAPred and miRNApre is that most of information about negative samples was abandoned and their discriminative ability on negative samples was reduced greatly.In addition, a pairedt-test was used to determine whether MiRNAClassify’s global performance (Gm) over the 3 groups of merged datasets and its accuracy (SE) over the 4 updated datasets is higher than other methods. The corresponding p values are listed in Table 6. The statistic results confirm that MiRNAClassify outperforms other methods significantly at the significance level 0.05.Table 6
The statistic results obtained by using pairedt-test over the prediction performance of MiRNAClassify and that of another method.
Different datasets
microPred
PlantMiRNAPred
HuntMi
miRNApre
p values on three groups of datasets
0.0019
4.9374
e
-
04
9.7070
e
-
04
0.0050
p values on four updated datasets
0.0339
0.0284
0.0354
0.0108


#### 3.3.3. Comparison with Other Classification Models

We use 5-fold cross validation to compare MiRNAClassify with other well-known classification models including the standard SVM, naive Bayes, and Random Forest. These models under SMOTE method were further developed to compare with MiRNAClassify. All the models were trained by the merged datasets and the same set of features, and their classification results are demonstrated in Table7.Table 7
The classification results of MiRNAClassify and three classification models over the merged datasets.
Classification models
SE (%)
SP (%)
G
m (%)
Human
SVM
69.18
99.83
83.11
SVM + SMOTE
92.25
95.70
93.96
Naive Bayes
87.43
96.12
91.67
Naive Bayes + SMOTE
90.24
94.43
92.31
Random Forest
67.78
99.82
82.26
Random Forest + SMOTE
91.51
95.34
93.41
MiRNAClassify
97.93
98.30
98.11
Animal
SVM
69.03
98.14
82.31
SVM + SMOTE
91.61
94.85
93.21
Naive Bayes
85.04
95.03
89.90
Naive Bayes + SMOTE
90.83
92.61
91.71
Random Forest
69.52
98.72
82.84
Random Forest + SMOTE
91.12
95.01
93.05
MiRNAClassify
95.85
97.62
96.73
Plant
SVM
68.51
99.24
82.45
SVM + SMOTE
89.50
93.10
91.28
Naive Bayes
82.91
96.75
89.57
Naive Bayes + SMOTE
87.20
92.61
89.86
Random Forest
68.32
99.35
82.39
Random Forest + SMOTE
89.18
92.87
91.01
MiRNAClassify
93.37
97.91
95.61For the datasets with severe class imbalance, MiRNAClassify demonstrated its ability to process the imbalanced data and achieved the best performance. As expected, the standard classification models overlearned the information of majority class and obtained the small SEs and the great SPs. After the SMOTE method was applied to balance the positive and negative samples, these models except naive Bayes obtained decent improvement onGm values. The main reason is that naive Bayes has intrinsic resistance to the class imbalance. However, the classification models under SMOTE method, SVM, and Random Forest obtained nearly consistent performances which were slightly better than naive Bayes. It confirmed that SVM has the excellent generalization ability. In particular, each classification instance of MiRNAClassify was established based on SVM, which was one of the important factors that MiRNAClassify could perform well. After exerting t-test on the Gm values obtained by MiRNAClassify and other models, the statistic results in Table 8 indicated that MiRNAClassify achieved significantly better performance.Table 8
The statistic results obtained by using pairedt-test over the prediction performance of MiRNAClassify and that of another classification model.
p value
SVM
Naive Bayes
Random Forest
7.3053
e
-
04
6.2651
e
-
04
0.0015
p value
SVM + SMOTE
Naive Bayes + SMOTE
Random Forest + SMOTE
0.0019
0.0010
0.0028


## 3.1. Data Preparation

Both the sample set and the feature set are important factors influencing the pre-miRNA classification. Furthermore, the previous methods extracted various features because they focused on the different characteristics of pre-miRNAs. Therefore, in order to fairly compare with each of previous methods, our classification method was also trained by the sample set and feature set which were the same as its ones. The information about the sample and feature of each compared method is listed as follows.MicroPred collected 691 real pre-miRNAs from the early version of miRNA database miRBase [27] (12.0 version, abbreviated as miRBase12.0) as the positive samples. The 660 pre-miRNAs have the stem-loop secondary structures and 31 pre-miRNAs have multiple stem-loops. These positive samples formed a positive dataset which is referred to as “MP_positive_set.” MicroPred obtained the 8494 pseudo hairpins as the negative samples and these hairpins were extracted from the human protein-coding regions. In addition, microPred collected 754 other types of noncoding RNAs, such as tRNA and rRNA, as the negative samples. The total 9248 negative samples formed a negative dataset named “MP_negative_set.” In terms of feature set, microPred obtained 21 features to train its classification model.PlantMiRNAPred was a classical method used to classify the plant pre-miRNAs. It collected the 2043 plant pre-miRNAs in the miRBase14.0 as the positive samples and extracted the 2122 pseudo hairpins from the protein-coding regions ofArabidopsis thaliana andGlycine max as the negative samples. The positive samples and negative samples of PlantMiRNAPred formed its positive dataset “PMP_positive_set” and negative one “PMP_negative_set.” PlantMiRNAPred extracted 68 features from plant pre-miRNAs.HuntMi obtained 1406 human pre-miRNAs from miRBase17.0 and extracted 81228 human pseudo hairpins to form the human positive dataset “HM_hsa_positive_set” and negative one “HM_hsa_negative_set.” It also obtained 7053 real animal pre-miRNAs and 218154 animal pseudo hairpins to construct the animal positive dataset “HM_animal_positive_set” and negative one “HM_animal_negative_set,” respectively. In addition, it obtained 2172 plant pre-miRNAs and 114929 plant pseudo hairpins to form the plant positive dataset “HM_plant_positive_set” and negative one “HM_plant_negative_set.” HuntMi used 28 features to train its model.MiRNApre obtained 1496 human pre-miRNAs from miRBase17.0 as the positive samples and extracted 1446 pseudo hairpins similar with the positive samples. These pseudo hairpins are used as the negative samples. These positive samples and negative samples formed the positive and negative datasets “MP_positive_set” and “MP_negative_set.” MiRNApre extracted 98 features from each positive and negative sample.Moreover, the new pre-miRNAs about human,Arabidopsis lyrata,Oryza sativa, andGlycine max were reported by miRBase when our work was almost completed. These pre-miRNAs were also used to test the classification methods, which can further validate their ability of discovering the new pre-miRNAs.In addition, to compare our method with other methods simultaneously, all the positive and negative samples about human, animal, and plant were merged, respectively. The 1496 human pre-miRNAs and 81982 pseudo hairpins formed the human-related positive and negative datasets which are referred to as “Merged_hsa_positive_set” and “Merged_hsa_negative_set.” The animal-related positive and negative datasets contain 7053 real animal pre-miRNAs and 218154 pseudo hairpins and they are named “Merged_animal_positive_set” and “Merged_animal_negative_set.” The plant-related datasets are “Merged_plant_positive_set” and “Merged_plant_negative_set” which are composed of 2172 plant real pre-miRNAs and 117051 pseudo hairpins. All the classification methods were tested by performing 5-fold cross validation on the three groups of datasets. The 355 new human pre-miRNAs (updated human dataset), 68Arabidopsis thaliana pre-miRNAs (updated ath dataset), 169Oryza sativa pre-miRNAs (updated osa dataset), and 302Glycine max pre-miRNAs (updated gma dataset) were also used for testing. In terms of features, the 139 features described in Section 2.1 are used to represent each positive sample and negative one.


## 3.2. Performance Evaluation Metrics

Suppose that TP and TN represent the number of the correctly classified positive samples (real pre-miRNAs) and that of the correctly classified negative samples (pseudo pre-miRNAs), respectively. FP and FN represent the numbers of the misclassified positive and negative samples, respectively. Sensitivity (SE) represents the proportion of the positive samples that are classified successfully accounting for the total positive samples. Specificity (SP) represents the proportion of the successfully classified negative samples accounting for the total negative samples. Consider(2)SE=TPTP+FN,SP=TNTN+FP.For the pre-miRNA classification, if a classifier has higher SE and lower SP, it has poorer ability to identify the pseudo pre-miRNAs. Thus, many pseudo pre-miRNAs will be misclassified as the real pre-miRNAs, which reduces the possibility that the biological experiments can successfully identify pre-miRNAs. On the contrary, the classifier has poorer ability to identify the real pre-miRNAs, and many real pre-miRNAs will be misclassified as the pseudo pre-miRNAs, which reduces the possibility that the real pre-miRNAs are discovered by the experimental study. Therefore, the classifier should have both high SE and high SP.The global classification accuracy based on machine learning is usually evaluated by the parameter Acc. However, the number of pseudo pre-miRNAs is usually much greater than that of real pre-miRNAs, which causes TN and FP to be much higher than TP and FN. Then we have(3)Acc=TP+TNTP+TN+FP+FN≈TNTN+FP=SP.Therefore, besides SE and SP, we also compute the geometric mean of SE and SP, denoted as Gm, to evaluate the global classification performance:(4)Gm=SE×SP.


## 3.3. Comparison with Other Methods and Classification Models

In order to compare MiRNAClassify with the state-of-the-art methods microPred [19], PlantMiRNAPred [16], HuntMi [17], and miRNApre [18], the 5-fold cross validation is performed. During the process of the cross validation, the positive and negative samples are randomly divided into 5 parts. 4 parts are used as the training samples, and the remaining part is used for testing. The testing dataset does not intersect with the training dataset. So it can objectively test the classification performances of the methods. Besides cross validation, the newly added human, animal, and plant pre-miRNAs are used to test the ability of discovering the new pre-miRNAs.MiRNAClassify is compared with other methods in two different ways. For one thing, in terms of each compared method, we choose its sample set and feature set to train MiRNAClassify. In this way, we compare MiRNAClassify with other methods, respectively. For another, MiRNAClassify is compared with all the other methods simultaneously by using the merged datasets and our feature set. In addition, MiRNAClassify is compared with other classification models, including SVM, naive Bayes, and Random Forest.


### 3.3.1. Comparison Using the Same Dataset and Feature Set

MicroPred focused on classification of the human pre-miRNAs, and it generated new simulated positive samples based on SMOTE. In order to compare with microPred in a fair way, we used the sample set and feature set of microPred to train the model of MiRNAClassify. As shown in Figure2 (details in Table 1), the first 3 columns are the results based on 5-fold cross validation, and the last column is the result for testing the newly added human pre-miRNAs. MicroPred obtained the lower sensitivity (SE=90.65%) and specificity (SP=92.16%) in the cross validation. The possible reason is that generating samples based on SMOTE introduced the noise data. MiRNAClassify is 5.31% better than microPred in overall accuracy Gm. SE increased by 5.75% and SP increased by 4.87%. For the dataset that composed of 1186 newly added human pre-miRNAs, MP_updated_set, the SEs of MiRNAClassify and microPred are 90.56% and 85.16%, respectively. These SEs are not as good as those in the cross validation. The main reason is that MicroPred used the pre-miRNAs in the early version of miRBase and the new pre-miRNAs for testing are even more than the pre-miRNAs used to train its classification model.Table 1
Datasets and detailed classification results of MiRNAClassify and microPred.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
MP_positive_set
Real
691
96.40
97.03
96.71
microPred
MP_negative_set
Pseudo
9248
90.65
92.16
91.40
MiRNAClassify
MP_updated_set
Real
1186
90.56
microPred
85.16Figure 2
Comparison of the performances of MiRNAClassify and microPred.PlantMiRNAPred mainly studied the classification of plant pre-miRNAs. The model of MiRNAClassify was constructed by using the same sample and feature sets with PlantMiRNAPred. As shown in Figure3 (details in Table 2), the first 3 columns are the comparison results of cross validation. MiRNAClassify achieved slightly better performance than PlantMiRNAPred. PlantMiRNAPred only extracted a small number of pseudo hairpins. The number of negative samples is close to that of positive samples, and there is no obvious class imbalance. It confirms that MiRNAClassify is still effective for this type of data. Actually, PlantMiRNAPred only selected partial positive and negative samples to train their model, which results in some of the sample information was lost. On the contrary, MiRNAClassify fully exploited the information from all the positive and negative samples by adjusting the sample weights and constructing multiple classification instances. In Figure 3, the last 3 columns are the results on 138 newly addedArabidopsis thaliana pre-miRNAs, 178Oryza sativa pre-miRNAs, and 488Glycine max pre-miRNAs. MiRNAClassify obtained consistently better classification accuracies.Table 2
Datasets and detailed classification results of MiRNAClassify and PlantMiRNAPred.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Plant
PMP_positive_set
Real
2043
96.57
98.35
97.46
PlantMiRNAPred
PMP_negative_set
Pseudo
2122
95.10
97.17
96.13
MiRNAClassify
Arabidopsis thaliana
PMP_ath_updated_set
Real
138
92.75
PlantMiRNAPred
90.58
MiRNAClassify
Oryza sativa
PMP_osa_updated_set
Real
178
77.53
PlantMiRNAPred
67.42
MiRNAClassify
Glycine max
PMP_gma_updated_set
Real
488
90.78
PlantMiRNAPred
86.07Figure 3
Comparison of the performances of MiRNAClassify and PlantMiRNAPred.HuntMi was constructed to distinguish the real and pseudo pre-miRNAs about human, animal, and plant. The model of MiRNAClassify was also trained by using its samples and features. As Figure4 shows (details in Table 3), MiRNAClassify and HuntMi performed the cross validation on three groups of datasets, respectively. It is obvious that the number of negative samples is far more than that of positive samples. There is a severe imbalance between the positive samples and negative samples. Although there are quite a lot of negative samples, HuntMi selected a small number of negative samples based on the ROC values to train its model. Apparently, it lost a large amount of information of negative samples. MiRNAClassify completely presented its advantage in processing the class imbalance and achieved superior performance. In addition, we performed further testing using the newly added human,Arabidopsis thaliana,Oryza sativa, andGlycine max pre-miRNAs. The result was demonstrated in Figure 5, which confirms that MiRNAClassify can discover more new pre-miRNAs. We found that the classification performances of these two methods on newly added plant pre-miRNAs were worse than the performances on the new human pre-miRNAs. One of the important reasons is that the sequences and structures of plant pre-miRNAs are more complicated than those of human pre-miRNAs.Table 3
Datasets and detailed classification results of MiRNAClassify and HuntMi.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
HM_hsa_positive_set
Real
1406
97.15
98.35
97.75
HuntMi
HM_hsa_negative_set
Pseudo
81228
95.02
96.94
95.98
MiRNAClassify
Animal
HM_animal_positive_set
Real
7053
95.74
97.55
96.64
HuntMi
HM_animal_negative_set
Pseudo
218154
94.11
95.95
95.03
MiRNAClassify
Plant
HM_plant_positive_set
Real
2172
93.32
97.82
95.54
HuntMi
HM_plant_negative_set
Pseudo
114929
91.71
95.87
93.77
MiRNAClassify
Homo sapiens
HM_hsa_updated_set
Real
445
93.93
HuntMi
92.14
MiRNAClassify
Arabidopsis thaliana
HM_ath_updated_set
Real
68
92.65
HuntMi
91.18
MiRNAClassify
Oryza sativa
HM_osa_updated_set
Real
169
75.74
HuntMi
69.82
MiRNAClassify
Glycine max
HM_gma_updated_set
Real
302
92.72
HuntMi
88.41Figure 4
Comparison of the performances of MiRNAClassify and HuntMi based on 5-fold cross validation.Figure 5
Comparison of the performances of MiRNAClassify and HuntMi on the newly added data.Although miRNApre was tested on human, animal, and plant, only the training and testing samples about human could be obtained. Therefore, the human-related dataset was used to train the model of MiRNAClassify. Since miRNApre only selected the negative samples which are similar to the positive ones, the negative dataset is only composed of 1446 samples. The positive dataset contains 1496 samples (Table4). There is no class imbalance because the number of positive samples nearly equals that of negative samples. In this case, the cross validation performance of MiRNAClassify is still better than miRNApre, as shown in the first 3 columns of Figure 6. In addition, we obtained 355 new human pre-miRNAs for further testing. The SEs of MiRNAClassify and miRNApre are 91.2% and 90.1%, respectively. The SEs are not as good as those obtained in the cross validation. The reason may be that the extracted negative samples were very similar to the positive samples which resulted in the low robustness of the classification models.Table 4
Datasets and detailed classification results of MiRNAClassify and miRNApre.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
MP_positive_set
Real
1496
98.33
98.27
98.29
miRNApre
MP_negative_set
Pseudo
1446
97.66
97.23
97.44
MiRNAClassify
MP_updated_set
Real
355
91.27
miRNApre
90.14Figure 6
Comparison of the performances of MiRNAClassify and miRNApre.


### 3.3.2. Comparison over the Merged Datasets

Besides comparison with each method, respectively, we tested MiRNAClassify and other methods simultaneously by using the merged datasets about human, animal, and plant and the newly updated datasets and extracting a set of same features. It is worth noting that the number of negative samples is much greater than that of positive samples in each merged dataset. The classification results are demonstrated in Table5.Table 5
The classification results of MiRNAClassify and other methods over the merged datasets and the updated datasets.
Method
Accuracy (%)
Merged human dataset
Merged animal dataset
Merged plant dataset
Updated human dataset
Updated ath dataset
Updated osa dataset
Updated gma dataset
MiRNAClassify
SE
97.93
95.85
93.37
SP
98.30
97.62
97.91
94.08
92.65
79.29
93.05
G
m
98.11
96.73
95.61
MicroPred
SE
92.25
91.61
89.50
SP
95.70
94.85
93.10
91.27
89.71
66.86
85.76
G
m
93.96
93.21
91.28
PlantMiRNAPred
SE
93.58
92.70
91.39
SP
91.20
88.60
87.10
92.11
89.71
70.42
88.41
G
m
92.38
90.63
89.22
HuntMi
SE
95.32
94.14
91.76
SP
97.11
96.07
95.94
92.68
91.18
72.78
89.07
G
m
96.21
95.10
93.83
miRNApre
SE
97.39
93.49
91.71
SP
90.90
89.80
88.10
90.14
89.71
71.01
86.09
G
m
94.09
91.63
89.89MiRNAClassify performed the best not only over the merged datasets but also over the updated datasets. HuntMiRNA also achieved decent prediction performance. MicroPred, PlantMiRNAPred, and miRNApre obtained the inferior results, especially over the negative samples. In terms of MicroPred, more simulated positive samples have to be generated because there are so many negative samples, which also introduced more noisy data. The possible reason for the worse performance of PlantMiRNAPred and miRNApre is that most of information about negative samples was abandoned and their discriminative ability on negative samples was reduced greatly.In addition, a pairedt-test was used to determine whether MiRNAClassify’s global performance (Gm) over the 3 groups of merged datasets and its accuracy (SE) over the 4 updated datasets is higher than other methods. The corresponding p values are listed in Table 6. The statistic results confirm that MiRNAClassify outperforms other methods significantly at the significance level 0.05.Table 6
The statistic results obtained by using pairedt-test over the prediction performance of MiRNAClassify and that of another method.
Different datasets
microPred
PlantMiRNAPred
HuntMi
miRNApre
p values on three groups of datasets
0.0019
4.9374
e
-
04
9.7070
e
-
04
0.0050
p values on four updated datasets
0.0339
0.0284
0.0354
0.0108


### 3.3.3. Comparison with Other Classification Models

We use 5-fold cross validation to compare MiRNAClassify with other well-known classification models including the standard SVM, naive Bayes, and Random Forest. These models under SMOTE method were further developed to compare with MiRNAClassify. All the models were trained by the merged datasets and the same set of features, and their classification results are demonstrated in Table7.Table 7
The classification results of MiRNAClassify and three classification models over the merged datasets.
Classification models
SE (%)
SP (%)
G
m (%)
Human
SVM
69.18
99.83
83.11
SVM + SMOTE
92.25
95.70
93.96
Naive Bayes
87.43
96.12
91.67
Naive Bayes + SMOTE
90.24
94.43
92.31
Random Forest
67.78
99.82
82.26
Random Forest + SMOTE
91.51
95.34
93.41
MiRNAClassify
97.93
98.30
98.11
Animal
SVM
69.03
98.14
82.31
SVM + SMOTE
91.61
94.85
93.21
Naive Bayes
85.04
95.03
89.90
Naive Bayes + SMOTE
90.83
92.61
91.71
Random Forest
69.52
98.72
82.84
Random Forest + SMOTE
91.12
95.01
93.05
MiRNAClassify
95.85
97.62
96.73
Plant
SVM
68.51
99.24
82.45
SVM + SMOTE
89.50
93.10
91.28
Naive Bayes
82.91
96.75
89.57
Naive Bayes + SMOTE
87.20
92.61
89.86
Random Forest
68.32
99.35
82.39
Random Forest + SMOTE
89.18
92.87
91.01
MiRNAClassify
93.37
97.91
95.61For the datasets with severe class imbalance, MiRNAClassify demonstrated its ability to process the imbalanced data and achieved the best performance. As expected, the standard classification models overlearned the information of majority class and obtained the small SEs and the great SPs. After the SMOTE method was applied to balance the positive and negative samples, these models except naive Bayes obtained decent improvement onGm values. The main reason is that naive Bayes has intrinsic resistance to the class imbalance. However, the classification models under SMOTE method, SVM, and Random Forest obtained nearly consistent performances which were slightly better than naive Bayes. It confirmed that SVM has the excellent generalization ability. In particular, each classification instance of MiRNAClassify was established based on SVM, which was one of the important factors that MiRNAClassify could perform well. After exerting t-test on the Gm values obtained by MiRNAClassify and other models, the statistic results in Table 8 indicated that MiRNAClassify achieved significantly better performance.Table 8
The statistic results obtained by using pairedt-test over the prediction performance of MiRNAClassify and that of another classification model.
p value
SVM
Naive Bayes
Random Forest
7.3053
e
-
04
6.2651
e
-
04
0.0015
p value
SVM + SMOTE
Naive Bayes + SMOTE
Random Forest + SMOTE
0.0019
0.0010
0.0028


## 3.3.1. Comparison Using the Same Dataset and Feature Set

MicroPred focused on classification of the human pre-miRNAs, and it generated new simulated positive samples based on SMOTE. In order to compare with microPred in a fair way, we used the sample set and feature set of microPred to train the model of MiRNAClassify. As shown in Figure2 (details in Table 1), the first 3 columns are the results based on 5-fold cross validation, and the last column is the result for testing the newly added human pre-miRNAs. MicroPred obtained the lower sensitivity (SE=90.65%) and specificity (SP=92.16%) in the cross validation. The possible reason is that generating samples based on SMOTE introduced the noise data. MiRNAClassify is 5.31% better than microPred in overall accuracy Gm. SE increased by 5.75% and SP increased by 4.87%. For the dataset that composed of 1186 newly added human pre-miRNAs, MP_updated_set, the SEs of MiRNAClassify and microPred are 90.56% and 85.16%, respectively. These SEs are not as good as those in the cross validation. The main reason is that MicroPred used the pre-miRNAs in the early version of miRBase and the new pre-miRNAs for testing are even more than the pre-miRNAs used to train its classification model.Table 1
Datasets and detailed classification results of MiRNAClassify and microPred.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
MP_positive_set
Real
691
96.40
97.03
96.71
microPred
MP_negative_set
Pseudo
9248
90.65
92.16
91.40
MiRNAClassify
MP_updated_set
Real
1186
90.56
microPred
85.16Figure 2
Comparison of the performances of MiRNAClassify and microPred.PlantMiRNAPred mainly studied the classification of plant pre-miRNAs. The model of MiRNAClassify was constructed by using the same sample and feature sets with PlantMiRNAPred. As shown in Figure3 (details in Table 2), the first 3 columns are the comparison results of cross validation. MiRNAClassify achieved slightly better performance than PlantMiRNAPred. PlantMiRNAPred only extracted a small number of pseudo hairpins. The number of negative samples is close to that of positive samples, and there is no obvious class imbalance. It confirms that MiRNAClassify is still effective for this type of data. Actually, PlantMiRNAPred only selected partial positive and negative samples to train their model, which results in some of the sample information was lost. On the contrary, MiRNAClassify fully exploited the information from all the positive and negative samples by adjusting the sample weights and constructing multiple classification instances. In Figure 3, the last 3 columns are the results on 138 newly addedArabidopsis thaliana pre-miRNAs, 178Oryza sativa pre-miRNAs, and 488Glycine max pre-miRNAs. MiRNAClassify obtained consistently better classification accuracies.Table 2
Datasets and detailed classification results of MiRNAClassify and PlantMiRNAPred.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Plant
PMP_positive_set
Real
2043
96.57
98.35
97.46
PlantMiRNAPred
PMP_negative_set
Pseudo
2122
95.10
97.17
96.13
MiRNAClassify
Arabidopsis thaliana
PMP_ath_updated_set
Real
138
92.75
PlantMiRNAPred
90.58
MiRNAClassify
Oryza sativa
PMP_osa_updated_set
Real
178
77.53
PlantMiRNAPred
67.42
MiRNAClassify
Glycine max
PMP_gma_updated_set
Real
488
90.78
PlantMiRNAPred
86.07Figure 3
Comparison of the performances of MiRNAClassify and PlantMiRNAPred.HuntMi was constructed to distinguish the real and pseudo pre-miRNAs about human, animal, and plant. The model of MiRNAClassify was also trained by using its samples and features. As Figure4 shows (details in Table 3), MiRNAClassify and HuntMi performed the cross validation on three groups of datasets, respectively. It is obvious that the number of negative samples is far more than that of positive samples. There is a severe imbalance between the positive samples and negative samples. Although there are quite a lot of negative samples, HuntMi selected a small number of negative samples based on the ROC values to train its model. Apparently, it lost a large amount of information of negative samples. MiRNAClassify completely presented its advantage in processing the class imbalance and achieved superior performance. In addition, we performed further testing using the newly added human,Arabidopsis thaliana,Oryza sativa, andGlycine max pre-miRNAs. The result was demonstrated in Figure 5, which confirms that MiRNAClassify can discover more new pre-miRNAs. We found that the classification performances of these two methods on newly added plant pre-miRNAs were worse than the performances on the new human pre-miRNAs. One of the important reasons is that the sequences and structures of plant pre-miRNAs are more complicated than those of human pre-miRNAs.Table 3
Datasets and detailed classification results of MiRNAClassify and HuntMi.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
HM_hsa_positive_set
Real
1406
97.15
98.35
97.75
HuntMi
HM_hsa_negative_set
Pseudo
81228
95.02
96.94
95.98
MiRNAClassify
Animal
HM_animal_positive_set
Real
7053
95.74
97.55
96.64
HuntMi
HM_animal_negative_set
Pseudo
218154
94.11
95.95
95.03
MiRNAClassify
Plant
HM_plant_positive_set
Real
2172
93.32
97.82
95.54
HuntMi
HM_plant_negative_set
Pseudo
114929
91.71
95.87
93.77
MiRNAClassify
Homo sapiens
HM_hsa_updated_set
Real
445
93.93
HuntMi
92.14
MiRNAClassify
Arabidopsis thaliana
HM_ath_updated_set
Real
68
92.65
HuntMi
91.18
MiRNAClassify
Oryza sativa
HM_osa_updated_set
Real
169
75.74
HuntMi
69.82
MiRNAClassify
Glycine max
HM_gma_updated_set
Real
302
92.72
HuntMi
88.41Figure 4
Comparison of the performances of MiRNAClassify and HuntMi based on 5-fold cross validation.Figure 5
Comparison of the performances of MiRNAClassify and HuntMi on the newly added data.Although miRNApre was tested on human, animal, and plant, only the training and testing samples about human could be obtained. Therefore, the human-related dataset was used to train the model of MiRNAClassify. Since miRNApre only selected the negative samples which are similar to the positive ones, the negative dataset is only composed of 1446 samples. The positive dataset contains 1496 samples (Table4). There is no class imbalance because the number of positive samples nearly equals that of negative samples. In this case, the cross validation performance of MiRNAClassify is still better than miRNApre, as shown in the first 3 columns of Figure 6. In addition, we obtained 355 new human pre-miRNAs for further testing. The SEs of MiRNAClassify and miRNApre are 91.2% and 90.1%, respectively. The SEs are not as good as those obtained in the cross validation. The reason may be that the extracted negative samples were very similar to the positive samples which resulted in the low robustness of the classification models.Table 4
Datasets and detailed classification results of MiRNAClassify and miRNApre.
Method
Species
Dataset
Type
Size
SE (%)
SP (%)
G
m (%)
MiRNAClassify
Homo sapiens
MP_positive_set
Real
1496
98.33
98.27
98.29
miRNApre
MP_negative_set
Pseudo
1446
97.66
97.23
97.44
MiRNAClassify
MP_updated_set
Real
355
91.27
miRNApre
90.14Figure 6
Comparison of the performances of MiRNAClassify and miRNApre.


## 3.3.2. Comparison over the Merged Datasets

Besides comparison with each method, respectively, we tested MiRNAClassify and other methods simultaneously by using the merged datasets about human, animal, and plant and the newly updated datasets and extracting a set of same features. It is worth noting that the number of negative samples is much greater than that of positive samples in each merged dataset. The classification results are demonstrated in Table5.Table 5
The classification results of MiRNAClassify and other methods over the merged datasets and the updated datasets.
Method
Accuracy (%)
Merged human dataset
Merged animal dataset
Merged plant dataset
Updated human dataset
Updated ath dataset
Updated osa dataset
Updated gma dataset
MiRNAClassify
SE
97.93
95.85
93.37
SP
98.30
97.62
97.91
94.08
92.65
79.29
93.05
G
m
98.11
96.73
95.61
MicroPred
SE
92.25
91.61
89.50
SP
95.70
94.85
93.10
91.27
89.71
66.86
85.76
G
m
93.96
93.21
91.28
PlantMiRNAPred
SE
93.58
92.70
91.39
SP
91.20
88.60
87.10
92.11
89.71
70.42
88.41
G
m
92.38
90.63
89.22
HuntMi
SE
95.32
94.14
91.76
SP
97.11
96.07
95.94
92.68
91.18
72.78
89.07
G
m
96.21
95.10
93.83
miRNApre
SE
97.39
93.49
91.71
SP
90.90
89.80
88.10
90.14
89.71
71.01
86.09
G
m
94.09
91.63
89.89MiRNAClassify performed the best not only over the merged datasets but also over the updated datasets. HuntMiRNA also achieved decent prediction performance. MicroPred, PlantMiRNAPred, and miRNApre obtained the inferior results, especially over the negative samples. In terms of MicroPred, more simulated positive samples have to be generated because there are so many negative samples, which also introduced more noisy data. The possible reason for the worse performance of PlantMiRNAPred and miRNApre is that most of information about negative samples was abandoned and their discriminative ability on negative samples was reduced greatly.In addition, a pairedt-test was used to determine whether MiRNAClassify’s global performance (Gm) over the 3 groups of merged datasets and its accuracy (SE) over the 4 updated datasets is higher than other methods. The corresponding p values are listed in Table 6. The statistic results confirm that MiRNAClassify outperforms other methods significantly at the significance level 0.05.Table 6
The statistic results obtained by using pairedt-test over the prediction performance of MiRNAClassify and that of another method.
Different datasets
microPred
PlantMiRNAPred
HuntMi
miRNApre
p values on three groups of datasets
0.0019
4.9374
e
-
04
9.7070
e
-
04
0.0050
p values on four updated datasets
0.0339
0.0284
0.0354
0.0108


## 3.3.3. Comparison with Other Classification Models

We use 5-fold cross validation to compare MiRNAClassify with other well-known classification models including the standard SVM, naive Bayes, and Random Forest. These models under SMOTE method were further developed to compare with MiRNAClassify. All the models were trained by the merged datasets and the same set of features, and their classification results are demonstrated in Table7.Table 7
The classification results of MiRNAClassify and three classification models over the merged datasets.
Classification models
SE (%)
SP (%)
G
m (%)
Human
SVM
69.18
99.83
83.11
SVM + SMOTE
92.25
95.70
93.96
Naive Bayes
87.43
96.12
91.67
Naive Bayes + SMOTE
90.24
94.43
92.31
Random Forest
67.78
99.82
82.26
Random Forest + SMOTE
91.51
95.34
93.41
MiRNAClassify
97.93
98.30
98.11
Animal
SVM
69.03
98.14
82.31
SVM + SMOTE
91.61
94.85
93.21
Naive Bayes
85.04
95.03
89.90
Naive Bayes + SMOTE
90.83
92.61
91.71
Random Forest
69.52
98.72
82.84
Random Forest + SMOTE
91.12
95.01
93.05
MiRNAClassify
95.85
97.62
96.73
Plant
SVM
68.51
99.24
82.45
SVM + SMOTE
89.50
93.10
91.28
Naive Bayes
82.91
96.75
89.57
Naive Bayes + SMOTE
87.20
92.61
89.86
Random Forest
68.32
99.35
82.39
Random Forest + SMOTE
89.18
92.87
91.01
MiRNAClassify
93.37
97.91
95.61For the datasets with severe class imbalance, MiRNAClassify demonstrated its ability to process the imbalanced data and achieved the best performance. As expected, the standard classification models overlearned the information of majority class and obtained the small SEs and the great SPs. After the SMOTE method was applied to balance the positive and negative samples, these models except naive Bayes obtained decent improvement onGm values. The main reason is that naive Bayes has intrinsic resistance to the class imbalance. However, the classification models under SMOTE method, SVM, and Random Forest obtained nearly consistent performances which were slightly better than naive Bayes. It confirmed that SVM has the excellent generalization ability. In particular, each classification instance of MiRNAClassify was established based on SVM, which was one of the important factors that MiRNAClassify could perform well. After exerting t-test on the Gm values obtained by MiRNAClassify and other models, the statistic results in Table 8 indicated that MiRNAClassify achieved significantly better performance.Table 8
The statistic results obtained by using pairedt-test over the prediction performance of MiRNAClassify and that of another classification model.
p value
SVM
Naive Bayes
Random Forest
7.3053
e
-
04
6.2651
e
-
04
0.0015
p value
SVM + SMOTE
Naive Bayes + SMOTE
Random Forest + SMOTE
0.0019
0.0010
0.0028


## 4. Conclusions

A new method based on cost-sensitive and ensemble learning (MiRNAClassify) was developed to classify the imbalanced real and pseudo pre-miRNAs. The multiple classification instances were constructed and integrated to classify a query sequence. Each instance was trained by the same number of positive and negative samples, which effectively relieved the negative effect of class imbalance. At the same time, the information of all the positive and negative samples was completely exploited. The weight of each sample embodied the possibility that it would be misclassified. Based on the sample weight, the new classification instance could focus on the samples that are easy to be misclassified. All of the above contribute to the more accurate classification performance.MiRNAClassify has been compared with the previous methods, microPred, PlantMiRNAPred, HuntMi, and miRNApre. Not only the human data but also the animal and plant data were used to test their performance. MiRNAClassify outperformed better than other methods during the cross validation and could discover more newly added pre-miRNAs. In addition, we compared MiRNAClassify with other methods and several well-known classification models by using the merged and imbalanced datasets. MiRNAClassify demonstrated its ability of processing the imbalance and achieved significantly better performance.



---

*Source: 960108-2015-11-10.xml*