# A Novel Approach to Ensemble Classifiers: FsBoost-Based Subspace Method

Authors: Adeeb Noor; Muhammed Kürşad Uçar; Kemal Polat; Abdullah Assiri; Redhwan Nour
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

In this article, an algorithm is proposed for creating an ensemble classifier. The name of the algorithm is the F-score subspace method (FsBoost). According to this method, the features are selected with the F-score and classified with different or the same classifiers. In the next step, the ensemble classifier is created. Two versions that are named FsBoost.V1 and FsBoost.V2 have been developed based on classification by the same or different classifiers. According to the results obtained, the results are consistent with the literature. Besides, a higher accuracy rate is obtained compared with many algorithms in the literature. The algorithm is fast because it has a few steps. It is thought that the algorithm will be successful due to these advantages.


---

## Body

## 1. Introduction

An ensemble classifier is a method in which multiple classifications are used together to improve classification performance [1, 2]. For example, when three classifiers are used to classify an object, the classifier works like this: if the first classifier is classified as a cat, the second classifier is classified as a dog, and the third classifier is classified as a cat, the ensemble classifier generates the result by taking the average of these decisions. There are many ways to create an ensemble classifier. Some of the most commonly used ones are (1) adaptive resampling and combining (boosting) [3], (1.1) AdaBoost (adaptive boosting) [4], (2) bagging (bootstrap aggregating) [5], and (3) random subspace [6].The boosting method can create powerful classifiers by combining and training weak classifiers [3]. The most commonly used boosting method is AdaBoost [4]. The AdaBoost method tries to improve performance by focusing on misclassified instances [4]. In the bagging method, classifiers trained with different training sets randomly selected (random sampling method) from the dataset are combined [5]. Outputs of classifiers are combined with majority voting or weighted voting [5]. In the random subspace method, feature subsets are generated by randomly selecting from N samples [6]. Each subset has an M element and an m feature [6]. In other words, a subset of features is created, not a subset of instances [6]. In this way, the training process is accelerated. These subclasses and classifiers are trained to form ensemble classifiers. The outputs of the classifiers are combined with majority voting or weighted voting.These algorithms have a disadvantage. As the education levels for AdaBoost increase, the number of samples decreases, and training becomes more difficult. There is a need for more samples for training. It is quite slow because the training stages are too much [7, 8]. The bagging method involves complex calculations [7]. Both methods require many iterations [1]. So, the success rate is usually lower than the random forest method [1]. These models cannot explain the dataset by modeling it as decision trees [7]. When these disadvantages are taken into consideration, these methods are still in need of improvement. In this study, a new ensemble algorithm based on the F-score feature selection algorithm has been developed to reduce the processing load of existing ensemble algorithms and to increase the accuracy rate.Feature selection algorithms are often used in the machine learning field to improve the performance of systems [9–11]. In the field of machine learning, datasets are used in a variety of sizes and types [12–15]. Large size data will cause the classifier to lengthen the training duration. Feature selection algorithms have been developed to solve this problem [9, 16, 17]. They do this by clearing irrelevant data when holding relevant data [9]. Thus, data size, process load, and training time decrease while classification accuracy increases [9, 18]. Many feature selection algorithms have been developed in the literature [1, 16]. However, in this study, the F-score feature selection algorithm is used because it can work fast, and its performance is good [16]. Feature selection algorithms can be used in many places such as health areas [19–22].In this study, two different methods have been developed, namely, FsBoost.V1 and FsBoost.V2, based on the F-score feature selection algorithm that can enhance training performance for ensemble classifiers. The FsBoost.V1 method is like the random subspace method. However, the features are chosen concerning the data label, not random. Selected datasets are classified with a single classifier, and then ensemble classifier 1 is created. This process is repeated for three or more different classifiers. Eventually, ensemble classifiers for three different classifiers are merged. In this way, it is ensured that unnecessary data are removed from the training process. The operation can be interrupted first in the ensemble classifier. In the FsBoost.V2 method, all data are classified with different classifiers. In the second step, the subfeature space is created by the F-score feature selection algorithm and reclassified. The ensemble classifier is created because of classification. This process was repeated a second time. Eventually, ensemble classifiers for three different classifiers are merged. The use of a single classifier reduces the cost. Only relevant features are retrieved by using the F-score feature selection algorithm. This process accelerates the training process. Complexity is less than other algorithms.


## 2. Materials and Methods

The operation was performed according to the flow in Figure1. Firstly, the records to be used in the study were collected. Then, features were selected with the F-score feature selection algorithm. Finally, the data are classified with different classifiers, and their performances are calculated. When these operations are performed, ensemble classifiers are created, and their performances are calculated at different levels and formations.Figure 1
Process flow diagram.


### 2.1. Collection of Data

The data used in the study were downloaded from the Machine Learning Repository website of the University of California, Irvine (UCI) [23, 24]. The data consist of 4 groups (A/B/C/D) belonging to epilepsy patients (Table 1). Records include EEG records of individuals. Each record is 23.6 seconds. 2300 EEG recordings were taken during the epileptic seizure. The other 2300 records (nonepilepsy) were recorded while in a healthy condition. However, the records belong to epileptic patients. The epilepsy data in each set are the same. However, nonepilepsy records are different. The database contains 178 features for each EEG recording.Table 1
Dataset distributions.
DatasetGroupTotalNumber of featuresEpilepsyNonepilepsyA115011502300178B115011502300178C115011502300178D115011502300178


### 2.2. F-Score Feature Selection Algorithm

The F-score is one of the feature selection algorithms that helps distinguish classes from each other [25]. To select the feature, an F-score value (Fi) is calculated for each feature (equation (1)). The F-score threshold value (FE) is determined by taking the average of all F-score values. For the ith feature, if Fi>FE, ith feature is selected. This step is repeated for each feature.(1)Fi=x¯i+−x¯i2+x¯i−−x¯i21/n+−1∑k=1n+xk,i+−x¯i+2+1/n−1∑k=1n−xk,i−−x¯i−2.The variables in equation (1) are (1) feature vector xk,i, and k=1,2…,m. (2) n+, n−, respectively, are the positive (+) and negative (−) total number of elements in the class, and m=n++n−. (3) i is the feature number. (4) x¯i, x¯i−, and x¯i+ are the average value, mean value in the negative class, and mean value in the positive class of the ith property, respectively. (5) xk,i+ represents the kth positive example of the ith feature. xk,i− represents the kth negative example of the ith feature.In the study,A, B, C, and D dataset features were selected with the F-score (Table 2). Feature selection has been applied twice.Table 2
Selected features with the F-score for datasets.
DatasetABCDNumber of features1781781781781st feature selection606862542nd feature selection28252325


### 2.3. Ensemble Classifier

The ensemble classifier is a system created by combining different classifiers to produce safer and more stable estimates [26]. The system is built with N classifiers. N can be single or double. While classifying according to the feature vector, for each feature vector 1, each classifier generates an output value. The output values produced are counted. Then, the output of the ensemble classifier is determined by the number of votes. If the number of classifiers is even, the average of the decision values of the classifiers is rounded off, and the decision of the ensemble classifier is determined. This process applies to all feature vectors. The ensemble classifier was prepared in MATLAB using three different classifiers: kNN, PNN, and SVMs [27].The kNN is one of the machine learning classification methods with advisory learning [28, 29]. Under the structure of the training dataset, classification is done according to nearest k of the new classifier. In this study, k=5 was selected, and ten distance calculation formulas were used. These include Spearman, Seuclidean, Minkowski, Mahalanobis, Jaccard, Hamming, Euclidean, Cosine, Correlation, and Cityblock.PNN is a statistical classification algorithm based on kernel and Bayesian [30]. The method is developed based on feedforward networks [30]. The classifier takes care of all class elements when processing [31]. The radial-based kernel function calculates the distance between class samples. The user in the PNN classifier can manipulate the spread parameter. As the spread parameter approaches zero, the network begins to behave like the nearest neighbor classifier [32]. This value when farther away from zero, the classifier classifies, considering several vectors that separate data from each other [32]. In the study, PNN networks were designed with a total of 500 different values ranging from 0.01 to 5 steps of the spread parameter, with 0.01 step range. At the end of the study, the best performing network parameters and performance criteria were calculated.SVMs are among the best machine learning algorithms [33]. They can be used in the regression analysis as well as classification [33]. SVMs try to separate datasets from each other with a linear and nonlinear line. The purpose of the SVM algorithm is to be able to distinguish between the data with the minimum error [34]. Gaussian or radial basis function (RBF) kernel (rbf) was used in the study. The BoxConstraint box limit is set between 1 and 100 so that the best performance can be achieved.


### 2.4. Ensemble Classifier Powered by the F-Score

In this study, two different ensemble classifiers, namely, Classifier-FsBoost.V1 and Feature-FsBoost.V2, were developed.


#### 2.4.1. Classifier-Based Ensemble Classifier: FsBoost.V1

The implementation steps of this method are shown in detail in Figure2. Accordingly to this, firstly, a dataset (A) is classified in a classifier (kNN). In the second step, the first feature selection is performed and again classified in the same classifier (kNN). In the third step, the first and second feature selection are performed and again classified in the same classifier (kNN). Thus, it is classified in three different steps, but only in a classifier (kNN). These three results are combined to form the kNN ensemble. The same process is repeated in PNN and SVMs. Eventually, the kNN ensemble, the PNN ensemble, and the SVM ensemble are combined into a single ensemble classifier.Figure 2
FsBoost.V1—classifier-based ensemble classifier.


#### 2.4.2. Feature-Based Ensemble Classifier: FsBoost.V2

The steps for this method are shown in Figure3. Accordingly to this, firstly, a dataset (A) is classified by each classifier (kNN, PNN, and SVMs). These three classifiers are combined to obtain ensemble classifier 1. In the second step, the first feature selection is performed, and the process in the first step is repeated. In the third step, the first and second property selection steps are performed together, and then the first process is repeated. Ensemble 1, 2, and 3 classifiers are combined to create the ensemble classifier.Figure 3
FsBoost.V2—feature-based ensemble classifier.


### 2.5. Performance Evaluation Criteria and Distribution of Data for Classification

Different performance evaluation criteria were used to test the accuracy rates of the proposed systems. These are accuracy rates, sensitivity, specificity, kappa value, receiver operating characteristic (ROC), area under a ROC (AUC), andk (10-fold) cross-validation accuracy.While classifying the datasets, they were divided into two groups: training (50%) and test (50%) (Table3).Table 3
Training and testing data distribution.
ClassForA datasetForB datasetForC datasetForD datasetTraining (50%)Test (50%)TotalTraining (50%)Test (50%)TotalTraining (50%)Test (50%)TotalTraining (50%)Test (50%)TotalEpilepsy115011502300115011502300115011502300115011502300Nonepilepsy115011502300115011502300115011502300115011502300Total230023004600230023004600230023004600230023004600


## 2.1. Collection of Data

The data used in the study were downloaded from the Machine Learning Repository website of the University of California, Irvine (UCI) [23, 24]. The data consist of 4 groups (A/B/C/D) belonging to epilepsy patients (Table 1). Records include EEG records of individuals. Each record is 23.6 seconds. 2300 EEG recordings were taken during the epileptic seizure. The other 2300 records (nonepilepsy) were recorded while in a healthy condition. However, the records belong to epileptic patients. The epilepsy data in each set are the same. However, nonepilepsy records are different. The database contains 178 features for each EEG recording.Table 1
Dataset distributions.
DatasetGroupTotalNumber of featuresEpilepsyNonepilepsyA115011502300178B115011502300178C115011502300178D115011502300178


## 2.2. F-Score Feature Selection Algorithm

The F-score is one of the feature selection algorithms that helps distinguish classes from each other [25]. To select the feature, an F-score value (Fi) is calculated for each feature (equation (1)). The F-score threshold value (FE) is determined by taking the average of all F-score values. For the ith feature, if Fi>FE, ith feature is selected. This step is repeated for each feature.(1)Fi=x¯i+−x¯i2+x¯i−−x¯i21/n+−1∑k=1n+xk,i+−x¯i+2+1/n−1∑k=1n−xk,i−−x¯i−2.The variables in equation (1) are (1) feature vector xk,i, and k=1,2…,m. (2) n+, n−, respectively, are the positive (+) and negative (−) total number of elements in the class, and m=n++n−. (3) i is the feature number. (4) x¯i, x¯i−, and x¯i+ are the average value, mean value in the negative class, and mean value in the positive class of the ith property, respectively. (5) xk,i+ represents the kth positive example of the ith feature. xk,i− represents the kth negative example of the ith feature.In the study,A, B, C, and D dataset features were selected with the F-score (Table 2). Feature selection has been applied twice.Table 2
Selected features with the F-score for datasets.
DatasetABCDNumber of features1781781781781st feature selection606862542nd feature selection28252325


## 2.3. Ensemble Classifier

The ensemble classifier is a system created by combining different classifiers to produce safer and more stable estimates [26]. The system is built with N classifiers. N can be single or double. While classifying according to the feature vector, for each feature vector 1, each classifier generates an output value. The output values produced are counted. Then, the output of the ensemble classifier is determined by the number of votes. If the number of classifiers is even, the average of the decision values of the classifiers is rounded off, and the decision of the ensemble classifier is determined. This process applies to all feature vectors. The ensemble classifier was prepared in MATLAB using three different classifiers: kNN, PNN, and SVMs [27].The kNN is one of the machine learning classification methods with advisory learning [28, 29]. Under the structure of the training dataset, classification is done according to nearest k of the new classifier. In this study, k=5 was selected, and ten distance calculation formulas were used. These include Spearman, Seuclidean, Minkowski, Mahalanobis, Jaccard, Hamming, Euclidean, Cosine, Correlation, and Cityblock.PNN is a statistical classification algorithm based on kernel and Bayesian [30]. The method is developed based on feedforward networks [30]. The classifier takes care of all class elements when processing [31]. The radial-based kernel function calculates the distance between class samples. The user in the PNN classifier can manipulate the spread parameter. As the spread parameter approaches zero, the network begins to behave like the nearest neighbor classifier [32]. This value when farther away from zero, the classifier classifies, considering several vectors that separate data from each other [32]. In the study, PNN networks were designed with a total of 500 different values ranging from 0.01 to 5 steps of the spread parameter, with 0.01 step range. At the end of the study, the best performing network parameters and performance criteria were calculated.SVMs are among the best machine learning algorithms [33]. They can be used in the regression analysis as well as classification [33]. SVMs try to separate datasets from each other with a linear and nonlinear line. The purpose of the SVM algorithm is to be able to distinguish between the data with the minimum error [34]. Gaussian or radial basis function (RBF) kernel (rbf) was used in the study. The BoxConstraint box limit is set between 1 and 100 so that the best performance can be achieved.


## 2.4. Ensemble Classifier Powered by the F-Score

In this study, two different ensemble classifiers, namely, Classifier-FsBoost.V1 and Feature-FsBoost.V2, were developed.


### 2.4.1. Classifier-Based Ensemble Classifier: FsBoost.V1

The implementation steps of this method are shown in detail in Figure2. Accordingly to this, firstly, a dataset (A) is classified in a classifier (kNN). In the second step, the first feature selection is performed and again classified in the same classifier (kNN). In the third step, the first and second feature selection are performed and again classified in the same classifier (kNN). Thus, it is classified in three different steps, but only in a classifier (kNN). These three results are combined to form the kNN ensemble. The same process is repeated in PNN and SVMs. Eventually, the kNN ensemble, the PNN ensemble, and the SVM ensemble are combined into a single ensemble classifier.Figure 2
FsBoost.V1—classifier-based ensemble classifier.


### 2.4.2. Feature-Based Ensemble Classifier: FsBoost.V2

The steps for this method are shown in Figure3. Accordingly to this, firstly, a dataset (A) is classified by each classifier (kNN, PNN, and SVMs). These three classifiers are combined to obtain ensemble classifier 1. In the second step, the first feature selection is performed, and the process in the first step is repeated. In the third step, the first and second property selection steps are performed together, and then the first process is repeated. Ensemble 1, 2, and 3 classifiers are combined to create the ensemble classifier.Figure 3
FsBoost.V2—feature-based ensemble classifier.


## 2.4.1. Classifier-Based Ensemble Classifier: FsBoost.V1

The implementation steps of this method are shown in detail in Figure2. Accordingly to this, firstly, a dataset (A) is classified in a classifier (kNN). In the second step, the first feature selection is performed and again classified in the same classifier (kNN). In the third step, the first and second feature selection are performed and again classified in the same classifier (kNN). Thus, it is classified in three different steps, but only in a classifier (kNN). These three results are combined to form the kNN ensemble. The same process is repeated in PNN and SVMs. Eventually, the kNN ensemble, the PNN ensemble, and the SVM ensemble are combined into a single ensemble classifier.Figure 2
FsBoost.V1—classifier-based ensemble classifier.


## 2.4.2. Feature-Based Ensemble Classifier: FsBoost.V2

The steps for this method are shown in Figure3. Accordingly to this, firstly, a dataset (A) is classified by each classifier (kNN, PNN, and SVMs). These three classifiers are combined to obtain ensemble classifier 1. In the second step, the first feature selection is performed, and the process in the first step is repeated. In the third step, the first and second property selection steps are performed together, and then the first process is repeated. Ensemble 1, 2, and 3 classifiers are combined to create the ensemble classifier.Figure 3
FsBoost.V2—feature-based ensemble classifier.


## 2.5. Performance Evaluation Criteria and Distribution of Data for Classification

Different performance evaluation criteria were used to test the accuracy rates of the proposed systems. These are accuracy rates, sensitivity, specificity, kappa value, receiver operating characteristic (ROC), area under a ROC (AUC), andk (10-fold) cross-validation accuracy.While classifying the datasets, they were divided into two groups: training (50%) and test (50%) (Table3).Table 3
Training and testing data distribution.
ClassForA datasetForB datasetForC datasetForD datasetTraining (50%)Test (50%)TotalTraining (50%)Test (50%)TotalTraining (50%)Test (50%)TotalTraining (50%)Test (50%)TotalEpilepsy115011502300115011502300115011502300115011502300Nonepilepsy115011502300115011502300115011502300115011502300Total230023004600230023004600230023004600230023004600


## 3. Results

The work aims to develop a new algorithm to improve the ensemble classifier performance. We have developed an algorithm (FsBoost) that is similar to the random subspace method but with less workload, faster running, and better performance. F-score feature selection algorithm based on this method has two versions (FsBoost.V1 vs. FsBoost.V2). The ensemble classifier is created with a single classifier in FsBoost.V1 (Şekil 2, Level 1) and at least three different classifiers in FsBoost.V2 (Şekil 3, Level 1). The developed algorithms were tested with four two-class datasets (A, B, C, and D) (Table 3).According to the FsBoost algorithm, the dataset features were selected twice using the F-score feature selection algorithm. For example, according to FsBoost.V1, the dataset (A) is classified with the same classifier after each property selection (Figure 2, Level 1—kNN1, kNN2, and kNN3) (Table 4). kNN ensemble was formed by combining classifiers of three kNNs (Figure 2, Level 1) (Table 5). This process was repeated with three different classifiers to create PNN ensemble and SVM ensemble (Figure 2, Level 1) (Table 5). Then, the kNN ensemble, PNN ensemble, and SVM ensemble were combined to form the final ensemble classifier (Figure 2, Level 2) (Table 5). This process is repeated for each dataset (Tables 4–8).Table 4
Level 1 classification results for theA dataset.
k-nearest neighbor classification algorithmNPk = 2k = 2k = 2DFEuclideanSeuclideanEuclideanFS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.841.0092.170.870.8993.260.891.0094.57N-E1.000.841.001.001.000.89AUC0.920.870.94Kappa0.840.930.88F-measure0.9293.260.9310-fold (%)88.8591.2292.54Probabilistic neural networksNPSpreadSpreadSpread0.110.110.21FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.871.0093.260.940.7593.090.751.0087.70N-E1.000.870.921.001.000.75AUC0.930.860.94Kappa0.870.930.89F-measure0.9393.090.9410-fold (%)0.110.110.21Support vector machinesNPBoxConstraint3212FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.991.0099.650.990.9899.390.980.9998.83N-E1.000.991.000.990.990.98AUC1.000.990.99Kappa0.990.990.99F-measure1.0099.390.9910-fold (%)99.7699.5499.02DF: distance function, Sen: sensitivity, Spe: specificity, Acc: accuracy (%), NP: network parameters, FS: feature selection, NF: number of features, EC: ensemble classifier, E: epilepsy, and N-E: nonepilepsy.Table 5
Results for the FsBoost.V1 ensemble algorithm (Levels 1 and 2).
LevelLevel 1Level 2ClassifierkNN ensemblePNN ensembleSVM ensembleEnsembleClassSenSpeAccSenSpeAccSenSpeAccSenSpeAccFor A datasetE0.881.0093.870.891.0094.260.991.0099.480.931.0096.43N-E1.000.881.000.891.000.991.000.93AUC0.940.940.990.96Kappa0.880.890.990.93F-measure0.930.940.990.96For B datasetE0.851.0092.480.761.0087.830.960.9997.480.871.0093.52N-E1.000.851.000.760.990.961.000.87AUC0.920.880.970.94Kappa0.850.760.950.87F-measure0.920.860.970.93For C datasetE0.840.9991.910.800.9989.170.970.9897.430.880.9993.78N-E0.990.840.990.800.980.970.990.88AUC0.920.890.970.94Kappa0.840.780.950.88F-measure0.910.880.970.93For D datasetE0.830.9789.870.640.9680.130.950.9394.040.860.9691.30N-E0.970.830.960.640.930.950.960.86AUC0.900.800.940.91Kappa0.800.600.880.83F-measure0.890.770.940.91Sen: sensitivity, Spe: specificity, Acc: accuracy (%), E: epilepsy, and N-E: nonepilepsy.Table 6
Level 1 classification results for theC dataset.
k-nearest neighbor classification algorithmNPk = 2k = 2k = 2DFEuclideanEuclideanMinkowskiFS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.801.0090.000.830.8890.960.880.9692.04N-E1.000.800.990.960.960.88AUC0.900.820.92Kappa0.800.900.84F-measure0.8990.960.9110-fold (%)85.3987.5490.41Probabilistic neural networksNPSpreadSpreadSpread0.110.210.21FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.800.9889.040.790.7888.910.780.8882.96N-E0.980.800.990.880.880.78AUC0.890.780.89Kappa0.780.880.78F-measure0.8888.910.8810-fold (%)0.110.210.21Support vector machinesNPBoxConstraint4884FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.980.9998.130.970.9597.170.950.9695.57N-E0.990.980.970.960.960.95AUC0.980.940.97Kappa0.960.970.95F-measure0.9897.170.9710-fold (%)98.4897.3795.41DF: distance function, Sen: sensitivity, Spe: specificity, Acc: accuracy (%), NP: network parameters, FS: feature selection, NF: number of features, EC: ensemble classifier, E: epilepsy, and N-E: nonepilepsy.Table 7
Level 1 classification results for theD dataset.
k-nearest neighbor classification algorithmNPk = 2k = 1k = 5DFEuclideanMinkowskiEuclideanFS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.790.9788.040.820.8489.170.840.9288.13N-E0.970.790.970.920.920.84AUC0.880.780.90Kappa0.760.890.80F-measure0.8789.170.8910-fold (%)84.2486.3087.17Probabilistic neural networksNPSpreadSpreadSpread0.410.410.41FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.680.9782.430.640.5379.520.530.9574.09N-E0.970.680.950.950.950.53AUC0.820.590.80Kappa0.650.760.60F-measure0.8079.520.7710-fold (%)0.410.410.41Support vector machinesNPBoxConstraint1212FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.970.9394.610.930.9393.780.930.9292.30N-E0.930.970.940.920.920.93AUC0.950.880.94Kappa0.890.940.88F-measure0.9593.780.9410-fold (%)95.4394.5492.33DF: distance function, Sen: sensitivity, Spe: specificity, Acc: accuracy (%), NP: network parameters, FS: feature selection, NF: number of features, EC: ensemble classifier, E: epilepsy, and N-E: nonepilepsy.Table 8
Level 1 classification results for theB dataset.
k-nearest neighbor classification algorithmNPk = 2k = 2k = 4DFEuclideanEuclideanEuclideanFS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.841.0091.960.850.8392.300.830.9890.52N-E1.000.840.990.980.980.83AUC0.920.850.92Kappa0.840.920.85F-measure0.9192.300.9210-fold (%)88.1389.7290.50Probabilistic neural networksNPSpreadSpreadSpread0.110.210.31FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.840.9891.000.790.5889.170.581.0078.65N-E0.980.840.991.001.000.58AUC0.910.780.88Kappa0.820.880.76F-measure0.9089.170.8610-fold (%)0.110.210.31Support vector machinesNPBoxConstraint4153FS012NF682515ClassSenSpeAccSenSpeAccSenSpeAccE0.971.0098.480.960.9297.220.920.9794.48N-E1.000.970.990.970.970.92AUC0.980.940.97Kappa0.970.970.95F-measure0.9897.220.9710-fold (%)99.2297.3594.07DF: distance function, Sen: sensitivity, Spe: specificity, Acc: accuracy (%), NP: network parameters, FS: feature selection, NF: number of features, EC: ensemble classifier, E: epilepsy, and N-E: nonepilepsy.In FsBoost.V2, the dataset (A) is classified with different classifiers after each feature selection (Figure 3, Level 1—kNN1, PNN1, and SVM1) (Table 4). These three classifiers were combined to create ensemble 1 (Figure 3, Level 1—ensemble 1) (Table 9). Then, ensemble 1, ensemble 2, and ensemble 3 were combined to form the final ensemble classifier (Figure 3, Level 2) (Table 9). This process is repeated for each dataset (Tables 4–7 and 9). Finally, the FsBoost ensemble algorithm is also compared with the ensemble algorithms available in the literature (Table 10).Table 9
Results for the FsBoost.V2 ensemble algorithm (Levels 1 and 2).
LevelLevel 1Level 2ClassifierEnsemble 1Ensemble 2Ensemble 3EnsembleFor A datasetFS0120/1/2NF68251568/25/15ClassSenSpeAccSenSpeAccSenSpeAccSenSpeAccE0.921.0095.910.961.0098.000.911.0095.520.941.0097.22N-E1.000.921.000.961.000.911.000.94AUC0.960.980.960.97Kappa0.920.960.910.94F-measure0.960.980.950.97For B datasetFS0120/1/2NF68251568/25/15ClassSenSpeAccSenSpeAccSenSpeAccSenSpeAccE0.901.0094.830.890.9994.170.830.9991.090.881.0094.04N-E1.000.900.990.890.990.831.000.88AUC0.950.940.910.94Kappa0.900.880.820.88F-measure0.950.940.900.94For C datasetFS0120/1/2NF68251568/25/15ClassSenSpeAccSenSpeAccSenSpeAccSenSpeAccE0.871.0093.300.880.9993.390.900.9693.350.900.9994.52N-E1.000.870.990.880.960.900.990.90AUC0.930.930.930.95Kappa0.870.870.870.89F-measure0.930.930.930.94For D datasetFS0120/1/2NF68251568/25/15ClassSenSpeAccSenSpeAccSenSpeAccSenSpeAccE0.840.9790.390.850.9791.090.860.9489.570.870.9791.65N-E0.970.840.970.850.940.860.970.87AUC0.900.910.900.92Kappa0.810.820.790.83F-measure0.900.910.890.91Sen: sensitivity, Spe: specificity, Acc: accuracy (%), FS: feature selection, NF: number of features, EC: ensemble classifier, E: epilepsy, and N-E: nonepilepsy.Table 10
Comparing the FsBoost algorithm with single classifiers and boosting algorithms.
MethodsDatasetsABCDRankAccRankAccRankAccRankAccAdaBoostM1399.04297.22496.48394.35Bag299.43496.87296.83194.87GentleBoost498.96397.04396.78294.65LogitBoost598.96596.35596.35494.22LPBoost698.57696.35794.74792.83RobustBoost1295.741687.781786.351587.17RUSBoost1790.831784.481686.431684.39Subspace1594.201094.151392.021190.54TotalBoost798.35795.78695.87693.30FsBoost.V1Level 1—kNN ensemble1693.871392.481491.911389.87Level 1—PNN ensemble1494.261587.831589.171780.13Level 1—SVM ensemble199.48197.48197.43594.04Level 2—ensemble997.221194.04894.52891.65FsBoost.V2Level 1—ensemble 11195.91894.831293.301290.39Level 1—ensemble 2898.00994.171093.391091.09Level 1—ensemble 31395.521491.091193.351489.57Level 2—ensemble1096.431293.52993.78991.30kNN92.1791.9690.0088.04PNN93.2691.0089.0482.43SVMs99.6598.4898.1394.61Acc: accuracy (%).Accuracy rates for FsBoost.V1 and FsBoost.V2 are higher than those for single classifiers (Table10). The FsBoost algorithm is well ranked compared to other boosting algorithms in the literature (Table 10). FsBoost.V1—Level 1—SVM ensemble method is the best method when compared with the literature (Table 10, Rank).Three different datasets were used to reconfirm the results obtained. The distribution of datasets is shown in Table11.Table 11
Comparing the FsBoost algorithm with single classifiers and boosting algorithms for other datasets.
DatasetsNFTraining (50%)TotalTest (50%)TotalClass 1Class 2Class 1Class 2Basehock89497500997497499996Madelon9568062013006206801300PCMAC87491481972491480971NF: number of features.In order to compare the FsBoost algorithm with boosting algorithms, three different datasets were reanalyzed. The results obtained from the analysis are summarized in Table12. According to the results, the algorithm with the average best performance is the FsBoost.V1 Level 2 ensemble algorithm.Table 12
Comparing the FsBoost algorithm with single classifiers and boosting algorithms for other datasets.
MethodDatasetsMeanBasehockMadelonPCMACRankAccRankAccRankAccAccRankAdaBoostM1559.341752.92263.7558.675Bag1750.90855.001752.7352.8817GentleBoost1358.231553.54562.8258.209LogitBoost1258.431453.69363.5458.566LPBoost1656.22755.151652.8354.7416RobustBoost459.541353.77163.9559.092RUSBoost1556.33656.001256.0256.1214Subspace1158.551653.311554.0455.3015TotalBoost659.041254.38462.8258.754FsBoost.V1Level 1 kNN ensemble1058.73356.851454.2756.6212Level 1 PNN ensemble259.74256.921158.8158.498Level 1 SVM ensemble758.841154.541060.1457.8411Level 2 ensemble160.14456.62960.5659.101FsBoost.V2Level 1—ensemble 1958.841054.62662.2058.557Level 1—ensemble 2858.84954.69760.8758.1310Level 1—ensemble 31457.33157.381354.8956.5413Level 2 ensemble359.74556.54860.7659.013kNN59.0454.2354.58PNN57.2353.0058.39SVMs56.2254.6961.89Acc: accuracy (%).


## 4. Discussion and Conclusion

FsBoost is one of the best algorithms developed until now [4–7]. This method has very few steps. In this way, it provides results faster. A high accuracy rate is a distinct advantage. Algorithms with high accuracy and fast results are preferred in medical data classification. In this regard, FsBoost may be preferred.FsBoost contains fewer calculations and steps than the algorithms in the literature [4–7]. The accuracy rate is very good compared with other algorithms (Table 10) [4]. Considering these advantages, FsBoost may be a commonly used algorithm soon.FsBoost algorithms are also suitable for use in biomedical signal processing, deep learning, and communication [35–37].FsBoost can be used with three or more classifiers. Besides, FsBoost.V1 is a version of FsBoost that can be used with a single classifier. Achieving high performance with a single classifier is a distinct advantage of FsBoost.V1. The F-score feature selection algorithm creates this advantage. By combining different features, the same data can be interpreted differently. If the classifiers are strong, FsBoost increases in performance. Therefore, it is recommended that the algorithm is used with robust classifiers. Ensemble classifiers often bring out a strong classifier by combining weak classifiers. This is the weakness of FsBoost.As a result, we can say that FsBoost is an alternative method to create an ensemble classifier. A high-performance ensemble classifier can be created with a powerful classifier and the F-score feature selection algorithm.



---

*Source: 8571712-2020-07-15.xml*