# Novel Based Ensemble Machine Learning Classifiers for Detecting Breast Cancer

Authors: Taarun Srinivas; Aditya Krishna Karigiri Madhusudhan; Joshuva Arockia Dhanraj; Rajasekaran Chandra Sekaran; Neda Mostafaeipour; Negar Mostafaeipour; Ali Mostafaeipour
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Nowadays, for many industries, innovation revolves around two technological improvements, Artificial Intelligence (AI) and machine learning (ML). ML, a subset of AI, is the science of designing and applying algorithms that can learn and work on any activity from past experiences. Of all the innovations in the field of ML models, the most significant ones have turned out to be in medicine and healthcare, since it has assisted doctors in the treatment of different types of diseases. Among them, early detection of breast cancer using ML algorithms has piqued the interest of researchers in this area. Hence, in this work, 20 ML classifiers are discussed and implemented in Wisconsin’s Breast Cancer dataset to classify breast cancer as malignant or benign. Out of 20, 9 algorithms are coded using Python in Colab notebooks and the remaining are executed using the Waikato Environment for Knowledge Analysis (WEKA) software. Among all, the stochastic gradient descent algorithm was found to yield the highest accuracy of 98%. The algorithms that gave the best results have been considered in the development of a novel ensemble model and the same was implemented in both WEKA and Python. The performance of the ensemble model in both platforms is compared based on metrics like accuracy, precision, recall, and sensitivity and investigated in detail. From this experimental comparative study, it was found that the ensemble model developed using Python has yielded an accuracy of 98.5% and that developed in the WEKA has yielded 97% accuracy.


---

## Body

## 1. Introduction

Any environment can be made smart and intelligent with the help of technologies like AI and ML [1]. These technologies not only reduce the manpower and labor time but also improve the quality of life by automating society and creating more job opportunities [2, 3]. A system that can apply some logic, evaluate the different options available, and decide on its own can be considered to be an application of Artificial Intelligence. To provide this intelligence, the system has to be trained; i.e., the system has to learn. This is done through machine learning, a branch of AI, where the system makes a decision based on the data. In ML, the performance of a task improves based on previous data or experience. Thus, it can be understood that performance, task, and experience are the three main characteristics of ML. Figure 1 represents the basic flow of machine learning pictorially.Figure 1
The basic flow of machine learning.The applications of ML can be found in many areas such as object recognition, spam detection, speech recognition, and healthcare industries for survival predictions, tumor detection, and cancer detection. The use of AI and ML models assists doctors in the early detection and curing of cancer, tumors, and other deadly diseases. One such application in medicine is the detection of breast cancer using ML models. Breast cancer is the growth of unwanted cells in the breast and early detection of this cancer can lead to a better cure [4]. Many classification algorithms in machine learning can be used to detect breast cancer with high accuracy, but selecting a particular algorithm for breast cancer detection is a challenging task. There are many works done by researchers working in these areas and these kinds of works give rise to the establishment of smart hospitals and constructive exploitation of these technologies leads to the improvement in quality of life [5].Chandrasekaran et al. [6] developed a fully integrated analog-based machine learning classifier that uses Artificial Neural Network (ANN) algorithms for breast cancer prediction. The developed ANN model mainly focused on nine attributes of breast cancer, such as (1) bland chromatin, (2) mitosis, (3) uniformity of cell shape, (4) marginal adhesion, (5) single epithelial cell size, (6) uniformity of cell size, (7) clump thickness, (8) normal nucleoli, and (9) bare nuclei. The authors achieved a mean accuracy of 96.9% with optimal energy consumption. Later, a Deep Neural Network with Support Value (DNNS) was developed by Vaka et al. [7] for predicting breast cancer. The model was developed in three stages. In the first stage, preprocessing was carried out by removing noise from the images in the dataset. In the second stage, the geometrical and textual features were detected from the preprocessed image. In the third stage, each image was segmented using the histosigmoid based fuzzy classifier. The authors achieved an accuracy of 97.21%. Mojrian et al. [8] developed a multilayer fuzzy expert system by integrating an Extreme Learning Machine (ELM) classification model with Radial Basis Function (RBF). The authors compared the proposed methodology with the linear Support Vector Machine (SVM) model. Accuracy, Precision, True Positive (TP rate), False Positive (FP rate), True Negative (TN rate), and False Negative (FN rate) were considered as the performance metrics to evaluate the proposed ML model. The authors achieved an accuracy of 98% and the proposed methodology was effective in detecting breast cancer when compared with the linear SVM model. Chaurasia and Pal [9] predicted and compared six machine learning algorithms. Classification and Regression Trees (CART), Linear Regression, Multilayer Perceptron (MLP), Naive Bayes (NB), K Nearest Neighbors (KNN), and SVM were the basic learners considered for prediction. Along with basic learners, a few ensemble algorithms like AdaBoost M1, Random Forest (RF), and Support Vector Clustering (SVC) were integrated to yield high accuracies. The authors divided the training and test set in the ratio of 80 : 20, respectively. Accuracy was the metric considered by the authors and they achieved a good accuracy of 95.17%. Similar to [9], a comparative analysis using data visualization and ML applications for breast cancer detection and diagnosis was carried out by [10] and achieved an accuracy of 98% using the logistic regression model.Having studied the recent works of these authors, it is understood that individual algorithms like either deep neural networks or ensemble algorithms have been used at a time. To further improve the accuracy of existing methodologies, a novel combination of ensemble and optimization-based approaches is required to detect breast cancer with high accuracy. In this paper, 20 different ML classification algorithms have been selected for predicting breast cancer as benign or malignant using Wisconsin’s Breast Cancer dataset. Out of these 20, 9 were implemented in Google Colab notebooks using Python programming language and the remaining 11 were executed in WEKA data mining software. Along with the 20 algorithms, we propose a novel machine learning ensemble-based algorithm. The algorithms for ensemble were chosen based on the performance of the 20 algorithms. The ensemble model developed was made to run on both platforms and the results obtained were compared and investigated in detail. Thus, the work in this paper can assist doctors in accurately detecting and diagnosing breast cancer in the early stages. This is an attempt to improve and automate society, establish the concept of smart hospitals, and contribute to the development of these technologies.The rest of this paper is organized as follows. Section2 introduces the dataset and discusses the EDA performed to understand the importance and contribution of individual features. Section 3 describes the working of the selected 20 algorithms. Section 4 explains the evaluation parameters considered to investigate the performance of the algorithm and Section 5 discusses the results obtained and explains in detail the comparative study carried out in both platforms before concluding in Section 6.


## 2. Wisconsin’s Breast Cancer Dataset

The Breast Cancer dataset from the University of Wisconsin Hospital was considered for carrying out the performance analysis of 20 different classification algorithms. This is a small dataset consisting of data from 569 patients with 31 different input features and a single output feature. The 31 different features are independent of each other and the output feature called “diagnosis” is dependent on the input features. The output feature “diagnosis” has two labels: malignant and benign.


### 2.1. Exploratory Data Analysis (EDA)

The Breast Cancer dataset consists of data from 569 patients, out of which 359 patients were diagnosed as benign and 212 patients were diagnosed as malignant. The blue region from Figure2 represents the percentage share of patients who were diagnosed as malignant (37.3%) and the red region represents the percentage share of patients who were diagnosed as benign (62.7%).Figure 2
Benign and malignant cases.As the percentage shares are not equal, there is a class imbalance and the predictions may not be generalized. To overcome this class imbalance problem and to have generalized results, the number of data samples must be equal for both labels. Inferring from Figure3, if the mean radius is higher, then the patient is more likely to be diagnosed as malignant and if it is lower, then the patient is diagnosed as benign. So, the mean radius is directly proportional to the chances of a patient having breast cancer or not.Figure 3
Diagnosis vs. radius mean.Similarly, from Figure4 it is understood that people who were diagnosed as malignant have mean compactness in the range of 0.05 to 0.27 and those who have been diagnosed as benign have mean compactness in the range of 0 to 0.18. As the mean compactness increases, the chances of a person likely having cancer cells are more. When the cancer is detected as malignant, the mean area of spread of cancer is between 500 and 1800 mm and when the same has been detected as benign, the mean area of spread of cancer is between 0 and 700 mm and their respective sum is depicted in Figure 5. Early cancer detection helps to stop the spread of cancer when compared to malignant cases. Curing of malignant cases is very hard as the area of spread will be high.Figure 4
Diagnosis vs. compactness mean.Figure 5
Diagnosis vs. area mean.


## 2.1. Exploratory Data Analysis (EDA)

The Breast Cancer dataset consists of data from 569 patients, out of which 359 patients were diagnosed as benign and 212 patients were diagnosed as malignant. The blue region from Figure2 represents the percentage share of patients who were diagnosed as malignant (37.3%) and the red region represents the percentage share of patients who were diagnosed as benign (62.7%).Figure 2
Benign and malignant cases.As the percentage shares are not equal, there is a class imbalance and the predictions may not be generalized. To overcome this class imbalance problem and to have generalized results, the number of data samples must be equal for both labels. Inferring from Figure3, if the mean radius is higher, then the patient is more likely to be diagnosed as malignant and if it is lower, then the patient is diagnosed as benign. So, the mean radius is directly proportional to the chances of a patient having breast cancer or not.Figure 3
Diagnosis vs. radius mean.Similarly, from Figure4 it is understood that people who were diagnosed as malignant have mean compactness in the range of 0.05 to 0.27 and those who have been diagnosed as benign have mean compactness in the range of 0 to 0.18. As the mean compactness increases, the chances of a person likely having cancer cells are more. When the cancer is detected as malignant, the mean area of spread of cancer is between 500 and 1800 mm and when the same has been detected as benign, the mean area of spread of cancer is between 0 and 700 mm and their respective sum is depicted in Figure 5. Early cancer detection helps to stop the spread of cancer when compared to malignant cases. Curing of malignant cases is very hard as the area of spread will be high.Figure 4
Diagnosis vs. compactness mean.Figure 5
Diagnosis vs. area mean.


## 3. Brief Description of the Classification Algorithms

The chart given in Figure6 explains the types of ML. The algorithms listed in the chart below are the most common and popularly used classifiers. Instance-based algorithms such as KNN and statistical-based algorithms such as Naive Bayes, Bayes Net, Neural Network, Decision Trees, and more are experimentally verified in the next section.Figure 6
Types of machine learning and classification algorithms.


### 3.1. Logistic Regression

It is a widely used machine learning approach for categorization. The logistic regression algorithm draws a decision boundary between the different input features. The model then maps the inputs to the outputs by computing the cost function. The sigmoid function, also called a logistic function, is used for predicting the target labels. Equations (1)–(3) mathematically describe the sigmoid function [11].(1)hx=gθTx,(2)gθTx=11+e−θTx,(3)gz=11+e−z,where hx is defined as the model and gθTx is the sigmoid function. θT is the model parameters and x represents the input features. After the sigmoid function is computed, the cost function is calculated and the results from the cost function are later optimized using a gradient descent algorithm.


### 3.2.K Nearest Neighbors (KNN)

KNN is a widely used method for various real-time applications dealing with the classification of classes. The classification can be either binary in which only two output classes will be available or multiclass classification, in which more than two output classes will be available. The KNN algorithm is a lazy learning and highly computational algorithm [12]. This algorithm uses the principle of matching the characteristic label of an unknown data sample with the characteristic label of a neighboring data sample based on Euclidean distance or, in some cases, the Manhattan distance [13]. Euclidean distance is the square root of the distance between the two points in the sample space. The following equation (4) gives the Euclidean distance formula [14]:(4)dxi,xj=∑n=1mxin−xjn2,where dxi,xj is the distance between a known data sample and an unknown data sample and xi,xj are the location of the data samples. The “K” in KNN is the total number of neighbors that are selected for making predictions. It should be purely user-defined for fine prediction or else should be an odd square of the total number of data in the dataset chosen [15]. When a new, unknown data sample is introduced into the search space, the distance between the unknown data sample and the k nearest neighbors is determined. The least proximity value among the unfamiliar data sample to the nearest neighbor will determine the class for the unknown data sample.


### 3.3. Naive Bayes Classifier

It is a Bayes theorem-based conditional probabilistic machine learning model.


#### 3.3.1. Bayes Theorem

The Bayes theorem in probability and statistics defines the likelihood of every occurrence based on prior knowledge of circumstances that may be relevant to the event. The following equation expresses the Bayes theorem mathematically [16]:(5)PA|B=PB|APAPB,where A and B are events and PB≠0.Hence, utilizing this hypothesis, it is feasible to discover the likelihood of an incident, sayA, given that B has happened. In this case, B represents the evidence and A represents the hypothesis. The assumption here is that the existence of one trait has no effect on the other. As a result, it is said to be naive. Saritas and Yasar used the Naive Bayes model and ANN in Breast Cancer Coimbra Dataset and found that the classes were classified with an accuracy of 86.95% with ANN and 83.54% with Naive Bayes algorithms, thus proving that Naive Bayes algorithms could be used for early breast cancer detection [17].


#### 3.3.2. Types of Naive Bayes Classifier

(1) Gaussian Naive Bayes Classifier (GNB). When the predictors take on a continuous value in GNB, we assume that these values have a Gaussian or normal distribution. Consider training data with a continuous attribute “X.” After dividing the data by class, the mean and variance of “X” in each class are computed. The probability distribution of some observation value “Xi” given a class “Y” can be computed by using the following equation [16]:(6)Pxi|y=12πσy2e−xi−μy2/2σy2,where σy is the variance of values associated with class y and μy is the mean of values associated with the class y.Piryonesi and El-Diraby explored the performance of different classification algorithms and used it for the examination of asphalt pavement degradation data. According to their findings, GNB can significantly improve the classifier’s accuracy [18].(2) Bernoulli’s Naive Bayes Classifier (BNB). Features in the multivariate Bernoulli event model are independent Booleans (binary variables) that describe inputs. This model, like the multinomial model, is popular for document classification problems where binary term occurrence characteristics are utilized rather than term frequencies.(3) Complement Naive Bayes Classifier (CNB). The complement Naive Bayes classifier, also called CNB, works well with imbalanced datasets. The main principle behind the CNB is that the probability of a feature or data sample is calculated for all the classes to which the data sample belongs and not just for a single particular class. The main disadvantage of CNB is that the model tends to overfit easily when there is a large number of data samples in the dataset.


### 3.4. CART

#### 3.4.1. Decision Tree

The decision tree algorithm is used for many classification problems. The main advantage of the decision tree is that it finds hidden patterns from the data by pruning the dataset and delivers high accuracy [19]. Pruning is the process of removing unwanted nodes from the decision tree that does not provide meaningful insights into the classification process. The decision tree consists of three components, mainly the root, node, and leaves. Root and node classify the unknown data based on if-then criteria and deliver the output to the next layer of nodes. Leaves are the ends of a decision tree that provides the filtered result after pruning. There are different algorithms to implement a decision tree and one such simple way to implement the same is using an Iterative Dichotomiser 3 (ID3) algorithm. The ID3 algorithm uses information gain and entropy to determine the output labels. Entropy measures the impurity of data and information gain is used to determine the amount of information that the data contributes to the final outcome [20]. The information gain and the entropy are measured using equations (7) and (8) [21].(7)Hs=∑c∈C−Pclog2Pc,where Hs is the entropy, s is the input data, and c is the number of output classes.(8)IGA,S=HS−∑t∈TPtHt,where IGA,S is the information gain from the data S depending on output A, t is the subset after the data is split, Pt is the number of elements in t, and Ht is the entropy of the subset.


#### 3.4.2. Decision Stump

The decision stump algorithm is similar to decision trees. In this algorithm, there is only one decision tree which consists of one root and a maximum of two leaf nodes for output predictions [22]. Because of this reason, decision stumps are weak learners and do not subclassify like decision trees. The algorithm terminates as soon as the first classification is executed. The overall prediction can be determined using the following equation [23]:(9)Fx=∑i=1ngixi,where x is the input features, gi is the predicted value of the ith feature, and xi is the ith feature.


#### 3.4.3. Random Trees

Random tree classifiers are a combination of decision trees and random forests. It is also called Random Forest Tree (RFT). It belongs to the category of ensemble classifiers. Random Trees improve the performance of weak learners as well as the accuracy of the machine learning model. A random tree is defined in three steps as (i) defining the method of splitting the leaves in the tree; (ii) selecting the predictor type for every leaf in the tree; (iii) providing the randomness into the leaves. One major disadvantage of random trees is that the classifier often tends to overfit the model [24].


#### 3.4.4. Reptree

Reptree, also called representative tree algorithm, belongs to the family of decision trees and is one of the fast learner algorithms [25]. In this algorithm, multiple decision trees are created and the one with the highest probability is chosen as the representative, and based on this, other trees will be classified [26]. The reptree algorithm is a combination or ensemble of various individual trees or individual learners.


#### 3.4.5. Random Forests

Random forest is the combination of many decision trees by randomly selecting the features and bootstrapping [27]. The random forest will first select a limited number of features from the entire set of input features and split those random features into nodes. The nodes, in turn, are split into subnodes or daughter nodes and the process is carried out until the required nodes are satisfied. The same process is repeated for “n” number of times creating “D” decision trees. The pseudocode for random forest is given in Table 1.Table 1
Pseudocode for random forest.
(1)Start(2)Define the input features(3)Training the data(i) Select “i” random features from the entire set of input features i(ii) Determine the nodes from the “i” features using the best split point(4)Repeat(i) The above three steps until a specific number of nodes are reached(5)Repeat(i) The above steps forn times to create m decision trees(6)Testing the data(i) Feed-in a new data sample and predict the output label(ii) The node which gives high prediction among the other output nodes will be taken as the final prediction(7)Stop


### 3.5. Support Vector Machines (SVM)

SVMs provide highly accurate results with linear data as well as nonlinear data. In SVM, the input features are defined as input vectors, and multiple hyperplanes are drawn between these input features. The hyperplane acts as a decision boundary in SVM. The main objective of a hyperplane is to maximize the distance between two or more input features [28]. Because of this characteristic of the hyperplane, SVMs work well with nonlinear data. A hyperplane can be expressed using the following equation [29]:(10)WTx+b=0,where x is the input vector containing the input features, b is the bias, and W is the weight that defines the distance separating the hyperplane. The best hyperplane is decided by finding different hyperplanes which classify the labels in the best way; then it will choose the one which is farthest from the data points or the one which has a maximum margin as depicted in Figure 7.Figure 7
SVM hyperplane.


### 3.6. Stochastic Gradient Descent (SGD)

SGD is a sophisticated version of gradient descent (GD). The only difference between GD and SGD is that the gradient descent is calculated for only a limited number of data samples. SGD selects data samples in a randomized manner and computes the gradients. Once the gradient is computed, the weights and the corresponding output labels are updated for all the data samples that were considered. This is an iterative process and continues until a minimum cost is reached [30]. The following equation is used to calculate the gradient descent [31]:(11)∇fx=1n∑i=1n∇fix,where fx is the objective function, n is the total number of data samples, ∇fx is the gradient descent of objective function, and ∇fix is the gradient descent that is computed for limited data samples randomly. Once the gradients are computed for the selected data samples, the weights are updated using the following equation [32]:(12)x⟵x−α∇fix,where x is the data sample and α is the learning rate.


### 3.7. Sequential Minimal Optimization (SMO)

SMO is used for solving quadratic programming problems. It is used to train the SVM classifier using a Gaussian or polynomial kernel [32]. It converts the attributes or features into binary values and also replaces missing values in the input features. SMO breaks down a quadratic problem into linear complexity. Then this linear complexity is solved using Lagrange multipliers α1 and α2. The constraints for the selection of Lagrange multipliers are given in equations (13) and (14) [33]:(13)0≤α1,α2≤M,(14)y1α1+y2α2=K,where M is an SVM hyperparameter and K is the kernel function, both supplied by the user; and the variables α1,α2 are Lagrange multipliers. SMO can be used when there is a class imbalance between the input features and works well with binary as well as multiclass classification [34].


### 3.8.K Star

K star or K∗ is one of the lazy algorithms used for many complex classification problems. It is also called an instance-based classification algorithm [35]. Every data sample is an instance of this algorithm. The class of an instance of an unknown data sample depends on the entropic distance between that unknown data sample and the known data sample in the search space [36].


### 3.9. Bayes Net

A Bayesian Network, also called Bayes Net, is one of the classification algorithms used for solving machine learning problems. A Bayes Net is constructed using DAGs—Directed Acyclic Graphs. These DAGs consist of nodes and edges. The nodes represent a random variable, and the edges are used to establish the connection between two or multiple different nodes and carry the information regarding the probabilities between nodes. The only prerequisite of a DAG is that all the nodes must be connected in an open-loop and there must be no closed connection between the two nodes. The information sharing between the nodes depends on the Parental Markov Condition (PMC). PMC states that a variable represented by a node is independent of all variables in a nondescendant node and conditional on the parent node. The probabilities are calculated using the conditional probability formula as mentioned in the following equation [37]:(15)PA|B=PA∩BPB.


### 3.10. Classification via Regression

In classification via regression algorithm, first regression is carried out. Classification is performed from the output features obtained from the regression model and finally, the output class is predicted using logistic regression. The main disadvantage of this algorithm is that it provides fluctuating results. It is both a weak learner and a strong learner. The algorithm performs well when the number of input features is limited and if the input features are more, the accuracies may vary.


### 3.11. Gradient Boosting Classifier

Gradient boosting classifier is an ensemble of various weak learners which is used for both regressions and classification problems. A gradient boosting classifier performs gradient descent on a weak learner-like decision tree. This loss function can be defined as in the following equation[38]:(16)fx=px+q+e,where fx is the loss function, e is the error, x is the vector containing input features, and p,q are weights to the loss function. The gradient boosting classifier can be applied to more complex datasets and it provides highly accurate results for nonlinear data.


### 3.12. AdaBoost M1

AdaBoost, also called Adaptive Boosting, is an ensemble-based algorithm that can be used for both binary and multiclass classification. In this boosting algorithm, classification is carried out by defining a threshold or weightage value for every classifier. The final output class is determined based on weighted votes from the individual classifiers. If there are any weak classifiers, then extra weights are given to the samples of these classifiers. This addition of extra weights to weak classifiers converts the weak learner into a strong learner [39, 40]. The weights for the individual classifier are determined using the following equation [41]:(17)αi=12ln1−εiεi,where αi is the weight for sample i and εi is the weighted error for sample i. The final output prediction is determined using the following equation [42]:(18)Hs=∑i=1nαihis,where Hs is the final output prediction and his is the output prediction for each sample in a classifier.


### 3.13. LogitBoost

LogitBoost is also called an additive logistic regression algorithm [43]. LogitBoost is a weak classifier. LogitBoost takes different data samples while training and produces a weak output prediction. Boosting is applied to this weak model to improve performance and produce high output predictions. LogitBoost works well with both binary and multiclass classification. LogitBoost is similar to the AdaBoost algorithm. The only difference between the two algorithms is that the logit boost algorithm produces a weak classifier during the initial period of training the model. The output predictions are determined by using the following equation [44]:(19)Px=efxefx+e−fx,where Px are the probabilities of output class and fx is the fitness function


### 3.14. Bagging

Bagging is also called Bootstrap Aggregation. It is a powerful ensemble algorithm that works well with small datasets. Bagging is used to reduce the variance in a trained model and prevents overfitting of data while training [45]. In this algorithm, the dataset is split into clusters. Classification algorithms like decision trees and random forest are applied to this cluster, and predictions are computed. The predictions from the individual clusters are averaged and through a voting basis, the final output prediction is determined [46, 47]. The pseudocode for bagging is given in Table 2.Table 2
Pseudocode for bagging.
(1)Start(2)Define the input features and the output label(3)Split data into training and testing sets(4)Training the model(i) Perform bootstrap process on the training data(ii) Choose the number of clusters to be formed based on the data(iii) Apply a classification algorithm like a decision tree(5)Evaluate the model on the test set(i) Compute the accuracy from each cluster in the model and average the accuracy(ii) Determine the output prediction by a voting process(6)Stop


## 3.1. Logistic Regression

It is a widely used machine learning approach for categorization. The logistic regression algorithm draws a decision boundary between the different input features. The model then maps the inputs to the outputs by computing the cost function. The sigmoid function, also called a logistic function, is used for predicting the target labels. Equations (1)–(3) mathematically describe the sigmoid function [11].(1)hx=gθTx,(2)gθTx=11+e−θTx,(3)gz=11+e−z,where hx is defined as the model and gθTx is the sigmoid function. θT is the model parameters and x represents the input features. After the sigmoid function is computed, the cost function is calculated and the results from the cost function are later optimized using a gradient descent algorithm.


## 3.2.K Nearest Neighbors (KNN)

KNN is a widely used method for various real-time applications dealing with the classification of classes. The classification can be either binary in which only two output classes will be available or multiclass classification, in which more than two output classes will be available. The KNN algorithm is a lazy learning and highly computational algorithm [12]. This algorithm uses the principle of matching the characteristic label of an unknown data sample with the characteristic label of a neighboring data sample based on Euclidean distance or, in some cases, the Manhattan distance [13]. Euclidean distance is the square root of the distance between the two points in the sample space. The following equation (4) gives the Euclidean distance formula [14]:(4)dxi,xj=∑n=1mxin−xjn2,where dxi,xj is the distance between a known data sample and an unknown data sample and xi,xj are the location of the data samples. The “K” in KNN is the total number of neighbors that are selected for making predictions. It should be purely user-defined for fine prediction or else should be an odd square of the total number of data in the dataset chosen [15]. When a new, unknown data sample is introduced into the search space, the distance between the unknown data sample and the k nearest neighbors is determined. The least proximity value among the unfamiliar data sample to the nearest neighbor will determine the class for the unknown data sample.


## 3.3. Naive Bayes Classifier

It is a Bayes theorem-based conditional probabilistic machine learning model.


### 3.3.1. Bayes Theorem

The Bayes theorem in probability and statistics defines the likelihood of every occurrence based on prior knowledge of circumstances that may be relevant to the event. The following equation expresses the Bayes theorem mathematically [16]:(5)PA|B=PB|APAPB,where A and B are events and PB≠0.Hence, utilizing this hypothesis, it is feasible to discover the likelihood of an incident, sayA, given that B has happened. In this case, B represents the evidence and A represents the hypothesis. The assumption here is that the existence of one trait has no effect on the other. As a result, it is said to be naive. Saritas and Yasar used the Naive Bayes model and ANN in Breast Cancer Coimbra Dataset and found that the classes were classified with an accuracy of 86.95% with ANN and 83.54% with Naive Bayes algorithms, thus proving that Naive Bayes algorithms could be used for early breast cancer detection [17].


### 3.3.2. Types of Naive Bayes Classifier

(1) Gaussian Naive Bayes Classifier (GNB). When the predictors take on a continuous value in GNB, we assume that these values have a Gaussian or normal distribution. Consider training data with a continuous attribute “X.” After dividing the data by class, the mean and variance of “X” in each class are computed. The probability distribution of some observation value “Xi” given a class “Y” can be computed by using the following equation [16]:(6)Pxi|y=12πσy2e−xi−μy2/2σy2,where σy is the variance of values associated with class y and μy is the mean of values associated with the class y.Piryonesi and El-Diraby explored the performance of different classification algorithms and used it for the examination of asphalt pavement degradation data. According to their findings, GNB can significantly improve the classifier’s accuracy [18].(2) Bernoulli’s Naive Bayes Classifier (BNB). Features in the multivariate Bernoulli event model are independent Booleans (binary variables) that describe inputs. This model, like the multinomial model, is popular for document classification problems where binary term occurrence characteristics are utilized rather than term frequencies.(3) Complement Naive Bayes Classifier (CNB). The complement Naive Bayes classifier, also called CNB, works well with imbalanced datasets. The main principle behind the CNB is that the probability of a feature or data sample is calculated for all the classes to which the data sample belongs and not just for a single particular class. The main disadvantage of CNB is that the model tends to overfit easily when there is a large number of data samples in the dataset.


## 3.3.1. Bayes Theorem

The Bayes theorem in probability and statistics defines the likelihood of every occurrence based on prior knowledge of circumstances that may be relevant to the event. The following equation expresses the Bayes theorem mathematically [16]:(5)PA|B=PB|APAPB,where A and B are events and PB≠0.Hence, utilizing this hypothesis, it is feasible to discover the likelihood of an incident, sayA, given that B has happened. In this case, B represents the evidence and A represents the hypothesis. The assumption here is that the existence of one trait has no effect on the other. As a result, it is said to be naive. Saritas and Yasar used the Naive Bayes model and ANN in Breast Cancer Coimbra Dataset and found that the classes were classified with an accuracy of 86.95% with ANN and 83.54% with Naive Bayes algorithms, thus proving that Naive Bayes algorithms could be used for early breast cancer detection [17].


## 3.3.2. Types of Naive Bayes Classifier

(1) Gaussian Naive Bayes Classifier (GNB). When the predictors take on a continuous value in GNB, we assume that these values have a Gaussian or normal distribution. Consider training data with a continuous attribute “X.” After dividing the data by class, the mean and variance of “X” in each class are computed. The probability distribution of some observation value “Xi” given a class “Y” can be computed by using the following equation [16]:(6)Pxi|y=12πσy2e−xi−μy2/2σy2,where σy is the variance of values associated with class y and μy is the mean of values associated with the class y.Piryonesi and El-Diraby explored the performance of different classification algorithms and used it for the examination of asphalt pavement degradation data. According to their findings, GNB can significantly improve the classifier’s accuracy [18].(2) Bernoulli’s Naive Bayes Classifier (BNB). Features in the multivariate Bernoulli event model are independent Booleans (binary variables) that describe inputs. This model, like the multinomial model, is popular for document classification problems where binary term occurrence characteristics are utilized rather than term frequencies.(3) Complement Naive Bayes Classifier (CNB). The complement Naive Bayes classifier, also called CNB, works well with imbalanced datasets. The main principle behind the CNB is that the probability of a feature or data sample is calculated for all the classes to which the data sample belongs and not just for a single particular class. The main disadvantage of CNB is that the model tends to overfit easily when there is a large number of data samples in the dataset.


## 3.4. CART

### 3.4.1. Decision Tree

The decision tree algorithm is used for many classification problems. The main advantage of the decision tree is that it finds hidden patterns from the data by pruning the dataset and delivers high accuracy [19]. Pruning is the process of removing unwanted nodes from the decision tree that does not provide meaningful insights into the classification process. The decision tree consists of three components, mainly the root, node, and leaves. Root and node classify the unknown data based on if-then criteria and deliver the output to the next layer of nodes. Leaves are the ends of a decision tree that provides the filtered result after pruning. There are different algorithms to implement a decision tree and one such simple way to implement the same is using an Iterative Dichotomiser 3 (ID3) algorithm. The ID3 algorithm uses information gain and entropy to determine the output labels. Entropy measures the impurity of data and information gain is used to determine the amount of information that the data contributes to the final outcome [20]. The information gain and the entropy are measured using equations (7) and (8) [21].(7)Hs=∑c∈C−Pclog2Pc,where Hs is the entropy, s is the input data, and c is the number of output classes.(8)IGA,S=HS−∑t∈TPtHt,where IGA,S is the information gain from the data S depending on output A, t is the subset after the data is split, Pt is the number of elements in t, and Ht is the entropy of the subset.


### 3.4.2. Decision Stump

The decision stump algorithm is similar to decision trees. In this algorithm, there is only one decision tree which consists of one root and a maximum of two leaf nodes for output predictions [22]. Because of this reason, decision stumps are weak learners and do not subclassify like decision trees. The algorithm terminates as soon as the first classification is executed. The overall prediction can be determined using the following equation [23]:(9)Fx=∑i=1ngixi,where x is the input features, gi is the predicted value of the ith feature, and xi is the ith feature.


### 3.4.3. Random Trees

Random tree classifiers are a combination of decision trees and random forests. It is also called Random Forest Tree (RFT). It belongs to the category of ensemble classifiers. Random Trees improve the performance of weak learners as well as the accuracy of the machine learning model. A random tree is defined in three steps as (i) defining the method of splitting the leaves in the tree; (ii) selecting the predictor type for every leaf in the tree; (iii) providing the randomness into the leaves. One major disadvantage of random trees is that the classifier often tends to overfit the model [24].


### 3.4.4. Reptree

Reptree, also called representative tree algorithm, belongs to the family of decision trees and is one of the fast learner algorithms [25]. In this algorithm, multiple decision trees are created and the one with the highest probability is chosen as the representative, and based on this, other trees will be classified [26]. The reptree algorithm is a combination or ensemble of various individual trees or individual learners.


### 3.4.5. Random Forests

Random forest is the combination of many decision trees by randomly selecting the features and bootstrapping [27]. The random forest will first select a limited number of features from the entire set of input features and split those random features into nodes. The nodes, in turn, are split into subnodes or daughter nodes and the process is carried out until the required nodes are satisfied. The same process is repeated for “n” number of times creating “D” decision trees. The pseudocode for random forest is given in Table 1.Table 1
Pseudocode for random forest.
(1)Start(2)Define the input features(3)Training the data(i) Select “i” random features from the entire set of input features i(ii) Determine the nodes from the “i” features using the best split point(4)Repeat(i) The above three steps until a specific number of nodes are reached(5)Repeat(i) The above steps forn times to create m decision trees(6)Testing the data(i) Feed-in a new data sample and predict the output label(ii) The node which gives high prediction among the other output nodes will be taken as the final prediction(7)Stop


## 3.4.1. Decision Tree

The decision tree algorithm is used for many classification problems. The main advantage of the decision tree is that it finds hidden patterns from the data by pruning the dataset and delivers high accuracy [19]. Pruning is the process of removing unwanted nodes from the decision tree that does not provide meaningful insights into the classification process. The decision tree consists of three components, mainly the root, node, and leaves. Root and node classify the unknown data based on if-then criteria and deliver the output to the next layer of nodes. Leaves are the ends of a decision tree that provides the filtered result after pruning. There are different algorithms to implement a decision tree and one such simple way to implement the same is using an Iterative Dichotomiser 3 (ID3) algorithm. The ID3 algorithm uses information gain and entropy to determine the output labels. Entropy measures the impurity of data and information gain is used to determine the amount of information that the data contributes to the final outcome [20]. The information gain and the entropy are measured using equations (7) and (8) [21].(7)Hs=∑c∈C−Pclog2Pc,where Hs is the entropy, s is the input data, and c is the number of output classes.(8)IGA,S=HS−∑t∈TPtHt,where IGA,S is the information gain from the data S depending on output A, t is the subset after the data is split, Pt is the number of elements in t, and Ht is the entropy of the subset.


## 3.4.2. Decision Stump

The decision stump algorithm is similar to decision trees. In this algorithm, there is only one decision tree which consists of one root and a maximum of two leaf nodes for output predictions [22]. Because of this reason, decision stumps are weak learners and do not subclassify like decision trees. The algorithm terminates as soon as the first classification is executed. The overall prediction can be determined using the following equation [23]:(9)Fx=∑i=1ngixi,where x is the input features, gi is the predicted value of the ith feature, and xi is the ith feature.


## 3.4.3. Random Trees

Random tree classifiers are a combination of decision trees and random forests. It is also called Random Forest Tree (RFT). It belongs to the category of ensemble classifiers. Random Trees improve the performance of weak learners as well as the accuracy of the machine learning model. A random tree is defined in three steps as (i) defining the method of splitting the leaves in the tree; (ii) selecting the predictor type for every leaf in the tree; (iii) providing the randomness into the leaves. One major disadvantage of random trees is that the classifier often tends to overfit the model [24].


## 3.4.4. Reptree

Reptree, also called representative tree algorithm, belongs to the family of decision trees and is one of the fast learner algorithms [25]. In this algorithm, multiple decision trees are created and the one with the highest probability is chosen as the representative, and based on this, other trees will be classified [26]. The reptree algorithm is a combination or ensemble of various individual trees or individual learners.


## 3.4.5. Random Forests

Random forest is the combination of many decision trees by randomly selecting the features and bootstrapping [27]. The random forest will first select a limited number of features from the entire set of input features and split those random features into nodes. The nodes, in turn, are split into subnodes or daughter nodes and the process is carried out until the required nodes are satisfied. The same process is repeated for “n” number of times creating “D” decision trees. The pseudocode for random forest is given in Table 1.Table 1
Pseudocode for random forest.
(1)Start(2)Define the input features(3)Training the data(i) Select “i” random features from the entire set of input features i(ii) Determine the nodes from the “i” features using the best split point(4)Repeat(i) The above three steps until a specific number of nodes are reached(5)Repeat(i) The above steps forn times to create m decision trees(6)Testing the data(i) Feed-in a new data sample and predict the output label(ii) The node which gives high prediction among the other output nodes will be taken as the final prediction(7)Stop


## 3.5. Support Vector Machines (SVM)

SVMs provide highly accurate results with linear data as well as nonlinear data. In SVM, the input features are defined as input vectors, and multiple hyperplanes are drawn between these input features. The hyperplane acts as a decision boundary in SVM. The main objective of a hyperplane is to maximize the distance between two or more input features [28]. Because of this characteristic of the hyperplane, SVMs work well with nonlinear data. A hyperplane can be expressed using the following equation [29]:(10)WTx+b=0,where x is the input vector containing the input features, b is the bias, and W is the weight that defines the distance separating the hyperplane. The best hyperplane is decided by finding different hyperplanes which classify the labels in the best way; then it will choose the one which is farthest from the data points or the one which has a maximum margin as depicted in Figure 7.Figure 7
SVM hyperplane.


## 3.6. Stochastic Gradient Descent (SGD)

SGD is a sophisticated version of gradient descent (GD). The only difference between GD and SGD is that the gradient descent is calculated for only a limited number of data samples. SGD selects data samples in a randomized manner and computes the gradients. Once the gradient is computed, the weights and the corresponding output labels are updated for all the data samples that were considered. This is an iterative process and continues until a minimum cost is reached [30]. The following equation is used to calculate the gradient descent [31]:(11)∇fx=1n∑i=1n∇fix,where fx is the objective function, n is the total number of data samples, ∇fx is the gradient descent of objective function, and ∇fix is the gradient descent that is computed for limited data samples randomly. Once the gradients are computed for the selected data samples, the weights are updated using the following equation [32]:(12)x⟵x−α∇fix,where x is the data sample and α is the learning rate.


## 3.7. Sequential Minimal Optimization (SMO)

SMO is used for solving quadratic programming problems. It is used to train the SVM classifier using a Gaussian or polynomial kernel [32]. It converts the attributes or features into binary values and also replaces missing values in the input features. SMO breaks down a quadratic problem into linear complexity. Then this linear complexity is solved using Lagrange multipliers α1 and α2. The constraints for the selection of Lagrange multipliers are given in equations (13) and (14) [33]:(13)0≤α1,α2≤M,(14)y1α1+y2α2=K,where M is an SVM hyperparameter and K is the kernel function, both supplied by the user; and the variables α1,α2 are Lagrange multipliers. SMO can be used when there is a class imbalance between the input features and works well with binary as well as multiclass classification [34].


## 3.8.K Star

K star or K∗ is one of the lazy algorithms used for many complex classification problems. It is also called an instance-based classification algorithm [35]. Every data sample is an instance of this algorithm. The class of an instance of an unknown data sample depends on the entropic distance between that unknown data sample and the known data sample in the search space [36].


## 3.9. Bayes Net

A Bayesian Network, also called Bayes Net, is one of the classification algorithms used for solving machine learning problems. A Bayes Net is constructed using DAGs—Directed Acyclic Graphs. These DAGs consist of nodes and edges. The nodes represent a random variable, and the edges are used to establish the connection between two or multiple different nodes and carry the information regarding the probabilities between nodes. The only prerequisite of a DAG is that all the nodes must be connected in an open-loop and there must be no closed connection between the two nodes. The information sharing between the nodes depends on the Parental Markov Condition (PMC). PMC states that a variable represented by a node is independent of all variables in a nondescendant node and conditional on the parent node. The probabilities are calculated using the conditional probability formula as mentioned in the following equation [37]:(15)PA|B=PA∩BPB.


## 3.10. Classification via Regression

In classification via regression algorithm, first regression is carried out. Classification is performed from the output features obtained from the regression model and finally, the output class is predicted using logistic regression. The main disadvantage of this algorithm is that it provides fluctuating results. It is both a weak learner and a strong learner. The algorithm performs well when the number of input features is limited and if the input features are more, the accuracies may vary.


## 3.11. Gradient Boosting Classifier

Gradient boosting classifier is an ensemble of various weak learners which is used for both regressions and classification problems. A gradient boosting classifier performs gradient descent on a weak learner-like decision tree. This loss function can be defined as in the following equation[38]:(16)fx=px+q+e,where fx is the loss function, e is the error, x is the vector containing input features, and p,q are weights to the loss function. The gradient boosting classifier can be applied to more complex datasets and it provides highly accurate results for nonlinear data.


## 3.12. AdaBoost M1

AdaBoost, also called Adaptive Boosting, is an ensemble-based algorithm that can be used for both binary and multiclass classification. In this boosting algorithm, classification is carried out by defining a threshold or weightage value for every classifier. The final output class is determined based on weighted votes from the individual classifiers. If there are any weak classifiers, then extra weights are given to the samples of these classifiers. This addition of extra weights to weak classifiers converts the weak learner into a strong learner [39, 40]. The weights for the individual classifier are determined using the following equation [41]:(17)αi=12ln1−εiεi,where αi is the weight for sample i and εi is the weighted error for sample i. The final output prediction is determined using the following equation [42]:(18)Hs=∑i=1nαihis,where Hs is the final output prediction and his is the output prediction for each sample in a classifier.


## 3.13. LogitBoost

LogitBoost is also called an additive logistic regression algorithm [43]. LogitBoost is a weak classifier. LogitBoost takes different data samples while training and produces a weak output prediction. Boosting is applied to this weak model to improve performance and produce high output predictions. LogitBoost works well with both binary and multiclass classification. LogitBoost is similar to the AdaBoost algorithm. The only difference between the two algorithms is that the logit boost algorithm produces a weak classifier during the initial period of training the model. The output predictions are determined by using the following equation [44]:(19)Px=efxefx+e−fx,where Px are the probabilities of output class and fx is the fitness function


## 3.14. Bagging

Bagging is also called Bootstrap Aggregation. It is a powerful ensemble algorithm that works well with small datasets. Bagging is used to reduce the variance in a trained model and prevents overfitting of data while training [45]. In this algorithm, the dataset is split into clusters. Classification algorithms like decision trees and random forest are applied to this cluster, and predictions are computed. The predictions from the individual clusters are averaged and through a voting basis, the final output prediction is determined [46, 47]. The pseudocode for bagging is given in Table 2.Table 2
Pseudocode for bagging.
(1)Start(2)Define the input features and the output label(3)Split data into training and testing sets(4)Training the model(i) Perform bootstrap process on the training data(ii) Choose the number of clusters to be formed based on the data(iii) Apply a classification algorithm like a decision tree(5)Evaluate the model on the test set(i) Compute the accuracy from each cluster in the model and average the accuracy(ii) Determine the output prediction by a voting process(6)Stop


## 4. Evaluation Metrics

The performance of the above-discussed 20 classification algorithms is evaluated using the parameters discussed below. A confusion matrix with TP, TN, FP, and FN for actual and predicted data samples is formed as seen in Table3. A confusion matrix is a table that allows you to examine how an algorithm performs. Each column of the matrix contains examples from an actual class, whereas each row represents instances from a projected class [11].Table 3
Confusion matrix representation.
BenignMalignantBenignTPFPMalignantFNTNThe letter TP denotes that the actual value was positive and that the model projected a positive result. The letter TN denotes that the actual value was negative and that the model expected a negative result. In both of these examples, the projected value corresponds to the actual value. Whereas FP denotes a negative actual value but a positive prediction from the model, FN denotes a positive actual value but a negative prediction from the model. In both cases, the predicted value was falsely predicted. Based on these 4 values, accuracy, precision, recall, F1 measure, sensitivity, and specificity can be calculated.Accuracy gives the proportion of properly identified samples to total samples. It is obtained from the formula [48, 49],(20)Accuracy=TP+TNTP+TN+FP+FN.Precision tells how many of the correctly anticipated samples were positive. It is calculated by dividing all positive cases by all positive examples projected to be positive [48].(21)Precision=TPTP+FP.Similarly,recall indicates how many of the actual positive samples were properly predicted by the model. In medical circumstances, recall is critical since positive cases should not go undiscovered. In the case of this dataset, recall would be a preferable statistic because we do not want to mistake a malignant individual for a benign person. Precision and recall would decide whether or not the model is dependable [48].(22)Recall=TPTP+FN.TheF1 metric is a harmonic mean of recall and precision; hence it provides a comprehensive overview of these two measures. When we strive to improve our model’s precision, the recall decreases, and vice versa. The F1-score incorporates both trends into a single number [48].(23)F1−measure=2TP2TP+FP+FN.Sensitivity is the ability of a test to correctly classify a sample as malignant/positive. It is defined as the rate of the predicted positive case to the total positive cases [48].(24)Sensitivity=TPTP+FN.Specificity, on the other hand, is the ability of a test to correctly classify a sample as negative/benign. It gives the relationship of observed negative examples with all negative examples [48].(25)Specificity=TNTN+FP.In addition to the parameters mentioned above, three model evaluation metrics inR are utilized to examine the performance of implementing algorithms (i.e., Kappa value, Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE)). The Kappa value, often known as Cohen’s Kappa, is a measure of categorization accuracy. It is normalized to the dataset’s random chance baseline. It is a statistic that compares the observed and expected accuracy. The Kappa statistic is used to evaluate not just a single classifier but also subclassifiers.The MAE is the average of all absolute errors. It is also defined as the difference between the expected and actual values. References [50, 51] give the formula(26)MAE=∑i=1nyi−xin,wheren is the total number of samples, yi is the predicted value, and xi is the actual value.Root mean square error (RMSE). It is equal to the square root of the mean of the squares of all mistakes. RMSE is widely used and regarded as an excellent error metric for various sorts of predictions [51, 52].(27)RMSE=∑i=1nyi−xi2n.


## 5. Results and Discussions

In this study, 20 various classification algorithms were discussed in the previous section. To analyze its performance, all the algorithms were applied to Wisconsin’s Breast Cancer dataset to predict whether the cancer is benign or malignant. The analysis was justified based on various parameters such as precision, F1 measure, confusion matrix, recall, accuracy metrics, and run time discussed in Section4. As discussed in Section 2, the dataset contains 569 samples, each with 33 attributes. The dataset is preprocessed by removing the unnecessary attributes which do not contribute to the output and checking for any incomplete data. After that, the dataset is divided into 426 training samples and 143 testing samples. All algorithms are trained on training samples before being verified on test data. The evaluation parameters given in this section are obtained from the validation of test samples. The current method employs a data driven approach. Based on historical data, the model will be able to detect breast cancer. Also, this method is faster in breast cancer detection when compared to traditional diagnostic methods on basis of performance and time. The novelty of this method is attributed to the fact that the ensemble model developed for the current study consists of combination of boosting and optimization algorithms which will dynamically reduce the errors within few iterations.


### 5.1. Experimental Platform

In this study, an Intel Core i5 powered 2.30 GHz/8.00 GB system running under the Windows operating system is used for processing purposes. Out of the 20 discussed algorithms, 9 of them (Logistic regression; KNN; Naive Bayes–Complement, Bernoulli, Gaussian; Decision tree; Random forest; Gradient Booster; Support Vector Machine) are run on Google Colab notebooks and coded using Python language. The Scikit-learn package is used to train and test the samples. The remaining 11 algorithms (SGD, SMO, K star, Decision Stump, random tree, REPTree, BayesNet, classification via regression, AdaBoost M1, LogitBoost, and bagging) are run using WEKA software which is a workbench for machine learning.


### 5.2. Experimental Results

Train and test data are split in the ratio of 80 : 20. The TP, TN, FP, and FN values of the implemented algorithms are given in Table4. Based on these 4 values, accuracy, precision, recall, F1 measure, sensitivity, and specificity are calculated using equations (20)–(25) and tabulated in Table 5. Apart from the above-discussed parameters, three model evaluation metrics in “R” are also calculated using equations (26) and (27) and tabulated in Table 6. Also, the accuracy and run time of the algorithms are graphically depicted in Figures 8 and 9.Table 4
Confusion matrix (TP, TN, FP, FN) of the implemented classifier.
ClassifierTPTNFPFNKNN894716Logistic regression855152CNB894419BNB930053GNB884825Decision tree865043Decision stump804769Random tree805561REPTree775195Random forest885122SVM894716SGD855412SMO845422K star845026Bayes net815254Classification via regression825343Gradient booster885122AdaBoost M1825343LogitBoost845224Bagging835234Table 5
Performance of implemented classifiers.
ClassifierPrecisionRecallF1 measureAccuracySensitivitySpecificityRoc areaKNN0.970.880.930.950.930.980.94Logistic regression0.910.960.930.950.970.910.95CNB0.970.830.890.930.900.970.90BNB0.000.000.000.630.630.000.50GNB0.960.900.930.950.940.960.94Decision tree0.920.940.930.950.960.930.95Decision stump0.880.830.860.890.890.890.85Random tree0.900.980.940.950.980.900.96REPTree0.850.910.870.900.930.850.93Random forest0.960.960.960.970.980.960.97SVM0.980.890.930.950.930.970.93SGD0.980.960.970.980.970.980.97SMO0.960.960.960.970.970.960.97K star0.960.890.920.940.930.960.98Bayes net0.910.930.920.940.950.910.99Classification via regression0.930.940.930.950.960.930.98Gradient booster0.960.960.960.970.980.960.97AdaBoost M10.930.950.940.950.960.930.99LogitBoost0.960.920.950.960.950.960.99Bagging0.950.930.940.950.950.950.99Table 6
Kappa value, MAE, RMSE, and run time of implemented algorithms.
ClassifierKappa valueMAERMSEKNN0.8950.0480.221Logistic regression0.8920.0480.221CNB0.8490.0690.264BNB0.0060.3700.608GNB0.8970.0480.221Decision tree0.8910.0480.221Decision stump0.7730.1680.310Random tree0.8970.0490.222REPTree0.7990.1280.287Random forest0.9460.0270.167SVM0.8970.0500.221SGD0.9620.0210.145SMO0.9460.0280.167K star0.8850.0570.228Bayes net0.8780.0590.236Classification via regression0.8940.0850.207Gradient booster0.9420.0300.170AdaBoost M10.8960.0580.189LogitBoost0.9170.0540.174Bagging0.8950.0830.188Figure 8
Accuracy of implemented algorithms.Figure 9
Run time of implemented algorithms.


### 5.3. Discussions

The hyperparameters used to implement the algorithms are summarized in Table7. From Table 5 it is clearly understood that SGD yields a higher accuracy of 98%, followed by random forest, SMO, and gradient booster with 97%. BNB yields the least accuracy of 63%. This is due to the fact that BNB is used for discrete data and operates on the Bernoulli distribution. The key characteristic of BNB is that it only takes binary values such as true or false. The Breast Cancer dataset has a continuous range of values for all the features. This is why BNB is not able to produce higher accuracy. All other algorithms yield accuracy ranging from 90% to 96%. KNN was able to provide an accuracy of 95% with its K value set to 4. The K value was decided using a search method where, for each K value from 1 to 10, the error obtained was plotted and visualized in Figure 10.Table 7
Hyperparameters of the implemented 20 classifiers.
ClassifiersHyperparametersKNNNearest neighbors—4; brute algorithmLogistic regressionSolvers—Newton-cg, liblinear, saga; learning rate—0.001, 0.01, 0.1, 1; penalty—l2, max. iterations—2000CNB, BNB, GNBAlpha (smoothing parameter)—1.0 (default) var_smoothing—1.00e − 09Decision treeCriterion—“gini”; ccp_alpha (minimal cost-complexity pruning)—0.1; splitter—“best”Decision stumpBatch size—100Random treeK value—1; batch size—100; max. depth—2REPTreeMin variance prop—0.001; batch size—100; num folds—3Random forestn_estimators—2000; criterion—“gini”; bootstrap—true; n_jobs—−1; ccp_alpha—0.1SVMC (regularization parameter)—1.0; kernel—“sigmoid”SGDLoss—“log”; penalty—l2; alpha—0.01SMOCalibrator—“logistic”; kernel—“poly Kernel”;C—0.1; epsilon—1.00e − 12K starBatch size —100; global blend—20; entropic auto blend—trueBayes netBatch size—100; estimator—“simple”; search algorithm—“hill climber”Classification via regressionBatch size—100; regression type—“additive”Gradient boosting classifiern_estimators—2000; loss —“deviance”; learning rate—0.01AdaBoost M1Batch size—100; weight threshold—100; seed—1; iterations—0LogitBoostBatch size—100; weight threshold—100; seed—1; iterations—50;z max—5; shrinkage—1; pool size—1BaggingSeed—1; iterations—50; batch size—100; bag size percent—50Figure 10
Error rate vs.K value for KNN.SmallerK values can be noisy and have a greater effect on the outcome, but bigger K values have smoother decision boundaries, which means reduced variance but increased bias and is computationally intensive. Hence an optimal K value of 4 was selected. The type of solver and learning rate used for logistic regression was selected based on the grid search method. Three solvers “newton-cg,” “liblinear,” and “saga” were used and the learning rate was set in a range from 0.001 to 10. Logistic regression was able to yield an accuracy of 95% with a “Liblinear” solver and a learning rate of 10.Figure9 shows the run time of all implemented algorithms. The gradient booster algorithm was found to take more time than 7.74 seconds, followed by logistic regression with 4.3 seconds. This is because the gradient booster tends to overfit the model. Hence, setting a large number of boosting results increases the performance of the model. In this case, the boosting stages were set to 2000 which caused the algorithm to take more time. The same is in the case of a random forest. As discussed before, the grid search method was used in logistic regression to find the optimal solver and learning rate which resulted in more time. Decision tree was able to provide an accuracy of only 92%. The tree diagram for the dataset is given in Figure 11.Figure 11
Tree diagram before pruning.To increase the accuracy, pruning was executed. Pruning reduces the size of decision trees by removing nodes of the tree that do not contribute to model training. Pruning can also improve classification accuracy. The pruned decision was able to provide an accuracy of 95% and the same tree is shown in Figure12.Figure 12
Tree diagram after pruning.


### 5.4. Development of Novel Ensemble Model

The ultimate objective of every ML challenge is to find a model that can predict the desired outcome with high accuracy. Rather than creating a single model and hoping it is the best, we can use ensemble to take a variety of diverse models into consideration and average those models to get a single final model. In other words, the ensemble approach is a machine learning strategy that integrates numerous basic models to build a single highly accurate model. Common ensemble algorithms such as bagging, AdaBoost, LogitBoost, and gradient boosting are discussed above and implemented. In addition to that, a novel ensemble model has been developed combining logistic regression, stochastic gradient descent, and random forest. The hyperparameters of the developed ensemble model are given in Table8. The performance of this ensemble model is investigated by implementing it on both platforms (i.e., Colab notebook and WEKA software). The results obtained are compared and discussed further in the next section.Table 8
Hyperparameters of the ensemble model.
Classifiers in ensemble modelHyperparametersLogistic regressionSolvers—Newton-cg, liblinear, saga; learning rate—0.001, 0.01, 0.1, 1; penalty—l2, max. iterations—2000 (grid search)Random forestsn_estimators—2000; criterion—“gini”; bootstrap—true; min. sample split—2Stochastic gradient descent (SGD)Loss—“log”; penalty—l2; alpha—0.01


### 5.5. Comparative Study between WEKA and Python Coded Results

A comparative study has been performed between WEKA results and Python coded results. The ensemble model developed above has been used to perform this comparative study. Table9 summarizes the results obtained through WEKA and Python by ensemble of the three algorithms.Table 9
Comparison between WEKA and Python coded results.
Evaluation parametersWEKAPython codedAccuracy0.9700.985Precision0.9740.970Recall0.9740.990F1 score0.9740.980True positive rate (TPR)0.9700.986False positive rate (FPR)0.0300.015Run time (seconds)0.7804.940Kappa value0.9460.940Mean absolute error (MAE)0.0450.026Root mean square error (RMSE)0.1380.162From Table9, it is evident that the algorithm implemented in the Python programming language yielded an accuracy of 98.5%, while the model implemented in WEKA provided an accuracy of 97%. But the run time is more when executed in Python. This is because logistic regression uses a grid search technique to find the optimal solver and learning rate while the other 2 algorithms use a high value of boosting stages to yield better results which together accounts for the more computational time when compared to WEKA. Now, from Tables 5 and 9, it is evident that the ensemble model developed yields an accuracy of 98.5%, while the highest accuracy yielded by individual classifiers is 98% (SGD). Conventional ensembles like gradient booster, AdaBoost, LogitBoost, and bagging were able to achieve an accuracy of 97%, 95%, 96%, and 95% respectively. Hence, it is concluded that the performance of the developed ensemble model using Python is better than the traditional ML algorithms.


## 5.1. Experimental Platform

In this study, an Intel Core i5 powered 2.30 GHz/8.00 GB system running under the Windows operating system is used for processing purposes. Out of the 20 discussed algorithms, 9 of them (Logistic regression; KNN; Naive Bayes–Complement, Bernoulli, Gaussian; Decision tree; Random forest; Gradient Booster; Support Vector Machine) are run on Google Colab notebooks and coded using Python language. The Scikit-learn package is used to train and test the samples. The remaining 11 algorithms (SGD, SMO, K star, Decision Stump, random tree, REPTree, BayesNet, classification via regression, AdaBoost M1, LogitBoost, and bagging) are run using WEKA software which is a workbench for machine learning.


## 5.2. Experimental Results

Train and test data are split in the ratio of 80 : 20. The TP, TN, FP, and FN values of the implemented algorithms are given in Table4. Based on these 4 values, accuracy, precision, recall, F1 measure, sensitivity, and specificity are calculated using equations (20)–(25) and tabulated in Table 5. Apart from the above-discussed parameters, three model evaluation metrics in “R” are also calculated using equations (26) and (27) and tabulated in Table 6. Also, the accuracy and run time of the algorithms are graphically depicted in Figures 8 and 9.Table 4
Confusion matrix (TP, TN, FP, FN) of the implemented classifier.
ClassifierTPTNFPFNKNN894716Logistic regression855152CNB894419BNB930053GNB884825Decision tree865043Decision stump804769Random tree805561REPTree775195Random forest885122SVM894716SGD855412SMO845422K star845026Bayes net815254Classification via regression825343Gradient booster885122AdaBoost M1825343LogitBoost845224Bagging835234Table 5
Performance of implemented classifiers.
ClassifierPrecisionRecallF1 measureAccuracySensitivitySpecificityRoc areaKNN0.970.880.930.950.930.980.94Logistic regression0.910.960.930.950.970.910.95CNB0.970.830.890.930.900.970.90BNB0.000.000.000.630.630.000.50GNB0.960.900.930.950.940.960.94Decision tree0.920.940.930.950.960.930.95Decision stump0.880.830.860.890.890.890.85Random tree0.900.980.940.950.980.900.96REPTree0.850.910.870.900.930.850.93Random forest0.960.960.960.970.980.960.97SVM0.980.890.930.950.930.970.93SGD0.980.960.970.980.970.980.97SMO0.960.960.960.970.970.960.97K star0.960.890.920.940.930.960.98Bayes net0.910.930.920.940.950.910.99Classification via regression0.930.940.930.950.960.930.98Gradient booster0.960.960.960.970.980.960.97AdaBoost M10.930.950.940.950.960.930.99LogitBoost0.960.920.950.960.950.960.99Bagging0.950.930.940.950.950.950.99Table 6
Kappa value, MAE, RMSE, and run time of implemented algorithms.
ClassifierKappa valueMAERMSEKNN0.8950.0480.221Logistic regression0.8920.0480.221CNB0.8490.0690.264BNB0.0060.3700.608GNB0.8970.0480.221Decision tree0.8910.0480.221Decision stump0.7730.1680.310Random tree0.8970.0490.222REPTree0.7990.1280.287Random forest0.9460.0270.167SVM0.8970.0500.221SGD0.9620.0210.145SMO0.9460.0280.167K star0.8850.0570.228Bayes net0.8780.0590.236Classification via regression0.8940.0850.207Gradient booster0.9420.0300.170AdaBoost M10.8960.0580.189LogitBoost0.9170.0540.174Bagging0.8950.0830.188Figure 8
Accuracy of implemented algorithms.Figure 9
Run time of implemented algorithms.


## 5.3. Discussions

The hyperparameters used to implement the algorithms are summarized in Table7. From Table 5 it is clearly understood that SGD yields a higher accuracy of 98%, followed by random forest, SMO, and gradient booster with 97%. BNB yields the least accuracy of 63%. This is due to the fact that BNB is used for discrete data and operates on the Bernoulli distribution. The key characteristic of BNB is that it only takes binary values such as true or false. The Breast Cancer dataset has a continuous range of values for all the features. This is why BNB is not able to produce higher accuracy. All other algorithms yield accuracy ranging from 90% to 96%. KNN was able to provide an accuracy of 95% with its K value set to 4. The K value was decided using a search method where, for each K value from 1 to 10, the error obtained was plotted and visualized in Figure 10.Table 7
Hyperparameters of the implemented 20 classifiers.
ClassifiersHyperparametersKNNNearest neighbors—4; brute algorithmLogistic regressionSolvers—Newton-cg, liblinear, saga; learning rate—0.001, 0.01, 0.1, 1; penalty—l2, max. iterations—2000CNB, BNB, GNBAlpha (smoothing parameter)—1.0 (default) var_smoothing—1.00e − 09Decision treeCriterion—“gini”; ccp_alpha (minimal cost-complexity pruning)—0.1; splitter—“best”Decision stumpBatch size—100Random treeK value—1; batch size—100; max. depth—2REPTreeMin variance prop—0.001; batch size—100; num folds—3Random forestn_estimators—2000; criterion—“gini”; bootstrap—true; n_jobs—−1; ccp_alpha—0.1SVMC (regularization parameter)—1.0; kernel—“sigmoid”SGDLoss—“log”; penalty—l2; alpha—0.01SMOCalibrator—“logistic”; kernel—“poly Kernel”;C—0.1; epsilon—1.00e − 12K starBatch size —100; global blend—20; entropic auto blend—trueBayes netBatch size—100; estimator—“simple”; search algorithm—“hill climber”Classification via regressionBatch size—100; regression type—“additive”Gradient boosting classifiern_estimators—2000; loss —“deviance”; learning rate—0.01AdaBoost M1Batch size—100; weight threshold—100; seed—1; iterations—0LogitBoostBatch size—100; weight threshold—100; seed—1; iterations—50;z max—5; shrinkage—1; pool size—1BaggingSeed—1; iterations—50; batch size—100; bag size percent—50Figure 10
Error rate vs.K value for KNN.SmallerK values can be noisy and have a greater effect on the outcome, but bigger K values have smoother decision boundaries, which means reduced variance but increased bias and is computationally intensive. Hence an optimal K value of 4 was selected. The type of solver and learning rate used for logistic regression was selected based on the grid search method. Three solvers “newton-cg,” “liblinear,” and “saga” were used and the learning rate was set in a range from 0.001 to 10. Logistic regression was able to yield an accuracy of 95% with a “Liblinear” solver and a learning rate of 10.Figure9 shows the run time of all implemented algorithms. The gradient booster algorithm was found to take more time than 7.74 seconds, followed by logistic regression with 4.3 seconds. This is because the gradient booster tends to overfit the model. Hence, setting a large number of boosting results increases the performance of the model. In this case, the boosting stages were set to 2000 which caused the algorithm to take more time. The same is in the case of a random forest. As discussed before, the grid search method was used in logistic regression to find the optimal solver and learning rate which resulted in more time. Decision tree was able to provide an accuracy of only 92%. The tree diagram for the dataset is given in Figure 11.Figure 11
Tree diagram before pruning.To increase the accuracy, pruning was executed. Pruning reduces the size of decision trees by removing nodes of the tree that do not contribute to model training. Pruning can also improve classification accuracy. The pruned decision was able to provide an accuracy of 95% and the same tree is shown in Figure12.Figure 12
Tree diagram after pruning.


## 5.4. Development of Novel Ensemble Model

The ultimate objective of every ML challenge is to find a model that can predict the desired outcome with high accuracy. Rather than creating a single model and hoping it is the best, we can use ensemble to take a variety of diverse models into consideration and average those models to get a single final model. In other words, the ensemble approach is a machine learning strategy that integrates numerous basic models to build a single highly accurate model. Common ensemble algorithms such as bagging, AdaBoost, LogitBoost, and gradient boosting are discussed above and implemented. In addition to that, a novel ensemble model has been developed combining logistic regression, stochastic gradient descent, and random forest. The hyperparameters of the developed ensemble model are given in Table8. The performance of this ensemble model is investigated by implementing it on both platforms (i.e., Colab notebook and WEKA software). The results obtained are compared and discussed further in the next section.Table 8
Hyperparameters of the ensemble model.
Classifiers in ensemble modelHyperparametersLogistic regressionSolvers—Newton-cg, liblinear, saga; learning rate—0.001, 0.01, 0.1, 1; penalty—l2, max. iterations—2000 (grid search)Random forestsn_estimators—2000; criterion—“gini”; bootstrap—true; min. sample split—2Stochastic gradient descent (SGD)Loss—“log”; penalty—l2; alpha—0.01


## 5.5. Comparative Study between WEKA and Python Coded Results

A comparative study has been performed between WEKA results and Python coded results. The ensemble model developed above has been used to perform this comparative study. Table9 summarizes the results obtained through WEKA and Python by ensemble of the three algorithms.Table 9
Comparison between WEKA and Python coded results.
Evaluation parametersWEKAPython codedAccuracy0.9700.985Precision0.9740.970Recall0.9740.990F1 score0.9740.980True positive rate (TPR)0.9700.986False positive rate (FPR)0.0300.015Run time (seconds)0.7804.940Kappa value0.9460.940Mean absolute error (MAE)0.0450.026Root mean square error (RMSE)0.1380.162From Table9, it is evident that the algorithm implemented in the Python programming language yielded an accuracy of 98.5%, while the model implemented in WEKA provided an accuracy of 97%. But the run time is more when executed in Python. This is because logistic regression uses a grid search technique to find the optimal solver and learning rate while the other 2 algorithms use a high value of boosting stages to yield better results which together accounts for the more computational time when compared to WEKA. Now, from Tables 5 and 9, it is evident that the ensemble model developed yields an accuracy of 98.5%, while the highest accuracy yielded by individual classifiers is 98% (SGD). Conventional ensembles like gradient booster, AdaBoost, LogitBoost, and bagging were able to achieve an accuracy of 97%, 95%, 96%, and 95% respectively. Hence, it is concluded that the performance of the developed ensemble model using Python is better than the traditional ML algorithms.


## 6. Conclusion

Breast cancer is a very common type of cancer and it is predominantly found in women. In this paper, 20 different ML classification algorithms were trained on a Breast Cancer dataset from the University of Wisconsin to detect the same. From the results obtained it was found that SGD has yielded the highest accuracy of 98%, while BNB provided the least accuracy of 63%. Apart from accuracy, various other evaluation parameters like confusion matrix, precision, recall, F1 measure, sensitivity, specificity, Kappa value, MAE, RMSE, and run time were calculated and tabulated. Further, a comparative study was carried out to analyze the performance of the algorithm when coded in Python and when run on WEKA. An ensemble model has been developed combining logistic regression, random forest, and SGD algorithms and deployed in the comparative study. The performance of the ensemble model in both platforms has been compared and investigated in detail. It was found that the ensemble model developed using Python was able to achieve an accuracy of 98.5% and the model developed in the WEKA data mining tool yielded an accuracy of 97%. We believe that development and implementation of ensemble models can improve the quality of diagnosis and brings a great advantage compared to the recent works. Future directions of this work include the development of a real-time AI-based hardware prototype that can be deployed in hospitals around the world to assist doctors in detecting breast cancer. The use of AI and ML technology is limited not only to the detection of breast cancer, but also to the detection of other deadly diseases. The same technology can automate society and one such example is described in this paper. Thus, the work in this paper is an attempt to improve and automate society, establish the concept of smart hospitals, and contribute to the development of these technologies.



---

*Source: 9619102-2022-05-10.xml*