What is the Logistic Regression algorithm and how does it work?
By: Positive Stud
Tags: Python Programming, Machine Learning, Algorithms, Programming, Logistic Regression

Building a model using Scikit-learn

After obtaining knowledge about Logistic Regression, let us now learn to develop a model for predicting heart disease using a Logistic regression classifier. Before we start developing a model, first of all, we require data. Hence, for now, we can obtain the data for predicting heart diseases from Kaggle.

Note: Please go to the link provided below in order to obtain data for predicting heart diseases.

Implementation of Pandas library for loading the dataset

First and foremost, we are required to load the dataset. Therefore, for loading the dataset, we will be using the pandas library.

#importing pandas library as pd

import pandas as pd

Then, we can read the CSV file using DataFrame which we have recently downloaded from Kaggle.

#using dataframe for reading the csv file

dataframe = pd.read_csv ('heart.csv')

Note: If you want to learn more about CSV files, please go to the link provided below.

Now, for verifying if the CSV file has been loaded appropriately, you can add the following line of code.

dataframe.head()

The output which is obtained after executing ‘dataframe.head()’

As shown in the above image, ‘dataframe.head()’ can be used to display all the columns with the first five rows present in the dataset

Defining the feature columns from the obtained dataset

Now, the columns of the dataset have to categorize into two variables i.e. Dependent variable (Target variable) and Independent variable (Feature variable). Therefore, in the given dataset which comprises details regarding the details of heart disease, the ‘target variable’ is given as ‘target’ in the column, and the rest of the columns can be provided as the ‘feature column’.

#defining the feature column in feature_columns from the obtained dataset

feature_columns =['age','sex','cp','trestbps','chol','restecg','thalach','exang','oldpeak','slope','ca','thal']

After defining the ‘feature_columns’, the feature variable and target variable can be defined as ‘x’ and ‘y’ respectively as shown below.

#defining feature variable as x

x = dataframe[feature_columns] #defining target variable as y

y = dataframe.target

Data should be splitted into training and test set

From the obtained dataset regarding heart disease, the obtained data should be splitted into two parts: a training set and a testing set. In our case scenario, we have categorized data in the ratio of 80:20. Hence, 80% of the data is going to be used for training purposes and 20% of data is going to be used for testing purposes.

#splitting feature and target variable (x and y) into training and testing sets

from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test=train_test_split(x,y,test_size=0.20,random_state=0)

Before we proceed towards the splitting of data, first of all, we are required to import a module called ‘train_test_split’ from a package called ‘sklearn.model_selection’. From this package, the ‘train_test_split’ function is implemented which requires three values (features, target, and test_size) from its parameter. Also, random_state can also be used to randomly select the records of the dataset.

Model Development and Prediction

After we have splitted the data as a training set and a testing set, the next step is to develop the model and carry out the prediction.

#from sklearn.linear_model, importing LogisticRegression module

from sklearn.linear_model import LogisticRegression #instantiating the Logistic Regression model

logistic_regression = LogisticRegression() #fitting Logistic Regression model with training dataset

logistic_regression.fit(X_train,y_train) #performing prediction on the testing set

y_prediction = logistic_regression.predict(X_test)

In the above code, first of all, ‘LogisticRegression’ model is imported from the ‘sklearn.linear_model’. Then, we are required to instantiate the Logistic Regression classifier object in ‘logistic_regression’ from the ‘LogisticRegression()’ function. We can then use the ‘fit()’ function for fitting the Logistic Regression model with the training dataset. Now, the final step is to carry out the prediction on the testing set of data using the ‘predict()’ function, and the outcome is stored in ‘y_prediction’.

Confusion Matrix and its purpose

After the development of the model, its performance should be determined in order to determine its applicability. Hence, for measuring the performance of the model, a confusion matrix can be used. Now, for determining the characteristics of the classification problem, let’s refer to the following table.

Now, let’s have a look at what the above table represents:

Positive (P) represents that the actual value is positive .

represents that the is . Negative (N) represents that the actual value is not positive i.e. Negative .

represents that the is i.e. . True Positive (TP) represents that the actual value is positive and is predicted to be positive .

represents that the is and is to be . False Negative (FN) represents that the actual value is positive but is predicted to be negative .

represents that the is but is to be . True Negative (TN) represents that the actual value is negative and is predicted to be negative .

represents that the is and is to be . False Positive (FP) represents that the actual value is negative but is predicted to be positive.

For determining the Classification Rate or Accuracy, you have the following relation.

Additionally, for determining Recall, you have the following relation.

Likewise, for calculating the Precision, you have the following relation.

Finally, for determining the F–measure, you have the following relation.

Using Confusion Matrix to evaluate the model

After obtaining knowledge regarding the confusion matrix, let us now evaluate the developed model using the confusion matrix as shown in the code provided below.

#metrics module is being imported from sklearn

from sklearn import metrics #determining the performance of model through confusion matrix

confusion_matrix_outcome = metrics.confusion_matrix(y_test, y_prediction)

confusion_matrix_outcome

In the above code, at first, the ‘metrics’ module is being imported from ‘sklearn’. Then, the ‘confusion_matrix()’ function, ‘y_test’, and ‘y_prediction’ are being passed in the order to obtain the outcome of the confusion matrix which is being stored in the variable named ‘confusion_matrix_outcome’.

Now, let’s have a look at the outcome of the above code after it is executed in the Jupyter Notebook.

The above image shows that we obtain a 2 * 2 matrix as output which is stored in variable ‘confusion_matrix_outcome’. As we know, in this case, we have only two classes i.e. 0 and 1. So, in the obtained output the diagonal element comprises accurate precision and the non-diagonal element comprises inaccurate predictions that had been made by the model. Therefore, the obtained matrix indicates that 22 and 30 are accurate predictions, and 5 and 4 are inaccurate predictions.

Visualization of Confusion Matrix through Heatmap

Now that we have obtained the confusion matrix, let’s create the heatmap of the confusion matrix.

# numpy, matplotlib.pyplot, seaborn are imported

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline

In the above code, the required modules i.e. numpy, matplotlib.pyplot, seaborn are imported.

# name of classes is being represented by names_detail

names_detail=[0,1]

fig, ax = plt.subplots()

plt.xticks(np.arange(len(names_detail)), names_detail)

plt.yticks(np.arange(len(names_detail)), names_detail)

# plotting the heatmap as per the confusion matrix (confusion_matrix_outcome)

sns.heatmap(pd.DataFrame(confusion_matrix_outcome), annot=True, cmap="Reds" ,fmt='g')

ax.xaxis.set_label_position("top")

plt.title('Confusion matrix for model evaluation', y=1.5)

plt.ylabel('Actual label')

plt.xlabel('Predicted label')

The above code is responsible for plotting the heatmap of the obtained confusion matrix.

The above image represents the heatmap which is plotted after the python code had been executed.

Note: If you are willing to have a look at the heatmap in detail then please refer to the link provided below.

https://matplotlib.org/3.1.1/gallery/images_contours_and_fields/image_annotated_heatmap.html

Classification rate, precision, and recall of the model

After the heatmap is plotted, now the classification rate, precision, and recall of the model are to be determined as presented in the code provided below.

#calculating accuracy, precision and recall of the model

print("Accuracy of the developed model is:",metrics.accuracy_score(y_test, y_prediction))

print("Precision of the developed model is:",metrics.precision_score(y_test, y_prediction))

print("Recall of the developed model is:",metrics.recall_score(y_test, y_prediction))

In the above code, two parameters that comprise the test dataset and prediction result which had been obtained from the training dataset are provided in the function, ‘accuracy_score()’, ‘precision_score()’, and ‘recall_score()’. Now, let’s view the image provided below in order to interpret the obtained result

The above image shows that the accuracy of the developed model is about 85% which is a good accuracy rate. Moving on, the positive class predictions which belong to the positive class itself is represented from the precision. Hence, the precision rate of the developed model is about to 86%. Likewise, recall indicates the positive class predictions which are developed from the positive circumstances of the dataset. In our model, the test dataset of the patients having heart disease is about 88%.

AUC — ROC Curve

Now, let’s measure the performance of the classification problem using the AUC — ROC curve. The measure or degree of separability is represented through AUC. Likewise, the probability curve is represented through ROC. Hence, a model with high AUC is good for predicting 0 as 0 and 1 as 1. The code which is responsible for obtaining the AUC — ROC curve has been provided below.

y_prediction_probability = logistic_regression.predict_proba(X_test)[::,1]

fpr, tpr, _ = metrics.roc_curve(y_test, y_prediction_probability)

auc = metrics.roc_auc_score(y_test, y_prediction_probability)

plt.plot(fpr,tpr,label="auc="+str(auc))

plt.legend(loc=4)

plt.show()

In our case, as shown in the above image, the model has a higher AUC which is about 0.90. Hence, the developed model can be considered better for distinguishing patients having heart disease and patients who do not have heart disease.