Logistic Regression in Classification model using Python: Machine Learning
By: Kaushik Katari
Tags: Python, Sensitivity, Logistic Regression, Machine Learning, Classification

There are still some variables with high correlation, but we can remove them while building the model. Let’s do that.

Running Our First Training Model

We’ll use the stasmodel library to build our first model. Let’s see how our model looks like if we consider all the variables.

The statistics of our first model is,

Image by Author — Statistics of our first model

The above model is not a good one because some variables have high p-values, which means the variables are insignificant. Instead of removing variables one at a time and building a model, again and again, we’ll use a method called RFE to select the top 15 variables for the model and remove the less significant variables based on p-values and VIF values.

Feature Selection Using RFE

We’ll select the top 15 variables, which helps in building the model in our logistic regression.

The top 15 variables are,

Image by Author — Top 15 variables using RFE

The variables having showing True are the ones we are interested in, and if we want to add more than 15 variables, we can add them one by one based on their respective ranks.

Build the model using the above 15 variables

Let’s build our second model using the 15 variables we got from RFE.

The model statistics will be,

Image by Author — Logistic Regression model using RFE Variables

Here we are using the GLM (Generalized Linear Models) method from the statsmodels.api library. Binomial in the family argument tells the statsmodels that it needs to fit a logit curve to binomial data (i.e., the target variable will have only two values, in this case, ‘Churn’ and ‘Non-Churn’).

A sample logit curve looks like this,

By Stat math — Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=92801334

Predicting the output probabilities

The logistic regression curve gives us the probabilities of churning and not churning. We can get these probabilities by simply using the ‘predict’ function.

The predicted probabilities are

Image by Author — Predicted probabilities

Now, let’s create a dataframe with the actual churn column and the predicted probabilities.

The dataframe for the above program will be as follows.

Image by Author — Dataframe of actual churn and it’s probability

Since the logistic curve gives us the probabilities and not the actual classification of ‘Churn’ and ‘Non-Churn,’ we need to find a threshold probability to classify customers as ‘churn’ and ‘non-churn.’

Here, let’s choose 0.5 as an arbitrary cutoff wherein if the probability of a particular customer churning is less than 0.5, we’d classify it as ‘Non-Churn,’ and if it’s greater than 0.5, we will classify it as ‘Churn.’

Creating a new column

The choice of 0.5 is entirely arbitrary at this stage, and we’ll learn how to find the optimal cutoff in ‘Model Evaluation.’

The dataframe looks like this after choosing an arbitrary cutoff.

Image by Author — Dataframe after arbitrary cutoff

Here, we used a cutoff of 0.5 to classify the customers as ‘Churn’ and ‘Non-churn.’ Now, since these are the probabilities, there are bound to be errors.

We’ll encounter two types of errors:

‘Churn ’ customers being incorrectly classified as ‘ Non-Churn .’

’ customers being incorrectly classified as ‘ .’ ‘Non-Churn’ customers being incorrectly classified as ‘Churn.’

Confusion Matrix

To find these errors and the model’s wellness, we’ll use a phenomenon called ‘Confusion Matrix.’

A typical confusion matrix looks like the following.

Image by Author — Confusion matrix

Note: The values shown in the above matrix is just a sample

If we observe the above matrix:

The first row and first column (1250) is the number of customers who have actually ‘ not churned ’; the model has also predicted them as ‘ not churned .’ This column is known as True Negative(TN).

’; the model has also predicted them as ‘ .’ This column is known as The first row and second column (150) is the number of customers who have actually ‘ not churned ,’ but the model has predicted them as ‘ churned .’ This column is known as False positive(FP).

,’ but the model has predicted them as ‘ .’ This column is known as The second row and first column (290) is the number of customers who have actually ‘ churned ,’ but the model has predicted them as ‘ not churned .’ This column is known as False Negative(FN).

,’ but the model has predicted them as ‘ .’ This column is known as The second row and second column (343) is the number of customers who have actually ‘churned’; also, the model has predicted them as ‘churned.’ This column is known as True Positive(TP).

We can create this confusion matrix in python by importing the metrics library from the sklearn .

The confusion matrix for the above code looks like the following.

Image by Author — Confusion Matrix

There are many ways to evaluate the model; one of the metrics we measure for the above model is accuracy.

Accuracy

Accuracy is the percentage of correctly predicted labels. The correctly predicted labels from the matrix would be:

True Positive(TP): Churn customers are being predicted as churn

Churn customers are being predicted as True Negative(TN): Non-churn customers are being predicted as non-churn.

Accuracy = (Correctly predicted labels)/(Total number of labels)

Accuracy = (TN+TP )/ (TN+FP+FN+TP)

According to our model above,

TN = 3270

FP = 365

FN = 579

TP = 708

The accuracy will be,

acc = (3270+708)/(3270+365+579+708) = 0.808 = 80.8%

The model we have built is approximately 81% accurate.

We can calculate the accuracy using python directly,

The accuracy we get by the above code is

0.8082080455099553

Checking VIF

Now let’s check the VIF values for the above model to see whether any of the variables are highly correlated.

I’ve already explained VIF and it’s workings in my previous article, “Multiple Linear Regression.” We’ll be following the same procedure here.

Let’s see the VIF values for the variables in our model.

Image by Author — VIF values for finding multi-collinearity

There are a few variables with high VIF. It’s best to drop these variables. The variable PhoneService has the highest VIF. So let’s start by dropping that and build the model again.

The model for the above code is,

Image by Author — Logistic Regression model statistics

Next, we’ll predict the values and print the dataframe consisting of original churn values and predicted values.

The dataframe looks like this,

Image by Author — Dataframe of original values and predicted values

Let’s check the accuracy of the model.

The accuracy of the model will be

0.8051605038602194

Which is not a massive drop to the original one. We have to recheck the VIF values, drop the variables, build the model, and calculate the accuracy. We’ll follow this till the VIF values we get for every variable is less than 5.

For the final model, after following the above process again and again, the confusion matrix we get is,

The matrix looks like this,

array([[3269, 366],

[ 595, 692]],)

The accuracy for the final matrix = 0.804754164973588

Now, the question that arises is — is accuracy the only metric that we use to assess the goodness of the model we’ve built?

The answer is No.

Consider an example: Suppose we are trying to build a logistic regression model for cancer patients, where 1’s for patients’ having cancer’ and 0’s for patients are ‘not having cancer.’ In this case, if we incorrectly predict some of the patients as ‘not having cancer,’ it’ll be very risky. In this type of instance, instead of looking at the overall accuracy, we’ll be looking to predict the 1’s correctly.

In another case, if we are building a model for a bank to block the customers to identify frauds, where 1 represents blocking and 0 as not blocking, we care more about getting the 0’s right. It is because we don’t want to block the good customers.

It is crucial to understand the overall business problem we are trying to solve, to see which metric we want to use.

So, apart from Accuracy, we have another three more important metrics:

Sensitivity /Recall

Specificity

Precision

Sensitivity /Recall

For cancer types of problems, we use sensitivity. So we should build a model, which results in high sensitivity, i.e., the value of FN(False Negative) should be as low as possible.

Sensitivity = (TP)/(TP+FN)

Specificity

For fraudulent transaction types of cases, we build a model having high specificity, i.e., the value of FP(False Positive) should be as low as possible.

Specificity = (TN)/(TN+FP)

Precision

If we want to build a model to predict whether an email is spam or not, we’ll use the Precision metric. In this type of case, we have to develop the model having high precision, i.e., the value of FP(False Positive) should be as low as possible.

Precision = (TP)/(TP+FP)

Now, let’s find out the values of Sensitivity and Specificity for our final model, which we have built earlier.

First, we’ll assign the TP, TN, FP, and FN values from our final matrix.

Here, confusion is the name of the matrix we created earlier.

Now, let’s find out the values of our different metrics.

The sensitivity and specificity values for the above code is,

# Sensitivity

0.5376845376845377 # Specificity

0.8993122420907841

So our model has high accuracy (80.4%) and high specificity (89.9%) but low sensitivity (53.7%), and we are interested in identifying the customers who might churn, so we need to deal with this. But what was the cause of such low sensitivity?

If we remember, while predicting the model as 0 and 1, we chose a cutoff of 0.5, this cutoff is selected at random, and there was no particular logic to it.

The predicted labels depend entirely on the cutoff we have chosen. For low cutoff, we’d have a higher number of customers predicted as 1, which means the higher number of customers being identified as ‘Churn.’ Similarly, for high cutoff, we’d have a low number of customers predicted as 0, that means the higher number of customers being identified as ‘Non-churn.’

Finding Optimal Cutoff Point

In our problem, we are trying to find the optimal cutoff where there will be a balance between sensitivity and specificity. To do that first, we’ll find the predicted values for different cutoffs from 0.1 to 0.9

The churn probability at different cutoff values looks like,

Image by Author — Churn probability at different cutoffs

Now, let’s calculate the accuracy, sensitivity, and specificity of these cutoff values.

The different values of the metrics will be,

Image by Author — Accuracy, Sensitivity and Specificity values

Let’s plot a line graph and see how the accuracy, sensitivity, and specificity behave at those probabilities.

The line graph looks like the following,

Image by Author — Line graph

From the curve above, 0.3 or slightly more is the optimum point to take it as a cutoff probability. We’ll consider 0.3 as the cutoff for ease of understanding.

Let’s predict the model using this 0.3 as the final cutoff.

The output will be

Image by Author — Final prediction

Let’s find out the accuracy, specificity, sensitivity, and confusion matrix as well.

The outputs for the above code is:

# Accuracy

0.771434376269809 # Confusion Matrix

array([[2793, 842],

[ 283, 1004]]) # Sensitivity

0.7801087801087802 # Specificity

0.768363136176066

Since we have our final model, let’s evaluate our model on the test data.

Model Evaluation

Let’s make predictions on the test data using our final model. First, we’ll scale the test data similar to the one we did on train data after splitting.

The test set looks like this,