Logistic Regression in Classification model using Python: Machine Learning | by Kaushik Katari | Towards Data Science

Logistic regression is a commonly used model in various industries such as banking, healthcare because when compared to other classification models, the logistic regression model is easily interpreted.
Binary classification is the most commonly used logistic regression. Some of the examples of binary classification problems are:
A finance company wants to know whether a customer is default or not
Predicting an email is spam or not
Whether a person is diabetic or not
The binary classification always has only two possible outcomes, either ‘yes’ & ‘no’ or ‘1’ & ‘0’ etc.
Like in the previous article “Multiple Linear Regression model, “ one independent variable is often not enough to capture all the uncertainties of the logistic regression’s target variable.
Let’s now build a logistic regression model using python in the Jupyter notebook.
For the entire article, we use the dataset from Kaggle.
We’ll be looking at the telecom churn prediction dataset. It has 21 variables related to customer behavior to predict whether a particular customer will switch to another telecom provider or not (i.e., churn or not).
First, we’ll start by importing the necessary libraries to the notebook and convert the .csv file to the pandas data frame.
The dataset looks like this,
Our target variable in the above dataset is the Churn column.
Let’s inspect the dataframe using .shape, .describe and .info methods.
The output for the code will be,
As we can see, there are 7043 rows and 21 columns in the dataset, there is also no significant jump in the values for the numeric variables from minimum to maximum, and the dataset has no null values.
Now let’s prepare the data by converting the ‘yes’ & ‘no’ values in the variables to ‘1’ & ‘0’. We’ll map the values to ‘1’ and ‘0’ by defining a function and mapping the ‘yes’ & ‘no’ values to ‘1’ & ‘0’.
The dataset looks like this after mapping the values.
Create Dummy VariablesFor the columns having more than two values, we’ll create dummy variables similar to the one we did in the “Multiple Linear Regression model” article.
The output will be,
If we observe the above dataset, the number of columns increased from 21 columns to 29 columns.
We are creating dummy variables for the remaining categorical variables.
The dataset after creating dummy variables for all the variables will be,
Drop the original variablesNow we’ll drop the variables for which we created the dummy ones, and we’ll convert the datatype of the columns if there are any discrepancies in them.
The info for the dataset looks like,
Outliers and Missing ValuesNow let’s check once more for the outliers and null values in the dataset before proceeding to model building.
The output will be,
From the distribution data, we can see that there are no outliers in the data. There is a gradual increase in the numbers.
Now let’s check for missing values in the dataset and handle the data.
Let’s check the percentage of null values in the columns to make an informed decision on how to handle those values.
The output is,
If we observe, the Total Charges column has less than 0.2% null values. So, it is better to remove the rows instead of inputting some values into them.
The output after handling the null values will be,
Since we have handled the null values, now let’s move on to the model building.
Before building the model, we’ll split the data into train and test data, similar to the linear regression model. So that we’ll make the model using train data and evaluate the model on test data.
By using the sklearn library, we’ll import the test_train_split to split the data. But before splitting the data, we’ll divide the dataset into two dataframes X and y.
We’ll drop the customerID column since it is of no use to our model and the churn column since it is our target variable. The test data looks like this,
Let’s assign the target variable to y and split the data into test and train sets.
The ‘y’ dataset looks like this,
If we observe the data, all the numeric values in the dataset are in different ranges. One is in the range below 100, one is in the range of 200, and another is more than 1500. If we build the model using these values, the model’s coefficients will be of different units, so it isn’t easy to compare them to make an informed decision.
To do that, we’ll re-scale all the continuous variables. To do that, we’ll use the StandardScaler method from sklearn.
The dataset after scaling the features,
Let’s check the percentage of customers who are churned from the dataset to know how many customers have transferred to other networks.
The output is 26.578498293515356So, we have an almost 27% churn rate.
Let’s look at the correlation between the variables. Suppose the correlation is high between the variables. In that case, we can remove the columns because if two variables are highly correlated, then there is no need to build the model using both the variables. We can explain the target variable using one of these two variables.
The correlation heatmap between the variables will look like this,
If we observe the heatmap, some of the variables are highly correlated with each other. So we’ll drop those variables from the dataset.
Dropping highly correlated dummy variablesWe’ll drop the highly correlated variables from the X_train and X_test datasets we’ve created above.
Now, let’s recheck the correlation in the X_train dataset.
The correlation heatmap after dropping the variables,
There are still some variables with high correlation, but we can remove them while building the model. Let’s do that.
We’ll use the stasmodel library to build our first model. Let’s see how our model looks like if we consider all the variables.
The statistics of our first model is,
The above model is not a good one because some variables have high p-values, which means the variables are insignificant. Instead of removing variables one at a time and building a model, again and again, we’ll use a method called RFE to select the top 15 variables for the model and remove the less significant variables based on p-values and VIF values.
We’ll select the top 15 variables, which helps in building the model in our logistic regression.
The top 15 variables are,
The variables having showing True are the ones we are interested in, and if we want to add more than 15 variables, we can add them one by one based on their respective ranks.
Build the model using the above 15 variablesLet’s build our second model using the 15 variables we got from RFE.
The model statistics will be,
Here we are using the GLM (Generalized Linear Models) method from the statsmodels.api library. Binomial in the family argument tells the statsmodels that it needs to fit a logit curve to binomial data (i.e., the target variable will have only two values, in this case, ‘Churn’ and ‘Non-Churn’).
A sample logit curve looks like this,
Predicting the output probabilitiesThe logistic regression curve gives us the probabilities of churning and not churning. We can get these probabilities by simply using the ‘predict’ function.
The predicted probabilities are
Now, let’s create a dataframe with the actual churn column and the predicted probabilities.
The dataframe for the above program will be as follows.
Since the logistic curve gives us the probabilities and not the actual classification of ‘Churn’ and ‘Non-Churn,’ we need to find a threshold probability to classify customers as ‘churn’ and ‘non-churn.’
Here, let’s choose 0.5 as an arbitrary cutoff wherein if the probability of a particular customer churning is less than 0.5, we’d classify it as ‘Non-Churn,’ and if it’s greater than 0.5, we will classify it as ‘Churn.’
Creating a new columnThe choice of 0.5 is entirely arbitrary at this stage, and we’ll learn how to find the optimal cutoff in ‘Model Evaluation.’
The dataframe looks like this after choosing an arbitrary cutoff.
Here, we used a cutoff of 0.5 to classify the customers as ‘Churn’ and ‘Non-churn.’ Now, since these are the probabilities, there are bound to be errors.
We’ll encounter two types of errors:
‘Churn’ customers being incorrectly classified as ‘Non-Churn.’
‘Non-Churn’ customers being incorrectly classified as ‘Churn.’
To find these errors and the model’s wellness, we’ll use a phenomenon called ‘Confusion Matrix.’
A typical confusion matrix looks like the following.
Note: The values shown in the above matrix is just a sample
If we observe the above matrix:
The first row and first column (1250) is the number of customers who have actually ‘not churned’; the model has also predicted them as ‘not churned.’ This column is known as True Negative(TN).
The first row and second column (150) is the number of customers who have actually ‘not churned,’ but the model has predicted them as ‘churned.’ This column is known as False positive(FP).
The second row and first column (290) is the number of customers who have actually ‘churned,’ but the model has predicted them as ‘not churned.’ This column is known as False Negative(FN).
The second row and second column (343) is the number of customers who have actually ‘churned’; also, the model has predicted them as ‘churned.’ This column is known as True Positive(TP).
We can create this confusion matrix in python by importing the metrics library from the sklearn.
The confusion matrix for the above code looks like the following.
There are many ways to evaluate the model; one of the metrics we measure for the above model is accuracy.
Accuracy is the percentage of correctly predicted labels. The correctly predicted labels from the matrix would be:
True Positive(TP): Churn customers are being predicted as churn
True Negative(TN): Non-churn customers are being predicted as non-churn.
Accuracy = (Correctly predicted labels)/(Total number of labels)
Accuracy = (TN+TP )/ (TN+FP+FN+TP)
According to our model above,
TN = 3270FP = 365FN = 579TP = 708
The accuracy will be,
acc = (3270+708)/(3270+365+579+708) = 0.808 = 80.8%
The model we have built is approximately 81% accurate.
We can calculate the accuracy using python directly,
The accuracy we get by the above code is
0.8082080455099553
Now let’s check the VIF values for the above model to see whether any of the variables are highly correlated.
I’ve already explained VIF and it’s workings in my previous article, “Multiple Linear Regression.” We’ll be following the same procedure here.
Let’s see the VIF values for the variables in our model.
There are a few variables with high VIF. It’s best to drop these variables. The variable PhoneService has the highest VIF. So let’s start by dropping that and build the model again.
The model for the above code is,
Next, we’ll predict the values and print the dataframe consisting of original churn values and predicted values.
The dataframe looks like this,
Let’s check the accuracy of the model.
The accuracy of the model will be
0.8051605038602194
Which is not a massive drop to the original one. We have to recheck the VIF values, drop the variables, build the model, and calculate the accuracy. We’ll follow this till the VIF values we get for every variable is less than 5.
For the final model, after following the above process again and again, the confusion matrix we get is,
The matrix looks like this,
array([[3269,  366],       [ 595,  692]],)
The accuracy for the final matrix = 0.804754164973588
Now, the question that arises is — is accuracy the only metric that we use to assess the goodness of the model we’ve built?The answer is No.
Consider an example: Suppose we are trying to build a logistic regression model for cancer patients, where 1’s for patients’ having cancer’ and 0’s for patients are ‘not having cancer.’ In this case, if we incorrectly predict some of the patients as ‘not having cancer,’ it’ll be very risky. In this type of instance, instead of looking at the overall accuracy, we’ll be looking to predict the 1’s correctly.
In another case, if we are building a model for a bank to block the customers to identify frauds, where 1 represents blocking and 0 as not blocking, we care more about getting the 0’s right. It is because we don’t want to block the good customers.
It is crucial to understand the overall business problem we are trying to solve, to see which metric we want to use.
So, apart from Accuracy, we have another three more important metrics:
Sensitivity /Recall
Specificity
Precision
For cancer types of problems, we use sensitivity. So we should build a model, which results in high sensitivity, i.e., the value of FN(False Negative) should be as low as possible.
Sensitivity = (TP)/(TP+FN)
For fraudulent transaction types of cases, we build a model having high specificity, i.e., the value of FP(False Positive) should be as low as possible.
Specificity = (TN)/(TN+FP)
If we want to build a model to predict whether an email is spam or not, we’ll use the Precision metric. In this type of case, we have to develop the model having high precision, i.e., the value of FP(False Positive) should be as low as possible.
Precision = (TP)/(TP+FP)
Now, let’s find out the values of Sensitivity and Specificity for our final model, which we have built earlier.
First, we’ll assign the TP, TN, FP, and FN values from our final matrix.
Here, confusion is the name of the matrix we created earlier.
Now, let’s find out the values of our different metrics.
The sensitivity and specificity values for the above code is,
# Sensitivity0.5376845376845377# Specificity0.8993122420907841
So our model has high accuracy (80.4%) and high specificity (89.9%) but low sensitivity (53.7%), and we are interested in identifying the customers who might churn, so we need to deal with this. But what was the cause of such low sensitivity?
If we remember, while predicting the model as 0 and 1, we chose a cutoff of 0.5, this cutoff is selected at random, and there was no particular logic to it.
The predicted labels depend entirely on the cutoff we have chosen. For low cutoff, we’d have a higher number of customers predicted as 1, which means the higher number of customers being identified as ‘Churn.’ Similarly, for high cutoff, we’d have a low number of customers predicted as 0, that means the higher number of customers being identified as ‘Non-churn.’
In our problem, we are trying to find the optimal cutoff where there will be a balance between sensitivity and specificity. To do that first, we’ll find the predicted values for different cutoffs from 0.1 to 0.9
The churn probability at different cutoff values looks like,
Now, let’s calculate the accuracy, sensitivity, and specificity of these cutoff values.
The different values of the metrics will be,
Let’s plot a line graph and see how the accuracy, sensitivity, and specificity behave at those probabilities.
The line graph looks like the following,
From the curve above, 0.3 or slightly more is the optimum point to take it as a cutoff probability. We’ll consider 0.3 as the cutoff for ease of understanding.
Let’s predict the model using this 0.3 as the final cutoff.
The output will be
Let’s find out the accuracy, specificity, sensitivity, and confusion matrix as well.
The outputs for the above code is:
# Accuracy0.771434376269809# Confusion Matrixarray([[2793,  842],       [ 283, 1004]])# Sensitivity0.7801087801087802# Specificity0.768363136176066
Since we have our final model, let’s evaluate our model on the test data.
Let’s make predictions on the test data using our final model. First, we’ll scale the test data similar to the one we did on train data after splitting.
The test set looks like this,
Let’s make predictions now on test data, follow the below code carefully; we are making many changes to the datasets.
The final dataset is for after all the changes are,
Let’s rename the column and rearrange them for ease of understanding.
The dataset looks like,
Let’s predict the model using 0.3 as cutoff, which we got earlier on the train dataset.
The predicted values for the test set is,
Let’s check the accuracy, sensitivity, and specificity values for the test data and compare it.
The values of the test set are:
# Accuracy0.7440758293838863# Confusion Matrixarray([[1150,  378],       [ 162,  420]],)# Sensitivity0.7216494845360825# Specificity0.7526178010471204
If we compare the training dataset’s accuracy, sensitivity, and specificity values, there is not that much of a change. So the model we have built is good enough to predict any future telecom data.
To summarize, we have seen the steps we need to take to build a basic logistic regression model. There are some more concepts we can learn like the ROC curve, Precision, Recall, etc. But to build a basic logistic regression model, the process we have seen in this article is good enough. Once we master this process, we can explore more things in this model.
Thank you for reading and happy coding!!!
Multiple Linear Regression model using Python: Machine Learning
Simple Linear Regression Model using Python: Machine Learning
Exploratory Data Analysis(EDA): Python
Central Limit Theorem(CLT): Data Science
Inferential Statistics: Data Analysis
Seaborn: Python
Pandas: Python
Matplotlib: Python
NumPy: Python
Machine Learning — Logistic Regression: https://www.tutorialspoint.com/machine_learning_with_python/machine_learning_with_python_classification_algorithms_logistic_regression.htm
Logistic Regression: https://machinelearningmastery.com/logistic-regression-for-machine-learning/
Logistic Regression: https://ml-cheatsheet.readthedocs.io/en/latest/logistic_regression.html
Logistic Regression For Machine Learning and Classification: https://kambria.io/blog/logistic-regression-for-machine-learning/