Logistic Regression. A gentle introduction to Logistic... | by Sangeet Aggarwal | Towards Data Science

Logistic Regression is one of the fundamental models used in Machine Learning. It is a classification technique and is best suited for predicting the categorical response variable.
While linear regression works well with a continuous or quantitative output variable, the Logistic Regression is used to predict a categorical or qualitative output variable.
For example, target values like price, sales, temperature, etc are quantitative in nature and thus can be analyzed and predicted using any linear model such as linear regression.
But what if we have to predict, whether a mail is a spam or not spam, whether a credit card customer will default or not, whether a given cancer is stage 1, 2, or 3? We cannot use a trivial regression model for such cases, and we need something that can accommodate a qualitative response.
Consider the figure below. The plot shows the credit card default status (0 or 1) against the customer’s account balance. Default status 0 means that no default is shown by the customer, and default value 1 means that the customer defaulted.
As can be seen in the figure above, the linear regression fit (left) exceeds the possible limits of 0 and 1. This happens because linear regression does not take into account the fact that the values of the response variable are categorical and discrete. It may plot even higher values for a customer having high account balance and vice versa, which does not make any sense. This is why we cannot use linear regression for such problems.
Such a problem can be solved by Logistic Regression (right) as it only takes values between 0 and 1.
The Logistic Regression is mostly used and best suited for problems having 2 response classes, for example, → 0 or 1, true or false, spam or not spam, type A or type B, etc.
Although it can be extended to predict response with more than 2 classes, there are several other ways that are better than Logistic Regression to deal with those problems. Hence, for this post, we will only focus on problems that have two response classes.
To understand Logistic Regression, we must understand how it generates results of classification. Does it simply predict the response class? No, it does not.
Logistic Regression calculates the probability, by which a sample belongs to a class, given the features in the sample. This probability is calculated for each response class. The class with the highest probability is generally taken to be the predicted class.
For a problem with 2 response classes, the probabilities of both classes add to 1 and can be represented as below (taking the class values as 0 and 1).
Pr(Class = 1 | Feature(s)) = 1 — Pr(Class = 0 | Feature(s))
Here, Pr(Class=1|Feauture(s)) is read as → the probability of a sample belonging to class 1 with the given feature(s). Since it’s a probability, its value is always between 0 and 1, both inclusive.
Recall the equation of Simple Linear Regression.
For Logistic Regression, this equation takes such a form, that it only returns values between 0 and 1.
This is the Logistic Function, whose values range from 0 to 1 for any value of X. Hence the plot of Logistic Function takes an S-shape with its tails ending towards 0 and 1, with values in between.
With few manipulations, the above equation can be rewritten as:
The ratio expression on the left-hand side is called the odds and can take values from 0 to infinity. The value of odds close to 0 indicates a very low probability of class 1 (high probability for class 0). A large value of odds indicates a high probability of class 1.
Taking log on both sides:
The expression on the left is called Logit function or the log-odds function. This logit function is linear in X and can be used to interpret the coefficients of X terms, as we will see later.
We will first see how Logistic Regression works when there is a single input variable. We will then learn how the model can be extended to accommodate more input variables.
Let’s catch a glimpse of the data we are going to use. This data represents customers of a credit card issuer, with few details like whether or not they are students, their income, balance, and if they were defaulters.
The company wants to build a logistic model that can predict whether a customer will default or not. So the default column is our target variable. To build our first logistic model, let’s just consider the balance of a customer as our only predictor.
So our model equations will look like these:
Let’s build our model using SciKit Learn in Python.
from sklearn.linear_model import LogisticRegressionX = data.balance.values.reshape(-1,1)y = data.defaultlogistic = LogisticRegression()logistic.fit(X,y)
Plotting the results of the above model, we will get something like this.
Let’s check the intercept and coefficient terms.
print("Intercept: ",logi.intercept_)print("Coefficient: ",logi.coef_)
Output:
Intercept: [-5.75236929]Coefficient: [[0.00448445]]
These can be interpreted as →
For every unit change in balance, the log-odds of default gets multiplied by e0-00445.
The intercept term -5.75 can be read as the value of log-odds when the account balance is zero.
Let’s now move on to the case where we consider the effect of multiple input variables to predict the default status.
The equation for multiple logistic regression can be written as:
And the corresponding log-odds function will become:
For our model to work, we need to change all categorical inputs into numeric labels. In this case, the student variable has values “Yes ” and “No”. Thus, I have converted these values to 1 and 0, respectively. Let’s have another look at the data before moving on.
Let’s fit the model.
X_all = data.drop('default',axis=1)y = data.defaultmlr = LogisticRegression()mlr.fit(X_all, y)
Check the coefficients.
print("Coefficients for stuedent, income and balance are:")for i in range(3):    print(round(mlr.coef_[0][i],5))print("Intercept: ", mlr.intercept_[0])
Output:
Coefficients for stuedent, income and balance are:-1.63465-5e-050.0047Intercept: -3.8386588181706274
Here, the coefficient value of income is quite low, which indicates that the income of a customer does not really add to the odds of default. We can do further analysis by checking for correlation and other insights to improve the understanding of our data and model. But to maintain the scope of the topic, I’m not adding that part of the analysis here.
I hope this post helped you understand the working of Logistic Regression. You can also check the links below for knowing about Linear Regression. For more such posts, please stay tuned.
towardsdatascience.com
towardsdatascience.com
If you are new to Data Science and Machine Learning and wondering where to begin your journey from, do check the link below, where I have mentioned step by step method to learn Data Science, with lots of sources for you to choose from.
towardsdatascience.com
Can’t wait? If you want to dive right into a course, check out the career tracks in Data Science that suits you, from the link below.