This Is What Makes Logistic Regression A Classification Algorithm
By: Rashmi Ranu
Tags: Machine Learning, Classification, Data Science, Logistic Regression

This Is What Makes Logistic Regression A Classification Algorithm

Logistic Regression is an extension of Linear regression, except that, here, the dependent variable is categorical and not continuous. It predicts the probability of the outcome variable.

So, Logistic Regression in one of the machine learning algorithm to solve a binary classification problem.

Some real-life classification examples would be :

to classify the mail as spam or not spam

to classify the transaction as fraudulent or genuine

whether a person will buy insurance or not

In the above examples, Logistic Regression estimates the probability of an event occurring i.e either mail is spam (1) or not spam (0).

Although scikit learn library of machine learning makes our job easy but having an understanding about how the algorithms work, helps us in better optimization.

Importing Libraries

#import libraries

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

Reading dataset

Suppose we have a dataset of whether a person will buy insurance or not. In the dependent variable(bought_insurance), 1 means he will buy insurance i.e a positive class and 0 means he will not buy insurance i.e a negative class

If we plot this dataset, we can see that all the points lie on 0 and 1

Seeing the plot we find that there is nothing special about the dependent variable aside from being binary. So we use Linear Regression by assuming the threshold value as 0.5. Which means that if the probability(yes) >= 0.5, the person will buy the insurance and if probability(yes) < 0.5, the person will not buy the insurance.

Implementing Linear Regression

Problem in using Linear Regression

Problem 1.

On the best fit line, if we try to find whether the person with age=15 and age=75 will buy the insurance or not, we are getting the probability as negative and greater than 1 respectively.

And since the probability always lies between 0 and 1, it can never be negative or >1, Linear Regression cannot be used here.

Problem 2.

In case the dataset contains outliers.

Here, you can see that index 27 have new entries with age = 120 and bought_insurance=1, indicating an outlier.

# fitting linear regression model on the new dataset with outlier reg2= LinearRegression()

reg2.fit(data2[["age"]], data2[["bought_insurance"]])

From the plot, we can see that because of the outlier, both the intercept and the coefficient is changed resulting in the shift of regression line.

Now some datapoints which were in positive class (probability(yes)>= 0.5) comes in negative class (probability(yes)< 0.5), giving the incorrect prediction.

print("probability(yes=40) without outlier was {}. This person was buying the insurance.".format(reg.predict([[40]])))

probability(yes=40) without outlier was [[0.52641316]]. This person was buying the insurance.

print("probability(yes=40) with outlier is {}. Now this person will not buy the insurance.".format(reg2.predict([[40]])))

probability(yes=40) with outlier is [[0.49769262]]. Now this person will not buy the insurance.

In short: probability (p)= f(x), a linear function where x is an independent variables.

But we need to constrain probability (p) such that 0<= p <=1.

Solving the above problems of using Linear Regression

Let’s consider y as a linear function in the above univariate regression model, where x = independent variable and y = dependent variable.

To make probability(p) >= 0

The exponential of any number always gives a positive value. Now the probability (p) becomes

To make probability(p) < 1

Dividing any number by a number which is slightly greater than it gives a value less than 1. Now the probability (p) becomes

This Sigmoid Function will also take care of Outliers

On solving the Sigmoid Function we get

here, p / (1-p) = probability of odds

log( probability of odds) = logit transformation of probability

From the above equation,

The LOGIT of probability gives the LINEAR REGRESSION and hence the name LOGISTIC REGRESSION

Link to dataset

Hope you found this article useful and interesting.