Crux Of Logistic Regression
By: Sunil Kumar
Tags: Logistic Regression, Machine Learning

Introduction

Logistic Regression is a supervised learning method which help you to predict in case where output is categorical. Before we jump into logistic regression, first and foremost question come in our mind.

Photo by Luis Rocha on Unsplash

What is wrong with linear regression for classification?

Linear Regression works best for continuous variable but fails for classification. In case of two classes, where we have to predict the label as 0 or 1. We can use linear regression but model doesn’t output any probability value. Therefore we can’t predict whether a given data point belongs to class 0 or 1.

Therefore, the concept of linear Regression come into picture. Let’s understand logistic regression in detail:

Consider a data set of the list of patients with Blood Sugar Level.

Problem Statement: Classify a patient as diabetes or non-diabetes on the basis of sugar level.

table 1

Approach 1:

We can start to solve a classification problem from binary sharp decision boundary to classify the patient as Diabetes or Non-Diabetes. But the patient who has intermediate sugar level can be misclassify as Non-Diabetes and Diabetes.

Figure 1

Therefore sharp binary classification is not correct approach to solve above problem.

Approach 2 :

In order to solve the above problem we go with probabilistic approach. i.e Given sugar level what is the probability of having diabetes.

Let’s analyze the problem with the help of sigmoid curve.

Figure 2

Red mark arrow shows probability of blood sugar in non-diabetic patients(i.e low).

Blue mark arrow shows probability of blood sugar diabetic patients(i.e high).

In above data set we have single feature to decide whether a patient is diabetic or not. (i.e Blood sugar level). Therefore, we can also say it as Univariate logistic regression.

Now, here we head up with another question.

How to come up with best fit sigmoid curve that covers maximum correct classification?

Let’s deep dive into sigmoid curve in more details:

Figure

Red mark arrow shows probability of blood sugar in non-diabetic patients.

Blue mark arrow shows probability of blood sugar diabetic patients.

[(1-P1)(1-P2)(1-P3)(1-P4)(1-P6)…non diabetic patients][(P5)(P7)(P8)(P9)(P10)…diabetic patients]

This is something called likelihood function. The best fit sigmoid curve will maximise the likelihood function.

Let’s simplify the sigmoid curve equation and understand in the context of above example.

Figure 3

In above the figure, x denotes the blood sugar level of the paitent. As the level of sugar level i.e x increases the probability of patients being diabetic also increase. This is how the probability of having diabetes varies with blood sugar level (Univariate variable).

Multi-variate Logistic regression: Probability of an events depends upon more than one independent variables.

OR consider the equation of sigmoid curve for number of independent variables as x1, x2, x3……xm can be also written as :

which can futher simplify as follow:

In a similiar way, probabilistic value for target variable may vary with independent variables.

In the next post, I plan to implement logistic regression with statmodel library and apply different evaluation metrics to the model. So stay tuned, and happy analyzing!