Logistic Regression Math & Geometrical Intuition with Example
By: Shashi Kiran
Tags: Logistic Regression, Classification Models, Machine Learning, Classification Algorithms, Data Science

Logistic Regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval, or ratio-level independent variables.

Logistic Regression is a Classifier which is used to solve the classification problems. As it’s technically dependent on the Linear Regression & Logit function is a method for a classification problem.

Why Can’t we use Linear Regression for Classification?

If everything is fine we can use linear regression, why logistic regression is needed.

a. Outlier: If the data set has an outlier, linear regression will not perform better.

b. High-end classification: Can be classified as >1 or <0 which is an issue

What are the assumptions in Logistic Regression?

The assumption in logistic regression

1. Logistic regression requires the dependent variable to be binary.

2. Classes are almost linearly separable points.

3. Requires to be little or no multicollinearity among the independent variables.

4. Typically requires a large sample size.

For More assumptions: https://www.statisticssolutions.com/assumptions-of-logistic-regression/

Mathematical & Geometrical Intuition in Logistic Regression.

Step — 1 Plot the Independent & Dependent points & Consider the best fit line (almost Linearly Separable points)

Step — 2 Calculate the distance function from the best fit line on the linearly separable points.

Consider the above example have the best fit line value to be 0, points classified below the best fit line considered as -ve point & above the best fit line is +ve point.

Let’s calculate the distance function from the Linear algebra.

From Linear Algebra we have

Considering ||W|| as a unit value & the intercept C passing through 0. The above formula will be modified as

Based on the above formulation we will have the following distance cases based on the classification in the above pic

Case 1: As per the given data yi = +1 & wt * xi > 0

Yi * wt * xi > 0

Case 2: As per the given data yi = -1 & wt * xi < 0

Yi * wt * xi > 0

Case 3: As per the given data yi = -1 & wt * xi > 0

Yi * wt * xi < 0

Case 4: As per the given data yi = +1 & wt * xi < 0

Yi * wt * xi < 0

Here Case 1 & Case 2 are correctly classified points but Case 3 & Case 4 are wrongly classified points.

Great now we got the distances of the classification points from the best fit line.

Step 3: Summing up the calculated distances & finding the arg max.

As we have calculated the distances we are now going to sum & consider the max value to evaluate the best fit line by using the below formula.

The Best Fit line with max cost function will be considered. By Changing the W transpose value we can drive our best fit line, as we have the yi & xi are fixed (from the dataset).

Finally we need to Maximize correctly classified points & Minimize wrongly classified points.

What if the data having an outlier? Let’s consider an example.

Case 1:

Let’s implement the cost function for different best fit line

Will input the data for the cost function above,

Negative class classification are calculated as +1 ( because -1 * -1)

Case 2:

Let’s implement the cost function for different best fit line

As per the cost function, this is not the best fit still we will try differently

Will input the data as given above,

Negative class classification are calculated as -1 ( because -1 * +1)

As per the results on both cases, we see that the case 2 (CF = -40) is having a max cost function considering the case 1 (CF = 1), But visually we see that case 1 is having the best fit function than Case 2.

Due to an outlier, we have the best fit line but the majority of -ve classes are classified as +ve which may have a major impact on predicting the outcome. Fig below.

So how do we solve this Outlier issue?

The simple answer is to implement the Sigmoid function outlier & to optimize the formula as below.

Sigmoid Definition (f):

A sigmoid function is a mathematical function having a characteristic “S”-shaped curve or sigmoid curve. A common example of a sigmoid function is the Logistic function shown in the first figure and defined by the formula:

Sigmoid formula:

Sigmoid Graph:

plot(1/(1+e^-x)) — Google Search

Altered Cost function formula:

The sigmoid function will get our outlier values between within our range which will result in the squashing technique for reducing the outlier issue.

Thanks for Reading.

You can reach me at:

LinkedIn : https://www.linkedin.com/in/shashikiranc/