Logistic Regression- In depth explanation
By: Anushka S
Tags: Logistic Regression, Data Science, Deep Learning, Python, Machine Learning

In this blog, we will discuss the in depth explanation of Logistic Regression, it’s types, what kind of problems it helps us to solve, cost function, gradient descent and it’s python implementation.

Figure1: Logistic Regression Model(Source:https://medium.com/@mallrishabh52/logistic-regression-12fe50ffacb3)

So, before starting into Logistic Regression we must know, what exactly is Regression.

What is Regression?

Regression analysis is a form of predictive modelling technique which investigates the relationship between a dependent variable(target) & an independent variable(predictor).

Logistic Regression is used when the dependent variable is categorical. It produces results in binary format which is used to predict the outcome of a categorical independent variable. So, the outcome should be discrete/categorical such as:- 0 or 1, Yes or No, True or False, High and Low.

For example,

To predict whether the tumor is malignant (1) or not (0)

To predict whether it is raining (0) or not (1)

To predict whether an email is spam (1) or not (0)

Why not Linear Regression?

Logistic Regression helps us in solving classification problems. In linear regression, the value which we need to predict is in a range but in case of logistic regression we just need 2 values, either 0 or 1 , it can’t be below 0 or above 1.

Consider a scenario where we need to classify whether it is raining or not. If we use linear regression for this problem, there is a need for setting up a threshold based on which classification can be done. Say if the actual class is malignant, predicted continuous value 0.4 and the threshold value is 0.5, the data point will be classified as not malignant which can lead to serious consequence in real time.

Figure2: Linear v/s Logistic regression (Source: https://www.javatpoint.com/linear-regression-vs-logistic-regression-in-machine-learning)

Types of Logistic Regression

Binary Logistic Regression

The categorical response has only 2 possible outcomes. Example:- If a mail is spam or not.

2. Multinomial Logistic Regression

Three or more categories are there without ordering them. Example:- People’s occupation choices.

3. Ordinal Logistic Regression

Three or more categories with ordering. Example:- Movie rating from 1 to 5.

Model

Output = 0 or 1

Hypothesis => Z = WX + B

hΘ(x) = sigmoid (Z)

Sigmoid function

As we know that logistic regression helps in solving the binary classification problems, here the sigmoid function plays a role of an activation function. It takes the weighted sum of the input features as an input and outputs the probability value of the outcome.

Figure3: Graphical representation of sigmoid function

As we can see, sigmoid function is a S-shaped curve. For any value of x, the sigmoid function will give an output between the value of 0 and 1. The sigmoid function is represented by the formula:-

Figure4: Formula of sigmoid function

Limits of sigmoid function

Figure5

Based on the limits of the inverse of the exponential, we can now compute the limits of the sigmoid function. When x takes negative values, we already know that the inverse of the exponential goes towards +Ꝏ, and since the rest of the terms are constant we end up with 1 over Ꝏ which goes toward 0.This means that the sigmoid function goes toward 0 when x is negative.

Figure6

Now the case that X goes toward 0 as we have seen earlier, the inverse exponential goes towards the value of 1. This means that we end up with a ratio of 1 over (1 + 1) which is equal to 0.5. Consequently, when x equals 0, the sigmoid function takes the value of 0.5.

Figure7

Finally when x takes large positive values, we know that inverse exponential vanishes toward the value of 0. In this case, we end up with a ratio of 1 over 1 which is equal to 1. This means that the sigmoid function converges toward the value of 1 for large positive values of x.

In this case, the three values 0, 0.5 and 1 are the points of interests in the sigmoid function.

Decision Boundary

If a prediction function returns a probability score between 0 and 1, in order to map this to a discrete class(true/false, cat/dog), we select a threshold value or tipping point above which we classify values into class 1 and below which we classify values into class 2.

Decision boundary can be linear or non-linear. Polynomial order can be increased to get complex decision boundary.

Figure8

Cost function

Figure9: Cost function of logistic regression

Why the cost function which has been used for linear regression cannot be used for logistic?

Linear regression uses mean squared error as its cost function. If the same cost function is used for logistic regression, then it will be a non-convex function of parameters (theta). Gradient descent will converge into global minimum only if the function is convex.

Cost function explanation

Figure11: Cost function explanation

Simplified cost function

Figure12: Simplified cost function

Derivation of formula for Gradient Descent

Figure13: Gradient Descent Algorithm pt.1

Figure14: Gradient Descent Algorithm pt.2

Python implementation of Logistic Regression

Full code:-

https://github.com/anushka012/Logistic_Regression/blob/main/LogisticRegression.ipynb

This blog post is inspired from Prof. Andrew Ng’s deep learning course.