Logistic Regression Explained
By: Mazen Ahmed
Tags: Logistic Regression, AI, Learn Data Science, Data Science, Machine Learning

Logistic Regression Explained

Logistic Regression can be thought of as an extension of Linear Regression. With Linear Regression our final output for our model took a single value, however, with logistic regression, we apply an extra function to Linear Regression that puts our final value output into a group i.e. 1 or 0

Please consider watching this video if any section of this article is unclear.

Video Link

What is Logistic Regression?

Logistic regression is a very common supervised machine learning algorithm (see Episode 3) used by Data Scientists to categorize data into groups.

Overview

The job of logistic regression is take a bunch of input data and organise the data into different groups. For example take a look at the follow table of weather data gathered from Albury, Australia.

Our objective is to predict whether it will rain tomorrow in Albury.

in Albury. Our model therefore has two group outputs: 0 - No or 1 - Yes

- No or - Yes Since our model has group outputs we use logistic regression to achieve our objective

We build a logistic regression model that:

Step 1:

Calculates a value from the weather data using linear regression

𝑥₁,𝑥₂, …, 𝑥₈ are our weather inputs: Min temp, Maxtemp, … , RainToday

θ₀, θ₁,θ₂, …. , θ₈ are set to random values as with linear regression.

Step 2:

Converts this value (z) into a probability between 0 and 1 of it raining tomorrow. (We use the logistic function to do this).

Step 3:

Places this probability into a group.

For example:

If this probability is ≥50% then our model should predict yes (1).

If this probability is <50% our model should predict no (0). This is called a boundary condition.

Our model takes the following form:

This is essentially, a very simple neural network, we will be covering neural networks in much more detail in a future episode.

The Logistic Function

The logistic function is given by the following formula:

When plotting the logistic function for -6< z <6 our graph looks like this.

Essentially, the logistic function takes any value (z) and puts it on a scale between 0 and 1. This is useful since it can convert values outputted from our linear regression model (step 1) into a probability.

We can then implement a decision boundary:

To put the probability calculated by our logistic function into a group.

i.e.

If the probability is greater that 0.5 output 1 (yes)

If the probability is less than 0.5 output 0 (no)

The Cost Function

I will not go into detail as how the cost function is calculated as this is not necessary to know, however I will try explain the logic behind the logistic regression cost function:

Just as with linear regression, logistic regression also has cost function. This cost function is used to calculate our model’s parameters θ₀, θ₁,θ₂, …. , θ₈ by gradient descent.

The cost function gives an indication of how far away our model’s predicted values hθ(𝑥) are away from our actual values y. With linear regression we used mean squared error for this.

Note the first two steps of our model can be combined into the single function:

Since we have two outputs 0 or 1 we need a cost function that can record the model error for each. We have the two cost functions:

Why do these cost functions work?

For our first function where our actual output y= 1, if our model predicts a value far from this, i.e. 0.01 , we should have a high cost.

Let’s test:

log(0.01) = 2 is high ✓ reflecting bad model performance (high error).

Likewise of our model predicts a value close to 1, i.e 0.99 we should have a low cost.

Let’s test:

log(0.99) = 0.004 is low ✓ reflecting good model performance (low error).

The same applies for y = 0

Combining the cost functions:

We can do a sneaky maths trick to combine both cost functions into one:

This works since if y = 1 we get:

as before, and if y= 0 we get:

also as before.

Lastly we we iterate our cost function over all data observations to get an overall cost for out model.

We then divide this cost by our number of training examples m to obtain an average cost:

We can then apply gradient descent on our final cost function J(θ) to calculate our parameters θ₀, θ₁,θ₂, …. , θ₈ for our model.

Evaluating Logistic Regression Models

We can evaluate a logistic regression model using what's called a confusion matrix shown below:

Here we have used the confusion matrix on 100 data observations.

The accuracy of our model is given by:

We use this evaluation metric to evaluate and make improvements to our model.

Summary

Logistic Regression is used to categorize data into groups

It works similar to linear regression but contains:

The Logistic function to convert values into probabilities

to convert values into probabilities A Boundary condition to place probabilities into groups (0 or 1)

In the Next Episode we will implement logistic regression in python.

If you have any questions please leave them below!