The Beginner’s Guide to Logistic Regression
By: Andrew Plummer
Tags: Logistic Regression, Regression, Statistics, Data Science, Machine Learning

The Beginner’s Guide to Logistic Regression

Photo by Gia Oris on Unsplash

Logistic regression is the first binary response models that undergraduate statistics students learn to use. It is also the first classification algorithm that MOOCs teach to aspiring data scientists.

There are a few reasons why logistic regression is so popular:

It is easy to understand. It only requires a few lines of code. It is a great introduction to binary response models.

In this article, I will explain the math behind the logistic regression, including how to interpret the coefficients of the logistic regression model, and explain the advantages of logistic regression over a more naive method.

A Primer on Estimating Probabilities with Regression

Before delving into logistic regression, let’s first discuss a simpler, though intuitively similar, model:

Linear Probability Model (LPM).

The Linear Probability Model is identical to a regression model except for a key difference:

whereas the dependent variable for a regression model is continuous, the dependent variable for a LPM is binary.

If we have one independent variable, then our LPM is

Image by author

where Y is a binary response variable, X is an independent variable, and u is the error term.

For every one-unit change in X, the probability that Y equals 1 changes by B1. If X equals 0, then the probability that Y equals 1 is B0.

Let’s use a practical example. Suppose that Y equals 1 if a person is employed, and 0 if they are not. X is their years of post-secondary education.

We run our regression and estimate that B1 equals 0.07 and B0 equals 0.4. That is,

Image by author

If a person has had one year of post-secondary education, then there is a 47% probability that they will find a job. If a person has had two years of post-secondary education, then the probability that they will get find a job increases by seven percentage points (up to 54%).

A person with no years of post-secondary education will have a 40% probability of finding a job.

The advantage of the linear probability model is that it is easy to interpret. However, there are two major disadvantages of the LPM which makes it unusable in any serious research.

First, what happens if a person has ten years of post-secondary education (they have completed a Bachelors and a PhD)?

In that case, there is a 110% probability that this person will find a job!

It is tempting to interpret this estimate as to mean that we would all be better off with PhDs; however, remember that probabilities can only be between 0% and 100%. Probabilities outside this interval are meaningless.

A second problem with linear probability models is that they assume that changes in the probability that Y equals 1 are constant.

That is, a person who has three years of post-secondary education, and then completes a fourth year (they have just graduated from a Bachelor’s degree) increases the probability that they will find a job by the same amount as the person who has four years of post-secondary education, and then completes a fifth year (they have just graduated from their Master’s).

This is probably not true. Completing a Master’s degree likely has a bigger change on the probability of finding employment than completing a Bachelors degree (or so we would hope).

The Logistic Regression model solves all the problems of the linear probability model.

Logistic Regression

Graph and Equation for Logistic Regression

The logistic regression looks like the graph in the image below:

Image by Author (in Desmos)

The above image makes clear several points about logistic regression.

First, logistic regression is non-linear. Put more technically, changes in the dependent variable depend on the values of the independent variables, and the slope coefficients.

Second, the range (the interval of possible values that the dependent variable can take on) for logistic regression is restricted between 0 and 1, exclusive. Thus, logistic regression avoids the problem of having probabilities less than 0% or greater than 100%.

If we have one independent variable, then the equation for logistic regression is:

Image by Author

where

Image by Author

The logistic regression model grows fastest when:

z = 0 G(z) approaches 0 (as z approaches negative infinity) G(z) approaches 1 (as z approaches positive infinity)

Third, the logistic regression does not require homoscedasticity (errors unrelated to the independent variables) or Gaussian errors to be accurate.

Interpreting the Coefficients of Logistic Regression

The coefficients for the logistic regression are interpreted as ‘log odds’. Log odds are the natural log transform applied to odds ratio.

We have seen odds ratios before (they are common in betting arenas).

For example, say that we are at a horse race.

Photo by Mathew Schwartz on Unsplash

The odds that ‘Slowpoke the Horse’ wins the race is 1:10.

Thus, Slowpoke the Horse is ten times more likely to lose the race than he is to win the race.

We can write the odds ratio, OR, as:

Image by author

where P is the probability of success, given the values of X.

Applying a natural log transform to the odds ratio gives us:

Image by author

Let’s return to our first example, where our dependent variable equals one if we are employed, and zero if we are unemployed. The independent variable is the number of years of post-secondary education.

We estimate the logistic regression and find that the slope estimate for the years of post-secondary education is equal to 3. Thus, for every extra year of post-secondary education, the log odds increases by 3.

What does this mean?

If log(OR) = 3, then we can exponentiate both sides to get

OR = e³

Thus, the odds ratio is about 20.1. This tells us that a person who has completed, say, five years of post-secondary education is twenty times more likely to find a job than someone who has completed four years of education .

The odds of success grows exponentially as the difference between comparison groups widens. For example, a person with six years of post-secondary education is 400 times more likely to find a job than someone who has completed four years of education (the odds are extreme in this fictitious example, but it gets the point across).

Put another way, the difference in odds equals

OR^(Difference between comparison groups).

Logistic Regression Assumptions

The logistic regression model assumes that the probability of an event can be modelled by a sigmoid function (the S curve in the graph above). If you have any reason to believe that the probability does not follow an S curve, then you should not use the logistic regression function.

Conclusion

In this article, we went over the assumptions of logistic regression, the math behind it, and its advantages over the linear probability model.

In another article (who knows when?), I will explain how to use logistic regression with binary and continuous independent variables. I will also explain the Receiver Operating Characteristic (ROC), which is a common tool to illustrate the diagnostic ability of logistic regression (among other binary response models).

I use logistic regression whenever I have a binary dependent variable. It is a powerful tool both for classification problems and generating causal insights. Hopefully, this article encourages you to use it in your next data science project!