Why Is Logistic Regression the Spokesperson of Binomial Regression Models?
Tags: Classification, Logistic Regression, Data Science, R, Binomial

A lot of events in our daily life follow the binomial distribution that describes the number of successes in a sequence of independent Bernoulli experiments.

For example, assuming that the probability of James Harden making his shot is constant and each shot is independent, the number of field goals follows the binomial distribution.

If we want to find the relationship between the success probability (p) of a binomially distributed variable Y with a list of independent variables xs, the binomial regression model is among our top choices.

The link function is the major difference between a binomial regression and a linear regression model. Specifically, the linear regression model uses p directly as the response variable.

linear regression

The problem of the linear regression is that its response value is not bounded. However, the binomial regression uses a link function (l) of p as the response variable.

Binomial regression with a link function

The link function maps the linear combination of xs to a value that is between 0 and 1 but never reaches 0 or 1. Based on such criteria, there are mainly three common choices:

Binomial regression link functions

When the link function is the logit function, the binomial regression becomes the well-known logistic regression. As one of the most first examples of classifiers in data science books, logistic regression undoubtedly has become the spokesperson of binomial regression models. There are mainly three reasons for that.

1. Applicable to more general cases. 2. Easy interpretation. 3. Works in retrospective studies.

Let’s go over them in detail.

Extreme value distribution? Not really.

A major difference of the complementary log-log model is that probit and logit functions are symmetrical but the complementary log-log function is asymmetric.

# R code

eq = function(x){log(x/(1-x))}

curve(eq, xlab="probability", ylab="l(probability)",col="red",lwd=2)

curve(qnorm,add = T,lty="dashed",col="blue",lwd=2)

curve(cloglog,add=T,lty="dotted",col="green",lwd=2)

abline(v=0.5,lty="dashed",col="grey")

legend("topleft",legend=c("logit","probit","cloglog"),lty = c("solid","dashed","dotted"),col = c("red","blue","green"),lwd=2)

Link functions’ curves. Symmetric: logit (red) and probit (blue); Asymmetric: cloglog (green)

Specifically, we can see that the symmetric functions (logit and probit) cross at the position of p=0.5. However, the cloglog function has a different rate of approaching 0 and 1 on the probability. With such a feature, the cloglog link function is always used on extreme events where the probability of the event is close to either 0 or 1.

In reality, it’s nearly impossible for us to make the assumption of the extreme value distribution with a limited amount of data. Therefore, in most cases, the cloglog link function is not selected for the binomial regression model.

In other words, the logit and probit model could be applied to more general cases than the cloglog model. Let’s next look at the advantages of the logit model over the probit model.

β unit increase in Z-score?? Weird.

Logistic regression is more widely used than probit regression because of its easy interpretation, which benefits from the concept of odds.

Interestingly, odds are sometimes more frequently used than probability in our daily life because of its better scale to represent chance, eg. the NBA playoffs bet. The relationship between the odds (o) and the probability (p) can be described in the following equation.

odds and probability

And the logistic regression could be rewritten as,

The logistic regression model with odds.

It is super easy to interpret the model above. For example, with all the other independent variables fixed, a unit increase in x1 increases the log-odds of success by β1.

However, the interpretation of the probit model is not straightforward. The probit link function calculates the Z-score of the probability. Therefore, the interpretation of the same example should be that with all the other independent variables fixed, a unit increase in x1 increases the Z-score by β1.

If the playoff bet website says that “the Z-score that Lakers wins the semi-final series is 1.65!”, few basketball fans will praise the website runner for his/her profession in statistics.

Predictors fixed and outcome observed? Not the case.

In most practical situations, the data comes from retrospective sampling instead of prospective sampling. In retrospective studies, the outcome is fixed and the predictors are observed and collected, however, the predictors are fixed and the outcome is observed in prospective studies.

For example, if we are interested in the relationship between dog cancer and eating dog food only, we first collect 10 different breeds of dogs and feed half of the dogs with dog food only and the other half with mixed food. And then we track the health status of the dogs in five years from now. This type of study is a prospective study and it is very ideal but slow.

However, a cheaper and faster way should be that we find a number of dogs with cancer, we check the record of feeding in the past five years, and we compare them with a group of healthy dogs in our binomial model. This type of study is a retrospective study.

Assume α1 is the probability that a dog is included in the retrospective study if they do not have cancer, while α0 is the probability of inclusion if the dogs have cancer. We can always assume α1 = α0 in the prospective study because we have not seen the outcome yet.

But it’s not the case in the retrospective studies where α1 is usually much larger than α0.

Let’s use the Bayes theorem to describe the relationship between the conditional probability (p*)that a dog has cancer given that it is included in the study and the unconditional probability (p) that the dog has cancer.

The conditional probability that a dog has cancer given that it is included in the study

After applying the logit link function, we have

logit function of the conditional probability

This equation above shows clearly that the difference between the prospective and retrospective study is log(α1/α0) which only affects the intercept term of the binomial model.

The advantage of the logit link function is that we don’t need to worry about the intercept term even in a retrospective study because we are focused on the relative effect of the independent variables on the odds.

In other words, even though it is impossible to know log(α1/α0) (or adjustment on β0) in a retrospective study, it won’t disable us from estimating the logistic model’s coefficients (the other βs). This feature is not valid for the probit link function.

Summary

Logistic regression is one of the binomial regression models, and it uses logit as its link function. It is preferred over the other link functions because of its easy interpretation and usefulness in the retrospective study.

Bonus

Are the other link functions useless? No, try the Challenger Disaster data using the code below, and you will find the cloglog model gains the smallest AIC among the binomial models with the link functions.

# R code

library(faraway)

data(orings)

logit_m = glm(cbind(damage, 6-damage) ~ temp, family=binomial(link = logit), orings)

probit_m = glm(cbind(damage, 6-damage) ~ temp, family=binomial(link=probit), orings)

clog_m = glm(cbind(damage, 6-damage) ~ temp, family=binomial(link=cloglog), orings) summary(logit_m)

summary(probit_m)

summary(clog_m)

I hope you enjoy reading this article.