Why Sigmoid: A Probabilistic Perspective
By: Logan Yang
Tags: Machine Learning, Sigmoid, Video Tutorial, Statistics, Logistic Regression

This question of “why sigmoid” used to bug me for a long time. Many answers online are not to the point. The kind of answers I found most frequently mentioned the keywords “logit” and “log odds” and simply transformed the sigmoid to its inverse, which not only explains nothing about why we chose the log odds as the thing our linear predictor aims for, it also says nothing about the implications such a choice has. Some better ones mentioned “generalized linear models”, but they share the same weakness as introductory classes where concepts are mentioned but the inner connections that really answer the “why” aren’t there. The real answer should help you get to the point where you can design this algorithm from scratch without knowing anything about it beforehand. When you face a binary classification problem with only basic probability and statistics knowledge, you should be able to think “okay, one of the most logical ways to tackle this problem is to follow this exact model design”.

In this post, I will try my best to arrange the flow of logic in an easy to read manner so it becomes clear that the sigmoid is a natural design choice for probabilistic binary classification with some important assumptions. To tell it like a story, the logic is not necessarily nice and linear, some points may appear to be parallel but they all contribute to the design motivation of the logistic model. So if you care about this topic, sit back and bear with me for a moment. This is going to be a long post with an amount of information comparable to an entire chapter in a machine learning book.

Some keywords and topics upfront:

Probability interpretation of linear regression, maximum likelihood estimation

Gaussian discriminant analysis

Latent variable formulation of logistic regression

Gaining insights from an alternative: the probit model

Exponential family, generalized linear models, and canonical link function

1. Probabilistic Interpretation of Linear Regression, Maximum Likelihood Estimation

The reason for mentioning linear regression here is to see how we can look at it as a probabilistic model of the data and whether we can apply similar ideas on classification.

We assume our target variable y and the inputs x are related via (superscript i is the index of the data point)

where epsilon is an error term that captures either unmodeled effects or random noise. We assume the noise comes from different sources and is not correlated, so it should be Gaussian based on Central Limit Theorem. We can write out the distribution and express the error as the difference between the target and the linear predictor,

We call this the distribution of y given x parametrized by θ. We are not conditioning on θ because it’s not a random variable, it is the parameter to learn. Next, we define the likelihood as

The likelihood is a function of θ. When viewed as a function of y and X with a fixed θ, it is just the probability density function. But when viewed as a function of θ, it means that by varying θ we can “fit” a distribution to the data observed. The process of finding that best fit is called maximum likelihood estimation (MLE). In other words, MLE is the attempt to find the distribution that maximizes the probability of observing the data, with the assumption of the type of distribution (in this case a Gaussian) and parameters (in this case, θ, notice we only care about the mean and not the variance/covariance matrix here). We further write it out as a product for individual data points in the following form because we assume independent observations,

Since the log transformation is monotonic, we use the log-likelihood below for the optimization of MLE.

To find the best Gaussian that describes the true underlying model which generates our data, in other words, the best θ, we need to find the peak that gives us the maximum log-likelihood. Maximizing the expression above is equivalent to minimizing the term below,

Now we see the magic: this is exactly least-squares!

In short, why does linear regression fit the data using least-squares?

Because it tries to find the best model in the form of a linear predictor plus a Gaussian noise term that maximizes the probability of drawing our data from it.

The probabilistic formulation of linear regression is not only an inspiring example for our formulation of logistic regression later, but it also shows what a proper justification for model design looks like. We mapped a linear predictor with Gaussian noise to the target variable. For binary classification, it would be nice if we can do something similar, i.e. map a linear predictor with something to the probability of being in one of the two classes (the posterior p(y=1|x)), and use MLE to justify the model design by saying it’s maximizing the probability of drawing the observed data out of our parametrized distribution. I will show how to do that in section 3, but next, let’s look at a motivating example.

2. A Motivating Example: Gaussian Discriminant Analysis

Let’s consider a binary classification task on 1D data where we already know the underlying generative distribution for the two classes: Gaussians with the same variance 1 and different means 3 and 5. Both Gaussians have 50k data points, i.e. equal priors, p(C0) = 0.5 = p(C1). (Ck represents the class of y)

Since we only have 1 dimension in the data, the best we can do is to draw a vertical boundary somewhere that separates the two classes as much as it can. It’s visually obvious that the boundary should be around 4. Using a generative approach where we know the class conditionals p(X|Ck), which are the two Gaussians, and the priors p(Ck), we can use Bayes rule to get the posterior

The result is plotted below

We can clearly see the boundary in the posterior, i.e. the final probability prediction of our algorithm. The red region is classified as class 0, the blue region is class 1. This approach is a generative model called Gaussian Discriminant Analysis (GDA). It models continuous features. You may have heard of its sibling for discrete features: the Naive Bayes classifier.

Now look at the S shape of the posterior around the boundary, it describes the transition of uncertainty between the two classes.

Wouldn’t it be cool if we can model this S shape directly without knowing the class conditionals beforehand?

But how? Let’s work through some math.

Notice that the red and blue curves are symmetric, and they always sum to 1 because they are normalized in Bayes theorem. Let’s look at the red one. It’s simply p(C0|X) which is a function of X. We massage the previous equation a bit by dividing the top and bottom with the top to the following form,

For the bottom right term, we can cancel the priors because they are equal, and plug the Gaussians in for the class conditionals.

Okay, this is nice! We have a linear function of x inside the exp(), if we set z = -2x + 8, write it out for the posterior, it becomes,

This is the logistic sigmoid function! If you ask why we have that negative sign for z, it’s because we want p and z to be monotonic in the same direction for convenience, meaning increasing z will increase p. The inverse of this is called the log odds or logit, which is the part that we can use a linear function to model.

Looking back at the flow of logic above, what really happened that made it possible to have a sigmoid form posterior and a linear function of x for z? That will give us some insights to decide when we can model the classification this way.

For the sigmoid form, you see that it came naturally from the Bayes rule for two classes, i.e. a Bernoulli distribution of the target variable. It doesn’t require the class conditionals to be Gaussians! There can be a family of distributions that have a similar exponential form, fitting into the same derivation we came through above! As long as the outcome y is binary, the input X can have some flexibility in their class conditional distribution.

Next, the linear form of z. In this example, we had two Gaussians with the same variance and prior. These are the facts that let us cancel out the priors and the quadratic term of X in the derivation. This requirement looks quite strict. Indeed, if we change the shape of our Gaussians, the decision boundary can no longer be a straight line. Consider the 2D examples below. If the two Gaussians have the same covariance matrix, the decision boundary is linear; in the second graph they have different covariance matrices, the decision boundary is parabolic.