The math behind Logistic regression
By: Keith Satuku
Tags: Sigmoid Function, Classification Algorithms, Logistic Regression

The math behind Logistic regression

Let’s start with what Logistic Regression is not. A regression algorithm is supposed to predict a set of continuous values. This is not a regression algorithm. Logistic regression is mainly a binary classification algorithm which can be extended to classify more groups.

Assuming one is familiar with the Linear Regression formula, Logistic Regression builds on that with the assumption that the probability of Y given X can be approximated by applying a sigmoid function to a linear combination of input features.

A sigmoid function is brilliant at taking inputs from negative infinity to positive infinity then convert them into output values between zero and one. That plus the obvious fact that a sigmoid function can easily be derived makes it a good choice for classification purposes.

Figure 1 A linear combination of inputs and weights

Figure 2 the input of the sigmoid function is the output of a linear combination of features

After applying the sigmoid function to our inputs we can then use the Probability Mass function of a Bernoulli to the resulting value that was between zero and one. After trying y equal to zero and y equal to one we can pick the one with the highest probability.

There should be a default threshold such that if the value from the equation below is less than that the algorithm picks zero, anything above that then the result will be one.

This threshold value is normally set to 0.5 by default but a machine learning engineer or anyone applying this algorithm can customize this value. Now that we know the probability mass function, we can write the likelihood of all the entries as:

And if we take the log of this function, we get the log likelihood for logistic regression. The log

Likelihood equation is:

Now that we have the Log-likelihood function, we simply need to choose the values of theta that maximizes the function. Unfortunately, if we try just setting the derivative equal to zero, we’ll quickly get frustrated: there’s no closed form for the maximum. However, we can find the best values of theta by using an optimization algorithm.

When training our model, our goal is to choose the right weights (a set of thetas) that maximize likelihood equation. Similar to Linear regression, we do not solve this analytically. Instead we use iterative methods to choose the optimum set of theta.

To do so we employ an algorithm called gradient ascent. The idea behind gradient ascent is that gradients point “uphill”. If you continuously take small steps in the direction of your gradient, you will eventually make it to a local maximum.

In the case of logistic regression you can prove that the result will always be a global maximum. The update to our parameters that results in each small step can be calculated as:

The second equation is simply eta, our small learning rate (typically less than 0.1) needed to avoid overshooting past the global maximum) and the difference inside our summation formula is simply the difference between the algorithm’s estimate and the actual y value.

To show that this numerical method results in the maximum log-likelihood we can derive the summation factor glued to eta above.

Taking the derivative of the sum of terms we get:

From there is we take the derivative of the log parts and apply the laws of logarithms we get:

Using chain rule and deriving the sigmoid function f(x) we get:

Then combining the quotient terms in the first bracket onto one common denominator we get:

After that we can cancel like terms and simplify to: