Maximum Likelihood Estimation and Logistic Regression From Scratch
By: Idriss Jairi
Tags: Logistic Regression, Logistic Regression Mle, Sigmoid Function, Maximum Likelihood, Machine Learning

Logistic regression is a generalized linear model that we can use to model or predict categorical outcome variables. For example, we might use logistic regression to predict whether a tumor is malignant or benign, but probably not to predict the price of someone’s house.

How are we going to do so? Well, In Logistic Regression, we’re essentially trying to find the weights(coefficients) that maximize the likelihood of producing our given data and use them to categorize the response variable

Since the likelihood maximization in logistic regression doesn’t have a closed-form solution, I’ll solve the optimization problem with gradient ascent. Gradient ascent is the same as gradient descent, except I’m maximizing instead of minimizing a function. Before I do any of that, though, I need some data.

Link Function — Sigmoid Function:

Generalized linear models usually transform a linear model of the predictors by using a link function. In logistic regression, the link function is the Sigmoid function.

Sigmoid Function

We can implement this really easily using python:

Maximizing the Likelihood:

In order to choose values for the parameters of logistic regression, we use maximum likelihood estimation (MLE). As such we are going to have two steps: (1) Write the log-likelihood function and (2) find the values of θ that maximize the log-likelihood function. The labels that we are predicting are binary, and the output of our logistic regression function is supposed to be the probability that the label is one.

Likelihood of Bernoulli

Calculating the log-likelihood:

Log-Likelihood

where y is the target class (0 or 1), xi is an individual data point, and β is the weights vector.

I can easily turn that into a python function.

Recall that in MLE the only remaining step is to choose parameters (θ or β) that maximize log-likelihood.

The gradient of Log-Likelihood:

Now that we have a function for log-likelihood, we simply need to choose the values of theta that maximize it. Unfortunately, if we try just setting the derivative equal to zero, we’ll quickly get frustrated: there’s no closed form for the maximum. However, we can find the best values of theta by using an optimization algorithm. The optimization algorithm we will use requires the partial derivative of log-likelihood with respect to each parameter.

The Gradient of Log-Likelihood

do not forget that:

Sigmoid Function

Like the other equation, this is really easy to implement. It’s so simple I don’t even need to wrap it into a function.

Gradient Ascent Optimization:

Our goal is to choose parameters (θ) that maximize the likelihood, and we know the partial derivative of log-likelihood with respect to each parameter. We are ready for our optimization algorithm. In the case of logistic regression, we can’t solve for θ mathematically. Instead, we use a computer to chose θ. To do so we employ an algorithm called gradient ascent (a classic in optimization theory). The idea behind gradient ascent is that gradients point “uphill”. If you continuously take small steps in the direction of your gradient, you will eventually make it to a local maximum. In the case of logistic regression, you can prove that the result will always be a global maximum. The update to our parameters that results in each small step can be calculated as:

Updating Our Weights (Parameters)

Building the Logistic Regression Function

Source Code & Dataset:

https://github.com/jairiidriss/MaximumLikelihoodEstimation-LogisticRegression

References:

https://beckernick.github.io/logistic-regression-from-scratch/

http://www.cs.cmu.edu/~guestrin/Class/10701-S05/slides/LogRegress-1-24-05.pdf