Multiclass logistic regression from scratch
By: Sophia Yang
Tags: Logistic Regression, Data Science, Softmax, Python, Editors Pick

Multiclass logistic regression from scratch

Photo by Amy Shamblen on Unsplash

Multiclass logistic regression is also called multinomial logistic regression and softmax regression. It is used when we want to predict more than 2 classes. A lot of people use multiclass logistic regression all the time, but don’t really know how it works. So, I am going to walk you through how the math works and implement it using gradient descent from scratch in Python.

Disclaimer: there are various notations on this topic. I am using the notation that I think is easy to understand and visualize. You may find other notations in other places such as matrices and vectors being transposed.

Problem statement

Image by author

Let’s assume we have N people/observations, each person has M features, and they belong to C classes. We are given:

A matrix 𝑋 is ℝ𝑁×𝑀. 𝑋𝑖𝑗 represents person i with feature j.

A vector 𝑌 is ℝ𝑁. 𝑌𝑖represents person i belonging to class k.

We do not know:

The weight matrix 𝑊 is ℝ𝑀×𝐶.𝑊𝑗𝑘 represents the weights for feature j and class k.

We want to figure out 𝑊 and use 𝑊 to predict the class membership of any given observation X.

Multiclass logistic regression workflow

If we know 𝑋 and 𝑊 (let’s say we give 𝑊 initial values of all 0s for example), Figure 1 shows the workflow of the multiclass logistic regression forward path.

First, we calculate the product of 𝑋 and 𝑊, here we let 𝑍=−𝑋𝑊.

Sometimes people don’t include a negative sign here. It doesn’t matter if there is a negative sign here or not.

Sometimes we would also add a bias term. For simplicity, let’s only look at the weights in this article.

Second, we take the softmax for each row 𝑍𝑖: 𝑃𝑖=softmax(𝑍𝑖)=𝑒𝑥𝑝(𝑍𝑖)/∑𝑒𝑥𝑝(𝑍𝑖𝑘). Each row of 𝑍𝑖 should be the product of each row of 𝑋(i.e., 𝑋𝑖) and the entire matrix of 𝑊. Now each row of 𝑃 should add up to 1.

Third, we take the argmax for each row and find the class with the highest probability.

Figure 1. Multiclass logistic regression forward path (Image by author)

Figure 2 shows another view of the multiclass logistic regression forward path when we only look at one observation at a time:

First, we calculate the product of 𝑋𝑖 and W, here we let 𝑍𝑖=−𝑋𝑖𝑊.

Second, we take the softmax for this row 𝑍𝑖: 𝑃𝑖=softmax(𝑍𝑖)=𝑒𝑥𝑝(𝑍𝑖)/∑𝑒𝑥𝑝(𝑍𝑖𝑘).

Third, we take the argmax for this row 𝑃𝑖 and find the index with the highest probability as 𝑌𝑖.

Figure 2. Operation on one row. (Image by author)

Likelihood

Recall that in the problem statement that we said we are given 𝑌. So for a given observation, we know the class of this observation, which is 𝑌𝑖. The likelihood function of 𝑌𝑖 given 𝑋𝑖 and 𝑊 is the probability of observation i and class 𝑘=𝑌𝑖, which is the softmax of 𝑍𝑖,𝑘=𝑌𝑖. And the likelihood function of 𝑌 given 𝑋 and 𝑊 is the product of all the observations. Figure 3 helps us understand this process from 𝑌𝑖 trace backward to 𝑊𝑘=𝑌𝑖.

Figure 3. Calculate likelihood. (Image by author)

Loss function

Next, we calculate the loss function. We use the negative log-likelihood function and normalized it by the sample size. One thing to note here is that

𝑇r means the sum of elements on the main diagonal. Figure 3 visualizes this calculation.

Figure 4. Matrix calculations. (Image by author)

We often add an 𝑙2 regularization term to the loss function and try to minimize the combined function. In fact, the default of scikit-learn uses 𝑙2 penalities. 𝑙1 regularization is also very commonly used. Here we use the 𝑙2 regularization.

Gradient Descent Implementation

Now we have calculated the loss function and the gradient function. We can implement the loss and gradient functions in Python, and implement a very basic gradient descent algorithm.

We try our model on the iris dataset. We fit the model and then plot the loss against the steps, we see that our loss function goes down over time. When we look at the prediction of our data, we see that the algorithm predicts most of the classes correctly.

This is a basic overview of the math and gradient descent of multiclass logistic regression. Hope you find this article helpful.

By Sophia Yang on April 18, 2021