Multinomial Logistic Regression In a Nutshell
By: Wilson Xie
Tags: Neural Networks, Python, Logistic Regression, Data Science, Machine Learning

Now that we understand multinomial logistic regression, let’s apply our knowledge. We’ll be building the MLR by following the MLR in the graph above (Figure 1).

Data

Our data will be the Fashion MNIST dataset from Kaggle. The dataset is stored as a DataFrame with 60,000 rows, where each row represents an image. The DataFrame also has 785 columns, where the first column of the DataFrame represents the label of the image (Figure 3.2). The rest of the 784 columns contain the RGB-values for the pixels of each training image (Figure 3.1). Each image pixel number ranges from 0 to 255 based on its RGB value.

Figure 3.1. Sample image of a shirt from the training set.

Figure 3.2. Labels

Task:

Split the DataFrame into DataFrame X and DataFrame Y

Convert DataFrame X to an array

One-hot encoding Y values and convert DataFrame Y to an array

We are using one-hot encoder to transform the original Y values into one-hot encoded Y values because our predicted values are probabilities. I will explain this later in the next step.

Figure 4. Given X- and Y-values and desired X- and Y-values

Score & Softmax

Task:

Compute the score values

Define an activation function

Run the activation function to compute errors

Looking at Figure 1, the next step would be computing the dot product between the vectors containing features and weights. Our original weight vector will be an array of 0s because we do not have any better values. Don’t worry, our weight will be constantly updating as the loss function is minimized. The dot product is called the score. This score is the deciding factor that predicts whether our image is a T-shirt/top, a dress or a coat.

Figure 5. Softmax function. Photo credit to Wiki Commons

Before we utilize the score to predict the label, we have 2 problems. Remember that we one-hot encode our scores because our predicted values are probabilities? Our current scores are not probability values because they contain negative values and thus do not range from 0 to 1. So, we need to implement the Softmax function in order to normalize the scores. This exponent normalization function would convert our scores into positives and turn them into probabilities (Figure 5).

In an array of probability values for each possible result, the argmax of the probability values is the Y value. For example, in an array of 10 probabilities, if the 5th element has the highest probability, then the image label is a coat, since the 5th element in the Y values is the coat (Figure 3.1).

Figure 6. Score and Softmax functions in Python

Gradient Descent & Loss Function

Task:

Define a gradient function

Define a loss function

Optimize the loss function

After the Softmax function computes the probability values in the initial iteration, it is not guaranteed that the argmax matches the correct Y value. We need to iterate multiple times until we are confident about our argmax. To validate our argmax, we need to set up a loss function. We will use cross-entropy loss.

Figure 7. Cross-entropy loss in Python

The way to maximize the correctness is to minimize the loss in cross entropy function. To do that, we will apply gradient descent. Specifically, we will use stochastic ٖgradient descent. Stochastic gradient descent is no different than regular gradient descent. The term “stochastic” means random, meaning the gradient descent will be done by randomly selecting a sample of features. Then, instead of taking the gradient of all features, we are only calculating the gradient for the sample features. The purpose of stochastic gradient descent is to decrease the number of iterations and save time.

Figure 8. Gradient descent and stochastic gradient descent formulas

In order to achieve randomness, we will disrupt the order inside the X array (permutation). Each time we sample an image from the X array, we’ll compute the stochastic gradient descent and update the weight. Then the updated weight will be used to find the minimum value of the loss function. The number of times the minimum is found in the loss function is known as the number of epochs. Typically, more epochs would lead to better results since there is more training involved. However, too many epochs would lead to overfitting. Choosing a good epoch depends on the loss values, there is an article that talks about how to choose a good number of epochs here.

Train & Test

Task:

Define a training set and a test set

Train our samples

Visualize our loss values

Now that we have optimized the loss function, let’s test our model on our data. Our training sample has 60,000 images, we will split 80% of the images as the train set and the other 20% as the test set based on the Pareto principle. While we fit the model, let’s keep track of the loss values to see if the model is working correctly.

Figure 9. Losses after iterations

We can clearly see that the value of the loss function is decreasing substantially at first, and that’s because the predicted probabilities are nowhere close to the target value. That means our loss values are far from the minimum. As we are getting close to the minimum, the error is getting smaller because the predicted probabilities are getting more and more accurate.

Accuracy

After fitting the model on the training set, let’s see the result for our test set predictions.

Figure 10. Prediction result

It looks like our accuracy is about 85% (Figure 11), which is not so bad.

Figure 11. Accuracy Score

Challenge

Now that we have our initial predictions, see if you can improve the accuracy by adjusting the parameters in the model or adding more features. Tuning parameters like the learning rate and epoch is something to start with. I’ve attached the code and datasets for you to play around with. Enjoy!