Multiclass Classification Using Logistic Regression from Scratch in Python: Step by Step Guide | by Rashida Nasrin Sucky | Towards Data Science

Logistic regression is a very popular machine learning technique. We use logistic regression when the dependent variable is categorical. This article will focus on the implementation of logistic regression for multiclass classification problems. I am assuming that you already know how to implement a binary classification with Logistic Regression.
If you haven’t worked on binary classification with logistic regression yet, I suggest, please go through this article before you dive into this one.
Because multiclass classification is built on the binary classification.
You will learn the concepts, formulas, and a working example of binary classification in this article:
towardsdatascience.com
The implementation of Multiclass classification follows the same ideas as the binary classification. As you know in binary classification, we solve a yes or no problem. Like in the example in the above-mentioned article, the output answered the question if a person has heart disease or not. We had only two classes: heart disease and no heart disease.
If the output is 1, the person has heart disease, and if the output is 0 the person does not have heart disease.
In multi-class classification, we have more than two classes. Here is an example. Say, we have different features and characteristics of cars, trucks, bikes, and boats as input features. Our job is to predict the label(car, truck, bike, or boat).
How to solve this?
We will treat each class as a binary classification problem the way we solved a heart disease or no heart disease problem.
This approach is called the one vs all method
In the one vs all method, when we work with a class, that class is denoted by 1 and the rest of the classes becomes 0.
For example, if we have four classes: cars, trucks, bikes, and boats. When we will work on the car, we will use the car as 1 and the rest of the classes as zeros. Again, when we will work on the truck, the element of the truck will be one, and the rest of the classes will be zeros.
It will be more understandable when you will implement it. I suggest, you keep coding and running the codes as you read.
Here I will implement this algorithm in two different ways:
The gradient descent approach.
The gradient descent approach.
2. The optimization function approach.
Important equations and how it works:
Logistic regression uses a sigmoid function to predict the output. The sigmoid function returns a value from 0 to 1. Generally, we take a threshold such as 0.5. If the sigmoid function returns a value greater than or equal to 0.5, we take it as 1, and if the sigmoid function returns a value less than 0.5, we take it as 0.
z is the input features multiplied by a randomly initialized value denoted as theta.
Here, X is the input feature. In most cases, there are several input features. So, this formula becomes big:
X1, X2, X3 are input features and one theta will be randomly initialized for each input feature. Theta0 in the beginning is the bias term.
The goal of this algorithm will be to update this theta with each iteration so that it can establish a relationship between the input features and the output label.
Cost Function and Gradient Descent
The cost function gives the idea that how far is our prediction from the original output. Here is the formula for that:
Here,
m is the number of training examples or the number of training data,
y is the original output label,
h is the hypothesis or the predicted output.
This is the equation for the gradient descent. Using this formula, we will update the theta values in each iteration:
Prerequisites:
a. You need to be able to read and write python code comfortably.
b. Basic Numpy and Pandas libraries.
Here I am going to show the implementation step by step.
Import the necessary packages and the dataset. I took the dataset from Andrew Ng’s Machine Learning course in Coursera. This is a handwriting recognition dataset. There are digits from 1 to 10.
Import the necessary packages and the dataset. I took the dataset from Andrew Ng’s Machine Learning course in Coursera. This is a handwriting recognition dataset. There are digits from 1 to 10.
From the dataset of pixels, we need to recognize the digits. In this dataset input variables and output-variables are organized in different sheets in an Excel file. Please feel free to download the dataset from the link at the end of this page.
Please run each piece of code if you are reading this to learn this algorithm.
Let’s import the necessary packages and the dataset,
import pandas as pdimport numpy as npxl = pd.ExcelFile('ex3d1.xlsx')df = pd.read_excel(xl, 'X', header=None)
2. Import y, which is the output variable
y = pd.read_excel(xl, 'y', header = None)
3. Define the hypothesis that takes the input variables and theta. It returns the calculated output variable.
def hypothesis(theta, X):    return 1 / (1 + np.exp(-(np.dot(theta, X.T)))) - 0.0000001
4. Build the cost function that takes the input variables, output variable, and theta. It returns the cost of the hypothesis. That means it gives the idea about how far the prediction is from the original outputs.
def cost(X, y, theta):    y1 = hypothesis(X, theta)    return -(1/len(X)) * np.sum(y*np.log(y1) + (1-y)*np.log(1-y1))
5. Now, it’s time for data preprocessing.
The data is clean. Not much preprocessing is required. We need to add a bias column in the input variables. Please check the length of df and y. If the length is different, the model will not work.
print(len(df))print(len(y))X = pd.concat([pd.Series(1, index=df.index, name='00'), df], axis=1)
6. y column has the digits from 1 to 10. That means we have 10 classes.
y is a DataFrame which is not necessary. I will only keep the column as a Series that contains the value.
y = y.iloc[:, 0]
We will make one column for each of the classes with the same length as y. When the class is 5, make a column that has 1 for the rows with 5 and 0 otherwise.
Check, how many classes we have,
y.unique()
output:
array([10,  1,  2,  3,  4,  5,  6,  7,  8,  9], dtype=int64)
So, we have 10 classes. Initiate a DataFrame that has 10 columns and df.shape[0] number of rows.
y1 = np.zeros([df.shape[0], len(y.unique())])y1 = pd.DataFrame(y1)
We will do it programmatically with some simple code:
for i in range(0, len(y.unique())):    for j in range(0, len(y1)):        if y[j] == y.unique()[i]:            y1.iloc[j, i] = 1        else:             y1.iloc[j, i] = 0y1.head()
7. Define the function ‘gradient_descent’ now. This function will take input variables, output variable, theta, alpha, and the number of epochs as the parameter. Here, alpha is the learning rate.
You should choose it as per your requirement. A too small or too big learning rate can make your algorithm slow. I like to run the algorithm for different learning rates and get the idea of the right learning rate. It may take a few iterations to select the right learning rate.
For each of the columns in y1, we will implement a binary classification.
For example, when I am considering the digit 2, it should return 1 for digit 2 and 0 for the rest of the digits. So, as we have 10 classes, we have run each epoch(iteration) 10 times. So, we have a nested for loop here.
def gradient_descent(X, y, theta, alpha, epochs):    m = len(X)    for i in range(0, epochs):        for j in range(0, 10):            theta = pd.DataFrame(theta)            h = hypothesis(theta.iloc[:,j], X)            for k in range(0, theta.shape[0]):                theta.iloc[k, j] -= (alpha/m) * np.sum((h-y.iloc[:, j])*X.iloc[:, k])            theta = pd.DataFrame(theta)    return theta, cost
7. Initialize the theta. Remember, we will implement logistic regression for each class. There will be a series of theta for each class as well.
I am running this for 1500 epochs. I am sure the accuracy rate will be higher with more epochs.
theta = np.zeros([df.shape[1]+1, y1.shape[1]])theta = gradient_descent(X, y1, theta, 0.02, 1500)
8. With this updated theta, calculate the output variable.
output = []for i in range(0, 10):    theta1 = pd.DataFrame(theta)    h = hypothesis(theta1.iloc[:,i], X)    output.append(h)output=pd.DataFrame(output)
9. Compare the calculated output and the original output variable to calculate the accuracy of the model.
accuracy = 0for col in range(0, 10):    for row in range(len(y1)):        if y1.iloc[row, col] == 1 and output.iloc[col, row] >= 0.5:            accuracy += 1accuracy = accuracy/len(X)
The accuracy is 72%. I am sure, accuracy will be better for more epochs. Because it takes so much time, I did not rerun the algorithm.
If you are running this, feel free to try for more epochs and let me know in the comment section, how much accuracy, you have got.
Instead of a gradient descent approach, you can also use an optimization function already built-in for you.
In this approach, you use an optimization function to optimize the theta for the algorithm. It’s a lot faster approach.
1. We are going to use the same datasets as before. Import the dataset with a different name if you are using the same notebook:
xls = pd.ExcelFile('ex3d1.xlsx')df = pd.read_excel(xls, 'X', header=None)
2. We still need to add a column of all ones for the bias term in df.
X = np.c_[np.ones((df.shape[0], 1)), df]
3. Import the data for ‘y’.
y = pd.read_excel(xls, 'y', header=None)
As this is a DataFrame, just take the column zero as a series and make it two -dimensional to match the dimension with the dimension of X.
y = y[0]y = y[:, np.newaxis]
Here, ‘y’ has one column only. Make it 10 columns for 10 classes. Each column will deal with one class. For example, when we will deal with class 10, we will keep 10 in its place and replace the rest of the values with zeros. Here is the function y_change that will take y itself and a class(such as 3). Then it will replace 3 with 1 and 0 with all other classes. This function will be used soon in the later steps.
def y_change(y, cl):    y_pr=[]    for i in range(0, len(y)):        if y[i] == cl:            y_pr.append(1)        else:            y_pr.append(0)    return y_pr
Data preparation is completed. Now develop the model:
4. Define the hypothesis function. This is the same as the previous method.
def hypothesis(X, theta):    z = np.dot(X, theta)    return 1/(1+np.exp(-(z)))
5. Develop the cost function. This one is also the same as the previous method:
def cost_function(theta, X, y):    m = X.shape[0]    y1 = hypothesis(X, theta)    return -(1/len(X)) * np.sum(y*np.log(y1) + (1-y)*np.log(1-y1))
6. Define the gradient. This one is different. This function defines how to update the theta.
def gradient(theta, X, y):    m = X.shape[0]    y1 = hypothesis(X, theta)    return (1/m) * np.dot(X.T, y1 - y)
7. Now, import the optimization function and initialize the theta. I am taking zeros as initial theta values. Any other values should work as well.
from scipy.optimize import minimize, fmin_tnctheta = np.zeros((X.shape[1], 1))
8. Let’s make a fit function that will take X, y, and theta as input. It will use an optimization function and output the optimized theta for us.
It takes these three parameters:
i. A function that needs to be minimized,
ii. A parameter to be optimized and,
iii. Arguments to use for optimization.
In this example, cost function should be minimized and theta needs to be optimized for that. Input and output variables X and y are the arguments to use.
This optimization function takes another parameter, that is the gradient. But this is optional. Here, we have a formula or function for the gradient. So we are passing it.
def fit(X, y, theta):    opt_weigths = fmin_tnc(func=cost_function, x0=theta,                          fprime=gradient, args=(X, y.flatten()))    return opt_weigths[0]
9. Use this fit method to find the optimized theta. We have to optimize the theta for each class separately. Let’s develop a function where for each class, ‘y’ will be modified accordingly using the y_change method in step 3.
def find_param(X, y, theta):    y_uniq = list(set(y.flatten()))    theta_list = []    for i in y_uniq:        y_tr = pd.Series(y_change(y, i))        y_tr = y_tr[:, np.newaxis]        theta1 = fit(X, y, theta)        theta_list.append(theta1)    return theta_list
Use this method to find the final theta
theta_list = find_param(X, y, theta)
10. It’s time to predict the output. We have to predict the classes individually as well.
def predict(theta_list, x, y):    y_uniq = list(set(y.flatten()))    y_hat = [0]*len(y)    for i in range(0, len(y_uniq)):        y_tr = y_change(y, y_uniq[i])        y1 = hypothesis(X, theta_list[i])        for k in range(0, len(y)):            if y_tr[k] == 1 and y1[k] >= 0.5:                y_hat[k] = y_uniq[i]    return y_hat
Use the predict method above and calculate the predicted output y_hat:
y_hat = predict(theta_list, X, y)
11. Calculate the accuracy
accuracy=0for i in range(0, len(y)):    if y_hat[i] == y.flatten()[i]:        accuracy += 1print(accuracy/len(df)*100)
This process gives a 100% accuracy. Now. you decide for yourself, which method of logistic regression you want to use for your projects.
This same problem is solved using a neural network as well in this article that shows how to develop a neural network from scratch:
medium.com
Please ask me if you have any questions in the comment section. Check this GitHub page for the dataset:
github.com
Here is the link for the code of the gradient descent method
github.com
Here is the link for the Github link of the optimization function method: