A Complete Logistic Regression Algorithm From Scratch in Python: Step by Step | by Rashida Nasrin Sucky | Towards Data Science

Logistic regression is a popular method since the last century. It establishes the relationship between a categorical variable and one or more independent variables. This relationship is used in machine learning to predict the outcome of a categorical variable. It is widely used in many different fields such as the medical field, trading and business, technology, and many more.
This article explains the process of developing a binary classification algorithm and implements it on Kaggle’s Heart disease dataset.
In this article, we will use a dataset from Kaggle that contains the health data of a population. It has a column at the end that contains if a person has heart disease or not. Our goal is to see if we can predict if a person has heart disease or not using the other columns in the table.
Here I will load the dataset. I will use pandas for that:
import pandas as pdimport numpy as npdf = pd.read_csv('Heart.csv')df.head()
The dataset looks like this:
Look at the last column of the dataset. It is ‘AHD’. This denotes heart disease. We will use the rest of the columns to predict heart disease. So in the future, if we have all the data, we will be able to predict if a person has heart disease without a medical checkup.
Our output will be 0 or 1. If a person has heart disease our algorithm will return 1 and if a person does not have heart disease the algorithm should return 0.
Remember the linear regression formula. That very basic formula of a straight line:
Y= A+BX
Where A is the intercept and B is the slope. If we avoid the ‘intercept’ A in this equation, the formula becomes:
Y = BX
Traditionally, in machine learning, it is pressed as,
Here, ‘h’ is the hypothesis or the predicted value and X is the predictor or input variable. Theta is initialized randomly in the beginning and updated it later.
For the logistic regression, we need to transform this simple hypothesis using a sigmoid function that returns a value from 0 to 1. A sigmoid function can be called a logistic function as well. Logistic regression uses the sigmoid function to predict the output. Here is the sigmoid activation function:
z is the input features multiplied by a randomly initialized term theta.
Here, X is the input feature and theta is the randomly initialized values that will be updated in this algorithm.
The reason we need to use a logistic function is, the curve of a logistic function looks like this:
As you can see from the picture above, it returns a value between 0 to 1. So, it is very helpful for classification. As we will work on a binary classification today,
we will return a zero if the logistic function returns a value that is less than 0.5 and we will return 1 if the logistic function returns a value greater than or equal to 0.5
Cost Function
The cost function gives you the measure of how far the predicted output(calculated hypothesis ‘h’) is from the original output(‘AHD’ column from the dataset).
Before diving into the cost function of logistic regression, I want to remind you of the cost function of a linear regression which was much more straightforward. The cost of linear regression is:
Where,
y is the original label (‘AHD’ column of the dataset)
The average cost function is:
Where,
m is the number of training data
The equation above takes the difference between the predicted label ‘h’ and the original label ‘y’ first. The formula includes squared to avoid any negative values and uses 1/2 to optimize that squared.
This simple and straightforward equation works for linear regression because linear regression uses a simple linear equation: (Y= A+BX).
But logistic regression uses a sigmoid function which is not linear.
We can not use that simple cost function here because it will not converge to global minima. To address this issue, we will use a log to regularize the cost function so that it converges to a global minimum.
Here is the cost function we will use to guarantee a global minimum:
if y = 1,
Cost(h, y) = -log(h)
if y = 0,
Cost(h, y) = -log(1 — h)
Simplified and combined cost function:
Here is the cost function expression:
Why this equation? Look, we can have only two cases y = 0 or 1. And in the cost function equation above, we have two terms:
y*logh and(1-y)*log(1-h).
y*logh and
(1-y)*log(1-h).
If y = 0, the first term becomes zero, and the second term becomes log(1-h). In the equation, we already put a negative sign in the beginning.
If y = 1, the second term becomes zero and only ylogh term remains, with the negative sign in the beginning.
I hope it makes sense now!
Gradient Descent
We need to update our randomly initialized theta values. Gradient descent equation does just that. If we take a partial differential of the cost function with respect to theta:
Using this expression above the gradient descent formula becomes:
Here, alpha is the learning rate.
Using this equation theta values will be updated in each iteration. It will be more clear to you when you will implement thin in python.
This is the time to use all the equations above to develop the algorithm
Step 1: Develop the hypothesis.
The hypothesis is simply the implementation of the sigmoid function.
def hypothesis(X, theta):    z = np.dot(theta, X.T)    return 1/(1+np.exp(-(z))) - 0.0000001
I deducted 0.0000001 from the output here because of this expression in the cost function:
If the outcome of the hypothesis expression comes out to be 1, then this expression will turn out to be the log of zero. To mitigate that, I used this very small number at the end.
Step 2: Determine the cost function.
def cost(X, y, theta):    y1 = hypothesis(X, theta)    return -(1/len(X)) * np.sum(y*np.log(y1) + (1-y)*np.log(1-y1))
This is just a straightforward implementation of the cost function equation above.
Step 3: Update the theta values.
Theta values need to keep updating until the cost function reaches its minimum. We should get our final theta values and the cost of each iteration as output.
def gradient_descent(X, y, theta, alpha, epochs):    m =len(X)    J = [cost(X, y, theta)]     for i in range(0, epochs):        h = hypothesis(X, theta)        for i in range(0, len(X.columns)):            theta[i] -= (alpha/m) * np.sum((h-y)*X.iloc[:, i])        J.append(cost(X, y, theta))    return J, theta
Step 4: Calculate the final prediction and accuracy
Use the theta values that come out of the ‘gradient_descent’ function and calculate the final prediction using the sigmoid function. Then, calculate the accuracy.
def predict(X, y, theta, alpha, epochs):    J, th = gradient_descent(X, y, theta, alpha, epochs)     h = hypothesis(X, theta)    for i in range(len(h)):        h[i]=1 if h[i]>=0.5 else 0    y = list(y)    acc = np.sum([y[i] == h[i] for i in range(len(y))])/len(y)    return J, acc
The final output is the list of costs in each epoch and the accuracy. Let’s implement this model to solve a real problem.
I already showed the dataset in the beginning. But for your convenience I am including it again here:
Notice, there are a few categorical features in the dataset. We need to convert them to numerical data.
df["ChestPainx"]= df.ChestPain.replace({"typical": 1, "asymptomatic": 2, "nonanginal": 3, "nontypical": 4})df["Thalx"] = df.Thal.replace({"fixed": 1, "normal":2, "reversable":3})df["AHD"] = df.AHD.replace({"Yes": 1, "No":0})
Add one extra column for the bias. This should be a column of ones because any real number remains unchanged if multiplied by one.
df = pd.concat([pd.Series(1, index = df.index, name = '00'), df], axis=1)
Define the input features and output variables. The output column is the categorical column that we want to predict. The input features will be all the columns except the categorical columns that we modified earlier.
X = df.drop(columns=["Unnamed: 0", "ChestPain", "Thal"])y= df["AHD"]
Finally, initialize the theta values in a list and predict the result and calculate the accuracy. Here I am initializing the theta values like 0.5. It can be initialized as for any other value. As each feature should have a corresponding theta value, one theta value should be initialized for each feature in the X, including the bias column.
theta = [0.5]*len(X.columns)J, acc = predict(X, y, theta, 0.0001, 25000)
The final accuracy is 84.85%. I used 0.0001 as the learning rate and 25000 iterations.
I ran this algorithm a few times to determine that.
Please check my GitHub link for this project provided below.
‘predict’ function also returns the list of costs in each iteration. Cost should keep going down in each iteration in a good algorithm. Plot the cost for each iteration to visualize the trend.
%matplotlib inlineimport matplotlib.pyplot as pltplt.figure(figsize = (12, 8))plt.scatter(range(0, len(J)), J)plt.show()
Cost decreased very rapidly in the beginning and then the rate of decrease slowed down. This is the behavior of a perfect cost function!
I hope, this was helpful. If you are a beginner, it will be a bit hard to grasp all the concepts in the beginning. But I suggest please run all the code by yourself in a notebook and observe the output carefully. That will be immensely helpful. This type of logistic regression is helpful for a lot of real-world problems. I hope, you will be using this to develop some cool projects!
If please let me know in the comment section if you have a problem running any piece of code.
Here you will find the complete code: