Implementation of Logistic Regression without using Built-In Library
By: Adesh Dalvi
Tags: Machine Learning, Python, Classification Algorithms, Data Science, Logistic Regression

Implementation from scratch

Gradient descent is an optimization technique where we minimize the error by changing the values of coefficients repeatedly. To understand how logistic regression works, we will use gradient descent.

For this implementation, we are going to use the Breast cancer data set. By importing dataset module of sklearn library we can easily use this data set. It is a binary classification data set.

Now, let me run you through the implementation details — step by step.

First, we will import the numpy library for data manipulation. We will use dataset module of sklearn library to import data. Train Test Split module of sklearn library will be used for splitting the data into training and testing data. As well as we will use matplotlib for visualization.

Here is the github link to the implementation code in python.

Fig 4. Importing Libraries and splitting data

We will store the independent variables in x and dependent/ output variable in y. Using train test split module of sklearn we will split our data.

The logistic sigmoid function is,

Fig 5. Logistic sigmoid function

As I already mentioned, t is an equation consists of variables (Attributes) and coefficients. Our main aim is to find the coefficients of the equation in order to obtain good classification results.

Initially, we do not know the values of the coefficients. So, we will assign zero values for all weights and bias and small value for learning rate. Number of weights is equal to the number of independent variables. i.e. in the beginning, all coefficient value for the equation will be zero (b0, b1, …, bn =0).

Fig 6. Initializing weights and bias to zero

As, values of all coefficient is 0 thus,

Hence,

Thus, irrespective of the value of x, we will receive 0.5 as an output, and we will get a straight line as s curve.

We have defined a sigmoid function, where we can map our output into real numbers of range 0 to 1.

Fig 7. Defining Sigmoid function

Now, we have to change the values for coefficients so, we can get a better model with a higher accuracy. After finding the better values we will create a linear model using those values. We will pass this model to sigmoid function which will give us the predicted value output variable. By defining a threshold, we can easily do the classification.

Let’s assume, by using gradient descent we got new values for coefficients. Using these values we will create a model (An equation). We will pass this model to the sigmoid function to get predicted sigmoid values, which are between 0 and 1. By comparing these values to threshold we will perform the classification.

To change the values of the coefficient, we will take derivative with respect to weights and bias to change the values of the coefficients. Then we will update the weights and bias by multiplying the negative learning rate to the derivatives we got w.r.t weights and bias.

Fig 8. Loop to update the Weights and Bias

The above code shows the loop used for gradient descent to change the coefficient values.

Selecting range for for loop is an important choice you should do with this model. Bigger range for for loop will mean more number of iterations and may give better accuracy, but the time complexity will increase. In the other hand, smaller range may give poor fit as the final error achieved may not be near the point of minima. Also, using bigger range we might over-fit the model which results in less accuracy for test data.

There are many stopping criteria for gradient descent like initializing the number of iteration and stop when change in prediction error is less than epsilon etc. I will try to explain detail gradient descent in another blog. Here we are defining the range for loop as stopping criteria.

Fig 9. Creating object and fitting the model

We will initialize learning rate as 0.0001 and number of iteration for loop is 1000. Learning rate is a small value between 0 and 1 which controls the speed of change of weights. Bigger value of learning rate indicates rapid changes and fewer epochs but, it might skip the minima.

After running the for loop, we will get some values of the coefficients and bias used in the model which will give better results.

Fig 10. Updated Weights and Bias

This model with new coefficient and bias values will be used for testing data to see the accuracy of the model.

Fig 11. Defining Predict function

Predict function takes input data x and will give the predicted value of y. This function will first create a linear model using new weights and bias. Then we will pass this model to sigmoid function which will return the y predicted values in the range of 0 to 1.

We will set our threshold as 0.5. We will predict the label as 1 if the sigmoid function gives value greater than 0.5 else, we will predict it as 0. We will store these labels into y test predicted.

Fig 12. Accuracy Function

Accuracy function will give the accuracy of the model. It will take actual and predicted values and return the accuracy of the model.

This model gives more than 92% accuracy with 1000 iterations. The accuracy of your model depends on many factors like threshold, Number of iteration, and type of data (right skewed, left skewed etc.). Different values of threshold or range for loop may give a better output. One way to find value of threshold and number of iterations is the Elbow method. I have plotted the number of iterations and the respective accuracy. We can select an elbow point from which accuracy decreases or stops changing significantly.

I have tried implementing this model using different values for number of iteration. I have received maximum accuracy i.e. 92.98% with 1000 iteration. For 2000 iteration, accuracy for model is 92.10%. Here, we can see the decrease in the accuracy even after increasing the number of iterations beyond 1000.

At 500 iterations, the accuracy obtained was ~91.23% . Hence depending on our accuracy goal, 500 iterations can also be a reasonable choice as a further increase by putting in twice the number of iterations only gives ~0.9% increase in accuracy.

Fig 13. Number of iteration and respective Accuracy

Fig 14. Plot of number of iteration and Accuracy

In this way, we can implement the logistic regression without using built-in libraries.