Logistic Regression from Scratch with NumPy | by Levent Baş | Towards Data Science

Welcome to another post of implementing machine learning algorithms! Today, the algorithm we will be implementing from scratch is Logistic Regression. Alongside its beloved sister algorithm linear regression, this one is highly used in machine learning as well, due to its simplicity and robustness. Even though its called logistic regression, it’s actually a classification algorithm that is used to classify input data into its classes (labels).
This powerful machine learning model can be used to answer some questions such as;
Whether an e-mail is spam or not
If the customer will churn
Whether a tumor is benign or malignant
All of the questions above are simply yes-no questions, therefore they can be used to classify input data into two classes. Hence, the term binary classification is used when the data can be categorized into two distinct classes.
Obviously, multi-class classification deals with data that has more than two labels (classes). After grasping the ins and outs of logistic regression to make binary classification, transition to a multi-class classification is pretty straight-forward, as a consequence, we will deal with data that has two classes only, for now.
Remember that in linear regression we predict numerical values based on the inputs and model parameters. Here, in logistic regression we can also approach the model as we are trying to predict numerical values but this time these values correspond to the probabilities of input data belonging to a particular class.
The term logistic in logistic regression is used because we are applying another function to the weighted sum of input data and parameters of the model and this function is called logit (sigmoid) function.
Sigmoid function always outputs values between 0 and 1, mapping values to a range and thus can be used to calculate probabilities of input data belonging to a certain class:
sigmoid(x) = 1 / (1 + e−x)
Without further ado, let’s start writing the code for this implementation. I will explain the code as I go, whenever deemed necessary.
We start off by importing necessary libraries. As always, NumPy is the only package that we will use in order to implement the logistic regression algorithm.
All the others will only help us with small tasks such as visualizing the data at hand or creating a dataset. Hence, we won’t be using already implemented package solutions for logistic regression.
Here, we write the code for the aforementioned sigmoid (logit) function. It is important to note that this function can be applied to all of the elements of a numpy array individually, simply because we make use of the exponential function from the NumPy package.
Next, we write the cost function for logistic regression. Note that the cost function used in logistic regression is different than the one used in linear regression.
Remember, in linear regression we calculated the weighted sum of input data and parameters and fed that sum to the cost function to calculate the cost. When we plotted the cost function it was seen to be convex, hence a local minimum was also the global minimum.
However, in logistic regression, we apply sigmoid function to the weighted sum which makes the resulting outcome non-linear.
If we fed that non-linear result to the cost function, what we get would be a non-convex function and we wouldn’t be assured to find only one local minimum that is also the global minimum.
As a result, we use another cost function to calculate the cost which is guaranteed to give one local minimum during the optimization.
Gradient descent implementation here is not so different than the one we used in linear regression. Only difference to be noted is the sigmoid function that is applied to the weighted sum, obviously.
While writing out the prediction function, let’s not forget that we are dealing with probabilities here.
Hence, if the resulting value is above 0.50, we round it up to 1, meaning the data sample belongs to the class 1. Consequently, if the probability of a data sample belonging to the class 1 is below 0.50, it simply means that it is part of the other class (class 0).
Remember that this is binary classification, so we have only two classes (class 1 and class 0).
After writing the code for the necessary functions, let’s create our very own dataset with make_classification function from sklearn.datasets. We create 500 sample points with two classes and plot the dataset with the help of seaborn library.
Initial Cost is: [[0.69312718]] Optimal Parameters are:  [[-0.45293068] [ 3.26552327] [ 0.03334871]]
Now, let’s run our algorithm and calculate the parameters of our model.
Seeing the plot, we can now be sure that we have implemented the logistic regression algorithm without a fault, since it decreases with every iteration until the decrease is so minimal that the cost converges to a minimum which is what we want indeed.
0.966
After running the algorithm and getting the optimal parameters, we want to know how successful our model is at predicting the classes of our data. Turns out the accuracy score we have obtained is a pretty high value, so make sure to pat yourself on the back!
Now, for the sake of visualization, let’s plot our dataset along with the decision boundary of our model.
We simply calculate the intercept and slope values using the optimal parameters and plot the boundary that classifies the data into two classes.
We can see from the plot that the classification is not 100% correct since the separation of classes is not linear naturally. Nevertheless, misclassified points (false positives and false negatives) are really few, so we have done a pretty good job with this implementation of logistic regression.
Feel free to go ahead and code all of this once again.
But this time, do it on your own.
You can also check out my GitHub profile to read the code along a jupyter notebook or simply use the code for implementation.
I’ll be back with more implementations in the future, for sure.