Neural Networks for Absolute Beginners with Numpy from scratch — Part 3: Logistic Regression | by Suraj Donthi | Towards Data Science

In both Machine Learning and Deep Learning, you could encounter two kinds of problems which are regression and classification.
In regression problems, you predict a continuous real-valued number while in classification problems, you predict different classes of objects.
In the previous tutorials, we went in-depth into what a perceptron is and later learned how it learns to predict with the use of linear regression. In case, you’re an absolute beginner to this domain I recommend you to go through short and easy previous tutorials of mine.
towardsdatascience.com
towardsdatascience.com
In this tutorial, you will learn about logistic regression that is used for classification problems.
As before a bit of math will be involved, but I’ll ensure to cover from the basics so that you can understand easily.
If you are unsure of which environment to use for implementing the code in this tutorial, I recommend Google Colab. The environment comes with many important packages already installed. Installing new packages and also importing and exporting the data is quite simple. Most of all, it also comes with GPU support. So, go ahead and get coding!
Lastly, I recommend that you go through the first two tutorials before you start this. However, if you already know well about linear regression and a bit about Neural Networks or just want to learn about Logistic Regression, you could start right away!
One of the earliest and most popular activation functions utilized is the Sigmoid function.
The equation of the sigmoid function is as follows:
and the graph for the function is as below:
Hence, a perceptron with a sigmoid activation function does binary classification on a given set of data. This process of binary classification is popularly called Logistic Regression. In the next section, we’ll dive deeper into logistic regression and also understand how the model is trained.
So, what is logistic regression?
Logistic regression is a technique used for binary classification. It creates a decision boundary between data points to categorize them in any one of the two classes. Such an illustration can be seen in the figure below. We shall go deeper into understanding and training a logistic regression model through a hands-on implementation. Understanding logistic regression will provide the base for understanding a Neural Network model.
The computation graph for logistic regression to be implemented is shown in the below figure.
We have two inputs x1 and x2 which are multiplied by the weights w1 and w2 respectively. An additional bias b is added to their sum to obtain z. The parameters (w1, w2, b) are learned during gradient descent.
Now let’s get coding!
The first step would be to import the required packages.
You’ll use the sklearn package to perform two tasks:
Generate datasets of blobsSplit the data into train and test sets.
Generate datasets of blobs
Split the data into train and test sets.
You’ll use matplotlib for visualizing the results.
Before you start defining code, in any machine learning project your first task will be to define the hyperparameters. The hyperparameters can be the dataset size, the learning rate, number of epochs, etc. You must be wondering why these variables are called hyperparameters! You’ll find out later that parameters are what is learned in the model and we use hyperparameter to fine-tune the model for achieving better accuracy.
Here we also define the number of input features and the number of clusters as we need to create the dataset.
Go ahead and define them!
Next, we import the dataset from scikit-learn with the make_blobs function which creates blobs of classes. We will visualize the same too.
Now lets, plot a graph to visualize the data generated.
The two sets of data points belong to the two classes. Our goal is to find an optimal decision boundary that separates these two classes.
The next step will be to split the data into train set and test set. We do this so that we can verify the accuracy of the learned algorithm.
Shape of X_train (400, 2) Shape of y_train (400, 1) Shape of X_test (100, 2) Shape of y_test (100, 1)
You’ll now randomly initialize the parameters W and b, which are learned during training.
Initializing weights... W: [[-0.59134456] [-0.31427067]] b: -1.6078558751252707
After we have randomly initialized the parameters, the next step is to perform forward propagation and see how the network predicts.
You will now utilize the generate_mesh_grid function to plot the dataset along with the contour.
It’s now time to define the loss function!!
Unlike the MSE for Linear Regression, here we use a logistic loss function. This is because when we pass the sum z through a sigmoid activation function, the output is non-linear (as the sigmoid function is non-linear). This results in an error that is non-convex.
But what do you mean by non-convex?
In unsophisticated terms, the non-convex function appears to be like the graph on the right.
In such situations, while our goal is to find the minimal point, it becomes extremely difficult to find one because of the troughs and crests. Instead, we could make our problem simpler if we could just have a simple function (like the one on the left) where we can easily find the minimum point. This is the convex function on the left.
Hence, we use a log function that actually converts the non-linear function back to a linear one, which in turn results in a convex function!
Loss can be defined as:
The graph of ᅳlog(x) is shown in the above figure. You can infer that given y = 1 if y → 0, then loss → 0 and when y’ → 0, then loss → ∞. Similarly, if y’ → 0, then loss → 0 and when y’ → 1, then loss → ∞ given y = 0. It implies that when the prediction is wrong, the parameters are heavily penalized and when the prediction is right, the parameters are not penalized.
Here, the function reduces to ᅳlog(y’) when y = 1 and log(1 ᅳ y’) when y = 0 just as we had defined earlier.
While the above is only the error for only one example (datapoint), we need to account for all the examples. Therefore, we find the mean of the errors by summing all of them and dividing by the number of examples as shown below.
The code for this would be as below:
1.0579769986979133
Let’s plot the loss. You’ll keep track of the losses for all the iterations (epochs) after which you will be able to visualize the decrease in error.
[1.0579769986979133]
This will usually be the toughest part to understand, but I have put in very easy terms.
Gradient descent in layman words is like walking down a hill. This hill refers to the error in our case. Higher the error, higher the hill!! So as you roll down the hill, the error (Loss) goes down.
Therefore, our goal is to roll down to the point of least error.
Hey, but wait a sec! How can I change the loss? 🤔
Now you must observe that if you tweak the parameters W and b the Loss also changes!!
Hence, we find the derivative (i.e., the rate of change) with respect to the parameters W & b and update them accordingly.
Well, that’s simple! Just find its derivative or the slope. Mathematically, the slope is zero at the minimal point(minima) [You can refer to this wonderful Khan Academy video if you do not know about derivatives and the slope.] and if you observe keenly, the slope is negative on the left of minima and positive on the right of minima. And when p is to the left of minima, we need to add some value so that it moves towards optimal p and when p is to the right subtract some value.
We find the derivative of the Loss with respect to W and b in our case.
I’ve skipped a few steps which are unnecessary, but if interested, you can refer to this video below.
As to the last part of updating the parameters, with the slopes, we multiply the slope by a dampening factor α so that the parameters don’t overshoot when having a very high error.
You’ll now implement the same gradient_descent algorithm as below.
Now there’s one last thing to do i.e., update combine both the plot functions into one.
We have obtained a beautiful plot of the decision boundary as well as the Loss.
Now as your final task, let’s put together everything you had done until now and see the results for yourselves.
So, you will first define the hyperparameters. You can play around with these values especially the learning rate l_r and epoch to observe how the model learns to predict.
Now, you can write the code from creating a dataset to training your logistic regression model in a single go as below.
Shape of X_train (400, 2) Shape of y_train (400, 1) Shape of X_test (100, 2) Shape of y_test (100, 1) Initializing weights... W: [[ 0.43753829] [-1.70958537]] b: -0.8793505127088026--------------------------------------------------------------------Iteration: 0 W = [[ 0.43753829] [-1.70958537]] b = -0.8793505127088026 Loss = 0.3991070405361789
-------------------------------------------------------------------- Iteration: 5 W = [[ 0.09634223] [-1.73455902]] b = [[-0.87943027]] Loss = 0.10346899489337041
--------------------------------------------------------------------Iteration: 10 W = [[-0.02440044] [-1.73555684]] b = [[-0.87945069]] Loss = 0.06810528539660123
--------------------------------------------------------------------Iteration: 15 W = [[-0.09215724] [-1.73617853]] b = [[-0.87945614]] Loss = 0.0569509026646708
--------------------------------------------------------------------Iteration: 20 W = [[-0.13607178] [-1.737859 ]] b = [[-0.87945457]] Loss = 0.05224655857924122
--------------------------------------------------------------------Iteration: 25 W = [[-0.16649094] [-1.74055496]] b = [[-0.87944893]] Loss = 0.04996979153280091
We had earlier created a test dataset, you’ll now test your regression model against it and determine the accuracy.
The formula for accuracy is:
Prediction: Loss = 0.2929627097697204 Accuracy = 95.0%
Hence W = [[ 3.9538485] [-3.9152981]] b = [[-0.68703193]]
We have obtained an accuracy of 95% which is pretty good!
In this tutorial, you learned
Sigmoid Activation FunctionCreating a Logistic Regression ModelTraining a logistic regression model
Sigmoid Activation Function
Creating a Logistic Regression Model
Training a logistic regression model
In the next tutorial, you’ll learn to implement Neural Networks from scratch using only Numpy!