Neural Networks from scratch with Numpy — Part 2: Linear Regression | by Suraj Donthi | Towards Data Science

In the previous tutorial, you got a very brief overview of a perceptron.
towardsdatascience.com
In this tutorial, you will dig deep into implementing a Linear Perceptron (Linear Regression) from which you’ll be able to predict the outcome of a problem!
This tutorial will apparently include a bit more of math as it is inevitable, but there’s no need to worry as I will explain them ground up. Regardless of this, it must be realized that all machine learning algorithms are basically mathematical formulations that are finally implemented in the form of code.
Before we start off, remember that we had used the threshold activation function to mimic the function of AND and NOR Gates?!
Here we will use another extremely simple activation function called linear activation function (equivalent to not having any activation!).
Let us find out the wonders that this activation function can do!
Let’s assume that there is only one input and bias to the perceptron as shown below:
The resulting linear output (i.e., the sum) will be
. This is the equation of a straight line, as shown in the below figure.
It must be noted here that when no activation function is used, we can say that the activation function is linear.
This is a multivariate(multiple variables) linear equation.
Let us see how this is utilized for predicting the actual output of y in the next section i.e., Linear Regression.
Fitting a linear equation on a given set of data in n-dimensional space is called Linear Regression. The below GIF image shows an example of Linear Regression.
In simple words, you try to find the best values of m and b that best fits the set of points as shown in the above figure. When we have obtained the best possible fit, we can predict the y values given x.
A very popular example is the housing price prediction problem. In this problem, you are given a set of values like the area of the house and the number of rooms, etc. as features and you must predict the price of the house given these values.
So, the big question is... How does the prediction algorithm work? How does it learn to predict?
Let’s learn this on the go!
Let’s start by importing the required packages.
You’ll use the sklearn dataset generator for creating the dataset. You will also use the package for splitting the data into training and test data. If you are not aware of sklearn, it is a rich package with many machine learning algorithms. Although you get pre-built functions for performing linear regression, you are going to build it from scratch in this tutorial.
For creating the dataset, you must first set a list of hyperparameters — while m and b are parameters, the number of samples, the number of input features, the number of neurons, the learning rate, the number of iterations/epochs for training, etc. are called hyperparameters. You shall learn about these hyperparameters as you implement the algorithm.
For now, you shall set the number of training samples, the number of input features, the learning rate, and epochs. You shall understand the learning rate and epochs in a short while.
Your first task would be to import or generate data. In this tutorial, you’ll generate the dataset using sklearn's make_regression function.
For the purpose of learning, we shall keep the number of features minimal so that it is easy to visualize. Hence, you must choose only one feature.
Now, it’s time to visualize what the data generator has cooked up!
Let’s check the shape of the vectors for consistency.
Shape of vector X: (200, 1) Shape of vector y: (200,)
We need reset the size of y to (200, 1) so that we do not get errors during vector multiplications.
(200, 1)
Next, you will have to split the dataset into train and test sets, so that you can test the accuracy of the regression model using a part of the dataset once you have trained the model.
Now let’s split the data into train set and test set.
In our case, the training set is 80% and the test set is 20%.
Let’s check the shape of the Train and Test datasets created.
(160, 1) (160, 1) (40, 1) (40, 1)
As you can see, 80% of the data i.e., 80% of 200 data points is 160 which is correct.
So, what have we achieved till now?
We have done the initial data preprocessing and also explored the data through visualizing it. This is typically the first step while modeling any machine learning algorithm. We have also split the data for testing the accuracy of the model once it is trained.
What do we do next?
Clearly as shown in the above Linear Regression GIF image, we need to consider a random line at first and then fit it on the data through training.
Therefore, the next step is to randomly generate a line with a random slope and an intercept(bias). The goal is to achieve the best fit for the line.
Now, given m & b, we can plot the line generated.
Let’s update the function plot_graph to show the predicted line too.
Since the line is now generated, you’ll need to predict the values it is producing for a given value of x. From this value, all there is to do is to calculate their mean squared error. Why?
How could we find the difference between the actual output and the predicted output?
The simplest way would be to just subtract these two differences. We have a random line that gives an output y_pred for every x that is given, but it’s surely not the actual output. Luckily, we have the actual output of all x too! So what we do is instead of taking the difference directly (which is technically called absolute distance or L1 distance), we square it (called the Euclidean distance or L2 distance) and take the mean for all the given points & this is called Mean Squared Error.
Let us now predict the values of y_pred from the parameters m & b given the datapoints X_train by defining a function forward_prop.
As mentioned earlier, now that you have both the corresponding values for X_train and the predicted values for y_pred you’ll calculate the Cost/Error/Loss Function.
The Loss(Mean Squared Error) is:
Summing over all M examples, we obtain the Loss fn. as below:
Our goal is to obviously minimize the Loss so the regression line predicts more accurately.
Let us now codify this.
You will also save each value of Loss that will be computed to graphically visualize how it changes during training.
4005.265725705774
Let’s modify the above plot_graph function defined above to plot the Loss too.
You’ll visualize the line created from the parameters m and b.
Now that you have computed the loss, let’s minimize it.
Since Loss is the dependent variable and m & b are the independent variables, we’ll have to update m & b so as to find the minimum Loss.
So, the immediate question would be...
How can I update the parameters m and b?
Let us for instance consider just a single parameter p as shown below and let t(target) be the value that has to be predicted. We see that as cost converges to the minima, the parameter p reaches a specific value called the optimal value. Let’s say the optimum value of p is a.
You can make a few observations from this graph.
It is clear from the graph, that as p moves towards a (minima), the Cost decreases, and as it moves away from it, the cost increases.
Now, how can we make p move towards a, regardless of whether it is on the left or to the right of a as shown in the figure?
Let us consider the p of the curve. From calculus, we know that the slope of a curve at a point is given by dy/dx (here it is dL/dp where L → Loss). From the fig., when p is to the left of a, the slope is obviously ‒ve and when it’s to the right, the slope would be +ve. But we see that if p is to the left of a, some value must be added to p. Likewise, some value must be subtracted when p is to the right of a.
This means that when the slope is ‒ve implies p = p + (some val.) and when the slope is +ve implies p = p ‒ (some val.) to move towards a.
∴ We subtract the slope from p. This way, the slope is negated and it ensures that it always moves towards a. The resulting equation would be,
It must also be observed that if the cost is too high, the slope will be too high. Hence, while subtracting the slope from p, p value might overshoot a. Hence, it is necessary to decrease the value of the slope so that p does not overshoot a. Therefore, we introduce a dampening factor called the Learning Rate (α) to the slope. You’ll see later that by varying α the rate of decrease in error varies.
What we finally obtain would be,
As shown in the figure, the trajectory taken by p against cost is that of a Bell curve.
This method is called Gradient Descent.
In our case, we use two parameters m and b. Therefore, the Bell curve would be 3-dimensional as shown in the below figure.
As mentioned, you’ll compute the partial derivative of the loss function w.r.t to the parameters m & b. [Note: It is usually expected that you know the basic concepts of partial derivatives. However if you do not, you can refer this wondeful Khan Academy video]
Now we subtract the slope of the parameters m and b from their respective derivatives along with the dampening factor α(alpha).
From decreasing the values of m and b, they are incrementally moving towards the minima. So updating the parameters this way has to be done for many iterations, which is called epochs.
Let us define a function grad_desc, which calls both gradient and update_params.
We have now defined everything that we need, so let’s compile all the functions into one and see how our algorithm works. So, before you can actually run the code, you’ll have to set the hyperparameters.
Epoch: 0 Loss = 2934.082243250548Epoch: 10 Loss = 1246.3617292447889
Epoch: 20 Loss = 546.310951004311
Epoch: 30 Loss = 255.88020867147344
Epoch: 40 Loss = 135.36914932067438
Epoch: 50 Loss = 85.35744394597806
Epoch: 60 Loss = 64.60029693013243
Since you have trained the parameters for 60 epochs and the regression line looks to be fitting the data, you can move forward to the last phase, i.e., prediction on our test data and checking the accuracy.
For checking the accuracy, you can take the mean of percentage error for all the test data points.
Prediction: Loss = 56.53060443946197 Accuracy = 80.1676%Hence m = 82.34083095217943 b = 0.46491578390750576
The accuracy is 80% which is “ok” considering the variance in the data as is seen in the above graphs.
I was hoping to introduce something really interesting in the article and as a bonus, I have also added an intro to Neural Networks. But this surely comes with a catch!
The Neural Network is shown below.
From the image, we observe that there are two inputs each to the two neurons in the first layer and an output neuron in the second layer.
We will be using matrices for representing our above equations. We can represent them in vector (single column matrix) form as:
While doing matrix computations, we’ll need to take care of the dimensions and multiply. Hence, we rearrange a bit to arrive at the required output.
The expansion of the equation is not required and hence let’s stick to
Similarly, the value of
Now the output from the 2nd layer will be:
From the above set of equations, we see that a neural network with a linear activation function reduces to a linear equation.
The whole purpose of neural networks was to create a very complex function that can fit on any sort of data and as it can be clearly seen, a neural network with linear activation functions fails the purpose. Hence, it should be strictly noted that a linear function cannot be used as an activation function for the neural network, although it can be used only in the last layer for regression problems.
Then I guess you’ll have to hold your horses until the next tutorial to implement one!
Go ahead clone it and start running the cells on your Colab to see the miracles of Gradient Descent!!
github.com
In this tutorial, you learned
Linear Activation functions perform the tasks of regression i.e., learn to predict and forecast values. This method is called Linear Regression everywhere.An MLP(Multi-Layer Perceptron) with a linear activation function reduces to a normal Linear Regression task. Hence, linear activations must not be used in the hidden layers of a network. However, it can be used in the last layer for regression/prediction tasks.
Linear Activation functions perform the tasks of regression i.e., learn to predict and forecast values. This method is called Linear Regression everywhere.
An MLP(Multi-Layer Perceptron) with a linear activation function reduces to a normal Linear Regression task. Hence, linear activations must not be used in the hidden layers of a network. However, it can be used in the last layer for regression/prediction tasks.
In the next tutorial, you’ll learn about Sigmoid Activation Function and perform Logistic Regression which is the most important key to implement neural networks.
You can read through the next article on Logistic Regression here:
towardsdatascience.com
Are you working on any cool Deep Learning project?
You can connect with me on Linkedin:
www.linkedin.com
OR message me on Twitter for any queries: