Linear regression from scratch in Python.
By: Joanna Trojak
Tags: Python, Machine Learning, Artificial Intelligence, Scikit Learn, Machine Intelligence

A. What is the linear regression?

Linear regression is one of the basic machine learning models. We may treat the learning algorithm as a black box and do not bother with the internal details of the implementation but having a good understanding of how the linear regression engine works allow can give you the following benefits. First of all, you can quickly choose the appropriate model and finetune the hyperparameters. Second of all, a good understanding of linear regression will help you debug the model. Finally, most of the theory required to understand the linear regression will bear its fruits once you start with neural networks.

Without much further ado, let us begin with the question — what is a linear regression model? A linear model makes a prediction using a weighted sum of the input features and a constant called the bias term (also called the intercept term).

In its normal form, it has the following equation:

We can further reduce the equation to:

Okay, so this is the model. Now, how can we train it? Training a model is basically, finding the right parameters which fit the training set. To achieve it we need to a performance measure, how well the model fits the data. The most common performance measure for the regression model is RMSE (Root Mean Square Error). In practice, MSE is used because the value that minimizes a function also minimizes its square root.

A.1 Normal equation

There is a direct solution which gives the answer to the question of how to minimize the cost function. It is called the normal equation and it looks like this:

To illustrate how the linear regression from normal equation works we can generate some random and linear points.

Now we can use the equation to find the solution. To do that we are going to use inv() function from numpy library to compute the inverse of the matrix and dot() for matrix multiplication.

Now we can find what is the result of the function.

Now when we know the theta we can use it to make predictions.

We can plot the predictions to see how the linear regression looks like.

A.2 Computational complexity

The computational complexity of linear regression with the normal equation is usually about O(n2.4) to O(n3). If we get a very large set of features, like 100 000, then the normal equation gets really slow. On the other site, the equation is linear therefore it handles large training sets well, on condition they can fit into memory. Furthermore, once the model is trained the new predictions are really fast.

B. Gradient descent

Gradient descent is the second way to obtain a solution to a linear regression problem. It is an optimization algorithm capable of solving a wide variety of problems. The general idea is to tweak the parameters iteratively in order to find the optimal cost function.

A good way to understand gradient descent is to imagine you are on the mountain peak in the dense fog and you want to go down. You can only feel the slope of the ground below your feet. A good strategy to get to the bottom of the valley quickly is to go downhill in the direction of the steepest slope. This is exactly what gradient descent does. It measures the local gradient of the error function with regards to the parameter vector θ, and it goes in the direction of descending gradient. Once the gradient is zero, you have reached a minimum.

You initialize θ as random values and then you improve one step at a time in an attempt to decrease cost function.

An important parameter in GD is the size of the steps determined by the learning rate parameter. If we choose learning rate to be too small the algorithm will have to go through many iterations in order to achieve the minimum.

In case when the learning rate is too high, we may encounter another problem. You may jump across the valley finding yourself even higher than before.

Finally, not all functions have a nice regular shape. Some functions may have local minimums which are not as good as the global minimum.

Fortunately, the MSE cost function for a Linear Regression model happens to be a convex function, which means that if you pick any two points on the curve, the line segment joining them never crosses the curve. This implies that there are no local minima, just one global minimum. It is also a continuous function with a slope that never changes abruptly. This is very good news. GD is guaranteed to approach close to the global minimum.

B.1 Batch gradient descent

In order to calculate GD we need to compute the gradient of the cost function in regards to each parameter θj. To do that we need partial derivatives. We can compute one derivative at a time or we can compute them all in one go.

Once we have completed the calculation of the gradient vector which points upfill we need to go in the other direction. We are subtracting the gradient vector from θ. We also need the learning rate which has to be multiplied by the gradient vector. The result is the size of the step.

The simple implementation in Python looks like this:

What if we used different learning rates? The following image shows the gradient descent algorithm with three different learning rates.

You can find the perfect learning rate with grid search but you have to limit the number of iterations so the search can eliminate those rates which take too long to converge.

B.2 Stochastic gradient descent

Batch gradient descent can be very slow since it uses the whole training set to find the solution. Stochastic gradient descent on the other hand does not keep the whole training set in the memory and only requires a single instance. Its random nature can be a disadvantage. Instead of gently increasing it jumps up and down. Over time it may end up very close to the minimum but in the next iteration, it may bounce back. Once the algorithm finishes the solution will be good enough but not optimal.

In the situation when the cost function is very irregular this may be quite useful because the SGD can help the algorithm leave the local minimum and has a better chance of finding global minimum than the batch gradient descent.

To solve this problem we can gradually decrease the learning rate. The algorithm will start with the large steps and eventually, the steps will become smaller and smaller which will help the algorithm to find the global minimum. This process is called the simulated annealing because it resembles the process in metallurgy when the hot metal is gradually cooled down.

B.3 Mini-batch gradient descent

The last linear regression algorithm is the combination of the batch and stochastic gradient descent. Mini-batch gradient descent computes the gradient-based on small sets of instances called mini-batches. The main advantage over stochastic gradient descent is the performance boost. You can optimize the performance of the algorithm especially using GPU’s. The algorithm progress is less erratic than SGD especially with big mini-batches but it may be harder for it to escape the local minimum.

C. Polynomial regression

There is one case when the data is actually more complex to fit into linear regression. In that case, we have to use polynomial regression which is add power to each feature as a new feature. Scikt-learn has the class PolynominalFeature which allows using this technique on the data. After we have transformed the data we can use the linear regression to fit the data perfectly.