2 ways to train a Linear Regression Model-Part 2
By: Jeffy Merin Jacob
Tags: Machine Learning, Stochastic Gradient, Gradient Descent, Batch Gradient Descent, Linear Regression

In the previous post we discussed how to compute model parameters (θ) that best fit the Linear Regression model to the training set directly using Normal Equation. In this post we will talk about the second way of training using Gradient Descent.

Calculating the normal equation sounded simple and it gave us model parameters that were close to that of the original function that was used to generate the data. So why do we need this second method of training?

Normal Equation calculates theta values 1.9 and 4.0, close to the original parameters 2 & 4

Well the reason is, the Normal equation and SVD approaches are very slow when you have a dataset with large number of instances or large number of features. What if your dataset is too big to even fit in memory? You would have to adopt an out-of-core-learning technique. Fortunately, Gradient Descent is one of them.

Gradient Descent

Gradient Descent is an optimization technique (not specific to Linear Regression) that is capable of finding an optimal solution for a wide range of problems.

“T he general idea of Gradient Descent is to tweak the parameters iteratively in order to minimize the cost function” — Hands-on Machine Learning by Aurelien Geron

Steps in Gradient Descent

Start by assigning random values to θ — this step is called the random initialization step. Next, tweak the values iteratively until the algorithm converges to a minimum cost function.

There are few different variants of Gradient Descent. Each with its own pros and cons. They are listed below:

We will get into more detail later. But now let’s talk about the cost function MSE (Mean Squared Error) and how it affects our search. Depending on the cost function, the search for the optimal solution becomes more challenging or less challenging. For instance, MSE cost function looks like a nice, regular bowl shape. Other cost functions may have plateaus, ridges and holes making it harder for Gradient Descent to find the optimal solution.

2 challenges facing GD (I will refer to Gradient Descent this way from now) are:

Converging to a local minimum instead of a global minimum. Getting stuck on a plateau for a long time

The following diagram describes both these challenges:

Lucky for us, the cost function MSE for Linear regression is bowl shaped and therefore the optimal solution lies at the bottom of the bowl. It is what we call a convex and continuous function. What does it mean to us? We are guaranteed to have only one global optimum!

Let’s reiterate what we are trying to do here: we are trying to find the best model parameters that minimize the cost function over the training set. We are using an optimization technique called Gradient Descent to search the model parameter space iteratively. With me so far? Okay, great!

Batch Gradient Descent

Batch GD uses the whole batch of training data at every GD step. This can be very slow if you have large number of instances, but it scales very well with large number of features (much faster than Normal Equation).

Batch GD uses the whole batch of training data at every GD step.

Let me also tell you about a very important parameter in Gradient Descent called “learning rate”. It is important that you choose a good learning rate since a small learning rate would result in the algorithm converging too late and a large learning rate would possibly miss the global optimum. You can treat the learning rate as a hyperparameter and determine it using Grid Search.

To implement Gradient Descent, you need to calculate the partial derivative of cost function with respect to each model parameter. Partial derivate of MSE with respect to θ0 and θ1 are given below:

These are calculated and stored in a vector called Gradient vector. The gradient vector is given below:

Note that the gradient vector points uphill. To go down-hill (end of the bowl), we need to subtract it from learning rate. Looks like we have all the equations to implement it. So let’s try it out!

eta = 0.1

n_iterations = 1000

m = 100 # number of instances theta = np.random.randn(2,1)

for itr in range(n_iterations):

gradient_vector = 2/m *X_b.T.dot(X_b.dot(theta) - y)

theta = theta - eta *gradient_vector

print(theta)

We got the exact same results as Normal Equation. An advantage of using Batch GD in Linear Regression is that we are guaranteed that with the right learning rate, the algorithm will eventually converge to global optimum.

Disadvantage of using Gradient Descent

We know that Batch GD requires the entire dataset at each step. Hence it can become very slow if you have a training set that’s large. Secondly, Batch GD, in general, can gets stuck in local minima since the cost function decreases very gently and is very regular compared to other variants like Stochastic GD that can jump out of local minima because if its irregularity. This, of course, it not a problem for us because our cost function has no local minima.

Stochastic Gradient Descent

The term “stochastic” means to randomly determine. Stochastic GD does exactly that. It picks a random instance in the training set for every step. The gradient is computed based on that single instance. Hence it is possible to train on huge datasets using this method.

It picks a random instance in the training set for every step.

Stochastic GD is very irregular, meaning IT’S ALL OVER THE PLACE! The cost function keeps jumping up and down. Eventually it will end up close to the minimum but even then, it will keep bouncing instead of settling down. Hence, we cannot guarantee that the model parameter values are optimal. But they may be very close.

However, because of its irregularity, it is less prone to get stuck in local optima. It can just bounce out of there! Let’s see a sklearn implementation of Stochastic GD:

Randomness is a good thing and bad thing!

#sklearn implementation of Stochastic GD from sklearn.linear_model import SGDRegressor

sgd_reg = SGDRegressor(max_iter = 500, tol = 1e-3, penalty = None, eta0 = 0.1)

sgd_reg.fit(X, y.ravel())

print(sgd_reg.intercept_)

print(sgd_reg.coef_)

Mini-batch Gradient Descent

If you have understood Batch and Stochastic GD, then Mini-Batch is not that hard. It picks a small random set of instances called “mini-batches”. The advantage of mini-batch is that it is slightly more regular than Stochastic meaning it can get a bit more closer to global optimum.

It picks a small random set of instances called “mini-batches”.

The only major difference between each of these types is the way they traverse through the parameter space. Batch GD can be thought of as a smooth line toward the minimum and it stops there. Stochastic and Mini-batch GD are more zigzag in the way they traverse, and they continue to walk around.

You can implement mini-batch GD using SGDRegressor’s partial_fit method. For more details on implementation, check out this article.

Conclusion

In this blog we learnt:

2nd Method of training a Linear Regression model (when the dataset is large) — using Gradient Descent. For the 1st method, refer to my previous blog. The properties of the MSE cost function — convex and continuous function The different challenges of Gradient Descent. A simple implementation of Batch GD A Scikit-Learn implementation of Stochastic GD Batch, Stochastic, Mini-batch Gradient Descent — its upsides and downsides.

If you enjoyed this blog please leave a clap or comment. It lets me know that my blog was helpful :) Happy learning, folks!

References

1. Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. Book by Aurelien Geron.