Optimization: Ordinary Least Squares Vs. Gradient Descent — from scratch
By: Chayan Kathuria
Tags: Machine Learning, Optimization, Gradient Descent, Linear Regression, Data Science

Now back to our optimization problem that we defined using OLS. Let’s do the solution using Gradient Descent. Again, the loss function will be the same. But this time we will be iterating step-by-step to reach the optimal point. W start with any arbitrary values of the weights and check the gradient at the point. Our aim is to reach the minima which is the valley bottom. So our gradient should be negative always.

Next, we need to update the weights to get them closer to the minima. We have the following equation for it:

This means that weight in next iteration will be weight in previous iteration minus the update. Now this update has 2 components: direction — which is the slope or the gradient, and the value — which is the step size. The gradient will be:

So if at our initial weights, the slope is negative, we are in the right direction. We just need to increase the value of the weights to get it closer. That is exactly what the above equation does. If the slope is negative at the particular point, the second term gets added to the value of weights in previous iteration. Conversely, if it is positive, that means we need to go in the opposite direction to get to the minima. In this case, the equation subtracts the second term from the value of weights in the previous equation. Neat.

The second component we need to consider is the step size — α. This is a Hyperparameter which you need to decide prior to the start of the algorithm. If the α is too large then your optimizer will be jumping big leaps and never find the minima. Conversely, if you set it to be too small, the optimizer will take it forever to reach the minima. Hence we need to set an optimum value for α beforehand.

If you understood correctly, you would appreciate that the gradient we are talking about here is essentially the Sum of Error. And by we are in essence taking a fraction of that error. And we are propagating that error to update our weights.

Therefore, our equation for weight update becomes:

So let’s jot down the clear steps we performed:

Initiate the values of the weights W0, W1 — which can be any value and the step size α — which needs to be a good value. Find the predictions of target Ŷ = W0 + W1.X for all X. Calculate the error values (Ŷ-Y) and the MSE. Update the weights as per the Gradient Descent update rule. Repeat 2–4.

This is known as Batch Gradient Descent as we are taking the error of the complete batch or the data set per iteration. We also have the following variants of Gradient Descent:

Stochastic Gradient Descent , where the updates in the weights are done in every iteration

, where the updates in the weights are done in every iteration Mini Batch Gradient Descent, which is a midway between Batch and Stochastic, divides the complete data set into mini batches and then applies weight updates after each batch.

Now finally let’s get all of this done in a few lines of code in Python!

Let’s begin by initializing our tiny little data set:

Now onto Step 1, initializing weights and the step size which I have chosen as 0.04. You can try tweaking the value and see the results for yourself:

After initializing, we iterate through the complete data set multiple times and calculate Mean Square error per iteration and update the weights:

So we iterate 10 times and hope that our algorithm has converged sufficiently. Let’s take a look at our final weights and see how close they got to our OLS solution:

Pretty close! Now let’s check the predicted target variable, Ŷ and the Error:

As you can see the predicted variable got pretty close to the actual values. Finally, let’s plot the Mean Square Error values per iteration and see how did our algorithm performed:

Great! So we covered :