Polynomial Regression — Gradient Descent from Scratch | by Mark Garvey | Towards Data Science

Gradient descent is an important algorithm to understand, as it underpins many of the more advanced algorithms used in Machine Learning and Deep Learning. Getting to grips with the inner workings of gradient descent will therefore be of great benefit to anyone who plans on exploring ML algorithms further.
The best way to learn is by doing, so in this article I will be walking through the steps of how the gradient descent process works, without using ML libraries such as scikit-learn for example. In day-to-day work, it is of course quicker and neater to make use of such libraries, but regarding the learning process I have found the exercise of implementing by hand to be invaluable for this particular algorithm.
The goal of gradient descent is to minimize the error of a model’s prediction, relative to the original data. In this article’s context, we will be looking at a second-degree polynomial model, also known as a quadratic equation:
Second degree polynomials tend to look something like this, when plotted:
We’re specifically looking at polynomial regression here, where the relationship between the independent variable x and the dependent variable y is modeled as an nth degree polynomial in x. Simply put, the coefficients of our second degree polynomial a,b and c will be estimated, evaluated and altered until we can accurately fit a line to the input x data. Gradient descent is the optimization step in this process that alters and improves on the values of these coefficients.
We will now look at how to create and plot such a curve, and then build an initial model to fit to this data, which we will then optimize and improve on using gradient descent. If we can get a model that accurately describes the data, the hope is that it should be able to accurately predict the y values of another set of x values.
We can start be choosing coefficients for a second degree polynomial equation (ax2+bx+c) that will distribute the data we will try to model:
These will be the coefficients for our base/ground truth model we hope to get our predictive model as close as possible to. Next, we need an evaluation function for a second degree polynomial, which, given a set of coefficients and a given input x returns the corresponding y:
We can see it in action when x=3:
7
Define some x data (inputs) we hope to predict y (outputs) of:
This is good, but we could improve on this by making things more realistic. You can add noise or ‘jitter’ to the values so they can resemble real-world data:
Test it out:
7Should get value in the range 3 - 116.233537936801398
This updated function will take in the inputs for the second-order polynomial and a jitter value j to add noise to this input, to give us a more realistic output than just a perfect curve:
When we build our predictive model, and optimize it with gradient descent, hopefully we will get as close to these values as possible.
The first pass at modelling involves generating and storing random coefficients for a second degree polynomial (y=ax2 +bx+c). This will be our initial model which will most likely not be that accurate, which we will aim to improve upon until it fits the data well enough.
(7, 6, 3)
Inspect this model’s accuracy by calculating the predicted output values from your input values:
It is evident from the above plot that this new model with random coefficients does not fit our data all that well. To get a quantifiable measure of how incorrect it is, we calculate the Mean Squared Error loss for the model. This is the mean value of the sum of the squared differences between the actual and predicted outputs:
47922.39790821987
Quite a large number. Let’s now see if we can improve on this fairly high loss metric by optimizing the model with gradient descent.
We wish to improve our model. Therefore we want to alter its coefficients a, b and c to decrease the error. Therefore we require knowledge about how each coefficient affects the error. This is achieved by calculating the partial derivative of the loss function with respect to each of the individual coefficients.
In this case, we are using MSE as our loss function — this is the function we wish to calculate partial derivatives for:
With output predictions for our model as:
Loss can therefore be reformulated as:
In this specific case, our partial derivatives for that loss function are the following:
If you calculate the value of each derivative, you will obtain the gradient for each coefficient.
These are the values that give you the slope of the loss function with regards to each specific coefficient.
They indicate whether you should increase or decrease it to reduce the loss, and also by how much it should be safe to do so.
Given coefficients a, b and c, calculated gradients ga, gb and gc and a learning rate lr, typically one would update the coefficients so that their new, updated values are defined as below:
Once you have applied that new model to the data, your loss should have decreased.
We need a gradient calculation function which, given a second degree polynomial’s coefficients, as well as a set of inputs x and a corresponding set of actual outputs y will return the respective gradients for each coefficient.
We’re now going to:
Use the above function to calculate gradients for our poor-performing random model,
Adjust the model’s coefficients accordingly,
Verify that the model’s loss is now smaller — that G.D. has worked!
Let’s set an initial learning rate to experiment with. This should be kept small to avoid the missing the global minimum, but not so small that it takes forever or gets stuck in a local minimum. lr = 0.0001 is a good place to start.
New model coeffs: 5.290395171471687 5.903335222089396 2.9704266522693037Now have smaller model loss: 23402.14716735533 vs 47922.39790821987
Visualize this improvement by plotting the training data, original random model and updated lower-loss model together:
We’re almost ready. The last step will be to perform gradient descent iteratively over a number of epochs (cycles or iterations.) With every epoch we hope to see an improvement in the form of lowered loss, and better model-fitting to the original data.
Let’s improve on the calc_gradient_2nd_poly function from above, to make it more usable for an iterative gradient descent process:
This will be called as part of the gradient_descent function:
Finally, let’s train for 1500 epochs and see if our model has learned anything:
This trained model is showing vast improvements after it’s full training cycle. We can examine further by inspecting its final predicted coefficients a,b and c:
Final Coefficients predicted: (2.0133237089326155, -4.9936501002139275, 3.1596042252126195)Original Coefficients: [2, -5, 4]
Not too far off! A big improvement over the initial random model. Looking at the plot of the loss reduction over training offers further insights:
We observe that the model loss reached close to zero, to give us our more accurate coefficients. We can also see there was no major improvement in loss after about 400 epochs — definitely no need for 1500 epochs. An alternative strategy would be to add some kind of condition to the training step that stops training when a certain minimum loss threshold has been reached. This would prevent excessive training and potential over-fitting for the model.
I hope you enjoyed this dive into gradient descent for polynomial regression. Certain concepts can be daunting to understand at first glance, but over time we become familiar with the ‘nuts and bolts’ of a problem if we stick at it long enough. I found this was certainly the case for me with this exercise, and feel it was a worthwhile learning experience overall.
If you liked this story, please consider following me on Medium. You can find more on https://mark-garvey.com/
Find me on LinkedIn: https://www.linkedin.com/in/mark-garvey/