Implementing Linear and Polynomial Regression From Scratch | by Chris Tam | Towards Data Science

In this tutorial, we’ll go through how to implement a simple linear regression model using a least squares approach to fit the data. After that, we’ll extend the model to a polynomial regression model in order to capture more complex signals. We’ll be using the mean squared error to measure the quality of fit for every model we generate. You can download all the resources I used to write this article from my Github repo 👍.
Let’s start by loading the training data into the memory and plotting it as a graph to see what we’re working with. Think of train_features as x-values and train_desired_outputs as y-values. The graph below is the resulting scatter plot of all the values.
Now it’s time to write a simple linear regression model to try fit the data. Our goal is to find a line that best resembles the underlying pattern of the training data shown in the graph. We’re going to use the least squares method to parameterize our model with the coefficients that best describe the training set before seeing how well the model generalizes to data it hasn’t seen before. Recall that the simple linear regression model is parameterized by a y-intercept and the slope of the regression line.
The process of finding parameters so that our model fits the training data is called ‘training’ our model. Given a design matrix X and a column vector of target outputs y, we can use the following equation to find the best intercept and slope coefficients for our linear model through least squares regression (for an in-depth view into the linear algebra concepts behind this equation, see this post):
Here’s how we code it up. We first create a design matrix X which holds a column of ones (in order to estimate the y-intercept) and another column to hold the values of our explanatory variable x. Then we take the inverse of the dot product of X with its transpose, and dot product it with the dot product of the X-transpose and y (the y-values for training_desired_outputs).
Here’s how our model fit the training data.
We can quantify how well our model fits the data by using the mean squared error (MSE) to calculate the average squared difference between our line and the actual data point in the training set. The MSE is calculated as follows:
Mean squared error on the training set:  2.1739455790492586
Not bad! The mean squared error is a measure of the quality of an estimator — in our case the estimator is our linear regression model. The MSE is always non-negative and values closer to zero are better.
Now let’s see how well our model predicts on data it hasn’t seen before. We call this step the testing phase. Remember, our model is defined by the coefficients returned by
So the function above returns w which contains coefficients for the y-intercept and the slope. We’re going to test our model using the same values we used to train it.
Here’s how our model fit the testing data:
Again we’ll use the MSE to evaluate how well our model fit the data (I’m using the same code provided earlier in the MSE.py gist).
Mean squared error on the testing set:  2.3118753456727985
Our testing error is slightly above our training error. This makes sense because training errors should usually be lower than testing errors. It’s rare that a model would predict more accurately on data it hasn’t seen before than the data it was trained on.
All we need to do to implement polynomial regression is to take our linear regression model and add more features. Recall the form of the linear regression model and compare it to the form of the polynomial regression model
You can see that we need an extra coefficient for every additional feature, denoted by x2...xm. The order of the polynomial regression model depends on the number of features included in the model, so a model with m features is an mth-degree or mth-order polynomial regression. We’ll start with 2nd-order polynomial regression and you’ll notice that it’s quite easy to increase the complexity of your regression model (increasing model complexity isn’t always a good thing and can lead to overfitting!!!).
Since we’re including another feature into our model, we’re going to have to account for it by adding another term into our design matrix. The general form of the design matrix with m-degrees looks like this
Let’s go ahead and build this in code:
Notice how we included a column for the x2 features on the right hand side of the design matrix X. The resulting three coefficients are stored in coeffs. Let’s apply the model to our training data and print out the regression line.
Here’s how our 2nd-order polynomial regression model fit the training data:
Mean squared error on the training set:  0.4846845031271553
And now let’s apply our model to the testing data.
Here’s how it fit.
Mean squared error on the testing set:  0.7573635655518132nd-order polynomial regression is a better fit than linear regression. 
When comparing the MSE’s of our linear regression model and 2nd-order polynomial regression model, we see that the latter fit the testing set better than former. Hooray! We were able to improve the accuracy of our model by increasing the complexity. Be warned, however, that increasing model complexity doesn’t always lead to better accuracy.
In order to extend this model further, try implementing a 3rd-order polynomial regression by adding in a cubed term for the feature x in the design matrix X, like this:
X = np.c_[np.ones(N), train_features_vals, np.square(train_features), np.power(train_features, 3)]
Make sure that when you plot your regression line, you modify y_pred to take this extra term into account.
y_pred = b[3]*np.power(x_line, 3) + b[2]*np.square(x_line) + b[1]*x_line + b[0]
I hope you’ve found this tutorial helpful! I am happy to respond to any questions or comments so feel free to leave thoughts below or private notes in the text. Cheers!