Polynomial Regression with a Machine Learning Pipeline
By: Rukshan Pramoditha
Tags: Data Science, Machine Learning, Dimensionality Reduction, Polynomial Regression, Scikit Learn

Polynomial Regression with a Machine Learning Pipeline

Photo by Joshua Sortino on Unsplash

Welcome back! It’s very exciting to apply the knowledge that we already have to build machine learning models with some real data. Polynomial Regression, the topic that we discuss today, is such a model which may require some complicated workflow depending on the problem statement and the dataset.

Today, we discuss how to build a Polynomial Regression Model, and how to preprocess the data before making the model. Actually, we apply a series of steps in a particular order to build the complete model. All the necessary tools are available in Python Scikit-learn Machine Learning library.

Prerequisites

If you’re not familiar with Python, numpy, pandas, machine learning and Scikit-learn, please read my previous articles that are prerequisites for this article.

Without further delay, let’s go into the problem definition.

Problem Definition

We have a dataset which contains information about Age, Height, and Weight of 71 people. We want to build a regression model which captures the Weight of a person depending on Age and Height and evaluate its performance. Then we use that model to make predictions for new cases.

Actually, we don’t know the nature or complexity of our data until we plot them. Sometimes, we can easily fit a straight line and describe the model. But most of the time, this is not the case for real-world data. The data may be complicated and you need to consider a different approach to tackle the problem.

The Dataset

We have a dataset called person_data.csv (download here) which contains information about Age, Height, and Weight of 71 people. Let’s load it using the pandas read_csv() function and store it in the df variable.

Let’s check whether the data has missing values.

Great! All the values of the 3 columns have 71 non-null values meaning that there are no missing values.

Let’s define our feature matrix and the target vector. According to the problem definition, the feature matrix — X contains the values of Age and Height. The target vector — y contains the values Weight. Building a Regression Model is a supervised learning task so that we map the input X to the output y=f(X).

The dimension of our data is 2 because X is 2-dimensional. So, how can we plot 2-dimensional data of X with y? Obviously, we need to create a 3D plot. But there is another way. We can combine Age and Height into one variable called Z and then plot Z with y in a 2D plot. Combining Age and Height into one variable by reducing the number of features in the X is called the Dimensionality Reduction and the technique that we use to perform dimensionality reduction is the Principal Component Analysis (PCA) which uses the correlation of these two variables.

We have two advantages of performing Dimensionality Reduction before making the model:

It is extremely useful for visualizing our data in a 2D plot which makes easier to see important patterns It is extremely useful to remove correlated variables in the feature matrix, X.— doing so will avoid misleading predictions

Let’s check whether Age and Height are correlated or not.

It seems that the two features are highly correlated. The condition when there is a significant dependency or association between the independent variables (the variables in the feature matrix X) is called multicollinearity. Having multicollinearity will lead to misleading predictions.

Before making the model, our next task is to remove these correlated variables. As I said earlier, we use Principal Component Analysis (PCA) to do this. But before running PCA, it is essential to perform feature scaling if there is a significant difference in the scale between the features of the dataset. This is because PCA is very sensitive to the relative ranges of the original features. So, We apply z-score standardization to get all features into the same scale by using Scikit-learn StandardScaler() class which is in the preprocessing submodule in Scikit-learn.

Note: We apply feature scaling only for the feature matrix X. We don’t need to apply it for our target vector y.

So, the workflow for building our model is as follows. It contains a series of steps that should be applied in the given order. Building a machine learning model is not a one time task and you may come back to an earlier step and make some modifications and then you go again through the next steps.

The general workflow is:

Apply feature scaling for the feature matrix X [Use Scikit-learn StandardScaler() class]. Run PCA algorithm [Use Scikit-learn PCA() class]. Create a scatterplot and identify the nature of the relationship [Use Seaborn scatterplot() function]. Can we fit a straight line to our data or do we need to add polynomial features to fit the model accurately? If we can fit a straight line to our data, then run Linear Regression algorithm [Use Scikit-learn LinearRegression() class] If we cannot fit a straight line to our data, then we need to add polynomial features [Use Scikit-learn PolynomialFeatures() class] After adding the polynomial features, run Linear Regression algorithm [Use Scikit-learn LinearRegression() class]

Image by Author

Apply feature scaling

We apply z-score standardization to get all features into the same scale by using Scikit-learn StandardScaler(). All the features are scaled according to the following formula.

The scaled values of X is stored in the X_scaled variable which is a 2-dimensional numpy array.

Let’s check the mean and standard deviation of the scaled X values.

Now, you can see that mean is zero and the standard deviation is 1 for each variable in the X_scaled matrix.

Now, our data is ready to run PCA.

Run PCA algorithm

Now, we are ready to apply PCA for our dataset. We need to reduce two-dimensional data in the X_scaled matrix into one-dimensional data.

In Scikit-learn, PCA is applied using the PCA() class. The most important hyperparameter in that class is n_components. Since we are interested in getting one-dimensional data, the value of n_components is 1.

We have applied PCA for X_scaled values and the output values are stored in X_pca variable which is now a one-dimensional array.

We have reduced the dimensionality of our data, and also with some information loss compared to our original dataset. In PCA, the algorithm finds a low-dimensional representation of the data while retaining as much of the variation as possible.

Let’s check how much of variation retained by the algorithm while running it.

Wow! Awesome value! You can see that the first principal component keep about 97.36% of the variability in the dataset while reducing 1 (2–1) feature in the dataset. So the values of X_pca more accurately represent the values of the feature matrix X.

Let’s plot our data in a scatterplot.

Plot the data

Now we have one-dimensional data (X_pca) which represent X. Now we can plot X_pca with our y (Weight) values in a 2D plot.

Clearly, a straight line will never fit this data properly. Let’s see what will happen if we would try to fit a straight line to our data.

Where θ1 and 𝛼 are model parameters which learn during the training process.

The fit is so bad that the straight-line approach will never give us the best model for our data.

Only about 61% of the variability observed in the Weight is explained or captured by our model. RMSE value is 10.99. It means that on average, the predictions of the model are 10.99 units away from the actual values.

Let’s try out a different approach called Polynomial Regression to get the best fit for our data.

Polynomial Regression

You can use a linear model to fit nonlinear data. A simple way to do this is to add powers of each feature as new features, then train a linear model on this extended set of features. This technique is called Polynomial Regression.

So, polynomial regression that uses polynomials is still linear in the parameters. This is because you build the equation by only adding the terms together. So, the performance metrics like R-squared (R²-coefficient of determination) are still valid for polynomial regression. Do not get confused polynomial regression with non-linear regression where R² is not valid!

Adding powers for each variable

In our model, the only variable is X_pca. After adding powers for that variable, the model becomes:

Where θ1, θ2, θ3 and 𝛼 are model parameters which learn during the training process.

Let’s add polynomial features to our data using Scikit-learn PolynomialFeatures() class.

The most important hyperparameter in the PolynomialFeatures() class is degree. We set degree=4 so that it creates 3 additional features called X_pca², X_pca³, X_pca⁴ when the input (X_pca) is one-dimensional. The X_poly variable holds all the values of the features.

Running the algorithm

Now, we have transformed our data into polynomial features. So, we can use the LinearRegression() class again to build the model.

Wow! It seems that the polynomial approach gives us a better model.

About 90% of the variability observed in the Weight is explained or captured by our model. RMSE value is 5.526. It means that on average, the predictions of the model are 5.526 units away from the actual values.

So, we have significantly increased the performance of the model with Polynomial Regression.

Let’s check the distribution of the residuals (errors).

By looking at the histogram, we can verify that the residuals are approximately normally distributed with mean 0.

Hyperparameter Tuning

Model parameters vs hyperparameters

Model parameters are the parameters which learn during the training process. We do not manually set values for the parameters and they learn from the data we provide. For example, θ1, θ2, θ3 and 𝛼 are parameters in our polynomial regression model.

In contrast, the model hyperparameters are the parameters that do not learn from data. So, we have to set values for them manually. We always set values for the model hyperparameters at the creation of a particular model and before we start the training process. For example, we have set n_components=1 manually at the creation of the pca object from the PCA() class. We have also set degree=4 manually at the creation of poly_features object from the PolynomialFeatures() class.

In most cases, setting up the correct values for the model hyperparameters is one of the most challenging tasks. There is no magic formula behind this. You may try different values and get visual representations to verify your choice. Sometimes, you may try different values and evaluate the model performance and choose the best one. Sometimes, the domain knowledge of the problem will help you to deduce the correct values for the hyperparameters.

Let’s try out some different values (2–9) for the hyperparameter degree (Initially, I have set this as degree=4) and get some visual representations and model evaluation metrics.

Image by Author

High degrees can cause overfitting. The problem of overfitting is a condition where a statistical model begins to describe the random error in the data rather than the relationships between variables. In overfitting, the model fits training data very well but fails to generalize for new input data which are not in our dataset.

Lower degrees can cause underfitting. In underfitting, the model does not fit training data very well and also the new data.

When we set the value for the degree hyperparameter, we should always try to avoid both overfitting and underfitting conditions

By looking at the visual representations and the values of the performance metrics, we can decide that degree=4 or degree=5 is the ideal value for the degree hyperparameter.

Making predictions on new data

Imagine that we have 5 new observations and we want to predict Weights using our model. The new input data is stored on X_new array which is two dimensional. The first column is the Age and the second column is the Height.

New input data

When we build our model, we have scaled and transformed our original data 3 times. So, when we use our model to make predictions on new data, it is necessary to scale and transform new data using the same methods. So, we have to call fit_transform() method 3 times and then call the predict() method 1 time. So, this is annoying for us. To overcome this problem, we can build a machine learning pipeline for our polynomial regression model.

Building a machine learning pipeline

Scikit-learn refers to machine learning algorithms as estimators. There are three different types of estimators: classifiers, regressors, and transformers. The classifiers and regressors are called predictors.

As our analysis and workflow become more complicated, you may need to apply multiple transformations to your data before it is ready for a supervised machine learning model. Pipelines sequentially apply a list of transformers and a final predictor (classifier or regressor). Intermediate steps of the pipeline must be ‘transformers’, that is, they must implement fit() and transform() methods. The final predictor only needs to implement the fit() method.

In our workflow:

StandardScaler() is a transformer. PCA() is a transformer. PolynomialFeatures() is a transformer. LinearRegression() is a predictor.

So, we can build a pipeline for our model using Scikit-learn Pipeline() class. It sequentially applies the above list of transformers and the final predictor. Here is the code.

By using pipelines, we can easily build complex models with less code!

Let’s call the fit() method of our pipeline.

When calling poly_reg_model.fit(X, y), the following process occurs:

X_scaled = sc.fit_transform(X)

X_pca = pca.fit_transform(X_scaled)

X_poly = poly_features.fit_transform(X_pca)

lin_reg.fit(X_poly, y)

Let’s call the predict() method of our pipeline.

When calling poly_reg_model.predict(X_new), the following process occurs:

X_new_scaled = sc.transform(X)

X_new_pca = pca.transform(X_scaled)

X_new_poly = poly_features.transform(X_pca)

lin_reg.predict(X_new_poly)

So, poly_reg_model.predict(X_new) returns predictions for our new data.

Saving our model

Let’s save our model with the Python dill package.

After running this code, the poly_reg_model.dill (download here) file will appear in your current working directory.

Let’s load our model.

That’s it. You can share the poly_reg_model.dill file with others and they can use the model without building it again. It is just 743 bytes in memory!

This tutorial was designed and created by Rukshan Pramoditha, the Author of Data Science 365 Blog.

Technologies used in this tutorial

Python (High-level programming language)

(High-level programming language) numPy (Numerical Python library)

(Numerical Python library) pandas (Python data analysis and manipulation library)

(Python data analysis and manipulation library) matplotlib (Python data visualization library)

(Python data visualization library) seaborn (Python advanced data visualization library)

(Python advanced data visualization library) Scikit-learn (Python machine learning library)

(Python machine learning library) Jupyter Notebook (Integrated Development Environment)

Machine learning used in this tutorial

Principal Component Analysis (PCA)

Polynomial Regression

2020–10–12