Polynomial Regression with Scikit learn: What You Should Know | by Angela Shi | Towards Data Science

Polynomial regression is an algorithm that is well known. It is a special case of linear regression, by the fact that we create some polynomial features before creating a linear regression. Or it can be considered as a linear regression with a feature space mapping (aka a polynomial kernel). With this kernel trick, it is, sort of, possible to create a polynomial regression with a degree that is infinite!
In this article, we will deal with the classic polynomial regression. With scikit learn, it is possible to create one in a pipeline combining these two steps (Polynomialfeatures and LinearRegression). I will show the code below. And let’s see an example, with some simple toy data, of only 10 points. Let’s also consider the degree to be 9. You can see the final result below.
Do you see anything wrong?
Well, in theory, this is wrong! For 10 points, a 9th-degree polynomial should fit them perfectly!
Or maybe, I am sure that some of you are thinking: why are you saying that this is wrong? This may be the right model. You think that the model should fit perfectly, but no, you are confused with polynomial interpolation!
First, you can try it for yourself using the following code to create the model.
import pandas as pdxdic={'X': {11: 300, 12: 170, 13: 288, 14: 360, 15: 319, 16: 330, 17: 520, 18: 345, 19: 399, 20: 479}}ydic={'y': {11: 305000, 12: 270000, 13: 360000, 14: 370000, 15: 379000, 16: 405000, 17: 407500, 18: 450000, 19: 450000, 20: 485000}}X=pd.DataFrame.from_dict(xdic)y=pd.DataFrame.from_dict(ydic)import numpy as npX_seq = np.linspace(X.min(),X.max(),300).reshape(-1,1)
from sklearn.preprocessing import PolynomialFeaturesfrom sklearn.pipeline import make_pipelinefrom sklearn.linear_model import LinearRegressiondegree=9polyreg=make_pipeline(PolynomialFeatures(degree),LinearRegression())polyreg.fit(X,y)
import matplotlib.pyplot as pltplt.figure()plt.scatter(X,y)plt.plot(X_seq,polyreg.predict(X_seq),color="black")plt.title("Polynomial regression with degree "+str(degree))plt.show()
Before talking about the difference between polynomial regression and polynomial interpolation. Let’s first talk about an answer that I got from the scikit learn team: you should not be doing this, expansion to a 9th-degree polynomial is nonsense. And scikit learn is built for practical use cases, and it works with finite-precision representations, not theoretical representations.
Yes, they are totally right! Just look at the numbers, how big they become: 1e24!
But if they cannot handle big numbers, shouldn’t they throw an error or a warning? Without any message, one will just consider that the model is correct, whereas, well, it is actually not.
OK OK, I know, some of you are not convinced that the result is wrong, or maybe it is impossible to handle big numbers, let's see with another package, numpy!
For the same example, polyfit from numpy has no problem finding the model. You can see the plot and the code below.
coefs = np.polyfit(X.values.flatten(), y.values.flatten(), 9)plt.figure()plt.plot(X_seq, np.polyval(coefs, X_seq), color="black")plt.title("Polyfit degree "+str(degree))plt.scatter(X,y)plt.show()
Now I know some of you are thinking: polyfit is a very different thing, it is an interpolation and not regression.
Because when asking around, I got some answers like this (but they are not accurate, or wrong):
polyfit is doing an altogether different thing. It is performing a univariate polynomial fit for some vector x to a vector y. Here we are performing a polynomial expansion of some feature space X in order to represent high-order interaction terms (equivalent to learning with a polynomial kernel) for a multivariate fit.
OK, what is a polynomial interpolation?
Well, for this kind of question, Wikipedia is a good source.
In numerical analysis, polynomial interpolation is the interpolation of a given data set by the polynomial of lowest possible degree that passes through the points of the dataset.
And we have this result that is proven: given n+1 distinct points x_0,x_0,... ,x_n and corresponding values y_0,y_1,... ,y_n, there exists a unique polynomial of degree at most n that interpolates the data (x_0,y_0),... ,(x_n,y_n).
Going back to our example: there are 10 points, and we try to find a 9th-degree polynomial. So technically, we are doing polynomial interpolation. And polyfit found this unique polynomial! This is not the case for scikit learn’s polynomial regression pipeline!
And this is precisely why some of you are thinking: polyfit is different from scikit learn’s polynomial regression pipeline!
Now, wait!
In polyfit, there is an argument, called degree. So you can modify the degree, let’s try with 5.
Yes, with polyfit, it is possible to choose the degree of the polynomial and we are doing polynomial regression with it. And degree 9, chosen by the user, is the special case of polynomial interpolation.
And it is reassuring because the linear regression tries to minimize the squared error. And we know that if there are 10 points, and we try to find a polynomial of degree 9, then the error can be 0 (can’t be lower!) because of the theorem of polynomial interpolation.
For those who are still doubting, there is the official document for polyfit: Least squares polynomial fit. Fit a polynomial p(x) = p[0] * x**deg + ... + p[deg] of degree deg to points (x, y). Returns a vector of coefficients p that minimizes the squared error in the order deg, deg-1, ... 0.
OK, time to go back to our scikit learn’s polynomial regression pipeline. So now, why the difference? Are there really two different polynomial regression (or fit), using both Least Squares, but using them differently?
I found this answer, but I am not getting it yet.
Both models uses Least Squares, but the equation on which these Least Squares are used is completely different. polyfit applies it on the vandemonde matrix while the linear regression does not.
While digging around, another important transformation of features should be mentioned: feature scaling.
In several books on machine learning, when performing polynomial regressions, the features are scaled. Maybe from the beginning, some of you were saying that it should be done.
And yes, scikit learn’s polynomial regression pipeline with the feature scaling, seems to be equivalent to polyfit! according to the plot (I didn’t really check, but visually they are the same).
You can use the code below:
from sklearn.preprocessing import PolynomialFeaturesfrom sklearn.pipeline import make_pipelinefrom sklearn.linear_model import LinearRegressionfrom sklearn import preprocessingscaler = preprocessing.StandardScaler()degree=9polyreg_scaled=make_pipeline(PolynomialFeatures(degree),scaler,LinearRegression())polyreg_scaled.fit(X,y)
Now, we didn’t answer our previous questions, and we have more questions: does feature scaling have an effect on linear regression?
Well, the answer is No.
To discuss this, there can be another article written and for our discussion about the effect of polynomial regression, we can just do another transformation.
X=pd.DataFrame.from_dict(xdic)/1000
That’s right, you just divide the predictors by 1000. Now, you know that the effect on the linear regression model is only proportional, but in practice, the difference is huge.
So that is why we can conclude that the initial numbers are too big for scikit learn.
In the end, we can say that scikit learn’s polynomial regression pipeline (with or without scaling), should be equivalent to numpy’s polyfit, but the difference in terms of big number handling can create different results.
And personally, I think that scikit learn should throw an error or at least a warning in this case.
I really would like to know your opinions!
If you like to know more about how polynomial regression is related to other supervised learning algorithms, you can read this article:
towardsdatascience.com
You will see that the polynomial regression is a special kind of feature space mapping.