An Introduction to Polynomial Regression | by Xichu Zhang | Towards Data Science

Before introducing polynomial regression, we need to have an idea, what linear regression does. The main idea of linear regression is to use the linear combination of base functions to build a model. In order to introduce this concept, there is one question to answer.
What does “linear combination” mean? It is a combination of several terms with addition and multiplication by scalars, in the form
where a1, a2,... are scalars and x1, x2,... are the terms, or in the language of linear algebra, vectors. No matter what those terms are, the model is linear to the parameters.
A polynomial regression model can be considered as a linear combination of several monomials and takes the following form
In the realm of linear regression, several variations can be encountered and it’s sometimes very easy to get confused by the similar-looking terminologies.
Multiple linear regression: it attempts to use a linear model to describe the relationship between one dependent variable and two or more independent variables. In this article, we will focus on polynomial regression, which is considered a special case of multiple linear regression.
Simple linear regression: it’s a special case of multiple linear regression as well, which involves only one independent variable. It uses a line to model the data, which is a polynomial of degree one.
General multivariate regression model: it’s several multiple linear regression simultaneously written together in the form
where Y is an n by p matrix, with p dependent variables in each column and n individuals placed in each row, X is an n by q known matrix with observed data, and B and U denote unknown parameters and noises respectively.
Generalized linear model: it applies a link function to link the parameter of the distribution of the dependent variable to the linear predictors (linear combination of the dependent variables). Differ from other linear models, it allows the error term of the dependent variable to have a distribution other than the normal distribution.
When the link function is identity, this model reduces to ordinary linear models with Normal errors.
To find a suitable model to fit the data using polynomial, there are two things to complete:
we need to know what’s the degree of it.we need to find the coefficients of each monomial.
we need to know what’s the degree of it.
we need to find the coefficients of each monomial.
The first task described above is in fact a tougher one, so we can look at the second first. To illustrate the principle of linear regression, let’s generate some data with white noise
X <- seq(from=-10, to=10, by=1)Y <- vector(mode = "double", length = length(X))lenY <- length(Y)for (i in seq(from = 1, to = lenY)){  noise <- runif(1, min=-1, max=1)[1]  rangeNoise <- 25  Y[i] <- X[i]^2 + 2*X[i] + 2 + noise*rangeNoise}df <- data.frame(X, Y)
of course, the best model for fitting this data is a quadratic polynomial. Let’s try to fit the data and see how it looks like. In R it’s easy, this is all we need to do
model <- lm(Y ~ I(X^2)+ I(X), df)
In fact, using poly(X, 2) is better, since it uses orthogonal polynomials and can avoid the problem caused by the correlation between X2 and X. However in order to compare the result from lm to the result from the manual calculation, we stay with Y ~ I(X^2) + I(X).
The optimal polynomial for fitting the data looks like this (apparently a linear model doesn’t necessarily have to appear to be a line, since it is just a linear combination of some terms, and the terms can be whatever, the linear model can look like whatever. But it is linear with regard to the parameters.)
where the grey region depicts the confidence interval.
What method did the lm function use, what results have we acquired and how can we interpret them? Let’s use summary to check it out
Firstly let’s have a look at the Estimate Std., the numbers there are the coefficients of our model, which means that the resulting polynomial is f(x) = 1.0341x2 + 1.7416x-0.7320. The way to calculate the coefficients is to minimize a cost function, in this case, the cost function is chosen to be the mean square error function
where yi is an observed value and yi hat is the predicted value. Since yi is a function of the monomials x, x2, x3,... and coefficients (I will also use “weights” interchangeably) w1, w2, ..., determining when the cost function achieves minimum gives us the expression of the w in terms of x, the x-es are known, thus we can solve the coefficients. This is the least square method.
The coefficients can also be calculated by means of pseudoinverse. Write the input data, weights, and observed values into matrices
The predicted values will be XW, and we want XW to be as close to Y as possible. The weights can be acquired as
Thus the loss function is
This is how we can calculate the weights
Now we want to prove that this is the something as the least square method, which means we can derive (2.3) from (2.2).
Then we take the derivative of f with regard to W.
after transposing both sides multiplying by the inverse of (X^T X), we immediately acquire (2.3).
To verify this
The result stored in Wm corresponds to the coefficients found using lm.(here intercept is the constant).
[,1]X^2  1.0340966X    1.74155111   -0.7320286
Std. Error is the standard errors of the estimated parameters. Since the data we have is a sample from the population, the optimal regression model for the whole population could fall in a range-the optimal solution we acquired is not exact (for the sample it is the best, but for the whole population, the underlying relation could differ a bit). In statistics, it’s always important to be aware, what data are we talking about, the whole population, or only the sample data. Those errors are used in calculating the range. Then how do we calculate those errors?
Write done our model in the form
It means
where εi is the error term, it’s the difference between the observed value and the predicted value.
According to this, we can calculate the coefficients in the following way
where the hats on the variables mean that they are estimations, σ(X) is the standard deviation of X, and X_hat is the mean of X. The degree of freedom is n-3 (in this example n-3 = 21–3 = 18, we have 21 points in the sample). s is the Residual standard error, here s = 15.11.
In this source the notation doesn’t seem to be very common and clear, so Wikipedia is referred to as well. In both sources, only the formulas for the simplest model (a line Y = aX + b) are given, but the formulas for our quadratic example can be derived. But of course, the reader is encouraged to try to implement them and verify them with some data.
The last two columns in Coefficients tell us whether a component in our model is significant. The approach is to calculate a t statistic, namely the t value. Then we calculate the p-value, namely Pr(>|t|), which is evidence against the null hypothesis (H0: the component in the model is not statistically significant, we don’t need it in the model). If the p-value is small enough, we can reject the null hypothesis (it’s worth noticing that we can never accept the null hypothesis, since we have only evidence against it).
Taking the term X as an example, the coefficient of X is w1, the t value for w1 is
t follows Student’s t distribution with degree of freedom n-3. In the case of this example, the t value is simply calculated by coefficient / standard error.
And the result is 3.197. To find the p-value, we need to refer to the table of critical values of student’s t distribution. We can see that our t value is between 2.878 and 3.610. So the cumulative probability is between 0.002 and 0.01 — the result is 0.00499. And the interval of the parameter in the underlying relation (2.4) in the whole population can be determined by
where t*n−3 is the (1-α/2)-th quantile of the tn−3 distribution, this value can also be found in this table. By default, α = 0.05, and the confidence interval is 1-α=0.95. If the p-value is less than α, we can reject the null hypothesis. Therefore, the term X is significant in our example.
The prediction will be discussed in the last section. It is very convenient in R, we just need to apply the function predict.
Here we come back to the first task, what kind of linear model do we want to use? What the terms of the linear combination should be there?
When using multiple linear regression, we often need to reduce the number of dependent variables, since if a more complex model is not much better than a simpler one, we should use the simpler one. For a very complex model, we might get high performance, but it is also much more difficult to interpret. Sometimes overfitting can happen as well. There are several methods for simplification, for example, the principal component analysis is a useful tool for this.
Since here we are only dealing with polynomials, PCA is not very suitable, stepwise regression will be a much better approach we can apply to reduce the degree of a model. But the mission of reducing the degree of the polynomial model is analogical to reducing the number of independent variables in any multivariable linear regression model. For a polynomial regression model, the higher the degree, the better the polynomial can fit the data, but in the meantime, more noise will be included and this will lead to very poor prediction (overfitting).
We can use a degree 5 polynomial for our data, of course, it is not necessary, and it is also not good for prediction. If we check the summary of this model, we will find out that the components have large p-values — they are not statistically significant. The following code can be used for conducting the stepwise regression (for which library MASS is necessary)
fullModel <- lm(Y ~ I(X^5) + I(X^4) + I(X^3) + I(X^2) + I(X) + 1, df)stepModel <- stepAIC(fullModel, direction = "backward", trace = FALSE)summary(stepModel)
The result stepModel is a quadratic polynomial, as we already know.
1.Speed & distance
The example used will be the built-in dataset car in R, which records the speed of cars and the distance covered when stopping. Polynomial regression is suitable here. In fact, it’s the correct answer according to our knowledge of physics.
We split the data into train and test sets and plot the linear model with testing data.
If we run an R2 test, it will tell us that this model explains 80.089% of the variability of the data in the test set.
2. Oil price
Here is when we shouldn’t use linear regression. Suppose we want to predict oil prices based on historical data (data source: https://finance.yahoo.com/quote/OIL/history?p=OIL).
We used a polynomial with degree 3, the plot seems good. ( index is indices of days, CloseP is the close price every day). But on the test data, it doesn’t seem to be plausible
In fact, in this example, it will never work since if we use a polynomial with an even degree, the curve will go up when the index gets bigger, with an odd degree it will go down. But apparently, it’s not how prices of commodities change.
Why is it so much worse than the first example? Figure 3.1 doesn’t look much less chaotic and noisy than Figure 3.3. However, in Figure 3.1 there is a time series, the relation between distance and speed and the relation between oil price and time are completely different stories. Much more things complicate the price change, for example, time lag, periodicity, etc.
This article tries to make an introduction to polynomial regression and connect it to simple linear regression and multivariable linear regression. It also briefly describes what is linear regression and what variations are there.
In the part which focuses on polynomial regression, firstly, we came to know how to fit the data and how to settle the coefficients (implementation using the built-in functions in R but also manually). After that errors in the model and the evaluation are introduced, mainly the calculation of t-value and p-value. Also, the statistical inference was briefly talked about.
At last, we had a look at two examples, where the polynomial regression model is suitable and unsuitable respectively.
[1] Linear combination(2015), nLab.
[2] Michelle Lacey, Multiple Linear Regression
[3] Kent, J. T., Bibby, J., & Mardia, K. V. Multivariate analysis (1979). Amsterdam: Academic Press.
[4] Nelder, J. A., & Wedderburn, R. W . Generalized linear models (1972). Journal of the Royal Statistical Society: Series A (General), 135(3), 370–384.
[5] Jonathan Bartlett, Assumptions for linear regression (2013)
[6] Alboukadel Kassambara, Stepwise Regression Essentials in R (2018)