Simple Linear vs Polynomial Regression | by Andrew Hershy | Towards Data Science

Fish get bigger as they get older. How predictive is fish length (cm) with age (yr)? Is the relationship best fit with a simple linear regression?
Pull in data, visualize, and preliminary analysesLinear regression analysisQuadratic and high-degree polynomial regression analysisSegment data into training and testingTest models per regression type (Linear, Quadratic, Sextic)
Pull in data, visualize, and preliminary analyses
Linear regression analysis
Quadratic and high-degree polynomial regression analysis
Segment data into training and testing
Test models per regression type (Linear, Quadratic, Sextic)
First, let’s bring in the data and a few important modules:
%matplotlib inlineimport numpy as npimport pandas as pdfrom sklearn.metrics import r2_scoreimport matplotlib.pyplot as pltfrom scipy import statsimport seaborn as snsdata = pd.read_excel(r”C:\Users\...\fish polynomial.xlsx”)df = data[[‘age’,’length’]]df
There are 77 instances in the data set. Below is the head of the original data:
Now let’s visualize the scatter-plot. We’ll be trying to predict length from age, so the axes are in their respective positions:
x = df['age']y = df['length']plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Scatterplot of Length vs Age')plt.scatter(x,y)plt.show()
There looks to be a trend. As the fish get older, there seems to be a relationship with length. To gain further perspective, let’s break down each axis into its own univariate distribution histogram.
Y — Length of Fish:
stdy = y.std()meany = y.mean()plt.hist(y)plt.xlabel('Length (cm)')plt.ylabel('Quantity')plt.title('Length Distribution Histogram')print ("The std of y is: " + str(stdy) + " The mean of y is: " + str(meany))
X — Age of Fish:
stdx = x.std()meanx = x.mean()plt.hist(x)plt.xlabel('Age (yr)')plt.ylabel('Quantity')plt.title('Age Distribution Histogram')print ("The std of x is: " + str(stdx) + " The mean of x is: " + str(meanx))
We can see the average length is 143.6 cm and the average age is 3.6yr. The standard deviation for age is proportionately higher than that of length, implying higher overall spread in the age distribution. This is despite length’s distribution having a higher range.
A simple linear regression is one of the cardinal types of predictive models. To put simply, it measures the relationship between two variables by fitting a linear equation to the data. One variable is considered to be explanatory (age), and the other is considered to be dependent (length). The regression model minimizes distance between the straight line and every data point to find the optimal fit.
Below, we’re using the scipy stats module to calculate key metrics for our line. Outputted beneath are the intercept and the slope, respectively.
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)intercept,slope
Now we will plot this line on the original scatter-plot by running every value of x through the linear equation: y = b0 + b1x
sns.set(color_codes=True)def linefitline(b):    return intercept + slope * bline = linefitline(x)plt.scatter(x,y)plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Scatterplot of Length vs Age - Linear Regression')plt.plot(x,line, c = 'g')plt.show()
r2_lin = r_value * r_valueprint('The rsquared value is: ' + str(r2_lin))
The rsquared value for this regression line shows that age explains 73% of the change in length. R-squared calculates how similar a regression line is to the data it’s fitted to. I wrote another article about rsquared you can reference here.
Other independent variables that could explain the remaining 27% variance in length could be food-availability, water quality, sunlight, fish genetics, etc. If we had the data on all those attributes, we could run a multivariate regression and have a better model. But, alas, we live in a world with limited data.
Intuitively, our simple linear model doesn’t go through the middle point of each cluster of y points. For years 1, 2 , 5 and 6, it goes above the median of the cluster. This is most likely because the majority of the sample is of the age 3 and 4, as we saw earlier. This shifts the line upwards.
It seems like the optimal line would need to curve to match the data more accurately. That’s where the polynomial regression comes in.
Simply put, polynomial regression models can bend. They can be constructed to the nth-degree to minimize squared error and maximize rsquared. Depending on the nth degree, the line of best fit can have more or less curves. The higher the exponent, the more numerous the curves.
Below we have some code to create the new line and to graph it on our scatter-plot. This line is a quadratic function because it’s only raised to the second power. Quadratic lines can only bend once. As we can see on the plot below, the new polynomial model matches the data with more accuracy.
x = df['age']y = df['length']plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Scatterplot of Length vs Age')p = np.poly1d(np.polyfit(x,y,2))xp = np.linspace(1,6,100)plt.plot(xp,p(xp),c = 'r')plt.scatter(x,y)plt.show()
r2 = r2_score(y, p(x))print('The rsquared value is: ' + str(r2))
The rsquared value is 0.80 compared to the 0.73 value we saw in the simple linear model. This means that 80% of length is explained by their age in this new model.
We can now experiment in changing the nth value of our model to see if we can find a better-fit line. However, we must keep in mind that over-fitting is a risk we will face the higher we go.
Below is an example of a polynomial raised to the 6th degree:
x = df['age']y = df['length']plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Scatterplot of Length vs Age')p = np.poly1d(np.polyfit(x,y,6))xp = np.linspace(1,6,100)plt.plot(xp,p(xp),c = 'b')plt.scatter(x,y)plt.show()
It caters a bit too extreme to the outliers and is too closely aligned to the data. The rsquared value for this model is 0.804, not too much higher than the quadratic model.
For this data set, most would agree that the quadratic function matches best.
How do we know that the quadratic polynomial regression line is likely the optimal fit for this data set? That’s where the concept of testing our data comes in!
First, let’s shuffle the data set. Then we need to divide it into training and testing segments. The training data is used for the purpose of creating our model; the testing data is used to see how well the model matches. There is a rule of thumb to divide into 70% training and 30% testing. Because this is a relatively small sample size of fish (n = 78), I decided to go a bit heavier on the testing side. I did 50 instances as training and the last 28 as testing. That’s a 65/35 split.
shuffled = df.sample(frac=1).reset_index(drop=True)shuffx = shuffled['age']shuffy = shuffled['length']trainX = shuffx[:50]testX = shuffx[50:]trainY = shuffy[:50]testY = shuffy[50:]plt.scatter(trainX, trainY)plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Testing Data - Length vs Age')axes = plt.axes()axes.set_xlim([1,6])axes.set_ylim([50, 200])plt.show()plt.scatter(testX, testY)plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Testing Data - Length vs Age')axes = plt.axes()axes.set_xlim([1,6])axes.set_ylim([50, 200])plt.show()
There are more points in the training plot compared to the testing, this is due to our 65/35 split:
Linear model test:
Now we will create a linear model based on the training data:
p1 = np.poly1d(np.polyfit(trainx, trainy, 1))xp = np.linspace(0, 6, 100)axes = plt.axes()axes.set_xlim([1,6])axes.set_ylim([50, 200])plt.scatter(trainx, trainy)plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Training Linear Regression')plt.plot(xp, p1(xp), c='r')plt.show()
The line matches nicely with the training set. A ‘0.75’ rsquared value is pretty good:
r2_train = r2_score(trainy, p1(trainx))print('The rsquared value is: ' + str(r2_train))
Now let’s see how that line matches to the testing data we saved:
xp = np.linspace(0, 6, 100)axes = plt.axes()axes.set_xlim([1,6])axes.set_ylim([50, 200])plt.scatter(testx, testy)plt.xlabel('Age (yr)')plt.ylabel('Length (cm)')plt.title('Testing Linear Regression')plt.plot(xp, p1(xp), c='r')plt.show()
The model looks decent. It doesn’t factor in the outlier at age 6, and it also looks to be a bit low with regard to the median length of each age cluster. These concerns are justified in the lower rsquared value:
r2_test = r2_score(testy, p1(testx))print('The rsquared value is: ' + str(r2_test))
Quadratic model test:
Let’s see how the quadratic regression compares with the simple linear regression. The code for these calculations is very similar to the calculations above, simply change the “1” to a “2” in when defining the regression in the numpy.polyfit method:
p2 = np.poly1d(np.polyfit(trainx, trainy, 2)).
The quadratic regression has a higher training rsquared. Additionally, it didn’t drop nearly as much when compared to the testing data.
Sextic (n6) regression test:
Lastly, let’s see how a high-degree polynomial regression performs:
This model starts off with the highest training rsquared value, but it plummets when actually tested. This is a symptom of over-fitting.
The quadratic regression best fits the data in this example. The shuffling prior to training/testing segmentation can affect these results, however. Depending on which rows end up in which segment, rsquared results can change slightly. This is particularly true with smaller data-sets such as this one.
Source of original data: Penn State.
Please subscribe, clap, or comment if you found this helpful.
Thanks for reading!
Other articles by me if you want to learn more: