Regression Plots with Pandas and Numpy | by Alan Jones | Towards Data Science

I like the plotting facilities that come with Pandas. Yes, there are many other plotting libraries such as Seaborn, Bokeh and Plotly but for most purposes, I am very happy with the simplicity of Pandas plotting.
But there is one thing missing that I would like and that is the ability to plot a regression line over a complex line or scatter plot.
But, as I have discovered, this is very easily solved. With the Numpy library you can generate regression data in a couple of lines of code and plot it in the same figure as your original line or scatter plot.
So that is what we are going to do in this article.
First, let’s get some data. If you’ve read any of my previous articles on data visualization, you know what’s coming next. I’m going to use a set of weather data that you can download from my Github account. It records the temperatures, sunshine levels and rainfall over several decades for London in the UK and is stored as a CSV file. This file has been created from public domain data recorded by the UK Met Office.
We are going to check whether the temperatures in London are rising over time. It’s not obvious from the raw data but by plotting a regression line over that data we will be better able to see the trend.
So to begin we need to import the libraries that we will need.
import pandas as pdimport numpy as npimport matplotlib.pyplot as plt
Nothing very unusual there, we are importing Pandas to help with data analysis and visualization, Numpy will give us the routines we need to create the regression data and Matplotlib is used by Pandas to create the plots.
Next, we download the data.
weather = pd.read_csv('https://raw.githubusercontent.com/alanjones2/dataviz/master/londonweather.csv')
(As you probably guessed, that’s all supposed to be on one line.)
We have read the CSV file into a Pandas DataFrame and this is what it looks like — a table containing monthly data that records the maximum and minimum temperatures, the rainfall and the number of hours of sunshine, starting in 1957 and ending part way through 2019.
I posed the question about whether summers were getting hotter, so I’m going to filter the data to give me only the data for the month of July when the hottest temperatures are normally recorded. And, for convenience, I’m going to add a column that numbers the years starting at year 0 (you’ll see how this is used later).
july = weather.query('Month == 7')july.insert(0,'Yr',range(0,len(july)))
The code above applies a query to the weather dataframe which returns only the rows where the Month is equal to 7 (i.e.July) and creates a new dataframe called july from the result.
Next, we insert a new column called Yr which numbers the rows from 0 to the length of the table.
july looks like this:
Now we can plot the maximum temperatures for July since 1957.
july.plot(y='Tmax',x='Yr')
There is a lot of variation there and high temperatures are not limited to recent years. But there does seem to be a trend, temperatures do seem to be rising a little, over time.
We can try and make this a bit more obvious by doing a linear regression where we attempt to find a straight line graph that represents the trend in the rise in temperature. To do this we use the polyfit function from Numpy. Polyfit does a least squares polynomial fit over the data that it is given. We want a linear regression over the data in columns Yr and Tmax so we pass these as parameters. The final parameter is the degree of the polynomial. For linear regression the degree is 1.
We then use the convenience function poly1d to provide us with a function that will do the fitting.
d = np.polyfit(july['Yr'],july['Tmax'],1)f = np.poly1d(d)
We now use the function f to produce our linear regression data and inserting that into a new column called Treg.
july.insert(6,'Treg',f(july['Yr']))
Next, we create a line plot of Yr against Tmax (the wiggly plot we saw above) and another of Yr against Treg which will be our straight line regression plot. We combine the two plot by assigning the first plot to the variable ax and then passing that to the second plot as an additional axis.
ax = july.plot(x = 'Yr',y='Tmax')july.plot(x='Yr', y='Treg',color='Red',ax=ax)
That’s it, done!
We can now see much more clearly the upward trend of temperature over the years.
And here is the same thing done with a scatter chart.
ax=july.plot.scatter(x='Yr', y='Tmax')july.plot(x='Yr',y='Treg',color='Red',legend=False,ax=ax)
That was fairly straightforward, I think, and I hope you found it useful.
For an introduction to plotting with Pandas see this: