Python Pandas and SQLite. Using SQLite to store your Pandas... | by Alan Jones | Towards Data Science

The SQLite database is a built-in feature of Python and a very useful one, at that. It is not a complete implementation of SQL but it has all the features that you need for a personal database or even a backend for a data-driven web site.
Using it with Pandas is simple and really useful. You can permanently store your dataframes in a table and read them directly into a new dataframe as you need them.
But it isn’t just the storage aspect that is so useful. You can select and filter the data using simple SQL commands: this saves you having to process the dataframe itself.
I’m going to demonstrate a few simple techniques using SQLite and Pandas using my favourite London weather data set. It’s derived from public domain data from the UK Met Office and you can download it from my Github account.
All the code here was written in a Jupyter notebook but should run perfectly well as a standalone Python program, too.
Let’s start by importing the libraries:
import sqlite3 as sqlimport pandas as pdimport matplotlib.pyplot as plt
Obviously, we need the SQLite and Pandas libraries and we’ll get matplotlib as well because we are going to plot some graphs.
Now let’s get the data. There’s about 70 years worth of temperature, rainfall and sunshine data in a csv file. We download it like this:
weather = pd.read_csv('https://github.com/alanjones2/dataviz/raw/master/londonweather.csv')
And this is what it looks like. The data is recorded for each month of the year. The temperatures are in degrees Celsius, the rainfall is in millimetres and ‘Sun’ is the total number of hours sunshine for the month.
Now we are going to save the dataframe in an SQLite database.
First we open a connection to a new database (this will create the database if it doesn’t already exist) and then create a new table in that database called weather.
conn = sql.connect('weather.db')weather.to_sql('weather', conn)
We don’t need to run this code ever again unless the original data changes and, indeed, we shouldn’t, because SQLIte will not allow us to create a new table in the database if one of the same name already exists.
Let’s assume that we have run the code above once and we have our database, weather, with the table, also called weather, in it. We could now start a new notebook or Python program to do the rest of this tutorial or simply comment out the code to download and create the database.
So, we have a database with our weather data in it and now and we want to read it into a dataframe. Here is how we load the database table into a dataframe.
First, we make a connection to the database in the same way as before. Then we use the sql_read method from Pandas to read in the data. But to do this we have to send
a query to the database:
SELECT * FROM weather
This is just about the simplest SQL query you can make. It means select all columns from the table called weather and return that data. Here’s the code (the query is passed as a string).
conn = sql.connect('weather.db')weather = pd.read_sql('SELECT * FROM weather', conn)
That, of course, is just reproducing the original dataframe, so let’s use the power of SQL to load just a subset of the data.
We’ll get the data for a couple of years, 50 years apart, and compare them to see if there was any clear difference.
Let’s start by getting the data for the year 2010.
We are going to create a new dataframe called y2010 using read_sql, as before, but with a slightly different query. We add a WHERE clause. This only selects the data where the condition following the keyword WHERE is true. So, in this case, we only get the rows of data where the value in the year column is ‘2010’.
y2010 = pd.read_sql('SELECT * FROM weather WHERE Year == 2010', conn)
You can see that the only data that has been returned is for the year 2010.
Here’s a line plot for Tmax.
Now let’s do that again but for 1960, 50 years earlier
y1960 = pd.read_sql('SELECT * FROM weather WHERE Year == 1960', conn)
And now let’s plot the Tmax for the two individual years.
ax2010 = y2010.plot(y='Tmax')ax = y1960.plot(y='Tmax',color = 'red', ax=ax2010)ax.legend(['2010','1960'])
Interesting, 2010 was both hotter and colder than 1960 it seems. Is the weather getting more extreme? We need to do a bit more analysis to come to that conclusion.
Maybe we could start by finding the hottest years, say the ones where the temperature was over 25 degrees.
We do this by using a WHERE clause with a condition, Tmax > 25, that is, select the rows where Tmax was more than 25 degrees
But also since we are only interested in the max temperature, we will only select the columns that we are interested in. We do this by modifying the SELECT clause in the SQL query.
Instead of selecting * (i.e. all columns), we list the columns that we want returned: Year, Month and Tmax.
high = pd.read_sql('SELECT Year,Month,Tmax FROM weather WHERE Tmax > 25', conn)
That’s in year order. Maybe it would be more interesting to have it in order of temperature.
We can do that with SQL, too. We add the clause ORDER BY and give the column which should dictate the order. DESC means in descending order (omit that and you’ll get the default, ascending order).
high = pd.read_sql('SELECT Year,Month,Tmax FROM weather WHERE Tmax > 25 ORDER BY Tmax DESC', conn)
So, topping the chart is 2018 with 2006 not far behind. But 1983 and 1995 are in third and fourth place so there is no obvious pattern here that shows that temperatures are getting hotter over time.
One thing we can see is that the hottest months are in the summer! (Who’d have thought it.) But to be sure let’s plot a histogram.
high.plot.hist(y='Month', xticks=high['Month'])
Yes, the hottest month tends to be July. So, if we are looking for a trend why don’t we take a look at the months of July throughout the decades and see how the temperature changes.
Here’s another query:
july = pd.read_sql('SELECT Year,Month,Tmax FROM weather WHERE month == 6', conn)
This gives us a table of temperature for all years but only where the month is equal to 6, i.e. July.
Now we’ll make a bar plot of the Tmax values over time.
july.plot.bar(x='Year', y='Tmax', figsize=(20,5));
We could do a regression plot to see if there is any trend but frankly looking at this bar chart there is no obvious trend.
So, with this extremely limited analysis of temperature data in London, we can’t come to any conclusion as to whether climate change is affecting Londoners (but from personal experience, I can tell that it’s feeling pretty warm there lately).
But I hope that I’ve demonstrated that SQLite is a useful tool when used alongside Pandas and, if you haven’t done any SQL before, you can see how, even the tiny part of it that I’ve used here, might benefit your data analysis.
As always, thanks for reading. If you would like to know when I publish new articles, please consider signing up for an email alert here.
If you are not a Medium subscriber, how about signing up so you can read as many articles as you like for $5 a month. Sign up here and I’ll earn a small commision.
You can find sample code for this and other articles on my Github page.