Simple Time Series EDA using Pandas and Seaborn
By: Emanuele Fumagalli
Tags: Meteorology, Seaborn, Visualization, Time Series Analysis, Pandas

Simple Time Series EDA using Pandas and Seaborn

Introduction

This article shows how to perform a quick analysis on simple time series by using basic Pandas and Seaborn commands to generate heatmaps.

The similar techniques can be used on any dataset containing just a date and value columns (number of sales, users accesses…).

I hope the same steps can be useful when analyzing any time series dataset.

Just look at the visualizations and if interested go ahead with the article.

The Jupyter Notebook and dataset used can be found on Github and Kaggle.

Note: sometimes in the code blocks in the article I mix the code and the output for better readability, for the actual code always refer to the Jupyter Notebook.

Dataset

I’m using a dataset downloaded from the National Centers for Environmental Information (NCEI), the data is in the public domain and can be used freely.

In case using the same dataset or generating a new one from NCEI you need to cite the origin.

The station used is located at Heathrow Airport in London.

If interested in generating a similar dataset from another station you can start from the Search Tool, select Daily Summaries, the time range of interest, search for Cities and in the Search Term put the city you’re looking for.

When selected you need to add to Cart like an order but there is no charge for ordering data from Climate Data Online as explained in their FAQs.

The dataset is a csv file that contains one entry for each day and if we take only the fields we’re analysing it will look like:

"DATE","PRCP","TAVG"

"2010-01-01","0.0","0.8"

DATE: is the year of the record (4 digits) followed by month (2 digits) and day (2 digits).

PRCP: Precipitation (mm)

TAVG: Average temperature (°C)

Load dataset

First import the needed libraries, and set figure format to retina to have higher resolution plots:

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

import calendar

import matplotlib.style as style

import missingno as msno %config InlineBackend.figure_format = 'retina'

%matplotlib inline

style.use('ggplot')

Then load the csv file in a Pandas DataFrame, the date is converted from string into a datetime64 using parse_dates. Doing this it will be possible to use date methods.

Refer to read_csv documentation for more options, you might need to use to_datetime for non standard date formats.

Columns are then renamed to lowercase strings.

df = pd.read_csv('HeathrowMeteo2010-2019.csv',

usecols=['DATE', 'PRCP', 'TAVG'],

parse_dates=['DATE'])

df.columns = ['date', 'precipitation', 'avg_temp']

Checking missing values

It can be interesting to know is there are null values:

len(df) # total number of observations, rows in the dataset 3621 df.isnull().sum() # total number of rows where the value is null date 0

precipitation 30

avg_temp 0 round(df.isnull().mean() * 100, 2) # percentage of null values date 0.00

precipitation 0.83

avg_temp 0.00

So precipitation column has less than 1% of missing values, other columns don’t have any missing values.

If we use the wonderful missingno library we can see a nice visualization of missing values:

In this case we will fill the null values with 0.0 (meaning no rain), this is to make the rest of the analysis simpler (like when summing up) but this may change depending on your specific dataset.

df['precipitation'].fillna(0, inplace=True)

Check missing observations

Since this is a time series with a minimum and maximum date we also want to make sure all the days in the range are present, we can use date_range.

print(f"Data Available from {df.date.min()} to {df.date.max()}") Data Available from 2010-01-01 00:00:00 to 2019-12-31 00:00:00 idx = pd.date_range(df.date.min(), df.date.max())

print(f"Days present {len(df)} out of {len(idx)}") Days present 3621 out of 3652

So in this case some days are missing completely, for the rest of the analysis this is not going to be a problem but in case interested to know which ones are not in the datset but in the range you can use Index.difference, we can create a DataFrame just for better visualisation in Jupyter.

pd.DataFrame(data=idx.difference(df.date), columns=['dates']).sample(3) # shows 3 samples of missing dates dates

0 2015-04-08

22 2018-11-18

1 2015-04-09

Add additional columns useful for analysis

In order to create the needed visualizations it can be useful to augment the dataframe with additional columns representing date infos. This step is not strictly needed because we could just use the same methods when later grouping.

Note: On Kaggle Pandas is on an older version, if using Pandas 1.0 use day_name.

df['month'] = df.date.dt.month

df['year'] = df.date.dt.year

df['day'] = df.date.dt.day

df['weekdayName'] = df.date.dt.weekday_name # df.date.dt.day_name()

df['weekday'] = df.date.dt.weekday

df['week'] = df.date.dt.week

df['weekend'] = df.date.dt.weekday // 5 == 1

We can also add a column to represent if the day was dry, according to the MET Office a day can be considered dry if precipitation is less than 1mm.

MIN_PRECIPITATION_MM_DRY = 1.0

df['raining'] = df['precipitation'].gt(MIN_PRECIPITATION_MM_DRY).astype('int')

Samples of the dataframe after adding new columns

Display average precipitation for each month for every year

We could create a heatmap representing the average precipitation for every month in the dataset.

To do so we can use the pivot_table function:

all_month_year_df = pd.pivot_table(df, values="precipitation",

index=["month"],

columns=["year"],

fill_value=0,

margins=True) named_index = [[calendar.month_abbr[i] if isinstance(i, int) else i for i in list(all_month_year_df.index)]] # name months all_month_year_df = all_month_year_df.set_index(named_index)

By setting margins to True we can get the subtotals. Since we didn’t specify aggfunc the function used on the grouped values is the mean.

The resulting dataframe is the following:

We can then use seaborn heatmap to get a nice visualization:

ax = sns.heatmap(df, cmap='RdYlGn_r',

robust=True,

fmt='.2f',

annot=True,

linewidths=.5,

annot_kws={'size':11},

cbar_kws={'shrink':.8,

'label':'Precipitation(mm)'})



ax.set_yticklabels(ax.get_yticklabels(), rotation=0, fontsize=10)

ax.set_xticklabels(ax.get_xticklabels(), rotation=0, fontsize=10)

plt.title('Average Precipitations', fontdict={'fontsize':18}, pad=14);

Most relevant parameters used:

cmap=’RdYlGn_r’

There are predefined colormaps available in matplotlib, depending on your needs you can try with different color maps. By adding ‘_r’ it’s possible to invert the order. In the notebook I show one way to create a one with custom colours and uneven starting points.

robust=True

It helps to reduce the colormap range with robust quantiles instead of the extreme values.

This can help to avoid outliers to “flatten” the other values and get a more representative colouring.

Alternatively you can set the minimum and maximum values to anchor the colormap using vmin and/or vmax (if present they override robust parameter). This may be needed when comparing datasets with a different distribution of values but keeping same colouring for same values.

For the others parameters refer to seaborn heatmap documentation.

It’s quite easy to spot visually the extreme values of precipitation in the last 10 years and get a summary for each year and each month

Aggregate with a different function or grouping by different periods

By changing the aggfunc on pivot_table creation we can have a different aggregation, for instance we can use aggfunc=np.sum to calculate the total amount of rainfall per month.

And if we wanted to calculate the average amount of rainfall per weekday we just need to change index=[“weekday”] (Code in the notebook).