Plotting and Data Representation using Python. A guided walk through using a Coursera question
By: Emmanuel Sibanda
Tags: Exploratory Data Analysis, Data Visualization, Data Science

This post is based off an assignment from a Coursera course, 'Applied Plotting, Charting & Data Representation in Python'. This is part 2 of the University of Michigan's Applied Data Science with Python Specialization. This is meant to run you through one of my solutions to one of the assignments explained in a way that makes it easier to understand for someone with limited Python experience. Please note that if you are taking this course, you can learn from my solution but it is in your best interest not to copy it as it will only hurt you in the long run. Additionally, there is a better solution which gives you a better visualization for the purposes of the assignment.

The Scenario

Imagine you are measuring temperature in weather stations in a particular city or region. From my research, albeit limited, this weather is often measured to tenths of a degree (for whatever reason). You start collecting data over a ten year period, measuring daily and monthly maximum and minimum temperatures and this data is compiled on an excel spreadsheet.

A new year passes, let's assume its 2015 and you continue measuring the temperature as you always do. You have been observing record highs and lows and heatwaves around the country during 2015. You start wondering how temperatures of this year are measuring against temperatures over the previous 10 years. You want to answer the question; how do the record highs and lows for each month that has passed this year measure against record highs for each month set over the last 10 years.

You decide that the best way to present these findings is through a visualization of a line chart that will show the record highs and lows of each month over the last 10 years. To answer your current question you also want to superimpose a scatterplot of the record highs and lows from 2015 that either exceed the record lows or highs set over the last 10 years.

The Code

So now you have a question and you know what you need to develop to answer your question. You decide to write out the code in Python. In order to help with your visualization you will need to import certain libraries that will help you perform many complex actions without writing lengthy and complex code.

Importing Libraries and reading data

import datetime

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

#read data from a csv

df = pd.read_csv('data/C2A2_data/BinnedCsvs_d400/fb441e62df2d58994928907a91895ec62c2c42e6cd075c2700843b89.csv')

For this visualization we would want to use the datetime library to stipulate a cut-off date so that we can divide the data into 2 tables; i.) temperature measurements between 2005 and the 31st of December 2014 and ii.) measurements occurring after the cutoff date.

Numpy which is short for numerical python will allow us to carry out mathematical functions on multi-dimensional matrices and arrays.

The pandas and library will allow us to carry out manipulations and calculations on the dataframes we will be interacting with. While matplotlib will be used for plotting our graphs. We use the aliases pd and plt (and np for numpy) to interact with these libraries for ease of reference.

The next line of code then uses the read_csv module included in the pandas library to read the data we want from our csv. This is automatically saved as a pandas dataframe — we named this dataframe 'df'.

At this point I like to use the head method to read the top 5 rows of our dataframe. This helps me understand what columns the dataframe has and just get a better feel of what the data looks like

df.head()

Output

Now we know that the dataframe has 4 columns; ID, Date, Element, and Data_Value. There is also an index column listing the row number starting from zero. The dataframe contains a total of 165 085 rows

You notice from this that the measurements are categorised as either TMAX (the maximum temperature) and TMIN (the minimum temperature). The ID shows the station identification code which presumably shows which station took the measurements.

Removing leap year data

Since 2015 is not a leap year, we want to remove all leap year data. In the event that a leap year had record high/low temperatures over the 10 year period as this could have an impact on record high/low temperature measurements. Since 2015 would have 1 less day than years with leap years (2008 and 2012) this wouldn’t be an accurate comparison.

#converting date to date timestamp

df['Date'] = pd.DatetimeIndex(df['Date']).date

lis = ['2922008','2922012'] #list of 29 Febs appearing in each leap year

dates = [datetime.datetime.strptime(i, '%d%m%Y').date() for i in lis] #converting into list of datetime.date objects

df = df[~df['Date'].isin(dates)] #remove all 29 February dates

In this code we convert the entire Date column using the DatetimeIndex module. This converts the Date column into a timestamp object. We then specify with .date that we want to return a datetime.date object (‘the date time of the timestamp without the timezone information’).

Using the datetime.date library, we convert our list of leap year dates into a datetime.date object. We then only select the dates that are not listed as leap year dates (under the list ‘lis’) and filter out all leap year dates from our dataframe.

Dates, cutoff dates and dividing dataframe into 2 tables

Next, since we have ensured that all the Dates in the date column are the same data type (that they are all dates and not string for example), we want to divide the data into two tables based on whether the measurements occurred before or after the cutoff date.

#creating a separate column for the month

df['Month'] = pd.DatetimeIndex(df['Date']).month

#we are colleting dates between 2005 - 2014, so want to remove any data from dates after 31 Dec 2014

#creating datetime.date format of cutoff date

a= '31122014'

#converting a string into datetime.date object

cutoff_date = datetime.datetime.strptime(a,'%d%m%Y').date()

#dataframe for values after 31 Dec 2014

df2 = df[df['Date'] <= cutoff_date]

#returning dates before cutoff date

df3 = df[df['Date'] > cutoff_date]

Since we want our table to show us monthly highs over a ten year period we also want to create a month column. Listing only the month of the Date column as an integer.

For the cutoff date we added the date as an integer and used the datetime.date module to convert the string into a datetime.date object (similar to our Date column), in 'date/month/year' format.

We then proceed to create 2 new dataframes; df2- which contains all the measurements before or on the cutoff date and df3- which contains all the measurements after the cutoff date.

Finding record highs and lows

Now we need to answer the question, how do we find the record high and low temperatures over the ten years between 2005 and 2014 by month.

From looking at our dataframe with the head method we already know that there is a column that specifies whether a measurement is recorded as a minimum of maximum temperature. Additionally, since we have created a month column, we know that this contains the month in which the measurement was recorded. In order to help us carry out this computation we will use groupby. This is a pandas module that allows us to split data into groups based on one or more criterions. This can be combined with the aggregation method, 'aggregate' which lets us carry out aggregations on our data such as returning the maximum or minimum values. Combined with groupby we can identify the maximum and minimum values between the groups we have created grouped by month.

df_min = df2[df2['Element'] =='TMIN'].groupby('Month').aggregate({'Data_Value':np.min})

df_max = df2[df2['Element'] == 'TMAX'].groupby('Month').aggregate({'Data_Value':np.max})

In this code, we created 2 dataframes; df_min- here we filtered measurements classified as minimum temperatures, grouped them by month and found the lowest temperature for each month over the ten year period and df_max- we filtered measurements classified as maximum temperatures, grouped them by month and found the highest temperature for each month.

The Visualization process

Now we want to start making use of the matplotlib library to visualize our data.

plt.plot(df_max, '-o',df_min,'-o')

plt.show() output

We start off by plotting the record highs and lows over the ten year period as a line chart. These two will automatically be plotted in different colours. We used '-o' to identify each data value plotted on this chart, this is represented as a circle on our line chart.

For visualization purposes we want to shade the area between the highest and lowest point grey. I believe this will make the data easier on the eye, helping you focus on the highest and lowest temperatures and the variability of record temperatures over the ten year period.

x = df_max.index.values

plt.fill_between(x,df_min.values.flatten(),df_max.values.flatten(), color='grey',alpha='0.5')

plt.show() output

We use the fill_between module to fill the area between the two lines. For fill_between to function you need to have the length of the x coordinates that you want to fill and the coordinates/values from both df_min and df_max.

To find the length of the x coordinates we have created an x variable that contains the values of the index of df_max. This will return an numpy array containing the numbers 1–12 corresponding with the months of the year.

Since df_min and df_max are dataframes and we only want the values in the dataframes (the temperatures) of both df_min and df_max. However, this will return an array of arrays whereas we need a one dimensional array to use the fill_between module. We then opt to flatten the array into a one dimensional array. Color and alpha simply allow us to specify the colour and hue of the fill_between.

Super imposing scatterplot

To answer our central question we need to start looking at the recordings after the cutoff date. In order to do this we need to group the data by month and look for measurements that exceed the lowest and highest recorded temperatures on our line chart.

#superimpose scatterplot

df3_max = df3[df3['Data_Value'] > df3['Month'].map(df_max['Data_Value'])]

df4_min = df3[df3['Data_Value'] < df3['Month'].map(df_min['Data_Value'])]

plt.scatter(df3_max.Month.tolist(),df3_max['Data_Value'],s=10,c='red') #values over max

plt.scatter(df4_min.Month.tolist(),df4_min['Data_Value'],s=10,c='black') #values under min

plt.show() output

In order to compare each temperature grouped my month to the lowest and highest temperatures of each month, we use the map function to iterate through the df3 database, identify the month of each column and compare the temperature listed under the Data_Value column to the value listed for that particular month in the df_min and df_max dataframes (depending on whether we are looking at the minimum or maximum temperatures). If the value is higher or lower than the highest and lowest temperatures recorded over the ten year period, we add that row to our new dataframes; df3_max and df4_min.

Now we want to superimpose our scatterplot over our line chart. For the scatter plot we need to specify the x (months) and y axis (data values). We also want to specify the size of the circle representing each value and the colour to differentiate it from the line chart circles.

Making the chart easier to interpret

If you are planning on showing this chart to anyone else, it is important to make sure that the chart is easy to interpret and shows what you want it to show without misleading your audience.

The first problem with the chart as it stands is that the x-axis incrementing in 2s skipping odd number months.

plt.xticks(list(df_max.index),['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sept','Oct','Nov','Dec']) #we want to show all the x values and rename according to the month

In order to rectify this I specify the xticks and rename them as abbreviations of the months they represent.

The next issue pertains to how the temperature is recorded. We would want to show the temperature in degrees celsius and no as tenths of a degree.

df_max['Data_Value'] = df_max['Data_Value']/10

df_min['Data_Value'] = df_min['Data_Value']/10

df4_min['Data_Value'] = df4_min['Data_Value']/10

df3_max['Data_Value'] = df3_max['Data_Value']/10

plt.show() output — I mistakenly left out April bur corrected this error

I then divide all the temperature values by ten to convert tenths of a degree to degrees celsius.

The final issue is that the graph is not labelled, and the size makes it difficult for us to know how many days in were recorded as exceeded the monthly high or low temperature.