Code and Techniques for Exploratory Data Analysis | Towards Data Science

Disclaimer: you are reading Part 1 “Exploratory Data Analysis Framework”. Part 2 shows how to build a baseline model based on the insights of the analysis below.
It is a very common situation when you need to show the value of the data to your clients by generating insights and building a first baseline model. As a Data Science Consultant, you might be getting such requests on a weekly basis. Therefore, it makes sense to build a basic framework for Exploratory Data Analysis that helps you to evaluate the data quality and show first insights to the client. In this article, I describe an Exploratory Data Analysis Framework that I use while analysing new datasets. The framework includes:
· Assessment of the data quality
· Overview of the data distributions
· Analysis of the different types of data (geolocation, time series, nominal etc.)
· Association (or Correlation) analysis
· Building a baseline model (which is described in Part 2 of the analysis)
The proposed framework is not exhaustive, but covers a wide range of cases and can be used as a starting point for the development of a more elaborate framework. It serves as a first sanity check before we approach any complex modelling. The full notebook can be found on GitHub.
In this example, I use the road accident safety data found at data.gov.uk. This dataset contains different types of data: Geo, Time Series, Categorical, and Continues variables that are very close to a real world scenario where the data come from multiple sources and might be of different types.
The end goal is to develop a model to predict the probability of a police officer to attend an accident. We also should evaluate the quality of data at our disposal and demonstrate which factors have the highest impact on the model’s decision.
First I am importing the packages that we need for the exploratory data analysis.
import pandas as pd
import numpy as np

import missingno as msno

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

ACCEDENTS_PATH = "data/Acc.csv"
acc_df = pd.read_csv(ACCEDENTS_PATH, 
                     parse_dates=["Date"], 
                     dtype={"Accident_Index": object})

Apart from the standard packages like pandas, numpy, and matplotlib I imported missingno which is a pandas compatible library for visualising missing values.
As I do not want the accident dataframe to be changed during the data exploration phase, the data are copied to a separate variable. All the subsequent steps of the data analysis will be performed on the copy and I will get back to the original dataset once entering the modelling phase (in Part 2).
data = acc_df.copy()

data.head()

5 rows × 32 columns
The dataset contains 32 variables most of which are categorical. We can check the meaning of each variable and its values in the Variable Lookup. First, we need to assess the variables datatypes, as some of them might need type casting.
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 129982 entries, 0 to 129981
Data columns (total 32 columns):
Accident_Index                                 129982 non-null object
Location_Easting_OSGR                          129963 non-null float64
Location_Northing_OSGR                         129963 non-null float64
Longitude                                      129953 non-null float64
Latitude                                       129953 non-null float64
Police_Force                                   129982 non-null int64
Accident_Severity                              129982 non-null int64
Number_of_Vehicles                             129982 non-null int64
Number_of_Casualties                           129982 non-null int64
Date                                           129982 non-null datetime64[ns]
Day_of_Week                                    129982 non-null int64
Time                                           129979 non-null object
Local_Authority_(District)                     129982 non-null int64
Local_Authority_(Highway)                      129982 non-null object
1st_Road_Class                                 129982 non-null int64
1st_Road_Number                                129982 non-null int64
Road_Type                                      129982 non-null int64
Speed_limit                                    129982 non-null int64
Junction_Detail                                129982 non-null int64
Junction_Control                               129982 non-null int64
2nd_Road_Class                                 129982 non-null int64
2nd_Road_Number                                129982 non-null int64
Pedestrian_Crossing-Human_Control              129982 non-null int64
Pedestrian_Crossing-Physical_Facilities        129982 non-null int64
Light_Conditions                               129982 non-null int64
Weather_Conditions                             129982 non-null int64
Road_Surface_Conditions                        129982 non-null int64
Special_Conditions_at_Site                     129982 non-null int64
Carriageway_Hazards                            129982 non-null int64
Urban_or_Rural_Area                            129982 non-null int64
Did_Police_Officer_Attend_Scene_of_Accident    129982 non-null int64
LSOA_of_Accident_Location                      122851 non-null object
dtypes: datetime64[ns](1), float64(4), int64(23), object(4)
memory usage: 31.7+ MB

Most of the variables in the accident dataset are numerical. There is also one variable that I explicitly set up as a datetime object while reading, and we also have four object variables:
Accident_Index — index of the accident
Time — time in a format HH:mm
Local_Authority_(Highway), LSOA_of_Accident_Location — categorical variables that contain information about location of the accident.
Before we jump into a deeper analysis of the variables in the dataset, let’s get an overview of the missing values.
As mentioned earlier, to get a quick impression of the pattern of missing values I use missingno package because of the graphical representation of the dataset that it provides. It allows us to quickly detect patterns in missing data and then come-up with a judgment whether the data are missing at random. Based on that we would need to develop a strategy on how to handle the missing values.
In the accident dataset, the missing values are indicated with a value of “-1”. First of all, we need to replace it with a NaN value, so we can get an overview of a pattern of missing observations.
As I am only interested in the variables that contain missing values I visualise only the columns that contain NaN values. The diagram shows missing observations in each column and the bar on the right shows each row’s data completeness.
As we can see, there are a few columns where missing values seem to be associated:
Junction_Control with 2nd_Road_Class
Pedestrian_Crossing-Human_Control with Pedestrian_Crossing-Physical_Facilities
Road_Surface_Conditions with Special_Conditions_at_Site and Carriageway_Hazards
That means that the values are not missing at random which we need to keep in mind while working on the strategy for handling missing values.
Also some variables like Junction_Control will be shown to have a significant impact on the response variable. Therefore in future, the missing values should be completed, e.g. by vehicle sensor data.
To get a better feel of the data we are dealing with, I will plot a histogram of each numerical value.
As we have been seeing, the dataset contains different types of data: Geographical, Categorical, Time Series. All types of data should be addressed and analysed individually.
Most of the variables are categorical with a low cardinality. We also see that some pairs of features like
Longitude and Location_Easting_OSGR
Latitude and Location_Northing_OSGR
have very similar distributions. They likely duplicate the information. Therefore we can concentrate on analysing only Latitude as well as Longitude and omit Location_Easting_OSGR as well as Location_Northing_OSGR for now.
Let’s take a brief look at the variable that we want to forecast, as further analysis will be performed in its context.
The response variable contains three classes. I will merge class 2 and 3 to a negative class, as I am interested only in predicting a probability of the police officer to attend an accident.
As we can see the classes in the response variable are imbalanced which is something to keep in mind when training the model.
Since there are quite some variables describing the location of the accident, I will check if location is an important factor to predict whether a police officer will attend the accident.
Red — Not Attended
Grey — Attended
The plot shows that there are areas where police officer are less likely to attend an accident. When we train a model, Longitude and Latitude might be variables with a high discriminative power.
Hypotheses testing
In this section, I will test a few hypotheses about the relations between the variables in the dataset. A common way to check for the relations between the variables is to calculate correlation (e.g. Pearson Correlation Coefficient). However, the vast majority of the variables in the accident dataset are categorical. We can not use correlation on our dataset, as correlation measures linear relationships between two quantitative variables. Therefore, I use Chi-Squared Test to identify a significant association between two categorical variables.
After staring at the data for a while, I came up with a hypothesis that one of the most important factors for a police officer to make a decision wether or not to attend an accident would be the accident severity. Then our first hypothesis to test would be the following:
H0: The severity of the accident has no impact on the probability of a police officer to attend an accident.
Remember, the null hypothesis is always a hypothesis of no change and our job now is to check whether we have enough evidence to reject it.
We can observe a difference between the groups, but we need to quantify these relationships. For that, I will perform a Chi-Squared Test at a 5% level of significance.
To perform a Chi-Squared Test I use the researchpy package. Researchpy produces Pandas DataFrames that contain relevant statistical testing information that is commonly required for academic research.
Chi-Squared test run by researchpy provides us with three values:
Pearson Chi-Square — is computed by taking a sum of squared differences between observed and expected frequencies normalised by expected frequency. It is used to calculate a p-value, which is another output of Chi-Squared Test
p-value — probability of the observed result assuming that H0 is correct
Cramer’s V — is an effect size which quantifies the association between two categorical variables. We could have used Pearson Chi-Square statistic to quantify the relationships, however as with covariance, it is not bounded and tends to grow with the number of dimensions and samples. Therefore, to make a comparable measure of the effect size across datasets of different sizes, we need to normalise the Pearson Chi-Square statistic by dividing it by the sample size and the minimum dimension minus 1. It will give us a value between 0 and +1 which is called Cramer’s V.
The result of the Chi-Squared Test suggests that there is a statistically significant evidence to show that H0 is false. Given evidence and data we reject H0 that there is no association between the variables Accident_Severity and Did_Police_Officer_Attend_Scene_of_Accident with a p-value < 0.005 and effect size 0.1.
Now that we know that the association is significant, we want to test where the relationship is between the levels of the variables. In order to do this, we need to conduct multiple 2×2 Chi-Squared Tests. However, conducting multiple tests on the same dependent variable increases the chance of false positives, thus increasing the likelihood of getting a significant result by chance alone. To account for false positives we need to adjust a significance to the number of comparisons being performed by correcting its level. This correction calls Bonferroni correction. I will use the stats package from scipy for that, as an alternative method to researchpy. Something to keep in mind is that scipy does not provide the result in a Pandas Dataframe.
The chi-squared test using the Bonferroni-adjusted level of significance=0.017 shows that all comparisons are significant. However, it is quite time consuming to run Chi-Squared Tests for each pair of variables in our dataset. Therefore, I will build an association matrix as the last step of the data exploration.
There are Date and Time variables in the dataset and it is always a good idea to check for the time dependent patterns and add some new variables to help our model to make a decision. First, let’s take a glimpse at the Time variable.
We can observe two rush hour periods where accidents are more likely to happen: between 7–9 AM and 15–18 PM. Knowing that the amount of police officers is constant across the time, I would assume that during the rush hours the attendance would be lower, as there might be not enough police officers to attend all the accidents. Therefore, I will create a variable indicating a rush hour and test for its association with the probability of the police officer to attend an accident.
There is a slight difference between the groups. In order to quantify these relationships I will run another Chi-Squared Test.
H0: Rush hour has no impact on the probability of a police officer to attends an accident
The result of the Chi-Squared Test suggests that there is a statistically significant evidence that H0 is false. Given evidence and data we reject H0 that there is no association between the variables Rush_Hour and Did_Police_Officer_Attend_Scene_of_Accident with a p-value<0.005 and effect size 0.04.
Another time related variable in the accident dataset is Date. I will plot the total number of accidents per day to analyse a time depended trend.
The plot shows a cyclical pattern, which might suggest that there is some sort of seasonality. I will plot an ACF and PACF plots to check whether there is a significant seasonal component in the time series.
The ACF plot shows a significant trend component at lag 1 and significant seasonal component at lags 6 and 7, which might indicate a weekend. I will plot the number of accidents per weekday to check if the behaviour on the weekends is different.
We can observe a slight drop in overall accidents and those that were attended during the weekends. I will model weekends explicitly and then check if a newly created variable has a significant association with the probability of a police officer to attend an accident.
H0: Weekends have no impact on the probability of a police officer to attend an accident.
The result of the Chi-Squared Test suggests that there is statistically significant evidence to show that H0 is false. Given the evidence and data, we reject H0 that there is no association between the variables Weekend and Did_Police_Officer_Attend_Scene_of_Accident with a p-value<0.005 and effect size 0.04.
We have already tested a few hypotheses and engineered two new features, Rush_Hour and Weekend. Now let’s join the other data that are available at data.gov.uk. First, I will join the data from the casualties dataset.
The casualties dataset mainly contains categorical variables. From the first three rows we can see that the Accident_Index is not unique for each row, as one accident may lead to multiple casualties. For us that implies that we can not join the accidents and casualties data tables directly, but rather need to do some feature engineering. For the first evaluation let’s join just a few variables from the casualties dataset.
I will shape a few hypotheses and assume that police officers are more likely to attend an accident if one of the following is true:
Casualties are pedestrians
People below 18 or above 65 years old
The driver is below 18, or above 65 years old
Therefore, I will create new binary variables, join them to the accident dataset, and test whether there is a significant association with the response variable.
The Chi-Squared Test showed a significant association between newly created variables and probability of a police officer to attend an accident at a significance level of 0.05 and p-value<0.005.
Now let’s join a third datasource available at data.gov.uk, which is a dataset of vehicles that being involved in the accident.
From the first few rows, we can see that Accident_Index is not unique for each row, as one accident may involve several vehicles. Again we need to do feature engineering to join the data.
I will shape a few hypotheses and assume that police officers are more likely to attend an accident if one of the following is true:
Cycle is involved in the accident
Taxi, Van, or Public transport is involved in the accident
The driver is below 18, or above 65 years old
As in a case with the casualties dataset, I will create related binary variables, join them to the accident dataset, and test whether there is a significant association with the response variable.
The Chi-Squared Test shows a significant association between the newly created variables and the probability of a police officer to attend an accident at a significance level of 0.05.
Undoubtedly, much more sophisticated feature engineering can be performed on these three datasets, including building categorical embeddings. However, our objective is to build a first prototype model and to show to the client the value of the data by generating quick insights and spotting interesting relationships between the variables (e.g. geographical data with accident attendance, accidents seasonality etc.). To stay pragmatic I will end the feature engineering for now. The last thing I would like to do before entering the modelling phase is to build an association matrix for the categorical variables to summarise the analysis that has been done. I will build an association matrix by calculating Cramer’s V statistic for cross categorial association, which is the effect size provided by the researchpy package. As mentioned already, an important detail to keep in mind is that Cramer’s V tends to overestimate the strength of association for the large and high dimensional samples. Therefore we need to implement a bias correction, which is described on Wikipedia. I use a very nice implementation of the bias correction from Ziggy Eunicien.
The variable we are interested in forecasting, “Did_Police_Officer_Attend_Scene_of_Accident”, is located in the first column (and row) of the matrix.
Unsurprisingly, the time-related variables are strongly associated with each other: Date, Day_of_Week, and Weekend.
The location-related variables in the dataset, Police_Force, Local_Authority_(District), Local_Authority_(Highway), and LSOA_of_Accident_Location have a high association with the probability of a police officer to attend an accident. This is consistent with the findings presented under the analysis of geographical data.
In Part 1 of Exploratory Data Analysis I analysed the UK the road accident safety data. A data quality assurance was conducted with the result that there is a pattern in missing data. Sensor data should be used to improve the accuracy of the forecasts.
The analysis of geographical data showed that there are relationships between the location of the accident and the probability of a police officer to attend it. This implies that there are districts (for example East London) and regions that require a higher number of police officers to be on staff. It is especially important during rush hours when the percentage of not attended accidents increases.
The analysis of variables from casualties and vehicles datasets showed that a police officer is more likely to attend an accident if pedestrians and people under 18 or above 65 years old were casualties. Also, a certain type of vehicles (bicycles and public transport) lead to a higher likelihood of a police officer to attend the accident. That is a valuable insight for predictive analytics.
In Part 2 I will show how to build a baseline model based on the insights we learned during the exploratory phase.