Perform Better Data Analysis in Python with Pandas
By: Johar M. Ashfaque
Tags: Python, Programming, Big Data, Data Science, Data Analysis

Perform Better Data Analysis in Python with Pandas

A Python library that can be used to better understand a dataset is Pandas.

Data Analysis

Data analysis is about asking and answering questions about the dataset.

You need to spark questions about the dataset that you can pursue to better understand it. One way to do this is by summarizing and visualizing the dataset.

Pandas

The Pandas Python library is built for fast data analysis and manipulation. The strength of Pandas seems to be in the data manipulation side, but it comes with very handy and easy to use tools for data analysis, providing wrappers around standard statistical methods in statsmodels and graphing methods in matplotlib.

Onset of Diabetes

The UIC Machine Learning repository provides a vast array of different standard machine learning datasets that can use to study and practice applied machine learning. Lets consider the Pima Indians diabetes dataset.

The dataset describes the onset or lack of onset of diabetes in female Pima Indians using details from their medical records. Download the dataset and save it into your current working directory with the name pima-indians-diabetes.data.

Summarize Data

We will start out by understanding the data that we have got by looking at its structure.

Load The Data

Start by loading the CSV data from file into memory as a data frame. We know the names of the data provided, so we will set those names when loading the data from the file.

import pandas as pd names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] data = pd.read_csv('pima-indians-diabetes.data', names=names)

Describe Data

We can now look at the shape of the data. We can take a look at the first 60 rows of data by printing the data frame directly.

print(data)

We can see that all of the data is numeric and that the class value on the end is the dependent variable that we want to make predictions about. At the end of the data dump we can see the description of the data frame itself as a 768 rows and 9 columns. So now we have idea of the shape of our data.

Next we can get a feeling for the distribution of each attribute by reviewing summary statistics.

print(data.describe())

This displays a table of detailed distribution information for each of the 9 attributes in our data frame. Specifically: the count, mean, standard deviation, min, max, and 25th, 50th (median), 75th percentiles.

We can review these statistics and start noting interesting facts about our problem. Such as the average number of pregnancies is 3.8, the minimum age is 21 and some people have a body mass index of 0, which is impossible and a sign that some of the attribute values should be marked as missing.

Visualize Data

A graph is a lot more telling about the distribution and relationships of attributes.

Nevertheless, it is important to take your time and review the statistics first. Each time you review the data a different way, you open yourself up to noticing different aspects and potentially achieving different insights into the problem.

Pandas uses matplotlib for creating graphs and provides convenient functions to do so.

Feature Distributions

The first and easy property to review is the distribution of each attribute.

We can start out and review the spread of each attribute by looking at box and whisker plots

import matplotlib.pyplot as plt data.boxplot()

Box and Whisker Plots

We can see that the test attribute has a lot of outliers. We can also see that the plas attribute seems to have a relatively even normal distribution. We can also look at the distribution of each attribute by discretization the values into buckets and review the frequency in each bucket as histograms.

data.hist()

This lets you note interesting properties of the attribute distributions such as the possible normal distribution of attributes like pres and skin.

Histogram Matrix

Feature-Class Relationships

The next important relationship to explore is that of each attribute to the class attribute.

One approach is to visualize the distribution of attributes for data instances for each class and note and differences. You can generate a matrix of histograms for each attribute and one matrix of histograms for each class value, as follows:

data.groupby('class').hist()

The data is grouped by the class attribute (two groups) then a matrix of histograms is created for the attributes is in each group. The result is two images:

Histogram Matrix for Class 0

Histogram Matrix for Class 1

Feature-Feature Relationships

The final important relationship to explore is that of the relationships between the attributes.

We can review the relationships between attributes by looking at the distribution of the interactions of each pair of attributes.

from pandas.plotting import scatter_matrix scatter_matrix(data, alpha=0.2, figsize=(6, 6), diagonal='kde')

This uses a built function to create a matrix of scatter plots of all attributes versus all attributes. The diagonal where each attribute would be plotted against itself shows the Kernel Density Estimation of the attribute instead.

Scatter Plot Matrix

This is a powerful plot from which a lot of inspiration about the data can be drawn. For example, we can see a possible correlation between age and preg and another possible relationship between skin and mass.