pandas for Data Science: Part 2
By: Rukshan Pramoditha
Tags: Dataset, Data Visualization, Pandas, Data Science, Explore

Hello! Welcome to the 2nd tutorial of pandas: Exploring a Dataset. In this tutorial, I discuss the following topics with examples.

Topics discussing

Reasons for data exploring

Reading the data: pandas read_csv() function

pandas function Viewing the first few rows of the dataset: head() method of pandas DataFrame

method of pandas DataFrame Viewing the last few rows of the dataset: tail() method of pandas DataFrame

method of pandas DataFrame Viewing the dimensionality of the dataset: shape attribute of the DataFrame class

attribute of the DataFrame class Getting a concise summary of the dataset: info() method of pandas DataFrame

method of pandas DataFrame Getting descriptive statistics of the data: describe() method of pandas DataFrame

method of pandas DataFrame Viewing the levels of a categorical variable

Viewing the counts of categorical variable levels: frequency table — pandas crosstab() function, bar chart — plot() method of pandas DataFrame, bar chart — catplot() function in seaborn

frequency table — pandas function, bar chart — method of pandas DataFrame, bar chart — function in seaborn Components of the dataset: Values — values attribute of the DataFrame class, Columns — columns attribute of the DataFrame class, Index — index attribute of the DataFrame class

Values — attribute of the DataFrame class, Columns — attribute of the DataFrame class, Index — attribute of the DataFrame class Viewing the distribution of continuous variables: Histograms with mean and median — seaborn distplot() function with customizations

Histograms with mean and median — seaborn function with customizations Viewing the relationship between continuous variables: scatterplots — seaborn scatterplot() function

scatterplots — seaborn function Viewing pairwise relationships: matrix of plots — pairplot() function in seaborn

matrix of plots — function in seaborn Summary statistics of continuous variables: box plots of a continuous variable by a categorical variable — seaborn boxplot() function with x and y parameters, box plot of a continuous variable — seaborn boxplot() function with the y parameter

box plots of a continuous variable by a categorical variable — seaborn function with and parameters, box plot of a continuous variable — seaborn function with the parameter Saving the dataset as an Excel spreadsheet: to_excel() method of pandas DataFrame (diamonds.csv file → diamonds.xlsx file)

Dependencies

Dataset used

Why explore your data?

5 Key Steps in a Data Science Project

The above figure shows 5 key steps in a Data Science project. The Prepare step can further be divided into 2 steps.

Explore

Pre-process

The main goal of exploring a dataset is to understand your data. You may consider the following things.

Get the dimensionality (number of rows x columns) of the dataset

Have an idea about variables in the dataset

Identify missing values in the dataset

Create frequency tables and bar charts to explore categorical variables

Plot some graphs like histograms to see the distributions of continuous variables

Plot some graphs like scatterplots to explore the correlation between continuous variables

Get summary statistics of variables in the dataset

Doing some preliminary analysis to explore the dataset is very useful for data pre-processing which includes data cleaning and transform. All of these are very important for the next critical step, data analysis.

Reading the data

In this tutorial, I use the dataset called diamonds.csv which can be read by executing the following command. I use pandas read_csv() function with relevant parameters. Since I’ve saved the dataset in the current working directory, the filepath parameter contains just the name and file extension of the dataset.

After executing the above code, the dataset is loaded into the format called DataFrame, which is a two-dimensional pandas data structure composed of rows and columns — exactly like a simple spreadsheet. This DataFrame is assigned to a new variable called df which can be used for future references.

Now df is an object of pandas DataFrame class. We can check the type of df as in:

All the methods and attributes of the DataFrame class can be applied to df object. For example, the head() method of pandas DataFrame class can be applied to the df object as df.head() which returns the first 5 rows of df.

Viewing the first few rows of the dataset

If you want to quickly view the first few rows of the dataset, you can use the head() method of pandas DataFrame. This returns the first n rows of the object df. The default is 5.

The first 5 rows of diamonds.csv

If you want to see the first 10 rows, use head(10).

The first 10 rows of diamonds.csv

Viewing the last few rows of the dataset

If you want to quickly view the last few rows of the dataset, you can use the tail() method of pandas DataFrame. This returns the last n rows of the object df. The default is 5. If you use tail(10), the last 10 rows are returned.

The last 5 rows of diamonds.csv

Viewing the dimensionality of the dataset

The dimensionality of the dataset is the number of rows x columns in the dataset. The rows represent the observations and the columns represent the variables. The shape attribute of the DataFrame class returns a tuple representing rows x columns.

There are 53940 observations (rows) and 10 variables (columns) in the dataset.

Variable information

price: Price in US dollars

Price in US dollars carat: Weight of the diamond

Weight of the diamond cut: Quality of the cut (fair, good, very good, premium, ideal)

Quality of the cut (fair, good, very good, premium, ideal) color: Diamond color, from J (worst) to D (best)

Diamond color, from J (worst) to D (best) clarity: A measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))

A measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best)) x: Length in mm

Length in mm y: Width in mm

Width in mm z: Depth in mm

Depth in mm depth: Total depth percentage — 2 * z / (x + y)

Total depth percentage — 2 * z / (x + y) table: Width of the top of the diamond relative to the widest point

Getting a concise summary of the dataset

The info() method prints information about df including the index data type and column data types, non-null values and memory usage.

According to this summary, the dataset contains 7 continuous variables (carat, depth, table, price, x, y, z) and 3 categorical variables (cut, color, clarity). The data types of continuous variables are float64 and int64. The data type of categorical variables is object. There are no null-values for any variable. The dataset occupies about 4.1 MB of memory in the computer.

Getting descriptive statistics of the data

The describe() method generates descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN (Not a Number) values.

Descriptive statistics for continuous variables

This works for continuous variables. However, for categorical variables, we need to use the include=object parameter inside the describe() method.

Descriptive statistics for categorical variables

The categorical variable cut has 5 levels, color has 7 levels and clarity has 8 levels.

Viewing the levels of a categorical variable

To view the levels of a categorical variable, first, we need to extract that variable (column) from the DataFrame. The result is a pandas Series object. Then we can call the unique() method of this Series object. This method returns the unique values of the categorical variable and these values represent the levels.

The levels of the cut categorical variable are: Ideal, Premium, Good, Very Good, Fair.

categorical variable are: Ideal, Premium, Good, Very Good, Fair. The levels of the color categorical variable are: E, I, J, H, F, G, D.

categorical variable are: E, I, J, H, F, G, D. The levels of the clarity categorical variable are: SI2, SI1, VS1, VS2, VVS2, VVS1, I1, IF.

Viewing the counts of categorical variable levels

By using a frequency table

The crosstab() function of pandas computes a frequency table of a given categorical variable. Now, I create a frequency table for the categorical variable cut.

By using a bar chart

We can use the plot() method of pandas DataFrame to create a bar chart for each level of the variable cut.

We see that most of the diamonds in the dataset are of the Ideal cut quality, followed by Premium, Very Good, Good, and Fair.

We can generate the same bar chart using the seaborn data visualization library with more visual aesthetics and customizations. Here we use the catplot() function of seaborn with the parameter kind=count.

Components of the dataset

Since the dataset is stored as a pandas DataFrame object, it has 3 components: Values, Columns and Index.

Values

We can use the values attribute of the DataFrame class to get all values of the dataset into a two-dimensional numpy array.

Columns

We can use the columns attribute of the DataFrame class to get all variable names in the dataset.

Index

We can use the index attribute of the DataFrame class to get the row index of the dataset.

Viewing the distribution of continuous variables

We can plot histograms for each continuous variable to view the distribution. The following figure shows the histogram created for the price variable. Again, I’ve used seaborn to draw the plot. This time, seaborn distplot() function works well.

Histogram of diamond price with mean and median

The following figure shows the histogram created for the depth variable.

Histogram of diamond depth with mean and median

The following figure shows the histogram created for the carat variable.

Histogram of diamond carat with mean and median

Viewing the relationship between continuous variables

To see if there is a relationship between two continuous variables, we can draw a scatterplot. The following figure shows the scatterplot between carat (x-variable) and price (y-variable). I’ve used seaborn scatterplot() function.

Scatterplot between carat and price

Viewing pairwise relationships

The pairplot() function in seaborn creates a plot that shows pairwise relationships of all continuous variables in the dataset. It is also possible to show a subset of variables by specifying the vars parameter inside the pairplot() function. To do that, we pass a list of variables that should be included in the plot.

This is a matrix of plots. The diagonal shows histograms for univariate distributions. The rest of the matrix shows scatterplots for joint relationships.

Summary statistics of continuous variables

Box plots are an excellent way to examine the relationship between the summary statistics of a continuous variable in relation to other categorical variables. Now, suppose we want to see the summary statistics of the price variable classified by another variable— cut quality. A popular way to show such information is to use box plots. This is very easy to do with the seaborn library with just two lines of code!

Box plot of price by cut quality

The following box plot shows the summary statistics of the price variable. This time, price is not classified by a categorical variable.

Box plot of price

You can compare this box plot with the exact summary statistics obtained using the describe() method of pandas DataFrame.

Summary statistics of price

Saving the dataset as an Excel spreadsheet

What if your friend requests this diamonds.csv dataset as an Excel spreadsheet? You should convert diamonds.csv file into diamonds.xlsx file. To do that, you can use the to_excel() method of pandas DataFrame. The to_excel() method stores the data as an excel file. By setting index=False, the row index labels are not saved in the spreadsheet.