Pivot Tables — Data Aggregation Tool in Python
By: Tanya Dayanand
Tags: Data Science, Pandas, Machine Learning, Eda, Pivot Tables

Pivot Tables — Data Aggregation Tool in Python

Multiple ways to effectively use the pivot_table() function in Python

Photo by Markus Winkler on Unsplash

Exploratory data analysis is an important phase of machine learning projects. The wonderful Pandas library is equipped with several useful functions for this purpose. One among them is pivot_table that summarizes a feature’s values in a neat two-dimensional table. The data summarization tool frequently found in data analysis software, offering a ton of flexibility. DataFrame has a pivot_table method (pandas.DataFrame.pivot_table), and additionally, there is a top-level pandas.pivot_table function(any of them can be used as per the convenience, both results in the same output). Most often we end up using pivot_table with the default parameters. This article will help you achieve more by optimal usage of the default parameters.

pivot_table()

The pivot_table() method returns a DataFrame which is an Excel-style pivot table. The pivot table aggregates a table of data by one or more keys, arranging the data in a rectangle with some of the group keys along the rows and some along the columns into a two-dimensional table that provides a multidimensional summarization of the data.

Syntax

pandas.DataFrame.pivot_table()

Parameters

Basic usage of pivot_table()

Let’s look into the basic level usage of this method by applying it on a dataset. Online Shopping Dataset used in the article can be downloaded from Kaggle. To get into the codes directly, an accompanying notebook is published on Kaggle.

Importing the dataset

Let’s begin with importing the necessary libraries and loading the dataset. This is a requisite step in every data analysis process.

# Importing necessary libraries import pandas as pd

import numpy as np # Importing and Loading the data into data frame market_df = pd.read_csv("../input/market.csv")

customer_df = pd.read_csv("../input/customer.csv")

product_df = pd.read_csv("../input/product.csv")

shipping_df = pd.read_csv("../input/shipping.csv")

orders_df = pd.read_csv("../input/order.csv") # Merging the dataframes to create a master_df df_1 = pd.merge(market_df, customer_df, how='inner', on='Cust_id')

df_2 = pd.merge(df_1, product_df, how='inner', on='Prod_id')

df_3 = pd.merge(df_2, shipping_df, how='inner', on='Ship_id')

master_df = pd.merge(df_3, orders_df, how='inner', on='Ord_id')

Extracting first few rows

master_df.head()

Calculating the number of null values

#Identifying Missing Values in Column

master_df.isnull().sum()

Thus, the column Product_Base_Margin has null values.

At this point, we barely have any idea of what dataset looks like. So let’s now look into the usage of pivot_table() in multiple different ways to explore the data further.

1. group data using index in a pivot table

The index is the column, grouper, array, or list we’d like to group our data by. The simplest pivot table requires a dataframe and an index . The index features will be displayed in the index column in the resultant table. By default, it will average all the numerical columns data when the value and aggfunc parameters are not specified.

I will be using the Customer_Segment column as the index here:

#a single index

#Using pandas.DataFrame.pivot_table

master_df.pivot_table(index = 'Customer_Segment') master_df.pivot_table(index = 'Customer_Segment') #Same as above - results in same output

#Using pandas.pivot_table

pd.pivot_table(master_df, index = 'Customer_Segment') pd.pivot_table(master_df, index = 'Customer_Segment')

The function returns a pivot table with Customer_Segment as index and averages the numerical columns. We can see that the pivot table is smart enough to start aggregating the data and summarizing it by grouping based on Customer_Segment.

Run a pivot_table() with multiple indexes

pivot_table can take multiple features as indexes through a list. This increases the level of granularity in the resultant table and we can get more specific with our findings:

#multiple indexes

master_df.pivot_table(index =['Customer_Segment','Product_Category'])

Here the pivot_table is grouping the Product_Category with Customer_Segment and giving us a summarized table of the combinations. Now we start to get a glimpse of what a pivot table can do for us.

2. Aggregate on specific features with values

The value parameter is where we tell the function which features to aggregate on. It is an optional field and if we don’t specify this value, then the function will aggregate on all the numerical features of the dataset. In the previous example of indexes, we saw that the aggregation was done for all numerical columns. Since the value parameter was not specified, pivot_table, by default considered all numerical columns.

Below example compares average Sales across Customer_Segment:

#Single value

master_df.pivot_table(values = 'Sales', index = 'Customer_Segment')

The Sales column automatically averages the data but we can do a count or a sum as well( by adding a parameter aggfunc and np.sum or count ).

Aggregate on multiple features

Alike index parameter, the values parameter can also take multiple features via a list.

#multiple value

master_df.pivot_table(values = ['Order_Quantity','Sales'], index = 'Customer_Segment')

The pivot table aggregates data of both the features Order_Quantity and Sales and groups it with Customer_Segment.

3. Manipulating the data using aggfunc

As mentioned before, pivot_table uses mean function ( numpy.mean ) for aggregating or summarizing data by default. But there are other important function or list of functions to consider. aggfunc is an aggregate function that pivot_table applies to our grouped data. aggfunc (optional) accepts a function or list of functions we’d like to use in our group. The aggregation specification can be a string such as 'sum' , 'mean' , 'count' , 'min' , 'max' , etc or a function that implements an aggregation (e.g. np.sum() , min() , sum() , etc). Now its time to experiment with the aggfunc parameter.

This first example aggregates values by taking the mean:

#Single aggrigate function(mean) and single value

master_df.pivot_table(values = 'Sales', index = 'Customer_Segment', aggfunc = 'mean')

The second example aggregates values by taking the sum. It compares the total number of orders across regions.

#Single aggrigate function(sum) and single value

master_df.pivot_table(values = 'Order_Quantity', index = 'Region', aggfunc = 'sum')

The following example aggregates by taking the sum across multiple columns.

#Sum aggregate function is applied to both the values

master_df.pivot_table(values = ['Order_Quantity','Sales'], index = 'Product_Category', aggfunc='sum')

pivot_table with multiple Aggregating Function

Not only we can specify what aggregating function we want, but we can also specify more than one aggregating function. For example, if we are interested in both the sum and count of Sales, we can specify the functions as a list to the argument aggfunc. Let’s try it out.

#multiple Aggregating Function applied to single column

master_df.pivot_table(values = 'Sales', index = 'Product_Category', aggfunc=['sum', 'count'])

We can provide a list of aggregate functions to apply to each value too. Below is the example for the same.

#Sum and Mean aggregate function is applied to both the values

master_df.pivot_table(values = ['Order_Quantity','Sales'], index = 'Product_Category', aggfunc=[np.sum, np.mean])

Different aggregation function for different features

We can use different aggregate functions for different features too. Just provide a dictionary mapping as an input to the aggfunc parameter with the feature name as the key and the corresponding aggregate function as the value.

In the below example, I will be using sum for the Order_Quantity feature and mean for the Sales feature. Also, notice that I’ve omitted the values keyword; when specifying a mapping for aggfunc , this is determined automatically.

#different aggregate applied to different values

master_df.pivot_table(index = 'Product_Category', aggfunc = {'Order_Quantity':sum, 'Sales':'mean'})

When we give multiple aggregating functions, we would get a multi-indexed data frame as output.

4. relationship between features with columns parameter

The column parameter displays the values horizontally on the top of the resultant table. Most of the time there is confusion with the pivot_table related to the use of columns and values . Bear in mind, columns are optional, they provide a supplementary way to segment the actual values we care about. The aggregation functions are applied to values we list.

If we want to see Profit broken down by the Customer_Segment, the columns parameter allows us to define one or more columns. Here we are comparing the total profit across product categories and customer segments.

#Single column

#Grouping by both rows and column

master_df.pivot_table(values = 'Profit',

index = 'Product_Category',

columns = 'Customer_Segment',

aggfunc = 'sum')

pivot_table() with multiple columns

Using multiple features as indexes is fine, but using some features as columns will help us to quickly understand the relationship between them. Also, the resultant table can always be better viewed by incorporating the columns parameter of the pivot_table.

#multiple columns

master_df.pivot_table(values = 'Profit',

index = 'Customer_Segment',

columns = ['Product_Category','Ship_Mode'],

aggfunc = 'count')

Both columns and the index parameters are optional, but using them effectively will help us to intuitively understand the relationship between the features.

5. Adding total rows/columns

At times it’s useful to compute totals for each grouping. Now, if we want to see some totals of the data, margins=True does that for us. margins is type boolean that adds all rows and columns (e.g. for subtotal / grand totals) and defaults to ‘False’.

#Margin

master_df.pivot_table(values = ‘Profit’,

index = ‘Product_Category’,

columns = ‘Customer_Segment’,

aggfunc = ‘sum’, margins=True)

The margin label can be specified with the margins_name keyword, which defaults to "All" . margins_name is of type string and accepts the name of the row/column that will contain the totals when margins equal True.

#margins_name

master_df.pivot_table(values = 'Profit',

index = 'Product_Category',

columns = 'Customer_Segment',

aggfunc = 'sum',

margins=True,

margins_name ='TOTAL')

These two parameters are both optional and often useful to improve the display.

6. Handling missing data

The NaN’s are a bit distracting. pivot_table helps us to deal with it through the parameters dropna and fill_value. Two of these options, fill_value and dropna , have to do with missing data and are fairly straightforward.

dropna is type boolean, and allows us to drop the null values in the grouped table whose entries are all NaN . It defaults to True.

is type boolean, and allows us to drop the null values in the grouped table whose entries are all . It defaults to True. fill_value is type scalar, and can be used to replace the NaN values in the table with the values that we provide. It defaults to None.

#Displaying NaN values in the table

#These can be imputed using fill_value

master_df.pivot_table(values = 'Product_Base_Margin',

index = 'Customer_Name',

columns = 'Customer_Segment',

aggfunc = 'mean')

I will be replacing the NaN values with the mean value from the Product_Base_Margin column:

#imputing with mean using fill_value

master_df.pivot_table(values = 'Product_Base_Margin',

index = 'Customer_Name',

columns = 'Customer_Segment',

aggfunc = 'mean', fill_value=np.mean(master_df['Product_Base_Margin']))

We don’t have any columns where all entries are NaN , but it’s worth knowing that if we did pivot_table would drop them by default according to dropna definition.

Conclusion

Thus, we can see that pivot_table() is a handy tool, and we can do some interesting analysis with this single line of code. As you build up the pivot table, I think it’s easiest to take it one step at a time. Add items and check each step to verify if you are getting the results you expect and see what presentation makes the most sense for your needs. As soon as you start playing with the data and slowly add the items, you can get a feel for how it works.