Pivot Tables — Data Aggregation Tool in Python | by Tanya Dayanand | Towards Data Science

Exploratory data analysis is an important phase of machine learning projects. The wonderful Pandas library is equipped with several useful functions for this purpose. One among them is pivot_table that summarizes a feature’s values in a neat two-dimensional table. The data summarization tool frequently found in data analysis software, offering a ton of flexibility. DataFrame has a pivot_table method (pandas.DataFrame.pivot_table), and additionally, there is a top-level pandas.pivot_table function(any of them can be used as per the convenience, both results in the same output). Most often we end up using pivot_table with the default parameters. This article will help you achieve more by optimal usage of the default parameters.
The pivot_table() method returns a DataFrame which is an Excel-style pivot table. The pivot table aggregates a table of data by one or more keys, arranging the data in a rectangle with some of the group keys along the rows and some along the columns into a two-dimensional table that provides a multidimensional summarization of the data.
pandas.DataFrame.pivot_table()
Let’s look into the basic level usage of this method by applying it on a dataset. Online Shopping Dataset used in the article can be downloaded from Kaggle. To get into the codes directly, an accompanying notebook is published on Kaggle.
www.kaggle.com
Let’s begin with importing the necessary libraries and loading the dataset. This is a requisite step in every data analysis process.
# Importing necessary librariesimport pandas as pdimport numpy as np# Importing and Loading the data into data framemarket_df = pd.read_csv("../input/market.csv")customer_df = pd.read_csv("../input/customer.csv")product_df = pd.read_csv("../input/product.csv")shipping_df = pd.read_csv("../input/shipping.csv")orders_df = pd.read_csv("../input/order.csv")# Merging the dataframes to create a master_dfdf_1 = pd.merge(market_df, customer_df, how='inner', on='Cust_id')df_2 = pd.merge(df_1, product_df, how='inner', on='Prod_id')df_3 = pd.merge(df_2, shipping_df, how='inner', on='Ship_id')master_df = pd.merge(df_3, orders_df, how='inner', on='Ord_id')
master_df.head()
#Identifying Missing Values in Columnmaster_df.isnull().sum()
Thus, the column Product_Base_Margin has null values.
At this point, we barely have any idea of what dataset looks like. So let’s now look into the usage of pivot_table() in multiple different ways to explore the data further.
The index is the column, grouper, array, or list we’d like to group our data by. The simplest pivot table requires a dataframe and an index . The index features will be displayed in the index column in the resultant table. By default, it will average all the numerical columns data when the value and aggfunc parameters are not specified.
I will be using the Customer_Segment column as the index here:
#a single index#Using pandas.DataFrame.pivot_tablemaster_df.pivot_table(index = 'Customer_Segment')#Same as above - results in same output#Using pandas.pivot_tablepd.pivot_table(master_df, index = 'Customer_Segment')
The function returns a pivot table with Customer_Segment as index and averages the numerical columns. We can see that the pivot table is smart enough to start aggregating the data and summarizing it by grouping based on Customer_Segment.
pivot_table can take multiple features as indexes through a list. This increases the level of granularity in the resultant table and we can get more specific with our findings:
#multiple indexesmaster_df.pivot_table(index =['Customer_Segment','Product_Category'])
Here the pivot_table is grouping the Product_Category with Customer_Segment and giving us a summarized table of the combinations. Now we start to get a glimpse of what a pivot table can do for us.
The value parameter is where we tell the function which features to aggregate on. It is an optional field and if we don’t specify this value, then the function will aggregate on all the numerical features of the dataset. In the previous example of indexes, we saw that the aggregation was done for all numerical columns. Since the value parameter was not specified, pivot_table, by default considered all numerical columns.
Below example compares average Sales across Customer_Segment:
#Single valuemaster_df.pivot_table(values = 'Sales', index = 'Customer_Segment')
The Sales column automatically averages the data but we can do a count or a sum as well( by adding a parameter aggfunc and np.sum orcount ).
Alike index parameter, the values parameter can also take multiple features via a list.
#multiple valuemaster_df.pivot_table(values = ['Order_Quantity','Sales'], index = 'Customer_Segment')
The pivot table aggregates data of both the features Order_Quantity and Sales and groups it with Customer_Segment.
As mentioned before, pivot_table uses mean function (numpy.mean) for aggregating or summarizing data by default. But there are other important function or list of functions to consider. aggfunc is an aggregate function that pivot_table applies to our grouped data. aggfunc (optional) accepts a function or list of functions we’d like to use in our group. The aggregation specification can be a string such as 'sum', 'mean', 'count', 'min', 'max', etc or a function that implements an aggregation (e.g. np.sum(), min(), sum(), etc). Now its time to experiment with the aggfunc parameter.
This first example aggregates values by taking the mean:
#Single aggrigate function(mean) and single valuemaster_df.pivot_table(values = 'Sales', index = 'Customer_Segment', aggfunc = 'mean')
The second example aggregates values by taking the sum. It compares the total number of orders across regions.
#Single aggrigate function(sum) and single valuemaster_df.pivot_table(values = 'Order_Quantity', index = 'Region', aggfunc = 'sum')
The following example aggregates by taking the sum across multiple columns.
#Sum aggregate function is applied to both the valuesmaster_df.pivot_table(values = ['Order_Quantity','Sales'], index = 'Product_Category', aggfunc='sum')
Not only we can specify what aggregating function we want, but we can also specify more than one aggregating function. For example, if we are interested in both the sum and count of Sales, we can specify the functions as a list to the argument aggfunc. Let’s try it out.
#multiple Aggregating Function applied to single columnmaster_df.pivot_table(values = 'Sales', index = 'Product_Category', aggfunc=['sum', 'count'])
We can provide a list of aggregate functions to apply to each value too. Below is the example for the same.
#Sum and Mean aggregate function is applied to both the valuesmaster_df.pivot_table(values = ['Order_Quantity','Sales'], index = 'Product_Category', aggfunc=[np.sum, np.mean])
We can use different aggregate functions for different features too. Just provide a dictionary mapping as an input to the aggfunc parameter with the feature name as the key and the corresponding aggregate function as the value.
In the below example, I will be using sum for the Order_Quantity feature and mean for the Sales feature. Also, notice that I’ve omitted the values keyword; when specifying a mapping for aggfunc, this is determined automatically.
#different aggregate applied to different valuesmaster_df.pivot_table(index = 'Product_Category', aggfunc = {'Order_Quantity':sum, 'Sales':'mean'})
When we give multiple aggregating functions, we would get a multi-indexed data frame as output.
The column parameter displays the values horizontally on the top of the resultant table. Most of the time there is confusion with the pivot_table related to the use of columns and values . Bear in mind, columns are optional, they provide a supplementary way to segment the actual values we care about. The aggregation functions are applied to values we list.
If we want to see Profit broken down by the Customer_Segment, the columns parameter allows us to define one or more columns. Here we are comparing the total profit across product categories and customer segments.
#Single column#Grouping by both rows and columnmaster_df.pivot_table(values = 'Profit',                       index = 'Product_Category',                       columns = 'Customer_Segment',                       aggfunc = 'sum')
Using multiple features as indexes is fine, but using some features as columns will help us to quickly understand the relationship between them. Also, the resultant table can always be better viewed by incorporating the columns parameter of the pivot_table.
#multiple columnsmaster_df.pivot_table(values = 'Profit',                       index = 'Customer_Segment',                       columns = ['Product_Category','Ship_Mode'],                       aggfunc = 'count')
Both columns and the index parameters are optional, but using them effectively will help us to intuitively understand the relationship between the features.
At times it’s useful to compute totals for each grouping. Now, if we want to see some totals of the data, margins=True does that for us. margins is type boolean that adds all rows and columns (e.g. for subtotal / grand totals) and defaults to ‘False’.
#Marginmaster_df.pivot_table(values = ‘Profit’,  index = ‘Product_Category’,  columns = ‘Customer_Segment’,  aggfunc = ‘sum’, margins=True)
The margin label can be specified with the margins_name keyword, which defaults to "All".margins_name is of type string and accepts the name of the row/column that will contain the totals when margins equal True.
#margins_namemaster_df.pivot_table(values = 'Profit',                       index = 'Product_Category',                       columns = 'Customer_Segment',                       aggfunc = 'sum',                       margins=True,                      margins_name ='TOTAL')
These two parameters are both optional and often useful to improve the display.
The NaN’s are a bit distracting. pivot_table helps us to deal with it through the parameters dropna and fill_value. Two of these options, fill_value and dropna, have to do with missing data and are fairly straightforward.
dropna is type boolean, and allows us to drop the null values in the grouped table whose entries are all NaN. It defaults to True.
fill_value is type scalar, and can be used to replace the NaN values in the table with the values that we provide. It defaults to None.
#Displaying NaN values in the table#These can be imputed using fill_valuemaster_df.pivot_table(values = 'Product_Base_Margin',                       index = 'Customer_Name',                      columns = 'Customer_Segment',                       aggfunc = 'mean')
I will be replacing the NaN values with the mean value from the Product_Base_Margin column:
#imputing with mean using fill_valuemaster_df.pivot_table(values = 'Product_Base_Margin',                       index = 'Customer_Name',                      columns = 'Customer_Segment',                       aggfunc = 'mean', fill_value=np.mean(master_df['Product_Base_Margin']))
We don’t have any columns where all entries are NaN, but it’s worth knowing that if we did pivot_table would drop them by default according to dropna definition.
Thus, we can see that pivot_table() is a handy tool, and we can do some interesting analysis with this single line of code. As you build up the pivot table, I think it’s easiest to take it one step at a time. Add items and check each step to verify if you are getting the results you expect and see what presentation makes the most sense for your needs. As soon as you start playing with the data and slowly add the items, you can get a feel for how it works.
We’ve covered the powerful parameters of pivot_table , so you can get a lot out of it if you go experiment using these methods on your project.