A Practical Introduction to Pandas pivot_table() function | by B. Chen | Towards Data Science

Pivot tables are one of Excel’s most powerful features. A pivot table allows us to draw insights from data. Pandas provides a similar function called pivot_table(). Pandas pivot_table() is a simple function but can produce very powerful analysis very quickly.
In this article, we’ll explore how to use Pandas pivot_table() with the help of examples. Examples cover the following tasks:
Simplest Pivot tableSpecifying values and performing aggregationSeeing break down using columnsReplacing missing valuesDisplaying multiple values and adjusting viewShowing totalGenerating a monthly report
Simplest Pivot table
Specifying values and performing aggregation
Seeing break down using columns
Replacing missing values
Displaying multiple values and adjusting view
Showing total
Generating a monthly report
Please check out the Notebook for the source code. More tutorials are available from Github Repo.
In this tutorial, we will be working on coffee sales data. You can download it from my Github repo.
Let’s import some libraries and load data to get started.
import pandas as pddef load_data():    return pd.read_csv('coffee_sales.csv', parse_dates=['order_date'])df = load_data()df.head()
We created a function load_data() to load coffee_sales.csv file with column 'order_date' as date datatype.
All columns are pretty self-explanatory.
The simplest pivot table must have an index. In our example, let’s use the region as our index. By default, it is performing the 'mean' aggregation function on all available numerical columns.
df.pivot_table(index='region')
To display multiple indexes, we can pass a list to index:
df.pivot_table(index=['region', 'product_category'])
The values to index are the keys to group by on the pivot table. You can change the order of the values to get a different visual representation, for example, we want to take a look at average values by grouping the region with the product_category.
df.pivot_table(index=['product_category', 'region'])
By default, pivot_table performs the mean aggregation function on all numerical columns and returns the result. To explicitly specify the columns we care about, use the values argument.
df.pivot_table(index=['region'], values=['sales'])
To perform an aggregation other than mean , we can pass a valid string function to aggfunc , for example, let’s do a sum:
df.pivot_table(index=['region'], values=['sales'], aggfunc='sum')
aggfunc can be a dict, and below is the dict equivalent.
df.pivot_table(    index=['region'],     values=['sales'],     aggfunc={ 'sales': 'sum' })
aggfunc can be a list of functions, and below is an example to display the sum and count
df.pivot_table(    index=['region'],     values=['sales'],     aggfunc=['sum', 'count'])# The dict equivalent# aggfunc={ 'sales': ['sum', 'count']}
If we would like to see sales broken down by product_category, the columns argument allows us to do that
df.pivot_table(    index=['region'],     values=['sales'],     aggfunc='sum',     columns=['product_category'])
You probably notice a NaN value from the previous output. We have got that because there aren't any Tea sales in the South. If we want to replace it, we could use fill_value argument, for example, to set NaN to 0.
df.pivot_table(    index=['region'],     values=['sales'],     aggfunc='sum',     columns=['product_category'],    fill_value=0)
If we would like to see the sum of cost as well, we can add the cost column to the values list.
df.pivot_table(    index=['region'],     values=['sales', 'cost'],     aggfunc='sum',     columns=['product_category'],     fill_value=0)
This does work, but the visual representation doesn’t seem to be useful when we would like to compare the cost and sales side by side. To get a better visual representation, we can move product_category from the columns and add to the index.
df.pivot_table(    index=['region', 'product_category'],     values=['sales', 'cost'],     aggfunc='sum',     fill_value=0)
Now, the pivot table is much convientent to see the difference between sales and cost side by side.
Another common display in pivot table is to show the total. In Panda pivot_table() , we can simply pass margins=True:
df.pivot_table(    index=['region', 'product_category'],     values=['sales', 'cost'],     aggfunc='sum',     fill_value=0,     margins=True)
Raw sales data is rarely aggregated by month for us. This type of data is often captured by the day. However, managers often want reports by month instead of detail by day. To generate a monthy sales report with Panda pivot_table(), here are the steps:
(1) defines a groupby instruction using Grouper() with key='order_date' and freq='M'
(2) defines a condition to filter the data by year, for example 2010
(3) Use Pandas method chaining to chain the filtering and pivot_table().
month_gp = pd.Grouper(key='order_date',freq='M')cond = df["order_date"].dt.year == 2010(    df[cond]    .pivot_table(index=['region','product_category'],        columns=[month_gp],        values=['sales'],        aggfunc=['sum']))
Thanks for reading.
Please checkout the notebook on my Github for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
When to use Pandas transform() function
Using Pandas method chaining to improve code readability
Working with datetime in Pandas DataFrame
Working with missing values in Pandas
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More can be found from my Github