Summarizing data with Pandas crosstab | by B. Chen | Towards Data Science

The most immediate effect of summarizing data is to take data that may be overwhelming to work with, and reduce it to a few key summary values that can be viewed, often in a table or plot.
Source from Introduction to Data Science [1]
Data summarization is a process to present the summary of generated data in an easily comprehensible and informative manner. Pandas provides several options for summarizing data: groupby, pivot_table, and crosstab.
We have previously covered the Pandas pivot_table() and groupby(), this article will explore the Pandas crosstab() and how to use it for your data analysis. This article is structured as follows:
Simplest crosstab tableDisplaying multiple rows and columnsShowing total with marginsGetting the percentage of occurs with normalizeSpecifying values and performing aggregationHandling missing valuesVisualization with Seaborn heatmap
Simplest crosstab table
Displaying multiple rows and columns
Showing total with margins
Getting the percentage of occurs with normalize
Specifying values and performing aggregation
Handling missing values
Visualization with Seaborn heatmap
Please check out the Notebook for source code. More tutorials are available from Github Repo.
For demonstration, we will be working on coffee sales data. Let’s import some libraries and load data to get started:
import pandas as pddef load_data():    return pd.read_csv('coffee_sales.csv', parse_dates=['order_date'])df = load_data()df.head()
When you create a crosstab table, you’ll need to specify what you want on the rows and how to split the columns. By default, it is calculating the number of observations in each combination. In our example, let’s use the region as our row index and split the columns by product_category.
pd.crosstab(df['region'], df['product_category'])
In some cases, you would prefer a custom label for display purposes. To do that, we can include the specific rownames and colnames:
pd.crosstab(    df['region'],     df['product_category'],     rownames=['US Region'],     colnames=['Product Category'])
One of the most useful features of crosstab is that you can pass in multiple indexes and columns, and it does all the grouping for us. For example, if we want to see how the data is distributed by market, we can include the market column by including it in a list:
pd.crosstab(    df['region'],     [df['product_category'], df['market']])
Similarly, we can specify a custom label for display purposes. Both rownames and colnames accept a list of labels with each corresponding to the row and column.
pd.crosstab(    df['region'],     [df['product_category'], df['market']],    rownames=['US Region'],     colnames=['Product Category', 'Market'])
Another common display in a crosstab table is to show the subtotals for each row and column. We can add them by setting the margins=True argument:
pd.crosstab(df['region'], df['product_category'], margins = True)
By default, the subtotals are displayed with the label “All”. We can specify a custom label using the margins_name argument:
pd.crosstab(    df['region'], df['product_category'],     margins = True,     margins_name = 'Subtotals')
One of the common tasks in data analysis is to understand the percentage of occurs. This can be accomplished by setting the normalize argument to True:
pd.crosstab(df['region'], df['product_category'], normalize = True)
If we prefer the result to be formatted with a percent sign (%), we can call style.format('{:.2%}') as follows:
pd.crosstab(    df['region'],     df['product_category'],     normalize = True).style.format('{:.2%}')
In addition, this calculation is applied to the subtotals when the margins is True:
pd.crosstab(    df['region'],     df['product_category'],     margins = True,     normalize = True).style.format('{:.2%}')
The normalize argument accepts the other two values 'index' and 'columns' :
If 'index' is given, it will normalize over each row.
If 'columns' is given, it will normalize over each column.
pd.crosstab(    df['region'],     df['product_category'],     margins = True,     normalize = 'columns').style.format('{:.2%}')
pd.crosstab(    df['region'],     df['product_category'],     margins = True,     normalize = 'index').style.format('{:.2%}')
By default, crosstab performs the count aggregation function on all numerical columns and returns the result. To explicitly specify the columns we care about and perform aggregation, use the values and aggfunc arguments.
pd.crosstab(    df['region'],     df['product_category'],     values = df['cost'],    aggfunc = 'mean')
By specifying values = df['cost'] and aggfunc = 'mean' , we are telling Pandas to apply the mean function to the cost of all the combinations of the data.
The result will retain 6 decimal places by default. To round the result, we can call round():
pd.crosstab(    df['region'],     df['product_category'],     values = df['cost'],    aggfunc = 'mean').round(2)
You probably notice a NaN value from the previous output. We have got that because there aren't any Tea sales in the South.
Unlike privot_table has a built-in fill_value argument to replace any missing value, crosstab doesn’t support it. We have to use other methods to handle missing values, for example:
pd.crosstab(    df['region'],     df['product_category'],     values = df['cost'],    aggfunc = 'mean',).fillna(0)
However, if the crosstab output includes rows or columns whose entries are all NaN, we could use the argument drop_na=True to drop them.
Honestly, I didn't know this until I read the Pandas Crosstab Explained article [2]. Sometimes, a graph is worth a thousand words and this is a really handy feature to visually summarize the crosstab output. What we need to do is simply pass the crosstab output to a seaborn heatmap:
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsplt.figure(figsize=(8, 6))sns.heatmap(    pd.crosstab(        df['region'],         [df['product_category'], df['market']],        rownames=['US Region'],         colnames=['Product Category', 'Market'],        values = df['cost'],        aggfunc = 'mean'    ),     annot = True)plt.show()
In this article, we have covered 7 use cases about summarizing data with the Pandas crosstab() method. It’s a very useful tool and one of the top favorite methods for Data Summarization during Exploratory Data Analysis.
After reading this article, you should be able to incorporate it in your own data analysis. I hope you enjoyed this article and learned something new and useful.
Thanks for reading. Please check out the Notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning. More tutorials are available from the Github Repo.
[1] Introduction to Data Science: Data summaries
[2] Pandas Crosstab explained from pbpython.com