Pivot Tables with Pandas. An Excel favorite in Python | by Ryan Lewis | Towards Data Science

Most everyone who has worked with any type of data has likely used Microsoft Excel’s PivotTable function. It’s a quick, user-friendly tool that allows users to to calculate, aggregate, and summarize data sets enabling further analysis of patterns and trends. Excel provides an intuitive GUI that allows analysts to simply click, drag and drop data and easily apply whichever aggregation function they choose. It’s fantastic tool to use and aids when building Excel visualizations for business presentations.
Python’s Pandas library — which specializes in tabular data, similar to Excel — also has a .pivot_table() function that works in the same concept. It’s a powerful method, comes with a lot of customizable parameters, that should be in every analyst’s Python toolbox. It takes some time to understand the syntax behind the method, but once you’re familiar, it’s a lot faster and more efficient than Excel.
Let’s dive into an NBA stats dataset and see how Panda’s .pivot_table() function works!
#importing required librariesimport pandas as pdimport numpy as np#reading in the data & selecting needed columnsdf = pd.read_csv('../Data/NBA_Player_seasonal.csv', index_col = 0)df = df[['Player', 'Season', 'Age', 'Tm','WS', 'G','MP','3P','TRB','AST','STL','BLK', 'TOV', 'PF','PTS']]#displaying first 5 rows of dataframedf.head()
Above, we can see a preview of our dataset. It’s seasonal totals for every NBA & ABA player dating back to the beginning of the league in 1947. It’s pre-sorted and ranked by highest “WS” — win-share, a metric that calculates the estimated number of wins a player contributes to their team. Let’s say we wanted to see this information, but aggregated by team. We could do something like this:
#creating a pivot table where team is the indexdf.pivot_table(index = 'Tm')
By default, pivot_table() brings in all numerical columns and aggregates the data using it’s mean. Let’s say we only care about a few of these columns, and instead of the averages, we want to see totals.
#pivot table with selected columns & summeddf.pivot_table(df[['PTS', 'AST', 'TRB']], #selected columns               index = 'Tm', #indexed by team               aggfunc = np.sum) #aggregated by sum
Now we can view just the statistics we’re interested in and view totals instead of averages. You are able to pass in a library for the aggfunc argument, allowing users to choose which type of aggregation should be performed on each column.
Taking it a step further, pivot_table() allows users to have multiple indexes to further analyze data. Below is an example of the same data set but reshaped as teams statistics over the years.
#multi-index pivot tabledf.pivot_table(df[['PTS', 'AST', 'TRB', 'STL','BLK', 'TOV']],              index = ['Season', 'Tm'],              aggfunc = np.mean).head()
In the above pivot table, we’re aggregating on missing data, even some of the basic statistics weren’t recorded until a few seasons after the inaugural one in 1947. Let’s say we want to fill those with zero’s for our analysis. We can accomplish this with a simple addition to our pivot table parameters.
#filling in NaN values w/ 0'sdf.pivot_table(df[['PTS', 'AST', 'TRB', 'STL','BLK', 'TOV']],              index = ['Season', 'Tm'],              aggfunc = np.mean,              fill_value = 0).head()
This function also enables users to quickly visualize their data in fewer lines of code, than if they were trying to code it directly from the original data set. As an example, let’s plot league-wide seasonal averages over the years via a pivot table line plot.
#plotting pivot tabledf.pivot_table(df[['PTS', 'AST', 'TRB', 'STL','BLK', 'TOV']],              index = ['Season',],                aggfunc = np.mean,                fill_value = 0).plot()
It’s not the prettiest visual, but it’s quick and provides some vital information. You can quickly identity the 1999 lockout seasons where only 50 games were played in the season compared to the usual 82.
These are several of the pivot_table() parameters you can use to customize/reshape your dataset. There are other parameters such as margins which adds totals to columns and rows. Another is columns which provides an additional way to segment your data set. Lastly, each pivot table outputs a new dataframe, that allows you to perform any standard dataframe functions/methods such as filtering on specific criteria, for example if we had only wanted to see Boston Celtics data.
I hope this quick tutorial is helpful to understand how powerful Panda’s pivot table function is! Please feel free to reach out with any questions or leave any feedback.