Master the when and how of Pandas pivot_table() | Towards Data Science

Here is the worst-case scenario: You are watching a paid course and the instructor is talking about a certain topic. Then, all of sudden, he introduces a completely new function saying “This function/method is perfect in this case, it is very easy so just check out its documentation for more details”. You say OK, go to the documentation, and don’t even know what you are looking at.
You feel frustrated, go to read some articles or StackOverflow threads, and sometimes come back feeling even worse. Believe me, that happens to everyone. This article is specifically about the case of hard pandas functions.
Mostly, the reason why most sources do not cover some advanced functions of pandas is that they are very case-specific. When you learn the basic methods and functions, you learn them in their own context like on toy datasets. For harder functions, they are difficult to explain, and would be hard to create the context they are useful in.
Such functions are often in the toolbox of more experienced scientists. They make the difference where you use them in such a way that the function solves the problem you are having with one line of code so elegantly. This post is about the three of them: pivot_table(), stack() and unstack().
1. Introduction  2. Setup 3. Pandas pivot_table(), with comparison to groupby() 4. Pandas stack() 5. Pandas unstack()
I hyperlinked the documentation for each function at their first appearance. Check them out for more details.
There should be one — and preferably only one — obvious way to do it.
The above is a quote from the Zen of python. Python wants to have only one obvious solution for a single problem. But, pandas deliberately avoids this. Often in pandas, there are several ways to do one operation.
pivot_table() is an example. It is a complete and sometimes a better alternative to groupby() function. The difference is the shape of the result. groupby() returns a Series object while pivot_table() gives an easy-to-work dataframe.
Let’s work on a problem and give the solutions using both functions. I will load the tips dataset from seaborn:
tips = sns.load_dataset('tips')tips.head()
We want to find the sum of all bills for each gender:
Let’s compare the syntax of the two functions. In groupby(), we pass the column we want to group by in the parentheses and in pivot_table() the equivalent parameter is the index. In groupby(), to choose the column to aggregate, we use subsetting with brackets while in pivot_table() we pass it to values. Finally, to choose the aggregating function, we use method chaining in groupby() whereas, pivot_table() provides aggfunc argument.
When I wrote an article about project setup for DS and ML, I researched a lot of notebooks. What I found surprising was that many people used groupby() and used .reset_index() function to turn the results of groupby() into a dataframe, let's explore further to find out why:
result = tips.groupby('sex')['total_bill'].sum().reset_index()result
If you use the pivot_table() you don't have to use reset_index() to convert the result into a dataframe. groupby() results are not as easy to work with as dataframes. Let's see how to group by multiple columns and aggregate with multiple functions:
tips.groupby(['sex', 'day'])['total_bill']\            .agg([np.mean, np.median, np.sum]).reset_index()
tips.pivot_table(values='total_bill',                  index=['sex', 'day'],                  aggfunc=[np.mean, np.median, np.sum])
Both functions return a dataframe for multiple columns. But, even though for a single column pivot_table() is better, using the reset_index() on the groupby result gives a much nicer dataframe. Maybe that's why Kagglers prefer groupby().
In pivot_table(), sometimes you can use columns parameter instead of index (or sometimes both) to display each group as a column. But if you pass multiple arguments to columns, the result will be a long dataframe with a single row.
Another difference between groupby() and pivot_table() would be fill_value parameter. Sometimes, when you group by multiple variables, there won't be matching cells for the result. In such cases, groupby() puts NaNs but in pivot_table() you can control this behavior:
tips.head()
When would you want to use pivot_table()? As I said previously, it can sometimes be a better alternative for groupby(). It is also a personal preference when it comes to syntax. An obvious example would be choosing pivot_table() because it has some other parameters which are not available in groupby(). I already covered fill_value, but there are others like margins. You can learn more about it in the documentation 😁.
Please, read the second part of the article to find out more differences between functions that compute grouped summary statistics:
towardsdatascience.com
When used, stack() returns a reshaped dataframe with a multi-level index. The inner-most levels are created by pivoting the columns of the dataframe. It is best we start with an example. I will load the cars dataset and subset it for better understanding:
Let’s see how we pivot the dataframe so that the columns are now indexes:
When we use the stack() function on this dataframe, the result will have multi-level indexes, name being the outer level and weight, horsepower in the inner-level:
Here, the original dataframe had a single level column names. That’s why the resulting dataframe was pandas.Series rather than a dataframe.
Remember that stack() function always pivots columns to the inner level index. If there are no columns left, meaning if the final data is a Series, stack() will not work. Let's try stacking the above Series:
stacked_cars.stack()AttributeError: 'Series' object has no attribute 'stack'
A more complex example of stack() would be when the column names are given as multi-level indexes. Let's get back to one of our pivoted tables:
As you see, the column names have a two-level hierarchy. You can access a column with a multi-level name like this:
Let’s use stack() on this dataframe and see what happens:
multi_name.stack()
Now, total_bill which is the inner-level column name became an index. You can control which column level you want to stack. Let's see how you would stack the outer level column name:
multi_name.stack(level=0)
As you see, using different levels gives differently-shaped dataframes. By default, level is set to 1.
As the name suggests, unstack() does exactly the opposite of stack(). It takes multi-level indexed Series or a dataframe and pivots the indexes to become columns. If we unstack the stacked cars Series we will get back the original dataframe:
>>> print('Unstacked Dataframe:')>>> stacked_cars.unstack()Unstacked Dataframe:
Perhaps, the most obvious use case for unstacking is when we use the groupby() function. Although the process is not possible when we group by one variable, unstacking proves very useful for grouping by multiple variables. Let's get back to our tips dataset:
tips.head()
The result is a Series with 4-level indexes. This is not what we want. Let’s unstack so that it is easier to use:
multiple_groups.unstack()
The result still has multi-level indexes. That’s because unstack() works on one index-level at a time. Let's call it once more to get a dataframe with a single index level:
multiple_groups.unstack().unstack()
I think you already see the effect unstack() has when you use it with groupby(). Multi-level indexes are always hard to work with. Always try to avoid them unless you absolutely have to. One way you can do it is with unstack().
Although, stack() is not used often, I showed you the basic syntax and the general use so that you will have a better grasp on unstack().
This post was only about 3 functions. In the near future, I am planning to write similar articles for other functions which might be a little difficult to understand and integrate into daily use. So, stay tuned folks!
Read more articles related to the topic: