Reshape pandas dataframe | Towards Data Science

There are many different ways to reshape a pandas dataframe from long to wide form. But the pivot_table() method is the most flexible and probably the only one you need to use once you learn it well, just like how you only need to learn one method melt to reshape from wide to long (see my other post below).
towardsdatascience.com
This tutorial will walk you through reshaping dataframes using pd.pivot_table or the pivot_table method associated with pandas dataframes. In other languages like R, pivot is also known as spread or dcast.
I highly recommend you try the code in Python while you read this article. Try running this tutorial on my shared DeepNote notebook (you can only run but not edit this notebook).
Also, you might be interested a similar tutorial that describes the reverse (reshape from wide to long) with pd.melt and my numpy reshape tutorial.
towardsdatascience.com
It’s easiest to understand what a long dataframe is or looks like if we look at one and compare it with a wide dataframe.
And below is the corresponding dataframe (with the same information) but in the wide form:
Before we begin our pd.pivot_table tutorial, let’s recreate the wide dataframe above in Python with pd.DataFrame. Remember, you can also follow along with my shared notebook.
df_long = pd.DataFrame({        "student":            ["Andy", "Bernie", "Cindy", "Deb",             "Andy", "Bernie", "Cindy", "Deb",             "Andy", "Bernie", "Cindy", "Deb"],        "school":            ["Z", "Y", "Z", "Y",             "Z", "Y", "Z", "Y",             "Z", "Y", "Z", "Y"],        "class":            ["english", "english", "english", "english",             "math", "math", "math", "math",             "physics", "physics", "physics", "physics"],        "grade":            [10, 100, 1000, 10000,             20, 200, 2000, 20000,             30, 300, 3000, 30000]})
We often want to keep the identifier columns as they are (index=["student", "school"]), but pivot or “split” a column’s values (values="grade") based on another column (columns="class"). Compare the original and pivoted dataframes below and you’ll understand what that means.
df_long.pivot_table(index=["student", "school"],                     columns='class',                     values='grade')
Each unique value in the class column will be a new column (english, math, physics) in the pivoted/wide dataframe. We can also provide a list to the columns parameter.
To get rid of the multi-index, use reset_index().
You can also aggregate each resulting row and column by specifying margins=True (default False).
df_long.pivot_table(index=["student", "school"],                     columns='class',                     values='grade',                    margins=True,  # add margins                    aggfunc='sum')  # sum margins (rows/columns)
Here we aggregate by computing the sum via aggfunc='sum' (default 'mean').
There are many other aggregation functions you can use (e.g.,'median' 'sum' 'max'). You can also specify multiple functions as a list (e.g.,aggfunc=['mean', 'sum']).
If we don’t specify any columns via columns, all remaining non-identifier numeric columns (only grade in this dataframe) will be pivoted (long to wide).
df_long.pivot_table(index=["student", "school"])
In the original long data, each student has four grades (english, math, physics), yet in the pivot_tableexample above, each student only has one grade after pivoting.
Why and how does it work? If you remember from the example above, the default is aggfunc='mean'. Thus, what the function did was it grouped the data by student and school (via index=["student", "school"]), and computed the mean value for each group.
If you use the groupby method associated with the pandas dataframe, you will get the same result as above.
df_long.groupby(['student', 'school']).mean().reset_index()  student school  grade0    Andy      Z     201  Bernie      Y    2002   Cindy      Z   20003     Deb      Y  20000
If you change the default aggregation function (e.g., aggfunc='max'), you’ll get different results. The examples below show you how to specify different aggregation functions and also show you how groupby can be used to perform the same pivot.
Note that you’ll also see the class that is associated with each ‘max’ and ‘first’ value.
df_long.pivot_table(index=["student", "school"],                     aggfunc=['max', 'first'])# groupby equivalent# df_long.groupby(["student", "school"]).agg(['max', 'first'])                    max           first                         class  grade    class  gradestudent school                                Andy    Z       physics     30  english     10Bernie  Y       physics    300  english    100Cindy   Z       physics   3000  english   1000Deb     Y       physics  30000  english  10000
The final example shows you what happens when you pivot multiple columns (columns=['school', 'class']) and you can also deal with missing values after pivoting by replacing the NaN values with another value (-5 in the example below).
df_long.pivot_table(index="student",                     columns=['school', 'class'],                     values='grade',                     fill_value=-5)  # replace NaN with -5
The NaN values are expected because each student belongs to only one school (Y or Z). For example, Andy is in school Z and therefore doesn’t have grades in the Y columns.
I hope now you have a better understanding of how pd.pivot_table reshapes dataframes. I look forward to your thoughts and comments.
If you find this post useful, follow me and visit my site for more data science tutorials and also my other articles:
towardsdatascience.com
towardsdatascience.com
medium.com
towardsdatascience.com
medium.com
For more posts, subscribe to my mailing list.