Reshape pandas dataframe in Python | Towards Data Science

There are many different ways to reshape a pandas dataframe from wide to long form. But the melt() method is the most flexible and probably the only one you need to use once you learn it well, just like how you only need to learn one method pivot_table() to reshape from long to wide (see my other post below).
towardsdatascience.com
This tutorial will walk you through reshaping dataframes using pd.melt() or the melt method associated with pandas dataframes. In other languages like R, melt is also known as gather. Also, R also has a melt function that works in the same way.
towardsdatascience.com
I highly recommend you try the code in Python while you read this article. Open my DeepNote notebook (you can only run but not edit this notebook) and run the cells while you read this article.
Also, you might want to check out the official pandas documentation and my numpy reshape tutorial:
towardsdatascience.com
It’s easiest to understand what a wide dataframe is or looks like if we look at one and compare it with a long dataframe.
And below is the corresponding dataframe (with the same information) but in the long form:
Before we begin our pd.melt tutorial, let’s recreate the wide dataframe above in Python with pd.DataFrame.
import pandas as pd# create wide dataframedf_wide = pd.DataFrame(  {"student": ["Andy", "Bernie", "Cindy", "Deb"],   "school":  ["Z", "Y", "Z", "Y"],   "english": [10, 100, 1000, 10000],  # eng grades   "math":    [20, 200, 2000, 20000],  # math grades   "physics": [30, 300, 3000, 30000]   # physics grades  })
We melt the dataframe by specifying the identifier columns via id_vars. The “leftover” non-identifier columns (english, math, physics) will be melted or stacked onto each other into one column.
A new indicator column will be created (contains values english, math, physics) and we can rename this new column (cLaSs) via var_name. We can also rename the column in which all the actual grades are contained (gRaDe) via value_name.
print(df_wide)> student school  english   math  physics     Andy      Z       10     20       30   Bernie      Y      100    200      300    Cindy      Z     1000   2000     3000      Deb      Y    10000  20000    30000df_wide.melt(id_vars=["student", "school"],             var_name="cLaSs",  # rename             value_name="gRaDe")  # rename>  student school    cLaSs  gRaDe0     Andy      Z  english     101   Bernie      Y  english    1002    Cindy      Z  english   10003      Deb      Y  english  100004     Andy      Z     math     205   Bernie      Y     math    2006    Cindy      Z     math   20007      Deb      Y     math  200008     Andy      Z  physics     309   Bernie      Y  physics    30010   Cindy      Z  physics   300011     Deb      Y  physics  30000
You can use value_vars to specify which columns you want to melt or stack into column (here, we exclude physics column, so value_vars=["english", "math"]). We also drop the school column from id_vars.
print(df_wide)> student school  english   math  physics     Andy      Z       10     20       30   Bernie      Y      100    200      300    Cindy      Z     1000   2000     3000      Deb      Y    10000  20000    30000df_wide.melt(id_vars="student",              value_vars=["english", "math"],              var_name="cLaSs",  # rename             value_name="gRaDe")  # rename> student    cLaSs  gRaDe0    Andy  english     101  Bernie  english    1002   Cindy  english   10003     Deb  english  100004    Andy     math     205  Bernie     math    2006   Cindy     math   20007     Deb     math  2000
Finally, let’s see what happens if we specify only the student column as the identifier column (id_vars="student") but do not specify which columns you want to stack via value_vars. As a result, all non-identifier columns (school, english, math, physics) will be stacked into one column.
The resulting long dataframe looks wrong because now the cLaSs and gRaDe columns contain values that shouldn’t be there. The point here is to show you how pd.melt works.
print(df_wide)> student school  english   math  physics     Andy      Z       10     20       30   Bernie      Y      100    200      300    Cindy      Z     1000   2000     3000      Deb      Y    10000  20000    30000df_wide.melt(id_vars="student",             var_name="cLaSs",  # rename             value_name="gRaDe")  # rename>  student    cLaSs  gRaDe0     Andy   school      Z1   Bernie   school      Y2    Cindy   school      Z3      Deb   school      Y4     Andy  english     105   Bernie  english    1006    Cindy  english   10007      Deb  english  100008     Andy     math     209   Bernie     math    20010   Cindy     math   200011     Deb     math  2000012    Andy  physics     3013  Bernie  physics    30014   Cindy  physics   300015     Deb  physics  30000
I hope now you have a better understanding of how pd.meltreshapes dataframes. I look forward to your thoughts and comments.
If you find this post useful, follow me and visit my site for more data science tutorials and also my other articles:
towardsdatascience.com
medium.com
towardsdatascience.com
medium.com
towardsdatascience.com
For more posts, subscribe to my mailing list.