3 easy ways to reshape pandas DataFrame | by Zolzaya Luvsandorj | Towards Data Science

Data comes in different shapes and sizes. As professionals working with data, we often need to reshape the data to a form that is more suitable for the task at hand. In this post, we will look at 3 simple ways to reshape a DataFrame.
Let’s start by importing libraries and loading a sample wide dataset:
import numpy as npimport pandas as pdfrom seaborn import load_dataset# Load sample datawide = load_dataset('penguins')\        .drop(columns=['sex', 'island', 'culmen_length_mm'])\        .sample(n=3, random_state=1).sort_index()\        .reset_index().rename(columns={'index': 'id'})wide
We can reshape the data to a long format with stack() like this:
long = wide.set_index('id').stack().to_frame().reset_index()\           .rename(columns={'level_1': 'variable', 0: 'value'})long
It gets the job done but this is quite verbose and not very elegant. Luckily, transforming the data to a long format becomes easy with melt():
long = wide.melt(id_vars='id')long
Voila! It’s quite simple, isn’t it? Of note, wide.melt(id_vars=’id’) can also be written as pd.melt(wide, id_vars='id').
It’s always important to apply what we learn to consolidate our knowledge. One of my favourite practical application of melt() that you may also find useful is to use it to format correlation matrix. Although we only have three records in wide, to illustrate the idea, let’s do a correlation table:
corr = wide.drop(columns='id').corr()corr
This format is useful as we can turn the matrix into heatmaps to visualise the correlations. But often the matrix or the heatmap is not enough if you want to drill into specifics and find variables whose correlations are above a certain threshold. Turning the matrix into a long format makes that task a whole lot easier:
corr.reset_index().melt(id_vars='index')
Now with this long data, we can easily filter by ‘value’ to find correlations between desired values. We will format the data a bit more and filter correlations between 0.9 and 1:
corr.reset_index().melt(id_vars='index')\    .rename(columns={'index': 'variable1',                      'variable': 'variable2',                      'value': 'correlation'})\    .sort_values('correlation', ascending=False)\    .query('correlation.between(.9,1, inclusive=False)',            engine='python') # workaround of this bug
You can even go ahead and remove the duplicate correlations too. This correlation formatting is especially useful when you have bigger datasets with many numerical features.
On the other hand, sometimes the data comes in a long format and we need to reshape it to a wide data. Let’s now do the opposite of what we did previously. Similar to the previous section, we will start the transformation with unstack():
long.set_index(['id', 'variable']).unstack()
The same transformation can be done using pivot() as below:
long.pivot(index='id', columns='variable', values='value')
This is not necessarily more concise but it probably is little easier to work with compared to unstack(). By now, you probably have noticed that melt() is to pivot() as stack() is to unstack().
A possible practical application of reshaping data to wide format is if your data is in an Entity-Attribute-Value (a.k.a. EAV) format similar to this:
eav = pd.DataFrame({'entity': np.repeat([10,25,37, 49], 2),                    'attribute': ['name', 'age']*4,                    'value': ['Anna', 30, 'Jane', 40,                               'John', 20, 'Jim', 50]})eav
Reshaping the data into a format where each row represents entity (e.g. customer) can be done using pivot():
eav.pivot(index='entity', columns='attribute', values='value')
Next time you know how to reshape a long data!
We learned how to reshape from long to wide with melt(). But with wide_to_long() function, reshaping becomes easier compared to melt() in some instances. Here’s one example:
pop = pd.DataFrame({'country':['Monaco', 'Liechtenstein',                                'San Marino'],                            'population_2016' : [38070, 37658, 33504],                   'population_2017' : [38392, 37800, 33671],                   'population_2018' : [38682, 37910, 33785]})pop
Using melt(), we can reshape the data and format it as follows:
new = pop.melt(id_vars='country')\         .rename(columns={'variable': 'year',                           'value': 'population'})new['year'] = new['year'].str.replace('population_', '')new
With wide_to_long(), it’s much simpler to get the same output:
pd.wide_to_long(pop, stubnames='population', i='country', j='year',                 sep='_').reset_index()
When using the function, it’s good to understand these three main terms: a stub name (stubnames), a suffix and a separator (sep). While these terms may be self-explanatory, an example may clarify them: population is a stub name, 2017 is a suffix and _ is a separator. A new column name for the suffix is passed to parameter j and a unique identifier column name is passed to parameter i. Without reset_index(), the output would look like the following where the unique identifier and the suffix column are in the index:
pd.wide_to_long(pop, stubnames='population', i='country', j='year',                 sep='_')
By default, suffix is set up to be numerical values. So this worked fine in our previous example. But it may not work for a data like this:
iris = load_dataset('iris').head()iris
This time, there are two stub names: sepal and petal. We will pass both in a list to stubnames when reshaping. The suffixes (i.e. length and width) are no longer numeric so we will need to specify that pattern using regular expression in suffix argument.
pd.wide_to_long(iris.reset_index(), stubnames=['sepal', 'petal'],                 i='index', j='Measurement', sep='_', suffix='\D+')
Hopefully these two examples have illustrated how useful wide_to_long() can be when used in the right settings.
Voila! These were the 3 easy ways to reshape pandas data! Here’re their official documentation if you want to learn more: pd.melt(), pd.pivot() and pd.wide_to_long().
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading this article. If you are interested, here are links to some of my other posts on pandas:◼️️ Writing 5 common SQL queries in pandas◼️️ Writing advanced SQL queries in pandas◼️️ 5 tips for pandas users◼️️ 5 tips for data aggregation in pandas◼️️ How to transform variables in a pandas DataFrame
Bye for now 🏃 💨