Pandas melt and pivot: Shape tables like JELLY | Towards Data Science

I am not ashamed to admit. melt() and pivot() were the hardest of pandas to learn for me. It took me more than 3-4 months to wrap my head around. When I first saw them, I tried and tried but could not understand how or when they are used. So, I gave up, moved on, and met them again. Tried, failed, moved on, and met again. Repeated many times.
What I found baffling was that in this age of millions of video and blog tutorials, I could not find a simple good source that could explain these functions better. My favorite blog for pandas, Practical Business Python, which covers many hard functions of pandas, kindly left them out 😔😔. And, their documentations only make sense after you completely know the functions and some websites (which always rank high if you google these functions) just copy out the documentation😶😶.
So, just like I have been writing about hard functions of pandas recently (which are performing really well), the time has come for melt() and pivot(). I hope this post will be useful for people who faced similar difficulties.
∘ Introduction ∘ Setup ∘ Pandas melt() ∘ Pandas melt() on real-world data ∘ Pandas pivot()
You can download the notebook and the sample data from this GitHub repo. I also hyperlinked each function’s documentation at their first appearance.
Let’s start with a very stupid example. I will create a 1x1 dataframe that holds a city name and a temperature for a single day. Then, I will call melt() on it to see what effect it has:
>>> df.melt()
So, without any parameters melt() takes a column and turns it into a row with two new columns (excluding the index). Let's add two more cities as columns:
If you notice, this type of format for dataframes are not easy to work with and it is not clean. What would be ideal is to take the columns and turn them into rows with their temperature values on the right side:
df.melt()
Let’s add more temperatures for the cities:
What do you think will happen if we call melt() on this version of the dataframe? Watch:
df_larger.melt()
Just like expected, it converts each column value into a row. For example, let’s take a key-value pair. New York’s temperatures are [25, 27, 23, 25, 29]. This means there are 5 key-value pairs and when we use melt(), pandas takes each of those pairs and displays them as a single row with two columns. After pandas is done with New York, it moves on to other columns.
When melt() displays each key-value pair in two columns, it gives the columns default names which are variable and value. It is possible to change them to something that makes more sense:
var_name and value_name can be used to change the labels of the melted dataframe's columns.
If we keep adding columns, melt() will always convert each value into a row with two columns that contain the previous column's name and its value.
Now, let’s get a little serious. Say we have this dataframe:
This time, we already have the cities as a column. But still, this type of format for tables are not useful to work with. This dataset holds temperature information for 5 cities for 5 days. We can’t even perform simple computations like mean on this type of data. Let’s try melting the dataframe:
>>> temperatures.melt()
This is not what we want, melt() turned the city names into rows too. What would be ideal is if we kept the cities as columns and append the remaining columns as rows. melt() has a parameter called id_vars to do just that.
If we want to turn only some of the columns into rows, pass the columns to keep as a list (even if it is a single value) to id_vars . id_vars stands for identity variables.
temperatures.melt(id_vars=['city'])
After using id_vars, the city column stayed as a column. But it has become longer. The reason is that for each city there were 5 days of observations. When we take those observations from columns and display them as rows, pandas automatically adds new rows to fit the new values.
Even though we have the table in better shape, the column names are not exactly what we want. Instead of changing them manually after melting the table, we can directly do it with melt():
The same dataframe with different column labels.
Now, it is time we work on a real-world dataset to bring the point home. I will load in the NYC stocks dataset which can be downloaded from Kaggle using this link. It contains stocks information for the year 2016 for more than 501 companies:
I will subset it for only one month because there are observations for each day:
Now, so that I can show you how to use melt() on a real world-data, I will perform an operation using pivot() which I will teach you later in the article.
The data now is in this format:
It has got 500 rows for all companies and 22 columns for 22 days in February. Each cell contains the close prices of stocks on a given day. Why did I choose this format? Because often real-world data comes in this shape.
For example, when recording this dataset, let’s say they wanted to add a new value for a new day. There are 500 companies and if you don’t add the new day as a new column, you would have to write out 500 companies once again adding 500 more rows to the dataset. Adding new observations as new columns is very handy for people who are recording this data.
So, now we want to turn back this stocks_small dataset into its original format using melt() so that it is easier to work with. If you look at the data once again, we want to turn all the date columns into two columns which contain (date, close price) key-value pairs preserving the symbol column. Just like we did in the previous examples, if we pass symbol to id_vars, it stays a column and the other dates will become rows:
Number of rows in melted table: 10500
Comparing it with the original data:
In a nutshell, melt() takes wide dataframes and makes them longer and thinner.
pivot() is the complete opposite of melt(). Sometimes, there will be cases where you want to turn your clean, long formatted data into wide. Let's see how I turned that subset stocks into a wide format. Here is the data to remind you:
stocks_small.head()
Now, I will pivot the table:
>>> pivoted.shape(500, 42)
When you use pivot(), keep these in mind:
pandas will take the variable you pass for index parameter and displays its unique values as indexes.pandas will take the variable you pass for columns and display its unique values as separate columns.
pandas will take the variable you pass for index parameter and displays its unique values as indexes.
pandas will take the variable you pass for columns and display its unique values as separate columns.
If you noticed, the above dataframe is not the one we used with melt(). That's because it also contains the open prices of stocks not just close. In pivot(), there is a parameter called values which if not specified tells pandasto include all of the remaining columns to the pivoted dataframe. Let's choose only the close prices this time:
>>> pivoted2.shape(500, 21)
That’s pretty much it for pivot(). Surprisingly, it is one of the hardest functions in pandas and yet it only has 3 parameters, not even an extra parameter to fill missing values.
For pivot(), just remember that it takes two categorical variables and displays their unique values as index and columns. This resulting table will be a grid of those two variables. If the values parameter is not specified, all the remaining columns are given as cell values which will make the table even wider.
Read more articles related to the topic: