Reshaping DataFrames in Pandas. Your ultimate guide to reshaping... | by Anirudh Nanduri | Towards Data Science

Your ultimate guide to reshaping DataFrames in python
Pandas is easily one of the most used packages in python. This might not come as surprise to most of the folks reading this article. Many people use pandas to do different kinds on analysis on their data and much more. However, in certain situations we would want to reshape or visualize data in a different format than it was initially provided. This might be due to various reasons. It might be because one finds a different representation easier to understand. It might also be because you want to have a certain format to run your code. Whatever the reason might be, reshaping dataframes can be considered as a common task that most of us do in our journey as data scientists/analysts. This article would give a brief introduction on some useful functions which can be used to reshape a pandas dataframe.
I like to think of reshaping functions in two types. First type of functions simple reform the existing dataframe. For example, they change columns to rows and rows to columns. The second type of functions would aggregate the information along with reforming them. In most of the real world cases, we would be using the second type of functions as it would give us a peek into the higher level summaries or aggregation as needed.
Reforming without aggregation can and should ideally be applied on data where there is a unique combination of selections being made. Otherwise, there is a good chance that they would throw an error for certain functions.
To explain the reforming without aggregation, we would first declare a dataframe. The declaration and the dataframe would be as follows:
np.random.seed(100)
df=pd.DataFrame({"Date":pd.Index(pd.date_range(start='2/2/2019',periods=3)).repeat(3),
"Class":["1A","2B","3C","1A","2B","3C","1A","2B","3C"],
"Numbers":np.random.randn(9)})
df['Numbers2'] = df['Numbers'] * 2
df
Using this dataframe, we would look at some functions which are useful for reforming/reshaping.
Pivot method is typically used to create a pivot style view of data where the users can specify rows (in python it is called index) and columns. These two parameters would give a structure to the view whereas the information to be populated would be from the data that is being used to create pivot. The information can also be selectively populated by using values parameter. Let us now look at some examples to understand it better.
df.pivot(index='Date', columns='Class', values='Numbers')
As we can see from above, once index, columns, values are specified the structure for a new pivot style dataframe is formed and is filled with information unique to the combination of index, column pair accordingly.
It is to be remembered that index and columns parameters are suggested to be filled always as it would determine the structure of final dataframe/view. Let us look at what would happen if we do not declare values parameter.
df.pivot(index='Date', columns='Class')
If we do not specify values parameter, pandas would create all the various possible views while taking all column names apart from what were specified as index and columns as above. Hence, for a big dataset having multiple columns, it is suggested to specify value parameter as well.
There is, however, another way to extract information about one column using the above code and tweaking it a little. It would be as follows.
df.pivot(index='Date', columns='Class')['Numbers']
If we mention the column name as given above while not specifying the values parameter, the resultant output would be the same. Only downside to this would be that it would take longer time to execute it. It is to be noted that the column name declaration is similar to syntax for sub-setting the dataframe. That is because we are exactly doing that. Using above code, pandas first gets the result for all columns, then using the column name/s specified in the brackets, pandas subsets the dataframe and displays it. This is also why the execution time for this method is longer than the one where we use values parameter.
Note: While using pivot function, in case the combination of row*column has multiple entries the function would throw an error as pivot would not do any sort of aggregation.
Melt is a function which is used to convert columns to rows. That means that this function is useful for when the users would like to bring one or more columns information into rows. This function would create two new columns by removing all other columns apart from the ones mentioned in its id_vars parameter and displays the column name in one column and its value in another column. Let us have a look at an example to understand it better.
df.melt(id_vars=['Date','Class'])
In the dataframe df, we have four columns of which in id_vars parameter Date and Class were mentioned. This would leave us with columns Numbers and Number2. As we can see above, the function created two new columns called variable and value and populated them with column name and the unique value for the combination of Date+class+column name while removing columns Numbers and Numbers2.
In case only specific columns are to be converted in this way and not all, then the parameter value_vars would help us achieve this.
df.melt(id_vars=['Date','Class'], value_vars=['Numbers'])
As we can see, when specific column names are mentioned in value_vars parameter only that columns information would be melted as needed. There are two additional parameters value_name and var_name which would rename the columns value and variable by the string mentioned in the parameters. Let us have a look at an example to understand this better.
df.melt(id_vars=['Date','Class'], value_vars=['Numbers'], value_name="Numbers_Value", var_name="Num_Var")
As we can see, the columns are renamed to the ones specified in the parameters as expected.
Stack and Unstack perform columns to rows and rows to columns operations respectively. Both these functions are definitely one of the less used functions of reshaping in pandas as one would use pivot to achieve the result they want most of the time and hence it would not be needed. We will still look into them to understand how they work as they might be useful in some specific scenarios.
The main different between stack and melt would be that stack requires id_vars to be set as index instead of having to pass them as parameters separately. Let us have a look at stack to understand what it does.
df.set_index(["Date","Class"]).stack()
As we can observe, it has the same functionality of melt where it brings columns to rows.
Unstack on the other hand does the opposite of stack. It brings rows to columns and like stack, it needs index to function as expected.
df.set_index(["Date","Class"]).stack().unstack()
As we can see, unstack if no input provided would basically undo the stacking operation. The output will differ when changing the number input inside the brackets of unstack. This would be a good topic to explore if one is interested about the topic.
Unlike the type-1 functions, type-2 functions give an aggregated view of information. These will be very useful in case the users would like to have some type of summary around the data. We will be using the same dataframe that we used for Type -1 functions to look into type -2 functions as well.
To explain the reforming without aggregation, we would first declare a dataframe. The declaration and the dataframe would be as follows:
Using this dataframe, we would look at some functions which are useful for reforming/reshaping.
Group by is the function that I use the more often than any other function mentioned in this article. This is because, it is very intuitive to use and has very useful parameters that can help one to view different aggregations for different columns. Let us look into a simple group by statement.
df.groupby('Date')["Numbers"].mean()
In group by statement, we specify the column/columns that we need to group the data on inside the first parenthesis. Here, we can either give a single string or a list of strings pertaining to column names as well. The next parenthesis is to specify the data to aggregate, and finally we call the method using which the aggregation should happen. In above example, we are aggregating column Numbers on Date using mean aggregation.
Notice that the output is not a dataframe but looks similar to a pandas series. To convert it into a dataframe, we can use reset_index method, or we can also use a parameter available in group by statement. I will be using the latter.
df.groupby('Date', as_index=False)["Numbers"].mean()
As mentioned in the above image, as_index parameter can be used to convert the result of group by into a dataframe.
Let us have a look at one more example where we aggregate columns on multiple columns using a single method.
df.groupby(['Date','Class'], as_index=False)["Numbers"].mean()
As we can see above, we can aggregate multiple columns on multiple columns using a list of strings (column names) instead of specifying a single string (column name). However, we can also use different aggregation functions for different columns. Let’s have a look at the example below to understand how that can be done.
df.groupby(['Date'], as_index=False).aggregate({"Numbers":"sum", "Numbers2":"mean"})
In the above image we are aggregating Numbers column on Date column with the sum function and similarly we are aggregating Numbers2 column on Date column with mean function.
Pivot Table functions in the same way that pivots do. However, pivot table has one additional and significant argument/parameter which specifies the aggregation function we will be using to aggregate the data. Let us look at the below example to understand it better.
Let us start with a simple pivot table.
df.pivot_table(index="Date", columns="Class")
Notice the method call for pivot table is pivot_table which is different from that of pivot which was pivot. If the aggregation function is not specified, the default function mean would be used to aggregate the information.
df.pivot_table(index="Date", columns="Class", aggfunc="sum")
However, if the user wants to specify a particular function, they can do so in the parameter aggfunc as shown in the above image.
The last function in this article would be crosstab. This function by default would give the count or frequency of occurrence between values of two different columns. Let us look at an example.
pd.crosstab(df.Date, df.Class)
The view above means that there are two different entries are available in combination of 1A class and 2019–02–02 Date. Similarly, the other information can be interpreted.
In case instead of frequency of occurrence, if the user wants to have some other aggregation function aggfunc parameter can be used.
pd.crosstab(df.Date, df.Class, values=df.Numbers, aggfunc='sum')
pd.crosstab(df.Date, df.Class, values=df.Numbers, aggfunc='mean')
Above images give a sense of how it can be used. Even though the output may look similar to that of pivot tables, the way it is achieved is different. But more often than not people prefer using pivot tables over crosstab function.
We have looked at multiple functions in this article and have understood various parameters and their usage.
All said and done, everyone knows that practice makes man perfect. This saying applies to technical stuff too right? To make it easier for you to practice multiple concepts we discussed in this article I have gone ahead and created a Jupiter notebook that you can download here. Good time practicing!!!
Please do feel free to reach out to me here in case of any query, constructive criticism, and any feedback.