Functions That Generate a Multi-index in Pandas and How to Remove the Levels | by Susan Maina | Towards Data Science

Introduction
In this article, we will look at what a multiindex is, where and when to use it, functions that generate a multiindex, and how to collapse it into a single index.
But first, let’s get some basic definitions out of the way.
An index is a column in a DataFrame that ‘uniquely’ identifies each row. Think of it as the row labels.
A multiindex is when there is more than one index. Other names are multiple index and hierarchical index.
Multiindex can also refer to multiple header levels, or when you have a hierarchy of column names.
Advantages of multiple indices
For holding data that is higher-dimensional and contains hierarchies or levels.
Provides a helpful visual display of the hierarchy levels in a table format.
It allows for efficient selection and manipulation of hierarchical data using functions such as df.xs() and df.unstack().
Disadvantages of multiindex
The format does not allow for direct plotting of graphs.
There are no performance benefits when using the multiindex.
We will use the diamonds dataset from seaborn to demonstrate the various scenarios that result in a multiindex situation, and how to collapse the multiindex back into a single index DataFrame.
import pandas as pdimport seaborn as snsdiamonds = sns.load_dataset('diamonds')
So, how do we end up with multiple indices?
A1. Using df.set_index(col_list)
The code below manually sets the index to two columns (cut and clarity).
sorted_df = diamonds.sort_values([ 'cut’, ’clarity’])multiind_df = sorted_df.set_index([ 'cut','clarity'])multiind_df
What to look out for:
The hierarchies are displayed as expected. If not, remember to sort the DataFrame by these columns.
The number of rows of the resulting DataFrame does not change, but rather the DataFrame is rearranged so that the hierarchies are visible.
diamonds.shape###Results(53940, 10)multiind_df.shape###Results(53940, 8)
The number of the resulting columns is now less by two because of the columns lost to the index (see results of df.shape above).
The index is now a multiindex. Run df.index to display a multiindex list where each element is a tuple.
multiind_df.index
The previous index was dropped by default. See what happens below when we set the index to another column in the new DataFrame.
multiind_df.set_index('carat')
If you want to retain the previous index, first use df.reset_index() to make the index part of the existing columns, then use df.set_index(col_list).
A2. Multiindex resulting from groupby of many columns
df.groupby summarizes columns (features) based on a chosen column’s categories.
For example, we can group the diamonds by the cut and color to see how other features are distributed among these categories. We use max() as the aggregation function.
grouped_df = diamonds.groupby([’cut’, 'color’]).max()grouped_df
What to look out for:
The number of rows will substantially reduce. This is because only unique indices are displayed (unique cut and color combinations here). The specified aggregate function (max ) combines the other values that fall in these groups into one value.
diamonds.shape###Results(53940, 10)grouped_df.shape###Results(35, 7)
The number of columns also reduces to 7 because two columns now reside as the index, while clarity was dropped because the aggregate function — max — does not work with non-numeric features.
Let us now demonstrate how we end up with multiple header levels.
B1. Groupby more than two columns then unstack
We’ll continue with the code from the previous section. We performed a groupby using cut and color.
Now let’s unstack it so that the ‘cut’ categories appear as column headers. This is by flipping them from a row index to column headers.
grouped_df = diamonds.groupby(['cut','color']).max()unstacked_df = grouped_df.unstack('cut')unstacked_df
We now have a new header level below the original headers — carat, depth, price, etc.
B2. Groupby using several aggregation functions
In our previous groupbys, we used only max() as the aggregate function. However, we can include several aggregation functions and their names will be held at a new level.
The code below groups the data by one column — cut — but uses 3 aggregation functions —median, max and mean.
diamonds.groupby( 'cut').agg( ['median','max','mean'] )
B3. use pivot_table to convert rows into columns
A pivot_table provides a convenient way of reshaping a column’s values into column headers like the unstack method we used above. Here we will pivot on our original diamonds dataset.
diamonds.pivot_table(    values = 'price',    index = 'color',    columns = ['clarity','cut'],    aggfunc='median')
B4. Pandas Cross-tab
The pandas.crosstab function allows us to create a frequency table of data. In the code below, we want to find the distribution of clarity and cut for every color. We use normalize=’columns’ to show the percentage distributions per column.
pd.crosstab(    index = diamonds[’color’],    columns = [diamonds[’clarity’],            diamonds[’cut’]],    normalize = 'columns’)
C1. Use df.reset_index()
df.reset_index() resets the index by converting the existing index (or indices) into a normal column. A RangeIndex is generated as the new index.
grouped_df.reset_index()
In a Multiindex situation, we can choose the name (or position) of the index to reset by including level=n.
grouped_df.reset_index(    level='cut')
We can also reset the index and still maintain multiple header levels.
df = diamonds.groupby(    'cut’).agg(    [’median’,’max’,’mean’])df.reset_index()
C2. Remove multiindex using df.droplevel(level = level_to_drop, axis=0)
This is used when you want to entirely drop an index.
Using the grouped_df generated earlier, let’s drop the color index. Note that we can either use level=index_name or level=position (counting from 0 as the outermost level). This method returns the modified DataFrame.
grouped_df.droplevel(    level = 1,    axis=0)
C3. Remove multiindex using df.index.droplevel(level = level_to_drop)
This function drops the specified level from the index and returns the remaining index list. The next step is to assign this list to be the DataFrame’s index.
grouped_df.index = grouped_df.index.droplevel('color')
Print the DataFrame to display the same results like the image in the previous section.
D1. Merge the levels’ names for each column
Merging the header levels is a common solution because both levels might be useful and it is impossible to reset the index here.
Method 1: Using the map and join functions
The map function modifies every element in a list using a given function. Here, we have a list of tuples for each column name, and .map uses .join to merge the tuples into one name separated by an underscore.
df.columns.map('_'.join)
The code above returns an index object. We need to assign this to the column names and then print out the DataFrame.
df.columns = df.columns.map('_'.join)
Method 2: Using a list comprehension
A list comprehension also takes in a list, modifies each element through some operation, and returns a new list. We use .join to merge each tuple into one name using a different joining symbol (|).
df.columns = ['|'.join(s) for s in df.columns]
Another list comprehension example uses the f-string format. This is useful to change the order of the names after a merge. In the code below, the lower-level names come first.
df.columns = [f'{j}#{i}' for i,j in df.columns]
D2. Remove multiindex per column Using df.droplevel(level, axis=1)
If a header level is not useful at identifying the columns, you can choose to drop it. We use axis=1 (or axis= ’columns’) to signify column header levels.
Let us demonstrate. First we group the dataset by cut and use four aggregation functions. Then we select only the price columns using df.xs().
df = diamonds.groupby(    'cut').agg(    ['max', 'median', 'min','mean'])df.xs(    key='price',     axis=1,     level=0,     drop_level=False)
Now we can drop the top level ‘price’ since we already know all the values represent the price.
df_price.droplevel(    level=0,     axis=1)
D3. df.columns.droplevel(level_to_drop)
We can also drop a header level and return the desired level(s) as a list. (This is similar to df.index.droplevel earlier but with columns instead of index). We then assign this list to the column names.
df.columns = df.columns.droplevel(0)display(df)
Print the DataFrame to display the same results like the image in the previous section.
D4. df.columns.get_level_values(level_to_return)
This function returns the desired level(s) and drops the rest. The code below yields the same results as the image in the previous section.
df.columns = df.columns.get_level_values(1)df
In this article, we explored the various functions that generate a multiindex and how to collapse it back into a basic DataFrame with one index.
There are other ways of creating multiindex DataFrames from scratch and several sophisticated ways of accessing and selecting the data. I encourage you to experiment and practice with multi-dimensional data that contains many prominent categorical features and play around with the multiindex.
Find the code used in this blog post here.
If you liked this content and would like to get notified of more like it, subscribe here. If you are not yet a medium member, join here. Thank you for reading.
1. Pandas MultiIndex Tutorial by Zax Rosenberg, CFA
2. Accessing Data in a MultiIndex DataFrame in Pandas by B. Chen
3. Hierarchical Indexing from the Python Data Science Handbook by Jake VanderPlas