How to Use MultiIndex in Pandas to Level Up Your Analysis | by Byron Dolon | Towards Data Science

What if you could have more than one column as in your DataFrame’s index?
The multi-level index feature in Pandas allows you to do just that. A regular Pandas DataFrame has a single column that acts as a unique row identifier, or in other words, an “index”. These index values can be numbers, from 0 to infinity. They can also be more detailed, like having “Dish Name” as the index value for a table of all the food at a McDonald’s franchise.
But what if you owned two McDonald’s franchises, and wanted to compare the sales of one dish across both franchises?
While thegroupby() function in Pandas would work, this case is also an example of where a MultiIndex could come in handy.
A MultiIndex, also known as a multi-level index or hierarchical index, allows you to have multiple columns acting as a row identifier, while having each index column related to another through a parent/child relationship.
At the end of this piece, we’ll have answered the following questions by creating and selecting from a DataFrame with a hierarchical index:
Which characters speak in the first chapter of “The Fellowship of the Ring”? (answered with .loc)
Who are the first three elves to speak in the “The Fellowship of the Ring”? (answered with .loc)
How much do Gandalf and Saruman talk in “The Two Towers”? (answered with .loc)
How much does Isildur speak in all of the films? (answered with .xs)
Which hobbits speak the most in each film and across all three films? (answered with a pivot table and.loc)
You can find the data used in this article here. We will be using data from “The Lord of the Rings” films, specifically the “WordsByCharacter.csv” file in the data set. This file will have each character’s number of words spoken in each scene of every movie.
As always, don’t forget to import pandas before trying any of the code.
import pandas as pd# load datadf = pd.read_csv('WordsByCharacter.csv')
Let’s dive into how multi-level indexes can be used for data analysis.
A hierarchical index means that your DataFrame will have two or more dimensions that can be used to identify every row.
To get the original DataFrame’s index label, we can use this code:
df.index.names
This outputs a “FrozenList”, which is just a Pandas specific construct used to show the index label(s) of a DataFrame. Here, we see the value is “None”, as this is the default value of a DataFrame’s index.
To create a MultiIndex with our original DataFrame, all we need to do is pass a list of columns into the .set_index() Pandas function like this:
multi = df.set_index(['Film', 'Chapter', 'Race', 'Character'])
Here, we can already see that the new DataFrame called “multi” has been organized so that there are now four columns that make up the index. We can check this by looking at the index names once more:
multi.index.names
We now see that the value “None” earlier has been replaced by the names of the four columns we assigned to be our new index.
Each index value in the regular, unaltered DataFrame would just be a number from 0 to 730 (because the DataFrame has 731 rows). To show you what each index value is in our newly created MultiIndex, we can use this line of code:
multi.index.values
Now, we see that each row value in the “Words” column can be identified by which film it came from, what chapter it refers to, what race the character who spoke the word is, and that character’s name.
One thing to note before we dive into some analysis is the .sort_index() Pandas function. When creating a DataFrame with a MultiIndex, make sure to append that to the end of the line of code like this:
multi = df.set_index([‘Film’, ‘Chapter’, ‘Race’, ‘Character’]).sort_index()
The Pandas documentation has this note on it:
Indexing will work even if the data are not sorted, but will be rather inefficient (and show a PerformanceWarning). It will also return a copy of the data rather than a view.
It’s also worth noting that you can remove the hierarchical index simply by passing .reset_index() to your DataFrame, like this:
multi.reset_index()
This will just return the original DataFrame without the parent/child relationships on multiple index columns.
Now we’re ready to perform some analysis!
Here, we’re going to use the trusty .loc Pandas function. If you haven’t used it before, feel free to check out this quick guide on vectorized Pandas functions.
Here, we are interested in two components of our index, namely “Film” and “Chapter”. In our .loc selector, we write the following:
multi.loc[('The Fellowship Of The Ring', '01: Prologue'), :]
You can select your desired value of each part of the MultiIndex of your DataFrame by passing it into a tuple. In the first part of the selector we just wrote, we passed the tuple (‘The Fellowship Of The Ring’, ’01: Prologue’). We didn’t need to pass values for “Race” or “Character”, because we don’t know who spoke in the first chapter yet.
Notice that the “Film” and “Chapter” index columns are no longer present in the view. This is because we already input “The Fellowship Of The Ring” for “Film” and “01: Prologue” as values in the tuple, so the output does not need to duplicate these values in the view.
Also, the : is inserted as the final argument in the .loc selector to indicate that we want all columns to be displayed. In our case, the DataFrame only has one column called “Words”, as every other column in the DataFrame has been converted to an index column. However, if you had multiple columns in your DataFrame and only wanted to see a subset, you would the column names instead of :.
Now, we know that Elrond, Galadriel, Gollum, and Bilbo spoke in the first scene of the first movie. Let’s move on to a slightly more complicated example.
Here, we’ll still be using .loc, but now we’re going to use some syntax that will give us some flexibility when selecting from our MultiIndex values.
To answer this question, we’re interested in the “Film” and “Race” index columns. This time, we’re “skipping” an index column, as “Chapter” lies between “Film” and “Race”. We don’t yet know which “Chapters” elves first speak in, so we need to leave that blank.
To do so, we’re going to use this line of code:
multi.loc[(‘The Fellowship Of The Ring’,slice(None),’Elf’), :].head(3)
Again, we pass a tuple in with our desired index values, but instead of adding values for “Chapter”, we pass slice(None). This is the default slice command in Pandas to select all the contents of the MultiIndex level. So here, we are selecting all possible “Chapter” values.
Now, we know that Elrond, Galadriel, and Arwen are the first three elves to speak in the first book. But so far, all we’ve been doing is passing one value per index column into the .loc selectors. What if we wanted to get more than one value?
In this question, we’re interested in all “Chapters” of the second “Film”, both of which we already know how to select. But this time, we’re interested in two values for the “Character” index.
To get the answer, we write this line of code:
multi.loc[('The Two Towers',slice(None),slice(None),['Gandalf','Saruman']), :]
All we need to do here is pass a list of our desired index values to the tuple we insert into .loc. Since we don’t know which “Chapter” or “Race” to get, we pass slice(None) for these two index values before getting to “Character”.
So far, we’ve had multiple conditions in our selections, which is why using .loc was handy. But what if we only wanted to specify one level of the index? It would be tiresome to write slice(None) three times, so we’ll turn to another method.
To answer this question, we only need to specify one index level value for “Character”. We will use the xs() (cross-section) method in Pandas, which allows you to specify which part of the MultiIndex you want to search across.
We use the following code:
multi.xs('Isildur', level='Character').sum()
At it’s most basic, xs() requires an input for the index value you want to look for and the level you want to search in. In this case, we want the “Character” level, so we pass that into level=, and we want the value for “Character” to be “Isildur”. Then, to aggregate all the values, we add .sum() to the end of the line.
I cheated a bit so I could verify the accuracy of the data, since I knew the answer to this (the one word Isildur says in the whole trilogy is “No”). But you could also use this same line of code to find how much any character spoke, simply by substituting their name as a parameter in xs().
To answer this question, it would be great if we had one table with the “Words” values aggregated for every character across every film. This is a great place to create a pivot table!
We’re going to use the .pivot_table() function from Pandas, but you’ll see that we pass a list into the index= parameter setting to create a MultiIndex again. Our code to create the pivot table will look like this:
pivoted = df.pivot_table(index = ['Race','Character'],                         columns = 'Film',                         aggfunc = 'sum',                         margins = True, # total column                         margins_name = 'All Films',                         fill_value = 0).sort_index()order = [('Words', 'The Fellowship Of The Ring'),         ('Words', 'The Two Towers'),         ('Words', 'The Return Of The King'),          ('Words', 'All Films')]pivoted = pivoted.sort_values(by=('Words', 'All Films'), ascending=False)pivoted = pivoted.reindex(order, axis=1)
Here, our multi-level index has two components instead of the four from the previous example. This is because we want the “Film” values to also be a column and not an index. We also don’t need to see the “Chapter” values, so that’s excluded. As we’re interested in the total number of words, we want our aggfunc (aggregate function) parameter to be “sum”.
Pandas also has a built-in total column for the .pivot_table() function. All you need to do is pass margins=True to enable it, and optionally set the name of the total column in the margins_name parameter.
Because we want to see the total number of words for every “Character” in that has a “Race” value as “Hobbit”, we can select this condition with a really easy application of .loc.
pivoted.loc['Hobbit']
Boom!
Now we have a table ordered by which Hobbit spoke the most across all films. We can also see how much each character spoke in each film. Some useful insights include how Bilbo didn’t speak at all in “The Two Towers”, but still has more total dialogue than Pippin and Merry, who spoke in all three films.
What’s great about having this table is we can do the same kind of analysis for “Elf”, “Men”, “Ainur”, and the rest of the races in The Lord of the Rings, all by substituting one argument in the above .loc function.
I hope you found this introduction to hierarchical indexes in Pandas useful! I’ve found that having more than one level in a DataFrame index means that I can quickly group my data to specific levels, without having to write multiple groupby() functions.
As always, check out the official documentation for a more in-depth look at how you can use the MultiIndex feature in your analysis.
I like getting involved in data collection as well as analysis, which is why I follow a quick and easy method of gathering sample data discussed in that article. If you want to grab some of your own test data online to try this code, you can check out this piece on getting tables from a website with Pandas.
Good luck with your multi-level indexing adventures!