How to Quickly Create and Unpack Lists with Pandas | by Byron Dolon | Towards Data Science

Pre-processing and “data wrangling” take up a lot of time, but it’s not always the most fun part of a data analysis project.
Having to unpack a list comes up a lot when it comes to reformatting your initial data. Fortunately, Pandas comes with a lot of vectorized solutions to common problems, so we won’t have to stress too hard about unpacking lists in a DataFrame.
In this piece, we’ll be looking at two things:
How to use df.explode() to unnest a column with list-like values in a DataFrame;
How to use Series.str.split() to create a list from a string.
We’ll be using a modified version of this video game sales data, so you can download the csv file if you want to follow along. This time, I included the code to get the initial tables for each of the examples at the very bottom. I’d suggest going through the piece first, and then copying the code to get the inputs to try out the examples after.
Our initial table looks like this:
The goal is to separate all the values in the “Genre” column, so that each row only has one value. In terms of database normalization, this would be a step towards fulfilling the “first normal form”, where each column only contains atomic (non-divisible) values.
To do so, all we need to do is use the df.explode() function. We only need to pass one argument, which is the name of the column with the list like values.
Our code is like this.
df2 = df2.explode('Genre').drop_duplicates()
A subset of the resulting DataFrame looks like this:
Now we have a table with all the different Genres of each Publisher. We also made sure to only have unique values by passing .drop_duplicates() on the DataFrame.
You’ll notice that the index value from the original DataFrame was kept for each of the rows, which goes to show that all the df.explode() function does is separate elements in the iterable while keeping all other row values the same. We could also easily pass .reset_index() if we wanted new index values.
In the last problem, we worked with a DataFrame that had a column full of lists. However, oftentimes you’ll be working with data that doesn’t fit neatly into pre-defined Pandas functions. For example, say we have a similar table to the last problem’s that looked like this:
Can you spot the difference?
We still have a bunch of values in the “Genre” column, but they’re no longer in a Python list. The df.explode() function wouldn’t work, because as per its documentation:
This routine (df.explode) will explode list-likes including lists, tuples, Series, and np.ndarray.
We currently have multiple items for each column value, but they are not in the “list-like” format. All we have is a bunch of really long strings.
To combine the sub-strings into a list, we’ll make use of a simple line of code to convert the string into a list:
df3['Genre'] = df3['Genre'].str.split(',')
Here, we used the Series.str.split() method on the “Genre” column to create a list. As its name suggests, this method splits a string on a separator that you specify, which in our case is a comma. If you don’t pass anything into split(), the method will attempt to split a string on whitespace, but our strings don’t have any.
The resulting table looks like this:
Now we have a table almost identical to the previous one, except this time all the different values are in a list.
All we have to do now is run the same line of code as in the first problem:
df3 = df3.explode('Genre')
I hope you found this quick look at df.melt useful for your work with Pandas! If you need to create or unpack lists in your DataFrames, you can make use of the Series.str.split() and df.explode() methods respectively.
As promised, here’s the code to set up the two examples:
import pandas as pddf = pd.read_csv('vgsales.csv').dropna()df['Year'] = df['Year'].astype(int)# problem 1df2 = df.groupby('Publisher')['Genre'].apply(','.join).reset_index()df2 = df2.loc[(df2['Publisher']=='Nintendo') | (df2['Publisher']=='Ubisoft') | (df2['Publisher']=='Activision')]df2['Genre'] = df2['Genre'].str.strip().str.split(',')# problem 2df3 = df.groupby('Publisher')['Genre'].apply(','.join).reset_index()df3 = df3.loc[(df3['Publisher']=='Infogrames') | (df3['Publisher']=='Wanadoo')]
You can see I used the .loc[] function to set out some conditions for my DataFrames, so if you’re not familiar with using it already, you can check out this piece:
towardsdatascience.com
I also used .apply() here, which is a simple way to perform a function along an axis of a DataFrame. For a different way to edit a DataFrame row by row (which I would suggest using only when you have a problem that doesn’t have a vectorized Pandas solution), check out this piece:
towardsdatascience.com
Good luck on your adventures with Pandas!