Dealing with List Values in Pandas Dataframes | by Max Hilsdorf | Towards Data Science

Have you ever dealt with a dataset that required you to work with list values? If so, you will understand how painful this can be. If you have not, you better prepare for it.
If you look closely, you will find that lists are everywhere! Here are some practical problems, where you will probably encounter list values.
Audio- or video tags
Open-ended questions in survey data
Lists of all authors, artists, producers, etc. involved in a creative product
I have recently worked on multiple projects that required me to analyze this kind of data. After many painful hours of figuring out how to perform even the simplest operations, I realized had to share my knowledge here to save you some time.
I will show you the kinds of problems you might encounter and offer practical solutions for them.
To follow this tutorial, feel free to download the fictional dataset used for all the examples. All materials including the code can be found in this repo.
Also, please import all the necessary libraries and load the dataframe.
import pandas as pdimport numpy as npimport matplotlib as pltfruits = pd.read_csv("fruits.csv", delimiter = ";")
Allow me to get right to the point: List values mess up everything you know about data analysis. The simplest operations can not be performed without endless looping. Let me show you a quick example:
For the “age” column in the example dataset, we can easily use the value_counts() function to count how many times which age was observed.
fruits["age"].value_counts()## OUTPUT ##8     310    26     29     17     15     1
What if we want to find out which fruits were named the most? In principle, we have all the data we need for this in the “favorite_fruits” column. However, if we apply the same function, the results are less than unhelpful.
fruits["favorite_fruits"].value_counts()## OUTPUT ##["strawberry", "raspberry", "blueberry"]                      1["mango", "pineapple", "orange"]                              1["blueberry", "watermelon", "apple", "raspberry"]             1["apple", "pear", "peach"]                                    1["pear", "strawberry", "raspberry"]                           1["banana", "maracuja", "watermelon", "apple", "pineapple"]    1["peach", "strawberry", "apple", "blueberry"]                 1[]                                                            1["banana", "mango", "orange", "watermelon"]                   1["watermelon", "apple", "blueberry", "pear", "strawberry"]    1
The reason this does not work is that Pandas does not have direct access to every individual element of the lists. Thus, Pandas is unable to apply functions like value_counts() properly. So, what do we do instead? I will show you in the following!
One problem you will always encounter is that Pandas will read your lists as strings, not as lists.
for i, l in enumerate(fruits["favorite_fruits"]):    print("list",i,"is",type(l))## OUTPUT ##list 0 is <class 'str'>list 1 is <class 'str'>list 2 is <class 'str'>list 3 is <class 'str'>list 4 is <class 'str'>list 5 is <class 'str'>list 6 is <class 'str'>list 7 is <class 'str'>list 8 is <class 'str'>list 9 is <class 'str'>
This means that you can not even loop through the lists to count unique values or frequencies. Depending on how your lists are formatted in the dataframe, there is an easy or a more complex solution. In any case, you will simply be able to use the code I provide.
In that case, there is a quick method using the apply() and eval() functions.
fruits["favorite_fruits"] = fruits["favorite_fruits"].apply(eval)
This one is tougher, because the eval() function will not work, as the list is lacking the inner quotation marks in order for it to be recognized as a list object. The quick and dirty solution is to simply add the quotation marks to the string and apply eval() afterwards. Use this function:
def clean_alt_list(list_):    list_ = list_.replace(', ', '","')    list_ = list_.replace('[', '["')    list_ = list_.replace(']', '"]')    return list_
To apply this to your dataframe, use this pseudo code:
df[col] = df[col].apply(clean_alt_list)
Note that in both cases, Pandas will still assign the series an “O” datatype, which is typically used for strings. But do not let this confuse you. You can check the actual datatype using:
for i, l in enumerate(fruits[“favorite_fruits”]): print(“list”,i,”is”,type(l))## OUTPUT ##list 0 is <class 'list'>list 1 is <class 'list'>list 2 is <class 'list'>list 3 is <class 'list'>list 4 is <class 'list'>list 5 is <class 'list'>list 6 is <class 'list'>list 7 is <class 'list'>list 8 is <class 'list'>list 9 is <class 'list'>
After this first step, our lists are finally recognized as such by Pandas. Still, we can not use the standard functions, because they are not made for list applications.
At least we could use loops for everything now. This method works with small datasets, but can get awfully slow with large ones. For example, I had to analyze taglists of up to 999 tags for around 500k music tracks. This means, that the inner loop would have had hundreds of millions of iterations, which would have taken many hours and probably crashed my computer. I will show you a cleaner and quicker way to do this in a minute. However, if you really want to use loops, here is the code:
fruit_dict = {}for i in fruits["favorite_fruits"]:    for j in j:        if j not in fruit_dict:            fruit_dict[j] = 1        else:            fruit_dict[j] += 1
Here is the clean way that took me a while to figure out. If we conceptualize the favorite_fruits column as a 2D array, reducing its dimensions from 2 to 1 would allow us to apply the typical pandas functions again. For that, you can use this function:
def to_1D(series): return pd.Series([x for _list in series for x in _list])
If we use value_counts() now, we get the results we want.
to_1D(fruits[“favorite_fruits”]).value_counts()## OUTPUT ##apple         5blueberry     4watermelon    4strawberry    4raspberry     3pear          3banana        2pineapple     2mango         2peach         2orange        2maracuja      1
To get unique values, just extract them from the results above chaining .index() onto it.
At this point, we can produce our first meaningful visualization.
fig, ax = plt.subplots(figsize = (14,4))ax.bar(to_1D(fruits["favorite_fruits"]).value_counts().index,        to_1D(fruits["favorite_fruits"]).value_counts().values)ax.set_ylabel("Frequency", size = 12)ax.set_title("Children's Favorite Fruits", size = 14)
At this point, things are getting advanced. If you are happy with the results we got before, you can stop here. However, a deeper level of analysis might be required for your research goal. Maybe you want to correlate all list elements with each other to compute similarity scores. E.g. do kids who eat bananas typically also like mangos? Or maybe you want to find out which fruit has been ranked as the top favorite fruit by the most kids. These questions can only be answered at a deeper level of analysis.
For this, I will introduce two useful methods. They differ in complexity, but also in what you can do with their results.
This is a shockingly easy and fast method I stumbled upon. And it is so useful! All you need is one line of code.
fruits_expanded_v1 = fruits["favorite_fruits"].apply(pd.Series)
As you can see, this one-liner produced a dataframe where every list is split into its single elements. The columns indicate the order, in which the fruit was placed in the list. With this method, you will always get a dataframe with a shape of (n, len(longest_list)). In this case, two of the 10 children named five favorite fruits, which results a 10x5 dataframe.
Using this, we can find out which fruit was named most often as the number one favorite fruit.
fruits_expanded_v1.iloc[:,0].value_counts(normalize = True)## OUTPUT ##banana        0.222222 pear          0.111111 watermelon    0.111111 blueberry     0.111111 strawberry    0.111111 apple         0.111111 peach         0.111111 mango         0.111111
We can see that bananas are most often kids’ absolute favorite fruit.
Alternatively, we could target single fruits and find out how many times they were named at each position of the lists. This is the function I wrote for that:
def get_rankings(item, df):        # Empty dict for results    item_count_dict = {}        # For every tag in df    for i in range(df.shape[1]):                # Calculate % of cases that tagged the item        val_counts = df.iloc[:,i].value_counts(normalize = True)        if item in val_counts.index:            item_counts = val_counts[item]        else:            item_counts = 0                # Add score to dict        item_count_dict["tag_{}".format(i)] = item_counts        return item_count_dict
If we apply it, we get:
get_rankings(item = "apple", df = fruits_expanded_v1)## OUTPUT ##{'tag_0': 0.1111111111111111,  'tag_1': 0.1111111111111111,  'tag_2': 0.2222222222222222,  'tag_3': 0.2,  'tag_4': 0}
As you can see, we can perform rank-based analyses very well with this approach. However, this method is near useless for other approaches. Because the columns do not represent a single tag, but a rank, most tag-based operations can not be done properly. For example, calculating the correlation between bananas and peaches is not possible with the dataframe we got from method 1. If that is your research goal, use the next method.
This method is more complex and requires more resources. The idea is that we create a dataframe where rows stay the same as before, but where every fruit is assigned its own column. If only kid #2 named bananas, the banana column would have a “True” value at row 2 and “False” values everywhere else (see Figure 6). I wrote a function that will perform this operation. It relies on looping, which means that it will take lots of time with large datasets. However, out of all the methods I tried, this was the most efficient way to do it.
def boolean_df(item_lists, unique_items):# Create empty dict    bool_dict = {}        # Loop through all the tags    for i, item in enumerate(unique_items):                # Apply boolean mask        bool_dict[item] = item_lists.apply(lambda x: item in x)                # Return the results as a dataframe    return pd.DataFrame(bool_dict)
If we now apply the function
fruits_bool = boolean_df(fruits[“favorite_fruits”], unique_items.keys())
we get this dataframe:
From here, we can easily calculate correlations. Note that “correlation” is not really the correct term, because we are not using metric or ordinal, but binary data. If you want to be correct, use “association”. I will not.
Again, there are multiple ways to correlate the fruits. One straight forward way is the Pearson correlation coefficient, which can also be used for binary data. Pandas has a built-int function for this.
fruits_corr = fruits_bool.corr(method = "pearson")
Another way is to simply count how many times a fruit was named alongside all other fruits. This can be solved using matrix multiplication. For this, we will need to convert the boolean dataframe to an integer based on first.
fruits_int = fruits_bool.astype(int)
Then, we can calculate the frequencies.
fruits_freq_mat = np.dot(fruits_int.T, fruits_int)## OUTPUT ##array([[5, 3, 3, 2, 2, 1, 1, 1, 0, 2, 0, 1],       [3, 4, 2, 1, 1, 1, 1, 2, 1, 0, 1, 1],       [3, 2, 4, 3, 1, 2, 0, 0, 0, 1, 0, 0],       [2, 1, 3, 4, 2, 2, 0, 0, 0, 1, 0, 0],       [2, 1, 1, 2, 3, 1, 0, 0, 0, 1, 0, 0],       [1, 1, 2, 2, 1, 3, 0, 0, 0, 0, 0, 0],       [1, 1, 0, 0, 0, 0, 2, 1, 1, 0, 1, 1],       [1, 2, 0, 0, 0, 0, 1, 2, 1, 0, 1, 1],       [0, 1, 0, 0, 0, 0, 1, 1, 2, 0, 2, 0],       [2, 0, 1, 1, 1, 0, 0, 0, 0, 2, 0, 0],       [0, 1, 0, 0, 0, 0, 1, 1, 2, 0, 2, 0],       [1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1]])
All we need to do now is add labels and transform it back to a dataframe.
fruits_freq = pd.DataFrame(fruits_freq_mat, columns = unique_items.keys(), index = unique_items.keys())
If you are looking for a nice visualization, you can create a heatmap with the seaborn library.
import seaborn as snfig, ax = plt.subplots(figsize = (9,5))sn.heatmap(fruits_freq, cmap = "Blues")plt.xticks(rotation=50)plt.savefig("heatmap.png", dpi = 300)
With the Pearson matrix, we can easily build a fruit recommender system. For example, if you input that you like bananas, it will recommend you a maracuja, because those two have the highest correlation (0.67). You will be surprised by how powerful this simple approach is. I have used it successfully multiple times. If you want to do something like this with the frequency dataframe, you need to normalize the data first. However, that is a topic for another post.
I hope this guide was useful to you and saved you some time or inspired you. Thank you all for reading!