Dealing with List Values in Pandas Dataframes
By: Max Hilsdorf
Tags: Data Analysis, Programming, Python, Pandas, Data Science

Problem 3: Individual Columns for All Unique Values

At this point, things are getting advanced. If you are happy with the results we got before, you can stop here. However, a deeper level of analysis might be required for your research goal. Maybe you want to correlate all list elements with each other to compute similarity scores. E.g. do kids who eat bananas typically also like mangos? Or maybe you want to find out which fruit has been ranked as the top favorite fruit by the most kids. These questions can only be answered at a deeper level of analysis.

For this, I will introduce two useful methods. They differ in complexity, but also in what you can do with their results.

Method 1

This is a shockingly easy and fast method I stumbled upon. And it is so useful! All you need is one line of code.

fruits_expanded_v1 = fruits["favorite_fruits"].apply(pd.Series)

Figure 5 — Expanded version of the fruit lists using method 1.

As you can see, this one-liner produced a dataframe where every list is split into its single elements. The columns indicate the order, in which the fruit was placed in the list. With this method, you will always get a dataframe with a shape of (n, len(longest_list)). In this case, two of the 10 children named five favorite fruits, which results a 10x5 dataframe.

Using this, we can find out which fruit was named most often as the number one favorite fruit.

fruits_expanded_v1.iloc[:,0].value_counts(normalize = True) ## OUTPUT ##

banana 0.222222 pear 0.111111 watermelon 0.111111 blueberry 0.111111 strawberry 0.111111 apple 0.111111 peach 0.111111 mango 0.111111

We can see that bananas are most often kids’ absolute favorite fruit.

Alternatively, we could target single fruits and find out how many times they were named at each position of the lists. This is the function I wrote for that:

def get_rankings(item, df):



# Empty dict for results

item_count_dict = {}



# For every tag in df

for i in range(df.shape[1]):



# Calculate % of cases that tagged the item

val_counts = df.iloc[:,i].value_counts(normalize = True)

if item in val_counts.index:

item_counts = val_counts[item]

else:

item_counts = 0



# Add score to dict

item_count_dict["tag_{}".format(i)] = item_counts



return item_count_dict

If we apply it, we get:

get_rankings(item = "apple", df = fruits_expanded_v1) ## OUTPUT ##

{'tag_0': 0.1111111111111111, 'tag_1': 0.1111111111111111, 'tag_2': 0.2222222222222222, 'tag_3': 0.2, 'tag_4': 0}

As you can see, we can perform rank-based analyses very well with this approach. However, this method is near useless for other approaches. Because the columns do not represent a single tag, but a rank, most tag-based operations can not be done properly. For example, calculating the correlation between bananas and peaches is not possible with the dataframe we got from method 1. If that is your research goal, use the next method.

Method 2

This method is more complex and requires more resources. The idea is that we create a dataframe where rows stay the same as before, but where every fruit is assigned its own column. If only kid #2 named bananas, the banana column would have a “True” value at row 2 and “False” values everywhere else (see Figure 6). I wrote a function that will perform this operation. It relies on looping, which means that it will take lots of time with large datasets. However, out of all the methods I tried, this was the most efficient way to do it.

def boolean_df(item_lists, unique_items): # Create empty dict

bool_dict = {}



# Loop through all the tags

for i, item in enumerate(unique_items):



# Apply boolean mask

bool_dict[item] = item_lists.apply(lambda x: item in x)



# Return the results as a dataframe

return pd.DataFrame(bool_dict)

If we now apply the function

fruits_bool = boolean_df(fruits[“favorite_fruits”], unique_items.keys())

we get this dataframe:

Figure 6 — Boolean dataframe.

From here, we can easily calculate correlations. Note that “correlation” is not really the correct term, because we are not using metric or ordinal, but binary data. If you want to be correct, use “association”. I will not.

Again, there are multiple ways to correlate the fruits. One straight forward way is the Pearson correlation coefficient, which can also be used for binary data. Pandas has a built-int function for this.

fruits_corr = fruits_bool.corr(method = "pearson")

Figure 7 — Pearson correlation dataframe.

Another way is to simply count how many times a fruit was named alongside all other fruits. This can be solved using matrix multiplication. For this, we will need to convert the boolean dataframe to an integer based on first.

fruits_int = fruits_bool.astype(int)

Then, we can calculate the frequencies.

fruits_freq_mat = np.dot(fruits_int.T, fruits_int) ## OUTPUT ## array([[5, 3, 3, 2, 2, 1, 1, 1, 0, 2, 0, 1],

[3, 4, 2, 1, 1, 1, 1, 2, 1, 0, 1, 1],

[3, 2, 4, 3, 1, 2, 0, 0, 0, 1, 0, 0],

[2, 1, 3, 4, 2, 2, 0, 0, 0, 1, 0, 0],

[2, 1, 1, 2, 3, 1, 0, 0, 0, 1, 0, 0],

[1, 1, 2, 2, 1, 3, 0, 0, 0, 0, 0, 0],

[1, 1, 0, 0, 0, 0, 2, 1, 1, 0, 1, 1],

[1, 2, 0, 0, 0, 0, 1, 2, 1, 0, 1, 1],

[0, 1, 0, 0, 0, 0, 1, 1, 2, 0, 2, 0],

[2, 0, 1, 1, 1, 0, 0, 0, 0, 2, 0, 0],

[0, 1, 0, 0, 0, 0, 1, 1, 2, 0, 2, 0],

[1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1]])

All we need to do now is add labels and transform it back to a dataframe.

fruits_freq = pd.DataFrame(fruits_freq_mat, columns = unique_items.keys(), index = unique_items.keys())

Figure 8 — Frequency dataframe.

If you are looking for a nice visualization, you can create a heatmap with the seaborn library.

import seaborn as sn fig, ax = plt.subplots(figsize = (9,5))

sn.heatmap(fruits_freq, cmap = "Blues")

plt.xticks(rotation=50)

plt.savefig("heatmap.png", dpi = 300)

Figure 9 — Heatmap.

With the Pearson matrix, we can easily build a fruit recommender system. For example, if you input that you like bananas, it will recommend you a maracuja, because those two have the highest correlation (0.67). You will be surprised by how powerful this simple approach is. I have used it successfully multiple times. If you want to do something like this with the frequency dataframe, you need to normalize the data first. However, that is a topic for another post.