Python Sets are Underrated
By: Thomas Hikaru Clark
Tags: Getting Started, Computer Science, Python, Editors Pick, Data Science

Let’s now go over some common and helpful use cases for sets.

1. Remove duplicates from a list

A fundamental property of sets is that they do not store repeats. Suppose you have the list letters = [“a”, “b”, “d”, “e”, “a”, “b”, “b”, “c”, “d”] .

By converting it to a set and then back to a list, you can remove duplicates.

list(set(letters)) Output: ["a", "b", "d", "e", "c"]

You can use this to quickly get the total vocabulary of a text or to get all the unique characters used in a document. This can be very helpful in the preprocessing stage of a machine learning or NLP task. Of course, this also works with data types besides text, including numbers and immutable objects.

2. Find intersections, unions, and differences

The following are three important operations that can be done with sets:

Intersection (A & B)

Union (A | B)

Difference (A - B)

These operations are best understood using Venn diagrams. In each of the following figures, the top circle is A and the bottom circle is B. The green area represents the set that results from each operation.

Image by Author.

Intersection. Let’s say you want to know which words are shared in common between two texts. This corresponds to the left figure above, in which we select just the overlapping portions of the sets. Python sets make it easy to create vocabulary sets for two (or more) texts, and then calculate the intersection, as shown in the code below.

# read in two files and create a vocabulary set for each

# makes everything lowercase so we ignore capitalization with open(filename1) as f:

vocab1 = set(f.read().lower().split())

with open(filename2) as f:

vocab2 = set(f.read().lower().split()) print(vocab1 & vocab2)

Note the similarity between the & operator and the logical and operator && in many languages. You can remember this, since this operator only chooses items that are present in set A and in set B. As an alternative to the & operator, the .intersection() method can be called, as in the following example:

print(vocab1.intersection(vocab2))

Union. This operation merges two sets. It corresponds to the center figure above, in which we choose the total area covered by both sets together. Following the fundamental property of sets, duplicates will not be stored. We could use this to find, for example, all the unique words that appear in two (or more) different books.

set_one = {"a", "b", "c"}

set_two = {"c", "d", "e"}

set_union = set_one | set_two

print(set_union)



>> {"a", "b", "c", "d", "e"}

Note the similarity between the | operator and the logical or operator || in many languages. You can remember this, since this operator chooses items that are present in set A or in set B. As an alternative to the | operator, the .union() method can be called.

Difference. This operation removes elements of one set if they also belong to the other set. It corresponds to the right figure above. Unlike the previous two operations, set difference is not commutative, so A - B is not the same as B - A. Can you explain in words what the following code does?

# read in two files and create a vocabulary set for each

with open(filename1) as f:

vocab1 = set(f.read().lower().split())

with open(filename2) as f:

vocab2 = set(f.read().lower().split()) print(vocab1 - vocab2)

Answer: the code prints words in the first file that are not present in the second file. This could be useful, as one example, when analyzing performance for language models on out-of-vocabulary words. As an alternative to the - operator, the .difference() method can be called.

Chaining operations. Multiple operations can be chained together, like so:

a & b & c & d & ...

a | b | c | d | ...

a — b — c — d — ...

Additional operations. There are additional operations that check if one set is a (strict) superset or (strict) subset of another, using the following operators: >, >=, <, <= . I am not going to go into too much detail on these, because you can actually implement these operations using the three operations we have already discussed. For more information, check out the Python documentation on sets.

3. Fast query for membership

It is very fast to check membership of an element in a set. This is because the underlying data structure of a Python set is a hash table, making them similar to Python dictionaries, just with keys only instead of key-value pairs. A lot of algorithms that would take linear time for a Python list would take near-constant time for a set.

On the surface, using a list and a set to check membership might look pretty much identical:

# Read a bunch of words from a file and store them as a list and set

with open(filename) as f:

my_list = f.read().split()

my_set = set(my_list) if "hello" in my_list:

print("The word 'hello' was found in the list.") if "hello" in my_set:

print("The word 'hello' was found in the set.")

However, if there are thousands or millions of items in my_set and my_list , then we will start to notice differences. The following plot shows the results of some experiments I ran to compare membership check time for lists and sets. As the size of a list or set gets bigger, the difference in time between lists and sets grows. For applications where you will be repeatedly looking up whether an item exists or has been seen before, consider using a set rather than a list.

Image by Author.

Conclusion

There are lots of situations in which sets can make your life easier or speed up a computation. In addition, there is something very satisfyingly Pythonic about removing duplicates from a list using list(set(my_list)) or performing one-line set operations without needing to resort to for loops. This is not an exhaustive list of set applications, so feel free to leave a comment if you think I left out any other uses of sets!

References

[1] Python Built-in Types Documentation, https://docs.python.org/3/library/stdtypes.html#set