Five Python Gotchas!. What to expect when you’re least... | by Ehsan Khodabandeh | Towards Data Science

Many posts out there catalog the “gotchas” of Python and/or its most popular packages. This blog is yet another entry in that series, but with one difference: I’ve actually, genuinely made all these mistakes myself (some of them with embarrassing frequency). However, just knowing the definition and type of each classic Python object should help you avoid most (if not all) of these mistakes in your work!
Without any further ado, let’s begin.
You probably know that for checking if an object’s value is True or False, you can do something like the following:
lst = [1, 2, 3]a = None# rather than this ...if len(lst) > 0 or a is not None: print('success')# you can simply do this ...if lst or not a: print('success')
That’s because empty lists (and all other empty sequences/collections), False, None, 0 (of any numeric type) all evaluate to False. This set of objects and values are therefore known as “falsy”.
Consider the following example: you have a dictionary of items and their costs, which you use to build a dataframe for some analysis.
import pandas as pdd1 = {'item': ['foo', 'bar', 'baz'], 'cost': [100, None, 20]}df = pd.DataFrame(d1)# lots of analysis here ...# if an item has a cost, print the item and its costfor i, r in df.iterrows():    if r['cost']:        print(f"item = {r['item']} and cost = {r['cost']}")
You expect:
item = foo, cost = 100.0item = baz, cost = 20.0
But you get:
item = foo, cost = 100.0item = bar, cost = nanitem = baz, cost = 20.0
The reason is that Pandas considers None to be missing or unavailable, and thus represents it with nan. Since nan is not falsy, it flows through.
This may seem counterintuitive at first, but nans are missing values, and if something is missing, you don’t really know what it is. For example, could a given nan in a numeric column represent 100 (truthy) or 0 (falsy)? If it’s a string, is it ‘hello’ (truthy) or an empty string (which, you guessed it, is falsy)? Pandas isn’t sure, so it doesn’t assume falsiness.
Be careful to avoid considering nan falsy.
Consider the example below:
>>> s1 = {True, 1, 1.0}>>> s1{True}
This is expected, as we know that 1==1.0==True evaluates to True.
Now take a look at this case:
>>> s2 = {float('nan'), float('nan'), float('nan')}>>> s2{nan, nan, nan}
Consider the logic from the previous example: since nan is a missing value, it’s impossible to know if the three missing values are the same. Consequently, nan == nan always evaluates to False. (If you’d like to read more about this, make sure to check out the PEP 754 — IEEE 754 Floating Point Special Values.)
I had thought any and all work as follows: If you have an iterable, any will return True if any element of the iterable is true. And all will return True if all the elements of the iterable are true.
So, let’s check them out. First, any:
>>> any([0, None, False])  # expected FalseFalse>>> any([0, None, False, 'False'])  # expected TrueTrue>>> any([float('nan'), 0])  # expected TrueTrue>>> any([])  # expected FalseFalse
So far, so good! Let’s check all:
>>> all((1, 0, 0))  # expected FalseFalse>>> all([float('nan'), 1])  # expected TrueTrue>>> all([])  # expected FalseTrue
I thought all the elements should be true and an empty list is clearly falsy. So why did all([]) evaluate to True?
The answer is in Python documentation (my emphasis added):
all(iterable)
Return True if all elements of the iterable are true (or if the iterable is empty).
But why is that? Well, long story short, it’s because of vacuous truth. “If I am seven feet tall, I am also a trillionaire superhero” is always true for me. I’m about six inches short of that (disregard any of my colleagues’ testimony to the contrary), so the statement is true no matter what comes after the comma. It’s a statement that’s never false because I’m not seven feet tall, and therefore it’s impossible to evaluate my trillionaire superhero status.
Apart from reading the documentation more closely, the best way to remember this behavior of all() is to think of it not as “if all the elements of the iterable are true,” but as “if there is no false element in the iterable.” When the iterable is empty, there can be no false element within it, meaning all([]) evaluates to True.
I think this is by far the most common Python “gotcha”. Let’s just dive in.
Consider the following function:
def foo(a, a_list=[]):    a_list.append(a)    return a_list
Let’s use this function foo to create two separate lists:
>>> my_list = foo(10)>>> print(my_list)  # expected [10][10]>>> another_list = foo(20)>>> print(another_list)  # expected [20][10, 20]
You might have expected that each function call creates a new list, but the list created during the first invocation of foo is used in each successive call.
This happens because, in Python, default arguments are evaluated only when the function is defined, not each time the function is called (you can read more about that in Python’s documentation). If we use a mutable default argument (like the a_list=[] in foo) and change it in the function, that object is mutated whenever the function is called.
The best way to avoid any such confusion is to use an immutable default argument in the function. Here’s an accordingly updated version of foo:
def foo_v2(a, a_list=None):    if a_list is None:        a_list = []    a_list.append(a)    return a_list
This definition of foo assumes that having a new list in each call is desirable. However, there are cases in which you might wish to intentionally pass a mutable object. One such situation is when writing recursive functions, which require that an object’s state is saved from one call to the next. The following implementation of the depth-first search (DFS) algorithm is an example of one such case:
def dfs(graph, node):    """dfs from a given node"""    return _dfs(graph, node, [])def _dfs(graph, node, path)    """interior utility function"""    path += [node]    for neighbor in graph[node]:        if neighbor not in path:            path = _dfs(graph, neighbor, path)    return path>>> graph = {'A': ['B', 'C'], 'B': ['D', 'E'], 'C': ['G'],          'D': ['F'], 'E': [], 'F': [], 'G': []}>>> print(dfs(graph, 'A'))  # give the path starting from node 'A'['A', 'B', 'D', 'F', 'E', 'C', 'G']
I’ve already discussed this last “gotcha” in my bite-sized recipes post, but since I’ve seen several people fall into this trap firsthand, I’ll mention it here too.
Assume you want to remove all the numbers less than 5 from a list.
Wrong Implementation: Remove the elements while iterating!
nums = [1, 2, 3, 5, 6, 7, 0, 1]for ind, n in enumerate(nums):    if n < 5:        del(nums[ind])# expected: nums = [5, 6, 7]>>> nums[2, 5, 6, 7, 1]
Correct Implementation:
Use list comprehension to create a new list containing only the elements you want:
>>> id(nums)  # before modification 2090656472968>>> nums = [n for n in nums if n >= 5]>>> nums[5, 6, 7]>>> id(nums)  # after modification2090656444296
You can see above that id(nums) is checked before and after to show that in fact, the two lists are different. So, if the list is used in other places and it’s important to mutate the existing list, rather than creating a new one with the same name, assign it to the slice:
>>> nums = [1, 2, 3, 5, 6, 7, 0, 1]>>> id(nums)  # before modification 2090656472008>>> nums[:] = [n for n in nums if n >= 5]>>> id(nums)  # after modification2090656472008
I hope you find this blog useful. I can be reached on Twitter and LinkedIn and I welcome any feedback.