Bitwise operators and chaining comparisons in Pandas | by Àlex Escolà Nixon | Towards Data Science

When learning Pandas, it is natural to have a tendency to use Python’s boolean operators ( and , or etc.) to chain conditions, since that is how it is done in Python. However, these are not the operators we should be using in Pandas (or NumPy). Let me explain why...
In Python, chaining comparisons is straight forward:
x = 4x==4 and x%2==0True
Here we’re just checking if x is equal to 4 and that it is an even number. And in the same way, we can chain multiple conditions together as:
b = 'random string'x==4 and x%2==0 and b.endswith('ing')True
This works because we are comparing boolean values (True or False), or truthy/falsy values (values which are evaluated to True or False), which produce, for the above example, an expression as True and True and True.
However, a common mistake is to think that the same will work with arrays of boolean values, as a result from applying some comparison. For instance:
x = np.array([2,4,6])x%2==0 and x!=3ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
As you can see, the above yields a ValueError. What has changed in this case? The difference is that we are no longer comparing boolean values, since x%2==0 yields:
array([ True, True, True])
Which will clearly neither be True nor False. Python is telling us through the error that we need to apply an aggregation function to the result to disambiguate, a.any() or a.all() . Doing so will reduce the array to a single value:
(x%2==0).any()True
Which can now be compared with another boolean value.
When we’re working with an array or a Pandas DataFrame column and want to filter based on more than one comparison, we’ll need to compute element-wise operations, in this case, logical element-wise operations.
We’ve discussed how python’s boolean operators are not what we want, since they require that the terms being compared can be evaluated as True or False. For logical element-wise operations we should instead be using python’s bitwise operators <<, >>, &, |, ~, and ^.
You can think of python’s bitwise operators as a boolean operator, but applied on an element-wise basis (on boolean arrays):
x%2==0array([ True, True, True])x==4array([False, True, False])(x%2==0) & (x==4)array([False, True, False])
So here we’ve applied a bitwise AND (&) between the two resulting boolean vectors, which produces another array of the same shape resulting from the element-wise operation applied over its elements.
Python has a bitwise operator equivalent for all boolean operators, as well as other operators which are listed bellow:
x & y does a “bitwise AND”. Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it’s 0.
x | y does a “bitwise OR” . Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it’s 1.
~x returns the complement of x (NOT ) - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x-1 .
x ^ y does a “bitwise exclusive or” ( XOR ). Each bit of the output is the same as the corresponding bit in x if that bit in y is0, and it’s the complement of the bit in x if that bit in y is 1.
And we also have the bitwise shift operators >> and << (see here).
Going back to the above example, an important aspect to take into account is that with bitwise operators parenthesis are important!
This is because bitwise operators have a higher precedence than comparison operators, meaning that the bitwise operation will precede that of the comparison.
In the Expressions section of the Python documentation, under Operator Precedence, we’ll find a table containing the precedence of all Python operators, from lowest precedence (least binding) to highest precedence (most binding):
To more clearly understand the above, say we have an expression with a multiplication operator (*) and an exponentiation operator (**):
3*2**212
Since the exponentiation operator has a higher precedence (see how it appears in a lower position in the table), python will first evaluate 2**2 , which is 4 and then 3*4=12. If we wanted to give the multiplication a higher precedence, we’d have to parenthesise the expression as:
(3*2)**236
Just how we’d write the mathematical expression too in this case.
Now the same applies to bitwise operators. Since == has a lower precedence than & , we’d need to parenthesise the comparison expressions so that they are evaluated first:
(x%2==0) & (x==4)
Another common mistake, is to try to chain operators as:
x = np.array([2,4,6])1<x<10ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
This is not supported by NumPy (or Pandas), and yields a ValueError. The problem here, is that Python is internally translating the above to:
(1 < x) and (x > 10)
Which again, takes us to the same error as in the example above: and is implicitly calling bool, and NumPy only permits implicit coercion to a boolean value for single elements (not arrays with size>1), since a boolean array with many values does not evaluate neither to True or False.
It was actually proposed to make this possible in PEP 535, though it still remains deferred.
In an upcoming post, I’ll discuss more advanced comparison operations in Pandas and review some useful tips to keep in mind.
Thanks a lot for taking the time to read this post and I hope you enjoyed it :)