The Nullable NaN. When NA is not False. | Towards Data Science

When something is nothing, and nothing is something...for boolean data in Pandas, there is crucial difference between NaN, Null, NA, and bools — a brief on when and how to use them.
Task: Clean a Pandas DataFrame comprising boolean (true/false) values to optimize memory. A constraint is to retain all null values as nulls, i.e. don’t turn null values to False because that is a meaningful change.
Action: Explicitly transform column dtypes, i.e. use float32 instead of float64 to conserve memory and bool instead of object.
Problem: When transforming selected columns to bool, all rows evaluate to either all True or all False and returns a bad headache.
To preserve null-like values in combination with boolean values, replace null values explicitly with pd.NA and set dtype to ‘boolean’ instead of just ‘bool’ — this is the boolean array.
Takeaway: When the source column contains null values or non-boolean values such as floats like 1.0, applying the Pandas ‘bool’ dtype may erroneously evaluate all rows to True. Instead, replace null values explicitly with pd.NA and set dtype to ‘boolean’ instead of just ‘bool.’
While building and deploying apps, I have become hyper-focused on making things run as fast as possible. To that end, I initially started taking big swings by broadly setting Pandas dtypes to categorical, bool, or float32 wherever it was an option. However, although I saved massively on memory, I later found out that I took a hit on data integrity.
Read on to learn how to fix and avoid this mistake.
When forcing a column to a bool dtype, there is a nearly instant memory savings on the order of about half. While that’s enticing, there is another side to the coin. When the source data has something other than only True/False values, Pandas will return some unwanted but logically correct results.
For example, let’s say you have a few ‘flag’ columns in a DataFrame that indicate some kind of true/false combination.
# unique values of columns in a DataFrame# col1 -> False True[False True]# col2 -> True True                     [nan   True]    # col3 -> True True              [nan   1.] 
In col1, the unique values are only False and True — this is ideal and applying a dtype of bool works perfectly.
In col2, we have a problem. Although we have a True value, we also have a nan value. The ‘nan’ represents the Pandas “Not A Number” which is a computer’s way of knowing there is supposed to be nothing there. While I won’t go deep into the logical hell (TDS has been there already), it should suffice to say that setting col2 to dtype bool will evaluate each row to True. This is bad because rows that should not be True are now True and rows that are supposed to be nothing now have something.
In col3, we have the same problem as col2 but now have a floating point number of 1. (1. is computer short-hand for 1.0). Much like col2, and without any intervention, the Pandas dtype bool will evaluate to True for this column and cause problems if we filter by this flag later.
To resolve this situation you may try, as I did, to map the data with a dictionary, then set the dtype to bool. For instance, if 1. should be True, then just map that and get over it, right? Not so fast.
# a tempting but spurious solution# given a dataframe as dfimport pandas as pdimport numpy as npkey = {'nan': np.nan,       1.: True}df['col1'] = df['col1].map(key)df['col1'] = df['col1].astype(bool)# this will not work like you might think
The above code snip does not work for the primary reason that it tries to map the string of ‘nan’ to a special type of NaN value — this will never work. To be sure, if you literally have ‘nan’ as a string, the dictionary map returns np.nan, but then what? As we know from the col3 example, it all just ends up as True values and we are right back at square one.
To illustrate how to get Nulls, NaNs, and bools to work for you, consider a sample table and code via GitHub Gist that covers the issues so far with actual data.
Usually, we might try to test for equivalence with if-statements, but for a lot of reasons that I won’t go into in this article, traditional conditionals will not work with nulls.
a = 1b = 1if b == a:   print('this will work but not for NaNs')
Although it may sound incredibly weird, the boolean array allows you to carry more than two values in a boolean column.
Instead, use the isnull() method in conjunction with something like np.where() to test for and find data. To replace data, use pd.NA instead of np.nan. Next, in the case of a floating point 1.0, we can nest a second np.where() statement and map 1.0 to True. Finally and lastly, use the dtype of ‘boolean’ instead of ‘bool’ to create a boolean array. Although it may sound incredibly weird, the boolean array allows you to carry more than two values in a boolean column.
Optimization is fantastic and turning Pandas objects into bools to save memory is a key component. However, if not done carefully, the results can be painful. In this article, I describe some pitfalls with transforming boolean or True/False data while retaining null values as pd.NA. The result is the ability to retain the source table’s data integrity while leveraging the memory savings and convenience of boolean in Pandas.
Although the concept of nullable boolean arrays with more than two values may seem strange, it totally works and can help you leverage the best of multiple worlds!
Pandas Stable Docs on Missing Data
Pandas Stable Docs on Boolean Data
Thanks for reading! Clap or Follow me to stay tuned for my stories on programming, data science, and artificial intelligence.