Working with missing values in Pandas | by B. Chen | Towards Data Science

The real-world data is rarely clean and homogeneous. In particular, many interesting datasets will have some amount of values missing.
In this article, we will discuss how missing value is represented in Pandas, how to deal with other characters representations and Pandas built-in methods for handling missing values.
Outline
Missing values in Pandas
Dealing with other characters representations
Handling missing values
Schemes for indicating the presence of missing values are generally around one of two strategies [1]:
A mask that globally indicates missing values.A sentinel value that indicates a missing entry.
A mask that globally indicates missing values.
A sentinel value that indicates a missing entry.
In the mask approach, it might be a same-sized Boolean array representation or use one bit to represent the local state of missing entry.
In the sentinel value approach, a tag value is used for indicating the missing value, such as NaN (Not a Number), null or a special value which is part of the programming language.
None of these approaches is without trade-offs [1]. The mask approach requires allocations of an additional Boolean array, which adds overhead in both storage and computation. A sentinel value reduces the range of valid values that can be represented and may require extra logic in CPU and GPU arithmetic.
Pandas use sentinels to handle missing values, and more specifically Pandas use two already-existing Python null value:
the Python None object.
the special floating-point NaN value,
The first sentinel value used by Pandas is None, a Python ‘object’ data that is most often used for missing data in Python code. Because it is a Python object, None cannot be used in any arbitrary NumPy/Pandas array, but only in arrays with data type ‘object’.
import numpy as npimport pandas as pdarr1 = np.array([1, None, 2, 3])arr1
That will output
array([1, None, 2, 3], dtype=object)
The use of Python object in an array also means that you will generally get an error if you perform aggregations like sum() or min().
The second sentinel value used by Pandas is NaN, is acronym for Not a Number and a special floating-point value use the standard IEEE floating-point representation.
arr2 = np.array([1, np.nan, 3, 4]) arr2.dtype
That will output
dtype('float64')
Keep in mind that NaN is specifically a floating-point value; there is no equivalent value for integers, strings or other types. And you should be aware that regardless of the operations, the result of arithmetic with NaN will be another NaN . For example
arr2.sum()arr2.min()arr2.max()
Will all output NaN. NumPy does provide some special aggregations that will ignore the missing data, such as nansum() , nanmin() , and nanmax() .
Pandas is built to handle the None and NaN nearly interchangeably, converting between them where appropriate:
pd.Series([1, np.nan, 2, None])0    1.01    NaN2    2.03    NaNdtype: float64
For types that don’t have an available sentinel value, Pandas automatically type-casts when NaN values are present.
For example, let’s create a Panda Series with dtype=int.
x = pd.Series(range(2), dtype=int)x0    01    1dtype: int64
If we set a value in an integer array to np.nan, it will automatically be upcast to a floating-point type to accommodate the NaN:
x[0] = Nonex0    NaN1    1.0dtype: float64
Not all missing values come in nice and clean np.nan . If we know what kind of characters used as missing values in the dataset, we can handle them while creating the DataFrame using na_values parameter:
df = pd.read_csv("source.csv", na_values = ['?', '&'])
When the DataFrame is already created, we can use pandas replace() function to handle these values:
df_clean = df.replace({ "?": np.nan, "&": np.nan })
Before we diving into the details, let’s create a DataFrame with some missing values:
import numpy as npimport pandas as pddf = pd.DataFrame({'A': [20, 15, 20],                  'B': [np.nan, 50, 4],                  'C': [30, 5, 30,],                  'D': [15, 2, np.nan],                  'E': [8, 5, 10],                  'F': [45, 7, np.nan],                  'G': [35, 10, 35]},                  index = ['Row 1', 'Row 2',  'Row 3'])df
Pandas has two useful methods for detecting missing values: isnull() and notnull() . Either one will return a Boolean mask over the data. For example:
df.isnull() returns a Boolean same-sized DataFrame indicating if values are missing
df.notnull() returns a Boolean same-sized DataFrame which is just opposite of isnull()
When the dataset is large, you can count the number of missing values instead.
df.isnull().sum() returns the number of missing values for each column (Pandas Series)
df.isnull().sum()A    0B    1C    0D    1E    0F    1G    0dtype: int64
df.isnull().sum().sum() returns the total number of missing values
df.isnull().sum().sum()3
In addition to the masking used before, there is the convenience method dropna() to remove missing values [2]. The method is defined as:
dropna(axis=0, how=’any’, thresh=None, subset=None, inplace=False)
axis: 0 for row and 1 for column
how: ‘any’ for dropping row or column if any NaN values are present. ‘all’ to drop row of column if all values are NaN.
thresh: require that many non-NaN values
subset: array-like value. Labels along other axis to consider, e.g. if you are dropping rows these would be a list of columns to include.
inplace: If True, do operation inplace and return None.
To recap, here is the DataFrame df we are using
To drop row if any NaN values are present:
df.dropna(axis = 0)
To drop column if any NaN values are present:
df.dropna(axis = 1)
To drop row if the number of non-NaN is less than 6.
df.dropna(axis = 0, thresh = 6)
Data is a valuable asset so we should not give it up easily. Also, machine learning models almost always tend to perform better with more data. Therefore, depending on the situation, we may prefer replacing missing values instead of dropping.
There is the convenience method fillna() to replace missing values [3]. The method is defined as:
fillna(value=None, method=None, axis=None, inplace=False)
value: value to use to replace NaN
method: method to use for replacing NaN. method='ffill' does the forward replacement. method='bfill' does the backword replacement.
axis: 0 for row and 1 for column.
inplace: If True, do operation inplace and return None.
To recap, here is the DataFrane df we are using
To replace all NaN values with a scalar
df.fillna(value=10) 
To replaces NaN values with the values in the previous row.
df.fillna(axis=0, method='ffill')
To replace NaN values with the values in the previous column.
df.fillna(axis=1, method='ffill')
The same, you can also replace NaN values with the values in the next row or column.
# Replace with the values in the next rowdf.fillna(axis=0, method='bfill')# Replace with the values in the next columndf.fillna(axis=1, method='bfill')
The other common replacement is to replace NaN values with the mean. For example to replace NaN values in column B with the mean.
df['B'].fillna(value=df['B'].mean(), inplace=True)
And that’s about it. Thanks for reading.
[1] Python Data Science Handbook
[2] Pandas Official documentation — dropna
[3] Pandas Official documentation — fillna