Cleaning Missing Values in a Pandas Dataframe | by Andrei Teleron | Towards Data Science

Ways to visualize missing data and what to do with them using Python.
When working on analyzing data, you’ll likely come across data that is missing (also called null values or NaNs). Data cleaning is an important part of the data analysis pipeline and making sure that it’s all tidy up will make your analysis much stronger.
The only library you’ll need to import is pandas:
import pandas as pd
If you’re using the pandas library in Python and are constantly dealing with data that has missing values and need to get to your data analysis faster, then here’s a quick function that outputs a dataframe that tells you how many missing values and their percentages in each column:
A little less readable version, but you can copy paste it in your code:
def assess_NA(data):    """    Returns a pandas dataframe denoting the total number of NA values and the percentage of NA values in each column.    The column names are noted on the index.        Parameters    ----------    data: dataframe    """    # pandas series denoting features and the sum of their null values    null_sum = data.isnull().sum()# instantiate columns for missing data    total = null_sum.sort_values(ascending=False)    percent = ( ((null_sum / len(data.index))*100).round(2) ).sort_values(ascending=False)        # concatenate along the columns to create the complete dataframe    df_NA = pd.concat([total, percent], axis=1, keys=['Number of NA', 'Percent NA'])        # drop rows that don't have any missing data; omit if you want to keep all rows    df_NA = df_NA[ (df_NA.T != 0).any() ]        return df_NA
The only parameter you need to pass is a dataframe object. Here’s an example where we call the function assess_NA() on a dataframe with missing data called training , which then outputs the dataframe: df_NA .
The rows represent the features of your dataframe and the columns provide information on your missing data. If there are no missing values, then it will just output an empty dataframe.
Knowing this, you can be more informed on what to do with null values such as:
Removing rows with themImpute using mean, median, 0, false, true, etc.
Removing rows with them
Impute using mean, median, 0, false, true, etc.
This method is a simple, but messy way to handle missing values since in addition to removing these values, it can potentially remove data that aren’t null. You can call dropna() on your entire dataframe or on specific columns:
# Drop rows with null valuesdf = df.dropna(axis=0)# Drop column_1 rows with null valuesdf['column_1'] = df['column_1'].dropna(axis=0)
The axis parameter determines the dimension that the function will act on. axis=0 removes all rows that contain null values. axis=1 does nearly the same thing except it removes columns instead.
Rather than dropping values with missing data, imputation looks to replace these values with another value — usually the mean or median of a specified column. There are benefits to using either. For example, if the column has a lot of outliers the median would probably be more useful since it is more resistant to them. This way, we are attempting to preserve aspects of the data. To do this, we can call the fillna() function on a dataframe column and specifying either mean() or median() as a parameter:
# Impute with mean on column_1df['column_1'] = df['column_1'].fillna( df['column_1'].mean() )# Impute with median on column_1df['column_1'] = df['column_1'].fillna( df['column_1'].median() )
Besides mean and median, imputing missing data with 0 can also be a good idea in some cases:
# Impute with value 0 on column_1df['column_1'] = df['column_1'].fillna(0)
Knowing what path to take when handling missing data is largely dictated by domain-knowledge and what your intuition tells you about the data. This is gained step-by-step and by working with datasets that allow you to ask meaningful questions about them.