Data Cleaning is an art..
By: Akshay Vijay
Tags: Seaborn, Pandas, Numpy, Python

Before any data scientist can concentrate on modeling, they must first master data cleansing. The complexity of your modeling will be determined by how successfully you clean your data. The cleaner your data sets are during the cleansing stage, the simpler the learning algorithms will be during the modeling stage.

Once you’ve mastered data cleaning, you may expect:

Reduced processing times More accurate forecasts Simplified algorithm functionality Increased model learning

Dirty data simply refers to incorrect data. Duplicate records, insufficient or obsolete data, and incorrect processing can make the data really messy . This information must be cleansed. Data cleaning (or data cleansing) refers to the process of “cleaning” this dirty data by finding and correcting mistakes.

Let’s begin.

Let’s import the required libraries.

import pandas as pd

import numpy as np

import seaborn as sn

Download the dataset and read it using read_csv() function. For this project we use the dataset from data.gov website. The dataset is provided by the National Cardiovascular Disease Surveillance System. Its integrated from multiple indicators from many data sources to provide a comprehensive picture of the public health burden of CVDs and associated risk factors in the United States.

df = pd.read_csv('Center_for_Medicare___Medicaid_Services__CMS____Medicare_Claims_data.csv')

Before we start cleaning the dataset, let’s conduct some fundamental analysis, such as analyzing the data’s structure, datatypes, and numerical / non-numerical columns.

df.shape

df.describe()

df.info()

What did we do now?

Shape: to obtain the data’s rows and columns. Describe() : returns the fundamental statistics of the data, such as mean, median, mode, and so on. Info() represents the datatypes

Let’s use the numpy library to distinguish the numerical and non-numerical columns using the code below,

df_numeric = df.select_dtypes(include=[np.number])

numeric_cols = df_numeric.columns.values

print(“Numeric columns: “, numeric_cols) df_non_numeric = df.select_dtypes(exclude=[np.number]) non_numeric_cols = df_non_numeric.columns.values

print(“Non numeric columns: “, non_numeric_cols)

Let’s start by visualizing the dataset with the heatmap command from the seaborn library to detect the missing values.

sn.heatmap(df.isnull(),yticklabels=False)

What do we consider as missing values?

Missing values can be represented by a ‘?’, ‘N/A’, 0 or a blank cell. This missing data might have an impact on machine learning models, resulting in inaccurate predictions.

So, how do we handle these missing values?

Check with the data collecting source to see if it is feasible to determine what the real value should be.

If you still have missing values to deal with, another approach is to just drop them. You may either remove the entire variable or only the data entry with the missing value.

If there aren’t too many missing values, it’s preferable to drop only that one item; otherwise, you’ll wind up wasting a lot of data, which is never a good thing.

Always ensure that the data you are deleting has the least possible influence on your target variable.

To prevent wasting data, replace the missing values. It lowers the accuracy to some amount because the value is merely an estimate.

Let’s calculate the proportion of missing values throughout the entire dataset.

rows, columns = df.shape

cell_count = rows * columns

number_of_nulls = df.isnull().sum().sum()

percentage_of_missing = (number_of_nulls / cell_count) * 100

print(f'Percentage of missing values: {percentage_of_missing}%')

As per the code, 7.005 % of the data set has missing values.

We must also ensure that no duplicate data is processed. Data de-duplication increases accuracy and efficiency. We utilize duplicated() in conjunction with sum to find duplicates ()

How do you identify duplicates? It’s pretty easy. We just use .duplicated() command along with .sum()

df.duplicated().sum()

To begin, consider the missing values in columns of float64 data types.

df.fillna({

'Data_Value': 0,

'LowConfidenceLimit': 0,

'HighConfidenceLimit': 0,

'GeoLocation': 0

}, inplace = True)

Let’s clear up the inconsistent values in other columns and replace them.

df.Data_Value_Footnote_Symbol.unique()



df.Data_Value_Footnote_Symbol.replace({

'~' : np.NaN

}, inplace = True)



df.Data_Value_Footnote_Symbol.unique() # Data_Value_Footnote_Symbol column is fixed



to_replace = ['Heart Failure', 'Diseases of the Heart (Heart Disease)']



df.Topic.replace(to_replace, 'Other - Heart Failure', inplace = True)

df.Topic.replace('Acute Myocardial Infarction (Heart Attack)',

'Acute Myocardial Infarction', inplace = True)

The code below shows the unique values in the column.

df.Data_Value_Footnote.unique()

We were able to view two types of data. One contains NaNs, whereas the other has “Statistically unstable estimations not provided [numerator 10].”

We need to look at the values for this data to see if the columns are usable or not. To filter down the results, use query().

num_less_than_10 = df.query('Data_Value_Footnote == "Statistically unstable estimates not presented [numerator <10]"') num_less_than_10.shape

A total of 529 entries were found, all of which are statistically unstable estimates that are not shown [numerator 10].

We use the code below to identify the states of these 529 entries.

num_less_than_10.LocationDesc.unique()

Let’s try to figure out which claim is linked to which of the 529 entries.

num_less_than_10.Topic.unique()

The code below displays the fraction of records that have statistically unstable estimates that are not shown [numerator 10].

# total dataframe

rows, columns = df.shape

tot_count = rows * columns



# subset of dataframe with low numerator

rows, columns = num_less_than_10.shape

low_tot_count = rows * columns



# identify the percentage

percentage_with_low_numerator = (low_tot_count / tot_count) * 100

print('Percentage of numerator < 10 in the dataset: ', percentage_with_low_numerator)

Let’s get rid of the columns that aren’t being used.

columns_to_drop = ['Data_Value_Footnote_Symbol',

'Data_Value_Footnote',

'CategoryId',

'TopicId',

'BreakOutId',

'BreakOutCategoryId',

'LocationID',

'IndicatorID',

'LocationAbbr',

'Data_Value_TypeID',

'Data_Value_Alt',

'Data_Value',

'Data_Value_Unit']



df.drop(columns_to_drop, axis = 1, inplace = True)

When we look at the data in heatmap, it seems lot cleaner now.

sn.heatmap(df.isnull(),yticklabels=False

The data is mostly structured by state across the United States, but I’m more interested in grouping it by Topic (which represents the claim) and also by buckets (age, gender, race)

We may split, apply, and merge the dataframe using the code below.

# group based on the Topic

g = df.groupby('Topic')



cor_df = g.get_group('Coronary Heart Disease')

other_df = g.get_group('Other - Heart Failure')

ami_df = g.get_group('Acute Myocardial Infarction')

major_df = g.get_group('Major Cardiovascular Disease')

stroke_df = g.get_group('Stroke')



# segregate coronary df by age, gender, race

cor_category = cor_df.groupby('Break_Out_Category')

age_cor = cor_category.get_group('Age')

gender_cor = cor_category.get_group('Gender')

race_cor = cor_category.get_group('Race')



# ssegregate other df by age, gender, race

other_category = other_df.groupby('Break_Out_Category')

age_other = other_category.get_group('Age')

gender_other = other_category.get_group('Gender')

race_other = other_category.get_group('Race')



# ssegregate ami df by age, gender, race

ami_category = ami_df.groupby('Break_Out_Category')

age_ami = ami_category.get_group('Age')

gender_ami = ami_category.get_group('Gender')

race_ami = ami_category.get_group('Race')



# segregate major df by age, gender, race

major_category = major_df.groupby ('Break_Out_Category')

age_major = major_category.get_group('Age')

gender_major = major_category.get_group('Gender')

race_major = major_category.get_group('Race')



# segregate stroke df by age, gender, race

stroke_category = stroke_df.groupby('Break_Out_Category')

age_stroke = stroke_category.get_group('Age')

gender_stroke = stroke_category.get_group('Gender')

race_stroke = stroke_category.get_group('Race')

Finally, we concatenate the data frame by bucket and reset the index before exporting it to Excel for further analysis.