The Art of Cleaning Your Data. Drop that bad data like Obama drops... | by George Seif | Towards Data Science

Want to be inspired? Come join my Super Quotes newsletter. 😎
Cleaning your data should be the first step in your Data Science (DS) or Machine Learning (ML) workflow. Without clean data you’ll be having a much harder time seeing the actual important parts in your exploration. Once you finally get to training your ML models, they’ll be unnecessarily more challenging to train. The main point is that if you want to get the most out of your data, it should be clean.
In the context of data science and machine learning, data cleaning means filtering and modifying your data such that it is easier to explore, understand, and model. Filtering out the parts you don’t want or need so that you don’t need to look at or process them. Modifying the parts you do need but aren’t in the format you need them to be in so that you can properly use them.
Here we’ll look at a few things we might typically want to clean up in our data along with the pandas code you can use to do it!
Large datasets are rarely perfectly full. By full I mean that all of the data points have values for all of their features variables. There is usually some missing values which, when loaded up in pandas with something like pd.read_csv() are labelled as NaN or None. There are a number of quite practical reasons why such data could be missing. The person collecting the data may have simply forgot or perhaps they didn’t start collecting that feature variable until halfway through the data collection process.
Missing data absolutely needs to be handled before working with your dataset. For example, let’s say you are in the middle of your data exploration process and you discover some key information about your data coming from a certain feature variable, say “variable F”. But later on you discover that 95% of the values for “variable F” in your dataset are NaNs; you can’t possibly make any concrete conclusions about your dataset from a variable which only properly represents 5% of it! When you get to training your ML model, NaNs might also be treated as 0 or infinity by your program, totally throwing off your training!
There are a few ways to handle missing data in pandas:
Check for NANs: pd.isnull(object) Detect missing values in your data; this detects both “NaN” and “None”
Drop missing data: df.dropna(axis=0, how='any') Returns your data frame where any data points containing NaNs have been removed.
Replace missing data: df.replace(to_replace=None, value=None) Replace values given in "to_replace" with "value". This is useful if you know some value that you’d like this feature variable to be.
Drop a feature: df.drop('feature_variable_name', axis=1) If you find that a certain feature variables has > 90% of NaN values in the dataset, it makes sense to just drop the whole thing from your data.
The outliers in your dataset are a mixed bag. One the one hand, they may contain key information since they are so different from the main group. On the other hand they throw off our view of the main group since we have to look so far out just to see the outliers! On the ML side of things, training with outliers included may allow your model to generalise well but also throw it off from the main group where most of your data is.
In general, it’s usually recommended to look at both sides. Explore your data both with and without outliers. If you decide that you need them for your ML model then select a method that will be robust enough to handle them. If you find that those outliers are really out there and not useful for getting the big picture information and modelling of your data, then it’s best to just drop them as shown in the last section.
In addition, if you want to filter out those outliers you can use the following method:
# Get the 98th and 2nd percentile as the limits of our outliersupper_limit = np.percentile(train_df.logerror.values, 98) lower_limit = np.percentile(train_df.logerror.values, 2) # Filter the outliers from the dataframedata[‘target’].loc[train_df[‘target’]>upper_limit] = upper_limit data[‘target’].loc[train_df[‘target’]<lower_limit] = lower_limit
Bad data means any data points or values that shouldn’t be there or are just plain wrong. For example, suppose one of your feature variables is called “gender” where most of the values are “male” or “female”. But then as you’re skimming through your dataset you notice that there are a couple of data points that have the value 67.3 for gender! Clearly 67.3 doesn’t mean anything in the context of that variable. Moreover, if you try to convert the “gender” feature variable to categorical floats: male = 0.0 and female = 1.0, you’a have an extra one: 67.3 = 2.0!
Duplicates are simply data points which are exactly repeated in your dataset. If there are too many of these it throws off the training of your ML model. Duplicates can simply be dropped from your data as we saw before.
Bad data can either be handled either by dropping or using some intelligent replacement. For example, we might look at the data points which have 67.3 for gender and see that for all of them the correct value should be “female”. Thus we would simply convert all 67.3 values to “female”. The nice thing about this is that we have effectively gained those data points back for ML training rather than just throwing them away. You can do such a conversion in pandas like this:
value_map = {'male': 'male', 'female': 'female', '67.3': 'female'}pd_dataframe['gender'].map(value_map)
Not all features are equal. Some things you might not even need! For example, maybe you’re looking at a dataset of books purchased from Amazon over the past year where one of the feature variables is called “font-type” indicating the type of font used in the book. This is pretty irrelevant to predicting the sales of a book! You can probably safely drop this feature all together like so:
df.drop('feature_variable_name', axis=1)
Doing so makes your data exploration easier as you aren’t looking through as much stuff. It also helps make training your ML model easier and faster as you are not processing as much data. If you’re unsure whether or not the variable will be important then you can always wait until you start exploring the dataset to decide. Computing the correlation matrix between feature variables and the target output may help in determining how important the variable is.
All of your data in each feature variable should be in the same standardised format. It’ll make the technical aspects of your data exploration and modelling much easier. For example, let’s take the example of the “gender” variable again with values “male” or “female”. If the data was collected by a human you might get many different values which you did not expect:
male, female (this one’s good)
MALE, FEMALE (entered with caps lock on)
Male, Female (some people will capitalise)
Make, Femall (typos!)
If we were to just go ahead and convert the feature variable to categorical floats, we’d get more than the 2 values of 0 and 1 that we want! We’d actually get something like this:
{    'male': 0,    'female': 1,    'MALE': 2,    'FEMALE': 3,    'Male': 4,    'Female': 5,    'Make': 6,    'Femall': 7}
There are a couple of ways to handle such a case. If it’s something simple that making something all lower case or capitalising the first letter, just do something like this:
# Make the whole string lower cases.lower()# Make the first letter capitaliseds.capitalize()
If there are typos then you’ll want to use the mapping function that we saw before:
value_map = {'Make': 'male', 'Femall': 'female'}pd_dataframe['gender'].map(value_map)
Follow me on twitter where I post all about the latest and greatest AI, Technology, and Science!