Data Cleaning
By: Chaquayla Halmon
Tags: Data Visualization, Data Cleaning, Python, Data Science, Jupyter Notebook

Data Cleaning

I believe that data cleaning is an essential part to being a data scientist. One of the few challenges I’ve faced is dealing with unnecessary data. I had to deal with duplicates, columns not needed for analysis, and datatype issues. Let’s look at how to handle some.

Unnecessary columns:

Looking at data from a movie dataset I was able to combine three datasets that had everything I needed and a few extras. Here’s a look at the original data frame versus the new data frame:

Before the cleaning

After cleaning

I used info() to look at all the columns information, then I used the drop() to get rid of the extra columns I didn’t need. That way you’re not scrolling through useless information and you can find information with ease. That being said know what you need before you check something off for deletion.

Converting Datatypes:

Sometimes you need to do some math with different information, but you can’t aggregate an object datatype. So what do you do? Staying with the dataframe above you can see from info() production budget, domestic gross, and worldwide gross are all object datatype. So if I want to find the profit or foreign gross I have to change the datatype to an integer. Let’s look at these columns:

Before cleaning

So the best way to convert is to first get rid of the thing making it an object: ($) and (,). We have to convert them to string with (str.) and replace with a nothing(“”), then change the datatype. So it would look something like:

df[col].str.replace(“$”, “”).str.replace(“,”,””).astype(“int64”)

We take that and apply it to each column. A for loop can help with that. Now we have column that can used in mathematical situations.

After cleaning

Duplicates:

There’s nothing worse then thinking you have a dataframe with no missing information until you start to plot values and see weird results. What causes them? Duplicates hiding in not so plain sight. What do we do? Back to my awesome movie dataset, let’s play with genres. Everybody has a favorite, so what does the world say:

Before cleaning

WOW! People REALLY love drama. That seems like a lot compared to the others. Did someone stuff the box just to crown Drama as Queen. Let’s see if there’s something going on(hint: there is). You can use a combination isin() and duplicated() to check a specific column for duplicates. Graph below: movies = df[‘column’], then, df[movies.isin(movies[movies.duplicated()])].sort_values(‘column’)

All the duplicates

Then I use drop_duplicates() to get rid of all the duplicates from this dataframe. On default it only keeps the first and drop all the other duplicates. You can change it if you like. Then go back to the original dataframe and use originaldf.drop_duplicates(subset=[‘column’], inplace=True) so it drops the duplicates here as well. Use the duplicated() and any() to check for duplicates again. See below:

Coding to drop duplicates

Now let’s look at our genres ratings now.

After cleaning

There! All the genres had a change, some more significant then others. This is what happens when you merge different datasets. A movie can get recorded with different genres depending on where you go. You have to be willing to go the extra mile and clean the data.

This is why as a data scientist it is important to supply our client with the most accurate and clean data as possible. It’s the only way to make recommendations that you feel best represent what they want. It might take a while but the results can be worth it if done properly. Think of it this way. If you want to buy something online the first thing you do is look up the details. If information is missing, you wont buy it or if the details are skewed you’ll end up with an outfit that fits a Barbie doll. Think about that.