Simplify your Dataset Cleaning with Pandas | by Ulysse Petit | Towards Data Science

I’ve heard a lot of analysts/data scientists saying they spend most of their time cleaning data. You’ve probably seen a lot of tutorials to clean your dataset but you probably know that already: it will never be 100% clean and you have to understand that point before continuing to read this article. So I’ll be honest with you: I won’t give you the magic recipe to get rid of all the data issues you might have with your dataset. The cleaning rules depend on the domain you are working on and the context of your project. The examples of this article come from my own experience with data manipulation in the real world.
I’ve dealt with all the issues/processes that I’m detailing in this article. The problem can come from the data source itself at times and you have to clean it, sometimes it’s just your colleague or your manager who requests some specific fields in the final file. Feel free to share the main issues you’ve seen from your experience in the comments.
You’ll need the latest Python release: 3.7+. I recommend using the Anaconda distribution to get Python, Pandas, and Jupyter. If you already have Anaconda installed, ignore the two following commands.
pip install seaborn
pip install pandas
pip install jupyter
Let’s start with a basic dataset I’ve found on Kaggle. This dataset is a search results report for “Flights” and “Tickets” queries searched on Google. When you search for a query on Google, the “organic” results will be ranked in the search result page and only 10 are listed for each query in this dataset. As you can guess, we might expect duplicates in some fields.
We will cover the following topics:
Remove useless charactersExtract relevant content from a SeriesCheck NaN valuesChange the type of your Series
Remove useless characters
Extract relevant content from a Series
Check NaN values
Change the type of your Series
Open a new Jupyter notebook and import the dataset:
import osimport pandas as pddf = pd.read_csv('flights_tickets_serp2018-12-16.csv')
We can check quickly how the dataset looks like with the 3 magic functions:
.info(): Shows the rows count and the types
df.info()
.describe(): Shows the main statistics for every numerical column in our dataset
df.describe()
.head(): Shows the first 5 rows of the dataset (you can change this number by passing an integer as a parameter)
df.head()
We can even show the column names because we have too many and they don’t fit in the screen:
df.columns
Now we have a better view of what we are dealing with. Starting from this point, depending on the context of your company and your objectives, you won’t be looking for the same thing. Moreover, if this dataset will be used to feed a Machine Learning algorithm for training or if you need to run an analysis for your manger, your output DataFrame won’t look the same. But it’s not the subject here. Let’s continue with the first topic.
Why should we remove characters in a dataset full of data? Well, a dataset is not always built for your personal use-case but for multiple uses in general. Again, depending on your project, your focus will be different.
In the case where you are working on an NLP (=Natural Language Processing) project, you will need to get your text very clean and get rid of the unusual characters that don’t change the meaning for instance. If we look closer at the “title” column,
df['title']
we can see a few special characters to remove like: , . ( ) [ ] + | -
If you want to be safe, you can use a complete list of special characters and remove them using a loop:
spec_chars = ["!",'"',"#","%","&","'","(",")",              "*","+",",","-",".","/",":",";","<",              "=",">","?","@","[","\\","]","^","_",              "`","{","|","}","~","–"]for char in spec_chars:    df['title'] = df['title'].str.replace(char, ' ')
Now you shouldn’t have any of those characters in your title column. Because we replaced the special characters with a plus, we might end up with double whitespaces in some titles.
Let’s remove them by splitting each title using whitespaces and re-joining the words again using join.
df['title'] = df['title'].str.split().str.join(" ")
We’re done with this column, we removed the special characters. Note that I didn’t include the currencies characters and the dot “.” in the special characters list above. The reason is that some results titles contain the price of the flights tickets they are selling (e.g. “$514.98”) and this information might be interesting to extract in the next section.
What is important when you work with a dataset is to make the dataset cleaner, readable. Your dataset might end up being ingested in a Machine Learning pipeline, that being said, the idea is to extract a lot of information from this dataset. I’ll show an example but the granularity of the information you will extract depends on your objectives.
Let’s say you want to extract all the prices in dollars from the results titles (i.e. the title column). We use a regex function to do that. You can do some tests with your regex function here if you want.
df['dollar_prices'] = df['title'].str.extract('(\$\.d*\.?\d*)')
We’ve now extracted all the prices in dollars in the title column.
We now display the unique values of the resulting column:
df['dollar_prices'].unique()
As you can see and it was expected, we have some NaN (=Not a Number) values (4th position in the array above). Some titles don’t have a dollar price so the regex rule couldn’t find it, instead, we have “nan”. We’ll see in the next section how to deal with the NaN values.
Most of the time, a big dataset will contain NaN values. What to do with them? Should we leave them? Absolutely not! You can’t leave them because of the calculations you might perform on numerical columns or just because of future modeling a Data Science team or even you could do. For those who know about Machine Learning, more or less, you know that your features/variables/predictors should be numerical before training a model. For all these reasons, we want to fix the NaN values. I won’t spend too much time dealing with the different strategies, there are plenty of articles explaining different strategies. What you can do:
Remove the corresponding rows: This can be done only if removing the rows doesn’t impact the distributions in your dataset or if they are not significant. If there are just 3 rows with some NaN values in your 1M dataset, it should be safe to remove the rows. Otherwise, you might have to go with the next options.
Use statistics to replace them (in numerical columns): You can replace the NaN values by the mean of the column. Be careful, you don’t want to skew the data. It might be more relevant to look at another column like a categorical one and replace the NaN values based on the mean for each category. Let me explain. In the Titanic dataset, for instance, each row represents a passenger. Some values in the Fares column are missing (NaN). In order to replace these NaN with a more accurate value, closer to the reality: you can, for example, replace them by the mean of the Fares of the rows for the same ticket type. You assume by doing this that people who bought the same ticket type paid roughly the same price, which makes sense.
Else: Try to know well your domain, your context which will help you to know how to replace in a safe way the missing variables. If you won’t use this dataset in a Machine Learning pipeline, you can replace the NaN with another value like a string (in case of a string type column) “None” or “NoData”, anything relevant for you and the people who will use the dataset.
For more examples about dealing with the NaN values, I recommend reading an article with a focus on that particular point.
About our dataset, let’s see what is the proportion of NaN values using a visualization library called seaborn. You can do it with Pandas functions directly but I think it’s good to go first with a visual way so you get to know your data:
sns.heatmap(df.isnull(), cmap='viridis')
In this heatmap, you can see in yellow (depending on the cmap you are using, but with mine it’s yellow) the NaN values in each column. It doesn’t mean that a column with no yellow doesn’t have any NaN, it happens to have some at the top of the heatmap or at the bottom so they are confused with the graphical borders of the heatmap (it’s due to the resolution of the image printed in the notebook). Anyway, don’t hesitate to show the raw data using the .isnull() Pandas function.
In our case, we will just keep the rows where we have a dollar_price in the title so we will just keep the rows with a value in the dollar_price column:
df = df.dropna(subset=['dollar_prices'])
The resulting DataFrame contains only the rows where dollar_price has a value, the others have been dropped.
In this dataset, we might not have a lot of type changes to do. It depends on your context and the quality of the dataset. If you’re happy with the column types as we saw with df.info(), that’s fine.
The input file might come from a colleague of yours who saved the data into an Excel file (.xlsx). In that case, and that’s because Excel is terrible for this, the format of your columns could be changed especially for your Ids columns. By “Ids” I mean a column that represents an Id for one of your listed entities. For instance, if you’re manipulating a dataset containing an inventory of products, you’ll probably have a ProductId column in it. Anyway, numerical Ids in Excel can be interpreted as a numerical column or text column and to be honest with you I don’t know when Excel chooses one or the other.
For your data manipulation, it depends on how you want to treat Ids in your script, do you want to manipulate text (string) or numerical values (integer)? That’s up to you, but be consistent in your script, if you need to join two DataFrames, based on an Id column, for instance, convert them first in the same format (string or integer but you have to choose one).
Let’s go back to our dataset. In the second section, we created a new column containing the prices in dollars. At this point, we know there are in dollars, right? Because this information about the currency is in the column name. That being said, we can get rid of the dollar characters:
df['dollar_prices'] = df['dollar_prices'].str.replace('$', '')
The last step is to check the type of this column. We should have a string as we extracted the data with the regex rule as strings:
df.dtypes
Here our dollar_prices is an object that means a string actually. Do we really want to consider our prices as strings? We will prefer to use floats for the prices, we just need to convert the column type.
df['dollar_prices'] = pd.to_numeric(df['dollar_prices'], errors='coerce')
If you show the dtypes one more time, you’ll notice that the dollar_prices column is no longer an object but it’s now a float64 type.
I hope you found some answers to the questions you might have asked yourself. As I said in the beginning, there is no universal way to clean a dataset, some checks have to be done like the NaN values for instance. The rest really depends on what level of quality do you need for the project? What data are relevant to extract?
Also, keep in mind that everything you can extract from your dataset (section 2) might be really helpful for visualization specialists who will build dashboards using your dataset as a source.
The Jupyter notebook is available in my GitHub just here.
Don’t hesitate to write comments and questions.