Treating Most Common Data Uniformity Problems with Pandas | Towards Data Science

In my previous articles on data cleaning, I talked about how out-of-range values, incorrect data types, uniqueness constraints are common problems and that when left untouched can seriously compromise our analysis.
In this article, we are going to tackle a problem that is just as dangerous and that can similarly skew our analysis which is unit uniformity.
Problems related to uniformity can come in many shapes:
temperature data in both Fahrenheit and Celsius
weight given in both kilograms and pounds
multiple date formats
money represented in different currencies
Such problems arise because of many reasons. For example, data is collected across many locations and merged together into one. This can result in the above problems. It is imperative that you know how to handle uniformity problems for accurate analysis.
people.head()
The notebook and data using in this article can be found on this Kaggle Notebook or on this GitHub repo.
Consider this temperature dataset imported earlier. It gives information about daily average temperatures for 3 months in Celsius:
temperatures.sample(10)
If we look at index 48, unless a major climate event occurred, we can guess that values are given in Fahrenheit:
When you are in such situations, use a scatter plot to spot them visually:
A scatter plot clearly shows the values which are Fahrenheit. A quick Google search shows us the formula for converting Fahrenheit to Celsius:
Now, we will use this equation to convert any values over 40 to Celsius:
We can check that the conversion is successful in 2 ways. First, with assert statement:
Again using a scatter plot:
Even though this was a simple example, your case may be completely different. Sometimes, you might not even notice the inconsistencies in the sea of data. Before looking for them, it is essential that you completely understand your data and learn how it was collected.
One other very common problem is date uniformity. Different countries have different standard DateTime formats and when you have data from multiple sources, you are going to have to deal with multiple date formats. In Python, there are 3 main DateTime formats you can work with:
25–12–2019 -> (%d-%m-%Y) -> day, month, year
October 31, 2020 -> (%c) -> literal dates
12–25–2019 -> (%m-%d-%Y) -> month, day, year
When you load data into your environment with pandas, it always imports dates as object data type. In the Setup section, we imported sample data containing people's full name and birthday:
people.sample(5)
The birthday column represents the dates as YY-MM-DD format but as strings. Converting date columns to datetime has a number of advantages to perform analysis on. We will use pd.to_datetime() function to convert the column into datetime:
We got an error! If we look closer, the error says that month must be in 1, 2, ..., 12. That means, somewhere in the data, there is an inconsistent value(s) that is preventing the function to run. pd.to_datetime has a good workaround for such cases:
This time, we did not get any errors. If you set infer_datetime_format to True, pandas will automatically identify the date format based on the first non-NaN element and converts the rest to that format. If there are any values that do not fit the conversion, errors parameter decides what to do with them. If set to coerce, pandas will put NaTs which are missing values for dates.
Based on our assumptions, the above line of code should have spotted date inconsistencies and put NaTs to inconsistent dates:
There are a few methods to handle such inconsistencies:
convert to NA and treat accordingly
infer format by looking at the data source, how it was collected
infer format by looking at other dates
If you have many inconsistencies, converting them to NaN is not always an option. You should try to come up with custom solutions by looking at their patterns and inferring their formats.
After you are done with error-handling and conversions, it is best practice to convert date columns to global DateTime standard:
Read more articles related to the topic: