Cross field validation with Pandas. Final Blows to the Dirty Data. | Towards Data Science

Today, data never comes from a single source. More often than not, it is collected in different locations and merged together. A common challenge when merging data is data integrity. In simple words, making sure our data is correct by using multiple fields to check the validity of another. In fancier terms, this process is called Cross Field Validation.
Sanity checking your dataset for data integrity is essential to have accurate analysis and running machine learning models. Cross field validation should come in after you dealt with most of the other cleaning issues like missing value imputation, ensuring field constraints are in place, etc.
I wrote the code snippets for this post with regard to execution time. Since cross-field validation may involve performing operations across multiple columns for millions of observations, the execution speed should is very important. The solutions suggested here should be scalable enough for even massive datasets.
I generated fake data to perform cross-field validation:
In the setup section, we loaded a fake people dataset. For example purposes, we are going to assume that this data was collected from two sources: from census data that gives each individual's full name and birthday which was merged with their hospital records later.
To have an accurate analysis, we should make sure our data is valid. In this case, we can check the validness of two fields: age and BMI (Body Mass Index).
Let’s start with age. We have to ensure when we subtract their birth year from the current year, the result matches the age column.
When you are performing cross-field validation, the speed should be the main concern. Unlike our little example, you may have to deal with millions of observations. One of the fastest methods for cross-field validation for datasets of any size is apply function of pandas.
Here is a simple example of apply:
The above was an example of a column-wise execution. apply takes a function name as an argument and calls that function on each element of the column it was called on. The function has an extra argument axis which is by default set to 0 or rows. If we set it to 1 or columns, the function shifts to row-wise execution.
Note on axis argument: axis='rows' means performing an operation along the row axis which is vertical because rows are stacked vertically. axis='columns' means performing an operation along the column axis which is horizontal because columns are stacked horizontally. These two terms confuse a lot of people because they look like they are doing the opposite of what they are told to. In reality, it just takes a shift in perspective, or using the words if you will.
Let’s create a function that validates a person’s age:
Since we are going to use apply for a row-wise operation, its inputs will be each row of the dataset. That's why we can easily access each column's value just like we would in a normal situation.
Using the datetime package we imported earlier, we will store today's date. Then, we calculate the age by subtracting the year components from each other. For this to work, you have to make sure the birthday column has a datetime data type.
In the return statement we compare the calculated age and the given age, which returns True if they match, False if otherwise:
The function works as expected. Now we can subset the data for invalid ages if any:
people[people['age_valid'] == False]
There were 75 rows with invalid age. If you do the math, the ages do not match. To correct these values, we could write a new function but that would involve a code repetition. We can update validate_age to replace any invalid values with valid ones:
We can make sure the operation was successful with an assert statement:
Next, we will validate the Body Mass Index column.
Body mass index is a value derived from the mass and height of a person.
A quick Google search gives us the formula to calculate BMI:
Using the ideas from the first example, we will create the function for BMI which replaces invalid BMI with correct values:
It is a good practice to start all of your validation functions with validate_. This gives a signal you are performing a validation to the readers of your code.
In this section, we will perform a speed comparison of different methods for cross-field validation. We will start off with the apply function for validating the bmi:
It took around 0.3 seconds for the 10k dataset. Next, we will use a for loop using pandas iterrows():
It took 10 times longer! Besides, this time difference will not be linear. For larger datasets, the difference becomes bigger and bigger. I don’t think any for loop can beat the apply function, but let's also try itertuples which is generally faster than iterrows():
Still much slower than apply. So, the general rule of thumb for cross-field validation is to always use apply function.
Read more articles related to the topic: