How Automated Data Validation using Pandera Made Me More Productive! | by Pratik Gandhi | Towards Data Science

Data is called as the new oil of 21st Century. It is very important to juggle with the data to extract and use the correct information to solve our problems. Working with data can be exciting and sometimes tedious for people. As it is rightly said, “Data Scientists spend 80% of their time cleaning the data”. Being a part of that pack, I go through the same process when encountering a new dataset. The same activity is not limited to until the Machine Learning(ML) system is implemented and deployed to production. When generating predictions in real time, the data might change due to unintuitive and unforseeable circumstances like error due to human interference, wrong data submitted, a new trend in data, problem while recording data, and many more. A simple ML system with multiple steps involved looks like shown in the diagram below:
This needs to be slightly shifted by introducing or labeling another component explicitly, after data preparation and before feature engineering we name as Data Validation:
The article is focused on why data validation is important and how can one use different strategies to seemlessly integrate it in their pipeline. After some work, I learnt how to implement scripts that would do data validation to save some of the time. Above that, I automated them using some of the pre-built packages, stepping up my game!
Almost 85% of projects will not make it to production as per Gartner. Machine Learning (ML) Pipelines usually face several hiccups when pushed in production. One of the major issues I have quite often experienced is the compromise of data quality. Spending multiple hours of a day, several times a month maybe, and figuring out that the data that came through was unacceptable because of some reason can be quite relieving but frustrating at the same time. Many reasons can contribute that leads to data type getting changed like, text getting introduced instead of an integer, an integer was on outlier (probably 10 times higher) or an entire specific column was not received in the data feed, to mention a few. That is why adding this extra step is so important. Validating manually can take some extra effort and time. Making it automated(to an extent) could reduce the burden of the Data Science team. There are some major benefits I see by integrating an automated data validation in the pipeline:
Saves time by a couple of hours
Less frustration among team members
More productive by focusing on other areas
Trying to make your production model more accurate ;)
Python community has already built some outstanding packages to take care of these issues in a very smart way. I am going to share a few of them which I have explored and how I engineered my scripts to make them part of my pipeline.
Data Source: I took a very recent dataset to show how one can implement these packages. Here is the link to get the data.
Pandera provides a very simple and flexible API for performing data validation for data frame or series data. It also helps in performing complex statistical validation like hypothesis testing like two_sample_ttest. More details about the package and usage can be found here.
Take a sample of 10 rows from the dataset:
As we can see, there are multiple columns of different data types: String, Int, Float, Datetime. We need to define the schema of the data we expect. I have defined a simple schema without any strict rules for data validation checks as seen in the code above. Based on the expected data type, we can either use pa.Int for Integer, pa.String for String, pa.Float for float or pa.DateTime if there is a date and time. Here we set nullable = True if we expect NaN values else False. Moreover, if we set coerce = True, it will convert the column into the expected datatype without giving any errors. For e.g. converting float to int, int to string, etc. On validating the data, the output will the data frame itself, as it passes all the checks (try it!!!). If it fails, the output would look something like:
Traceback (most recent call last):...ValueError: cannot convert float NaN to integer
There are some amazing capabilities that I have explored about Pandera which I will try to explain with as much detail as I can.
Sometimes, columns might be optional and not required. In such a case, we can set required = False. By default required = True for all columns
As we see in the screenshot below, it returned data as output without any errors.
Try setting required=True for the same column and run the above code again.
By default, if a new column is added to the data frame and not defined in the schema, no error will be raised. However, if you want to you can set strict = True.
Running the code above we see below that it throws the errors that ‘extra_column’ is not defined in the schema.
If you see my code, I have added lazy = True which gives a more detailed view as follows:
Index can also be validated if there is any specific pattern or holds any special importance in the dataset. For example, see below:
I created and set that column as an index to show how we can do Index validation. Outside the parenthesis of the schema, one can specify the index format that is expected. Again the output is the same as before
Here we performed a single index validation. MultiIndex validation can be performed in a similar way. Check out here.
Once we have defined the schema to validate the data, it is possible to transform the schema in the pipeline itself, after new columns have been introduced because of additional computation (which happens 100% of the time). Let’s say initially we only have Id, County, Province_State, Country_Region, Population as the columns. We add new columns Weight, Date, Target, TargetValue to the schema as follows:
On printing the schema as we show above, we get the following:
On the other hand, we can also remove some columns from the schema if needed as below:
One can do a more detailed check on the column using Check object as follows:
Pandera has some pre-built checks that can be directly used like greater_than_or_equal_to, less_than. A custom check can also be passed for e.g. here we have used lambda argument to calculate the length of the string. This is one of the best functionalities in Pandera and can bring a lot more value to the data validation strategy. Find more details here
I create separate scripts that run and gives a report of the validation, once the final data has been received before it goes to the next step. People can integrate them as a part of pre-processing as well.
One can also use them as decorators directly. See how here.
Apart from using directly on a data frame, it can also be applied to a series, a specific column check, or even an element-wise check.
Because of the flexibility regex operations can also be carried out on the data using Pandera.
Importantly, you would not have a perfect data validation schema right in first go. More than likely, you will make data validation schema “perfect” and ML pipeline more robust by incorporating different validation strategies as you encounter data in real time, over a long course of period.
As the name of the package suggests, you can set expectations for the data to be validated. Honestly, I got so comfortable with Pandera, that I have not got a chance to explore to the full potential. It seems to be quite easy to implement and straight forward package to use. Below is a small snippet of the implementation of the same data:
This is another package whose syntax is similar to Pandera. It can be more applied when data is in a dictionary form or JSON format.
More details about the package can be found here.
This is another great package if your data is obtained via XML, JSON or HTML form post. In other words, it is beneficial to validate any type of strings, mappings, and lists type of data. Below are some of their useful links:
Github: https://github.com/Pylons/colander
Documentation: https://docs.pylonsproject.org/projects/colander/en/latest/
JsonSchema is an implementation of JSON schema for Python. As the name suggests, this package is very helpful for validating JSON data. Take the following example with valid JSON data and invalid JSON data:
On running the invalid data, it will throw an error:
As we can see, the price is expected to be number but string was provided rather. More details about it can be found here.
The post might seem to be mostly dominated by Pandera, as it is one of the packages that I use often. Writing this post was purely to share the voyage I embarked on. I am sure one can benefit more from these packages for their specific use case. Often walking the extra mile can achieve extraordinary results. Data validation is definitely that “extra mile”. Developing data validation strategies might feel extra work but one can definitely benefit a lot if such a habit of creating automated scripts is formed, even for taking care of small things.
Please, feel free to share your comments and suggestions if you have any!