Machine Learning in Production using Apache Airflow
By: Danylo Baibak
Tags: Data Science, Data Engineering, Data Pipeline, Machine Learning, Airflow

2. Data Validation

Data Validation is the process of ensuring that data is present, correct, and meaningful. Ensuring the quality of your data through automated validation checks is a critical step in building data pipelines at any organization.

The data validation step is required before model training to decide whether you could train the model or stop the execution of the pipeline. This decision is automatically made if the following was identified by the pipeline [2]:

Data schema skews : these skews are considered anomalies in the input data, which means that the downstream pipeline steps, including data processing and model training, receive data that doesn’t comply with the expected schema. Schema skews include receiving unexpected features, not receiving all the expected features, or receiving features with unexpected values.

: these skews are considered anomalies in the input data, which means that the downstream pipeline steps, including data processing and model training, receive data that doesn’t comply with the expected schema. Schema skews include receiving unexpected features, not receiving all the expected features, or receiving features with unexpected values. Data values skew: these skews are significant changes in the statistical properties of data, which means that data patterns are significantly changed, and you need to check the nature of these changes.

Importance of the data validation:

The quality of the model depends on the quality of the data.

Increasing confidence in data quality by quantifying data quality.

Correcting the trained ML model can be expensive — prevent is better than cure.

Stopping training a new ML model if the data is invalid.

Airflow provides a group of check operators, that allows us easy verify data quality. Let’s look at how to use such Operators on practical examples.

2.1 CheckOperator

The CheckOperator expects a SQL query that will return a single row. Each value on that first row is evaluated using python bool casting. If any of the values return False the check is failed and errors out. So, simply added one task to the pipeline we can check if data exists for instance for a specific date.

2.2 IntervalCheckOperator

For more complex checks Airflow has IntervalCheckOperator. The Checks that the values of metrics given as SQL expressions are within a certain tolerance of the ones from “days_back” before.

One of the key-point for this check is ratio formula — which formula to use to compute the ratio between the two metrics. You can choose from two possible:

max_over_min: max(cur, ref) / min(cur, ref)

relative_diff : computes abs(cur-ref) / ref

IntervalCheckOperator allows you to check different metrics at the same time with the different ratios for some specific table.

2.3 ValueCheckOperator

The ValueCheckOperator — a simple and powerful operator. It performs a pure value check using SQL code. You can use a SQL query of any complexity to get any value. The check will pass if the computed value is equal to the passed value with some tolerance.

Using the check operators from Airflow, it is easier to understand where is a problem when you see that a check is failed, instead of dealing with error in the code.

The list of data checks at the beginning of the pipeline

In the current example, I use all checks to test input data. But you can also check any output values. Also, it will be useful to add the checks to ETL pipelines.