How Airbnb Built “Wall” to prevent data bugs
By: Subrata Biswas
Tags: Data, Data Engineering, Data Quality, Airflow

As shared in our Data Quality Initiative post, Airbnb has embarked on a project of massive scale to ensure trustworthy data across the company. To enable employees to make faster decisions with data and provide better support for business metric monitoring, we introduced Midas, an analytical data certification process that certifies all important metrics and data sets. As part of that process, we made robust data quality checks and anomaly detection mandatory requirements to prevent data bugs propagating through the data warehouse. We also created guidelines on which specific data quality checks need to be implemented as part of the data model certification process. Adding data quality checks in the pipeline has become a standard practice in our data engineering workflow, and has helped us detect many critical data quality issues earlier in the pipelines.

In this blog post we will outline the challenges we faced while adding a massive number of data checks (i.e. data quality, accuracy, completeness and anomaly checks) to prevent data bugs company-wide, and how that motivated us to build a new framework to easily add data checks at scale.

Challenges

When we first introduced the Midas analytical data certification process, we created recommendations on what kind of data quality checks need to be added, but we did not enforce how they were to be implemented. As a result, each data engineering team adopted their own approach, which presented the following challenges:

1. Multiple approaches to add data checks

In Airbnb’s analytical data ecosystem, we use Apache Airflow to schedule ETL jobs or data pipelines. Hive SQL, Spark SQL, Scala Spark, PySpark and Presto are widely used as different execution engines. However, because teams started building similar data quality checks in different execution engines, we encountered other inherent issues:

We did not have any centralized way to view the data check coverage across teams.

A change in data check guidelines would require changes in multiple places in the codebase across the company.

Future-proof implementations were nearly impossible to scale. Teams kept re-inventing the wheel and duplicated code spread across the codebase.

2. Redundant efforts

Different teams often needed to build tools to meet their own requirements for different data checks. Each Data Engineering (DE) team started to build data check tools in silos. Although each of these teams were building solid tools to meet their individual business needs, this approach was problematic for a few reasons:

We started to build multiple frameworks in parallel.

Data check frameworks became costly to maintain and introduced operational overhead.

Missing features and lack of flexibility/extensibility made these frameworks difficult to reuse across the company.

3. Complicated Airflow DAG code

Each check was added as a separate task in Airflow as part of the ETL pipeline. Airflow DAG files soon became massive. The operational overhead for these checks grew to the point that it became hard to maintain, because of a few different factors: