Data Quality at Airbnb
By: Vaughn Quoss
Tags: Analytics Engineering, Data, Data Engineering, Data Warehouse, Data Science

Introduction

At Airbnb, we’ve always had a data-driven culture. We’ve assembled top-notch data science and engineering teams, built industry-leading data infrastructure, and launched numerous successful open source projects, including Apache Airflow and Apache Superset. Meanwhile, Airbnb has transitioned from a startup moving at light speed to a mature organization with thousands of employees. During this transformation, Airbnb experienced the typical growth challenges that most companies do, including those that affect the data warehouse.

In the first post of this series, we shared an overview of how we evolved our organization and technology standards to address the data quality challenges faced during hyper growth. In this post we’ll focus on Midas, the initiative we developed as a mechanism to unite the company behind a shared “gold standard” that serves as a guarantee of data quality at Airbnb.

Defining the Gold Standard

As Airbnb’s business grew over the years, the company’s data warehouse expanded significantly. As the scale of our data assets and the size of the teams developing and maintaining them grew, it became a challenge to enforce a consistent set of standards for data quality and reliability across the company. In 2019, an internal customer survey revealed that Airbnb’s data scientists were finding it increasingly difficult to navigate the growing warehouse and had trouble identifying which data sources met the high quality bar required for their work.

This was recognized as a key opportunity to define a consistent “gold standard” for data quality at Airbnb.

A Multi-dimensional Challenge

While all stakeholders agreed that data quality was important, employee definitions of “data quality” encompassed a constellation of different issues. These included:

Accuracy: Is the data correct?

Is the data correct? Consistency: Is everybody looking at the same data?

Is everybody looking at the same data? Usability: Is data easy to access?

Is data easy to access? Timeliness: Is data refreshed on time, and on the right cadence?

Is data refreshed on time, and on the right cadence? Cost Efficiency: Are we spending on data efficiently?

Are we spending on data efficiently? Availability: Do we have all the data we need?

The scope of the problem meant that standards focused on individual data quality components would have limited impact. To make real headway, we needed an ambitious, comprehensive plan to standardize data quality expectations across multiple dimensions. As work began, we named our initiative Midas, in recognition of the golden touch we hoped to apply to Airbnb’s data.

End-to-end Data Quality

In addition to addressing multiple dimensions of data quality, we recognized that the standard needed to be applicable to all commonly consumed data assets, with end-to-end coverage of all data inputs and outputs. In particular, improving the quality of data warehouse tables was not sufficient, since that covered only a subset of data assets and workflows.

Many employees at Airbnb will never directly query a data warehouse table, yet use data on a daily basis. Regardless of function or expertise, data users of all types are accustomed to viewing data through the lens of metrics, an abstraction which does not require familiarity with the underlying data sources. For a data quality guarantee to be relevant for many of the most important data use cases, we needed to guarantee quality for both data tables and the individual metrics derived from them.

In Airbnb’s data architecture, metrics are defined in Minerva — a service that enables each metric to be uniquely defined in a single place — and broadly accessed across company data tools. A metric defined in Minerva can be directly accessed in company dashboarding tools, our experimentation and A/B testing framework, anomaly detection and lineage tools, our ML training feature repository, and for ad-hoc analysis using internal R and Python libraries.

For example, take Active Listings, a top-line metric used to measure Airbnb’s listing supply. An executive looking up the number of Active Listings in a Apache Superset dashboard, a data scientist analyzing the Active Listings conversion funnel in R, and an engineer reviewing how an experiment affected Active Listings in our internal experiment framework will all be relying on identical units for their analysis. When you analyze a metric across any of Airbnb’s suite of data tools, you can be sure you are looking at the same numbers as everybody else.

In Airbnb’s offline data architecture, there is a single source of truth for each metric definition shared across the company. This key architectural feature made it possible for Midas to guarantee end-to-end data quality, covering both data warehouse tables and the metric definitions derived from them.

The Midas Promise

To build data to meet consistent quality standards, we created a certification process. The goal of certification was to make a single, straightforward promise to end users: “Midas Certified” data represents the gold standard for data quality.

In order to make this claim, the certification process needed to collectively address the multiple dimensions of data quality, guaranteeing each of the following:

Accuracy: certified data is fully validated for accuracy, with exhaustive one-off checks of all historical data, and ongoing automated checks built into the production pipelines.

certified data is fully validated for accuracy, with exhaustive one-off checks of all historical data, and ongoing automated checks built into the production pipelines. Consistency : certified data and metrics represent the single source of truth for key business concepts across all teams and stakeholders at the company.

: certified data and metrics represent the single source of truth for key business concepts across all teams and stakeholders at the company. Timeliness: certified data has landing time SLAs, backed by a central incident management process.

certified data has landing time SLAs, backed by a central incident management process. Cost Efficiency: certified data pipelines follow data engineering best practices that optimize storage and compute costs.

certified data pipelines follow data engineering best practices that optimize storage and compute costs. Usability: certified data is clearly labeled in internal tools, and supported by extensive documentation of definitions and computation logic.

certified data is clearly labeled in internal tools, and supported by extensive documentation of definitions and computation logic. Availability: certification is mandatory for important company data.

As a last step, once data was certified, that status needed to be clearly communicated to internal end users. Partnering with our analytics tools team, we ensured data that was “Midas Certified” would be clearly identified through badging and context within our internal data tools.

Fig 1: Midas badging next to metric and table names in Dataportal, Airbnb’s data discovery tool.

Fig 2: Midas badging for metrics in Airbnb’s Experimentation Reporting Framework (ERF).

Fig 3: Pop-up with Midas context in Airbnb’s internal data tools.

The comprehensive Midas quality guarantee, coupled with clear identification of certified data across Airbnb’s internal tools, became our big bet to guarantee access to high quality data across the company.

The Midas Certification Process

The certification process we developed consists of nine steps, summarized in the figure below.